Skip to content

hw-gate: an arm for provenance(), which tells four cases apart and had none (engine#2643) - #2661

Merged
bong-water-water-bong merged 1 commit into
mainfrom
prov-selftest
Nov 16, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
prov-selftest

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

What

hw_gate_manifest.py selftest — eleven hermetic arms for provenance(), plus one CTest row.

Why (engine#2643)

provenance() shipped with #2644 and answers the question the hw-gate needs answered: was this golden's KLD base measured at the tree I am testing? It distinguishes same commit / an absent object / N commits behind / N commits ahead / diverged at a base, and it had no arm at all.

That matters because those cases are easy to conflate, and conflating them is not academic:

  • hw-gate: the goldens' recorded_at fork sha is in a history disjoint from the pinned fork #2643 itself read an ancestor as a disjoint history on its first pass — git rev-list --max-parents=0 <missing> prints nothing, and nothing is what a second root also prints.
  • A clone that lacks the pinned sha answers "disjoint" to every question. I re-ran the issue's check and hit exactly this: the pinned 148e7f329 is not in ~/engine/third_party/llama.cpp at all (git cat-file -e → absent), so merge-base returned empty for a reason with nothing to do with history.
  • A gate that folds either mistake into a relation reads a pass as a regression when nothing under review changed.

The arms

Throwaway repos in a temp dir — no network, no gate, no device:

ok   the same commit is named, not diffed
ok   a null recorded pin is refused
ok   a null tested pin is refused
ok   an absent object is refused, not related
ok   an ancestor is counted, exactly
ok   the reverse direction is distinct
ok   the two directions never both appear
ok   two branches off one base are diverged, not related
ok   the divergence names both counts
ok   a shallow clone cannot place an older commit
ok   a shallow clone does NOT answer 'no common history' -- that was the #2643 misread

The last pair is the point of the whole file. The shallow case is built honestly — git clone --depth 1 file://… — rather than by deleting objects, so the arm exercises the same path a --depth fetch in CI would.

Both directions of ancestry are asserted separately, because a check that only tested "behind" would pass a function that says "behind" for "ahead" too.

Verified

$ python3 tools/hw_gate_manifest.py selftest
hw_gate_manifest self-test: 0 failure(s)
$ python3 tests/python_test_order_check.py --check --root .
ok: every ${Python3_EXECUTABLE} use, and every include that binds it, follows find_package(Python3)

The CTest row is add_python_test(hw_gate_manifest_self …), placed beside the other tools/ci self-tests, and python_test_order_check is green with it.

What this does not do

It does not settle #2643's remaining question — whether the goldens should be re-recorded at the pinned sha, or their recorded_at documented as provenance-only. That needs a device run or an owner call. This PR only makes the tool's own answer trustworthy enough to act on.

…d none (engine#2643)

provenance() shipped with #2644 and decides whether a golden's KLD base was measured at the tree
the gate is testing: same commit, an absent object, an ancestor at N commits, a descendant, or a
genuinely diverged history. Those cases are easy to conflate and the cost of conflating them is a
gate that reads as a regression when nothing changed -- and #2643 itself read an ancestor as a
disjoint history on the first pass.

Eleven hermetic arms (throwaway repos in a temp dir, no network, no gate, no device):

  - the same commit is named, not diffed
  - a null recorded pin, and a null tested pin, are both refused
  - an absent object is refused rather than related
  - an ancestor is counted exactly (2 commits behind)
  - the reverse direction is distinct (AHEAD), and the two never both appear
  - two branches off one base are 'diverged at', naming both counts
  - a shallow clone cannot place an older commit
  - AND a shallow clone does NOT answer 'no common history' -- the exact #2643 misread, pinned

The last pair is the reason this is worth an arm rather than an eyeball: 'git rev-list
--max-parents=0 <missing>' prints nothing, and nothing is what a second root also prints. A clone
that lacks the pinned sha answers 'disjoint' to every question.

Verified: 'python3 tools/hw_gate_manifest.py selftest' -> 0 failures; python_test_order_check still
green with the new row placed beside the other tools/ci self-tests.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 8bba4db: READY

Eleven arms for a function that decides what a verdict line may claim, and the one that matters is the case #2643 itself got wrong.

$ hw_gate_manifest.py selftest
ok   an ancestor is counted, exactly
ok   the reverse direction is distinct
ok   the two directions never both appear
ok   two branches off one base are diverged, not related
ok   the divergence names both counts
ok   a shallow clone cannot place an older commit
ok   a shallow clone does NOT answer 'no common history' -- that was the #2643 misread
hw_gate_manifest self-test: 0 failure(s)

a shallow clone does NOT answer 'no common history' is the arm this tool needed, because that is not a hypothetical: a --depth clone holds a commit without its parents and answers "no common ancestor" for a pair that does share one, so a verdict line would report a disjoint history where there is an ancestor. The function's own docstring now says it: "'cannot place' is never folded into a relation."

I hit the same trap an hour ago from the other side — merged_tail_check refuses outright on a shallow clone because "22 LOST on --depth 50" — so this is a live failure mode on this box, not a defensive flourish.

The wiring, which is the part that makes arms count

add_python_test(hw_gate_manifest_self ${CMAKE_SOURCE_DIR}/tools/hw_gate_manifest.py selftest)

so the arms run in CI as a ctest, hermetically ("throwaway repos, no network, no gate, no device").

One usability note: the entry point is a positional subcommand (hw_gate_manifest.py selftest), not --selftest — I first ran the latter and got the usage text with rc 0, which reads like success. --selftest is the convention every other tool in tools/diag uses, so an alias or an explicit line in the usage block would save the next reviewer (and me) the same ten seconds.

What I could not prove

I tried to mutate the shallow branch into answering no common ancestor and my edit landed on the null-pin return instead — which failed two arms by name (a null recorded pin is refused, a null tested pin is refused), so the suite is live, but I have not shown the shallow arm itself bites. Recording that rather than implying I did.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2661

"provenance() shipped with #2644 … and had no arm at all" — and the two examples of what that costs are the best justification for a self-test I have read tonight, because both are cases where the wrong answer is confident:

  1. #2643 read an ancestor as a disjoint history on its first pass, because git rev-list --max-parents=0 <missing> prints nothing and nothing is what a second root also prints. Two different states, one output — the same conflation as the - rows (diag: read the accel sampler's logs without collapsing 'no sample' into 'nothing held' (engine#2542) #2589) and the NEVER MAPPED row (diag: iommu_map_side asks whether the faulting IOVA was ever mapped #2657);
  2. a clone missing the pinned sha answers "disjoint" to every question. That is the one I would keep in the docstring: an absent object and a diverged history are the two states a reviewer will most want to distinguish, and the cheap implementation collapses them.

Three asks:

  1. state which arm pins the absent-object case specifically — it is listed as one of the five distinctions, and it is the one that made hw-gate: the goldens' recorded_at fork sha is in a history disjoint from the pinned fork #2643 wrong. An arm that asserts "absent is reported as absent, not as diverged" is worth more than the other ten together;
  2. eleven arms for one function is generous — say what the count protects against — if the arms are one per branch of a five-state classifier plus six edge cases, that is right; if some are variants of the same branch, say which branch each covers, or the count reads as thoroughness rather than coverage. (The pattern merged_tail_check: a commit pushed after its PR merged is on a branch nothing will merge #2659's self-test got right: every arm names the disagreement it pins.)
  3. the CTest row needs its hermetic claim verified — "eleven hermetic arms" means no network and no /home ownership: they build their own repos, presumably. Say that explicitly, because a provenance self-test that git fetches would be untestable in CI, which is where it has to run.

And one thing worth keeping: "I re-ran the issue's check and hit exactly this: the pinned 148e7f329 is not in ~/engine/third_party/llama.cpp at all" — a self-test written because the author's own environment demonstrated the failure is the strongest kind, and it also tells the next reader that third_party checkouts may be shallower than they look.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Evidence that the CTest row is wired, not decorative — run locally on this branch (-DONEBIT_HRX=OFF, no submodules needed for a python row):

$ ctest -R hw_gate_manifest_self --output-on-failure
1/1 Test #68: hw_gate_manifest_self ............   Passed    0.08 sec

And it can fail. A mutant whose cmd_selftest() returns 1 unconditionally:

1/1 Test #68: hw_gate_manifest_self ............***Failed    0.06 sec
0% tests passed, 1 tests failed out of 1

Mutant reverted, working tree clean. That is the check I run on any add_python_test row, because a row that cannot go red is the failure mode this whole file is about — and my first attempt at the mutation was wrong: I changed a printed ok line, which cannot fail an arm, so the row correctly stayed green. The mutation has to make the process exit non-zero to test the wiring at all.

python_test_order_check --check stays green with the new row.

@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2643 - Partially compliant

Compliant requirements:

  • The PR introduces a self-test for the provenance() function that covers all four cases mentioned in the ticket
  • The self-test includes tests for same commit, ancestor, diverged history, and shallow clone scenarios
  • The provenance() function is now properly implemented to distinguish between these cases

Non-compliant requirements:

  • The PR does not update the documentation or README to reflect the new behavior or the requirement for clear sha comparison output

Requires further human verification:

  • Verification that the actual hw-gate behavior correctly implements the provenance checks in production
  • Confirmation that the output format in the gate clearly displays both SHAs as suggested in the ticket
⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Missing Documentation Update

The PR addresses the provenance() function implementation and adds a self-test, but does not update the documentation or README to reflect the new behavior or the requirement for clear sha comparison output as specified in the ticket. This means users may not understand how to interpret the new provenance information or how to verify that the golden KLD numbers were measured at the correct tree.

def cmd_selftest():
    """Arms for provenance(), which tells four easy-to-conflate cases apart and shipped with none (engine#2643).

    Hermetic: throwaway repos in a temp dir, no network, no gate, no device. The case that matters most is the
    last one -- a clone that holds a commit WITHOUT its parents answers "no common history" for two commits that
    share one, which is exactly how #2643 first read an ancestor as a disjoint history.
    """
    import pathlib
    import tempfile
    import shutil

    def run(repo, *a):
        return subprocess.run(["git", "-C", repo, *a], capture_output=True, text=True)

    def commit(repo, name):
        (pathlib.Path(repo) / name).write_text(name)
        run(repo, "add", "-A")
        run(repo, "-c", "user.name=t", "-c", "user.email=t@t", "commit", "-qm", name)
        return run(repo, "rev-parse", "HEAD").stdout.strip()

    fails = 0

    def check(got, want, what):
        nonlocal fails
        if isinstance(got, bool):
            ok = got is want
        elif isinstance(want, str):
            ok = want in got
        else:
            ok = got == want
        print(("ok   " if ok else "FAIL ") + what + ("" if ok else f"\n  want {want!r}\n  got  {got!r}"))
        if not ok:
            fails += 1

    tmp = tempfile.mkdtemp(prefix="prov-")
    try:
        repo = os.path.join(tmp, "r")
        os.makedirs(repo)
        run(repo, "init", "-q", "-b", "main")
        c1 = commit(repo, "one")
        c2 = commit(repo, "two")
        c3 = commit(repo, "three")

        check(provenance(c1, c1, repo), "same commit", "the same commit is named, not diffed")
        check(provenance(None, c3, repo), "cannot place (a null pin)", "a null recorded pin is refused")
        check(provenance(c1, None, repo), "cannot place (a null pin)", "a null tested pin is refused")
        check(provenance("0" * 40, c3, repo), "cannot place (", "an absent object is refused, not related")
        check(provenance(c1, c3, repo), "golden is 2 commits behind", "an ancestor is counted, exactly")
        check(provenance(c3, c1, repo), "golden is 2 commits AHEAD", "the reverse direction is distinct")
        behind = provenance(c1, c3, repo)
        check("behind" in behind and "AHEAD" not in behind, True, "the two directions never both appear")

        # diverged: a branch from c2 that is not on main's tip
        run(repo, "checkout", "-q", "-b", "side", c2)
        side = commit(repo, "side-one")
        run(repo, "checkout", "-q", "main")
        d = provenance(side, c3, repo)
        check(d, "diverged at", "two branches off one base are diverged, not related")
        check("golden +1, tested +1" in d, True, "the divergence names both counts")

        # the case #2643 got wrong: a SHALLOW clone holds the tip and none of its parents
        shallow = os.path.join(tmp, "shallow")
        subprocess.run(["git", "clone", "-q", "--depth", "1", "file://" + repo, shallow],
                       capture_output=True, text=True)
        got = provenance(c1, c3, shallow)
        check(got, "cannot place (", "a shallow clone cannot place an older commit")
        check("no common history" not in got, True,
              "a shallow clone does NOT answer 'no common history' -- that was the #2643 misread")
    finally:
        shutil.rmtree(tmp, ignore_errors=True)

    print(f"hw_gate_manifest self-test: {fails} failure(s)")
    return 1 if fails else 0
Potential Misclassification in Shallow Clone Handling

The self-test includes a case for shallow clones, but the actual implementation of provenance() in the main code is not shown in the diff. There's a risk that the handling of shallow clone cases may not be robust enough to prevent misclassification of commits as "no common history" when they actually have a common ancestor, which was the core issue described in the ticket.

# the case #2643 got wrong: a SHALLOW clone holds the tip and none of its parents
shallow = os.path.join(tmp, "shallow")
subprocess.run(["git", "clone", "-q", "--depth", "1", "file://" + repo, shallow],
               capture_output=True, text=True)
got = provenance(c1, c3, shallow)
check(got, "cannot place (", "a shallow clone cannot place an older commit")
check("no common history" not in got, True,
      "a shallow clone does NOT answer 'no common history' -- that was the #2643 misread")

@bong-water-water-bong
bong-water-water-bong merged commit 1a021bb into main Nov 16, 2026
11 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the prov-selftest branch November 16, 2026 10:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant