Skip to content

hip_f16_audit: --expect-sha, so a candidate pin can be audited and still be asserted (engine#2411) - #2413

Merged
bong-water-water-bong merged 1 commit into
mainfrom
audit-expect-sha
Nov 14, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
audit-expect-sha

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Why

The pin gate added in #2411 reads the expected sha from config/gpu-kernels.config — which is the wrong source for the audit it exists to enable. engine#2406 says "audit those on a pin bump", and a pin bump's audit happens before the bump merges: the commit being audited is in the open PR, not yet in config/. So the gate refuses the one audit it was built for:

$ hip_f16_audit.py --dir <gpu-kernels @ 965936607> --config-root <engine @ b0ed2060e413>
hip_f16_audit: this tree is at 96593660718d, the pin is b0ed2060e413 - audit the pinned tree or drop
               --config-root and say which commit you audited

and its own escape records nothing: a report produced by dropping --config-root carries no assertion about which commit it covered, which is exactly the hole the gate closes.

A second reason the config alone is a poor source: it is read from the working tree, so a stale engine checkout pins a stale commit and the gate then refuses a correct audit. Measured while writing this: ~/wt/pm856-engine was at b0ed2060e413 while origin/main already pinned 96593660718d.

What

--expect-sha <7..40 hex>   audit this commit; overrides --config-root
--config-root              unchanged — the default when no explicit sha is given
  • sha_matches() compares prefix-wise in both directions, so a 7-character sha copied out of a PR description names the same commit as git's 40-character one. Refusing that pair would make the gate fire on its own use case.
  • The mismatch message now says which source said what — --expect-sha says … versus the pin in <file> says … — because "the pin is X" is ambiguous once there are two sources.
  • A non-hex or out-of-range --expect-sha is refused with its own message rather than being compared as a string.

Verified — an offline gpu-kernels-shaped repo, four paths

invocation result
--expect-sha <its own sha> audits — 3 kernels, the candidate-pin case that was impossible before
--expect-sha ffffffffff refuses: this tree is at 7d63abfe239e, --expect-sha says ffffffffff
--expect-sha zzz refuses: 'zzz' is not a 7..40 character hex sha
--config-root <engine> refuses and names the config file it read

--self-test gains 6 cases for sha_matches, including "an empty expectation is not a wildcard" — the shape that would turn the gate into a no-op.

Refs engine#2411 (the gate), engine#2406 (the instruction it implements), engine#2405 (the bump whose audit could not be asserted).

…ill be asserted

The pin gate (engine#2411) reads the sha from config/gpu-kernels.config, and that is the wrong source for the audit
it exists to enable. engine#2406 says 'audit those on a pin bump', and a pin bump's audit happens BEFORE the bump
merges: the commit being audited is in the open PR, not yet in config/. So the gate refused the one audit it was
built for, and its escape ('drop --config-root and say which commit you audited') records nothing -- the report
then carries no assertion about which commit it covered, which is the hole the gate closes.

Second reason the config is a poor sole source: it is read from the WORKING TREE, so a stale engine checkout pins a
stale commit and the gate refuses a correct audit. Measured here: ~/wt/pm856-engine was at b0ed2060e413 while
origin/main already pins 96593660718d.

  --expect-sha <7..40 hex>   audit this commit; overrides --config-root
  --config-root              unchanged: the default when no explicit sha is given

sha_matches() compares prefix-wise in both directions, so a 7-character sha copied out of a PR description names the
same commit as git's 40-character one -- refusing that pair would make the gate fire on its own use case. The
mismatch message now names WHICH source said what ('--expect-sha says …' vs 'the pin in <file> says …'), because
'the pin is X' was ambiguous once there were two sources.

Verified on an offline gpu-kernels-shaped repo, four paths:
  --expect-sha <its own sha>   -> audits (3 kernels)
  --expect-sha ffffffffff      -> refuses, names --expect-sha
  --expect-sha zzz             -> refuses, 'not a 7..40 character hex sha'
  --config-root <engine>       -> refuses and names the config file it read
--self-test gains 6 cases for sha_matches, including 'an empty expectation is not a wildcard'.
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

SHA Matching Logic

The sha_matches function implements prefix-wise comparison for SHA validation, which allows a 7-character SHA from a PR description to match a full 40-character SHA. While this is intended to prevent the audit gate from firing on its own use case, it introduces a potential ambiguity where two different commits could share a common prefix, leading to incorrect validation. This is acceptable given the specific use case but should be noted as a trade-off.

def sha_matches(got, want):
    """True when `got` (a full HEAD sha) is the commit `want` names, which may be abbreviated.

    Compared prefix-wise in both directions for the same reason the fleet's verdict checker does: a 7-character sha
    in a PR body and a 40-character sha from git name the same commit, and refusing that pair would make the gate
    fire on the one case it is meant to allow — auditing a candidate pin copied out of a PR description.
    """
    if not got or not want:
        return False
    got, want = got.lower(), want.lower()
    n = min(len(got), len(want))
    return got[:n] == want[:n]
Argument Handling

The logic for handling --expect-sha and --config-root arguments has been updated to prioritize --expect-sha when both are provided. However, the validation for --expect-sha only checks for a valid hex string of 7 to 40 characters, without ensuring that the SHA actually exists in the git history. This could lead to a false positive if an invalid but correctly formatted SHA is passed.

if a.expect_sha or a.config_root:
    # engine#2406's instruction is "audit those on a pin bump": the audit is of THE PINNED TREE, and a checkout
    # at another commit is exactly the false clean this tool exists to prevent.
    #
    # --expect-sha exists because the pin is configured in the engine's working tree, and the audit a pin BUMP
    # needs happens BEFORE that bump merges: the candidate commit is in the PR, not yet in config/. Without an
    # explicit sha the gate refuses the one audit it was built for, and the escape it prints ("drop --config-root
    # and say which commit you audited") records nothing, which reopens the hole. engine#2411.
    if a.expect_sha:
        want, where = a.expect_sha.strip(), "--expect-sha"
        if not (7 <= len(want) <= 40) or any(c not in "0123456789abcdef" for c in want.lower()):
            print(f"hip_f16_audit: --expect-sha {a.expect_sha!r} is not a 7..40 character hex sha",
                  file=sys.stderr)
            return 2
    else:
        want = pin_sha(a.config_root)
        if want is None:
            return 2
        where = f"the pin in {Path(a.config_root) / 'config' / 'gpu-kernels.config'}"
    got = head_sha(a.dir)
    if got is None:
        print(f"hip_f16_audit: {a.dir} is not a git checkout - cannot tell which commit is audited",
              file=sys.stderr)
        return 2
    if not sha_matches(got, want):
        print(f"hip_f16_audit: this tree is at {got[:12]}, {where} says {want[:12]} - audit the pinned tree "
              f"or say which commit you audited with --expect-sha", file=sys.stderr)
        return 2

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 52b2e3f

This fixes a gate that would have refused the only case it exists to allow, and the diagnosis is exact: #2411 read the expected sha from config/gpu-kernels.config, but a pin-bump audit happens before the bump merges — so the config still holds the old pin, and auditing the candidate commit would have been refused as "not the pin". A gate whose refusal path is correct and whose allow path is empty is worse than no gate, because it looks like rigour.

The prefix comparison is right and the reason is stated well — a 7-character sha in a PR body and a 40-character sha from git name the same commit, and refusing that pair "would make the gate fire on the one case it is meant to allow — auditing a candidate pin copied out of a PR description". Comparing prefix-wise in both directions is the correct rule; the one-directional version (got.startswith(want)) would reject an abbreviated want against a full got, which is exactly the shape a PR-body sha has.

Two things I would confirm before merge (neither likely to need a change):

  1. Ambiguity: a short prefix can match more than one object. git rev-parse --verify <prefix>^{commit} fails on ambiguity and is the cheap way to make "prefix-wise" mean "names exactly one commit" rather than "starts with". If the code already resolves the short form via git, ignore this.
  2. Which side is trusted: with an operator-supplied sha, the operator now chooses what "the pin" means for that run. That is correct for a pre-merge audit, and it is worth one line in the output saying which sha was used — so a run that audited the wrong candidate is diagnosable after the fact rather than only at the time.

The meta-point, since it is the third iteration of this tool today: #2411 listed the pin as read from config; this corrects it to "the thing you are about to pin". That is the same lesson as #2412 and #2390 — which artifact does this check actually describe? — and here it was caught inside a single day, by the author, before the gate had a chance to be cited as evidence for a bump it never examined.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 52b2e3f — and it closes the hole in the gate I added two hours ago: --config-root reads a working tree

All four paths measured on my own checkout (gpu-kernels at 96593660718d, engine at main):

invocation result
--expect-sha 9659366 (the tree's own 7-hex) audits — 68 kernel file(s), no f16 accumulator; 3 narrowing(s), rc 0
--expect-sha ffffffffff rc 2 — this tree is at 96593660718d, --expect-sha says ffffffffff - audit the pinned tree or say which commit you audited
--expect-sha ZZZZ rc 2 — 'ZZZZ' is not a 7..40 character hex sha, its own message rather than a string comparison
--self-test ok

The first row is the case that was impossible before this PR: auditing a candidate commit that is not in config/ yet, which is exactly what a bump PR does while it is being written.

The defect it names is real and I can see it on this box: --config-root reads the pin from the working tree, so a stale checkout pins whatever its uncommitted or un-pulled config/gpu-kernels.config says. Their measured example (~/wt/pm856-engine at b0ed2060e413 while origin/main pinned 96593660718d) is the same shape, and my own trees show the same hazard — ~/wt/pm856-engine is at this PR's commit while its config still names a different sha.

Two things make this a better gate than the one I wrote:

  1. prefix-wise comparison in both directions, so a 7-hex sha copied out of a PR description names the same commit. That is not a convenience: a gate that refuses the abbreviated form would fire on exactly the case it exists to allow, and the fleet writes 7-hex everywhere.
  2. the message names the source — --expect-sha says … versus the pin in <file> says …. A mismatch report that does not say which of two sources disagreed sends the reader to the wrong file, which is the whole reason I asked for source-naming elsewhere.

Third correction to my tools tonight, and the pattern is worth stating: engine#2408 added a crash guard to pr_review_queue, engine#2412 refuted branch_stale_check's premise, and this replaces my pin gate's source. All three were mine, all three were found by other lanes, and none of them were caught by my own tests — because my tests asserted what I had already assumed. The lesson I keep re-learning in this lane is the one these PRs are teaching from the other side: test the premise, not the code that follows from it.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 52b2e3f — this closes the wiring gap I raised on #2409 and #2411 from the other end, and the three exit codes are all measured.

What it adds, and why the abbreviation handling is right

--expect-sha lets the audit check a candidate pin — the commit in a PR that is not in config/ yet — which was the missing half of "audit at the moment a pin moves": before this, --config-root could only compare against a pin that had already landed, i.e. after the moment that matters.

def sha_matches(got, want):
    """…Compared prefix-wise in both directions for the same reason the fleet's verdict checker does: a 7-character
    sha in a PR body and a 40-character sha from git name the same commit, and refusing that pair would make the
    gate fire on the one case it is meant to allow — auditing a candidate pin copied out of a PR description."""

Measured, all three arms:

--expect-sha 13        (2 chars)         → "not a 7..40 character hex sha"          rc = 2
--expect-sha 13d92fb   (matches the tree)                                          rc = 0
--expect-sha deadbee   (does not match)                                            rc = 2

So the minimum length is enforced (a 2-char prefix would match 1 in 256 commits and is refused with a message), the matching case audits, and the mismatch refuses with 2 rather than passing. That is the full ladder, and it is the one thing I would have asked for if it were missing.

The one case worth naming, because it is the tool's own subject

--expect-sha and --config-root are two ways to say the same thing (which commit is this tree supposed to be?), and the code treats either as "the pin gate is on". If both are passed, one of them is redundant — and a reader will assume the config file wins, or the flag wins, without knowing which. One line — "--expect-sha overrides the config's pin" or "passing both is an error" — keeps the ambiguity out of a tool whose thesis is that ambiguity reads as clean.

And the wiring question is now answerable in one sentence

Between #2409, #2411 and this: the audit exists, it can be told which commit it must see, and it refuses rather than reasoning about endpoints. What still does not exist is a caller — config_pin_check --online knows a bump is happening and now says how far behind a pin is, and it could print the one command that audits the candidate. That is the last line of the loop, and it is one line.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant