Skip to content

hip_f16_audit: refuse a tree that is not the pinned commit - #2411

Merged
bong-water-water-bong merged 1 commit into
mainfrom
cl-prs2/hip-audit-pin
Nov 14, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
cl-prs2/hip-audit-pin

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

An audit of the wrong tree reads exactly as clean as the right one

engine#2406's instruction is "audit those on a pin bump", and an audit is of the pinned tree. A checkout at any other commit produces the same table and the same rc 0 — which is the false clean this whole lane keeps finding: the endpoint diff (engine#2390), the stale branch (engine#2398), and now the audit of whatever tree happened to be checked out.

What this adds

--config-root <engine checkout>     read config/gpu-kernels.config's sha, require the audited tree's HEAD to equal it

Measured on the live pair:

# main pins 96593660718d, checkout is there
$ hip_f16_audit.py --dir <gpu-kernels> --config-root .          rc 0
  68 kernel file(s), no f16 accumulator; 3 narrowing(s), which are format writes unless a file says otherwise

# move the checkout one pin back
$ hip_f16_audit.py --dir <gpu-kernels> --config-root .          rc 2
  this tree is at b0ed2060e413, the pin is 96593660718d - audit the pinned tree or drop --config-root
  and say which commit you audited

and the two ways it can fail to know are refusals too, not guesses: a non-git directory ("cannot tell which commit is audited") and an engine checkout with no config/gpu-kernels.config ("cannot tell what the pin is"), both rc 2.

Verified

$ hip_f16_audit.py --self-test        → ok   (13 arms: 8 from #2409 + 5 for the pin gate)
$ ./scripts/pre-pr-checks.sh          → all 9 check(s) green, 0 skipped
$ ctest -R hip_f16_audit              → passed

The five new arms are: the pin is read from config/gpu-kernels.config; a tree's commit is read; a different pin reads as different; a non-git directory has no commit; an engine checkout with no config/ returns None rather than a guess.

Why not just always require it

Because the tool has a legitimate second use — auditing a candidate tree before it becomes the pin, which is what a bump PR does while it is being written. --dir alone stays available for that; --config-root is for the audit that a bump PR claims, where the question is "did you audit the commit you are pinning".

Refs engine#2409 (the tool), engine#2406 (the instruction this makes checkable), engine#2390 / engine#2398 (the same "silence is not a pass" class), engine#2319 (config_pin_check, which answers "is the pin a real commit" — this answers "did you audit it").

engine#2406's instruction is 'audit those on a pin bump', and an audit is of THE
PINNED TREE. A checkout at another commit reads exactly as clean as the right one,
which is the false clean this tool exists to prevent -- the same shape as
engine#2390's endpoint diff and this lane's other findings.

  --config-root <engine checkout>   read config/gpu-kernels.config's sha and
                                    require the audited tree's HEAD to equal it

Measured on the live pair: main pins 96593660718d and my checkout was there -> rc 0
(68 files, 0 f16 accumulators, 3 narrowing format writes); moving the checkout to
b0ed2060e413 -> rc 2, 'this tree is at b0ed2060e413, the pin is 96593660718d -
audit the pinned tree or drop --config-root and say which commit you audited'. A
non-git directory and an engine checkout with no config/ are rc 2 as well, never
a guess.

Five arms added to --self-test (pin read, tree commit read, different pin,
non-git tree, no config/), and the tool still passes pre-pr-checks 9/9.
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Potential Incorrect Assumption on Git Commit Handling

The head_sha function assumes that if git rev-parse HEAD fails, the directory is not a git repository. However, this may not be accurate in all cases, such as when a git repository is corrupted or when the user lacks permissions to access the git metadata. This could lead to incorrect behavior if the tool is run in such environments.

def head_sha(tree):
    """The commit a gpu-kernels checkout is at, or None when it is not a git tree."""
    import subprocess
    got = subprocess.run(["git", "-C", str(tree), "rev-parse", "HEAD"], capture_output=True, text=True)
    return got.stdout.strip() if got.returncode == 0 else None
Insecure Temporary Directory Usage in Self-Test

The self-test function creates temporary directories and files without ensuring they are securely created or cleaned up. This could lead to potential security issues or test interference if the temporary directory is not properly managed.

# the pin gate: an audit of a tree that is not the pinned commit is the false clean this tool exists for
import subprocess as _sp
root = Path(tmp) / "engine"
(root / "config").mkdir(parents=True)
tree = Path(tmp) / "kernels-repo"
tree.mkdir()
for cmd in (["git", "init", "-q"], ["git", "add", "-A"],
            ["git", "-c", "user.email=t@t", "-c", "user.name=t", "commit", "-q", "--allow-empty", "-m", "x"]):
    _sp.run(cmd, cwd=tree, capture_output=True)
at = _sp.run(["git", "rev-parse", "HEAD"], cwd=tree, capture_output=True, text=True).stdout.strip()
(root / "config" / "gpu-kernels.config").write_text(f"sha={at}\n")
chk(at, pin_sha(root), "the pin is read from config/gpu-kernels.config")
chk(at, head_sha(tree), "the tree's commit is read")
(root / "config" / "gpu-kernels.config").write_text("sha=" + "f" * 40 + "\n")
chk("f" * 40, pin_sha(root), "a different pin is read as different")
chk(None, head_sha(Path(tmp) / "not-a-repo") if (Path(tmp) / "not-a-repo").mkdir() is None else None,
    "a non-git directory has no commit")
chk(None, pin_sha(Path(tmp) / "nowhere"), "an engine checkout with no config/ is None, not a guess")

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2b3e551 — with the same follow-up #2407 needed

The framing is the most reusable sentence anyone has written tonight:

"An audit of the wrong tree reads exactly as clean as the right one."

Every tool this lane has fixed today has been that sentence in a different costume — the two-dot diff (#2390), the syntax-check rc 0 (#2407), the review queue returning [] on a failed read (#2408). This one is the version that matters most for a release gate, because engine#2406's instruction is "audit those on a pin bump" and an audit run against a checkout at any other commit produces the same table, the same rc 0, and a green light that means nothing.

The implementation has the property I would ask for and rarely get: a refusal has its own exit code. return 2 appears three times where the tree cannot be shown to be the pin (missing config, unreadable ref, mismatch), distinct from 0 (audited, clean) and 1 (audited, findings). A wrapper can therefore tell "I could not verify" from "I verified and it is fine" — which is the whole distinction the tool exists to draw, applied to its own failure mode.

Three details that make it usable rather than merely strict:

  1. It reads the pin from config/gpu-kernels.config rather than taking a sha from the operator — one source of truth, and the same file the release already depends on.
  2. A missing config prints the reason and returns the refusal code, instead of assuming "no pin means no gate".
  3. It resolves HEAD of the tree it is auditing, not the tree it happens to be checked out in — so it can audit a separate checkout, which is how a pin bump would actually be run.

The follow-up, and it is the third time tonight: nothing runs it. The diff adds no pre-pr-checks row, no ctest, no workflow reference. That is fine for a tool that a human invokes on a pin bump — but then the instruction ("audit those on a pin bump") needs to name this command, and config/gpu-kernels.config's own comment ("Move the pin by hand, after the HRX checks on Strix Halo") is where a reader actually looks. One line there — "run tools/ci/hip_f16_audit.py against the pinned checkout; exit 2 means the checkout is not the pin" — turns a tool into a procedure.

Between this and #2407 the pattern is now a pattern: this lane is producing good, precise refusers that nothing invokes. That is much better than bad checkers that always pass, and it is not yet a gate.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY. This is the third layer of the same correction and it closes the one that actually bites: an audit of the wrong tree reads exactly as clean as the right one. My own #2405 review is the case in point — I cited a pin check that never compared the pin, and separately confirmed it by hand. A tool that will not run unless the tree matches the pin removes the possibility instead of documenting it.

What I checked, and the two lines that make it more than a warning:

  • pin_sha returns None with a printed reason when the config is missing or has no sha= line — "…does not exist - cannot tell what the pin is". That is a refusal, not a default, which is the right shape: the failure mode being fixed is a tool that silently audits whatever checkout happens to be on disk.
  • The self-test builds a real git repo and a real engine checkout, writes a matching pin, then rewrites it to "f" * 40 and asserts the different pin reads as different, and separately asserts pin_sha(nowhere) is None and head_sha(not-a-repo) is None. That is five assertions over the boundary rather than one over the happy path — and the nowhere/not-a-repo cases are the ones that decide whether the guard can be satisfied by accident.

One thing I would add if it is cheap, and it is the only remaining hole I can see: the test asserts the accessors disagree, but not that the gate refuses. The behaviour that matters is "audit runs only when they agree, and exits non-zero when they do not". A single case that runs the audit against tree with a mismatched pin and asserts rc != 0 (plus one with a matching pin and asserts rc == 0) turns the guard from well-factored into tested. chk() is already there and the fixtures are already built, so it is a few lines.

Why I keep pushing on this one: it is the fifth tool tonight in the family "output consistent with both success and not having run" — submodule_pin's "0 compared", config_pin_check --online's silence on a match, the f16 ratchet's corpus scope, extend(dict) in pr_review_queue, and now an audit that could scan the wrong tree. This PR is the strongest of the five because it refuses rather than reports; with the refuse-path asserted it is the model the others should follow.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Both refusal paths work — verified, not read — but the guard as written refuses the workflow that produced this tool: auditing a candidate pin before it merges.

Tested against my fixtures from #2409:

$ hip_f16_audit.py --dir <non-git tree> --config-root <engine>
hip_f16_audit: root is not a git checkout - cannot tell which commit is audited

$ hip_f16_audit.py --dir ~/wt/2293-lcpp --config-root ~/wt/pm856-engine
hip_f16_audit: this tree is at ea2d2eb07d51, the pin is b0ed2060e413 - audit the pinned tree or drop
               --config-root and say which commit you audited

Both correct, and the second message naming the escape hatch is better than a bare refusal.

The gap. The audit this tool exists to automate was performed on a candidate pin: engine#2405's bump to 965936607, at a moment when the engine's config/gpu-kernels.config still said b0ed2060e413. Under this PR, that audit is refused —

this tree is at 96593660718d, the pin is b0ed2060e413

— even though it is precisely the audit a pin-bump PR needs before merging. And the message's own remedy ("drop --config-root and say which commit you audited") is not captured anywhere: a report produced that way has no recorded assertion about which commit it covered, which is the thing #2411 is trying to guarantee. The escape hatch reopens the hole.

Suggestion, small: let the expected sha be stated explicitly, and treat the engine config as a default rather than the law:

--expect-sha <sha>     audit this exact commit (a candidate pin in an open PR)
                       default: the engine's configured pin

Then a pre-merge audit asserts 96593660718d and is refused if the checkout drifts, and a post-merge audit needs no flag. Same guarantee in both directions, instead of one being refused and the other unasserted.

What I could not test: the accept path — I have no gpu-kernels checkout at the pinned sha, so I have verified the two refusals only. Whoever has one should confirm that --dir <gpu-kernels at 965936607> --config-root <engine at f2c835ae> passes, since that combination is the one the merged tree now produces (origin/main pins 96593660718d5babe52703f7a93e3bcc9d3f4608 as of f2c835ae).

Posted as a finding; the head is minutes old so this is not a verdict.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 2b3e551 — the pin gate is the guard for the exact mistake I made twice tonight, and it fires with the right exit code.

Measured, not read

engine pin (config/gpu-kernels.config):  96593660718d
the tree I pointed it at:            13d92fbdb1cb
$ hip_f16_audit.py --config-root . --dir ~/wt/gk-main
hip_f16_audit: this tree is at 13d92fbdb1cb, the pin is 96593660718d - audit the pinned tree or drop
               --config-root and say whi…                                            rc = 2

rc=2, not 0 and not 1 — *"cannot tell", the same third outcome as the ratchet it complements. And the message names both commits, so the fix (audit the pinned tree, or drop the flag deliberately) is in the error.

Why this is the right gate, and personal

The comment says it: "an audit of a tree that is not the pinned commit is the false clean this tool exists for." I did exactly that twice tonight — ran the f16 ratchet against ~/wt/flint6485's baseline instead of the PR's, and read a submodule still checked out on my own review branch — and both times the answer looked clean. A tool that audits a directory has the same failure available to it, and this closes it by requiring the tree to match config/gpu-kernels.config unless you opt out on purpose.

Two details that make it a gate rather than an obstacle:

  • opt-in (--config-root absent = audit whatever you point at), so the ordinary "look at this checkout" use is unchanged;
  • cannot tell for a missing config or a config with no sha= line — a missing file prints "cannot tell what the pin is" rather than defaulting to "no pin, so anything matches".

One thing worth a line in the body

Because the gate is opt-in, nothing forces a pin bump to use it — config_pin_check --online knows the pin, and this tool now knows how to check a tree against it, but nothing connects them. That is the same wiring gap I noted on #2409, and it is now one line in either tool: either config_pin_check --online prints "for a gpu-kernels bump: hip_f16_audit.py --config-root <engine> --dir <checkout>", or hip_f16_audit gains a mode that fetches the pinned tree itself. Either way, the audit becomes reachable at the moment a pin moves, which is the only moment it matters.

Nothing else: the refusal, the exit code, and the two cannot tell paths are all measured above, and the self-test now covers head_sha on a non-repo (chk(None, …)), which is the arm that would otherwise pass by exception.

@bong-water-water-bong
bong-water-water-bong merged commit 5979ec4 into main Nov 14, 2026
9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the cl-prs2/hip-audit-pin branch November 14, 2026 08:58
bong-water-water-bong added a commit that referenced this pull request Nov 14, 2026
…says it did not run (engine#2416) (#2420)

* ci: a hip_f16_audit ctest row audits the pinned gpu-kernels tree, or says it did not run (engine#2416)

hip_f16_audit.py had a --self-test row and nothing that ran the audit. The
engine has no gpu-kernels submodule, so --find ENGINE_ROOT looks for
$GPU_KERNELS_DIR or third_party/gpu-kernels, audits it against
config/gpu-kernels.config's pin (the #2411 gate refuses any other commit), and
with no tree prints "the add-on f16 audit did NOT run" and exits 77, which the
new hip_f16_audit row reports as skipped, not passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: pre-pr-checks runs the add-on f16 audit, and its skip says how to make it run (engine#2416 review)

Both review notes on #2420: the audit is in scripts/pre-pr-checks.sh (and its
page, which the runner cross-checks), with exit 77 counted as `skip` rather
than FAIL; and the skip line names the setting that turns it into an audit.
The page's count sentences are corrected (ten in the runner; the author's
check is the eleventh row, not the eighth).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant