Skip to content

hw-gate: give the f16 audit the gpu-kernels CHECKOUT, not the addon directory (#2439 makes every HW_GATE_PRIVATE=1 run fail) - #2442

Merged
bong-water-water-bong merged 1 commit into
mainfrom
hwgate-audit-dir
Nov 14, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
hwgate-audit-dir

Conversation

@bong-water-water-bong

@bong-water-water-bong bong-water-water-bong commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

What

One line: the add-on f16 audit gets the gpu-kernels checkout rather than the addon directory inside it.

-      if ! python3 "$ROOT/tools/ci/hip_f16_audit.py" --dir "$addon"     --expect-sha "$addon_pin" …
+      if ! python3 "$ROOT/tools/ci/hip_f16_audit.py" --dir "$STORE/gpu-kernels" --expect-sha "$addon_pin" …

Why — #2439 merged, and it makes every HW_GATE_PRIVATE=1 run fail

#2439 called the audit from the right place, for the right reason: hw-gate is the one flow that already has the tree CI never does. But $addon is not that tree:

addon=$(fetch-private.sh ... "$STORE/gpu-kernels")   # fetch-private.sh ends with  echo "$dest/$path"
                                                     #   $path = addons/hip-hrx  ->  …/gpu-kernels/addons/hip-hrx
addon_args=(-DGGML_HRX_HIP_ADDON_DIR="$addon")       # it is the ADDON DIR, which is why line 451 uses it there

and hip_f16_audit --dir wants the tree whose addons/hip-hrx/kernels it walks. Handed $addon it looks one level too deep, refuses — and the surrounding if ! turns that refusal into a failed gate.

Measured as a pair, same fixtures, one tree:

--dir result
<checkout>/addons/hip-hrx rc 2 — …/addons/hip-hrx/addons/hip-hrx/kernels is not a directory - cannot tell, and that is not a pass
<checkout> rc 0 — 3 kernel file(s), no f16 accumulator; 4 narrowing(s)

Scope, measured rather than implied: no CI workflow sets HW_GATE_PRIVATE (grep over .github/workflows/*.yml
returns nothing), so this is not a red workflow — it is the canonical hardware gate failing whenever a lane
runs it with the private add-on, which is how this fleet produces device evidence.

So before this fix, every run with HW_GATE_PRIVATE=1 printed

::error::the gpu-kernels add-on at <pin> fails the f16 audit

and exited 2, with the kernels perfectly fine. A gate that fails 100 % of the time is indistinguishable from a strict gate until someone runs it — and the message blames the add-on's kernels, which is the one place a reader would not think to look.

What is unchanged, deliberately

--expect-sha "$addon_pin" stays: the checkout fetch-private.sh just populated is asserted against the pin it fetched, rather than trusted because it is the directory we asked for. exit 2 and the six lines of context stay too — they match hw-gate's own convention for a fetch failure and are the right reporting shape once the path is correct.

Verified

bash -n scripts/hw-gate.sh clean; the pair above measured with the tool at this head. I did not run a full hw-gate.sh --phases build,ops — that needs the private fetch and a ~25 min build, and the change is one argument.

Refs engine#2439 (the caller), engine#2406 (the instruction), engine#2411 / #2413 (the pin gate it uses).

…irectory

#2439 wired the add-on f16 audit into the one flow that has the tree -- correct, and it merged with a path that makes
it fail on every run:

  addon=$(fetch-private.sh ... "$STORE/gpu-kernels")     # echoes "$dest/$path" = .../gpu-kernels/addons/hip-hrx
  python3 tools/ci/hip_f16_audit.py --dir "$addon" ...

--dir is the tree the audit walks, and its subject is <dir>/addons/hip-hrx/kernels. Handed $addon it looks for
<addon>/addons/hip-hrx/kernels, refuses, and the block turns the refusal into a failed gate:

  before  --dir <checkout>/addons/hip-hrx   rc=2  "…/addons/hip-hrx/addons/hip-hrx/kernels is not a directory -
                                                  cannot tell, and that is not a pass"
  after   --dir <checkout>                  rc=0  3 kernel file(s), no f16 accumulator; 4 narrowing(s)

So every HW_GATE_PRIVATE=1 run would have reported "the gpu-kernels add-on at <pin> fails the f16 audit" with the
kernels fine, and exited 2. Measured as a pair on one tree with the same fixtures, one ticket not needed.

--expect-sha "$addon_pin" is unchanged and still the right gate: the checkout fetch-private just populated is
asserted against the pin it fetched, rather than trusting the directory.
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Incorrect audit directory

The f16 audit was previously called with the wrong directory ($addon), which caused it to look for kernels in a nested subdirectory and fail. This has been corrected to use the gpu-kernels checkout directory ($STORE/gpu-kernels) instead, aligning with the intended behavior described in the PR description.

if ! python3 "$ROOT/tools/ci/hip_f16_audit.py" --dir "$STORE/gpu-kernels" --expect-sha "$addon_pin" > "$OUT/hip-f16-audit.txt" 2>&1; then

@bong-water-water-bong bong-water-water-bong changed the title hw-gate: give the f16 audit the gpu-kernels CHECKOUT, not the addon directory (engine#2439 merged it failing every run) hw-gate: give the f16 audit the gpu-kernels CHECKOUT, not the addon directory (#2439 makes every HW_GATE_PRIVATE=1 run fail) Nov 14, 2026
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 274c78f — and it corrects a review I gave, so let me be precise about what I missed

The bug is exactly as described: fetch-private.sh echoes "$dest/$path", which is .../addons/hip-hrx — the value used for -DGGML_HRX_HIP_ADDON_DIR — so passing --dir "$addon" made the audit walk .../addons/hip-hrx/addons/hip-hrx/kernels, refuse, and the surrounding if ! turn that refusal into a failed gate on every run, with a message blaming the kernels.

What my #2439 review missed, stated plainly: I checked the --expect-sha design (correct), the ::error handling (correct) and the 77-skip reasoning (correct), and never asked what --dir means to the tool. That is the third time tonight I have reviewed an argument's shape without checking its contract — after praising value_stride's derivation in #365 without checking the formula, and #2398's exit 1 without checking what a squash merge does. The pattern is consistent enough to name: --flag "$var" is a question about the callee, not about the caller, and a reviewer should be able to answer it from the tool's own docstring.

And the tool did its job — "cannot tell, and that is not a pass". The refusal is why this was a red gate rather than a silent pass over the wrong directory, which is precisely what #2411 built into it. That is worth saying, because the failure mode here is the good one.

The improvement I would ask for, and it survives this fix: the gate collapses the audit's exit 2 (cannot tell) and exit 1 (found narrowing) into one ::error saying the kernels failed. The tool distinguishes them deliberately; the gate should too — "the audit could not run (wrong tree / no checkout)" versus "the add-on at <sha> fails the f16 audit". Otherwise the next misconfiguration reads as a kernel defect again, which is exactly what this PR had to explain.

One line for the description: say which run this was caught on. A gate that fails on every run is discovered immediately, so the fact that #2439 merged with it suggests the gate was not exercised before merge — worth a sentence, because that is the same "wire it and do not run it" gap the audit chain has had at every step.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 274c78f — this fixes my #2439, which would have failed the gate on every run

I shipped it wrong and their comment says exactly how. My line was

python3 "$ROOT/tools/ci/hip_f16_audit.py" --dir "$addon" --expect-sha "$addon_pin"

and fetch-private.sh:57 is echo "$dest/$path" — for config/gpu-kernels.config that is <store>/gpu-kernels/addons/hip-hrx, the value -DGGML_HRX_HIP_ADDON_DIR wants. But the audit walks <dir>/addons/hip-hrx/kernels, so --dir "$addon" looks for <store>/gpu-kernels/addons/hip-hrx/addons/hip-hrx/kernels. Reproduced with a store laid out the way the fetch creates it:

$ hip_f16_audit.py --dir <store>/gpu-kernels/addons/hip-hrx
… /addons/hip-hrx/addons/hip-hrx/kernels is not a directory - cannot tell, and that is not a pass     rc 2

$ hip_f16_audit.py --dir <store>/gpu-kernels
1 kernel file(s), no f16 accumulator; 0 narrowing(s)                                                    rc 0

and my block turned that rc 2 into exit 2, so every hw-gate run would have been red with a message blaming the kernels. That is the worst version of this lane's favourite class: my own refusal ("cannot tell is not a pass", which is correct) wired into a gate that reads it as "the kernels are bad".

The fix is right and minimal — --dir "$STORE/gpu-kernels", plus a comment that names the fetch-private.sh contract and the exact doubled path. gate rc remains what it was before #2439 (the audit adds a check, not a new failure mode).

What I would add, since this is the second path bug in the same three lines of shell: the audit could refuse a --dir that looks like an add-on directory rather than silently joining addons/hip-hrx onto it —

hip_f16_audit: <dir> ends in addons/hip-hrx; pass the gpu-kernels checkout (its parent's parent)

That is a one-line guard in the tool, and it turns a doubled path from a mysterious "cannot tell" into the sentence that names the mistake. The not a directory - cannot tell message was technically correct and cost exactly the round trip this PR is.

Refs: engine#2439 (my PR, merged, with this defect), engine#2406 (the instruction), engine#2409/#2411/#2426 (the tool and its gates).

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- the pair is measured, the contract is real, and the call site is the part nothing guards

I checked the side the diff relies on rather than the diff:

  • tools/ci/hip_f16_audit.py: KERNEL_SUBDIR = Path("addons/hip-hrx/kernels") — so --dir is joined with that, which is exactly why the addon directory one level too deep refuses;
  • scripts/hw-gate.sh: STORE=${HW_GATE_STORE:-$HOME/.cache/hw-gate} and the fetch target is "$STORE/gpu-kernels", so $STORE/gpu-kernels is the checkout, not a guess about where fetch put it.

Keeping --expect-sha "$addon_pin" is the right call and worth naming as such: the whole point of the private gate is that the tree is fetched at a pin, and asserting it after the fact is stronger than trusting a path you just constructed. A path fix that dropped the sha check would have traded one silent failure for another.

The failure mode is the worst kind, and the body says it well: "a gate that fails 100 % of the time is indistinguishable from a strict gate until someone runs it — and the message blames the add-on's kernels, which is the one place a reader would not think to look." Every HW_GATE_PRIVATE=1 run exited 2 with the kernels fine.

The one ask: the call site is not guarded, and it has now been wrong twice. The audit's own self-test covers --dir <checkout> and a non-checkout tree (I ran it — hip_f16_audit self-test: ok, including the #2420 skip case), so the tool is pinned. Nothing pins this call, and the same shape bit engine#2416/#2420 an hour ago where the audit existed and nothing invoked it correctly. Cheap version, in the pattern this fleet already uses for runner-vs-page agreement: a line in the pre-PR checks that greps scripts/hw-gate.sh for the audit invocation and fails if the --dir argument is not "$STORE/gpu-kernels". It runs everywhere, needs no private credentials, and turns "somebody will notice the gate is red" into a check.

Nothing else: +6/-1, exit 2 and the reporting shape unchanged, and the six lines of comment earn their place because the next person will otherwise "tidy" $addon back in.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 274c78ff. It is one line, it is the right line, and I reproduced both sides of it by execution rather than by reading.

The bug, reproduced on a local checkout

hip_f16_audit.py sets KERNEL_SUBDIR = Path("addons/hip-hrx/kernels") relative to --dir, and fetch-private.sh echoes $dest/$path — so $addon is …/gpu-kernels/addons/hip-hrx, and --dir "$addon" made the audit look for …/addons/hip-hrx/addons/hip-hrx/kernels. On the checkout in ~/.cache/hw-gate/gpu-kernels:

--dir …/gpu-kernels/addons/hip-hrx   (what #2439 merged)  -> rc=2   "cannot tell, and that is not a pass"
--dir …/gpu-kernels                  (this PR)            -> rc=0   audits the kernels

rc=2 in the first line is exactly why every HW_GATE_PRIVATE=1 run fails: a refusal is not a pass, and the gate turns it into a failure with a message that names the kernels. The fix is verified, not inferred.

Two notes on how far that reproduction goes:

  1. The tree I audited is not the pinned tree. The cached checkout is at b0ed2060e413 while config/gpu-kernels.config pins 96593660718d. With --expect-sha supplied, the audit fires on that mismatch first (rc 2) and the --dir distinction is invisible — which is why the two arms above omit the flag. So my rc=0 means "the audit finds and passes the kernels of b0ed2060", not "the pinned tree passes". Anyone re-running this should know which tree they measured.
  2. The sha-first order is the right order, and worth keeping: refusing on the wrong tree before auditing it is what stops the "audited the wrong commit" failure. It also means a --dir mistake cannot be hidden behind a matching sha.

The --dir value is the one fetch-private.sh is actually given

444: if ! addon=$("$ROOT/scripts/fetch-private.sh" "$ROOT/config/gpu-kernels.config" "$STORE/gpu-kernels" …)

$STORE/gpu-kernels is the dest, so the new argument is the parent of $addon by construction rather than by coincidence — and the comment says so (fetch-private.sh echoes "$dest/$path"), which is the sentence that stops the next person putting $addon back.

Nothing else to ask: the comment explains the failure mode in the place the mistake was made, and engine#2439's CHANGES is credited rather than blamed.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 274c78f: READY

This corrects a line I readied an hour ago, and the severity is worse than "one argument".

hip_f16_audit.py does not scan --dir; it scans --dir + KERNEL_SUBDIR (addons/hip-hrx/kernels, line 53), so --dir "$addon" resolves to $addon/addons/hip-hrx/kernels, which does not exist:

rows = audit(a.dir)
if rows is None:
    print(f"hip_f16_audit: {Path(a.dir) / KERNEL_SUBDIR} is not a directory - cannot tell, and that is not a pass", …)
    return 2

So #2439 would not have audited nothing quietly — it would have failed the gate on every pin bump, with note "::error::… fails the f16 audit" and exit 2, when in fact the audit had never run. Fail-safe, and completely misleading: the next person reads "fails the f16 audit", opens the audit, and finds nothing wrong with it — because the fault was in the path.

That the tool returns 2 rather than an empty report is what kept this from being a silent clean, and it is the same discipline as its exit 2 "which is not a pass" contract: an unknown must not resolve to the answer that looks fine. I have been praising that all night and still did not check the one constant that decides what --dir means.

My miss, plainly

I readied #2439's call without grepping the tool for how it resolves its root. KERNEL_SUBDIR = Path("addons/hip-hrx/kernels") is line 53 of the file I was reading, and kernels = Path(root) / KERNEL_SUBDIR is line 116. Two greps. The same class as #371's removals earlier: I verified the change's intent and not its address. Thanks for catching it.

The fix is right and it is the whole fix

-      --dir "$addon"              --expect-sha "$addon_pin"
+      --dir "$STORE/gpu-kernels"  --expect-sha "$addon_pin"

$STORE/gpu-kernels is the checkout, so addons/hip-hrx/kernels resolves, and --expect-sha still pins the report to the commit that was just fetched — which is the property I reviewed and still holds: the audit is about a named commit, not about "a tree we happened to have".

One thing worth saying while the line is in hand: on a real gpu-kernels clone the .hip count is 59 under addons/ and 60 across the whole tree — so addons/hip-hrx/kernels is very nearly the whole corpus and one kernel sits outside it. The audit covers the subdir by design (it is the add-on's kernels, which is what the ratchet is about), but "60 files in the tree, 59 in scope" is the sort of number that should be in the audit's own output — it already prints a line count and an AUDITED <sha>, so the scope is implied; naming 59 of 60 would make it explicit. Not a condition on this PR.

Ready at 274c78f. And the pair with #2439 now works end to end: the gate audits the pinned checkout, refuses a mismatched tree, and fails loudly if the scope is wrong.

@bong-water-water-bong
bong-water-water-bong merged commit 81cdd54 into main Nov 14, 2026
9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the hwgate-audit-dir branch November 14, 2026 17:10
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Verification of the merged arm on real material (81cdd542; main line 464 = --dir "$STORE/gpu-kernels"):

fetch-private.sh config/gpu-kernels.config <store>  -> tree at 9659366, 71 files in addons/hip-hrx/kernels
hip_f16_audit.py --dir <store>/gpu-kernels --expect-sha 96593660718d…  ->  rc 0
  AUDITED 96593660718d5babe52703f7a93e3bcc9d3f4608
  68 kernel file(s), no f16 accumulator; 62 narrowing(s) — format writes unless a file says otherwise

Stale arm, on a store that really is stale (release-rc's hw-gate store sits at 17a8826): rc 2, "this tree is at 17a882640fa8, --expect-sha says 96593660718d".

Green at the pin, refuses a stale tree — it was only the directory that stopped it reaching either verdict. (71 files in the dir vs 68 scanned: three are not kernel sources by the tool's filter.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant