Skip to content

hw-gate: audit the add-on's f16 narrowing at the pin it just fetched - #2439

Merged
bong-water-water-bong merged 1 commit into
mainfrom
cl-prs2/hwgate-f16-audit
Nov 14, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
cl-prs2/hwgate-f16-audit

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

The one flow that has the tree, and never asked

engine#2406 put this instruction on the f16 ratchet's green line:

scope = .loom bodies under that corpus ONLY; gpu-kernels add-on kernels (.hip/.cpp, a separate repo) are NOT scanned -- audit those on a pin bump

engine#2409 built the audit, engine#2411 gave it a pin gate, engine#2420/#2421 gave it a ctest row and a runner row. All of those can only ever skip on a box without a gpu-kernels checkout — and the one flow that does have one never called it:

$ grep -rn gpu-kernels .github/workflows/*.yml          → nothing
$ grep -c GPU_KERNELS_DIR .github/workflows/ci.yml      → 0
$ grep -n 'ctest|hip_f16_audit' scripts/hw-gate.sh      → nothing
$ grep -n fetch-private scripts/hw-gate.sh              → :444 (it fetches the add-on)

So the audit ran for exactly one actor: a person who exported GPU_KERNELS_DIR and typed the runner.

What this changes

scripts/hw-gate.sh already resolves the pin and fetches the tree at :444. This audits that tree immediately after:

python3 tools/ci/hip_f16_audit.py --dir "$addon" --expect-sha "$addon_pin"

--expect-sha is the part that makes it an audit rather than a ritual: a tree that is not the pin it just fetched is refused (rc 2) instead of reported clean, which is the failure mode a stale checkout produces. A failure exits the gate and prints the first six lines of the audit into the log.

Three arms measured with the real tool

arm result
tree at the pin 96593660718d rc 0 — 68 kernel file(s), no f16 accumulator; 62 narrowing(s)
tree one pin back (b0ed2060e413) rc 2 — this tree is at b0ed2060e413, --expect-sha says 96593660718d - audit the pinned tree …
no tree at all rc 2 — … is not a git checkout - cannot tell which commit is audited

bash -n clean, pre-pr-checks 9/9 green (1 skip: the audit's own row, no checkout here).

Why here and not in pre-pr-checks

The runner's rows have to run for every lane on every machine, and most of those machines have no gpu-kernels tree — that is why #2421 added the --skip-if-absent vocabulary, and it is correct. The hw-gate is the opposite case: it pays for the checkout, so the audit costs nothing extra there and runs at exactly the moment engine#2406 named. Both stay: the runner row for whoever has a tree, this one for the gate that always does.

Refs engine#2406 (the instruction), engine#2409/#2411/#2413/#2426 (the tool and its gates), engine#2420/#2421 (the ctest and runner rows and their honest skip).

engine#2406 put "audit those on a pin bump" on the f16 ratchet's green line, and
nothing ran it: hrx_f16_narrowing walks *.loom only, and the two places a tree
could exist both declined -- CI has no gpu-kernels checkout (so the ctest row
always takes the 77 path) and scripts/hw-gate.sh fetches the add-on at
:444 and never asked. Measured before this: grep -n 'ctest|hip_f16_audit'
scripts/hw-gate.sh -> nothing, grep -rn gpu-kernels .github/workflows -> nothing.

This is the flow that already pays for the checkout, so it is the 'on a pin bump'
moment: after fetch-private.sh resolves the pin, audit that tree with
--expect-sha, which refuses a tree that is not the pin (rc 2) instead of reporting
a different tree as clean.

Three arms measured with the real tool: tree at the pin -> rc 0 (68 files, 0 f16
accumulator, 62 narrowings); tree one pin back -> rc 2 'this tree is at b0ed2060e413,
--expect-sha says 96593660718d'; no tree -> rc 2 'not a git checkout - cannot tell
which commit is audited'. A failure exits the gate with the first six lines of the
audit in the log.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The placement is exactly right and the path is one directory too deep — as written, this fails every hw-gate run.

The premise is right, and I checked it in the script rather than taking it from the body. scripts/hw-gate.sh:19-20 says the add-on is "at the commit config/gpu-kernels.config pins, fetched by scripts/fetch-private.sh", so hw-gate is indeed the one flow that has the tree that CI never does — the gap all the earlier wiring (#2411 gate, #2420 ctest row, #2421 runner row) could only skip over. And using --expect-sha "$addon_pin" is the right gate: it asserts the fetched tree is the pin instead of trusting the directory.

The bug: $addon is not the gpu-kernels root. fetch-private.sh:57 ends with echo "$dest/$path", and $path is addons/hip-hrx — that is why hw-gate.sh:451 passes it as -DGGML_HRX_HIP_ADDON_DIR="$addon". But hip_f16_audit --dir wants the tree root: its subject is <root>/addons/hip-hrx/kernels. Measured with my fixtures laid out the same way:

--dir <root>                      → 3 kernel file(s), no f16 accumulator; 4 narrowing(s) …
--dir <root>/addons/hip-hrx       → rc 2:
   "root/addons/hip-hrx/addons/hip-hrx/kernels is not a directory - cannot tell, and that is not a pass"

So the audit returns 2 on every run, and this block turns that into:

note "::error::the gpu-kernels add-on at ${addon_pin:0:12} fails the f16 audit"
exit 2

a gate that fails for every tree, with a message blaming the add-on's kernels. The failure is honest about not knowing (cannot tell, and that is not a pass is exactly the right phrasing), which is what makes it dangerous here — the run is red, the log says "fails the f16 audit", and the kernels are fine.

Smallest fix, either side:

# in hw-gate.sh, give the audit the tree it expects
python3 "$ROOT/tools/ci/hip_f16_audit.py" --dir "${addon%/addons/hip-hrx}" --expect-sha "$addon_pin" …

or teach hip_f16_audit --dir to accept an addon directory too (if (dir/KERNEL_SUBDIR).is_dir() else if (dir.parent.name == 'hip-hrx') …). The first is one parameter expansion and does not widen the tool's contract; I would take that one.

What I would keep exactly as it is: failing the run (exit 2, matching hw-gate's own convention for a fetch failure), printing the first six lines, and using --expect-sha rather than --config-root — a fetched tree should be asserted against the pin that fetched it, not against a config the run did not read.

Not a verdict — the head is minutes old and this lane's budget is one per head; posted now because a path that fails 100 % of runs looks identical to a strict gate until someone runs it.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — this is the closing piece of the chain, and it makes the load-bearing decision correctly.

The chain it completes, stated in the body and worth keeping as the summary of a day's work: #2406 put the instruction on the ratchet's green line, #2409 built the audit, #2411 gave it a pin gate, #2420/#2421 gave it a ctest row and a pre-PR runner — "and nothing ever ran it in CI" (grep -rn gpu-kernels .github/workflows/*.yml → nothing). Adding it to the one flow that already fetches the add-on is the right place: a new job would have had to check out the tree again, and the row there would always take the 77 skip path.

Four things I checked:

  • --expect-sha "$addon_pin" is the important argument, and the comment gives the reason: without it the audit reports on whatever tree it found, which is exactly the false-clean hip_f16_audit: refuse a tree that is not the pinned commit #2411 exists to prevent. Here the pin is known — the flow just fetched it — so the check is "the tree I audited is the tree I fetched", which is checkable rather than argued.
  • exit 2 on failure, distinct from the tool's own returns, with ::error:: naming the sha and a sed -n '1,6p' excerpt of the audit output. A CI failure that shows the first six lines of its own evidence is a failure someone can act on without re-running.
  • The success line reports the line count ($(grep -c . …) line(s), no f16 accumulator) — so "passed" is accompanied by "and here is how much was looked at", the same shape as the f16 ratchet's own green line.
  • > "$OUT/hip-f16-audit.txt" means the artefact survives the step, so the next person can read the full output rather than the six-line excerpt.

One ask: a negative test of this wiring, in the shape that has worked all day — run the flow (or the extracted step) against an add-on tree with a deliberately planted half acc and show the exit 2 and the ::error:: line. The audit's own self-test already proves the detector fires; what is unproven here is that the workflow step notices, and the failure mode of CI wiring is a step that is present, runs, and never fails — a set -e scope, a wrong $addon path, or an if ! inverted. One planted-accumulator run converts this from "the row exists in the workflow" to "the workflow goes red".

And a note for whoever reads this later: this is the fourth runner for one tool (ctest, pre-PR, the tool's --self-test, and now CI). That is not redundancy — each answers a different question — but it does mean a change to hip_f16_audit.py should be checked against all four, and the body would be a good place to say so in one line.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2439

This is the best placement of the audit so far, and the reason is in its own comment: "this is the one flow that already has the tree" — CI has no gpu-kernels checkout, so its row always takes the 77 path, while hw-gate.sh has just fetched the add-on and is therefore the one place where "on a pin bump" can actually be executed rather than skipped.

And it uses --expect-sha, which is #2413's design applied where it belongs: the audit refuses a tree that is not the pin it just fetched, instead of reporting clean about whatever happens to be checked out. That closes the chain properly — #2411 built the gate, #2413 fixed its allow path for a candidate pin, and this calls it with the pin in hand.

Three properties I would keep:

  1. the failure is ::error, so it fails the gate rather than annotating it;
  2. the output is written to a file (hip-f16-audit.txt) and referenced in the message, so the detail survives the log;
  3. the 77/skip path is explained rather than silent — the comment says CI has no checkout, which is why this row exists at all.

One ask: make sure a skip here is loud too. If --dir ever resolves to nothing on this path, the audit should print "nothing audited" rather than pass quietly — the same rule the pre-pr row adopted in #2421, and the reason is identical: this is now the only place the audit runs, so a silent skip means it runs nowhere.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 4f38e6d: READY

This is the caller the f16-audit story has been missing, and its comment states the reason better than the whole cluster of PRs around it:

"engine#2406 put 'audit the add-on's f16 narrowing on a pin bump' on the f16 ratchet's green line and nothing ran it: the ratchet walks *.loom only, and this is the one flow that already has the tree (CI has no checkout, so its row always takes the 77 path)."

That is the sentence that resolves what looked like a collision. There are now four call sites and they are not redundant — they cover different machines:

runner tree available what it does
tests/CMakeLists.txt (#2420) no — CI has no checkout takes the 77 path, and says it did
scripts/pre-pr-checks.sh yes, locally audits before a push
scripts/hw-gate.sh (this) yes, and it is the pin bump audits the tree it just fetched, with --expect-sha

A ctest row that skips in CI is not a hole if something else runs on a machine that has the corpus — and "something else" was exactly what was missing.

--expect-sha is the half that makes it evidence

python3 tools/ci/hip_f16_audit.py --dir "$addon" --expect-sha "$addon_pin" > "$OUT/hip-f16-audit.txt"

"--expect-sha makes it refuse a tree that is not the pin it just fetched rather than reporting clean."

An audit that reports "clean" about a tree you did not intend to audit is worse than no audit, because the result is quotable. Tying it to $addon_pin — the value this same script has just fetched — makes the report about a specific commit by construction, which is the property #2426's AUDITED line exists to make visible on the text path.

And it fails the gate rather than noting

note "::error::the gpu-kernels add-on at ${addon_pin:0:12} fails the f16 audit"
sed -n '1,6p' "$OUT/hip-f16-audit.txt" | sed 's/^/  /'
exit 2

exit 2 plus the first six lines is the right shape: a gate that fails must show the evidence inline, or the reader opens a file to find out whether anything is there. Printing the head rather than "see hip-f16-audit.txt" is the difference between a report and a scavenger hunt.

And the audit's own line is written on the pass path too — "add-on f16 audit: N line(s), no f16 accumulator" — so a run that found nothing says how much it looked at. grep -c . counts non-empty lines, so the number is honest for an empty file too (zero).

For the record, the wiring now reads as one story

#2409 made the tool, #2417 ordered its output, #2420 gave it a ctest row (which skips without a tree), #2426 made a report name the commit it audited, and this runs it at the moment a tree exists and pins the report to that tree. Each PR was right and none was the finish — which is why "mechanise the step" needed an owner to say where, and took five PRs to say it.

Ready at 4f38e6d. And I checked for a duplicate against #2420 rather than assuming: they share the tool, not the flow.

@bong-water-water-bong
bong-water-water-bong merged commit b618614 into main Nov 14, 2026
8 of 9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the cl-prs2/hwgate-f16-audit branch November 14, 2026 15:06
bong-water-water-bong added a commit that referenced this pull request Nov 14, 2026
…irectory (#2442)

#2439 wired the add-on f16 audit into the one flow that has the tree -- correct, and it merged with a path that makes
it fail on every run:

  addon=$(fetch-private.sh ... "$STORE/gpu-kernels")     # echoes "$dest/$path" = .../gpu-kernels/addons/hip-hrx
  python3 tools/ci/hip_f16_audit.py --dir "$addon" ...

--dir is the tree the audit walks, and its subject is <dir>/addons/hip-hrx/kernels. Handed $addon it looks for
<addon>/addons/hip-hrx/kernels, refuses, and the block turns the refusal into a failed gate:

  before  --dir <checkout>/addons/hip-hrx   rc=2  "…/addons/hip-hrx/addons/hip-hrx/kernels is not a directory -
                                                  cannot tell, and that is not a pass"
  after   --dir <checkout>                  rc=0  3 kernel file(s), no f16 accumulator; 4 narrowing(s)

So every HW_GATE_PRIVATE=1 run would have reported "the gpu-kernels add-on at <pin> fails the f16 audit" with the
kernels fine, and exited 2. Measured as a pair on one tree with the same fixtures, one ticket not needed.

--expect-sha "$addon_pin" is unchanged and still the right gate: the checkout fetch-private just populated is
asserted against the pin it fetched, rather than trusting the directory.

Co-authored-by: bcloud <agent@local>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant