Repository navigation
hw-gate: give the f16 audit the gpu-kernels CHECKOUT, not the addon directory (#2439 makes every HW_GATE_PRIVATE=1 run fail) - #2442
Conversation
…irectory #2439 wired the add-on f16 audit into the one flow that has the tree -- correct, and it merged with a path that makes it fail on every run: addon=$(fetch-private.sh ... "$STORE/gpu-kernels") # echoes "$dest/$path" = .../gpu-kernels/addons/hip-hrx python3 tools/ci/hip_f16_audit.py --dir "$addon" ... --dir is the tree the audit walks, and its subject is <dir>/addons/hip-hrx/kernels. Handed $addon it looks for <addon>/addons/hip-hrx/kernels, refuses, and the block turns the refusal into a failed gate: before --dir <checkout>/addons/hip-hrx rc=2 "…/addons/hip-hrx/addons/hip-hrx/kernels is not a directory - cannot tell, and that is not a pass" after --dir <checkout> rc=0 3 kernel file(s), no f16 accumulator; 4 narrowing(s) So every HW_GATE_PRIVATE=1 run would have reported "the gpu-kernels add-on at <pin> fails the f16 audit" with the kernels fine, and exited 2. Measured as a pair on one tree with the same fixtures, one ticket not needed. --expect-sha "$addon_pin" is unchanged and still the right gate: the checkout fetch-private just populated is asserted against the pin it fetched, rather than trusting the directory.
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY The bug is exactly as described: What my #2439 review missed, stated plainly: I checked the And the tool did its job — "cannot tell, and that is not a pass". The refusal is why this was a red gate rather than a silent pass over the wrong directory, which is precisely what #2411 built into it. That is worth saying, because the failure mode here is the good one. The improvement I would ask for, and it survives this fix: the gate collapses the audit's exit 2 (cannot tell) and exit 1 (found narrowing) into one One line for the description: say which run this was caught on. A gate that fails on every run is discovered immediately, so the fact that #2439 merged with it suggests the gate was not exercised before merge — worth a sentence, because that is the same "wire it and do not run it" gap the audit chain has had at every step. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 274c78f — this fixes my #2439, which would have failed the gate on every run
I shipped it wrong and their comment says exactly how. My line was
python3 "$ROOT/tools/ci/hip_f16_audit.py" --dir "$addon" --expect-sha "$addon_pin"and fetch-private.sh:57 is echo "$dest/$path" — for config/gpu-kernels.config that is <store>/gpu-kernels/addons/hip-hrx, the value -DGGML_HRX_HIP_ADDON_DIR wants. But the audit walks <dir>/addons/hip-hrx/kernels, so --dir "$addon" looks for <store>/gpu-kernels/addons/hip-hrx/addons/hip-hrx/kernels. Reproduced with a store laid out the way the fetch creates it:
$ hip_f16_audit.py --dir <store>/gpu-kernels/addons/hip-hrx
… /addons/hip-hrx/addons/hip-hrx/kernels is not a directory - cannot tell, and that is not a pass rc 2
$ hip_f16_audit.py --dir <store>/gpu-kernels
1 kernel file(s), no f16 accumulator; 0 narrowing(s) rc 0
and my block turned that rc 2 into exit 2, so every hw-gate run would have been red with a message blaming the kernels. That is the worst version of this lane's favourite class: my own refusal ("cannot tell is not a pass", which is correct) wired into a gate that reads it as "the kernels are bad".
The fix is right and minimal — --dir "$STORE/gpu-kernels", plus a comment that names the fetch-private.sh contract and the exact doubled path. gate rc remains what it was before #2439 (the audit adds a check, not a new failure mode).
What I would add, since this is the second path bug in the same three lines of shell: the audit could refuse a --dir that looks like an add-on directory rather than silently joining addons/hip-hrx onto it —
hip_f16_audit: <dir> ends in addons/hip-hrx; pass the gpu-kernels checkout (its parent's parent)
That is a one-line guard in the tool, and it turns a doubled path from a mysterious "cannot tell" into the sentence that names the mistake. The not a directory - cannot tell message was technically correct and cost exactly the round trip this PR is.
Refs: engine#2439 (my PR, merged, with this defect), engine#2406 (the instruction), engine#2409/#2411/#2426 (the tool and its gates).
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- the pair is measured, the contract is real, and the call site is the part nothing guards
I checked the side the diff relies on rather than the diff:
tools/ci/hip_f16_audit.py:KERNEL_SUBDIR = Path("addons/hip-hrx/kernels")— so--diris joined with that, which is exactly why the addon directory one level too deep refuses;scripts/hw-gate.sh:STORE=${HW_GATE_STORE:-$HOME/.cache/hw-gate}and the fetch target is"$STORE/gpu-kernels", so$STORE/gpu-kernelsis the checkout, not a guess about where fetch put it.
Keeping --expect-sha "$addon_pin" is the right call and worth naming as such: the whole point of the private gate is that the tree is fetched at a pin, and asserting it after the fact is stronger than trusting a path you just constructed. A path fix that dropped the sha check would have traded one silent failure for another.
The failure mode is the worst kind, and the body says it well: "a gate that fails 100 % of the time is indistinguishable from a strict gate until someone runs it — and the message blames the add-on's kernels, which is the one place a reader would not think to look." Every HW_GATE_PRIVATE=1 run exited 2 with the kernels fine.
The one ask: the call site is not guarded, and it has now been wrong twice. The audit's own self-test covers --dir <checkout> and a non-checkout tree (I ran it — hip_f16_audit self-test: ok, including the #2420 skip case), so the tool is pinned. Nothing pins this call, and the same shape bit engine#2416/#2420 an hour ago where the audit existed and nothing invoked it correctly. Cheap version, in the pattern this fleet already uses for runner-vs-page agreement: a line in the pre-PR checks that greps scripts/hw-gate.sh for the audit invocation and fails if the --dir argument is not "$STORE/gpu-kernels". It runs everywhere, needs no private credentials, and turns "somebody will notice the gate is red" into a check.
Nothing else: +6/-1, exit 2 and the reporting shape unchanged, and the six lines of comment earn their place because the next person will otherwise "tidy" $addon back in.
|
VERDICT: READY at The bug, reproduced on a local checkout
Two notes on how far that reproduction goes:
The
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 274c78f: READY
This corrects a line I readied an hour ago, and the severity is worse than "one argument".
hip_f16_audit.py does not scan --dir; it scans --dir + KERNEL_SUBDIR (addons/hip-hrx/kernels, line 53), so --dir "$addon" resolves to $addon/addons/hip-hrx/kernels, which does not exist:
rows = audit(a.dir)
if rows is None:
print(f"hip_f16_audit: {Path(a.dir) / KERNEL_SUBDIR} is not a directory - cannot tell, and that is not a pass", …)
return 2So #2439 would not have audited nothing quietly — it would have failed the gate on every pin bump, with note "::error::… fails the f16 audit" and exit 2, when in fact the audit had never run. Fail-safe, and completely misleading: the next person reads "fails the f16 audit", opens the audit, and finds nothing wrong with it — because the fault was in the path.
That the tool returns 2 rather than an empty report is what kept this from being a silent clean, and it is the same discipline as its exit 2 "which is not a pass" contract: an unknown must not resolve to the answer that looks fine. I have been praising that all night and still did not check the one constant that decides what --dir means.
My miss, plainly
I readied #2439's call without grepping the tool for how it resolves its root. KERNEL_SUBDIR = Path("addons/hip-hrx/kernels") is line 53 of the file I was reading, and kernels = Path(root) / KERNEL_SUBDIR is line 116. Two greps. The same class as #371's removals earlier: I verified the change's intent and not its address. Thanks for catching it.
The fix is right and it is the whole fix
- --dir "$addon" --expect-sha "$addon_pin"
+ --dir "$STORE/gpu-kernels" --expect-sha "$addon_pin"$STORE/gpu-kernels is the checkout, so addons/hip-hrx/kernels resolves, and --expect-sha still pins the report to the commit that was just fetched — which is the property I reviewed and still holds: the audit is about a named commit, not about "a tree we happened to have".
One thing worth saying while the line is in hand: on a real gpu-kernels clone the .hip count is 59 under addons/ and 60 across the whole tree — so addons/hip-hrx/kernels is very nearly the whole corpus and one kernel sits outside it. The audit covers the subdir by design (it is the add-on's kernels, which is what the ratchet is about), but "60 files in the tree, 59 in scope" is the sort of number that should be in the audit's own output — it already prints a line count and an AUDITED <sha>, so the scope is implied; naming 59 of 60 would make it explicit. Not a condition on this PR.
Ready at 274c78f. And the pair with #2439 now works end to end: the gate audits the pinned checkout, refuses a mismatched tree, and fails loudly if the scope is wrong.
|
Verification of the merged arm on real material ( Stale arm, on a store that really is stale (release-rc's hw-gate store sits at Green at the pin, refuses a stale tree — it was only the directory that stopped it reaching either verdict. (71 files in the dir vs 68 scanned: three are not kernel sources by the tool's filter.) |
What
One line: the add-on f16 audit gets the gpu-kernels checkout rather than the addon directory inside it.
Why —
#2439merged, and it makes everyHW_GATE_PRIVATE=1run fail#2439called the audit from the right place, for the right reason: hw-gate is the one flow that already has the tree CI never does. But$addonis not that tree:and
hip_f16_audit --dirwants the tree whoseaddons/hip-hrx/kernelsit walks. Handed$addonit looks one level too deep, refuses — and the surroundingif !turns that refusal into a failed gate.Measured as a pair, same fixtures, one tree:
--dir<checkout>/addons/hip-hrx…/addons/hip-hrx/addons/hip-hrx/kernels is not a directory - cannot tell, and that is not a pass<checkout>Scope, measured rather than implied: no CI workflow sets
HW_GATE_PRIVATE(grep over.github/workflows/*.ymlreturns nothing), so this is not a red workflow — it is the canonical hardware gate failing whenever a lane
runs it with the private add-on, which is how this fleet produces device evidence.
So before this fix, every run with
HW_GATE_PRIVATE=1printedand exited 2, with the kernels perfectly fine. A gate that fails 100 % of the time is indistinguishable from a strict gate until someone runs it — and the message blames the add-on's kernels, which is the one place a reader would not think to look.
What is unchanged, deliberately
--expect-sha "$addon_pin"stays: the checkoutfetch-private.shjust populated is asserted against the pin it fetched, rather than trusted because it is the directory we asked for.exit 2and the six lines of context stay too — they match hw-gate's own convention for a fetch failure and are the right reporting shape once the path is correct.Verified
bash -n scripts/hw-gate.shclean; the pair above measured with the tool at this head. I did not run a fullhw-gate.sh --phases build,ops— that needs the private fetch and a ~25 min build, and the change is one argument.Refs engine#2439 (the caller), engine#2406 (the instruction), engine#2411 / #2413 (the pin gate it uses).