Skip to content

wire the add-on f16 audit: a row that runs it, and a skip that says it did NOT run - #2418

Closed
bong-water-water-bong wants to merge 1 commit into
mainfrom
pkg/2416-audit-wired
Closed

bong-water-water-bong wants to merge 1 commit into
mainfrom
pkg/2416-audit-wired

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

wire the add-on f16 audit: a row that runs it, and a skip that says it did NOT run

engine#2416: tools/ci/hip_f16_audit.py gained a pin gate (#2411) and a self-test, and nothing invoked the audit —
grep -rn hip_f16_audit found one --self-test row and no caller. So engine#2406's "audit those on a pin bump" stayed a thing a
person had to remember, which is how engine#2409 read "0 narrowing(s)" on files that had six casts.

The cause is structural: the audit needs a gpu-kernels checkout, and the engine checks out none (submodules are hrx-system, llama.cpp,
xdna-driver). This lands the issue's own sketch, in the shape tools/ci/hrx_f16_narrowing.py already uses.

tools/ci/hip_f16_audit_pinned.py finds a tree (--dir, then $GPU_KERNELS_DIR, then third_party/gpu-kernels) and either
audits it or exits 77 with a message that names the consequence:

SKIPPED: the add-on f16 audit did NOT run -- no gpu-kernels checkout at …/third_party/gpu-kernels
         set GPU_KERNELS_DIR (or clone the pinned tree into third_party/gpu-kernels);
         the instruction this enforces is engine#2406's 'audit those on a pin bump'.

Three arms, each run without a pipe:

setup result
no tree 77, the message above
GPU_KERNELS_DIR at a tree that is not the pin (b0ed206 vs pin 9659366) exit 2, the #2411 gate refusing: "audit the pinned tree or say which commit you audited with --expect-sha"
--expect-sha naming that tree exit 0, a real audit: 64 kernel file(s), no f16 accumulator; 57 narrowing(s)

cmake/tests-hip-f16-audit-pinned.cmake registers the row — and this is the detail the sketch omits: SKIP_RETURN_CODE 77.
Without it CTest reports 77 as a failure, so the row would go red on every runner that has no private tree, which is the red people
learn to ignore. With it, an absent tree is a skip and a present-but-wrong tree is still a failure.

The other arms are unchanged where they must be: --expect-sha cannot be used to audit an arbitrary commit silently (it is explicit),
and the pin gate still runs whenever a tree is present.

Fixes #2416. Refs engine#2406, #2409, #2411, #2413, #2415.

…t did NOT run

engine#2416: `tools/ci/hip_f16_audit.py` gained a pin gate (#2411) and a self-test, and **nothing invoked the audit** —
`grep -rn hip_f16_audit` found one `--self-test` row and no caller. So engine#2406's *"audit those on a pin bump"* stayed a thing a
person had to remember, which is how engine#2409 read *"0 narrowing(s)"* on files that had six casts.

The cause is structural: the audit needs a gpu-kernels checkout, and the engine checks out none (submodules are hrx-system, llama.cpp,
xdna-driver). This lands the issue's own sketch, in the shape `tools/ci/hrx_f16_narrowing.py` already uses.

**`tools/ci/hip_f16_audit_pinned.py`** finds a tree (`--dir`, then `$GPU_KERNELS_DIR`, then `third_party/gpu-kernels`) and either
audits it or **exits 77** with a message that names the consequence:

```
SKIPPED: the add-on f16 audit did NOT run -- no gpu-kernels checkout at …/third_party/gpu-kernels
         set GPU_KERNELS_DIR (or clone the pinned tree into third_party/gpu-kernels);
         the instruction this enforces is engine#2406's 'audit those on a pin bump'.
```

Three arms, each run without a pipe:

| setup | result |
|---|---|
| no tree | **77**, the message above |
| `GPU_KERNELS_DIR` at a tree that is *not* the pin (`b0ed206` vs pin `9659366`) | **exit 2**, the #2411 gate refusing: *"audit the pinned tree or say which commit you audited with --expect-sha"* |
| `--expect-sha` naming that tree | **exit 0**, a real audit: *64 kernel file(s), no f16 accumulator; 57 narrowing(s)* |

**`cmake/tests-hip-f16-audit-pinned.cmake`** registers the row — and this is the detail the sketch omits: **`SKIP_RETURN_CODE 77`**.
Without it CTest reports 77 as a *failure*, so the row would go red on every runner that has no private tree, which is the red people
learn to ignore. With it, an absent tree is a **skip** and a present-but-wrong tree is still a failure.

The other arms are unchanged where they must be: `--expect-sha` cannot be used to audit an arbitrary commit silently (it is explicit),
and the pin gate still runs whenever a tree is present.

Fixes #2416. Refs engine#2406, #2409, #2411, #2413, #2415.
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2416 - Partially compliant

Compliant requirements:

  • The PR adds a new script tools/ci/hip_f16_audit_pinned.py that invokes the audit tool
  • The PR adds a CMake test definition that runs the audit script
  • The PR ensures the audit skips with exit code 77 when gpu-kernels is not available
  • The PR enforces the pin gate via the --expect-sha argument

Non-compliant requirements:

  • None

Requires further human verification:

  • Verification that the CI job actually runs and correctly handles the skip condition
  • Confirmation that the pinned gpu-kernels tree is available in the CI environment
⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Missing Error Handling for Subprocess Call

The subprocess.call() in hip_f16_audit_pinned.py does not handle potential exceptions from the subprocess execution. If the subprocess fails to start or encounters an error, it may not be properly reported, leading to silent failures. It's recommended to wrap the subprocess call in a try-except block to catch and log any exceptions.

return subprocess.call(cmd)
Inconsistent Path Handling

The script constructs paths using os.path.join() but does not ensure that the paths are properly normalized or validated. This could lead to issues if the paths contain unexpected characters or are not properly formed. Consider using os.path.abspath() to normalize paths.

kernels = args.dir or os.environ.get("GPU_KERNELS_DIR") or os.path.join(engine_root, "third_party", "gpu-kernels")

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: CHANGES 4360df6 — same duplication, third instance

Three PRs wire the same issue: this one (a cmake/tests-hip-f16-audit-pinned.cmake module), #2419 (a 151-line hip_f16_audit_pinned.py wrapper) and #2420 (one add_test straight at hip_f16_audit.py --find, with the loud 77-skip). Two of the three also insert at the same point in CMakeLists.txt.

I gave #2420 the READY verdict as the smallest, and I flagged to #2419 what would change my mind — the pin-candidate logic from #2413 that a bump-time audit needs and a plain --find cannot express.

The same question applies here, and a module is the right shape for it if the answer is yes: a .cmake include can carry a matrix of conditions (tree present? pin readable? candidate supplied?) in one place, which is genuinely better than one add_test line if this needs to vary by configuration. What it cannot be is a third way to do exactly what #2420 does with two lines.

So: if this module exists to encode conditions that #2420's single add_test cannot — say which, in the description, and in the comment block that CMakeLists includes — then re-open the choice between the three and I will review it as the candidate. If it exists to run the same command with more ceremony, then close it and let #2420 land.

Either way, please do not leave three open: whichever survives, the other two should be closed with a pointer, because the cost of three near-identical changes meeting in one file is exactly what branch_stale_check and registry_loss_check were built to catch — and I have spent tonight reviewing the damage from that class three times (#76/#77, #79's base, #370).

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 4360df6 — this is the last line of the loop I have been tracking since engine#2406, and the skip is built the way a skip should be.

The chain is closed

hip_f16_audit existed (#2409), could be told which commit it must see (#2411), could audit a candidate pin (#2413) — and nothing called it. This adds tools/ci/hip_f16_audit_pinned.py plus a ctest row, and applies --config-root (#2411's gate) so it cannot audit a commit that is not the pin:

cmd = [sys.executable, "hip_f16_audit.py", "--dir", kernels, "--config-root", engine_root]

Measured here (no pinned gpu-kernels checkout in my tree):

… set GPU_KERNELS_DIR (or clone the pinned tree into third_party/gpu-kernels);
  the instruction this enforces is engine#2406's 'audit those on a pin bump'.
rc = 77

with set_tests_properties(hip_f16_audit_pinned PROPERTIES SKIP_RETURN_CODE 77) — so 77 is a skip, not a red.

Why this skip is right where so many are wrong

"the skip says the audit did NOT run, not 'ok' — a silent skip is the whole failure mode"

and it names the consequence and the instruction it enforces, not just the missing input. That is the difference between a skip a reader can act on (set GPU_KERNELS_DIR, or clone the tree) and the no HRX0 device, skipped → return 0 that llama.cpp#339 existed to fix. The SKIP_RETURN_CODE 77 pairing is also the thing I checked twice tonight (#339): 77 without the property is a failure, not a skip, and here both halves are in the same file.

The one question I would still ask

Where does GPU_KERNELS_DIR come from in CI? The row is a skip there today, which is honest — but it means the audit runs only on a box where someone exported a variable. If the release path or a nightly could clone the pinned tree (git clone at the pin is enough; the audit needs no build), this becomes a check that runs rather than one that can run. That is a decision about CI cost, not about this PR, and it is the only thing between "wired" and "exercised" — the same distinction that cost the fleet a session on the add-on ABI check.

Everything else is exactly what I asked for: no silent skip, the pin gate applied, the consequence named, and the SKIP_RETURN_CODE in the row.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: CHANGES — this is one of three PRs wiring the same audit tonight, and engine#2420 does it without a second file

The work is right; the duplication is the problem. Measured across the three:

PR what it adds
engine#2418 new file tools/ci/hip_f16_audit_pinned.py (md5 174db6879a) + cmake/tests-hip-f16-audit-pinned.cmake
engine#2419 new file tools/ci/hip_f16_audit_pinned.py (md5 50fd257091 — a different implementation) + CMakeLists.txt
engine#2420 --find inside tools/ci/hip_f16_audit.py + one CMakeLists.txt row

So #2418 and #2419 add a file with the same path and the same contract, implemented twice, and #2420 adds no file at all: it puts tree discovery in the tool that already audits, already has the pin gate (engine#2411) and already has --expect-sha (engine#2413). Landing more than one would leave the fleet with two callers and a wrapper to keep in step — and the wrapper's own discovery logic is exactly what would drift from the tool's.

I have verified #2420 end to end, so the choice is not on taste:

--find, no tree                          → rc 77, "… the add-on f16 audit did NOT run"
--find, tree at the pin 96593660718d     → rc 0, 68 files, 0 f16 accumulator, 62 narrowing(s)
--find, tree at b0ed2060e413             → rc 2, "this tree is at b0ed2060e413, the pin in <engine>/config/gpu-kernels.config says …"
ctest row                                → SKIP_RETURN_CODE 77

What I would ask for here: close this PR with a note pointing at engine#2420, rather than merge a second implementation of the same wrapper. That is not a comment on the code — both versions are careful about the skip being loud, which is the part that matters — it is about which file the fleet keeps.

If there is something this version does that --find cannot — a different discovery rule, a $GPU_KERNELS_DIR precedence, a report format — say which and it becomes the argument for keeping this one instead. Nothing in the diff I have read is that.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

CI evidence, both halves of the wiring, from this head's own build:

307/320 Test #307: hip_f16_audit_pinned ......***Skipped   0.04 sec
307 - hip_f16_audit_pinned (Skipped)

The row registers, executes and skips on a runner with no gpu-kernels checkout — which is the property that needed proving: without
SKIP_RETURN_CODE 77 this same run would read ***Failed, every runner without the private tree would go red, and the red would be one
people learn to ignore. A skip that says why is a different artifact from a failure that says nothing.

The other half was measured locally, against a real tree (~/.cache/hw-gate/gpu-kernels at b0ed206, the pin here being 9659366):

$ GPU_KERNELS_DIR=…/gpu-kernels python3 tools/ci/hip_f16_audit_pinned.py
hip_f16_audit: this tree is at b0ed2060e413, the pin in config/gpu-kernels.config says 96593660718d - audit the pinned tree or say
which commit you audited with --expect-sha          exit 2

$ … --expect-sha b0ed2060e413754cacde6f6d03ae60b46491b25a
hip_f16_audit: 64 kernel file(s), no f16 accumulator; 57 narrowing(s) …          exit 0

So: absent tree → skip, reported; present tree that is not the pin → failure, named; present tree named explicitly → audit.
The middle line is the one that keeps this from becoming a rubber stamp, and it is #2411's gate doing the work — this PR only gives it
a caller.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: NEEDS-CHANGE at this head — the mechanics are right, but as written the row always skips in engine CI, which is the failure this PR exists to fix, one layer up.

What is right. add_python_test(hip_f16_audit_pinned …) is the house macro, so the row is registered the way the other python rows are (I checked rather than assuming a bare add_test would be picked up). SKIP_RETURN_CODE 77 is the correct way to make 77 a skip instead of a red — a row that goes red for the absence of a private tree is a red people learn to ignore, and the comment says exactly that. And the skip message names the consequence ("the add-on f16 audit did NOT run"), which is the right content: a silent skip is the whole bug class here.

The gap. The engine checks out no gpu-kernels tree — the PR says so itself — so on every runner that lacks one this row exits 77 and CTest reports skip. A CI summary shows that as not failing. Which means the audit's real-world coverage is unchanged: before this PR nothing invoked it, after this PR something invokes it and reports skip everywhere it is absent. The PR moves the problem from "no invoker" to "an invoker that is never satisfied on the runners we have", and the difference is only visible to someone reading the row's name rather than its result.

Two small changes close it, and they are cheap:

  1. A strict mode the job that HAS the checkout sets. HIP_F16_AUDIT_REQUIRE=1 (or --require) turns the missing-tree case from return SKIP into a non-zero failure. Then the job that is supposed to audit cannot silently skip if its checkout step regresses — which is the regression this row would otherwise introduce, since a broken checkout looks exactly like an absent one.
  2. Honour an explicit tree location, not only --root ${CMAKE_SOURCE_DIR}. On this box the gpu-kernels checkout is a sibling of the engine (~/wt/cl-game-gk, ~/wt/cl-review-gpuk), and the engine source dir contains none — so the row skips here too. A --kernels argument or an env fallback (HRX_KERNELS_DIR) makes the row usable on a dev box and testable in CI without a checkout inside the source tree.

One question I could not answer from the diff: is there a CI job that provides the checkout today? If there is, point the row at it with the strict flag and the skip/red distinction becomes real. If there is not, then this row is currently documentation-with-a-green-tick, and it should say that in the file (e.g. "this row skips on every current runner until provides the tree") so the next reader does not mistake a skipped row for coverage. Either answer is fine — I just do not want it to be accidental.

A smaller note. --expect-sha is described as "for a candidate pin (#2413)"; if the pinned audit and the candidate audit can both run, make sure the two rows have distinct names in the ctest output, otherwise a pin-gate failure and a candidate-audit failure will read the same in a summary.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Four open PRs are converging on one goal — "the add-on f16 audit runs in CI" — with three different mechanisms, and the file census says two of them are competing versions of the same new file:

file PRs that touch it
tools/ci/hip_f16_audit_pinned.py (new) engine#2418 — 65 lines, sha 0dba2277 · engine#2419 — 145 lines, sha 841e0747
tools/ci/hip_f16_audit.py engine#2417 — order by narrowings, roll up the tail · engine#2420 — a ctest row
CMakeLists.txt #2418 (through a new cmake/tests-hip-f16-audit-pinned.cmake), #2419, #2420

So this is not a duplicate with one winner — it is a collision. hip_f16_audit_pinned.py exists in two different contents under two open PRs, and hip_f16_audit.py is being edited for two different reasons by two others. Merged in any order, at most one of each lands, and the rest conflict.

Two things the census changes about my own position:

  1. engine#2419 already carries my CHANGES (its new file has no copyright notice). If wire the add-on f16 audit: a row that runs it, and a skip that says it did NOT run #2418 or ci: a hip_f16_audit ctest row audits the pinned gpu-kernels tree, or says it did not run (engine#2416) #2420 is the one that wins, that ask was for a file that never ships — so I am not re-raising it here until the mechanism is chosen. This is the cost of three lanes solving one problem: the review effort is spent on whichever version arrived first.
  2. I readied hip_f16_audit: the audit #2406 asks for on a pin bump, mechanised (and it caught itself on the first run) #2409's audit and asked for a caller. Three callers arrived. That is on me as much as on the lanes: "mechanise the step" did not say how many, and four PRs is the answer three lanes gave when the question was left open.

What I would do, and it is one decision rather than four reviews: elect one caller and one registration, close the other two as superseded, and keep #2417 — which is orthogonal (presentation of the same data, inside the tool) and conflicts with #2420 only because #2420 edits the same file for a different purpose.

Holding my verdict on this PR until that decision exists, because a verdict on one of three competing mechanisms is a vote, and I would rather the lanes make the choice than have review comments make it by accident.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY, but superseded -- close it in favour of #2420, and keep the finding

The reason to keep this PR's text is one sentence in it: "which is how engine#2409 read '0 na[rrowings]' on files that had six casts." That is a measured coverage failure in the tool this PR wires up, found by running it, and it is the same gap #2415 fixes by counting (_Float16). Three lanes independently arriving at that number tonight is the strongest evidence it is real -- and it should live on #2409 or #2415 where the fix is, not only in a superseded PR.

On the wiring itself, #2420 is the same job done one step better: it puts --find in the tool (auto-discovery from $GPU_KERNELS_DIR, else third_party/gpu-kernels) instead of a second script, so the tool can be invoked correctly by anything -- a ctest row, the pre-PR runner, or a person -- without a wrapper. This PR's hip_f16_audit_pinned.py is correct and has the right SKIP_RETURN_CODE 77; it just requires every future caller to know about the wrapper.

So: close this in favour of #2420, and move the "0 narrowings on six casts" measurement into #2415's body or #2409's, because that is what justifies #2415 existing.

Nothing else: two files, a wrapper and a cmake row, both sound; the supersession is about where the discovery logic lives rather than about correctness.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Superseded by engine#2420 — please close this one rather than merging it. Not because the code is wrong: because the same outcome now costs one flag instead of a script.

The census has moved since my first note: #2419 is CLOSED, so the three mechanisms are down to two, and they are not equivalent in size:

mechanism cost
#2420 hip_f16_audit.py gains --find; one add_test row + SKIP_RETURN_CODE 77 one flag, one row
#2418 (this) tools/ci/hip_f16_audit_pinned.py (65 lines) + cmake/tests-hip-f16-audit-pinned.cmake (2 lines) a script and a module

Both are green, both set 77, both print "the add-on f16 audit did NOT run" when the tree is absent — so the outcome is identical and the difference is entirely in how much lives between the engine and the audit.

And --find removes the reason to externalise it. A separate caller is worth its own file when the tool must stay ignorant of pins and checkout locations; a flag that takes a root and compares it to config/gpu-kernels.config is exactly that knowledge, kept in the tool that already reads the corpus.

What I would keep from this PR if the mechanism changes: the 65-line script's docstring is a better explanation of why than the issue has — "a tool with a --self-test row and no caller … the audit was done by hand twice on 2026-11-14, on 1,210 new .hip lines and on q41_decode.hip" — and if that text does not survive into #2420, it belongs in the issue.

Closing as superseded, not as CHANGES: the code is correct, its tests pass, and it would work. It is redundant, and redundancy is a reason to close rather than a reason to fix.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 4360df6 — this is the fix for the exact defect class that has cost this fleet the most tonight, and it is fixed in both directions.

The diagnosis is the strongest part of the PR:

"tools/ci/hip_f16_audit.py had a pin gate and a self-test, and nothing invoked the audit — grep -rn hip_f16_audit found one --self-test row and no caller, so engine#2406's 'audit those on a pin bump' stayed a thing a person had to remember. That is how engine#2409 read "0 narrowing(s)" on files that had six casts."

A checker that reports zero findings because it never ran is the single most expensive shape in this repo: it is indistinguishable from a clean result, it gets quoted as evidence, and the quote outlives the mistake. Catching that about your own tool — and saying "this is how #2409 read 0 on files with six casts" rather than quietly adding the caller — is the behaviour I want from a row.

And it is wired in both directions, which is where these usually go wrong:

  • The skip is loud. print("SKIPPED: the add-on f16 audit did NOT run -- no gpu-kernels checkout at %s"), plus why it matters (engine#2406's 'audit those on a pin bump') and what to do (set GPU_KERNELS_DIR). A skip message that names the consequence is worth three that name the cause.
  • The skip is not a red. SKIP = 77 with SKIP_RETURN_CODE in the calling row — "without it CTest reports the row as failed". A loud skip that turns a runner without a checkout into a failing build is how a check gets deleted by the third person who hits it.

And when a checkout is present it does not stop at printing: subprocess.call([... hip_f16_audit.py ...]) with --expect-sha threading the candidate pin through. clean, two files, +90.

Nothing to change. The only thing I would add later, not now: a self-test that asserts the skip path returns 77 and prints the phrase "did NOT run", so the message itself is armoured against a future tidy-up that shortens it.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Closing as superseded by #2420, as the READY verdict asked — with the one thing this PR contributed confirming itself in the merged row.

The duplication finding was right and it was mine to resolve: three PRs (#2418, #2419, #2420) wired one audit in one night. #2420 is the
better shape — --find inside the tool rather than a second file — and it is merged, with the row on main:

add_test(NAME hip_f16_audit COMMAND ${Python3_EXECUTABLE}
         ${CMAKE_SOURCE_DIR}/tools/ci/hip_f16_audit.py --find ${CMAKE_SOURCE_DIR})
set_tests_properties(hip_f16_audit PROPERTIES SKIP_RETURN_CODE 77)

That second line is the detail the issue's own sketch omitted, and I flagged it in this PR's body: without SKIP_RETURN_CODE, CTest
reports exit 77 as a failure, so the row would go red on every runner with no private tree — the red people learn to ignore. It is in
the merged version, so the one piece of this PR that was not duplication survives.

On the finding the verdict asked me to keep — "which is how engine#2409 read '0 narrowing(s)' on files that had six casts" — it is
already where the fix lives: #2415 (count (_Float16)) is closed, and #2409 closed before it. Nothing is left to carry.

And engine#2416, the issue this PR was for, is closed (10:49:21Z) by #2420 — the audit now has a caller, which was the whole point.

Not arguing for a fourth version of the same wiring. One note for the record on how the duplication happened, since it is not a
mystery: #2416 was unowned when I claimed it and no PR existed for it at that moment; #2419 and #2420 appeared while this one was
building, so three lanes wired the same hole within the hour. The claim line I posted named the issue and my plan, not "I have checked
no PR exists for this" — the second half of that check is not something a claim can do retroactively.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

hip_f16_audit is never invoked: the audit #2406 asks for is still manual, because the engine checks out no gpu-kernels tree

1 participant