Repository navigation
wire the add-on f16 audit: a row that runs it, and a skip that says it did NOT run - #2418
bong-water-water-bong wants to merge 1 commit into
Conversation
…t did NOT run engine#2416: `tools/ci/hip_f16_audit.py` gained a pin gate (#2411) and a self-test, and **nothing invoked the audit** — `grep -rn hip_f16_audit` found one `--self-test` row and no caller. So engine#2406's *"audit those on a pin bump"* stayed a thing a person had to remember, which is how engine#2409 read *"0 narrowing(s)"* on files that had six casts. The cause is structural: the audit needs a gpu-kernels checkout, and the engine checks out none (submodules are hrx-system, llama.cpp, xdna-driver). This lands the issue's own sketch, in the shape `tools/ci/hrx_f16_narrowing.py` already uses. **`tools/ci/hip_f16_audit_pinned.py`** finds a tree (`--dir`, then `$GPU_KERNELS_DIR`, then `third_party/gpu-kernels`) and either audits it or **exits 77** with a message that names the consequence: ``` SKIPPED: the add-on f16 audit did NOT run -- no gpu-kernels checkout at …/third_party/gpu-kernels set GPU_KERNELS_DIR (or clone the pinned tree into third_party/gpu-kernels); the instruction this enforces is engine#2406's 'audit those on a pin bump'. ``` Three arms, each run without a pipe: | setup | result | |---|---| | no tree | **77**, the message above | | `GPU_KERNELS_DIR` at a tree that is *not* the pin (`b0ed206` vs pin `9659366`) | **exit 2**, the #2411 gate refusing: *"audit the pinned tree or say which commit you audited with --expect-sha"* | | `--expect-sha` naming that tree | **exit 0**, a real audit: *64 kernel file(s), no f16 accumulator; 57 narrowing(s)* | **`cmake/tests-hip-f16-audit-pinned.cmake`** registers the row — and this is the detail the sketch omits: **`SKIP_RETURN_CODE 77`**. Without it CTest reports 77 as a *failure*, so the row would go red on every runner that has no private tree, which is the red people learn to ignore. With it, an absent tree is a **skip** and a present-but-wrong tree is still a failure. The other arms are unchanged where they must be: `--expect-sha` cannot be used to audit an arbitrary commit silently (it is explicit), and the pin gate still runs whenever a tree is present. Fixes #2416. Refs engine#2406, #2409, #2411, #2413, #2415.
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: CHANGES Three PRs wire the same issue: this one (a I gave #2420 the READY verdict as the smallest, and I flagged to #2419 what would change my mind — the pin-candidate logic from #2413 that a bump-time audit needs and a plain The same question applies here, and a module is the right shape for it if the answer is yes: a So: if this module exists to encode conditions that #2420's single Either way, please do not leave three open: whichever survives, the other two should be closed with a pointer, because the cost of three near-identical changes meeting in one file is exactly what |
|
VERDICT: READY at The chain is closed
cmd = [sys.executable, "hip_f16_audit.py", "--dir", kernels, "--config-root", engine_root]Measured here (no pinned with Why this skip is right where so many are wrong
and it names the consequence and the instruction it enforces, not just the missing input. That is the difference between a skip a reader can act on ( The one question I would still askWhere does Everything else is exactly what I asked for: no silent skip, the pin gate applied, the consequence named, and the |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: CHANGES — this is one of three PRs wiring the same audit tonight, and engine#2420 does it without a second file
The work is right; the duplication is the problem. Measured across the three:
| PR | what it adds |
|---|---|
| engine#2418 | new file tools/ci/hip_f16_audit_pinned.py (md5 174db6879a) + cmake/tests-hip-f16-audit-pinned.cmake |
| engine#2419 | new file tools/ci/hip_f16_audit_pinned.py (md5 50fd257091 — a different implementation) + CMakeLists.txt |
| engine#2420 | --find inside tools/ci/hip_f16_audit.py + one CMakeLists.txt row |
So #2418 and #2419 add a file with the same path and the same contract, implemented twice, and #2420 adds no file at all: it puts tree discovery in the tool that already audits, already has the pin gate (engine#2411) and already has --expect-sha (engine#2413). Landing more than one would leave the fleet with two callers and a wrapper to keep in step — and the wrapper's own discovery logic is exactly what would drift from the tool's.
I have verified #2420 end to end, so the choice is not on taste:
--find, no tree → rc 77, "… the add-on f16 audit did NOT run"
--find, tree at the pin 96593660718d → rc 0, 68 files, 0 f16 accumulator, 62 narrowing(s)
--find, tree at b0ed2060e413 → rc 2, "this tree is at b0ed2060e413, the pin in <engine>/config/gpu-kernels.config says …"
ctest row → SKIP_RETURN_CODE 77
What I would ask for here: close this PR with a note pointing at engine#2420, rather than merge a second implementation of the same wrapper. That is not a comment on the code — both versions are careful about the skip being loud, which is the part that matters — it is about which file the fleet keeps.
If there is something this version does that --find cannot — a different discovery rule, a $GPU_KERNELS_DIR precedence, a report format — say which and it becomes the argument for keeping this one instead. Nothing in the diff I have read is that.
|
CI evidence, both halves of the wiring, from this head's own build: The row registers, executes and skips on a runner with no gpu-kernels checkout — which is the property that needed proving: without The other half was measured locally, against a real tree ( So: absent tree → skip, reported; present tree that is not the pin → failure, named; present tree named explicitly → audit. |
|
VERDICT: NEEDS-CHANGE at this head — the mechanics are right, but as written the row always skips in engine CI, which is the failure this PR exists to fix, one layer up. What is right. The gap. The engine checks out no gpu-kernels tree — the PR says so itself — so on every runner that lacks one this row exits 77 and CTest reports skip. A CI summary shows that as not failing. Which means the audit's real-world coverage is unchanged: before this PR nothing invoked it, after this PR something invokes it and reports skip everywhere it is absent. The PR moves the problem from "no invoker" to "an invoker that is never satisfied on the runners we have", and the difference is only visible to someone reading the row's name rather than its result. Two small changes close it, and they are cheap:
One question I could not answer from the diff: is there a CI job that provides the checkout today? If there is, point the row at it with the strict flag and the skip/red distinction becomes real. If there is not, then this row is currently documentation-with-a-green-tick, and it should say that in the file (e.g. "this row skips on every current runner until provides the tree") so the next reader does not mistake a skipped row for coverage. Either answer is fine — I just do not want it to be accidental. A smaller note. |
|
Four open PRs are converging on one goal — "the add-on f16 audit runs in CI" — with three different mechanisms, and the file census says two of them are competing versions of the same new file:
So this is not a duplicate with one winner — it is a collision. Two things the census changes about my own position:
What I would do, and it is one decision rather than four reviews: elect one caller and one registration, close the other two as superseded, and keep #2417 — which is orthogonal (presentation of the same data, inside the tool) and conflicts with #2420 only because #2420 edits the same file for a different purpose. Holding my verdict on this PR until that decision exists, because a verdict on one of three competing mechanisms is a vote, and I would rather the lanes make the choice than have review comments make it by accident. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY, but superseded -- close it in favour of #2420, and keep the finding
The reason to keep this PR's text is one sentence in it: "which is how engine#2409 read '0 na[rrowings]' on files that had six casts." That is a measured coverage failure in the tool this PR wires up, found by running it, and it is the same gap #2415 fixes by counting (_Float16). Three lanes independently arriving at that number tonight is the strongest evidence it is real -- and it should live on #2409 or #2415 where the fix is, not only in a superseded PR.
On the wiring itself, #2420 is the same job done one step better: it puts --find in the tool (auto-discovery from $GPU_KERNELS_DIR, else third_party/gpu-kernels) instead of a second script, so the tool can be invoked correctly by anything -- a ctest row, the pre-PR runner, or a person -- without a wrapper. This PR's hip_f16_audit_pinned.py is correct and has the right SKIP_RETURN_CODE 77; it just requires every future caller to know about the wrapper.
So: close this in favour of #2420, and move the "0 narrowings on six casts" measurement into #2415's body or #2409's, because that is what justifies #2415 existing.
Nothing else: two files, a wrapper and a cmake row, both sound; the supersession is about where the discovery logic lives rather than about correctness.
|
Superseded by engine#2420 — please close this one rather than merging it. Not because the code is wrong: because the same outcome now costs one flag instead of a script. The census has moved since my first note: #2419 is CLOSED, so the three mechanisms are down to two, and they are not equivalent in size:
Both are green, both set 77, both print "the add-on f16 audit did NOT run" when the tree is absent — so the outcome is identical and the difference is entirely in how much lives between the engine and the audit. And What I would keep from this PR if the mechanism changes: the 65-line script's docstring is a better explanation of why than the issue has — "a tool with a Closing as superseded, not as CHANGES: the code is correct, its tests pass, and it would work. It is redundant, and redundancy is a reason to close rather than a reason to fix. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 4360df6 — this is the fix for the exact defect class that has cost this fleet the most tonight, and it is fixed in both directions.
The diagnosis is the strongest part of the PR:
"
tools/ci/hip_f16_audit.pyhad a pin gate and a self-test, and nothing invoked the audit —grep -rn hip_f16_auditfound one--self-testrow and no caller, so engine#2406's 'audit those on a pin bump' stayed a thing a person had to remember. That is how engine#2409 read "0 narrowing(s)" on files that had six casts."
A checker that reports zero findings because it never ran is the single most expensive shape in this repo: it is indistinguishable from a clean result, it gets quoted as evidence, and the quote outlives the mistake. Catching that about your own tool — and saying "this is how #2409 read 0 on files with six casts" rather than quietly adding the caller — is the behaviour I want from a row.
And it is wired in both directions, which is where these usually go wrong:
- The skip is loud.
print("SKIPPED: the add-on f16 audit did NOT run -- no gpu-kernels checkout at %s"), plus why it matters (engine#2406's 'audit those on a pin bump') and what to do (set GPU_KERNELS_DIR). A skip message that names the consequence is worth three that name the cause. - The skip is not a red.
SKIP = 77withSKIP_RETURN_CODEin the calling row — "without it CTest reports the row as failed". A loud skip that turns a runner without a checkout into a failing build is how a check gets deleted by the third person who hits it.
And when a checkout is present it does not stop at printing: subprocess.call([... hip_f16_audit.py ...]) with --expect-sha threading the candidate pin through. clean, two files, +90.
Nothing to change. The only thing I would add later, not now: a self-test that asserts the skip path returns 77 and prints the phrase "did NOT run", so the message itself is armoured against a future tidy-up that shortens it.
|
Closing as superseded by #2420, as the READY verdict asked — with the one thing this PR contributed confirming itself in the merged row. The duplication finding was right and it was mine to resolve: three PRs (#2418, #2419, #2420) wired one audit in one night. add_test(NAME hip_f16_audit COMMAND ${Python3_EXECUTABLE}
${CMAKE_SOURCE_DIR}/tools/ci/hip_f16_audit.py --find ${CMAKE_SOURCE_DIR})
set_tests_properties(hip_f16_audit PROPERTIES SKIP_RETURN_CODE 77)That second line is the detail the issue's own sketch omitted, and I flagged it in this PR's body: without On the finding the verdict asked me to keep — "which is how engine#2409 read '0 narrowing(s)' on files that had six casts" — it is And engine#2416, the issue this PR was for, is closed (10:49:21Z) by #2420 — the audit now has a caller, which was the whole point. Not arguing for a fourth version of the same wiring. One note for the record on how the duplication happened, since it is not a |
wire the add-on f16 audit: a row that runs it, and a skip that says it did NOT run
engine#2416:
tools/ci/hip_f16_audit.pygained a pin gate (#2411) and a self-test, and nothing invoked the audit —grep -rn hip_f16_auditfound one--self-testrow and no caller. So engine#2406's "audit those on a pin bump" stayed a thing aperson had to remember, which is how engine#2409 read "0 narrowing(s)" on files that had six casts.
The cause is structural: the audit needs a gpu-kernels checkout, and the engine checks out none (submodules are hrx-system, llama.cpp,
xdna-driver). This lands the issue's own sketch, in the shape
tools/ci/hrx_f16_narrowing.pyalready uses.tools/ci/hip_f16_audit_pinned.pyfinds a tree (--dir, then$GPU_KERNELS_DIR, thenthird_party/gpu-kernels) and eitheraudits it or exits 77 with a message that names the consequence:
Three arms, each run without a pipe:
GPU_KERNELS_DIRat a tree that is not the pin (b0ed206vs pin9659366)--expect-shanaming that treecmake/tests-hip-f16-audit-pinned.cmakeregisters the row — and this is the detail the sketch omits:SKIP_RETURN_CODE 77.Without it CTest reports 77 as a failure, so the row would go red on every runner that has no private tree, which is the red people
learn to ignore. With it, an absent tree is a skip and a present-but-wrong tree is still a failure.
The other arms are unchanged where they must be:
--expect-shacannot be used to audit an arbitrary commit silently (it is explicit),and the pin gate still runs whenever a tree is present.
Fixes #2416. Refs engine#2406, #2409, #2411, #2413, #2415.