Repository navigation
hip_f16_audit: print AUDITED <sha> before the counts, so a pasted report is evidence (engine#2416) - #2426
Conversation
…ort is evidence
engine#2416's playbook now asks a pin-bump PR body for "the commit audited", and the tool never printed it: measured
on the from-pointer tree, the sha appeared 0 times in the output. So that line depended on the reporter remembering to
run rev-parse as well, which is the fragility the pin gate (engine#2411) exists to remove -- a report that does not
name its tree cannot be checked by the person reading it.
AUDITED 7d63abfe239e39436847334e70afd46b121b8b25
fa_prompt_q8w.hip 336 lines narrowings 2 f16 accumulators -
...
Same convention as the Loom ratchet's CORPUS and FINGERPRINT lines, and printed BEFORE the verdict for the same reason
stated there: a FAIL is when the identity of the tree matters most.
A tree that is not a git checkout prints "AUDITED no commit: <path> is not a git checkout" rather than a blank or a
guess -- an audit of a tree whose commit is unknown is still a legitimate thing to run (plain --dir), it just cannot
claim an identity.
--self-test gains two cases, both driven through main() rather than the helper: the report names the commit it audited,
and a non-checkout tree says so instead of printing a sha. Driving main() is deliberate -- the caller removed earlier
today passed its own self-test while its default path skipped the pin gate, because the test called the function
directly and never the entry point.
Verified:
--dir <git tree> --expect-sha <sha> -> AUDITED <sha> then the rows
--dir <non-git tree> -> AUDITED no commit: ... then the rows
--self-test -> ok
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY This is the ask I made on #2411 and repeated on #2413 — "print which sha was audited, so a run that audited the wrong candidate is diagnosable after the fact rather than only at the time" — and it is done the right way round: The sha comes first, before the counts. That ordering matters: a reader who sees a table of per-file narrowing counts has no way to know which tree produced them unless the identity is above the data, and a 40-character sha at the top is exactly the line that makes a saved output auditable months later. Printing it after the table (or in a trailer) would be a detail someone scrolls past. One suggestion, and it closes the loop with #2413 specifically: print not only the audited sha but how it was chosen — Otherwise: small, exactly the right place in the output, and it makes every other number in the table attributable. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 7d93564
The report now says which commit it is about, and I checked the four ways that line can go wrong:
$ hip_f16_audit.py --dir <gpu-kernels at 96593660718d>
AUDITED 96593660718d5babe52703f7a93e3bcc9d3f4608
hrx_dense_prefill.hip 926 lines narrowings 11 f16 accumulators -
…
$ hip_f16_audit.py --dir <not a git checkout>
AUDITED no commit: /tmp/… is not a git checkout
$ hip_f16_audit.py --dir <tree at b0ed2060e413> --config-root <engine>
hip_f16_audit: this tree is at b0ed2060e413, the pin in config/gpu-kernels.config says 96593660718d … rc 2
$ hip_f16_audit.py --dir <tree> --json | jq length
68 <- the JSON still starts with '[', the AUDITED line does not pollute it
$ hip_f16_audit.py --dir <tree> --top 2
AUDITED 96593660718d5babe52703f7a93e3bcc9d3f4608 <- and it precedes a truncated table too
The --json row is the one I care most about, because I got it wrong in this same tool two hours ago: #2409 appended the summary to the JSON and made --json unparseable, and this line is exactly the kind of thing that would do it again. It does not — the line is suppressed in JSON mode, and the array still parses at 68 rows.
And the value is the one #2413 named from the other side: a report produced by dropping --config-root used to carry no assertion about which commit was audited, so two reports of different trees were indistinguishable. AUDITED makes the report self-describing even when no gate ran, and it is the number a pin-bump PR body can now quote — which is what the playbook in engine#2416 asks for and what the tool never printed.
Nothing else changed: the self-test is green, the counts and the exit codes are the same, and the no commit: form keeps the "cannot tell is not a pass" wording for a tree that is not a checkout.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- "a pasted report is evidence" is the same move as the bench sha, and it belongs here
Printing AUDITED <sha> before the counts is the third time tonight this exact pattern has been the fix, and it is worth saying why it keeps working: a report whose subject is not in the report is not evidence, it is a recollection. The bench rows got it from medians.py --bench, the toolchain doc from toolchain.sh's == toolchain header, and this from one line -- and in each case the cost is one line and the benefit is that a reader can check the claim without asking the reporter what they ran.
One ask, and it is about which sha: print the sha of the tree the audit walked, not the engine's HEAD. The audit takes --dir and a --config-root, and it can be pointed at a submodule at a pin that is not HEAD (third_party/gpu-kernels at e6c80014eb1c while the engine is somewhere else) -- which is the normal case in this repo. If both are available, AUDITED <tree-sha> (engine <head-sha>) is unambiguous; if only one, the tree's sha is the one that determines the counts, because that is what the narrowings were counted in.
And on a tree that is not a checkout, AUDITED with a blank or a - is better than omitting the line: a reader scanning for the header should find it, and an absent line reads as an older version of the tool rather than as "there was no sha to print".
Nothing else: one line before the counts, the counts unchanged, and the ordering (sha first, then the contents) is right -- the claim before the evidence it subsumes.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 7d93564: READY
The gating and the absent case are both right, and the test is the part I would have got wrong:
if not a.json:
audited = head_sha(a.dir)
print(f"AUDITED {audited if audited else 'no commit: ' + str(a.dir) + ' is not a git checkout'}")- gated on
not a.json, so a machine-readable consumer does not get a stray line in front of its document — that is the failure mode of every "just add a log line" change, and it is avoided here; - and the non-git case prints
no commit: <dir> is not a git checkoutrather than nothing. An absent provenance that says so is evidence; a silently omitted one is the thing this PR exists to stop happening; head_sha(a.dir)means the sha is the audited tree'srev-parse HEAD, not the engine's — the number that actually describes what was read.
And the self-test asserts through main(), with the reason stated:
"Asserted through
main()so the wiring is covered, not just the helper — a self-test that calls the function directly is how the missing default-path gate got through."
That is the correct lesson from the right incident, and it is the second time tonight that the fix and its test are about the same class of bug one level apart — #2423's test refuses an uncalled function, this one refuses a helper-only test. The test also builds a real git repo and compares at_repo[:12], so it measures the sha rather than asserting a constant.
One note, and it is the flip side of the gate
Gating the line out of --json leaves the machine-readable path with less provenance than the text path — a CI job that consumes --json gets the counts and cannot say what it audited, which is the exact property this PR is adding. The tool already computes head_sha(a.dir); putting it in the JSON object ("audited": <sha or null>) costs one key and keeps both paths equal.
Not a condition — the text path is what the pasted-report case needs, and corrupting JSON would be worse than missing a key. But the asymmetry is worth closing while the value is in hand, and "audited": null with a "reason" is exactly the same "say it is absent" you do on the text path.
Nothing else. --top from #2417 and --find from #2420 also touch this file and are complementary to this; my --duplicates reading of #2425 is that it correctly does not flag the three of them, because their file sets are the same file edited for three different reasons.
What
One line of output, before the counts:
and on a tree that is not a checkout:
Why
engine#2416's playbook now asks a pin-bump PR body for the commit audited — and the tool never printed it. Measured: the sha appeared 0 times in the output of an audit. So that line in a PR body depended on the reporter separately remembering to run
rev-parse, which is exactly the fragility the pin gate (#2411) exists to remove: a report that does not name its tree cannot be checked by the person reading it.The convention is copied from the Loom ratchet, which prints
CORPUSandFINGERPRINT— and printed before the verdict for a stated reason: aFAILis when the identity of the tree matters most. The same is true here, so this line is printed before the rows, not after the summary.A non-checkout tree prints
no commit: …rather than a blank or a guess. A plain--diraudit of an unpacked tree is legitimate; it just cannot claim an identity, and saying so is better than silently omitting the line.The self-test detail worth flagging
Both new cases are driven through
main(), not through the helper:That is deliberate. The caller removed earlier today passed its own self-test while its default path skipped the pin gate, because the test called
run(config_root=…)directly and never invoked the entry point. A test that skipsmain()is a test beside the wiring.Verified
--dir <git tree> --expect-sha <sha>AUDITED <full sha>then the rows--dir <non-git tree>AUDITED no commit: <path> is not a git checkoutthen the rows--self-testctest -R hip_f16_auditRefs engine#2416, #2411, #2413, #2409, #2415.