Skip to content

hip_f16_audit: print AUDITED <sha> before the counts, so a pasted report is evidence (engine#2416) - #2426

Merged
bong-water-water-bong merged 1 commit into
mainfrom
audit-print-sha
Nov 14, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
audit-print-sha

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

What

One line of output, before the counts:

AUDITED      7d63abfe239e39436847334e70afd46b121b8b25
  fa_prompt_q8w.hip                    336 lines  narrowings   2  f16 accumulators -
  moe_down_q4k.hip                     324 lines  narrowings   1  f16 accumulators -

and on a tree that is not a checkout:

AUDITED      no commit: <path> is not a git checkout

Why

engine#2416's playbook now asks a pin-bump PR body for the commit audited — and the tool never printed it. Measured: the sha appeared 0 times in the output of an audit. So that line in a PR body depended on the reporter separately remembering to run rev-parse, which is exactly the fragility the pin gate (#2411) exists to remove: a report that does not name its tree cannot be checked by the person reading it.

The convention is copied from the Loom ratchet, which prints CORPUS and FINGERPRINT — and printed before the verdict for a stated reason: a FAIL is when the identity of the tree matters most. The same is true here, so this line is printed before the rows, not after the summary.

A non-checkout tree prints no commit: … rather than a blank or a guess. A plain --dir audit of an unpacked tree is legitimate; it just cannot claim an identity, and saying so is better than silently omitting the line.

The self-test detail worth flagging

Both new cases are driven through main(), not through the helper:

with contextlib.redirect_stdout(buf):
    main(["--dir", str(repo)])
chk(True, "AUDITED" in buf.getvalue() and at_repo[:12] in buf.getvalue(), ...)

That is deliberate. The caller removed earlier today passed its own self-test while its default path skipped the pin gate, because the test called run(config_root=…) directly and never invoked the entry point. A test that skips main() is a test beside the wiring.

Verified

invocation first line
--dir <git tree> --expect-sha <sha> AUDITED <full sha> then the rows
--dir <non-git tree> AUDITED no commit: <path> is not a git checkout then the rows
--self-test ok
ctest -R hip_f16_audit passes

Refs engine#2416, #2411, #2413, #2409, #2415.

…ort is evidence

engine#2416's playbook now asks a pin-bump PR body for "the commit audited", and the tool never printed it: measured
on the from-pointer tree, the sha appeared 0 times in the output. So that line depended on the reporter remembering to
run rev-parse as well, which is the fragility the pin gate (engine#2411) exists to remove -- a report that does not
name its tree cannot be checked by the person reading it.

  AUDITED      7d63abfe239e39436847334e70afd46b121b8b25
    fa_prompt_q8w.hip                    336 lines  narrowings   2  f16 accumulators -
    ...

Same convention as the Loom ratchet's CORPUS and FINGERPRINT lines, and printed BEFORE the verdict for the same reason
stated there: a FAIL is when the identity of the tree matters most.

A tree that is not a git checkout prints "AUDITED no commit: <path> is not a git checkout" rather than a blank or a
guess -- an audit of a tree whose commit is unknown is still a legitimate thing to run (plain --dir), it just cannot
claim an identity.

--self-test gains two cases, both driven through main() rather than the helper: the report names the commit it audited,
and a non-checkout tree says so instead of printing a sha. Driving main() is deliberate -- the caller removed earlier
today passed its own self-test while its default path skipped the pin gate, because the test called the function
directly and never the entry point.

Verified:
  --dir <git tree> --expect-sha <sha>   -> AUDITED <sha> then the rows
  --dir <non-git tree>                  -> AUDITED no commit: ... then the rows
  --self-test                           -> ok
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Missing Git Commit Check

The code assumes that if a directory is not a git checkout, it will not have a commit SHA. However, it does not validate that the directory is actually a git repository before attempting to retrieve the commit SHA. This could lead to incorrect behavior if the directory is not a git checkout but still has a valid commit hash in some other way.

audited = head_sha(a.dir)
print(f"AUDITED      {audited if audited else 'no commit: ' + str(a.dir) + ' is not a git checkout'}")

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 7d93564

This is the ask I made on #2411 and repeated on #2413 — "print which sha was audited, so a run that audited the wrong candidate is diagnosable after the fact rather than only at the time" — and it is done the right way round:

AUDITED      7d63abfe239e39436847334e70afd46b121b8b25
  fa_prompt_q8w.hip                    336

The sha comes first, before the counts. That ordering matters: a reader who sees a table of per-file narrowing counts has no way to know which tree produced them unless the identity is above the data, and a 40-character sha at the top is exactly the line that makes a saved output auditable months later. Printing it after the table (or in a trailer) would be a detail someone scrolls past.

One suggestion, and it closes the loop with #2413 specifically: print not only the audited sha but how it was chosen — --pin from config/gpu-kernels.config versus a candidate passed for a pre-merge bump. Those are the two modes, they answer different questions, and a run that audited the current pin when the operator meant the candidate is indistinguishable from a correct run in the output as it stands. One token ((from config) / (from candidate)) on the same line makes the provenance legible, and provenance is the entire subject of this tool's recent history.

Otherwise: small, exactly the right place in the output, and it makes every other number in the table attributable.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 7d93564

The report now says which commit it is about, and I checked the four ways that line can go wrong:

$ hip_f16_audit.py --dir <gpu-kernels at 96593660718d>
AUDITED      96593660718d5babe52703f7a93e3bcc9d3f4608
  hrx_dense_prefill.hip                926 lines  narrowings  11  f16 accumulators -
  …

$ hip_f16_audit.py --dir <not a git checkout>
AUDITED      no commit: /tmp/… is not a git checkout

$ hip_f16_audit.py --dir <tree at b0ed2060e413> --config-root <engine>
hip_f16_audit: this tree is at b0ed2060e413, the pin in config/gpu-kernels.config says 96593660718d …   rc 2

$ hip_f16_audit.py --dir <tree> --json | jq length
68                                    <- the JSON still starts with '[', the AUDITED line does not pollute it

$ hip_f16_audit.py --dir <tree> --top 2
AUDITED      96593660718d5babe52703f7a93e3bcc9d3f4608      <- and it precedes a truncated table too

The --json row is the one I care most about, because I got it wrong in this same tool two hours ago: #2409 appended the summary to the JSON and made --json unparseable, and this line is exactly the kind of thing that would do it again. It does not — the line is suppressed in JSON mode, and the array still parses at 68 rows.

And the value is the one #2413 named from the other side: a report produced by dropping --config-root used to carry no assertion about which commit was audited, so two reports of different trees were indistinguishable. AUDITED makes the report self-describing even when no gate ran, and it is the number a pin-bump PR body can now quote — which is what the playbook in engine#2416 asks for and what the tool never printed.

Nothing else changed: the self-test is green, the counts and the exit codes are the same, and the no commit: form keeps the "cannot tell is not a pass" wording for a tree that is not a checkout.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- "a pasted report is evidence" is the same move as the bench sha, and it belongs here

Printing AUDITED <sha> before the counts is the third time tonight this exact pattern has been the fix, and it is worth saying why it keeps working: a report whose subject is not in the report is not evidence, it is a recollection. The bench rows got it from medians.py --bench, the toolchain doc from toolchain.sh's == toolchain header, and this from one line -- and in each case the cost is one line and the benefit is that a reader can check the claim without asking the reporter what they ran.

One ask, and it is about which sha: print the sha of the tree the audit walked, not the engine's HEAD. The audit takes --dir and a --config-root, and it can be pointed at a submodule at a pin that is not HEAD (third_party/gpu-kernels at e6c80014eb1c while the engine is somewhere else) -- which is the normal case in this repo. If both are available, AUDITED <tree-sha> (engine <head-sha>) is unambiguous; if only one, the tree's sha is the one that determines the counts, because that is what the narrowings were counted in.

And on a tree that is not a checkout, AUDITED with a blank or a - is better than omitting the line: a reader scanning for the header should find it, and an absent line reads as an older version of the tool rather than as "there was no sha to print".

Nothing else: one line before the counts, the counts unchanged, and the ordering (sha first, then the contents) is right -- the claim before the evidence it subsumes.

@bong-water-water-bong
bong-water-water-bong merged commit 5050a90 into main Nov 14, 2026
9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the audit-print-sha branch November 14, 2026 11:21

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 7d93564: READY

The gating and the absent case are both right, and the test is the part I would have got wrong:

if not a.json:
    audited = head_sha(a.dir)
    print(f"AUDITED      {audited if audited else 'no commit: ' + str(a.dir) + ' is not a git checkout'}")
  • gated on not a.json, so a machine-readable consumer does not get a stray line in front of its document — that is the failure mode of every "just add a log line" change, and it is avoided here;
  • and the non-git case prints no commit: <dir> is not a git checkout rather than nothing. An absent provenance that says so is evidence; a silently omitted one is the thing this PR exists to stop happening;
  • head_sha(a.dir) means the sha is the audited tree's rev-parse HEAD, not the engine's — the number that actually describes what was read.

And the self-test asserts through main(), with the reason stated:

"Asserted through main() so the wiring is covered, not just the helper — a self-test that calls the function directly is how the missing default-path gate got through."

That is the correct lesson from the right incident, and it is the second time tonight that the fix and its test are about the same class of bug one level apart — #2423's test refuses an uncalled function, this one refuses a helper-only test. The test also builds a real git repo and compares at_repo[:12], so it measures the sha rather than asserting a constant.

One note, and it is the flip side of the gate

Gating the line out of --json leaves the machine-readable path with less provenance than the text path — a CI job that consumes --json gets the counts and cannot say what it audited, which is the exact property this PR is adding. The tool already computes head_sha(a.dir); putting it in the JSON object ("audited": <sha or null>) costs one key and keeps both paths equal.

Not a condition — the text path is what the pasted-report case needs, and corrupting JSON would be worse than missing a key. But the asymmetry is worth closing while the value is in hand, and "audited": null with a "reason" is exactly the same "say it is absent" you do on the text path.

Nothing else. --top from #2417 and --find from #2420 also touch this file and are complementary to this; my --duplicates reading of #2425 is that it correctly does not flag the three of them, because their file sets are the same file edited for three different reasons.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant