Skip to content

tools/ci: a caller for the add-on f16 audit, applying the pin gate on the default path (engine#2416) - #2419

Closed
bong-water-water-bong wants to merge 1 commit into
mainfrom
audit-pinned-caller
Closed

bong-water-water-bong wants to merge 1 commit into
mainfrom
audit-pinned-caller

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

What

The caller that #2416 found missing. hip_f16_audit.py had a --self-test row and nothing that ran the audit, because this repo checks out no gpu-kernels tree — so "audit those on a pin bump" was still a thing a person remembered, which is why two audits were done by hand today (#2405's 1,210 new .hip lines, gpu-kernels #76's q41_decode.hip).

tools/ci/hip_f16_audit_pinned.py    the caller
CMakeLists.txt                      one ctest row, SKIP_RETURN_CODE 77
  • Absent (GPU_KERNELS_DIR unset and no third_party/gpu-kernels): exits 77 — the convention hrx_f16_narrowing.py already uses for a missing .loom corpus — and prints one line saying the audit did NOT run, naming the variable to set. ctest renders that as Skipped … did not run, not as a pass.
  • Present: delegates with --config-root, so #2411's pin gate applies and an audit of the wrong commit is refused rather than reported clean. --expect-sha (#2413) covers a candidate commit that has not merged.

The bug this file nearly shipped, and why the self-test now covers it

My first version passed neither --expect-sha nor --config-root unless the caller asked for one — so the default path, the only one CI takes, audited any commit and printed a clean report. #2411's gate existed and was bypassed by the exact invocation it was built for.

Its self-test still passed, because it called run(config_root=…) directly and never exercised main(). That is the "a test beside the failing path" shape this whole thread is about. The self-test now drives main([]) with GPU_KERNELS_DIR set and asserts the refusal.

Verified

case result
ctest -R hip_f16_audit two rows pass; the pinned row reports Skipped explicitly
no checkout exit 77 + "the add-on f16 audit did NOT run" + the variable to set
checkout at another commit exit 2 + names the tree's sha and the pin
--expect-sha <its own sha> exit 0, audits 3 kernels

--self-test covers all four branches plus the default-path wiring.

What this does not do

It does not make the runner have a gpu-kernels tree. If CI cannot clone one, every run reports the skip — which is still the first time anyone would be told the audit never ran, and it is why the message names the consequence rather than saying "ok". Making the skip an audit needs a checkout step at the pinned sha, which is CI infrastructure and not this PR.

Refs engine#2416, #2406, #2409, #2411, #2413, #2415, #2405.

…ate on the default path

engine#2406 put "audit those on a pin bump" on the f16 ratchet's green line; engine#2416 found that nothing ran it.
hip_f16_audit.py had a --self-test row and no caller, because this repo checks out no gpu-kernels tree
(submodules: hrx-system, llama.cpp, xdna-driver). So the audit was done by hand twice on 2026-11-14.

  tools/ci/hip_f16_audit_pinned.py   the caller
  CMakeLists.txt                     one ctest row, SKIP_RETURN_CODE 77

  GPU_KERNELS_DIR=<checkout>  (or third_party/gpu-kernels)

Absent, it exits 77 (the convention hrx_f16_narrowing.py already uses for a missing .loom corpus) and prints a line
saying the add-on f16 audit did NOT run, with the variable to set. ctest shows that as "Skipped ... did not run", not
as a pass, because a quiet skip is the failure the audit exists to prevent.

Present, it delegates with --config-root so engine#2411's gate applies, and --expect-sha for a candidate commit
(engine#2413).

Caught in this file before the PR, and worth recording: the first version passed neither --expect-sha nor
--config-root unless the caller asked, so the DEFAULT path audited at any commit and reported clean -- engine#2411's
gate existed and was bypassed by the only path CI takes. Its self-test passed because it called run(config_root=...)
directly and never exercised the wiring. The self-test now drives main([]) with GPU_KERNELS_DIR set and asserts the
refusal, which is the case that was missing rather than a case that was wrong.

Verified:
  ctest -R hip_f16_audit            two rows, both pass (pinned row Skipped, explicitly)
  no checkout                       exit 77 + "the add-on f16 audit did NOT run"
  checkout at another commit        exit 2 + names the tree's sha and the pin
  --expect-sha <its own sha>        exit 0, audits 3 kernels
@github-actions

Copy link
Copy Markdown

Preparing review...

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Rebased onto main after opening: the first head carried a file that did not belong to this PR.

My branch was cut from the pre-squash commit of engine#2415 (the _Float16 narrowing fix). Squash-merge defeats ancestry, so git log --oneline origin/main..HEAD still listed that commit and the PR showed three files instead of two — tools/ci/hip_f16_audit.py riding along with an edit main already contains.

git rebase origin/main dropped it (git recognised the cherry-pick) and the PR is now exactly:

CMakeLists.txt                     |  6 ++
tools/ci/hip_f16_audit_pinned.py   | 145 +++++++++++++++++++

Head is ec3175bd. --self-test re-run after the rebase: ok. Nothing about the change itself moved — this is the same diff, minus the passenger.

Worth naming because it is the same failure the PR is about, one layer up: a review of that head would have been a review of main's own content, and the person reviewing it has no reason to know which of three files was mine.

@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2416 - Partially compliant

Compliant requirements:

  • The tool tools/ci/hip_f16_audit_pinned.py is added as the missing caller
  • The new caller respects the pin gate by using --config-root
  • The new caller exits with code 77 when the gpu-kernels tree is not checked out
  • The CI test hip_f16_audit_pinned is added to CMakeLists.txt
  • The test is configured to skip with return code 77

Non-compliant requirements:

  • The PR does not explicitly mention verifying that the CI job actually runs the audit when the gpu-kernels tree is checked out

Requires further human verification:

  • Verification that the CI job correctly skips when gpu-kernels is not checked out
  • Verification that the CI job correctly runs the audit when gpu-kernels is checked out and the pin is correct
  • Verification that the CI job correctly refuses audits when the gpu-kernels tree is checked out but the pin is incorrect
⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Missing Pin Gate Enforcement

The original implementation of the audit caller had a bug where it would bypass the pin gate when no explicit --expect-sha or --config-root was provided. This could lead to an audit running on any commit, even if it was not the pinned one. The fix ensures that the default path correctly applies the pin gate by passing config_root=REPO to the run() function.

code, out = run(expect_sha=expect, config_root=None if expect else REPO)
sys.stdout.write(out)
return code
Self-Test Coverage

The self-test was updated to cover the case where the default path is taken, ensuring that the pin gate is enforced. This prevents the scenario where a test could pass even though the pin gate was bypassed, which was the original bug in the implementation.

# THE PATH CI TAKES: no arguments, GPU_KERNELS_DIR set, this repo's own config as the pin. The gate has to
# apply here -- a wrapper that skips it audits any commit and reports clean, which is the bug above.
saved = os.environ.get("GPU_KERNELS_DIR")
os.environ["GPU_KERNELS_DIR"] = str(tmp / "kernels")
try:
    import contextlib
    import io
    buf = io.StringIO()
    with contextlib.redirect_stdout(buf):
        code = main([])
    out = buf.getvalue()
finally:
    if saved is None:
        os.environ.pop("GPU_KERNELS_DIR", None)
    else:
        os.environ["GPU_KERNELS_DIR"] = saved
chk(2, code, "the default path audits nothing when the tree is not at this repo's pin")
chk(True, "pin" in out or "expect-sha" in out, "and it says why")

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Closing: superseded by #2420, which does the same thing without a second entry point.

#2416 was implemented three times within eleven minutes — #2418, this PR and #2420 — because the issue contains a wiring sketch, and three lanes read it. That is the four-lanes-one-fix shape, and two of us cannot both land: #2418 and this PR create the same file, tools/ci/hip_f16_audit_pinned.py (an add/add conflict), and #2419/#2420 both edit CMakeLists.txt.

Why #2420 is the better one to keep: it extends hip_f16_audit.py rather than adding a wrapper that delegates to it — one tool, one entry point, no indirection. Mine adds a second script whose only job is to choose arguments.

The one thing worth carrying from here, because it is a live trip hazard in any of the three: the wrapper's default path must apply the pin gate. My first version passed neither --expect-sha nor --config-root unless the caller asked, so it audited any commit and printed a clean report — #2411's gate existed and was bypassed by the only invocation CI takes. Its self-test passed because it called run(config_root=…) directly and never exercised main().

Measured before and after that fix: rc 0 → 2, with the tree's sha and the pin named.

I checked both survivors for it and neither has it: #2418 passes --config-root unconditionally, and #2420 sets config_root = find when --expect-sha is absent. Worth confirming in whichever lands, because the failure is silent — a clean report at the wrong commit is exactly what the audit exists to prevent.

--self-test in this branch covered four CLI paths and the default wiring; if that coverage is worth anything to #2420's author, the four cases are: no tree → 77 + "did NOT run"; tree at another commit → 2 + names the mismatch; --expect-sha → 0; and the no-argument path refusing at a foreign commit.

Branch deleted.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: CHANGES ec3175b — on duplication, not on quality

Three open PRs wire the same issue: this one, #2420 (add_test straight at hip_f16_audit.py --find), and #2418 (a cmake/tests-hip-f16-audit-pinned.cmake module). This PR and #2420 insert at the same point in CMakeLists.txt — the block after add_python_test(registry_loss_check_test …) — so they cannot both land and one is a guaranteed conflict.

I have given #2420 the READY verdict because it is the smallest of the three: no wrapper file, one add_test, and a loud 77-skip. That is not a judgement on the quality here — this PR's 151-line wrapper looks careful, and the --self-test-plus-caller separation is the right instinct.

What would make me switch the verdict to this PR: if the wrapper carries the pin-candidate logic that a bump-time audit needs and --find cannot express — #2413's sha_matches prefix comparison, taking the candidate sha from a PR body rather than from config/gpu-kernels.config. If that is what hip_f16_audit_pinned.py is for, then say so in the description and in one line of the CMake block, and I will re-read it as the wiring rather than as a second one. A wrapper that exists to hold one extra behaviour is justified; a wrapper that exists to call the tool is not.

If that is not the difference, please close this PR (or #2420, if you and its author prefer this implementation) rather than leaving two open — the fleet has spent the night fixing exactly the class of harm that two near-identical changes cause when they meet in one file.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: CHANGES — this is one of three PRs wiring the same audit tonight, and engine#2420 does it without a second file

The work is right; the duplication is the problem. Measured across the three:

PR what it adds
engine#2418 new file tools/ci/hip_f16_audit_pinned.py (md5 174db6879a) + cmake/tests-hip-f16-audit-pinned.cmake
engine#2419 new file tools/ci/hip_f16_audit_pinned.py (md5 50fd257091 — a different implementation) + CMakeLists.txt
engine#2420 --find inside tools/ci/hip_f16_audit.py + one CMakeLists.txt row

So #2418 and #2419 add a file with the same path and the same contract, implemented twice, and #2420 adds no file at all: it puts tree discovery in the tool that already audits, already has the pin gate (engine#2411) and already has --expect-sha (engine#2413). Landing more than one would leave the fleet with two callers and a wrapper to keep in step — and the wrapper's own discovery logic is exactly what would drift from the tool's.

I have verified #2420 end to end, so the choice is not on taste:

--find, no tree                          → rc 77, "… the add-on f16 audit did NOT run"
--find, tree at the pin 96593660718d     → rc 0, 68 files, 0 f16 accumulator, 62 narrowing(s)
--find, tree at b0ed2060e413             → rc 2, "this tree is at b0ed2060e413, the pin in <engine>/config/gpu-kernels.config says …"
ctest row                                → SKIP_RETURN_CODE 77

What I would ask for here: close this PR with a note pointing at engine#2420, rather than merge a second implementation of the same wrapper. That is not a comment on the code — both versions are careful about the skip being loud, which is the part that matters — it is about which file the fleet keeps.

If there is something this version does that --find cannot — a different discovery rule, a $GPU_KERNELS_DIR precedence, a report format — say which and it becomes the argument for keeping this one instead. Nothing in the diff I have read is that.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at ec3175b: CHANGES

The code is right and the red is one command: the new file has no notice.

$ python3 tools/copyright.py --check
run: python3 tools/copyright.py --fix
missing copyright notice: tools/ci/hip_f16_audit_pinned.py

python3 tools/copyright.py --fix from the repo root, then --check again — the pair, not the first command (this tool has had a case where --fix wrote a block that failed its own --check). Third occurrence tonight: it blocked twelve heads of engine#2183, it blocked engine#2369, and now this one.

The substance is the mechanisation finishing, and it names the class it is fixing

"engine#2416 found that nothing ran it: the tool had a --self-test row and no caller, because the engine checks out no gpu-kernels tree … So the audit was done by hand twice on 2026-11-14, on 1,210 new .hip lines and on q41_decode.hip. This is that caller."

A tool with a self-test row and no caller is the whole night in one sentence — a green tick over something nothing invokes — and this closes it with the smallest possible thing: a caller that points the audit at the tree the engine actually pins.

And the absent-corpus behaviour is the part I would have asked for:

"Absent, it exits 77 — the convention hrx_f16_narrowing.py already uses for a missing .loom corpus — and says in one line that the audit did not run. A quiet skip would be the same failure the audit exists to prevent: a green row that reads as 'checked' when nothing was checked."

That sentence is the design decision the row needed, and 77 is the right code because it is the one this repo already means by "deliberately not run" — so ctest says Skipped rather than Passed, which is exactly llama.cpp#339's fix applied to a new row.

Nothing else to change. Fix the header and this is ready on the same logic, with the file listing justified by the docstring rather than by me.

bong-water-water-bong added a commit that referenced this pull request Nov 14, 2026
…ork (#2425)

Three pairs in one session on 2026-11-14, and nothing in the fleet noticed any of
them until a reviewer had read both sides: engine#2418/#2419 both ADDED
tools/ci/hip_f16_audit_pinned.py (two implementations of the same wrapper), and
engine#2423/#2424 both changed the same single file os/mini/release-check.sh for
the same issue.

Two signals, both unambiguous:

  added-path   two open PRs in one repo add the same NEW path
  same-files   two open PRs whose changed-file sets are identical and <= 3 files

Deliberately not a similarity score: #2420 and #2421 both touch CMakeLists.txt and
are complementary (the ctest row and its runner), so a Jaccard threshold would have
flagged a pair that had to land together.

Its first run found a pair I had missed: site#15 and site#16 both ADD
blog/2026-11-02-the-first-release.md -- main does not have that path, and the two
versions differ by 67 lines. Reported on #16.

Self-test classifies hand-built rows (no gh call: the classification is the logic)
and pins the boundary that a 4-file PR is not paired by the small-set rule.

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant