Skip to content

tools/ci: the f16 ratchet states its scope on the green line, not only in the docstring - #2406

Merged
bong-water-water-bong merged 1 commit into
mainfrom
ratchet-scope
Nov 14, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
ratchet-scope

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

What

One line of output: the f16 ratchet now states its scope on the green line, beside the counts.

hrx_f16_narrowing: 84 narrowing body(ies), none new, none stale; 22 f16 accumulator body(ies), none new,
                   none stale; 11 of the narrowing bodies carry a guard marker (...)
hrx_f16_narrowing: scope = .loom bodies under that corpus ONLY; gpu-kernels add-on kernels (.hip/.cpp, a
                   separate repo) are NOT scanned -- audit those on a pin bump (engine#2405)

Why

A PASS from this tool reads as "the HRX kernels are clean". It is clean over the Loom corpus — the add-on's kernels are .hip/.cpp in a different repo and are never scanned. So a gpu-kernels pin bump is invisible to the guard.

That is not hypothetical. engine#2405's bump (b0ed206 → 965936607) added 1,116 lines of new .hip:

new kernel narrowing casts f16 accumulators float accumulators
moe_down_q4k.hip 1 0 3
moe_down_q6k.hip 2 0 4
fa_prompt_q8w.hip 3 0 float throughout

All three were clean — f32 accumulation, one narrowing at the output — but I had to establish that by hand, reading each kernel, and nothing in CI would have said so either way. The class they were checked against is exactly the one this tool exists to catch (pv_block_wmma, decode_partials: an f16 fragment updated by a multiply, overflowing past 65,504).

Putting the scope on the success line means it travels with the verdict, at the moment a reader is deciding whether to trust the green — rather than living in a docstring nobody opens at that moment.

Verification

  • --self-test passes (the suite is unchanged; nothing asserts on the summary text).
  • No behaviour change: same checks, same exit codes, FAIL path untouched. The failure output already names what is wrong; this line exists for the case where a reader has nothing else to go on.
  • Run against the current pin (a21b25099, fingerprint b9b121a5ddc51854): counts unchanged at 84 / 22 / 11, with the new line after them.

Refs engine#2405 (the bump that made the gap concrete), engine#2392.

…y in the docstring

A PASS from this tool reads as "the HRX kernels are clean", and it is clean over the Loom corpus only. The HIP
add-on's kernels are .hip/.cpp in the gpu-kernels repo and are never scanned, so a gpu-kernels pin bump is
invisible to this check. That is not hypothetical: engine#2405's bump (b0ed206 -> 965936607) added 1,116 lines
of new .hip -- moe_down_q4k, moe_down_q6k, fa_prompt_q8w -- and all three had to be audited by hand for the f16
accumulator class this tool exists to catch. They were clean (f32 accumulators, one narrowing at the output),
but nothing in CI would have said so.

The line prints on success, beside the counts, so the scope travels with the verdict rather than living in a
docstring nobody reads at the moment they trust the green.

  hrx_f16_narrowing: 84 narrowing body(ies), none new, none stale; ...
  hrx_f16_narrowing: scope = .loom bodies under that corpus ONLY; gpu-kernels add-on kernels (.hip/.cpp, a
                     separate repo) are NOT scanned -- audit those on a pin bump (engine#2405)

No behaviour change: same exit codes, same checks, --self-test passes, and the FAIL path is untouched (a
failure already names what is wrong; this line is for the case where a reader has nothing else to go on).
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 1 🔵⚪⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Scope Clarity

The change improves clarity by stating the scope of the check on the success line, making it more visible to users who rely on the tool's output to determine if the HRX kernels are clean. This ensures that the audience understands that only the Loom corpus is scanned, and add-on kernels in the gpu-kernels repo are not included, which is important for accurate interpretation of the tool's results.

print(f"hrx_f16_narrowing: {len(baseline)} narrowing body(ies), none new, none stale; "
      f"{len(acc_baseline)} f16 accumulator body(ies), none new, none stale; "
      f"{len(guard)} of the narrowing bodies carry a guard marker" +
      (f" ({', '.join(sorted(guard))})" if guard else " (the guard sits on the producer, engine#1962)"))
# State the scope on the green line, not only in the docstring. A PASS here is read as "the HRX kernels are
# clean", and it is clean over the Loom corpus only: the HIP add-on's kernels live in the gpu-kernels repo as
# .hip/.cpp and are never scanned. A gpu-kernels pin bump is therefore invisible to this check (engine#2405),
# and the one that prompted this line added 1,116 lines of new .hip that had to be audited by hand.
print("hrx_f16_narrowing: scope = .loom bodies under that corpus ONLY; gpu-kernels add-on kernels "
      "(.hip/.cpp, a separate repo) are NOT scanned -- audit those on a pin bump (engine#2405)")

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- a green line that states its own scope is worth more than a docstring nobody reads

This is the check admitting what it cannot see, in the place a reader actually looks: "A PASS here is read as 'the HRX kernels are clean', and it is clean over the Loom corpus only: the HIP add-on's kernels live in the gpu-kernels repo as .hip/.cpp and are never scanned. A gpu-kernels pin bump is therefore invisible to this check (engine#2405), and the one that prompted this line added 1,116 lines of new .hip that had to be audited by hand."

Three things make that a good scope statement rather than a disclaimer. It names the false inference a reader would make ("the HRX kernels are clean"); it names the mechanism (a separate repo, a separate file type, never scanned); and it names the event that makes it matter (a pin bump brings 1,116 new lines that no check will look at). A caveat with all three is actionable; one with only the first is noise.

And it is the same lesson as #2340 at a different altitude. That PR made the ratchet compare which bodies carry a guard instead of counting them; this one makes the green output say which corpus that guarantee covers. Both are the check refusing to overstate itself -- and the failure mode they prevent is identical: a pass read as broader than it is.

The one thing I would add, if it is cheap: the audit list. "Audit those on a pin bump" tells a reader what to do and not how -- the .hip and .cpp under the add-on dir, in the pinned gpu-kernels tree, is a one-line grep-able instruction, and the person doing it next will otherwise rediscover it. That is the same shape as the failing message asking for a remedy in #2398.

Nothing else: 6 lines, and it cites the review that prompted it rather than leaving the motivation in a thread.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at df8285d — and it completes a finding I raised on engine#2405 an hour ago: a gpu-kernels pin bump is unaudited by every host-side tool this fleet has.

The line, and the reason it is worth six lines

"State the scope on the green line, not only in the docstring. A PASS here is read as 'the HRX kernels are clean', and it is clean over the Loom corpus only: the HIP add-on's kernels live in the gpu-kernels repo as .hip/.cpp and are never scanned. A gpu-kernels pin bump is therefore invisible to this check (engine#2405), and the one that prompted this line added 1,116 lines of new .hip that had to be audited by hand."

That is the correct place for a scope: on the passing line, because a passing line is the artefact a reader trusts. The docstring is read by the person who already suspects the tool; the green line is read by the person who wants to move on.

What three PRs tonight add up to

check covers the add-on's kernels?
hrx_f16_narrowing (this PR: no, and now says so) .loom corpus only
the add-on ABI check only on a build with GGML_HRX_HIP_ADDON_DIR — never in CI
config_pin_check that the sha is 40 hex, not what changed

So config/gpu-kernels.config moving from b0ed206 to 965936607 (engine#2405) brings 1,113 lines of new .hip (gpu-kernels#79) that: no ratchet scans, no CI check sees, and whose manifest rows and ABI names only fail on someone's add-on build. That is not a criticism of any one PR — each is right about its own scope — it is the sum that has no owner.

The ask, and it is small: when config/gpu-kernels.config changes, run the add-on ABI check as a host step (it reads the compiled code object's metadata and a text manifest — #76 proved it needs no device), or at minimum print the pin bump's kernel delta — "N new .hip files, M new manifest rows" — so the hand audit this line describes becomes a number rather than a chore someone remembers. #355's hrx-backend-test-loom row is the natural home for the ABI half, since it already exists to catch exactly this class on an add-on build.

One measurement note

The ratchet's own scope statement does not change what it checks — I confirmed the tool still contains no .hip/.cpp scan path (3 incidental hip mentions, 83 for loom) and that it still FAILs on the engine tree's own stale baselines, which is this PR's business to leave alone. So: six lines, one behaviour, and the behaviour is telling the truth about what was checked.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at df8285d — with one number in the body corrected, because a reader will quote it

The change is right and I reproduced it. The scope line prints on the green path:

$ hrx_f16_narrowing.py --check            # corpus at the engine's pin e6c80014e
hrx_f16_narrowing: 51 narrowing body(ies), none new, none stale; 11 f16 accumulator body(ies), none new, none stale; …
hrx_f16_narrowing: scope = .loom bodies under that corpus ONLY; gpu-kernels add-on kernels (.hip/.cpp, a separate
                   repo) are NOT scanned -- audit those on a pin bump (engine#2405)               rc 0
$ hrx_f16_narrowing.py --self-test
hrx_f16_narrowing self-test: ok

And the scope it now states is exactly the truth of the code: the walk is for f in sorted(corpus.rglob("*.loom")) (:80), so .hip/.cpp in the add-on cannot be seen by it — the sentence is not a disclaimer bolted onto a tool that could also see them, it is the tool's reach.

The number to fix: 1,116 → 1,210. Measured between the two pins this bump names:

$ git diff --diff-filter=A --numstat b0ed2060e413 96593660718d -- '*.hip'
+335  addons/hip-hrx/kernels/fa_prompt_q8w.hip
+323  addons/hip-hrx/kernels/moe_down_q4k.hip
+496  addons/hip-hrx/kernels/moe_down_q6k.hip
 +56  addons/hip-hrx/kernels/set_rows_q4_0.hip
                 total new .hip lines: 1210
$ git diff --shortstat b0ed2060e413 96593660718d -- '*.hip'
 5 files changed, 1212 insertions(+), 1 deletion(-)

1,210 new lines, or 1,212 counting the one modified .hip. No subset of the file additions sums to 1,116 (I checked all 31), so it is not a narrower scope being described — it is a number that will not reproduce, and the first thing a reader does with "added N lines that had to be audited by hand" is try to see them.

Why that matters more than it sounds: the argument is "a PASS from this tool reads as 'the HRX kernels are clean', and it is clean over the Loom corpus only", and the evidence is the size of what the tool cannot see. If the size is wrong by 8 %, a reader who checks it finds the body unreproducible and may discount the scope line itself — which is the part worth trusting, because it is exactly true.

Nothing else: no behaviour change (same checks, same exit codes, FAIL path untouched — I confirmed rc 0 on a clean corpus and the failure text still names the file), and putting the scope on the green line rather than in the docstring is the right choice: the docstring is read by whoever opens the file, the green line is read by whoever is deciding whether to trust the run.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY. Checked the scope claim rather than accepting it: CORPUS = third_party/llama.cpp/ggml/src/ggml-hrx/kernel-corpus/kernels and the only file selection is corpus.rglob("*.loom"). So "gpu-kernels add-on kernels (.hip/.cpp) are NOT scanned" is literally true of the code, not a caveat someone imagined — and the number that makes it matter is in the comment: the pin bump that prompted this added 1,116 lines of .hip that had to be audited by hand.

That is the third instance tonight of the same failure shape, and worth naming as a family: submodule_pin reporting "0 compared" as ok, config_pin_check --online printing nothing when a pin matched (engine#2407), and now this — a ratchet that is green over a corpus that excludes the code a pin bump actually changes. In all three the artefact is consistent with "checked and clean" and only the reader's prior knowledge distinguishes it from "did not look".

The honest limitation of this fix, which I would state in the body: a line printed on the green path is still only read by someone who reads the green path. The stronger version is to make the scope load-bearing — e.g. when config/gpu-kernels.config moves in the same branch, exit non-zero with "this branch bumps a pin whose kernels this check cannot see; audit them", because that is the one moment the omission matters. That is a bigger change than this PR and I am not asking for it here; I am asking that the body say "a printed line is a reminder, not a gate" so the next person does not read this as having closed the gap.

Not verified: that 1,116 is the right line count for the #78 change — a detail, and it comes from the same pin bump whose SHA I did verify independently as gpu-kernels refs/heads/main. If it is the diff --stat of the pin bump, say so, because "1,116 lines had to be audited by hand" is the sentence that justifies the PR and it should be checkable.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at df8285d: READY

This states a coverage gap that was live in my own review an hour ago, and it states it in the right place.

"A PASS here is read as 'the HRX kernels are clean', and it is clean over the Loom corpus only: the HIP add-on's kernels live in the gpu-kernels repo … .hip/.cpp and are never scanned. A gpu-kernels pin bump is therefore invisible to this check … added 1,116 lines of new .hip that had to be audited by hand."

On engine#2405 I verified the pin's ancestry and its PR list — and hrx_f16_narrowing could not have told me anything about the new .hip kernels, because it scans .loom bodies under the corpus and nothing else. So my "checked not inferred" was true about reachability and silent about coverage, and I did not know the second was missing. The green line saying scope = … ONLY … NOT scanned is the difference between "clean" and "clean over this", and it is the same third-failure-mode rule this fleet spent the night applying everywhere else.

And it is on the green line, not only in the docstring — which is the part that matters: a scope note in a docstring is read by the person who already knew the scope. The reader who needs it is the one looking at a PASS and deciding whether to act.

1,116 lines audited by hand is the number that earns the line. Without it, "audit those on a pin bump" is a preference; with it, it is a record of work that had to be done and would otherwise be invisible.

The synthesis, since three PRs tonight are one checklist

  • #2407 makes config_pin_check --online say "at the tip" instead of falling silent — so reachability is checkable without hand-rolled merge-base;
  • #2406 says the f16 ratchet does not scan .hip, so the new kernels' contents need a manual audit on a bump;
  • and #2405 is the bump where both applied.

A pin review now has two named steps and neither is a memory: run --online and quote its line; audit the .hip diff for f16 narrowing. One non-blocking ask: put that second step where a pin-bump author will meet it — a line in the pin PR description or config_pin_check's output, not only in this ratchet's scope line. A scope note on the checker tells the reviewer; a checklist tells the author, and the author is who can prevent the 1,116 lines from needing review at all.

@bong-water-water-bong
bong-water-water-bong merged commit 1d264d1 into main Nov 14, 2026
9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the ratchet-scope branch November 14, 2026 07:15
bong-water-water-bong added a commit that referenced this pull request Nov 14, 2026
…d it caught itself on the first run) (#2409)

* hip_f16_audit: the f16 properties the Loom ratchet enforces, over the HIP add-on's kernels

hrx_f16_narrowing walks corpus.rglob('*.loom') and cannot see the add-on, whose
kernels live in the gpu-kernels repo as .hip/.h. engine#2406 put that on the
ratchet's green line as 'audit those on a pin bump', which I did by hand twice on
2026-11-14 -- engine#2405's 1,210 new lines and gpu-kernels#76's q41_decode.hip.
This is that audit, mechanised, with the same two properties and the same
accounting: an f16 accumulator fails, a narrowing is reported per file because a
block's d scale is the format.

Measured on gpu-kernels 96593660718d: 68 kernel files, 0 f16 accumulators, 3
narrowings (f16_mask.hip, set_rows_q4_0.hip, set_rows_q8.hip -- all format
writes), rc 0.

Its own first run found a false positive in itself: kquant_tokens_q8.hip:181 is
the sentence 'sends half of them and keeps the other half (by bit M of its lane
id)', matched by the 'half (' pattern. Comments are blanked before scanning now
(blanked, not deleted, so line counts stay true) and a self-test arm pins it. An
unreadable tree is rc 2, never a clean result.

* hip_f16_audit: an f16 accumulator can be spelled half16, and the first regex missed it

The tool's own coverage, checked on the tree it audits: gpu-kernels 96593660718d
declares f16 vectors as

    typedef _Float16 half16 __attribute__((ext_vector_type(16)));

and glm_mla_decode.hip:352 writes its operands that way ('half16 a = load_half16(...)',
accumulating into 'float8 s'). The first version of ACCUMULATOR matched only the
scalar spellings -- half, __half, half_t -- so an accumulator written 'half16 acc'
would have been reported as clean. That is a false negative in the one property the
tool exists to enforce.

Now the pattern covers half*, _Float16, and the vector spellings, with two arms
pinning both directions: 'half16 acc' fails, and a half16 *operand* with a float8
accumulator stays clean (which is what glm_mla_decode.hip actually does).

Re-measured on the real tree: 68 kernel file(s), 0 f16 accumulators, 3 narrowings,
rc 0 -- unchanged, so the real tree was never misreported, only a hypothetical one
would have been.

---------

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant