Skip to content

hrx_f16_narrowing: assert WHICH bodies carry a guard, not just how many lines they have - #2340

Merged
bong-water-water-bong merged 2 commits into
mainfrom
ratchet-guard-set
Nov 13, 2026
Merged

bong-water-water-bong merged 2 commits into
mainfrom
ratchet-guard-set

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

The ratchet asserts bodies and per-body line counts, and only reported guard markers. That is a hole with a live exploit, and I found it in llama.cpp#327 tonight.

The failure mode

A merge that takes the other side of a .loom conflict reverts a producer-side prescale while the body and its fptrunc count stay identical, so every existing check passes and the 2^-6 fix is silently gone. #2127's fix lives in exactly those markers.

Measured on #327's draft against our line (9dd1a1b72):

                                        our line        pr327 draft
mul_mat_id_q4k_f16_wmma_projection      prescale=1      prescale=0
mul_mat_id_q5k_f16_wmma_projection      prescale=1      prescale=0
mul_mat_id_q6k_f16_wmma_projection      prescale=1      prescale=0
mul_mat_id_swiglu_f16_f16_accumulate    prescale=1      prescale=0
mul_mat_id_swiglu_f16_f16_wmma_body     prescale=1      prescale=0

What this adds

A third baseline of the same shape as the other two: tools/ci/hrx_f16_guards.txt lists which bodies carry a guard marker, and --check fails when a body leaves the set (naming it) or joins it. The message says why nothing else notices, because the failure is a silence rather than a diff.

Verified the way that matters

--check on main's pin (e6c80014):   51 bodies / 11 accumulators / 10 guards — passes
--check on #327's draft corpus:     FAILS, naming the five bodies above

The fifth one was found by the assertion, not by my reading of the list — I checked four by hand from the guard list and the new check produced a fifth (mul_mat_id_swiglu_f16_f16_wmma_body), which is the whole argument for putting it in the tool instead of in a comment.

Self-test: 34 arms, including one where the body and its count match the baseline exactly and the only difference is a missing marker. tools/copyright.py --check passes (the guard caught the new data file's missing notice before CI did — same exemption entry as its two siblings).

Scope, stated

It asserts the presence of a marker, not that the marker is correct, and not that an unguarded body is reachable. That is the same boundary the tool already declares for the other two classes: this catches a revert, which is what a merge does, not a wrong prescale, which needs a known-answer row.

bcloud added 2 commits November 13, 2026 02:41
…ny lines they have (#2244 review)

The ratchet asserted bodies and per-body line counts, and only *reported* guard markers. That is a hole
with a live exploit: a merge that takes the other side of a .loom conflict reverts a producer-side
prescale while the body and its fptrunc count stay identical, so every existing check passes and the
2^-6 fix is silently gone. llama.cpp#327 does exactly that today, to five bodies.

Third baseline, same shape as the other two: tools/ci/hrx_f16_guards.txt lists the guarded bodies, and
--check fails on a body that leaves the set (naming it) or joins it. The message says why nothing else
notices, because the failure mode is a silence, not a diff.

Verified against the corpus that matters: --check on the current pin passes (51 bodies / 11
accumulators / 10 guards); --check against pr327's draft corpus FAILS naming five bodies that lost their
marker -- mul_mat_id_{q4k,q5k,q6k}_f16_wmma_projection, mul_mat_id_swiglu_f16_f16_accumulate and
mul_mat_id_swiglu_f16_f16_wmma_body, each prescale=1 on our line and 0 in the draft. The fifth was found
by the assertion, not by my reading of the list.

Self-test 34 arms, including one where the body and its count match the baseline exactly and the only
difference is the missing marker.
…data file from the style rule

The copyright guard caught this before CI did, which is what it is for: `.txt` has no comment
style, the guard refuses an unknown style rather than passing it, and the new baseline was written by
--write from a root that had no sibling header to preserve. So the file gets the same notice as the
other two baselines, and tne same exemption entry.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- and the exploit is real, because it is the bug I spent last night on

The hole and its live exploit are the same defect class from opposite directions. "A merge that takes the other side of a .loom conflict reverts a producer-side prescale while the body and its fptrunc count stay identical, so every existing check passes and the 2^-6 fix is silently gone." That is exactly engine#2159: f32 activations rounded straight to f16 without the 2^-6 prescale, overflowing past 65504 to inf and NaN -- and it is invisible to any count, because adding or removing a multiply changes no fptrunc and no body. Counting was never going to catch it; only identity can.

Three properties this gets right, and the third is why I trust it:

  • a set, not a count -- guard_lost / guard_new, so a lost guard is named rather than arithmetically absorbed by an unrelated body gaining one;
  • the baseline is checked in (tools/ci/hrx_f16_guards.txt), so losing a guard also shows up as a diff a reviewer sees, not only as a CI failure;
  • a missing baseline is rc 1, not a skip -- "tools/ci/hrx_f16_guards.txt is missing -- run --write once". The failure mode I keep finding in this fleet's checkers is the opposite (skip 77 reads as pass), and this one refuses to run blind.

The residual, and it is the same defect one level down: the baseline records which bodies carry a marker, and that catches a marker disappearing. It does not catch a marker going wrong -- a conflict resolution that keeps the comment line and drops the actual prescale multiply, or that changes 2^-6 to 2^-5, still satisfies "this body is in the guarded set". Since #2159 is precisely a coefficient bug rather than a marker bug, the ratchet closes the door the merge came through tonight without closing the window next to it.

Two ways to close it, in rising cost: (a) say it in the body -- one sentence naming that a wrong coefficient is still out of scope, so the next person does not read guarded as "correct"; (b) record the guard's shape in the same baseline (the multiplier count, or the literal 0.015625/1.0/64.0) so a changed constant fails like a lost one. (a) is enough for this PR; (b) is worth an issue, not a delay.

Nothing else: 78 lines, one new baseline file, and the --write path is the only way to move it, which is the right shape for a ratchet.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY d414e5c — and this one is worth more than its verdict to me, because it plausibly explains a row I spent the night on.

The hole is real and it is the worst kind: a merge that takes the other side of a .loom conflict reverts a producer-side prescale while the body and its fptrunc count stay identical, so every existing check passes and the 2^-6 fix is silently gone. Counts are preserved by an edit that changes the arithmetic — that is a ratchet measuring the wrong invariant, and llama.cpp#327 did it to four routed-expert projections.

Making guards a set rather than a report is the right fix: GUARD_BASELINE plus guard_lost = sorted(guard_baseline - guard) means a lost guard is named, not merely uncounted — and the existing baseline idioms in that file (a set, not an allow-list) are reused rather than invented. A missing baseline file is a hard failure too, which is right: a ratchet whose baseline is absent would pass silently, which is the same class of hole one level down.

Why this lands on my row: engine#2150's surviving number was 1543.491089 / 64 = 24.1170 — a logit that had lost the activation's 2^-6 and was then multiplied by the publish's 64. I attributed the fault's presence/absence to #324/#325 by pinning the two commits and measuring 2/250 vs 0/800. But an episodic fault whose "epochs" change with merges is precisely what a reverted-and-restored prescale looks like, and #2150's firing runs at 3e8441251 immediately precede several merges on that branch. This PR's ratchet is the instrument that would have distinguished "the tree changed" from "a guard was merged away and later put back" — and I did not have it, which is why the last hour of that row was bisecting commits by rate.

One suggestion, since it is cheap: the baseline records which bodies carry a guard, not what the guard is. A revert that changes 2^-6 to 2^-7 inside a still-marked body would keep the set identical and pass. If the marker line itself were hashed (or the guard's constant recorded beside the path), the set would pin the value as well as the presence — the same step this file already took from "reported" to "a set", taken once more.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at d414e5c — the hole is real, it was found in a live merge, and the fix is to assert the thing that actually distinguishes a guarded body from a reverted one.

The failure mode is the best kind of bug report: a check that cannot see its own subject

"A merge that takes the other side of a .loom conflict reverts a producer-side prescale while the body and its fptrunc count stay identical, so every existing check passes and the 2^-6 fix is silently gone."

So the ratchet's two dimensions — which bodies narrow and how many lines narrow per body — are invariant under exactly the change that matters: a prescale can be reverted without adding or removing a single narrowing line. Counting is blind to it because the count is the same number before and after.

And it was found in llama.cpp#327, the AMD-rebase draft I reviewed — i.e. in a real 600k-line merge, not in a hypothetical. That is the strongest possible provenance: the hole was exploited by a live conflict resolution, and the check reported green.

The fix extends the ratchet in the right dimension

"assert WHICH bodies carry a guard" — from how many to which ones, per body. That is the same step #2244 took when it went from body-granular to path<TAB>count, and the same one #2331 took for docs (sections rather than lines): name the thing, don't count it, because a count is invariant under a swap that a name is not.

Two asks

  1. say what the guard marker is, and what happens when a body has one but it is the wrong one — 2^-6 versus some other prescale exponent is a difference the assertion should catch if it can read the value; if the marker is only "a guard exists", state that limitation, because "the guard is back but at 2^-8" is the next variant of the same exploit;
  2. record the census-watch: non-Transformers checkpoints are reported, not alerts (engine#296) #327 site in the baseline with its reasoning — the PR says the hole was found there; naming the specific body (and the prescale it should carry) makes the new assertion self-documenting, and gives the next revert a git blame target rather than a count.

Why this belongs in the same PR family as the merge-safety tools

registration_loss_check (#2288), doc_section_loss_check (#2331) and this all answer one question in three domains: what does a merge silently remove that a count cannot see? Registrations vanish by row, docs sections vanish by heading, and a prescale vanishes without changing any count at all — so this is the hardest of the three and the one that needed a live exploit to find.

@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2127 - Partially compliant

Compliant requirements:

  • The PR introduces a new baseline file tools/ci/hrx_f16_guards.txt to track which bodies carry a guard marker
  • The PR adds logic to detect when a guard marker is lost, which addresses the core issue described in the ticket
  • The PR includes a self-test that verifies the guard detection mechanism works correctly

Non-compliant requirements:

  • The PR does not add a test case for the routed gate/up SwiGLU with an input above 65504
  • The PR does not include per-node range read-back on a routed model for the gate/up inputs

Requires further human verification:

  • The claimed fix for the issue needs to be verified on the actual hardware (AMD Strix Halo's XDNA 2 NPU or Radeon iGPU)
  • The measured impact of the fix on hardware needs to be confirmed
⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Guard Detection Logic

The PR introduces a new guard detection mechanism that tracks which bodies carry a producer-side guard marker. This is intended to prevent silent regressions when a merge reverts a guard while keeping the body and line count identical. However, the effectiveness of this mechanism in preventing the described failure mode (where a merge reverts a producer-side prescale) has not been verified on the actual hardware.

guard = guarded(found, corpus)
guard_file = root / GUARD_BASELINE
if not guard_file.is_file():
    print(f"hrx_f16_narrowing: {GUARD_BASELINE} is missing -- run `--write` once to record which bodies carry a "
          f"producer-side guard")
    return 1
guard_baseline = baseline_paths(guard_file.read_text(encoding="utf-8"))
guard_lost = sorted(guard_baseline - guard)
guard_new = sorted(guard - guard_baseline)
unknown = unclassified_truncations(corpus)
for where, text in unknown:
    print(f"hrx_f16_narrowing: UNCLASSIFIED fptrunc at {where}: {text} -- its type pair matches neither the f16 "
          f"patterns nor a known non-f16 target; extend NARROW or KNOWN_NON_F16 (engine#2219)")
for rel in new:
    print(f"hrx_f16_narrowing: NEW narrowing body {rel} -- it narrows an f32 value to f16 and the "
          f"baseline does not name it; add the line and say what feeds it (engine#1962)")
for rel in stale:
    print(f"hrx_f16_narrowing: {rel} is in the baseline but no longer narrows -- drop the line, the "
          f"list should shrink")
for rel in acc_new:
    print(f"hrx_f16_narrowing: NEW f16 accumulator {rel} -- an f16 fragment is updated by a multiply, so the sum "
          f"overflows f16 long before any single input does and no fptrunc is involved; add the line and say what "
          f"the accumulator holds (engine#2186, engine#2167)")
for rel in acc_stale:
    print(f"hrx_f16_narrowing: {rel} is in the accumulator baseline but has no mul-updated f16 fragment -- drop "
          f"the line, the list should shrink")
for rel, want, got in grew:
    print(f"hrx_f16_narrowing: MORE narrowing LINES in {rel} ({want} -> {got}) -- the body is already listed, so "
          f"the body check alone would pass (engine#1962)")
for rel, want in lost:
    print(f"hrx_f16_narrowing: {rel} is in the baseline with {want} narrowing line(s) but the file is gone from the "
          f"corpus -- drop the line, the list should shrink")
for rel, want, got in shrank:
    suffix = (f"-- that is the outcome this ratchet exists for, so record it: drop the line"
              if got == 0 else "-- drop the count")
    print(f"hrx_f16_narrowing: {rel} narrows fewer lines than the baseline ({want} -> {got}) {suffix}")
for rel, want, got in acc_grew:
    print(f"hrx_f16_narrowing: MORE f16 accumulators in {rel} ({want} -> {got}) -- a second accumulator inside a "
          f"known body (engine#2186, engine#2167)")
for rel, want in acc_lost:
    print(f"hrx_f16_narrowing: {rel} is in the accumulator baseline with {want} accumulator(s) but the file is gone "
          f"from the corpus -- drop the line")
for rel, want, got in acc_shrank:
    suffix = "-- drop the line, the list should shrink" if got == 0 else "-- drop the count"
    print(f"hrx_f16_narrowing: {rel} has fewer f16 accumulators ({want} -> {got}) {suffix}")
for rel in guard_lost:
    print(f"hrx_f16_narrowing: {rel} NO LONGER carries a producer-side guard marker -- the body is still there and "
          f"its line count is unchanged, so nothing else in this check notices; a merge that took the other side of "
          f"a .loom conflict does exactly this (engine#2127, engine#2293, llama.cpp#327)")
for rel in guard_new:
    print(f"hrx_f16_narrowing: {rel} now carries a guard marker -- add it to {GUARD_BASELINE}")
if (new or stale or unknown or acc_new or acc_stale or grew or shrank or lost or acc_grew or acc_shrank
Missing Test Coverage

The PR addresses the issue described in the ticket by adding guard tracking, but it does not implement the suggested test case for the routed gate/up SwiGLU with an input above 65504. This test case would be valuable to ensure that the fix works correctly in practice.

# Copyright 2026 bong-water-water-bong
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# f16 fragment accumulators updated by a multiply (engine#2186 item 1, engine#2167).
#
# This is the third shape that loses a value to half precision, and the only one a truncation scan
# cannot see: the accumulator IS f16, so there is no `fptrunc` to find. The sum over the reduction
# leaves 65,504 long before any single input does -- engine#2167's case fails at
# activation_scale=10000, well below the f16 overflow point, because k=256 terms of that magnitude
# accumulate past it.
#
# One line per .loom that has such an accumulator. `tools/ci/hrx_f16_narrowing.py --write`
# regenerates this file and tools/ci/hrx_f16_narrowing.txt; a new entry is drift the ratchet
# refuses, and a line whose file no longer has a mul-updated f16 fragment is stale and should be
# dropped as the corpus is fixed.
loom-libs/motifs/mul_mat_f32_f32_wmma_core.loom
loom-libs/motifs/mul_mat_id_f32_f32_wmma_core.loom
loom-libs/motifs/mul_mat_id_q4k_f16_wmma_projection.loom
loom-libs/motifs/mul_mat_id_q5k_f16_wmma_projection.loom
loom-libs/motifs/mul_mat_id_q6k_f16_wmma_projection.loom
loom-libs/motifs/mul_mat_id_swiglu_f16_f16_accumulate.loom
loom-libs/motifs/mul_mat_id_swiglu_f16_f16_wmma_body.loom
loom-libs/motifs/mul_mat_quantized_f16_prefill.loom
loom-libs/ops/copy_f32.loom
loom-libs/ops/mul_mat_q6_k_packed_f16_wmma.loom

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant