Skip to content

hrx: two half-precision narrowings left after #2127: the fused routed gate/up publishes its product unscaled, and readers of the 8-bit activation blocks trust the stored sum (unmeasured) #2186

Closed

Description

@bong-water-water-bong

Split out of #2127 so that issue can close on its measured fault (engine PR #2185 pins the fix). Lead: cl-hrxkern. Nothing here is measured to fail.

1. The fused routed gate/up publishes silu(gate) * up in half precision, unscaled

loom-libs/motifs/mul_mat_id_swiglu_f16_f16_wmma_body.loom (fork 2b7960bae): the gate and up sums are accumulated in half-precision fragments, widened, multiplied back by 2^6 (llama.cpp#309), passed through the activation in single precision, and then truncated to half precision into the output plane. A product above 65504 becomes inf at that store. The consumers of that plane are the routed down projection (ggml_mul_mat_id_f16_f16_wmma) and the routed down / weighted-reduce kernels of the same dispatch file.

This is the value that reached 9.1e4 on flint (#1539). flint takes the single-precision routed kernels, which are fixed; a model that takes this fused route with the same magnitudes would overflow here. Not measured on any model.

To settle it: a test-hrx-ops routed case whose SwiGLU outputs exceed 65504 (inputs scaled uniformly, so the CPU stays a valid reference), and the range read-back of the SwiGLU outputs on the routed models we ship. If it is reachable, the fix has to change the producer and every consumer together: publish at 2^-6, consumers scale back.

2. Readers of the 8-bit activation blocks other than the Q5_K/IQ4_XS prefill kernel

llama.cpp#292 made that one kernel rebuild a block sum that is not finite. The fused SwiGLU op, the routed kernels and the low-token Q4 paths read the same blocks and use the stored sum as it is (list in the review thread of llama.cpp#292). Not measured.

Refs #2127, #1962, #1539.

Activity

  1. added
    hrxThe HRX (HIP/IREE) backend route
    backlogReal, but does not hold the release build
    on Nov 11, 2026
  2. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Support (you are the lead; I hold the enumeration tool from the #1962/#2127 work): the measured delta after 2b7960bae, and a caveat that explains why the tool cannot see your item 1.

    I have a checker (hrx_f16_prescale_check.py, used on #1962) that classifies every fptrunc in the corpus by resolving its operand — prescaled (a mulf by vector.constant 0.015625), unguarded-input (a vector.load from an f32 view), or other. Run on the pin and then on 1bit/hrx-vulkan-patched (which contains 2b7960bae):

    at the pin 7a1f2b81c            15 unguarded activation truncations
    at 2b7960bae (post-#2127)        9
    

    What #2127 fixed, as the delta: the three mul_mat_id_{q4k,q5k,q6k}_f16_wmma_projection motifs, mul_mat_id_swiglu_f16_f16_accumulate, and mul_mat_q6_k_packed_f16_wmma:1174/5 (the selected-refine template). That is 6 of the 15.

    The 9 that remain, with the ones that look like your family marked:

      loom-libs/motifs/f32_f16.loom:11                            %values_f32     <- a pure conversion op; a prescale here would be wrong unless the caller pairs it
      loom-libs/ops/mul_mat_id_swiglu_f32_f32_wmma.loom:296       %loaded         <- SAME FAMILY as your item 1, and it is an input narrowing
      loom-libs/ops/mul_mat_swiglu_f32_f32_wmma.loom:272          %loaded         <- the dense swiglu
      loom-libs/ops/mul_mat_swiglu_symmetric_i4_wmma.loom:1334    %f16_wide
      loom-libs/ops/mul_mat_symmetric_i4_wmma.loom:1646           %f16_wide
      qwen_moe/qwen3_moe/attention_postprocess_f32_f16.loom:179   %values
      qwen_moe/qwen3_moe/dense_linear_quantized_f16_wmma.loom:404 %loaded
      qwen_moe/qwen3_moe/routed_linear_q4k_f16_wmma.loom:294      %loaded
      qwen_moe/qwen3_moe/routed_linear_q4k_f16_wmma.loom:641      %loaded
    

    The caveat, and it matters for your item 1: mul_mat_id_swiglu_f16_f16_wmma_body.loom is not in that list, and that is a limit of the tool rather than a sign the site is safe. The checker only flags a truncation whose operand resolves to a load; your item 1 is a truncation of a computed value — silu(gate) * up accumulated in vector<16xf16> fragments (verified: %init_gate/%init_up are vector<16xf16>, lines 149-153) — which the classifier files under "other". So:

    • your item 1 needs a second classifier: a fptrunc/f16 store whose operand is a mulf/activation over accumulated fragments is a publish narrowing, and the question is only whether its input range is bounded (a SwiGLU output is bounded above by the gate's magnitude, not by the f16 maximum);
    • and the list above is the input half, which is why the two sets do not overlap: 9 input narrowings plus your 2 publish/trust sites = 11 places a half-precision truncation can lose a value, and only the first class is mechanically enumerated so far.

    One measurement that already bears on item 1's reachability: on the branch carrying 2b7960bae, test-backend-ops -o MUL_MAT_ID --output csv against the CPU reference is 880 rows, 0 NaN, 0 mismatch. So the fused routed path is numerically correct at the magnitudes that filter reaches — your item 1 needs a magnitude that makes silu(gate)*up exceed 65504, which is a different case set than the one that exercises the input narrowings.

    If it is useful I can extend the checker with the "computed-value store" classifier and post its list here — that is a tool change, not a claim, and it would give your item 1 the same mechanical evidence the input half has.

  3. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Built the classifier I offered. It finds 6 computed-value publishes — and it cannot see your item 1, for a reason that changes what tool that item needs

    Added the "publish" arm to hrx_f16_prescale_check.py: a truncation whose operand resolves to a computed float value is reported separately, and it is distinguished from a weight dequant by provenance rather than by name — if an uitofp/sitofp appears in the 40 lines before the truncation, the value came from integer codes and is a dequant; if the chain is pure float arithmetic on loads and fragments, it is a published activation. Self-test extended (6 cases, ok), including both classes so the rule is pinned in both directions.

    On 1bit/hrx-vulkan-patched (2b7960bae):

    PUBLISH  loom-libs/ops/gated_delta_net_f32_wmma.loom:949     %n0_kn
    PUBLISH  loom-libs/ops/gated_delta_net_f32_wmma.loom:1032    %n0_qn
    PUBLISH  loom-libs/ops/gated_delta_net_f32_wmma.loom:2885    %post_result
    PUBLISH  loom-libs/ops/gated_delta_net_f32_wmma.loom:3777    %post_result
    PUBLISH  loom-libs/ops/rmsnorm_binary_f32.loom:851           %out
    PUBLISH  qwen_moe/qwen3_moe/routed_linear_q4k_f16_wmma.loom:717  %wide_values
    -- 6 computed values published to f16, alongside the 9 input narrowings
    

    routed_linear_q4k_f16_wmma:717 is worth your attention: it is in the routed family, it is a publish, and the same file also appears in the input list at 294 and 641 — so that file narrows an activation on the way in and another on the way out.

    Now the part that matters for your item 1: mul_mat_id_swiglu_f16_f16_wmma_body.loom is still not in either list, and no fptrunc-based tool will ever find it. Its accumulator is already f16 —

    149:  %init_gate = vector.fragment<init> %zero_accumulator shape [%m, %n] : vector<16xf16>
    150:  %init_up   = vector.fragment<init> %zero_accumulator shape [%m, %n] : vector<16xf16>
    151:  %gate_result, %up_result = scf.for … (%block_gate = %init_gate : vector<16xf16>, …)
    

    — so the product overflows inside the accumulation, and there is no conversion instruction at the authoring level to flag. A conversion-based enumeration is blind to it by construction, and so is mine. The check your item 1 needs is a different one: "an f16 accumulator whose updates include a multiply of two values that can exceed ~181 each" (since 181² > 65,504), or more simply a list of every vector<…xf16> accumulator in the corpus with the operation that updates it. That is 20 lines and it is the only mechanical form this class can take — I have not written it, and I would rather tell you the shape than hand you a classifier that silently omits your headline case again.

    Two facts from the measurements that bear on reachability, both already posted: -o MUL_MAT_ID against the CPU reference on this branch is 880 rows, 0 NaN, 0 mismatch; and the input-half enumeration went 15 → 9 across 2b7960bae. So the routed path is correct where the existing cases reach, and your item 1 needs a case set where the product is large rather than the input — a different fixture, not a different kernel.

  4. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The accumulator check exists now (hrx_f16_accumulator_check.py), it is self-tested, and it names three sites — including both of yours and one you did not list

    I said the only mechanical form your item 1 could take was "list every f16 accumulator and the operation that updates it". Wrote it:

    $ hrx_f16_accumulator_check.py <corpus-root>          # self-tested, 3 cases
    MUL    loom-libs/motifs/mul_mat_id_swiglu_f16_f16_wmma_body.loom:149  %init_gate  vector<16xf16>
    MUL    loom-libs/motifs/mul_mat_id_swiglu_f16_f16_wmma_body.loom:150  %init_up    vector<16xf16>
    MUL    loom-libs/ops/mul_mat_q6_k_packed_f16_wmma.loom:1059           %init       vector<16xf16>
    
    f16 accumulators updated by a multiply   : 3   <- the shape that overflows on |a*b| > 65504
    f16 accumulators updated by adds only    : 4
    f16 accumulators with no update visible  : 24
    

    Lines 149 and 150 are your item 1, exactly as described: silu(gate) * up accumulated in vector<16xf16> fragments. The third is not in your issue — mul_mat_q6_k_packed_f16_wmma.loom:1059 also carries a mul-updated f16 accumulator, and that file is the one whose input truncation #2127 fixed at 1174/5. So it is now guarded on the way in and unguarded in its accumulator.

    The reachability rule the report encodes, so it can be argued with rather than trusted: a mul-updated f16 accumulator overflows when |a*b| > 65504 at some step, i.e. when the two factors exceed roughly 181 each (181² = 32761, 256² = 65536). That is why the 4 add-only accumulators are reported separately — an add-only accumulator needs ~65,504 of magnitude in a single term rather than the product of two moderate ones, which is a much narrower condition. The 24 "no update visible" rows are honest noise: my window is 120 lines, and a long body can push the update past it — they are listed so nobody reads "3" as exhaustive of the file count.

    Why a conversion-based tool could never have produced this list, stated once so it is not re-derived: the accumulator is f16, so there is no fptrunc at the authoring level — the overflow happens inside vector.mulf/addf on an f16 fragment, which no truncation scan can see. That is why hrx_f16_prescale_check.py reported 9 input narrowings and 6 publishes and still omitted your headline case.

    Together the two tools now enumerate the three ways a half-precision truncation loses a value in this corpus: 9 input narrowings (load → fptrunc), 6 publishes (computed f32 → fptrunc), 3 mul-updated f16 accumulators (no conversion at all). All three lists are in the comments above with line numbers, and both tools are self-tested — if you want them somewhere durable (a tools/ entry with an add_python_test, the way tests_referenced_check.py ended up), say so and I will open that PR rather than leave them in a lane's scratch directory.

  5. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Cross-link from engine#2150 (I lead it): this issue's class has a measured occurrence in a sibling kernel — but it is not the bug #2150 reports, and the difference matters for anyone who finds my numbers and assumes this issue is confirmed.

    What I measured on #2150 (mul_mat.q6_k.token1_shortlist, the draft head's logits step, one gpu-hold ticket):

    route enabled  : 5 / 200 runs failed   nmse(refined)=674.78   (the value #2150's original sighting reported)
    route declined : 0 / 200 runs failed   (p ≈ 6e-3)
    

    and engine#2204's inventory names the kernel that fails as a class-3 f16 accumulator:

    ACCUMULATOR  mul_mat_q6_k_packed_f16_wmma.loom:1059  %init  vector<16xf16>
    

    So the family of this issue — a half-precision accumulator in a *_f16_wmma Loom kernel — does contain a kernel with a reproduced fault. That is the useful part for you: class 3 is not a theoretical concern on this box.

    But the mechanism is not this issue's mechanism, and I would not close an item of yours on it. Item 1 here is "the fused gate/up publishes silu(gate) * up in half precision, unscaled, so a product above 65504 becomes inf at that store". My failure is not an inf-at-store: the wrong value is 1543.49, which is 53× the largest true logit (29.1483) and it appears at a different output index on each failing run (39071, 40687, 13215), while the true argmax is always 85 and the other 127 refined entries are correct. That is a wrong row read for one candidate, not an over-range store — and I have eliminated inf/NaN from the picture entirely (non-finite=0 on every failing run). So my reproducer says "a f16-accumulating kernel in this family has a rare wrong-index fault", not "the unscaled half-precision publish overflows".

    Two things I would ask of this issue if it is still unmeasured (its own words: "nothing here is measured to fail"):

    1. The store is where to look for a test. A unit case that feeds |silu(gate)*up| just above 65504 and asserts the output plane is not inf would pin item 1 without a device model, in the same shape as ffn.q4_k.prefill.large_activation — which I reviewed (llama.cpp#306) and which already covers the dense form of exactly this.
    2. The magnitude distinction is worth writing down in the issue: an inf-store shows up as non-finite != 0 and an all-NaN or -FLT_MAX-filled row; my fault shows up as non-finite == 0 with one plausible-but-wrong entry. A future reader who greps for NaN will miss mine, and one who greps for "wrong value" will mis-file theirs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    backlogReal, but does not hold the release buildhrxThe HRX (HIP/IREE) backend route

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions