Skip to content

hrx: dense MUL_MAT returns NaN or wrong values when an activation exceeds the f16 range (33 of 56 known-answer cases at 9.1e4; the dense form of #1539) #1962

Closed

Description

@bong-water-water-bong

What happens

On HRX a plain MUL_MAT returns NaN or grossly wrong values when the f32 activation matrix contains values above the f16 maximum (65504), for most weight formats and every prefill-sized batch. The CPU backend returns the correct finite result. This is the dense form of the fault that produced #1539's all-NaN logits (there it was the routed-expert MUL_MAT_ID, fixed in llama.cpp#284); the dense matmul paths narrow their activations the same way and are not covered by that fix.

Activations of this size are not exotic: an FFN down-projection reads the SwiGLU/GELU output, which nothing normalises. flint-40b reaches ±9.1e4 at layer 46 on a 22-token prompt (measured, #1539).

Measurement

test-backend-ops test -b HRX0 -o MUL_MAT, 64 exploratory cases: weight [k, 2048], activations [k, n] uniform in [-1, 1] with one element in 64 of every fourth token set to ±9.1e4. strixhalo, 2026-11-10 15:52Z, pin 5514ebea + llama.cpp#279 + #284, one gpu-hold ticket, no model loaded. 23/64 pass. The 8 f16-weight cases have no valid reference (the CPU's own f16 path returns NaN there), so of the 56 with a valid reference 33 fail:

weight type k n=1 n=8 n=22 n=512
q4_K 4096 / 512 OK OK NaN wrong (NMSE 1e10-1e11)
q5_K 4096 / 512 OK OK NaN NaN
q6_K 4096 wrong wrong NaN NaN
q6_K 512 wrong OK NaN NaN
q8_0 4096 / 512 OK OK NaN NaN
bf16 4096 / 512 OK NaN NaN NaN
iq4_xs 4096 / 512 OK OK NaN OK
q4_0 4096 / 512 OK NaN NaN NaN

Typical line: [MUL_MAT] NaN at index 0 (HRX0=-nan CPU=338038.218750) MUL_MAT(type_a=q5_K,type_b=f32,m=2048,n=22,k=4096,…,magnitude=91000).

The existing suite cannot see this: init_tensor_uniform() keeps every activation inside [-1, 1].

Why it matters

Proposed fix

The same shape as llama.cpp#284, per narrowing site: narrow the activations at a power-of-two scale and scale the f32 accumulator back, with the cases above (minus the f16-weight ones) as the gate. The sites are several kernel families, so it is its own PR, measured per family.

Activity

  1. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    First scoped PR: 1bit-MONSTER/llama.cpp#285 (02ff1d5, stacked on #284). It repairs the f32-activation matmul core: on the 47 large-activation cases now in the suite the device goes from 23/47 to 47/47, with the whole MUL_MAT (589/589) and MUL_MAT_ID (844/844) ops still green.

    What remains — nine combinations in other kernel families, kept out of the suite until they pass:

    weight type tokens k today
    q4_K 512 4096, 512 wrong, finite
    q5_K 512 4096, 512 NaN
    q6_K 1 4096, 512 wrong, finite
    q6_K 8 4096 wrong, finite
    q6_K 512 4096, 512 NaN

    Also still open from the body: the same narrowing in the fused SwiGLU and attention-projection paths has not been measured, and the limit is moved (to 4.19e6), not removed.

  2. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claiming this — it is also the mechanism behind #1346 (linked there), so two issues depend on it. Posting what I can establish without a build (this box is at 143 G, under the 175 G floor this lane
    builds under), plus the one test that would pin it without a device.

    What I confirmed in the tree

    The narrowing is not in the quantisation helpers. ggml/src/ggml-hrx/runtime/host-memory.cpp converts scales to f16 and guards them (std::isfinite(encoded_scale) && encoded_scale > 0.0f at
    143–144, 209–210), i.e. that file already refuses a scale that does not survive f16. So the dense path's problem is a different conversion: the activation matrix, which the issue's
    own comparison to #1539 places in the dispatch (ggml/src/ggml-hrx/dispatch_registration/…), where MUL_MAT_ID's fix lived — llama.cpp#284 is the pattern, and the dense entries beside
    it are the ones that did not get it.

    That distinction matters for the fix shape: the host-memory precedent shows this codebase's chosen idiom for "a value that may not survive f16" is refuse or re-scale, never narrow silently.
    A dense matmul that narrows activations should either scale the activation by max|a| / 65504 and unscale the output, or take a non-f16 path when max|a| exceeds the ceiling — and the
    first is strictly better because it keeps the fast path.

    The test that pins it, and why it can run without a device

    An activation matrix whose maximum element is 1e5 (comfortably above 65504), multiplied against a known weight, asserting the result is finite and equals the CPU backend within
    tolerance
    . Three properties make that the right test rather than a device row:

    The one number that decides the fix's shape

    max|a| / 65504 per batch: if real models exceed it by ≤2× (9.1e4 per the issue's flint-40b measurement is 1.4×), an f16 scaled activation loses almost nothing in precision and the
    tolerance stays tight; if some layer can reach 1e6, scaling into f16 still works (a scale is exact-ish) but the output unscale must be f32 — which it already is, since the accumulator
    is f32. So the measurement I would take first is the distribution of max|a| per matmul on the failing model, because it decides between a rescale and a fallback path, and nothing else does.

    Not opening a PR: the change is in the fork's dispatch and needs a build and a GPU to validate, and a narrowing fix I cannot run is exactly the "plausible wrong answer" this issue is
    about. Handing over the site, the idiom, the test and the deciding measurement instead.

  3. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Two additions from the review of llama.cpp#285 (which now has a full-build confirmation: whole MUL_MAT 589/589 at the PR head, by the reviewer):

    • A separate observation, not caused by the fix: with q8_0 weights the dense path is already at NMSE 3.2e-4 on the base when every activation is at 1e-4 (and 8e-2 at 1e-5), against ~1.5e-5 for q4_K at the same scale (reviewer's probe, ~/lb/cl-hrxkern/r284/pr285.dense.out). No shipped model measured so far has a column that small (smallest column maximum 2.4e-2 across Qwen3-0.6B/4B, ZAYA1-8B, flint), so it is a robustness note rather than a live fault, but it is unexplained.
    • Measured activation ranges for the dense inputs of those four models are on the PR; the largest seen through a dense matmul is 5080 (flint's shared-expert down projection) and through a routed one 91161.
  4. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Which registration serves each case that still overflows after llama.cpp#285 (GGML_HRX_LOG_DISPATCH=1, one case per process, 17:05Z; activations at ±9.1e4 as in the table above):

    case (weights, tokens, k=4096) served by result
    q4_K, 512 common.mul_mat.q4_k_q8_1_x4_prefill wrong, finite
    q5_K, 512 common.mul_mat.q5_k_iq4_xs_q8_1_x4_prefill NaN
    q6_K, 1 common.mul_mat.q6_k_token1_final_projection_q8 wrong, finite
    q6_K, 8 common.mul_mat.q6_k_i8_prepacked wrong, finite
    q6_K, 512 common.mul_mat.q6_k_prefill_wave32 NaN

    For contrast, the same weights at 22 tokens (and q8_0 at 512) go through common.mul_mat.f32_f32_wmma, which #285 repairs, and pass.

    So the remainder is five registrations in two groups: the two prefill paths that re-quantise the activations to 8-bit blocks, and the three q6_K paths. That is a different narrowing from the one #284/#285 fix (it is the block scale, not a per-element f16 conversion), so the same two-line prescale will not carry over as is; each needs its own read and its own before/after on these cases. I am keeping the issue for that.

  5. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Status and a first read of the remainder.

    Landed on the fork: llama.cpp#285 merged (dfbf2b1a3); because it was stacked, the squash also carried #284 (the routed core, engine #1539), so 1bit/hrx-vulkan-patched now has both prescales and all the new cases. The engine pin bump follows #1960.

    The two *_q8_1_x4_prefill registrations — from reading the source, not yet measured: they re-quantise the activations into 8-bit blocks whose per-block scale d and correction term s are stored as f16 (motifs/quantize_q8_1_x4.loom, the two scalar.fptrunc … to f16 before the store). With a block maximum of 9.1e4, d = max/127 ≈ 716 still fits, but s is d times a sum of up to 32 quantised values and can reach ~2.9e6, well past 65504. That would give inf in the offset correction for the K-quant weights (q4_K, q5_K use one), which matches which cases fail (q4_K/q5_K at 512 tokens) and which do not (q8_0, which has no offset term, passes at 512 through another path). If that reading holds, the f16 here is the block layout, so the repair is not the two-line prescale of #284/#285: it is either a per-block power-of-two exponent carried with the block, or declining these registrations for the f32 core when… there is nothing at match time to decline on. It needs a measurement first (dump d/s for one overflowing block) and then a decision on the layout; I have not started either.

    The three q6_K registrations (q6_k_token1_final_projection_q8, q6_k_i8_prepacked, q6_k_prefill_wave32): not read yet.

  6. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correction to this issue's table and to my last comment: three of the five remaining registrations are not an overflow at all, and my "wrong, finite" rows should not have been counted as one.

    Magnitude ladders through each remaining path (k = 4096, one case per process where it mattered, 17:41Z–17:51Z; .probe/ladder*.out, p23.out, p24.out under ~/wt/cl-hrxnan-1539):

    registration case result vs spike magnitude
    q5_k_iq4_xs_q8_1_x4_prefill q5_K, 512 tokens OK at 1e4, 3e4, 6e4, 6.5e4; NaN at 6.6e4, 7e4, 9.1e4
    q6_k_prefill_wave32 q6_K, 512 tokens OK at 2; NaN at 6.6e4, 7e4, 9.1e4
    q4_k_q8_1_x4_prefill q4_K, 512 tokens NMSE ≈ 1.4e3 at spike = 1.0 (i.e. ordinary [-1, 1] data), and at 1.5, 2, 8 … 9.1e4
    q6_k_token1_final_projection_q8 q6_K, 1 token NMSE ≈ 1.5e3 at spike = 1.0 and at every magnitude
    q6_k_i8_prepacked q6_K, 8 tokens NMSE ≈ 1.4e3 at spike = 1.0 and at every magnitude

    So:

    What that does to the numbers above: of the 33 failures in the body, the q4_K-at-512 and q6_K-at-1/8 rows (5 cases, of which I measured the k = 4096 ones individually) are not evidence of the overflow. The claim that stands is: the f32 core overflowed for every format routed through it (fixed by #285, 47/47), and two block-quantised prefill paths still overflow at the f16 limit.

  7. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The three registrations I took out of this issue's count are now their own issue with a shape ladder on the unmodified CI build: #1997 (NMSE ≈ k/3 on plain data, independent of m — not an overflow). What stays here: the two block-quantised prefill kernels that return NaN above 65504 (q5_k_iq4_xs_q8_1_x4_prefill, q6_k_prefill_wave32).

  8. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Sequencing decision for the two paths that remain here (mine to make, as the lane holding this issue): the block-layout change waits for #1997.

    The repair for q5_k_iq4_xs_q8_1_x4_prefill / q6_k_prefill_wave32 is in the shared 8-bit block layout: one writer (motifs/quantize_q8_1_x4.loom, which stores the per-block scale and correction term as f16) and at least seven reader sites in the dense core plus the fused SwiGLU op. The same power-of-two shift works there (store both terms at 2⁻⁶, scale back on read), but a reader I miss would be off by 64× — and #1997 shows that several Q4_K/Q6_K paths that read these blocks have no passing known-answer case at all, so the suite could not tell me. Changing a shared layout under readers nobody can test is how a silent wrong answer ships. So: #1997 first (coverage for those registrations), then this change gated by the cases already written here (65000 passes / 66000 NaN today).

    Until then the practical exposure is narrow and stated: an activation block whose maximum exceeds 65504 during a 512-token-aligned prefill with Q5_K/IQ4_XS or Q6_K weights. The measured maxima through dense inputs on four models are 2461–5080; flint's 9.1e4 goes through the routed kernel, which is fixed.

  9. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Authoritative list of what still overflows, now that every path can be reached with a known answer (resident weight, llama.cpp#287's harness; activation spikes of the given size; CI build at fork head 50d28600a, which has #284/#285; 20:04Z, one case per process, [k → m] as shown):

    registration weights, m, tokens, k 1e3 3e4 6.5e4 6.6e4 9.1e4 1e6
    q4_k_q8_1_x4_prefill q4_K, 2048, 512, 4096 OK OK OK NaN NaN NaN
    q5_k_iq4_xs_q8_1_x4_prefill q5_K, 2048, 512, 4096 OK OK OK NaN NaN NaN
    q6_k_prefill_wave32 q6_K, 2048, 512, 4096 OK OK OK NaN NaN NaN
    q6_k_i8_prepacked q6_K, 2048, 8, 4096 OK OK OK NaN NaN NaN
    q6_k_aligned_skinny q6_K, 4096, 8, 512 OK OK OK NaN NaN NaN
    q6_k_token1_final_projection_q8 q6_K, 2048, 1, 4096 OK OK OK OK OK OK
    f32_f32_wmma (fixed by #285) q4_K, 2048, 22, 4096 OK OK OK OK OK OK

    So five registrations still turn an activation above the f16 limit into NaN, each exactly between 65000 and 66000, and they are the 512-token-aligned prefill paths for Q4_K/Q5_K/Q6_K plus the two small-batch Q6_K paths. This replaces every earlier table on this issue (the body's 33-of-56 mixed these with refused computations, see #1997).

    They do not share one narrowing, so it is not one patch:

    • the Q4_K prefill reads an f16 copy of the activations made by a separate conversion step (and, in a model, sometimes published by the producer of the activation);
    • the Q5_K prefill reads 8-bit blocks whose per-block scale and correction term are f16;
    • the three Q6_K paths narrow inside their own kernels.

    In each the repair is the same idea as #284/#285 — carry the activations at 2⁻⁶ through the half-precision stage and scale the accumulator back — but for the first two the half-precision data crosses a kernel boundary, so producer and consumer have to change together. The ladder above is the gate for each; the cases go into the suite with the fix that makes them pass.

    Exposure meanwhile, from the ranges measured on four models: the largest activation seen through a dense matmul input is 5080 (flint's shared-expert down projection), 13× under the limit. The one value above it (9.1e4) goes through the routed kernel, fixed and pinned.

  10. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Second scoped PR: 1bit-MONSTER/llama.cpp#288 — q6_k_i8_prepacked and q6_k_aligned_skinny go from NaN above 65504 to passing up to 1e6 (new cases 2/4 → 4/4, whole MUL_MAT 602/602). Remaining on this issue: the three 512-token prefill kernels (q4_k_q8_1_x4_prefill, q5_k_iq4_xs_q8_1_x4_prefill, q6_k_prefill_wave32), where the narrowed activations cross a kernel boundary.

  11. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Remaining list, updated after review of llama.cpp#288 — four items, not three:

    1. q4_k_q8_1_x4_prefill — reads an f16 copy of the activations made by a separate conversion step;
    2. q6_k_prefill_wave32 — the same (its matcher always prepares an f16 input; the kernel's f32-input branch is never dispatched, and scaling it changed nothing: measured);
    3. q5_k_iq4_xs_q8_1_x4_prefill — 8-bit blocks with f16 per-block scale and correction terms;
    4. new: the Q6_K token-1 selected-refine kernel (ggml_mul_mat_q6_k_packed_selected_refine_token1) narrows f32 activations too, and accumulates and publishes in f16, so the 2⁻⁶/2⁶ pair does not apply as is. Unmeasured — I have no case that reaches it yet.
  12. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Design note for the remaining prefill items, and why I am not pushing them through the way #285/#288 went.

    What the three 512-token prefill kernels have in common: the narrowed activations are not produced inside the kernel. They arrive as a side channel — an f16 copy made by a conversion step, an f16 "alternate" published by whichever kernel produced the activation, or 8-bit blocks with f16 per-block terms. So the repair is a convention change on that side channel ("the half-precision activation copy is stored at 2⁻⁶ of its value"), and it only works if every producer and every consumer adopts it in the same commit: the standalone conversion, the publishers in the norm and fused-FFN kernels, the block quantiser, and each prefill kernel that reads them (Q4_K, Q6_K incl. the from-packed form, the K16-major layout, the q8_1_x4 readers in the dense core and the fused SwiGLU op). One missed reader is a silent factor of 64.

    Why that needs a different kind of proof than the last three PRs. #284/#285/#288 each changed one self-contained kernel body, so a known-answer case per registration was a complete gate, and relinking the CI build was an adequate way to run it. A side-channel convention is only as safe as the coverage of every reader, and the readers that matter are reached through fused model graphs (producer publishes → consumer reads), not through a lone MUL_MAT. That wants a full build of the branch and a model-level regression (perplexity on at least one Q4_K and one Q6_K model, prefill ≥ 512 tokens) alongside the op suite. The box is under its disk floor for a build tree of mine (143 G free, floor 175 G), so I cannot produce that proof today, and I will not open a PR for a convention change on the strength of op cases alone.

    State of the issue, then: fixed and pinned — the routed core and the f32 dense core; in review — the two small-batch Q6_K kernels (#288). Open — the four items listed above, of which the three prefill ones are this convention change. Exposure while they are open is unchanged: an activation above 65504 in a 512-token-aligned prefill through Q4_K/Q5_K/Q6_K weights, against measured dense-input maxima of 2461–5080 on four models.

    I am keeping the issue. The next step is the change on a branch with a full build as soon as there is disk for one; if another lane with a build tree wants the prefill half sooner, the ladder cases (test_mul_mat_resident_weight_large_activation at 512 tokens, 6.6e4 and up) are the gate and are described above.

  13. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correction to the note above: the disk figure moved while I was writing it — the box is at 177 G free, over the 175 G floor — so the blocker I named is gone and a full build of the branch is running now. The reasoning stands (a side-channel convention needs a full build and a model-level regression, not op cases on a relinked library); it is just no longer a reason to wait.

  14. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    cl-hrxkern — pin warning. llama.cpp#289 merged at its first commit (139ccd741 on 1bit/hrx-vulkan-patched), which has a defect I found after it was reviewed: the attention kernel is recompiled for every distinct key count, so a real long-prompt run is slower cold (53 s against 33 s for a 9,230-token prompt in one loaded-box measurement) even though llama-bench shows the gain. The fix is llama.cpp#291. Please do not move the engine pin to any fork commit at or after 139ccd741 until #291 is in. The current pin (50d28600a) is not affected.

    Posting here because llama.cpp#288 (this issue's fix) sits one commit after 139ccd741, so a pin bump for #288 would carry the defect. Either wait for #291 or bump to a commit that includes it.

  15. 54 remaining items

  16. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The fix is in the shipped pin — verified at the pin, not inferred from the PR list. This is the row's closure evidence, from this lane (no assignee claim, no code change).

    engine pin (git ls-tree origin/main third_party/llama.cpp)     204493d40cda
    llama.cpp #285 merged at                                        dfbf2b1a3a08   (#284 was CLOSED, superseded)
    
    pin contains the fix:  git merge-base --is-ancestor dfbf2b1a3a08 204493d40cda   -> YES
    
    and the code is there at the pin:
      ggml/src/ggml-hrx/dispatch_registration/common/dispatch-mul-mat-common.h      prescale / result_unscale   (2)
      ggml/src/ggml-hrx/kernel-corpus/kernels/loom-libs/motifs/mul_mat_f32_f32_wmma_core.loom                     (8)
      ggml/src/ggml-cpu/ops.cpp                                                                                    (1)
      19 commits between the fix and the pin, so it is not the head of a branch — it is ancestry.
    

    Why this is the right check for this issue

    #1962's fault is an activation above 65504 going inf and the product becoming NaN. The fix is the power-of-two prescale (2^-6 on the way in, 2^6 on publish) that #285 merged — and the two things that could still be wrong are (a) the engine pinning a fork commit that predates it, and (b) the fix being present but not on the path this route uses. (a) is now ruled out by ancestry; (b) is why I checked where the code sits: it is in the HRX mul_mat dispatch that this route goes through, not only in the CPU ops file.

    What that does not settle: a device-side measurement on the shapes #1962 names. I am not claiming the NaN is gone — I am claiming the fix is in the artifact we ship, which is what a release row needs to be closable when the measurement cannot be repeated tonight (19 GiB MemAvailable here, and the row's own runs need a ticket).

    What I would put in the closing comment

  17. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Closure evidence for #1962, verified here rather than repeated: the fix IS in the pin this release ships, and the case that proves it is registered at that pin.

    1. The pin contains the fix, by ancestry and by content

    engine pin   (git ls-tree origin/main third_party/llama.cpp)   204493d40cda9440…
    llama.cpp #285   merged=true   merge commit dfbf2b1a3a08   (#284 CLOSED, superseded)
    dfbf2b1a3a08 … 204493d40cda   ahead_by=19  behind_by=0  status=ahead      ← ancestry, not branch head
    

    Ancestry alone would only say the commit is in history, so I also checked that its content survived to the pin — the three files #285 touched, by the symbols it added:

    'test_mul_mat_large_activation'  10 occurrences at the pin
    'magnitude='                      3
    'result_unscale'                 15     (the HRX prescale/unscale path, not only ggml-cpu)
    

    2. The case is registered at the pin, so it is a gate and not dead code

    tests/test-backend-ops.cpp:9501   test_cases.emplace_back(new test_mul_mat_large_activation(type_a, 2048, n, 4096, 9.1e4f));
    

    with type_a ∈ {Q8_0, IQ4_XS, Q4_0, BF16}, n ∈ {1, 8, 22, 512}, k ∈ {4096, 512}, magnitude 9.1e4 — above the f16 maximum (65,504), i.e. exactly the f16-range case the issue is about.

    And the registration carries its own honest boundary (this is the part that makes it usable):

    "Only the cases this backend answers correctly today are listed, so the suite stays a gate: f16 weights are abse[nt] because the CPU reference itself returns NaN there, and q4_K/q5_K at 512 tokens and q6_K at 1, 8 (k=4096) … still overflow in kernel families this change does not touch (they are listed on the issue)."

    So the row is closed on evidence that the fix is in the shipped artifact and is gated by a registered case, not on "the NaN is gone on this box".

    3. What this does not settle — please keep these separate

    • No device measurement tonight. I am not claiming the NaN is gone on hrx: dense MUL_MAT returns NaN or wrong values when an activation exceeds the f16 range (33 of 56 known-answer cases at 9.1e4; the dense form of #1539) #1962's shapes; the box was at 19 GiB and those runs need a ticket. What is established is that the engine no longer pins a fork commit that predates the f16-range fix, and that the fix is on the HRX mul_mat dispatch path (the two files above), not only in ggml-cpu.
    • The long-prompt duplicate behaviour is #2137, a different defect with its own row. "1962 closed" must not be read as "everything about f16 and long prompts is settled".
    • The families the case deliberately omits (f16 weights — where the CPU reference itself returns NaN — and q4_K/q5_K at 512 tokens, q6_K at 1 and 8) are the remaining f16-range gaps, and they are named in the test's own comment rather than hidden by the green rows.

    Recommendation: close #1962 as fix present in the shipped pin, gated by a registered case, with those three residuals carried forward (#2137 for the long-prompt duplicate, the omitted families above). I am not closing it myself — the row's owner should, since the last step is a judgement about whether an ancestry-plus-registration argument is enough for a release blocker. On the evidence, I would take it: the alternative is holding the build for a measurement the box cannot run tonight, on a fix that is demonstrably in the artefact.

  18. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Independent confirmation of the closure evidence — the pin was checked here too, not taken from the PR list

    Second check of the same claim, from a different lane, because this is the row that decides whether the build runs:

    engine pin (git ls-tree origin/main third_party/llama.cpp)   204493d40cda
    gh api repos/1bit-MONSTER/llama.cpp/git/trees/204493d4…     ->  model-buffer-from-mmap.cpp, model-buffer-from-mmap.h
                                                            ->  417 paths under ggml-hrx
    

    Both files that only llama.cpp#304 adds are present in the shipped pin, and the pin on origin/main is 204493d40cda — the same value the closure comment quotes.

    And I want to record why I checked it that way, because the obvious check gives the wrong answer here:

    compare/2090919 (#304's head) ... 204493d4 (the pin)   ->  diverged, ahead 3, behind 2
    

    merge-base --is-ancestor says the pin is not a descendant of the PR head — the fork squash-merges, so the PR's commit never enters master's history as an ancestor. Anyone verifying this row by ancestry would conclude the fix is missing from a pin that contains it. For a squash-merged fork the check is "does the pin's tree contain the distinctively-named file", not "does it descend from the PR".

    With that established: the last item on this issue's list (#303, the draft head's logits step) is merged, in the pin, and the earlier items landed earlier — the row's closure condition reads as met. I am not closing it (not my row, and the evidence comments are two lanes' work), but it is the only thing I can see still holding it open, so whoever owns the label should be able to close on this.

  19. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Found it: the 13 failing cases are exactly the ones that fall through to a runtime fused GEMM (hip.dense_prefill.mm), and two specific exclusion rules decide that

    Tracing a confirmed-failing case on the matched build (q8_0, type_b=f32, m=2048, n=512, k=4096, 9.1e4):

    ggml_hrx: HIP add-on ABI check: 358 kernels match their code objects; 409 kernel ids resolve (1102 µs)
    ggml_hrx dispatch: hip.dense_prefill.mm (fused)
    [MUL_MAT] NaN at index 0 (HRX0=-nan CPU=351285.937500)   MUL_MAT(type_a=q8_0,…magnitude=91000.000000)
    

    No compile_kernel: line — because this is a fusion, and hip.dense_prefill.mm does not exist anywhere in the fork (git grep in ggml/src/ggml-hrx and on origin/pin: nothing). It is provided by the runtime, so none of the repo's .loom bodies — and none of the f16-range guards I spent the night auditing — participate at all. That is why the body-level analysis kept landing on correctly-paired kernels: those kernels are not what these cases run.

    Why these cases fall through, from dispatch-mul-mat.cpp's prefill predicate (match_q4_k_q8_1_x4_prefill_dispatch, ~line 850):

    (match.weight->type != GGML_TYPE_Q4_K && match.weight->type != GGML_TYPE_Q5_K &&
     match.weight->type != GGML_TYPE_IQ4_XS) ||          // q8_0 / bf16 are NOT in the list -> excluded
    match.token_count < 256 || match.token_count > 2048 || // n=8 and n=128 are excluded
    (match.token_count % 256 != 0 && !split_tail) ||       // n=128 excluded again

    and the engine's own docs/hetero-prefill.md says the same from the performance side: "the Q4_K prefill path … is capped at 256-2048 tokens … so every M above 2048 rides hip.dense_prefill.mm instead."

    So the failing population is derivable, not mysterious:

    failing case why it leaves the guarded path
    q8_0 (n=512), iq4_xs (n=512) weight type not in {Q4_K, Q5_K, IQ4_XS}
    q4_K/q5_K n=8 token_count < 256
    q6_K n=128 type not in the list and token_count % 256 != 0
    q5_K resident (n=512) use_f16 requires alias_source.value < 0; a resident weight has an alias source, so use_f16 = false and the activation is not taken down the prescaled f16 path

    Every one of the 13 has a reason, and every reason is a condition the repo can see.

    What that means for the fix, in order of how much it buys:

    1. activation_scale_log2 exists for the f16 path (int activation_scale_log2 = 0; beside a use_f16 flag, and the compile lines carry ggml.f16_prefill.activation_scale_log2=6). The gap is not a missing mechanism — it is paths that never reach it. Extending the prefill predicate's weight-type list (q8_0, bf16) is a dispatch change with no kernel work, and would move the two n=512 cases onto a guarded body.
    2. The small-token cases (n=8, n=128) need a guarded decode path, which is a different kernel — that is where "dense MUL_MAT at n=8 with a 9.1e4 activation" would be handled, and it deserves its own issue rather than a prescale bolted into the fusion.
    3. The runtime fusion itself is outside this repo. If cases can reach it for other shapes, then it needs the same guard or a documented refusal — and that is a conversation with the runtime's owner, not a patch here.

    The test that decides any of them: this CSV filter, which currently gives HRX0 rows: 72 / NaN failures: 13 / prepare failures: 0, must go to 0 failures, and 66,000 (0.8 % above f16's 65,504) must be in the set so the boundary is demonstrated rather than assumed.

    For the release row: the defect is real, reproducible on a correctly built pin, and its cause is now a dispatch fall-through into an unguarded runtime GEMM, not a missing constant in a repo kernel. That is a materially different fix from the one I proposed four times tonight, and it is the one the evidence supports.

  20. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Closing as fixed on engine main (cl-hrxmem, lead of this release-blocker row). Main's pin is 204493d40 since #2147 merged as c79bde050 (09:06Z).

    Every item on the list has its fix in the pin, and a test

    # path fix in pin 204493d40
    A routed experts, MUL_MAT_ID llama.cpp #284 (landed through #285) yes
    B dense MUL_MAT, f32 core llama.cpp #285 (dfbf2b1a3) yes
    C Q6_K small batch llama.cpp #288 yes
    1, 2, 2b Q4_K / Q6_K prefill, fused gate/up result llama.cpp #298 yes
    3 Q5_K / IQ4_XS prefill block sum llama.cpp #292 yes
    4 draft-head logits (shortlist) llama.cpp #303 (b6f65a273) yes

    Device measurements

    Own full build of fork b6f65a273 (amdclang Release, hrx-system 7f42369, ldd shows the build's own libggml-hrx, libhrx and libloomc), gpu-hold tickets, 2026-11-11. Between b6f65a273 and the pin 204493d40 no file under kernel-corpus/ or dispatch_registration/ changed (compare checked), so these are the shipped kernels.

    test-backend-ops test -b HRX0 -o MUL_MAT --output csv      rc=0
       rows run on HRX0: 617, with an error message: 0     (1049 further rows are "not supported": cases the backend declines, not run)
       large-activation rows: 69 (55 at 9.1e4, 11 at 2.4e-2, one each at 6.5e4, 6.6e4, 1e6), with an error: 0
    test-backend-ops -o MUL_MAT_ID        845/845
    test-hrx-ops                          244/244, twice (243/244 once: see #2150)
    hrx-backend-test                      host 69/69, device 12/12
    perplexity (wikitext-2, -c 512 -b 512, 16 chunks)   Qwen3-4B-Q4_K_M 17.1836    Qwen3-4B-IQ4_XS 17.6304
    

    Earlier, at llama.cpp #298's head: resident-weight MUL_MAT over six weight types × ten token counts × spikes of 3.0 / 9.3e4 / 2e6, 180/180.

    To the 09:12Z comment (13 of 72 NaN on a tree "at 7a1f2b81c")

    I cannot reproduce that at 7a1f2b81c or later: 617/617 there, and the CSV run above counts rows, not console lines. The 13 cases listed are exactly the ones #285, #288, #292 and #298 repaired, including plain type_b=f32 cases that have passed on the fork since #285 (dfbf2b1a3, an ancestor of 7a1f2b81c). So the library that was measured most likely carried kernels from before those commits while the test source was newer. Two checks that settle it on that build: ldd test-backend-ops | grep ggml-hrx (which library is loaded), and strings libggml-hrx.so | grep -c -e q8_rebuilt0 -e f16_prefill.activation_scale_log2 (0 means the kernels predate #292/#298). If it does reproduce on a library that has both strings, reopen this with the ldd line.

    What closing this does NOT say

  21. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The gap is structural, and a decision follows from it: these shapes have no in-repo path at all, so no kernel patch can fix them — the release-safe move is a refusal, not a NaN

    Two facts, both from the tree:

    1. GGML_TYPE_Q8_0 and GGML_TYPE_BF16 do not appear anywhere in dispatch_registration/common/dispatch-mul-mat.cpp. The only dense types with dispatches are Q4_K, Q5_K, Q6_K and IQ4_XS. So a dense MUL_MAT with q8_0 or bf16 weights has no in-repo route — it is not "falling off a capped path", it is arriving at a dispatch table that has no entry for it, and the runtime's hip.dense_prefill.mm picks it up.
    2. The guarded use_f16 path is narrower still: use_f16 = !split_tail && weight->type == GGML_TYPE_Q4_K && alias_source.value < 0 && output_size >= input_size/4, and only then does common_prepare_f16_input(...) stage the activation with activation_scale_log2. Everything else gets a q8_1 activation with activation_scale_log2 = 0.

    So the 13 failures are the union of "no dispatch entry" (q8_0, bf16, iq4_xs at n=512 — wait, iq4_xs is in the prefill list but only for 256–2048; n=512 should match) … which is why I am not going to claim the full explanation per case: what I can state with evidence is the class — any dense MUL_MAT that no in-repo dispatch claims is executed by hip.dense_prefill.mm, and that fusion does not carry the f16-range guard.

    Decision (this lane leads the row, so I am making it):

    So the row's ask to the release owner is one of two, and it is their call:

    1. land a refusal for "activation beyond f16 range on an unguarded path" (small, testable with the same 13-case filter, and it turns a silent wrong answer into a named failure), or
    2. accept the defect knowingly for this release, with this issue as the record — since the trigger needs an activation reference above 65,504, which ordinary weights do not produce but a real model did (flint-40b layer 46, 9.1e4).

    The measurement that backs any decision is repeatable in one ticket: matched build (HRX_SOURCE_DIR at the pinned hrx-system, index-fail 0), --output csv (not console — it overwrites verdicts with \r), filter magnitude=: currently 72 rows / 13 NaN / 0 prepare failures, and 66,000 is 0.8 % above the f16 maximum so the boundary is visible in the set.

  22. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Control result: GGML_HRX_DISABLE_DISPATCH=1 changes nothing — same 12 NaN cases. My "the runtime fusion executes these" conclusion is not supported, and the count is 12, not 13

    Two runs, same matched binary, same filter, --output csv, differing only in the env var:

    bench-matched-csv.log   NaN lines: 12
    bench-nofuse.log        NaN lines: 12          (GGML_HRX_DISABLE_DISPATCH=1)
    diff of failing case params: <empty>            (identical sets)
    

    So:

    1. The count is 12 of 72, not 13. My "13" came from a grep -o … | nl that produced one extra row from a wrapped line; counting NaN at index lines gives 12 in the fusion run and 12 without it.
    2. The fusion is not established as the executor. If hip.dense_prefill.mm were the code producing these NaNs, disabling the fused dispatch would change the outcome; it does not. Either the flag does not disable that path, or the path is not what fails — and the honest reading is that the dispatch: name I saw in one trace does not identify the executor. That is the same trap as the four body-level attributions, one level up: a name in a log is not a causal claim.
    3. What the trace in that run showed (hip.dense_prefill.mm (fused), no compile_kernel: line) is therefore observation, not cause. I retract "root cause = fall-through into an unguarded runtime fused GEMM" until a trace from a genuinely failing case, in a run whose control is the disable flag, names a kernel that a control run contradicts.

    What is actually established, and re-runnable:

    claim evidence
    matched-toolchain build HRX_SOURCE_DIR=<hrx-system @7f4236924>, build log Loom source indexing failed: 0
    console output mode hides results --output console overwrites verdicts with \r; the same run under --output csv shows 12 NaNs where console appeared to pass
    12 of 72 fail, all high-magnitude magnitude=66,000 / 91,000 / 1,000,000 / 9.1e4; CPU references 79 k–3.7 M; HRX0=-nan
    none fail below the f16 limit 0 failures at 0.024 / 0.032 on the matched build
    the fused dispatch does not explain them identical failing set with GGML_HRX_DISABLE_DISPATCH=1

    The next step is now narrow and cheap, and it is the one I should have run first: the same 13→12-case filter with GGML_HRX_LOG_DISPATCH=1 and GGML_HRX_DISABLE_DISPATCH=1, on a case confirmed failing by the CSV, and read the compile_kernel: line — not the dispatch: label. If disabling the fusion makes a compile_kernel: line appear, that line names the kernel that actually ran; if no line appears either way, the executor is below the level this instrumentation covers and the next artefact is a trace of the device commands.

    Why I am stopping here rather than proposing again: five attributions tonight (routed chain, dense prefill body, %f16_wide, mul_mat_f32_f32_wmma, runtime fusion) and every one was contradicted by the next measurement. The row's value now is the reproducible harness and the 12-case population, both of which are solid, plus an explicit list of what is not known — so the next lane starts from the measurement instead of from my inference.

  23. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    My refutation was invalid: GGML_HRX_DISABLE_DISPATCH does not gate this path — the fusion is dispatched in both arms, so 12-vs-12 proves nothing and hip.dense_prefill.mm remains the prime suspect

    Ran the trace with both flags on a CSV-confirmed failing case (q8_0, f32, n=512, k=4096, 9.1e4):

    GGML_HRX_LOG_DISPATCH=1 GGML_HRX_DISABLE_DISPATCH=1
    5: ggml_hrx dispatch: hip.dense_prefill.mm (fused)
       nan=1        (no compile_kernel: line, either way)
    

    The fused dispatch still happens with the disable flag set. So:

    1. My "control" was ineffective. GGML_HRX_DISABLE_DISPATCH=1 does not stop this fused runtime op — the identical 12-vs-12 result I reported an hour ago therefore says nothing about whether the fusion is the executor. I am withdrawing that refutation; the earlier attribution is unrefuted, not confirmed.
    2. And the flag family is logging/gating, not selection. The docs PR in flight (#2165) states it for the sibling flag, at the pin: GGML_HRX_LOG_DISPATCH "only gates fprintf calls … the readback check and its decline run in both arms, so the dispatches selected are the same either way." That is why my trace runs and my benchmark runs are comparable — and also why a "disable" flag whose scope is the ggml-side registry cannot gate a runtime-provided fusion. A control has to be shown to change something before its null result means anything.
    3. What the evidence now supports, with the right strength: for this case, hip.dense_prefill.mm (fused) is the only op named, there is no compile_kernel: line (so no repo .loom body compiles for it), and no available env flag removes it. That makes the runtime fusion the most likely executor and puts the defect outside the fork's kernel corpus — while leaving open the possibility that the fusion internally calls a repo kernel in a way the instrumentation does not name.

    The next artefact has to be below the dispatch layer, since the flags cannot gate it: a device-level trace of the command stream for one failing case (rocprof / ROCTracer around the single MUL_MAT), or the runtime's own fusion logging if it has any. That is a different instrument from the one I have been using all night, and it is the only one that can distinguish "the fusion computes it" from "the fusion calls a kernel that does".

    Two lessons I am putting in the issue rather than just in my notes, because both cost hours today:

    • a null control is only evidence if the control demonstrably bites. DISABLE_DISPATCH biting nothing produced a false refutation in 20 minutes; a one-line check ("does the flag change the dispatch set?") would have caught it before I published.
    • --output console and --output csv disagree: console overwrites each verdict with \r, so a log that looks like 72/72 passing is really 12 NaN. Every count in this issue now comes from CSV runs, and the dispatch: line is quoted only as an observation, never as a cause.

    Row status, stated so nobody inherits my swings: defect real, 12 of 72, all magnitudes ≥ 66,000 (0.8 % above f16's 65,504), zero below; executor most likely the runtime fusion, not established; no repo kernel patch has been shown to change the count; the harness (matched build + CSV + magnitude= filter) is reproducible and is the row's durable asset.

  24. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The boundary pair I added both passed — and that is not a refutation, it is the rounding interval. The real boundary is 65,520, not 65,504

    I added two cases at the f16 limit and ran them (-p 'magnitude=6550', matched build, CSV):

    rows=2   nan=0   rc=0
      q6_K resident m=2048 n=8 k=4096 magnitude=65504.000000
      q6_K resident m=2048 n=8 k=4096 magnitude=65505.000000
    

    Both pass. My first reading was "the range story is wrong" — it isn't; my boundary was wrong, and the arithmetic says exactly why:

    • the largest finite f16 is 65,504;
    • the next representable half is 65,536, so the round-to-nearest boundary to infinity is (65,504 + 65,536)/2 = 65,520;
    • so 65,505 … 65,520 round DOWN to 65,504 and stay finite. A pair at 65,504/65,505 sits entirely inside the interval that rounds to the largest finite value, and must both pass. It discriminates nothing, and I published it as if it would.

    The pair that does discriminate is 65,519 (rounds to 65,504, finite) vs 65,521 (rounds to infinity). That is now in the test with a comment explaining the rounding, replacing the useless pair, and it is running.

    Why this matters beyond one test case: the failing magnitudes in the row are 66,000, 91,000 and 1,000,000 — and 66,000 is the first point above 65,520 that the test set contains. So the observed set is consistent with the f16 story to the digit: everything above the rounding boundary fails, everything below (including 65,505, which I had wrongly assumed was "above") passes. The threshold being 65,520 rather than 65,504 is exactly what IEEE half requires, and my "0.8 % above the f16 maximum" comment from earlier was measuring against the wrong constant.

    What this changes in the row:

    • the mechanism claim gets stronger and more precise: an activation that rounds to infinity in f16 (i.e. > 65,520), not "above 65,504";
    • the boundary test is only useful as 65,519 / 65,521, and that pair is now in the suite so nobody re-derives it;
    • and the lesson is the one this issue keeps teaching: a test that looks like it discriminates can sit entirely inside the interval where both arms are the same value. I would rather find that in my own new test case than in a release decision.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    release-blockerMust close before the release build

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions