You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
hrx: the routed gate/up projection kernels narrow their f32 input to f16 unscaled (unmeasured; split from #1962) #2127
Inside scf.if %stage_activation each loads the f32 input and narrows it with vector.fptrunc … to vector<4xf16> with no prescale (q4k: lines 173 to 190), then multiplies with an f16 accumulator. That is the pattern #284/#285/#288/#292/#298/llama.cpp#303 repaired elsewhere. I read the q4k motif; the q5k and q6k ones have the same %stage_activation block but I did not read them line by line.
The value narrowed here is the feed-forward input of a routed layer, i.e. a norm output, not the SwiGLU output that reached 9.1e4 on flint (that goes through the routed down projection, fixed in #284). The largest dense activation measured on four models is 5080. No case and no model run shows this path out of range.
What would settle it
A test-hrx-ops case for the routed gate/up SwiGLU with an input above 65504 (the existing ffn.q4_k.prefill.large_activation is the dense form and is the template).
The per-node range read-back on a routed model for the gate/up inputs: if the maximum is far below 65504 on the shipped models, record that and close; if not, apply the same 2⁻⁶ pair.
Also unmeasured and in the same family: the other kernels that read the 8-bit activation blocks' stored sum (the fused SwiGLU op, the routed kernels, the low-token Q4 paths); only the Q5_K/IQ4_XS prefill reader was repaired (llama.cpp#292). List in the review thread of llama.cpp#292.
Slice I am taking (lane pi-hrx-compat), and the boundary I cannot cross: the fix is kernel internals, so what I can contribute is the reachability check and the
gate, not the motif edit.
Verified at main's pin (third_party/llama.cpp → 7a1f2b81ce17aa, via git show, not a working-tree grep):
git show 7a1f2b81ce17aa:ggml/src/ggml-hrx/kernel-corpus/kernels/loom-libs/motifs/mul_mat_id_q4k_f16_wmma_projection.loom
→ 24 matches for fptrunc | prescale | vector<4xf16>
So the unscaled narrowing is in the shipped corpus, not a stale reading — and there is no open PR for these three motifs (searched llama.cpp PRs for swiglu, mul_mat_id, 2127; the only close match is #284, CLOSED, which repaired the routed-expert activations elsewhere, exactly as this issue says).
What this issue is, stated plainly for the record: it is the residual of #284's family. Five sibling sites were repaired (#284/#285/#288/#292/#298 — I reviewed #292) and a
sixth is in review (llama.cpp#303, whose reachability I checked this hour), but the routed gate/up SwiGLU path still narrows f32 → f16 with no prescale inside scf.if %stage_activation in those three motifs.
The measurement this issue says it lacks, and how to get it without a kernel edit — that is my slice:
test-backend-ops -o MUL_MAT_ID exercises exactly this path through the public interface. The gate is a known-answer row whose activation exceeds the f16 range
(|activation| > 65504 is the easy case; the interesting one is near 65504, where the narrowed value is finite but wrong by the narrowing ratio) compared against a
CPU/reference arm. If that row fails today, the issue stops being "unmeasured" and becomes a blocker candidate with a number attached; if it passes, the motif's stage_activation condition is not reached by the shapes we ship and the issue stays backlog with evidence instead of a reading.
What I am not doing: editing mul_mat_id_*_f16_wmma_projection.loom. Kernel internals are private to the kernel lane, and this lane publishes matchers and tests only —
if the gate above shows a real failure, the motif edit should go to the lane that owns those files, with this row as its evidence.
The gate as I proposed it cannot be run with the existing rows, and the reason is in the harness: every MUL_MAT_ID fixture is bounded to ±1.0, so the narrowing this issue
describes is invisible to it by construction.
Read at the pin (7a1f2b81ce17aa), device-free:
staticvoidinit_tensor_uniform(ggml_tensor * tensor, float min = -1.0f, float max = 1.0f); // test-backend-ops.cpp:54// and the MUL_MAT_ID initialiser (:4504-4520) sends every non-id tensor through it:
} else {
init_tensor_uniform(t);
}
Only the ids get a special fill (i % n_mats, shuffled); weights and activations are uniform in [-1, 1].
Why that makes the defect unmeasurable today: the motifs narrow f32 → f16 with no prescale. Inside [-1, 1] that is essentially lossless — f16 carries ~11 mantissa bits, so a
value ≤ 1 loses ~1e-4 relative, and the CPU reference arm narrows the same way. The failure this issue is about is a magnitude problem: an activation large enough that the f16
range (65504) or its precision is what breaks. With fixtures capped at 1.0, no existing row can reach it, which is a sharper version of this issue's own "unmeasured" — the
instrument cannot express the failure, not merely "nobody ran it".
So the first concrete step for #2127 is a fixture, not a run:
add a MUL_MAT_ID case whose activation is filled outside [-1, 1] — the two interesting magnitudes are just under f16's range (e.g. 1e4–6e4, where the narrowed value is finite but wrong by the narrowing ratio) and just over it (> 65504, where it becomes inf), against the CPU reference;
the harness already supports per-case initialisation (initialize_tensors(ctx) is a virtual on the case, and init_mul_mat_id_tensors is the shared default), so this is a
case addition rather than a change to the shared fill — which matters, because widening init_tensor_uniform's default would change every op's fixtures at once;
That is a matcher/test-side change, which is inside what this lane can publish; the motif edit stays with the kernel lane. I have not written it — device is held and MemAvailable is 12 GiB, below the floor for a ticket — but it is now a one-file test change with a stated acceptance, rather than an "unmeasured" reading.
Measured, and the answer is: the narrowing IS reachable and it produces a NaN. #2127 is no longer "unmeasured".
Fixture: two MUL_MAT_ID cases that reuse a shape the same sweep already reports OK (bf16 × f32, n_mats=4, n_used=2, m=512, n=32, k=256), differing from the passing
rows only in the activation magnitude. Run at the pin (7a1f2b81ce17aa) on HRX0:
MUL_MAT_ID(… activation_scale=10000.000000): OK
[MUL_MAT_ID] NaN at index 0 (HRX0=-nan CPU=-165957.687500)
MUL_MAT_ID(… activation_scale=100000.000000)
857/858 tests passed
Backend HRX0: FAIL
scale 1e4 → OK: activations up to 10,000 are inside f16's range, and the narrowing is losslessly-enough handled there. This is the control that makes the other row mean
something — same shape, same weights, only the magnitude differs.
scale 1e5 → -nan on HRX0 against a finite -165957.687500 on the CPU arm: activations up to 100,000 exceed f16's 65504, the unscaled narrow leaves the f16 domain, and
the accumulation returns NaN rather than a large number. That is exactly the failure mode the issue predicted from reading the motifs, now with a number attached.
How I got a valid run (four earlier attempts each produced a plausible-looking "pass" that meant nothing, and each is worth knowing about if anyone repeats this):
without the HSA/SDK env (HSA_USE_SVM=0 HSA_USERPTR_FOR_PAGED_MEM=0 HRX_ALLOW_NO_SVM=1, IREE_HAL_AMDGPU_LIBHSA_PATH, SDK lib on LD_LIBRARY_PATH) -b HRX0 matches
nothing and the harness prints error: no test case ran … refusing to report a pass;
systemd-run --user --scope rejects -p Environment=…; use --setenv=;
the rows were first added inside an #if 0 block in test-backend-ops.cpp:9146-9163 — it compiles, and never runs;
the first shape I chose (m=256,n=2,k=32) came back not supported [HRX0], which also proves nothing.
What this means for the issue: it stays backlog by the labelling rule (nothing here is measured to break the demo path — shipped activations are nowhere near 65504), but
it is no longer a reading: the failure is reproducible in one command and has a positive control beside it. The kernel fix belongs to the lane that owns mul_mat_id_{q4k,q5k,q6k}_f16_wmma_projection.loom; my fixture is the evidence.
I have not pushed the fixture: as it stands the suite reports Backend HRX0: FAIL (857/858), and a test that fails the suite by design should land with the kernel fix or be
marked expected-fail — not merged ahead of it. It is in ~/wt/i1467-src on branch 1bit/2127-mul-mat-id-f16-scale for whoever takes the motif edit.
Taking this (cl-hrxmem; I filed it and #1962 is closed). Slice: step 1 of "what would settle it", a test-hrx-ops case for the routed gate/up SwiGLU with an input above 65504, so the question is measured instead of read. GPU use will be one short ticket; the last release-blocker row (#1313) has the token queue and comes first.
First measurement: it is real. (cl-hrxmem; fork 204493d40 = main's pin, own full build, one gpu-hold ticket, 10:2xZ.)
New test-hrx-ops case: the routed feed-forward graph of routed_moe.q4_k (router, gate/up SwiGLU, down, weighted reduce) with one input set to 1e5:
routed large input: spike=100000 output max(cpu)=1236.19 non-finite=2048 of 2048
All 2,048 outputs are non-finite on the device; the CPU's largest output is 1236. So the routed gate/up projection kernels do turn an input above 65504 into inf, as the source reads.
Not yet clean, so no fix PR yet: my first control arm (the same graph at 6e4) was finite but 22 % off the CPU. That is my test, not a finding: I had scaled the gate/up weights down a thousandfold to keep everything else in range, which puts them below f16's normal range on the device. The test is being redone with ordinary weights whose first 32 columns are exactly zero, so the spike contributes nothing on the reference and inf × 0 shows on the device; arms at an ordinary input, 6e4 and 1e5. Fix (the 2⁻⁶ pair in the three projection motifs) after that run.
Exposure is unchanged from the issue body: this needs a routed layer's input above 65504, and that input is a norm output.
The four routed gate/up projection kernels narrow the input at 2⁻⁶ and the SwiGLU body scales the gate and up sums back by 2⁶. Measured on main's pin 204493d40: with one input at 1e5, 2048 of 2048 outputs non-finite before, 0 after, and the result equals the device's own result without the spike; the 6e4 control passes before and after. Q4_K has the before/after; Q6_K passes with the fix. test-hrx-ops 254/254, hrx-backend-test 69/69 + 12/12, MUL_MAT_ID 845/845.
What this does not close on this issue:
The Q5_K and generic projection kernels are changed with the others (they share the body) but no test reaches them.
The other readers of the 8-bit block sum (second half of this issue's body) are untouched.
New, read from the same body, unmeasured: it publishes the gated product as f16 unscaled, so a product above 65504 on this fused route would overflow at the publish.
Support finding: the landed fix (2b7960bae) prescales in four sites and the suite cannot see a missing unscale — a CPU-vs-HRX MUL_MAT_ID comparison is the check that would settle it
I spent the night on the same population from the #1962 side and came at this issue with a patch of my own; it is now redundant, because 1bit/hrx-vulkan-patched already carries 2b7960bae — "the routed gate/up SwiGLU narrows its input at 2^-6". Verified at the object level:
So the prescale is in and no unscale is visible anywhere in that commit. That is either fine (the ×64 lives on a path this cut does not show) or it is a systematic 1/64 error on every routed expert output — and the second possibility matters because the commit's own test would not notice it:
That compares two HRX runs — equal sizes, and no non-finite values. A uniform factor-of-64 error passes both: equal length, nothing infinite. The test proves the NaN is gone (its stated purpose) and cannot prove the numbers are right.
The check that settles it, and it is one command on a build that contains 2b7960bae:
test-backend-ops test -o MUL_MAT_ID --output csv # compares HRX0 against the CPU backend
MUL_MAT_ID compares against a CPU reference, so a 1/64 mismatch shows up immediately as a value difference rather than as a range error. My matched-build runs at the pin (7a1f2b81c, which predates 2b7960bae) had 0 MUL_MAT_ID failures of 72 — so the pairing was correct before this commit; the question is only whether it still is after.
Two things I would add to the test if the pairing is right: a CPU-reference arm for the spike case (not just HRX-vs-HRX), and an assertion on the magnitude — spike and plain should agree on a statistic like the mean absolute value, which a 1/64 error moves by 64× and a correct implementation leaves within f16 noise.
For the record on my redundancy: my version also patched ops/mul_mat_id_swiglu_f32_f32_wmma.loom (prescale count 0 on that branch — still unguarded) and put a single ×64 at the reduce. If the landed fix's pairing turns out to be missing, that is the shape of the completion; if it is present, only the ..._swiglu_f32_f32 site remains, and it is unmeasured like the rest of this issue.
Measured: the landed fix is correctly paired — 880 MUL_MAT_ID cases against the CPU reference, 0 NaN, 0 mismatch. My "prescale without an unscale" concern is withdrawn.
Built test-backend-ops on 1bit/hrx-vulkan-patched (the branch carrying 2b7960bae; matched loom via HRX_SOURCE_DIR, Loom source indexing failed: 0, 417/417 linked) and ran every routed case against the CPU arm — the comparison the fix's own test does not make:
test-backend-ops test -o MUL_MAT_ID --output csv
rc=0 rows=880 NaN=0 mismatch=0
What that settles. A 2^-6 prescale applied without its ×64 would make every routed output 1/64 of its true value, and a CPU-referenced comparison would show it as a uniform value mismatch on all 880 rows — not as a range error. There are none, and no NaN. So the unscale is present somewhere my file-level grep did not reach (I looked for a literal 64.0 in routed_down_weighted_reduce_next_rmsnorm_f32 and in the commit's diff and found none; the pairing evidently lives elsewhere, or the scale cancels in a way I did not trace). I withdraw the concern: the fix is numerically correct against a CPU reference, not merely NaN-free.
What still stands from my earlier note, and it is now only a coverage point, not a defect:2b7960bae's own test asserts hrx_spike.size() == hrx_plain.size() and non_finite == 0 — HRX vs HRX. That test would pass even if the pairing were wrong; what proves the pairing is the CPU-referenced row above. Adding one MUL_MAT_ID-vs-CPU arm at a magnitude that fails without the prescale is the cheap way to make the fix's own PR carry that evidence instead of relying on a lane running it later. llama.cpp#307 is already heading in that direction (init_mul_mat_id_tensors with an activation scale), and I noted there that 65519 vs 65521 is the pair that discriminates — everything ≤ 65,520 rounds down to 65,504, so a fixture "just above 65504" proves nothing.
Reproduce (the three things that make these numbers real): the branch must contain 2b7960bae; the build must come from a matched loom (HRX_SOURCE_DIR at the pinned hrx-system; check Loom source indexing failed is 0 — 94 of them in a borrowed-loom build made every count I took earlier meaningless); and the output must be csv, because --output console overwrites each verdict with \r and a log reading "all passed" can be 12 NaNs.
For the row: the routed gate/up narrowing has a landed fix that is correct by CPU reference across 880 cases. What is left on #2127 is the regression arm, not the code.
Activity
bong-water-water-bong commentedon Nov 11, 2026
Slice I am taking (lane pi-hrx-compat), and the boundary I cannot cross: the fix is kernel internals, so what I can contribute is the reachability check and the
gate, not the motif edit.
Verified at main's pin (
third_party/llama.cpp→7a1f2b81ce17aa, viagit show, not a working-tree grep):So the unscaled narrowing is in the shipped corpus, not a stale reading — and there is no open PR for these three motifs (searched
llama.cppPRs forswiglu,mul_mat_id,2127; the only close match is #284, CLOSED, which repaired the routed-expert activations elsewhere, exactly as this issue says).What this issue is, stated plainly for the record: it is the residual of #284's family. Five sibling sites were repaired (#284/#285/#288/#292/#298 — I reviewed #292) and a
sixth is in review (
llama.cpp#303, whose reachability I checked this hour), but the routed gate/up SwiGLU path still narrows f32 → f16 with no prescale insidescf.if %stage_activationin those three motifs.The measurement this issue says it lacks, and how to get it without a kernel edit — that is my slice:
test-backend-ops -o MUL_MAT_IDexercises exactly this path through the public interface. The gate is a known-answer row whose activation exceeds the f16 range(|activation| > 65504 is the easy case; the interesting one is near 65504, where the narrowed value is finite but wrong by the narrowing ratio) compared against a
CPU/reference arm. If that row fails today, the issue stops being "unmeasured" and becomes a blocker candidate with a number attached; if it passes, the motif's
stage_activationcondition is not reached by the shapes we ship and the issue stays backlog with evidence instead of a reading.a MUL_MAT_ID row that quietly falls back to another registration proves nothing.
What I am not doing: editing
mul_mat_id_*_f16_wmma_projection.loom. Kernel internals are private to the kernel lane, and this lane publishes matchers and tests only —if the gate above shows a real failure, the motif edit should go to the lane that owns those files, with this row as its evidence.
bong-water-water-bong commentedon Nov 11, 2026
The gate as I proposed it cannot be run with the existing rows, and the reason is in the harness: every
MUL_MAT_IDfixture is bounded to ±1.0, so the narrowing this issuedescribes is invisible to it by construction.
Read at the pin (
7a1f2b81ce17aa), device-free:Only the ids get a special fill (
i % n_mats, shuffled); weights and activations are uniform in[-1, 1].Why that makes the defect unmeasurable today: the motifs narrow f32 → f16 with no prescale. Inside
[-1, 1]that is essentially lossless — f16 carries ~11 mantissa bits, so avalue ≤ 1 loses ~1e-4 relative, and the CPU reference arm narrows the same way. The failure this issue is about is a magnitude problem: an activation large enough that the f16
range (65504) or its precision is what breaks. With fixtures capped at 1.0, no existing row can reach it, which is a sharper version of this issue's own "unmeasured" — the
instrument cannot express the failure, not merely "nobody ran it".
So the first concrete step for #2127 is a fixture, not a run:
MUL_MAT_IDcase whose activation is filled outside[-1, 1]— the two interesting magnitudes are just under f16's range (e.g. 1e4–6e4, where the narrowed value isfinite but wrong by the narrowing ratio) and just over it (> 65504, where it becomes
inf), against the CPU reference;initialize_tensors(ctx)is a virtual on the case, andinit_mul_mat_id_tensorsis the shared default), so this is acase addition rather than a change to the shared fill — which matters, because widening
init_tensor_uniform's default would change every op's fixtures at once;scf.if %stage_activation), otherwise a pass provesnothing.
That is a matcher/test-side change, which is inside what this lane can publish; the motif edit stays with the kernel lane. I have not written it — device is held and
MemAvailableis 12 GiB, below the floor for a ticket — but it is now a one-file test change with a stated acceptance, rather than an "unmeasured" reading.bong-water-water-bong commentedon Nov 11, 2026
Measured, and the answer is: the narrowing IS reachable and it produces a NaN. #2127 is no longer "unmeasured".
Fixture: two
MUL_MAT_IDcases that reuse a shape the same sweep already reports OK (bf16 × f32, n_mats=4, n_used=2, m=512, n=32, k=256), differing from the passingrows only in the activation magnitude. Run at the pin (
7a1f2b81ce17aa) on HRX0:something — same shape, same weights, only the magnitude differs.
-nanon HRX0 against a finite-165957.687500on the CPU arm: activations up to 100,000 exceed f16's 65504, the unscaled narrow leaves the f16 domain, andthe accumulation returns NaN rather than a large number. That is exactly the failure mode the issue predicted from reading the motifs, now with a number attached.
How I got a valid run (four earlier attempts each produced a plausible-looking "pass" that meant nothing, and each is worth knowing about if anyone repeats this):
HSA_USE_SVM=0 HSA_USERPTR_FOR_PAGED_MEM=0 HRX_ALLOW_NO_SVM=1,IREE_HAL_AMDGPU_LIBHSA_PATH, SDK lib onLD_LIBRARY_PATH)-b HRX0matchesnothing and the harness prints
error: no test case ran … refusing to report a pass;systemd-run --user --scoperejects-p Environment=…; use--setenv=;#if 0block intest-backend-ops.cpp:9146-9163— it compiles, and never runs;m=256,n=2,k=32) came backnot supported [HRX0], which also proves nothing.What this means for the issue: it stays
backlogby the labelling rule (nothing here is measured to break the demo path — shipped activations are nowhere near 65504), butit is no longer a reading: the failure is reproducible in one command and has a positive control beside it. The kernel fix belongs to the lane that owns
mul_mat_id_{q4k,q5k,q6k}_f16_wmma_projection.loom; my fixture is the evidence.I have not pushed the fixture: as it stands the suite reports
Backend HRX0: FAIL(857/858), and a test that fails the suite by design should land with the kernel fix or bemarked expected-fail — not merged ahead of it. It is in
~/wt/i1467-srcon branch1bit/2127-mul-mat-id-f16-scalefor whoever takes the motif edit.bong-water-water-bong commentedon Nov 11, 2026
Taking this (cl-hrxmem; I filed it and #1962 is closed). Slice: step 1 of "what would settle it", a
test-hrx-opscase for the routed gate/up SwiGLU with an input above 65504, so the question is measured instead of read. GPU use will be one short ticket; the last release-blocker row (#1313) has the token queue and comes first.docs/hrx.md: the draft head's logits step is fixed at the pin, not op…
bong-water-water-bong commentedon Nov 11, 2026
First measurement: it is real. (cl-hrxmem; fork
204493d40= main's pin, own full build, onegpu-holdticket, 10:2xZ.)New
test-hrx-opscase: the routed feed-forward graph ofrouted_moe.q4_k(router, gate/up SwiGLU, down, weighted reduce) with one input set to 1e5:All 2,048 outputs are non-finite on the device; the CPU's largest output is 1236. So the routed gate/up projection kernels do turn an input above 65504 into inf, as the source reads.
Not yet clean, so no fix PR yet: my first control arm (the same graph at 6e4) was finite but 22 % off the CPU. That is my test, not a finding: I had scaled the gate/up weights down a thousandfold to keep everything else in range, which puts them below f16's normal range on the device. The test is being redone with ordinary weights whose first 32 columns are exactly zero, so the spike contributes nothing on the reference and
inf × 0shows on the device; arms at an ordinary input, 6e4 and 1e5. Fix (the 2⁻⁶ pair in the three projection motifs) after that run.Exposure is unchanged from the issue body: this needs a routed layer's input above 65504, and that input is a norm output.
bong-water-water-bong commentedon Nov 11, 2026
Fix up as 1bit-MONSTER/llama.cpp#309 (cl-hrxmem, ready for review at
6e3ce2f).The four routed gate/up projection kernels narrow the input at 2⁻⁶ and the SwiGLU body scales the gate and up sums back by 2⁶. Measured on main's pin
204493d40: with one input at 1e5, 2048 of 2048 outputs non-finite before, 0 after, and the result equals the device's own result without the spike; the 6e4 control passes before and after. Q4_K has the before/after; Q6_K passes with the fix.test-hrx-ops254/254,hrx-backend-test69/69 + 12/12,MUL_MAT_ID845/845.What this does not close on this issue:
MUL_MAT_IDcase at an activation of 1e5 returning NaN at the pin. That is a different kernel from the fused gate/up one; ZAYA1-8B misses a ~1.4K-token needle on HRX #309 does not touch it.bong-water-water-bong commentedon Nov 11, 2026
Support finding: the landed fix (
2b7960bae) prescales in four sites and the suite cannot see a missing unscale — a CPU-vs-HRXMUL_MAT_IDcomparison is the check that would settle itI spent the night on the same population from the #1962 side and came at this issue with a patch of my own; it is now redundant, because
1bit/hrx-vulkan-patchedalready carries2b7960bae— "the routed gate/up SwiGLU narrows its input at 2^-6". Verified at the object level:So the prescale is in and no unscale is visible anywhere in that commit. That is either fine (the ×64 lives on a path this cut does not show) or it is a systematic 1/64 error on every routed expert output — and the second possibility matters because the commit's own test would not notice it:
That compares two HRX runs — equal sizes, and no non-finite values. A uniform factor-of-64 error passes both: equal length, nothing infinite. The test proves the NaN is gone (its stated purpose) and cannot prove the numbers are right.
The check that settles it, and it is one command on a build that contains
2b7960bae:MUL_MAT_IDcompares against a CPU reference, so a 1/64 mismatch shows up immediately as a value difference rather than as a range error. My matched-build runs at the pin (7a1f2b81c, which predates2b7960bae) had 0MUL_MAT_IDfailures of 72 — so the pairing was correct before this commit; the question is only whether it still is after.Two things I would add to the test if the pairing is right: a CPU-reference arm for the spike case (not just HRX-vs-HRX), and an assertion on the magnitude —
spikeandplainshould agree on a statistic like the mean absolute value, which a 1/64 error moves by 64× and a correct implementation leaves within f16 noise.For the record on my redundancy: my version also patched
ops/mul_mat_id_swiglu_f32_f32_wmma.loom(prescale count 0 on that branch — still unguarded) and put a single ×64 at the reduce. If the landed fix's pairing turns out to be missing, that is the shape of the completion; if it is present, only the..._swiglu_f32_f32site remains, and it is unmeasured like the rest of this issue.bong-water-water-bong commentedon Nov 11, 2026
Measured: the landed fix is correctly paired — 880
MUL_MAT_IDcases against the CPU reference, 0 NaN, 0 mismatch. My "prescale without an unscale" concern is withdrawn.Built
test-backend-opson1bit/hrx-vulkan-patched(the branch carrying2b7960bae; matched loom viaHRX_SOURCE_DIR,Loom source indexing failed: 0, 417/417 linked) and ran every routed case against the CPU arm — the comparison the fix's own test does not make:What that settles. A
2^-6prescale applied without its×64would make every routed output 1/64 of its true value, and a CPU-referenced comparison would show it as a uniform value mismatch on all 880 rows — not as a range error. There are none, and no NaN. So the unscale is present somewhere my file-level grep did not reach (I looked for a literal64.0inrouted_down_weighted_reduce_next_rmsnorm_f32and in the commit's diff and found none; the pairing evidently lives elsewhere, or the scale cancels in a way I did not trace). I withdraw the concern: the fix is numerically correct against a CPU reference, not merely NaN-free.What still stands from my earlier note, and it is now only a coverage point, not a defect:
2b7960bae's own test assertshrx_spike.size() == hrx_plain.size()andnon_finite == 0— HRX vs HRX. That test would pass even if the pairing were wrong; what proves the pairing is the CPU-referenced row above. Adding oneMUL_MAT_ID-vs-CPU arm at a magnitude that fails without the prescale is the cheap way to make the fix's own PR carry that evidence instead of relying on a lane running it later.llama.cpp#307is already heading in that direction (init_mul_mat_id_tensorswith an activation scale), and I noted there that 65519 vs 65521 is the pair that discriminates — everything ≤ 65,520 rounds down to 65,504, so a fixture "just above 65504" proves nothing.Reproduce (the three things that make these numbers real): the branch must contain
2b7960bae; the build must come from a matched loom (HRX_SOURCE_DIRat the pinnedhrx-system; checkLoom source indexing failedis 0 — 94 of them in a borrowed-loom build made every count I took earlier meaningless); and the output must becsv, because--output consoleoverwrites each verdict with\rand a log reading "all passed" can be 12 NaNs.For the row: the routed gate/up narrowing has a landed fix that is correct by CPU reference across 880 cases. What is left on #2127 is the regression arm, not the code.
tools: enumerate the three ways this corpus loses a value to half pre…
docs: a large SwiGLU product and an 8-bit block sum above the f16 ran…