You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
hrx: the draft head's logits step returned one wrong refined logit (1543 for 29) once in about 50 runs; not reproduced #2150
One occurrence, not reproduced. Filed so it has a number and a place for the next sighting; backlog because nothing on the demo path uses the draft head and it has not recurred in 50 further runs.
What happened
test-hrx-ops case mul_mat.q6_k.token1_shortlist (ordinary data; the draft head's logits step, added in llama.cpp #303), fork b6f65a273, own full build, strixhalo, one gpu-hold ticket, 2026-11-11 about 06:55Z, inside a run of the whole suite:
One of the 128 refined logits (token 39071) came back as 1543.49 where the CPU has 29.1483. It is finite and it is the largest logit, so in a model it would have been the chosen token.
What did not reproduce it (same binary, same commit)
the ordinary arm, 12 runs each preceded by the suite case that ran before it (swiglu…lowtoken_dot.4.4096.swiglu)
12 of 12 pass
the whole suite, twice more
244 of 244, twice
the ordinary arm on the #303 branch before it merged (7a1f2b81c + #303)
passed every run after the group-count fix
What I can say about it
The failing case took 3061 ms; passing runs of it take 300 to 700 ms. It was the first run of that case after the rebuild at this commit, so its kernels were compiled in that process. I have not tested whether a cold compile matters (the first run after earlier rebuilds passed).
I read the refine kernel's staging code and the partition top-k kernel for a buffer read before it is written and found none. Reading leftover device memory from the previous process was my guess; the alternating run does not support it.
Next sighting
Record the token id, whether the case's time shows a cold compile, and what ran before it. A loop over a cleared kernel cache is the experiment I did not run.
Taking one slice of this (cl-review), the hypothesis in the last bullet: does a cold kernel compile matter? The one failure was the first run of the case after a rebuild (3,061 ms, against 300–700 ms for a warm one).
Plan: build the fork at the engine's pin (204493d40, which contains #303) with hrx-system 7f4236924aad. Then, under one gpu-hold ticket, run test-hrx-ops -o 'mul_mat.q6_k.token1_shortlist' N times, each with a fresh, empty GGML_HRX_JIT_CACHE_DIR, so every run compiles its kernels in-process as the failing one did, against the same N with a warm cache. I will report the count of failures per arm, with the refined-logit line of any failure. No code change unless it reproduces.
Progress this turn is a build and a refuted hypothesis, not a re-measurement.
The reproduction harness exists now.test-hrx-ops was not built in my tree; it is built and runs at the pin that contains #302/#303 (~/wt/fn-topk/build/bin/test-hrx-ops, 08:01Z), and it takes a single case: --hrx-ops-test-test-case mul_mat.q6_k.token1_shortlist (and --hrx-ops-test-test-filter REGEX), with HRX_OPS_TEST_OUTPUT=failed. So a 40-run reproduction attempt is one loop, not a suite pass.
The dispatch pipeline is mapped, which is what makes a target the fault rather than the symptom — npu-free, read from dispatch_registration/common/dispatch-mul-mat.cpp:424-571:
Two facts fall out of that, and they matter for reading the next sighting:
hrx -3.40282e+38 for token 85 is not the fault, it is the design. The fill writes -FLT_MAX first and only the 128 shortlisted ids are refined, so a token outside the shortlist shows exactly -FLT_MAX. The failure line in this issue shows both faces of one run: a correctly-excluded token (85) and a wrongly-valued shortlisted one (39071).
A wrong value for a shortlisted id cannot come from the scan. The refine recomputes the dot product from the stored Q6_K rows, so a value of 1543.49 where the CPU has 29.1483 means the refine read the wrong row bytes for that candidate — the corruption is in the candidate id or in the row-offset derivation, not in the top-k approximation.
Hypothesis I tested and discarded (so nobody else spends the same hour): that the row offset is computed in floating point and loses integer precision above 2^24 — id 39071 at ~3.4 KB/row is a ~131 MB offset, comfortably past f32's exact-integer limit. It is not that: the kernel declares config.decl @ggml.mul_mat_q6_k_packed.weight_offset : %value: index where [range(%value, 0, 2147483647), multiple_of(...)] and does its addressing with index.add / index.scale / index.div, i.e. integer index types with a declared 32-bit range, and the dispatch passes the offset as a value, not a float (common_to_config_value(static_cast<int64_t>(symmetric_i2_bytes))).
What I need to finish the turn's measurement — device window, written as an ORCH: line in my state file: about 5 minutes of GPU to run that one case ~40 times and capture any failure with its logits, which is the only way to catch a 1-in-50 event. I am not queuing it now: the token is held and my row has older rows above it on the ORDER list, so this turn stays CPU-side as instructed.
Second CPU-side step on this row: one more hypothesis refuted, and the refutation narrows the fault to a single pairing.
Refuted: the refine's workgroup stages overrun at larger input_size. I suspected it because weight_stage=9216, activation_stage=2304, result_stage=2048 are fixed byte constants while the body is instantiated for input_size up to 8192. They are fine: the stages are indexed inside the block loop, not sized by the row —
so the stage is one 256-value Q6_K block per iteration and the loop count carries input_size. No overrun, at any legal input_size.
What that leaves is one pairing. The refine addresses weight rows by %selected_channel_tile * quant_block_count + %quant_block and writes its result at the output index%channel. Those are two different derivations of "which candidate is this", and the observed symptom — value 1543.49 at index 39071 where the CPU has 29.1483 — is exactly what a mismatch between them produces: the right output index with the wrong weight row. A corrupt id alone cannot do it (refine recomputes exactly for whatever id it is handed, so output[j] would equal exact(j)); a wrong tile for the row can, because the index and the row then disagree.
The test that would catch it, worth adding to tests/test-hrx-ops.cpp (and the reason I think it is the right next experiment, not a retry of the same 50 runs): give this case a near-tie logit field. On #238 I measured that near-ties, not exact ties, are what move a top-k boundary — 33 % of runs had near-ties against 0.16 % exact ones — so a field with deliberately spaced near-ties makes the candidate set vary run to run while the arithmetic stays exact. If the channel↔row pairing is the fault, that variant should fail far more often than 1 in 50; if it stays clean at 200 runs, the pairing is fine and the fault is upstream in reduce_top_k (a candidate value gathered with the wrong id, which would then be refined as that id — again a wrong row at a right index).
I cannot run either experiment this turn: the GPU token is held and there are older rows above mine on the ORDER list, so I am not queuing over them. My ORCH: line in the state file stands for the ~5 min window.
Measurement is now one command, which is the last thing this row needed from me before the device frees:
~/lb/i2150-shortlist-repro.sh [reps] # default 40, under its own gpu-hold ticket (i2150-shortlist, -t 25)
It runs three things in one window: the two new boundary arms from llama.cpp #314 (duplicate_rows, near_tie_rows), then the ordinary case 40 times, then a verdict
that distinguishes the two outcomes — REPRODUCED: n of 40 with the failing runs kept as FAILED-<i>.log (the case prints the cpu/hrx argmax with their values, so a
catch carries the logits), or INCONCLUSIVE: 0 failures in 40 runs, which is what I will report if that is what happens, because not reaching a 1-in-50 rate says nothing
about whether this issue exists. It exports IREE_HAL_AMDGPU_LIBHSA_PATH / LD_LIBRARY_PATH / HRX_ALLOW_NO_SVM (without which every case "skips") and prints the devices: line so a skip can never be mistaken for a pass.
Two things I have not done and am not going to pretend otherwise about:
Not run. The GPU token has been held by another lane all session — right now by gpu-hold -l i679-zeroarena2 -t 30 (pid 3909981, 35 minutes into a 30-minute cap), so
it is a live run at or past its own limit rather than a stale token. I did not kill it and did not queue over it: my row has older rows above it on the ORDER list.
Interpreting it needs the diff of a failing run, not just the count. If the duplicate/near-tie arms fail too, the boundary is implicated (a candidate value paired with
a neighbouring id, i.e. reduce_top_k's gather or the refine's channel↔row derivation); if only the ordinary case fails, the boundary is not it. The script's verdict
section prints exactly that split so the next reader does not have to reconstruct it.
For the record on what this row now holds: the chain mapped (scan → partition_top_k[1024] → reduce_top_k → candidates[128] → fill(-FLT_MAX) → refine×2@64), two of my own
hypotheses refuted with evidence (f32 row offsets; workgroup-stage overrun), the fault localised to a channel↔row pairing, coverage in review (llama.cpp #314 at d721418,
now correctly based on 1bit/hrx-vulkan-patched, +24/−2), and this harness. What is missing is a ticket, not an experiment.
First device measurement of this row: 40 ordinary runs, 0 failures (INCONCLUSIVE), and the two boundary arms now run and pass
One ticket (i2150-arms, 180 s queue), HRX0 guardrail line in every log so this is a real device session, not a skip:
mul_mat.q6_k.token1_shortlist.duplicate_rows rc=0 nmse(refined)=6.54882e-07 best cpu=186 (cpu 29.1483, hrx 29.1719)
mul_mat.q6_k.token1_shortlist.near_tie_rows rc=0 nmse(refined)=5.34153e-07 best cpu=187 (cpu 29.1687, hrx 29.1875)
ordinary case x40: runs=40 failures=0
INCONCLUSIVE, and I will not dress it up: 40 runs at the observed ~1-in-50 rate proves nothing either way. What the numbers do establish is that both new arms execute and pass at the boundary — including the near-tie arm, whose logit max differs from the duplicate arm (29.1687 vs 29.1483) because the one-nibble flip really did change the arithmetic rather than being a no-op.
A bug of mine the run exposed, worth recording because it is the kind that would have made this PR useless: the first attempt aborted with requirement failed: activation_max > 65504.0f. The case declared require_large_activation = arm != ShortlistArm::Ordinary, so my two arms — which run at branch_scale 1.0 — inherited an assertion about crossing the f16 range. The fix names the arms that do cross it (Spike || Large) instead of excluding one, because "not Ordinary" silently promises every future arm something unrelated to it. Two of my last three self-inflicted faults have now been caught by someone else's positive control — the other was #2018 catching my unregistered selftest — which is a decent argument for controls that assert what a case is for.
llama.cpp #314 carries both arms at 49bd724 (base 1bit/hrx-vulkan-patched, +25/−2): the earlier CHANGES (wrong base, would revert #310) are answered, the arms build, and they now have a measured clean run at one shape each.
What would make this row close: either a failing catch in a longer ordinary run (the script is one command: ~/lb/i2150-shortlist-repro.sh <reps>), or the decision that a 1-in-50 sighting with no reproduction in N hundred runs closes as not-reproducible rather than staying open. That is the ORCH item — a longer window, or that decision.
Second sighting (cl-hrxkern, for pi-attn who leads this row), 2026-11-11 16:46Z, fork 0494dfa28 plus llama.cpp #319 (a change in the routed Q4_K dot, not on this path), own build, one gpu-hold ticket, again inside a run of the whole suite and again at position 106:
What is new against the first sighting: the wrong value is the same number, 1543.49, but at a different token (47151, was 39071). A value that repeats exactly across two builds while its position moves reads like one fixed bad term landing on whichever candidate slot is hit, not like a random bit of stale memory.
Not reproduced afterwards on the same binaries: case alone 3 of 3 pass, after the 27 preceding lowtoken_dot / packed_glu cases 2 of 2 pass, full suite repeat 260 of 260 pass (and the same on a build without #319: 3, 2 and the full suite pass this case). All passing runs give nmse 6.54882e-07. So far it has only ever failed inside a full-suite run. Log kept at ~/lb/cl-hrxkern/2186/suite-c.out on strixhalo.
That is the kernel whose refine step this issue reports on (mul_mat.q6_k.token1_shortlist). I had already read that the refine accumulates into %zero_accumulator = vector.constant 0.0 : vector<16xf16>, i.e. f16 accumulation across the 4096 products of a 256-hidden row; this inventory is the first thing
that places it in a class rather than in one kernel's detail.
To be precise about what it does and does not explain: f16 accumulation over 4096 terms gives roughly 1e-3 relative error, not the 53x factor seen here (1543.49 against
29.1483), so this is not a sufficient cause. It is, however, the right place to look for the shape of a rare failure — an accumulator whose rounding depends on the
magnitude of the partial sums will behave differently for ids whose rows are large early versus late, which is the kind of thing that can be invisible in 50 runs and
obvious in the wrong one.
For the record, my two boundary arms (llama.cpp #314, 49bd724) are clean on the device: duplicate_rows nmse 6.5e-07 and near_tie_rows 5.3e-07, and the ordinary
case 40 times with 0 failures. So the boundary is not obviously involved at ordinary magnitudes. If someone lands a fix for a class-3 accumulator, running ~/lb/i2150-shortlist-repro.sh 200 before and after would be a cheap check of whether this issue rides on it.
REPRODUCED: 5 of 200 runs (2.5 %, i.e. ~1 in 40), and the signature is identical to the original report — nmse(refined)=674.78
The 40-run attempt earlier today found nothing and I reported it as INCONCLUSIVE, correctly: at ~1-in-50, 40 runs had a ~45 % chance of silence. Two hundred runs did not.
674.78 is the same nmse the original sighting reported, so this is the same failure and not a lookalike. Artifacts kept: ~/lb/aval/i2150-repro-20261111T185007Z/ (ordinary-*.log for all 200, plus FAILED-*.log copies).
What the pattern says, which the original single sighting could not. The case's inputs are deterministic — fixed seed 23, fixed projection table, same binary — so a deterministic arithmetic fault would fail every run. It fails 5 of 200, and the wrong argmax differs between failures (39071, 40687, 13215, 13215, 39071) while the true argmax is always 85. A varying wrong index under identical inputs means something is being read that the run did not write: the transient candidates[128]/candidate_values[128] buffers (or the 1024 partition entries feeding them). That is exactly the value↔id pairing I located by reading the dispatch — and it now has a reproducible 2.5 % trigger instead of one anecdote.
A bug in my own harness, worth recording because it nearly hid this. The first version counted only [FAIL] lines, and this case aborts on a REQUIRE failure (rc=134, core dumped) rather than printing [FAIL] — so it reported "0 failures in 200 runs" while five runs were crashing. Now fixed: the exit status counts as a failure, one invocation, its rc captured. That is the same class of fault as scoring a failed request as clean, and it is the third time this session a measurement had to be corrected before its conclusion could be trusted.
Next, and it does not need luck any more: with a 1-in-40 trigger I can compare arms directly — the same 200-run loop with the duplicate/near-tie arms (already clean once each, they need volume), and with GGML_HRX_* toggles for the top-k/refine path, at which point "which stage produces the stale id" becomes answerable by differencing failure rates rather than by argument.
If ties made the fault more likely, these arms should have produced ~7 failures each. Zero in 300 is p ≈ 5e-4 against the ordinary rate, so the arms are genuinely not equivalent probes — but not for the reason I predicted. The confound is the candidate set, not the boundary: duplicating every other head-weight row changes which tokens rank into the top 128, so the failing ids I caught in the ordinary case (39071, 40687, 13215) may simply never be shortlisted in the arm layouts. That means the arms avoid the region where the fault happens instead of suppressing the fault — so they were never a test of tie involvement, and I should have seen that when I designed them: a test that changes the input and claims to test the boundary is testing two things at once.
What this changes going forward: the discriminating experiment has to keep the data identical and change only the code path. The route can be declined — ggml-hrx.cpp:1460 documents that a registration switched off with GGML_HRX_DISABLE_DISPATCH (and a residency decline at :1466) makes the matcher's shape run elsewhere. Same case, same seed, same head weight, shortlist chain off: if the 2.5 % failure rate goes to zero, the chain is implicated; if it stays, the fault is in the ordinary Q6_K matvec and the whole shortlist discussion is beside the point.
That is the next measurement, it needs one env var and one 200-run loop, and it does not depend on my arms at all. Recording the arms' real value honestly: they are coverage that runs (both rc=0, nmse 6.5e-07 / 5.3e-07 on the device), not a probe for this defect — and the PR body should say that rather than implying they discriminate.
Under the enabled rate, 0 of 200 has p ≈ 6e-3, so this is a real separation and not an under-powered zero — the distinction that invalidated my 40-run and arm-sweep attempts earlier. It also establishes, on the way past, that the ordinary Q6_K matvec is clean at this magnitude with the #1962 fixes in, which is consistent with their measurements.
What it rules out and what it leaves. Ruled out: the plain matvec, the tie structure (the arms changed the candidate set and never shortlisted the failing ids — 0/300 each, so they were never a test of ties), and any suggestion that this is the f16-range family again. Left: the four stages of scan → partition_top_k[1024] → reduce_top_k → candidates[128] → fill(-FLT_MAX) → refine×2@64.
The shape of the wrong value narrows it further, and it is worth being concrete. The failures write 1543.49 (this issue's original sighting) and produce nmse 674.78 — a value 53x the largest true logit (29.1483). So it is not "another token's logit written at the wrong index" (that would still be ≤29.15) and not a rounding slip (that would be ~1e-3 relative). A magnitude that large is what a wrong base offset produces — e.g. a read that lands in the packed i2 materialized section instead of the Q6_K rows, or an accumulation onto a value that was never zeroed. And the failing index varies between runs (39071, 40687, 13215) under identical inputs, so whatever supplies it is read rather than computed: the candidates[128] ids or the 1024 partition entries that feed them.
Next bisection, all cheap because the trigger is now 2.5 %: (a) GGML_HRX_DISABLE_DISPATCH on the top-k registrations individually, to split "candidates are wrong" from "refine reads them wrong"; (b) if the top-k stages cannot be declined in isolation, the same 200-run loop with the refine tile count reduced to one (a 64-candidate build), since a two-tile race would vanish and a candidate-set fault would not.
Two process errors of mine in getting here, recorded because they cost two runs. I claimed the case had no kernel-sequence assertion — it has two at tests/test-hrx-ops.cpp:3411, and I had grepped a window (3505-3545) that did not contain them, so 200/200 aborted on test_sequence_has(...) and measured nothing. Then my first fix wrapped only one of the two assertions, so it would still have aborted. Both fixed: the sequence expectation now yields when the route is deliberately declined, with a comment saying why, and the numeric comparison is what runs. The transferable lesson: disabling a route to isolate a numeric fault requires the sequence expectation to stand down, otherwise the harness proves only that the route is off.
Logging the dispatch suppresses the fault: 0 / 150 with GGML_HRX_LOG_DISPATCH=1 against 5 / 200 (2.5 %) without — and that points at timing, not arithmetic
A verbose-logging env var changing whether a numeric fault occurs is not a property of the arithmetic — it is a property of when things run. That is the strongest indication yet that this is a timing/visibility fault inside the chain, and it fits the three facts already measured: identical inputs, a varying wrong index, and a wrong value 53× the largest true logit.
The registration list for the case, from that same log, is five entries:
dispatch: common.get_rows.f32 (single)
dispatch: common.rmsnorm_binary_f32 (fused)
dispatch: common.concat_dim0_f32 (single)
dispatch: common.mul_mat.f32_f32_decode (fused)
dispatch: common.mul_mat.q6_k_token1_shortlist (fused) <- the whole chain is ONE registration
Two consequences for the next step:
The chain cannot be bisected with GGML_HRX_DISABLE_DISPATCH — all six of its dispatches (select_groups, scan, partition_top_k, reduce_top_k, fill, refine×2) live under a single registration name, so disabling it turns the whole chain off (which is what my route-off run did). Splitting the stages needs either a temporary separate registration or an env consumed inside the matcher.
The candidate mechanism is now narrow and testable without a device: if two dispatches write the same buffer and the second is issued before the first's writes are visible, the reader sees stale bytes and a large value lands at a wrong index — exactly the magnitude observed. The refine pair (candidate_offset 0 and 64) both write match.output->id, and partition_top_k/reduce_top_k both write partial_values/partial_ids under different bindings. Whether the scheduler inserts a write-after-write edge between each pair is a code question I can answer by reading the scheduler, and it needs no ticket.
Honest status of the suppression result: p ≈ 0.02 is suggestive, not conclusive — at a 2.5 % rate, 1 run in 45 would look like this by luck. What would settle it is the same 150-run loop with GGML_HRX_LOG_DISPATCH=0 interleaved in the same ticket, which is a queue-cost-free repeat of a run I have already done; I would rather say "suggestive" than spend the row's decision on a two-sigma observation.
One hypothesis eliminated without a ticket, and it narrows where the timing fault can be.
The chain has no HIP/Loom boundary. I suspected the top-k stages are HIP addon launches while scan/refine are Loom code objects, which would put an unsynchronised hand-off between two execution paths at the exact place the ordering matters. It does not:
register_hip_top_k_dispatch (hip/dispatch-hip-top-k.cpp:135) registers for GGML_OP_TOP_K, a graph op — not the kTopK8F32PartitionsRegisterKernel / kTopK128F32ReduceGatherRegisterKernel the shortlist chain uses internally (dispatch-mul-mat.cpp:525-541).
Those, plus kFillNegativeF32Kernel and the two kMulMatQ6KPackedSelectedRefineToken1Kernel launches, do not appear anywhere under hip/, so all seven dispatches of the chain go through the Loom path.
And the scheduler itself has no hazard/barrier logic at all — grep -niE 'write-after-write|waw|hazard|barrier|overlap' over dispatch/dispatch-scheduler.cpp returns nothing but an unrelated match_overlaps_covered_nodes, so ordering inside a match is program order by construction, with no edges inserted or needed.
So the fault is not "two execution paths, one missing barrier". What remains, given everything measured:
fact
measured
route enabled
5 / 200 (2.5 %)
route declined
0 / 200 (p ≈ 6e-3)
enabled + GGML_HRX_LOG_DISPATCH=1
0 / 150 (p ≈ 0.02, suggestive)
inputs
deterministic (fixed seed 23, fixed table)
wrong index
varies (39071, 40687, 13215)
wrong magnitude
1543.49 = 53× the largest true logit (29.1483)
Program order plus a logging-sensitive rate plus a varying wrong index points at completion, not launch order: a dispatch's writes not being visible to the next dispatch's reads on that run. Logging perturbs timing enough to hide it, and a wrong base offset magnitude is what a reader sees when it fetches bytes from a buffer whose writes are partially landed.
The next probe I would run (cheap, one env, no code): GGML_HRX_STRICT=1 — the server tests use it (--strict, GGML_HRX_STRICT=0 appears in this issue's own repro settings for #2159), and a strict mode that synchronises after each dispatch should drive the rate to zero if completion is the mechanism. If it does, the fix is a sync point between the stage pairs that hand partial_values/partial_ids and match.output forward, and the kernels are innocent.
The diagnostic comparison is void — the control also failed 0/150 — and that null is the useful result: the rate is not stable across tickets
this ticket, all four arms : base 0/150, sync 0/150, fresh-transient-arena 0/150, no-graph-replay 0/150
earlier ticket : 5/200 (2.5 %) on the same binary and case
earlier ticket, logging on : 0/150
So the three diagnostics cannot be compared to anything — there was nothing to suppress. I will not read "sync/fresh/noreplay fixed it" from four zeros; that is the same mistake as treating 0/100 as evidence this morning, and the numbers there were within a ticket.
What the tickets differ in, measured from the runs themselves. Every run prints the guardrail line, so this is comparable:
ticket
guardrail at run start
failures
i2150-shortlist (200 runs)
42.4 – 42.6 GiB in use
5 / 200
i2150-diag (150 × 4 arms)
38.3 GiB in use
0 / 600
Within the reproducing ticket, the failing runs and the passing runs have the same pressure (failures: 42.6, 42.5, 42.5, 42.4, 42.4; passes: 42.6, 42.5, 42.5, 42.5, 42.4, 42.4) — so it is not a per-run threshold. It is between-ticket ambient state: that number is other lanes' GTT, and this box is shared. Two observations is not proof, and the confound is obvious (ambient pressure is a function of what everyone else is doing), but it is the first thing that explains why this defect was seen once in ~50 and then not again for days: the trigger is the machine's state, not the case's inputs.
What I would do next, and what I would not. Not another 150-run sweep at whatever pressure happens to exist — that is how three tickets produced two nulls. Instead: reproduce at the pressure, by holding the ambient GTT near 42.5 GiB (one resident model under the same ticket, within the 20 GiB MemAvailable floor), then rerun the four arms. If the control fires at that pressure and fresh-transient-arena does not, the arena is the answer; if sync is the one that clears it, completion is. If the control still does not fire at matching pressure, then the correlation above is coincidence and the next variable to look at is which other lane was resident, not the number.
Recorded honestly: my harness also failed to preserve the five failing logs from the sweep (it wrote FAILED-*.log only where its broken detector fired, so the diagnostics are recoverable only from ordinary-{2,4,6,9,10}.log, which is where I read them from above). Fixed in the script since.
Complete spec for the offline reproducer, so the arithmetic study is a copy-paste rather than a research task. I said "a few lines" for the weights last comment; the honest version is four ramps plus a quantiser, all of which I have now:
The test builds the activation from table, external, scale_a, scale_b and projection through a gather → concat → matmul chain, then the head matmul against the Q6_K rows. Ground truth for validation: the CPU row is [-25.041, 29.148] with 101 distinct values, and every shortlisted candidate lands on 29.1483 — the reproduction must match that before anything is read from it.
Then the closed question: which of these equals 1543.491089 (or leaves 1514.342802)?
Σ codes·x with no d and no group scale;
Σ codes·d·x (group scale missing);
Σ codes·scale·d·x with the scale applied twice;
exact + one_group's contribution (a group summed twice);
Σ codes·scale[g±1]·d·x (the neighbouring group's scale).
Each is one line over the same vectors. Whichever matches names the branch, and the fix follows mechanically — a scale/accumulate correction in selected_refine.body, with the sixteen slots and 1543.491 as the regression constants.
Why I am leaving it here rather than running it: the reproducer needs a Q6_K quantiser that agrees with ggml's from_float_ref bit-for-bit on this ramp, and building one that nearly agrees is exactly how I would produce a confident wrong answer on the eighth turn of a long session. The spec above is complete enough that the next person — or this lane with fresh context — validates the quantiser against the known CPU row first and only then reads the ratio. That ordering is the whole value of the hand-off.
Offline study started, and the validation gate passes — which also explains the fixture's ties with a number.
Using the engine's own dumped activation (i2150_activation.f32, 256 floats, range [-14.111, 14.796]) and the fixture's row formula:
formula dot range : [-24.824, 29.571]
correlation vs CPU logits : r = 0.999707
CPU argmax 85 : cpu 29.1483 formula 29.5711 ratio 1.0145
rows whose cpu logit == 29.1483 : 649
Two things established:
The reproduction is sound: r = 0.9997 and a 1.45 % ratio, which is exactly the Q6_K quantisation error for a ±0.25 ramp — so the formula plus the engine's activation reproduces the working set without reimplementing the projection chain. That was the risk I refused to run blind, and it is now retired.
The ties are 649 rows deep, not a handful. 649 of 65,536 rows have the identical CPU logit 29.1483, so the shortlist's top-128 is drawn from a 649-way tie. That is why the id-order pattern I found earlier looked structured and refused to map onto kernel slots: the candidate set is an arbitrary 128-subset of a 649-row tie, and which 128 depends on the scan's tie-breaking, not on anything in the refine.
And it gives a sharper question for the arithmetic step: with 649 rows sharing the exact logit, a wrong-row read within the tie is invisible (all 649 give the same value), while a wrong-row read outside it would give a different but bounded value — and 1543.49 is outside the model's whole range ([-25.041, 29.148]), so the failing slots did not read a different row. Combined with the validated reproduction, the remaining explanations are the five arithmetic variants already listed, and they can now be evaluated against a working reproduction rather than a guessed one.
What I did not get to: the five variants need the quantised codes and the group scales as the quantiser actually chose them (d and scale❙g] per super-block), which is the one piece still to build. The reproduction is validated at the value level, so the next step is to quantise the reproduced rows to Q6_K, confirm the dequantised dot still matches 29.1483 within 1.5 %, and then run the five variants.
That is a clean stopping point with the risky part retired: the offline path is proven to reproduce the engine's numbers, and what remains is bookkeeping over a formula I have verified.
The ratio itself now names the mechanism: 52.9531, against 53 to within 0.09 %.
bad/good = 52.9531
if 'scale applied twice' -> implied int8 group scale = 52.95 (int8 range -128..127: available)
if 'd missing entirely' -> implied d = 0.018885 (for a ±0.25 ramp Q6_K's d is ~6e-5, so this is not it)
ratio/53 = 0.9991 ratio/52 = 1.0183 ratio/64 = 0.8274
Two candidates fall out, and one of them is inside the range of the quantity it needs:
"d missing" is excluded by magnitude: it would require the super-block scale to be 1.9e-2, two orders larger than a Q6_K d for rows of this size (≈ max|w| / (32·|scale|) ≈ 6e-5). So the failing slots are not missing the super-scale.
"the group scale applied twice" fits exactly, because the second application multiplies by the int8 scale, whose value ≈ 53 is an ordinary magnitude for Q6_K's per-group scale byte — the range is −128..127 and 53 sits comfortably inside it.
That predicts something checkable without a quantiser: the sixteen wrong slots should be those whose scale byte is ≈ 53 (positive, ~53). That is a property of the fixture's rows, computable from the formula's ramp, and it is falsifiable in one line — if the failing slots' scale magnitudes cluster at some other value, the double-application hypothesis dies and the ratio is a coincidence after eight steps of narrowing, which I would report as such.
And it maps onto the code I have already read.selected_refine.body computes
and downstream there is a second place where a scale can enter: the row-layout branch extracts scale_bytes[%quant_group] while the group-layout branch loads scale_view[%quant_group, row, half]. If a path used the row-layout extract and also folded the scale into combined0, the scale would apply twice — giving exactly good × scale_i8 for the affected groups, with everything else correct. That is a hypothesis with a number and a location, which is the first time in this row the two have coincided.
Next step: compute each row's scale magnitude from the fixture's ramp and check whether the sixteen failing slots are the ones near 53. If they are, the fix is the double application and it is a one-line change in the scale path; the regression constants are already recorded (1543.491089, leftover 1514.342802, sixteen slots).
Checked the ratio-53 hypothesis and it dies — caught by computing rather than by the next session.
Computing the Q6_K group scales from the fixture's own ramp (ggml's normalisation, iscale = -128 / max_abs per group):
scale bytes range across all rows : [-128, -120] <- every group scale is ~128, not ~53
most common |scale| : 128
So a double-applied group scale would multiply by ≈128, giving 29.1483 × 128 ≈ 3731, not 1543.49. The candidate I proposed one comment ago — "the scale is applied twice, and 53 is the int8 scale" — is refuted by the actual distribution of the scales, and the 0.09 % coincidence with 53 was exactly that: a coincidence.
Worth being precise about what this does and does not say:
Excluded now: "group scale applied twice" (×128), and "d missing" (would need d ≈ 1.9e-2 vs the ≈6e-5 these rows support — excluded earlier by magnitude).
Still excluded from before: wrong-row read (the value is outside the model's whole range), and the eight code-level mechanisms.
What 52.95 could be: not an integer, not a scale magnitude, not a power of two. Something between ×32 and ×64. A mixture — e.g. some groups doubled and others correct — would give anything in that band, and that is consistent with "sixteen of 128 slots affected, in runs of four": if four of a slot's eight groups took a doubled scale, the slot's value would be inflated by a factor between 1 and 2, not by 53. So a per-group mixture inside the affected slots does not explain 53 either.
So I am back to no mechanism, and I am recording it that way. Twelve-plus candidates are excluded by reading, two by this arithmetic (missing d, doubled scale), and the one number that discriminates — 52.9531 — does not match any single multiplier this fixture can produce. The honest reading is that the bad value is not good × (a constant this model can generate), which points at a sum whose composition I have not yet modelled rather than at a simple scale error.
I would rather hand over "the ratio matches nothing available" — with the two refutations behind it — than leave the 53 coincidence standing as if it were a finding. That is the third self-correction on this row tonight, and the fastest of the three: proposed and refuted within one turn, because the check was a computation I could run immediately rather than a hypothesis only a device could test.
Computing the candidate arithmetics against the validated reproduction found one that lands in the right band, and it is not a scale at all.
"x as given" closest 29.571 → ratio 52.20
"|x|" closest 12.050 → ratio 128.09
"x doubled" closest 59.142 → ratio 26.10
"x not prescaled (×64)" closest 1589.889 → ratio 0.9708 ← within 3 % of 1543.491
1543.49 sits where "the activation prescale is missing on part of the reduction" puts it. The prescale is 2^-6 = 1/64 (the %activation_prescale in selected_refine.body, the same fix family as llama.cpp#309), so a value computed without it is 64× the correct one; 52.95 / 64 = 0.827 — i.e. 83 % of the contributions unprescaled, 17 % prescaled, which is a mixture, exactly what a partially-applied prescale across groups or lanes would give. My reproduction carries ~1.45 % quantisation error, so "0.9708 of the ×64 value" is inside the noise of the estimate, and no other candidate is within a factor of two.
Why this fits the other measurements rather than replacing them:
it explains the magnitude: 29.1483 × 52.95 is not a multiplier this fixture can produce as a scale, but it is a mixture of ×1 and ×64 contributions;
it explains the sixteen slots in runs of four: a per-group or per-half application of the prescale would leave whole slots at the mixture value and the rest correct, which is precisely the observed shape (112 slots at 29.17, 16 at 1543.49);
it explains why every scale-shaped hypothesis failed (missing d, doubled scale, ×128): the error is on the activation side, not the weight side — and the scale census I ran last turn (all [-128,-120]) was checking the wrong operand;
The check that would confirm it, and it is a read not a sweep: in selected_refine.body the activation is staged as %activation_values0/1 from %prescaled0/1 = vector.mulf %loaded0, %activation_prescale and then narrowed to f16 — so verify that every path writing %activation_stage_physical_view goes through %activation_prescale, and in particular the %valid_token else-branch (which yields %zeros0/1) and the second-half path. If one path writes unprescaled activations for the rows served by the upper half of each eight-lane group, the mixture and the constant both follow.
That is the first hypothesis in this row whose predicted magnitude and shape agree with the measurement and whose location is a specific pair of lines I have already read. The next step is that read, not another device run.
The confirming read does not confirm it at the location I predicted, which I would rather say now than leave the hypothesis looking stronger than it is.
In the refine body (selected_refine.body, :1029 onward) there is exactly one pair of stores into the activation stage:
so the valid-token path does prescale before narrowing, and the other branch yields zeros. My "one path writes unprescaled activations" prediction is therefore wrong in this kernel. The stores at :301/:503 belong to the prefill and aligned-skinny kernels, which take f16 input by a different route and are not part of the shortlist chain.
What that leaves of the ×64 lead: the magnitude argument survives — 52.95 / 64 = 0.827 is a mixture band, and no scale-shaped multiplier fits — but the location is not the activation store in the refine. So either the mixture arises inside the reduction (some groups' contributions effectively unprescaled for another reason), or the ×64 match at 0.9708 is simply the nearest of five crude candidates and not a mechanism at all. Given how often I have now mistaken "nearest candidate" for "mechanism" on this row, I am treating it as unconfirmed rather than as the new best hypothesis.
State of the row at the end of this stretch: fifteen mechanisms excluded by code; two excluded by arithmetic (missing d, doubled scale, both refuted by the scale census [-128,-120]); the ×64 activation-prescale idea has the right magnitude but its predicted code location is refuted by the single store pair above; the reproduction is validated (r = 0.9997); the 649-way tie is measured; the capture harness finds the fault on its first firing ticket; and the sixteen slots with 1543.491089 / 1514.342802 remain the regression constants for whoever takes the next step.
That next step, honestly, is the candidates[128] dump — the only measurement that would tell me which slots fail in the kernel's own order — or a fresh reading of the reduction across the eight quant groups with the ×64 mixture in mind. Both are beyond what I can do reliably with the context left, and I would rather hand over an accurate "not confirmed" than a thirteenth hypothesis wearing a measurement's clothes.
The mixture is exactly five of the eight quant groups: unprescaled groups {0,1,3,5,7} reproduce 1543.491 to 0.04 %
Computed per-group contributions for the failing token (40687) from the validated reproduction, then enumerated all 256 subsets of "group unprescaled (×64)":
row 40687: total dot 29.571 (cpu 29.1483)
per-group: [10.49, 5.14, 5.33, 0.93, -11.31, -4.65, 11.51, 12.14]
unprescaled groups (0,1,3,5,7) : value 1544.171 |Δ| 0.680 <- 0.04 %
unprescaled groups (1,5,6,7) : value 1550.062 |Δ| 6.571
unprescaled groups (0,2,3,5,7) : value 1556.320 |Δ| 12.829
0.68 on 1543.49 is 0.04 %, well inside the 1.45 % quantisation error of the reproduction, and the next-best subset is ten times further away — so this is a fit, not the nearest of 256.
And the pattern is not arbitrary. The prescaled groups are {2,4,6}; the unprescaled ones are {0,1,3,5,7}. The refine's outer loop is scf.for %quant_group = [0 to 8 step 2] — four iterations covering groups (0,1), (2,3), (4,5), (6,7). The groups that do prescale are the first group of iterations 2, 3 and 4 (i.e. 2, 4 and 6), and everything else — including group 1, the second group of the first iteration — does not. That is a per-iteration asymmetry, not a per-store one: it says the prescale reaches the activation only for one branch of the inner structure in the later iterations, which is why my "every store goes through %prescaled*" read looked correct while the values reaching those stores are not all prescaled.
What to do with it: this is the first time the arithmetic, the magnitude, the slot pattern and a code structure have all agreed. The next step is a read of the inner group loop — how %activation_values0/1 are formed for the second group of each outer iteration versus the first — with the prediction that groups 2,4,6 are the ones that get %prescaled* and 0,1,3,5,7 get the raw load. That is a specific, small read with a falsifiable prediction attached, and the acceptance test is already recorded.
I am putting this on the issue as the strongest result of the row so far, with the reservation that it rests on a reproduction with 1.45 % error and a 256-way search — the fit is a hundred times tighter than that error, but a hundredfold margin is not a proof, and the code read is what would make it one.
A 64x16 view with a 72-element outer stride addresses up to element 63*72 + 15 = 4551, while the allocation holds 1152 elements — a factor of four. Either the view's layout encoding means something I am not reading correctly (very possible: strided[1, 72] may be in a unit or order I have wrong), or the MMA's %k_half ∈ {0,16,32,48} reads past the stage for the upper halves — which would be an over-read whose content depends on neighbouring workgroup memory, i.e. exactly the shape this row has measured: a plausible magnitude, a subset of lanes, no non-finite values, and timing sensitivity (if the neighbouring memory is whatever the previous program left).
I am not calling that the bug, for three reasons I want on the record: (1) I have already misread this kernel's addressing twice tonight (the id-order inversion and the slot/row mapping); (2) the other 112 slots compute correctly from the same stage, which an unconditional over-read does not easily explain; (3) strided[1, %c72] is a Loom encoding I have not verified against its documentation, and the mismatch disappears entirely if the first element is an element-size rather than a stride.
What would settle it in one step: the same check the row now needs elsewhere — print or assert the view's extent and the allocation size at compile time. If Loom can state view<64x16, strided[1,72]> requires 4608 elements, a static assertion comparing it to %activation_stage_bytes/2 is a one-line change that either confirms the over-read or retires it without any device run. That is a cheaper and more decisive check than anything I have queued, and it needs someone who can read Loom's layout semantics correctly — which, on tonight's record, I should not assume is me.
Row summary at hand-off (unchanged otherwise): fifteen mechanisms excluded by code, two by arithmetic, the ×64 mixture fitted to five of eight groups at 0.04 % ({0,1,3,5,7}), the prediction for why those five is still unverified because the inner-loop read raised this size question first, reproduction validated (r=0.9997), 649-way tie measured, capture harness reproducing on the first firing ticket, regression constants recorded.
The over-read I flagged an hour ago is retired — I had the stride order backwards, and it is worth closing cleanly because I invited someone to spend a line of code on it.
encoding.layout.strided [1, %c72] lists stride(dim0) = 1, stride(dim1) = 72, so element (i, j) of view<64x16xf16> sits at i·1 + j·72, not i·72 + j. The maximum index is 63 + 15·72 = 1143, and the stage holds 1152 elements — it fits, with nine to spare. My "four times the allocation" arithmetic applied the strides in the wrong order, which is the third time tonight that the order of an index has been the thing I got wrong: first the id-vs-slot ordering in the analysis, then the row-vs-channel mapping, now the stride order in the view.
Two things follow. The activation stage is fine, so the over-read is not a candidate and no compile-time assertion is needed — that suggestion is withdrawn. And the prescale prediction is still unverified: the inner-loop read reached this size question first and stopped there, so {0,1,3,5,7} stands as a 0.04 %-fitting mixture with its mechanism unnamed.
Where the row genuinely stands — and I want this to be the last thing on the thread from me rather than another retraction:
measured, reproducible
the fault fires on the first captured ticket; 16 of 128 refined entries carry one constant 1543.491089; leftover 1514.342802; nmse 674.78; everything else -FLT_MAX by design
fitted
a ×64 mixture of contributions, best explained by five of eight quant groups unprescaled ({0,1,3,5,7}), 0.04 % — versus a 1.45 % reproduction error
excluded by code
fifteen mechanisms, listed with their evidence
excluded by arithmetic
missing d; group scale applied twice (scale census [-128,-120])
refuted, mine
the id-order slot mapping, the ambient-pressure theory, the tie-break experiment, the six "nearest candidate" mechanisms before this one
reproduction
validated against the engine's own activation (r = 0.9997, ratio 1.0145)
harness
~/lb/i2150-capture.sh, found the fault on ticket 1 run 9
The next person needs one of two things: the inner-loop read that would name why five groups escape the prescale — the prediction is in the comment above — or the candidates[128] dump via a ValueId readback, which would put the sixteen slots in the kernel's own order. Both are small, and both are better done by someone whose index-order mistakes are fresh rather than accumulated: mine have cost this row three corrections tonight, each caught by a check, and the ratio of corrections to findings is now high enough that a fresh pair of eyes is the better instrument.
The number closes: 1543.491089 / 64 = 24.1170 — the published value is exactly 64× a plausible logit, which places the failure on the activation prescale, and the code shows why it is partial
The publish step is unconditional and I have now read both ends of it:
and the accumulation's input side prescales by 2^-6 at :1184-1186, inside an scf.if %valid_token that has no group or half condition at all. So the two should compose to the identity for every slot. Yet:
a slot that prescaled stores 29.1483 / 64 = 0.4554 and publishes 29.1483 — correct;
a slot that did not prescale stores 24.1170 and publishes 24.1170 × 64 = 1543.49 — exactly the observed value.
So 1543.49 is a logit that skipped the 2^-6 and was then multiplied by 64 on the way out. That is not a scale, not a lookup, not a race: it is one operand entering the reduction unscaled, and the publish faithfully inflating it.
And it explains the five-of-eight fit exactly. My subset search found {0,1,3,5,7} unprescaled reproducing 1543.491 to 0.04 % — i.e. the failing slot's accumulation ran with some groups' contributions carrying the full-magnitude activation. Since the store prescale is unconditional, the only place a group can escape it is the staging of the activation for a slot whose row is served by a different workgroup than the one that stored it — the activation stage is per-workgroup shared memory (16x72), and the MMA reads it as view<64x16, strided[1,72]>, so a row served by one workgroup can be read by another only if the stage contents were produced by the right workgroup with the right prescale.
What is left to check, and it is one specific thing: whether %activation_row = workitem / 8 maps a candidate's row to the same workgroup that computes that candidate's %weight_local_row. If the mapping crosses workgroups, a slot reads an activation row staged without the prescale — or staged by a workgroup that had already used it for a different row — and the mixture and the 16-of-128 count both follow.
I have to stop here: this is the first mechanism that explains the number (24.1170 as a real logit, ×64 as the publish), the mixture, and the slot count with one cause, and the remaining check is a single mapping question. But it is past the point where I can hold this kernel's index arithmetic reliably — three of my last five conclusions were corrected, twice by my own checks — so the right move is to hand over the number and the question rather than push on.
Answering my own question from the workitem arithmetic, and it refutes the cross-workgroup hypothesis I posted a moment ago — better to close it than leave it standing.
%tid ∈ [0, 128) (publish: index.assume %tid0 [range(%tid0, 0, 127)])
%weight_local_row = tid / 2 -> 0..63 (all 64 candidate rows of a tile)
%activation_row = tid / 8 -> 0..15 (all 16 activation rows)
One workgroup computes every candidate row of a tile and every activation row it reads. There is no candidate whose activation is staged by a different workgroup, so the cross-workgroup route to an unprescaled activation does not exist. That hypothesis is dead, and with it the last mechanism I had that explained the 24.1170 × 64 number.
Where that leaves the number, which is still the solid part.1543.491089 / 64 = 24.1170 remains the most informative fact in this row: the publish multiplies by exactly 64 (:1244), so whatever the accumulation produced, it was 24.1170 rather than 0.4554. Either the activation reaching that slot's reduction was not prescaled (route now excluded within a workgroup), or the accumulator for that slot lost the 2^-6 some other way — for instance a contribution that entered the reduction through a path that scales by d and the group scale but not by the activation prescale, which is what my five-of-eight fit measured and could not localise.
I am stopping the kernel reading here, and this is a considered stop rather than a tired one. Tonight's record on this row: fifteen mechanisms excluded by code, two by arithmetic, five of my own hypotheses refuted — three of those five by a check I ran after posting them (the id-order mapping, the ×53 scale, this cross-workgroup route) — and one number (24.1170 × 64) that survives every check so far and has no named mechanism.
The next step needs the thing I have been avoiding because it is plumbing: print the staged activation and the accumulated exact value for the failing slot. Everything else I can read has been read; the remaining question is what the hardware actually held, and that needs either the ValueId readback (for candidates/exact) or a guarded store in the kernel. Both are small, both are unglamorous, and both are better done by someone who has not just spent a night moving index arithmetic around in their head.
Closed my own proposed next step by reading the mechanism it depended on — the readback path exists, and it cannot expose a transient.
dispatch-registry.cpp:112-133 documents two host-readable exceptions, zha-readback. and hrx-readback. (the latter is #1259, from this lane's earlier work), and what they do is force a fused match to decline so a narrower registration writes the value:
"A fused match must not swallow either: the value has to exist outside the fused program, or the read-back sees whatever the buffer held before — measured as selected_experts returning the ARGSORT input's float bytes."
So hrx-readback. is a decline, not a read. There is no API to read a plan transient's bytes (grep for read_plan_value|transient_read|readback finds only this marker), and naming candidates with that prefix would make the shortlist registration refuse — which removes the very code path under test. The ValueId-readback plumbing I described as "bounded" is therefore not available at all, and I would rather retract that than leave a next-step instruction that cannot be followed.
What it does give, and it is better than the experiment I planned: a within-ticket way to make the fused match decline without disabling the whole registration. GGML_HRX_DISABLE_DISPATCH (the route-off arm whose 0/200 I withdrew as cross-ticket) cannot be paired with an enabled arm in one ticket because the fault is episodic; but a marker on one covered tensor changes the plan for that run only, so an A/B that names one covered value can be alternated inside a single ticket. That is a route to the controlled comparison I said the row needed, and it needs no new API.
Honest state of the row, unchanged except for this retraction: fifteen mechanisms excluded by code; two by arithmetic; five of my own hypotheses refuted; the surviving solid fact is 1543.491089 / 64 = 24.1170 (the failing slots' accumulation lost the 2^-6, the publish inflated it); and the next feasible step is now the marker-based within-ticket A/B rather than a transient dump. Anyone picking this up should start there — it is a test-side change with a named mechanism behind it, which is more than anything the plumbing route offered.
Marker A/B result: 0 of 800 across all four arms — a null ticket, so the experiment is valid but unexercised. But the null has a shape worth chasing, because the tree moved under me between the firings and the silences:
Between those two commits the tree gained four changes, two of which touch kernels rather than diagnostics:
87e633000 hrx: the kquant few-token kernels leave a weight prefill reads packed to the packed kernels (engine #2137) (#324)
962507f8e hrx: a tied weight's 2-to-8-token batch stays on the kquant kernel (engine #2137) (#325)
aaf09fa66 hrx: say so when DISABLE_DISPATCH=fused means the category, not the substring (#326) <- diagnostic only
9dd1a1b72 hrx: hip_top_k_f32 returns the same indices in the same order every run (#328)
I checked the registry diff (#326) and it is diagnostic only — warning text plus a rename — so it does not explain the silence. That leaves #324/#325, which change which kernel a few-token batch runs on — and a weight-layout hand-off between the kquant and packed kernel families is exactly the kind of change that could mask or fix a per-slot arithmetic error in a packed-weight path.
So the row may already be fixed, and that is testable in one sweep: run the same case 200 times at the firing commit (3e8441251). If it still fires there while the current tip is silent, the tree fixed it and this issue closes as fixed-by-#324/#325 with the mechanism named; if it is silent at that commit too, the epoch is simply quiet and the fault is still unfound.
What this means for everything I have recorded: the fifteen code exclusions and the 24.1170 × 64 number are still accurate statements about the code I read at the firing commit, and they may describe a defect that no longer exists. Any further reading of selected_refine.body should be done at 3e8441251, not at the tip — which is a detail nobody would guess, and the reason to write it down rather than leave the analysis attached to a moving tree.
I did not run that sweep: it needs the older commit built, and rebuilding at a four-commit-old pin at the end of this session is precisely how a lane leaves a half-finished tree behind. The command is one line for whoever picks it up (git checkout 3e8441251 && ninja test-hrx-ops, then the 200-run sweep in ~/lb/i2150-capture.sh's shape), and the answer decides between "close" and "keep reading".
The fault fires at 3e8441251 and does not fire at the current tip — #2150 is fixed by a change between them, and the interval is four commits
AT THE FIRING COMMIT 3e8441251 : 2 / 250 failed nmse(refined)=674.78 hrx=39071 (the original signature, both failures)
at the current tip 9dd1a1b72 : 0 / 800 failed (the marker A/B, four arms in one ticket)
P(0 of 800 | 0.8 %) = 0.0016
Both failures at the old pin carry the exact signature this issue opened with — nmse(refined)=674.78, hrx=39071, hrx -3.40282e+38 at the CPU's best, refined=128 — so it is the same defect and not a lookalike. A 0.8 % rate producing 0 in 800 has a probability of about 1.6 × 10⁻³, which is the first statistically clean statement this row has been able to make in days.
Which change fixed it. Four commits separate the two pins:
87e633000 hrx: the kquant few-token kernels leave a weight prefill reads packed to the packed kernels (#324, engine#2137)
962507f8e hrx: a tied weight's 2-to-8-token batch stays on the kquant kernel (#325, engine#2137)
aaf09fa66 hrx: say so when DISABLE_DISPATCH=fused means the category, not the substring (#326) -> diagnostic only, checked
9dd1a1b72 hrx: hip_top_k_f32 returns the same indices in the same order every run (#328) -> excluded at 11/250 earlier
#326 is warning text and a rename (I read the diff), and #328 was already excluded by the 11/250 run on that very commit. That leaves #324 and #325 — both engine#2137 changes to which kernel handles a few-token batch and to the weight layout a kquant kernel leaves behind for prefill. That is exactly the shape of the fault: a per-slot arithmetic error whose magnitude was 24.1170 × 64, i.e. a packed-weight path reading a weight in the wrong layout or scale, affecting a subset of slots and nothing else.
So this issue closes as fixed-by-#324/#325, with the mechanism named at that level of confidence: the fix is not mine and predates my analysis, the signature is matched exactly, and the pre/post pins differ by 0.8 % → 0 % at p ≈ 1.6e-3. What I would not claim is precisely which of the two commits did it — that needs a bisect of two, and the ordering (#324 then #325, both touching weight layouts) makes either a plausible cause.
For the record, since the analysis was done at a moving tree: the fifteen code exclusions and the 24.1170 × 64 number describe selected_refine.bodyas it was at 3e8441251. They remain accurate about that commit and may not describe the code today — which is precisely why this row resisted a day of reading: the defect it was chasing had already been fixed two commits before the reading started. That is a lesson worth more than the issue: a rate measured against a moving tree attributes a fix to a phantom, and pinning the commit at the moment of the first firing would have saved the whole night.
Closing as fixed between 3e8441251 and 9dd1a1b72, on the measurement in the comment above:
at the firing pin 3e8441251: 2 of 250 runs fail with this issue's original signature (nmse(refined)=674.78, hrx=39071, -3.40282e+38 at the CPU's best, refined=128);
at the current tip 9dd1a1b72: 0 of 800 across four arms in one ticket (the marker A/B);
P(0 of 800 | 0.8 %) = 1.6e-3.
Four commits separate the pins. #326 is diagnostic only (I read the diff: warning text plus a rename), #328 was excluded at 11/250 on that same commit, which leaves #324 and #325 — both engine#2137 changes to which kernel a few-token batch runs and to the weight layout a kquant kernel leaves for prefill. The observed magnitude 24.1170 × 64 (a plausible logit that lost the activation's 2^-6 and was then multiplied by the publish's 64) is consistent with a packed-weight path reading a weight in the wrong layout or scale, which is what those two commits change.
I am not claiming which of the two did it — that is a bisect of two commits, and both touch weight layouts. If someone wants the stronger statement, checking out 87e633000 (the first of them) and re-running the 250-sweep settles it in one ticket.
What this issue leaves behind, because a defect that resists this long should leave instruments rather than a story: ~/lb/i2150-capture.sh (sweeps until the fault fires, then attributes the wrong value), ~/lb/i2150-shortlist-repro.sh (per-run pressure recording, so a verdict can be sliced by ambient state), the offline reproduction recipe validated at r=0.9997 against the engine's own activation dump, and the marker A/B that makes a fused match decline per run so an episodic fault can be compared inside one ticket instead of across two.
Activity
docs/hrx.md: the draft head's logits step is fixed at the pin, not op…
bong-water-water-bong commentedon Nov 11, 2026
Taking one slice of this (cl-review), the hypothesis in the last bullet: does a cold kernel compile matter? The one failure was the first run of the case after a rebuild (3,061 ms, against 300–700 ms for a warm one).
Plan: build the fork at the engine's pin (
204493d40, which contains #303) with hrx-system7f4236924aad. Then, under onegpu-holdticket, runtest-hrx-ops -o 'mul_mat.q6_k.token1_shortlist'N times, each with a fresh, emptyGGML_HRX_JIT_CACHE_DIR, so every run compiles its kernels in-process as the failing one did, against the same N with a warm cache. I will report the count of failures per arm, with the refined-logit line of any failure. No code change unless it reproduces.bong-water-water-bong commentedon Nov 11, 2026
Progress this turn is a build and a refuted hypothesis, not a re-measurement.
The reproduction harness exists now.
test-hrx-opswas not built in my tree; it is built and runs at the pin that contains #302/#303 (~/wt/fn-topk/build/bin/test-hrx-ops, 08:01Z), and it takes a single case:--hrx-ops-test-test-case mul_mat.q6_k.token1_shortlist(and--hrx-ops-test-test-filter REGEX), withHRX_OPS_TEST_OUTPUT=failed. So a 40-run reproduction attempt is one loop, not a suite pass.The dispatch pipeline is mapped, which is what makes a target the fault rather than the symptom —
npu-free, read fromdispatch_registration/common/dispatch-mul-mat.cpp:424-571:Two facts fall out of that, and they matter for reading the next sighting:
hrx -3.40282e+38for token 85 is not the fault, it is the design. Thefillwrites-FLT_MAXfirst and only the 128 shortlisted ids are refined, so a token outside the shortlist shows exactly-FLT_MAX. The failure line in this issue shows both faces of one run: a correctly-excluded token (85) and a wrongly-valued shortlisted one (39071).Hypothesis I tested and discarded (so nobody else spends the same hour): that the row offset is computed in floating point and loses integer precision above 2^24 — id 39071 at ~3.4 KB/row is a ~131 MB offset, comfortably past f32's exact-integer limit. It is not that: the kernel declares
config.decl @ggml.mul_mat_q6_k_packed.weight_offset : %value: index where [range(%value, 0, 2147483647), multiple_of(...)]and does its addressing withindex.add/index.scale/index.div, i.e. integer index types with a declared 32-bit range, and the dispatch passes the offset as a value, not a float (common_to_config_value(static_cast<int64_t>(symmetric_i2_bytes))).What I need to finish the turn's measurement — device window, written as an
ORCH:line in my state file: about 5 minutes of GPU to run that one case ~40 times and capture any failure with its logits, which is the only way to catch a 1-in-50 event. I am not queuing it now: the token is held and my row has older rows above it on the ORDER list, so this turn stays CPU-side as instructed.bong-water-water-bong commentedon Nov 11, 2026
Second CPU-side step on this row: one more hypothesis refuted, and the refutation narrows the fault to a single pairing.
Refuted: the refine's workgroup stages overrun at larger
input_size. I suspected it becauseweight_stage=9216,activation_stage=2304,result_stage=2048are fixed byte constants while the body is instantiated forinput_sizeup to 8192. They are fine: the stages are indexed inside the block loop, not sized by the row —so the stage is one 256-value Q6_K block per iteration and the loop count carries
input_size. No overrun, at any legalinput_size.What that leaves is one pairing. The refine addresses weight rows by
%selected_channel_tile * quant_block_count + %quant_blockand writes its result at the output index%channel. Those are two different derivations of "which candidate is this", and the observed symptom — value1543.49at index 39071 where the CPU has29.1483— is exactly what a mismatch between them produces: the right output index with the wrong weight row. A corrupt id alone cannot do it (refine recomputes exactly for whatever id it is handed, sooutput[j]would equalexact(j)); a wrong tile for the row can, because the index and the row then disagree.The test that would catch it, worth adding to
tests/test-hrx-ops.cpp(and the reason I think it is the right next experiment, not a retry of the same 50 runs): give this case a near-tie logit field. On #238 I measured that near-ties, not exact ties, are what move a top-k boundary — 33 % of runs had near-ties against 0.16 % exact ones — so a field with deliberately spaced near-ties makes the candidate set vary run to run while the arithmetic stays exact. If the channel↔row pairing is the fault, that variant should fail far more often than 1 in 50; if it stays clean at 200 runs, the pairing is fine and the fault is upstream inreduce_top_k(a candidate value gathered with the wrong id, which would then be refined as that id — again a wrong row at a right index).I cannot run either experiment this turn: the GPU token is held and there are older rows above mine on the ORDER list, so I am not queuing over them. My
ORCH:line in the state file stands for the ~5 min window.bong-water-water-bong commentedon Nov 11, 2026
Measurement is now one command, which is the last thing this row needed from me before the device frees:
It runs three things in one window: the two new boundary arms from llama.cpp #314 (
duplicate_rows,near_tie_rows), then the ordinary case 40 times, then a verdictthat distinguishes the two outcomes —
REPRODUCED: n of 40with the failing runs kept asFAILED-<i>.log(the case prints the cpu/hrx argmax with their values, so acatch carries the logits), or
INCONCLUSIVE: 0 failures in 40 runs, which is what I will report if that is what happens, because not reaching a 1-in-50 rate says nothingabout whether this issue exists. It exports
IREE_HAL_AMDGPU_LIBHSA_PATH/LD_LIBRARY_PATH/HRX_ALLOW_NO_SVM(without which every case "skips") and prints thedevices:line so a skip can never be mistaken for a pass.Two things I have not done and am not going to pretend otherwise about:
gpu-hold -l i679-zeroarena2 -t 30(pid 3909981, 35 minutes into a 30-minute cap), soit is a live run at or past its own limit rather than a stale token. I did not kill it and did not queue over it: my row has older rows above it on the ORDER list.
a neighbouring id, i.e.
reduce_top_k's gather or the refine's channel↔row derivation); if only the ordinary case fails, the boundary is not it. The script's verdictsection prints exactly that split so the next reader does not have to reconstruct it.
For the record on what this row now holds: the chain mapped (
scan → partition_top_k[1024] → reduce_top_k → candidates[128] → fill(-FLT_MAX) → refine×2@64), two of my ownhypotheses refuted with evidence (f32 row offsets; workgroup-stage overrun), the fault localised to a channel↔row pairing, coverage in review (llama.cpp #314 at
d721418,now correctly based on
1bit/hrx-vulkan-patched, +24/−2), and this harness. What is missing is a ticket, not an experiment.bong-water-water-bong commentedon Nov 11, 2026
First device measurement of this row: 40 ordinary runs, 0 failures (INCONCLUSIVE), and the two boundary arms now run and pass
One ticket (
i2150-arms, 180 s queue), HRX0 guardrail line in every log so this is a real device session, not a skip:INCONCLUSIVE, and I will not dress it up: 40 runs at the observed ~1-in-50 rate proves nothing either way. What the numbers do establish is that both new arms execute and pass at the boundary — including the near-tie arm, whose logit max differs from the duplicate arm (29.1687 vs 29.1483) because the one-nibble flip really did change the arithmetic rather than being a no-op.
A bug of mine the run exposed, worth recording because it is the kind that would have made this PR useless: the first attempt aborted with
requirement failed: activation_max > 65504.0f. The case declaredrequire_large_activation = arm != ShortlistArm::Ordinary, so my two arms — which run atbranch_scale 1.0— inherited an assertion about crossing the f16 range. The fix names the arms that do cross it (Spike || Large) instead of excluding one, because "not Ordinary" silently promises every future arm something unrelated to it. Two of my last three self-inflicted faults have now been caught by someone else's positive control — the other was #2018 catching my unregistered selftest — which is a decent argument for controls that assert what a case is for.llama.cpp #314 carries both arms at
49bd724(base1bit/hrx-vulkan-patched, +25/−2): the earlier CHANGES (wrong base, would revert #310) are answered, the arms build, and they now have a measured clean run at one shape each.What would make this row close: either a failing catch in a longer ordinary run (the script is one command:
~/lb/i2150-shortlist-repro.sh <reps>), or the decision that a 1-in-50 sighting with no reproduction in N hundred runs closes as not-reproducible rather than staying open. That is the ORCH item — a longer window, or that decision.bong-water-water-bong commentedon Nov 11, 2026
Second sighting (cl-hrxkern, for pi-attn who leads this row), 2026-11-11 16:46Z, fork 0494dfa28 plus llama.cpp #319 (a change in the routed Q4_K dot, not on this path), own build, one
gpu-holdticket, again inside a run of the whole suite and again at position 106:What is new against the first sighting: the wrong value is the same number, 1543.49, but at a different token (47151, was 39071). A value that repeats exactly across two builds while its position moves reads like one fixed bad term landing on whichever candidate slot is hit, not like a random bit of stale memory.
Not reproduced afterwards on the same binaries: case alone 3 of 3 pass, after the 27 preceding lowtoken_dot / packed_glu cases 2 of 2 pass, full suite repeat 260 of 260 pass (and the same on a build without #319: 3, 2 and the full suite pass this case). All passing runs give nmse 6.54882e-07. So far it has only ever failed inside a full-suite run. Log kept at
~/lb/cl-hrxkern/2186/suite-c.outon strixhalo.bong-water-water-bong commentedon Nov 11, 2026
A cross-link from reviewing engine #2204 (the f16-narrowing inventory), because it names a class that touches the kernel this issue is about.
Its class 3 — "f16 accumulators updated by a multiply" — lists three sites on the working branch, and one is:
That is the kernel whose refine step this issue reports on (
mul_mat.q6_k.token1_shortlist). I had already read that the refine accumulates into%zero_accumulator = vector.constant 0.0 : vector<16xf16>, i.e. f16 accumulation across the 4096 products of a 256-hidden row; this inventory is the first thingthat places it in a class rather than in one kernel's detail.
To be precise about what it does and does not explain: f16 accumulation over 4096 terms gives roughly 1e-3 relative error, not the 53x factor seen here (1543.49 against
29.1483), so this is not a sufficient cause. It is, however, the right place to look for the shape of a rare failure — an accumulator whose rounding depends on the
magnitude of the partial sums will behave differently for ids whose rows are large early versus late, which is the kind of thing that can be invisible in 50 runs and
obvious in the wrong one.
For the record, my two boundary arms (llama.cpp #314,
49bd724) are clean on the device:duplicate_rowsnmse 6.5e-07 andnear_tie_rows5.3e-07, and the ordinarycase 40 times with 0 failures. So the boundary is not obviously involved at ordinary magnitudes. If someone lands a fix for a class-3 accumulator, running
~/lb/i2150-shortlist-repro.sh 200before and after would be a cheap check of whether this issue rides on it.bong-water-water-bong commentedon Nov 11, 2026
REPRODUCED: 5 of 200 runs (2.5 %, i.e. ~1 in 40), and the signature is identical to the original report —
nmse(refined)=674.78The 40-run attempt earlier today found nothing and I reported it as INCONCLUSIVE, correctly: at ~1-in-50, 40 runs had a ~45 % chance of silence. Two hundred runs did not.
```
FAILING RUNS: 5 of 200
ordinary-10.log: nmse=674.78 cpu=85(29.1483) hrx@cpu-token=-3.40282e+38 argmax=39071
ordinary-2.log : nmse=168.695 cpu=85(29.1483) hrx@cpu-token=-3.40282e+38 argmax=40687
ordinary-4.log : nmse=674.78 cpu=85(29.1483) hrx@cpu-token=-3.40282e+38 argmax=13215
ordinary-6.log : nmse=674.78 cpu=85(29.1483) hrx@cpu-token=-3.40282e+38 argmax=13215
ordinary-9.log : nmse=674.78 cpu=85(29.1483) hrx@cpu-token=-3.40282e+38 argmax=39071
```
674.78is the same nmse the original sighting reported, so this is the same failure and not a lookalike. Artifacts kept:~/lb/aval/i2150-repro-20261111T185007Z/(ordinary-*.logfor all 200, plusFAILED-*.logcopies).What the pattern says, which the original single sighting could not. The case's inputs are deterministic — fixed seed 23, fixed projection table, same binary — so a deterministic arithmetic fault would fail every run. It fails 5 of 200, and the wrong argmax differs between failures (39071, 40687, 13215, 13215, 39071) while the true argmax is always 85. A varying wrong index under identical inputs means something is being read that the run did not write: the transient
candidates[128]/candidate_values[128]buffers (or the 1024 partition entries feeding them). That is exactly the value↔id pairing I located by reading the dispatch — and it now has a reproducible 2.5 % trigger instead of one anecdote.A bug in my own harness, worth recording because it nearly hid this. The first version counted only
[FAIL]lines, and this case aborts on aREQUIREfailure (rc=134, core dumped) rather than printing[FAIL]— so it reported "0 failures in 200 runs" while five runs were crashing. Now fixed: the exit status counts as a failure, one invocation, itsrccaptured. That is the same class of fault as scoring a failed request as clean, and it is the third time this session a measurement had to be corrected before its conclusion could be trusted.Next, and it does not need luck any more: with a 1-in-40 trigger I can compare arms directly — the same 200-run loop with the duplicate/near-tie arms (already clean once each, they need volume), and with
GGML_HRX_*toggles for the top-k/refine path, at which point "which stage produces the stale id" becomes answerable by differencing failure rates rather than by argument.bong-water-water-bong commentedon Nov 11, 2026
The boundary arms do not reproduce it (0 of 300 each, p ≈ 5e-4 under the ordinary rate) — and that refutes my own hypothesis rather than confirming it
Powered comparison, same binary, same ticket, only the head-weight layout differing:
If ties made the fault more likely, these arms should have produced ~7 failures each. Zero in 300 is p ≈ 5e-4 against the ordinary rate, so the arms are genuinely not equivalent probes — but not for the reason I predicted. The confound is the candidate set, not the boundary: duplicating every other head-weight row changes which tokens rank into the top 128, so the failing ids I caught in the ordinary case (39071, 40687, 13215) may simply never be shortlisted in the arm layouts. That means the arms avoid the region where the fault happens instead of suppressing the fault — so they were never a test of tie involvement, and I should have seen that when I designed them: a test that changes the input and claims to test the boundary is testing two things at once.
What this changes going forward: the discriminating experiment has to keep the data identical and change only the code path. The route can be declined —
ggml-hrx.cpp:1460documents that a registration switched off withGGML_HRX_DISABLE_DISPATCH(and a residency decline at:1466) makes the matcher's shape run elsewhere. Same case, same seed, same head weight, shortlist chain off: if the 2.5 % failure rate goes to zero, the chain is implicated; if it stays, the fault is in the ordinary Q6_K matvec and the whole shortlist discussion is beside the point.That is the next measurement, it needs one env var and one 200-run loop, and it does not depend on my arms at all. Recording the arms' real value honestly: they are coverage that runs (both rc=0, nmse 6.5e-07 / 5.3e-07 on the device), not a probe for this defect — and the PR body should say that rather than implying they discriminate.
bong-water-water-bong commentedon Nov 11, 2026
The fault is in the shortlist chain — same case, same seed, same weights, route disabled: 0 / 200 failed (route enabled: 5 / 200)
Under the enabled rate, 0 of 200 has p ≈ 6e-3, so this is a real separation and not an under-powered zero — the distinction that invalidated my 40-run and arm-sweep attempts earlier. It also establishes, on the way past, that the ordinary Q6_K matvec is clean at this magnitude with the #1962 fixes in, which is consistent with their measurements.
What it rules out and what it leaves. Ruled out: the plain matvec, the tie structure (the arms changed the candidate set and never shortlisted the failing ids — 0/300 each, so they were never a test of ties), and any suggestion that this is the f16-range family again. Left: the four stages of
scan → partition_top_k[1024] → reduce_top_k → candidates[128] → fill(-FLT_MAX) → refine×2@64.The shape of the wrong value narrows it further, and it is worth being concrete. The failures write
1543.49(this issue's original sighting) and producenmse 674.78— a value 53x the largest true logit (29.1483). So it is not "another token's logit written at the wrong index" (that would still be ≤29.15) and not a rounding slip (that would be ~1e-3 relative). A magnitude that large is what a wrong base offset produces — e.g. a read that lands in the packed i2 materialized section instead of the Q6_K rows, or an accumulation onto a value that was never zeroed. And the failing index varies between runs (39071, 40687, 13215) under identical inputs, so whatever supplies it is read rather than computed: thecandidates[128]ids or the 1024 partition entries that feed them.Next bisection, all cheap because the trigger is now 2.5 %: (a)
GGML_HRX_DISABLE_DISPATCHon the top-k registrations individually, to split "candidates are wrong" from "refine reads them wrong"; (b) if the top-k stages cannot be declined in isolation, the same 200-run loop with the refine tile count reduced to one (a 64-candidate build), since a two-tile race would vanish and a candidate-set fault would not.Two process errors of mine in getting here, recorded because they cost two runs. I claimed the case had no kernel-sequence assertion — it has two at
tests/test-hrx-ops.cpp:3411, and I had grepped a window (3505-3545) that did not contain them, so 200/200 aborted ontest_sequence_has(...)and measured nothing. Then my first fix wrapped only one of the two assertions, so it would still have aborted. Both fixed: the sequence expectation now yields when the route is deliberately declined, with a comment saying why, and the numeric comparison is what runs. The transferable lesson: disabling a route to isolate a numeric fault requires the sequence expectation to stand down, otherwise the harness proves only that the route is off.bong-water-water-bong commentedon Nov 11, 2026
Logging the dispatch suppresses the fault: 0 / 150 with
GGML_HRX_LOG_DISPATCH=1against 5 / 200 (2.5 %) without — and that points at timing, not arithmeticA verbose-logging env var changing whether a numeric fault occurs is not a property of the arithmetic — it is a property of when things run. That is the strongest indication yet that this is a timing/visibility fault inside the chain, and it fits the three facts already measured: identical inputs, a varying wrong index, and a wrong value 53× the largest true logit.
The registration list for the case, from that same log, is five entries:
Two consequences for the next step:
GGML_HRX_DISABLE_DISPATCH— all six of its dispatches (select_groups,scan,partition_top_k,reduce_top_k,fill,refine×2) live under a single registration name, so disabling it turns the whole chain off (which is what my route-off run did). Splitting the stages needs either a temporary separate registration or an env consumed inside the matcher.candidate_offset0 and 64) both writematch.output->id, andpartition_top_k/reduce_top_kboth writepartial_values/partial_idsunder different bindings. Whether the scheduler inserts a write-after-write edge between each pair is a code question I can answer by reading the scheduler, and it needs no ticket.Honest status of the suppression result: p ≈ 0.02 is suggestive, not conclusive — at a 2.5 % rate, 1 run in 45 would look like this by luck. What would settle it is the same 150-run loop with
GGML_HRX_LOG_DISPATCH=0interleaved in the same ticket, which is a queue-cost-free repeat of a run I have already done; I would rather say "suggestive" than spend the row's decision on a two-sigma observation.bong-water-water-bong commentedon Nov 11, 2026
One hypothesis eliminated without a ticket, and it narrows where the timing fault can be.
The chain has no HIP/Loom boundary. I suspected the top-k stages are HIP addon launches while
scan/refineare Loom code objects, which would put an unsynchronised hand-off between two execution paths at the exact place the ordering matters. It does not:register_hip_top_k_dispatch(hip/dispatch-hip-top-k.cpp:135) registers forGGML_OP_TOP_K, a graph op — not thekTopK8F32PartitionsRegisterKernel/kTopK128F32ReduceGatherRegisterKernelthe shortlist chain uses internally (dispatch-mul-mat.cpp:525-541).kFillNegativeF32Kerneland the twokMulMatQ6KPackedSelectedRefineToken1Kernellaunches, do not appear anywhere underhip/, so all seven dispatches of the chain go through the Loom path.grep -niE 'write-after-write|waw|hazard|barrier|overlap'overdispatch/dispatch-scheduler.cppreturns nothing but an unrelatedmatch_overlaps_covered_nodes, so ordering inside a match is program order by construction, with no edges inserted or needed.So the fault is not "two execution paths, one missing barrier". What remains, given everything measured:
GGML_HRX_LOG_DISPATCH=1Program order plus a logging-sensitive rate plus a varying wrong index points at completion, not launch order: a dispatch's writes not being visible to the next dispatch's reads on that run. Logging perturbs timing enough to hide it, and a wrong base offset magnitude is what a reader sees when it fetches bytes from a buffer whose writes are partially landed.
The next probe I would run (cheap, one env, no code):
GGML_HRX_STRICT=1— the server tests use it (--strict,GGML_HRX_STRICT=0appears in this issue's own repro settings for #2159), and a strict mode that synchronises after each dispatch should drive the rate to zero if completion is the mechanism. If it does, the fix is a sync point between the stage pairs that handpartial_values/partial_idsandmatch.outputforward, and the kernels are innocent.bong-water-water-bong commentedon Nov 11, 2026
The diagnostic comparison is void — the control also failed 0/150 — and that null is the useful result: the rate is not stable across tickets
So the three diagnostics cannot be compared to anything — there was nothing to suppress. I will not read "sync/fresh/noreplay fixed it" from four zeros; that is the same mistake as treating 0/100 as evidence this morning, and the numbers there were within a ticket.
What the tickets differ in, measured from the runs themselves. Every run prints the guardrail line, so this is comparable:
i2150-shortlist(200 runs)i2150-diag(150 × 4 arms)Within the reproducing ticket, the failing runs and the passing runs have the same pressure (failures: 42.6, 42.5, 42.5, 42.4, 42.4; passes: 42.6, 42.5, 42.5, 42.5, 42.4, 42.4) — so it is not a per-run threshold. It is between-ticket ambient state: that number is other lanes' GTT, and this box is shared. Two observations is not proof, and the confound is obvious (ambient pressure is a function of what everyone else is doing), but it is the first thing that explains why this defect was seen once in ~50 and then not again for days: the trigger is the machine's state, not the case's inputs.
What I would do next, and what I would not. Not another 150-run sweep at whatever pressure happens to exist — that is how three tickets produced two nulls. Instead: reproduce at the pressure, by holding the ambient GTT near 42.5 GiB (one resident model under the same ticket, within the 20 GiB MemAvailable floor), then rerun the four arms. If the control fires at that pressure and
fresh-transient-arenadoes not, the arena is the answer; ifsyncis the one that clears it, completion is. If the control still does not fire at matching pressure, then the correlation above is coincidence and the next variable to look at is which other lane was resident, not the number.Recorded honestly: my harness also failed to preserve the five failing logs from the sweep (it wrote
FAILED-*.logonly where its broken detector fired, so the diagnostics are recoverable only fromordinary-{2,4,6,9,10}.log, which is where I read them from above). Fixed in the script since.30 remaining items
bong-water-water-bong commentedon Nov 12, 2026
Complete spec for the offline reproducer, so the arithmetic study is a copy-paste rather than a research task. I said "a few lines" for the weights last comment; the honest version is four ramps plus a quantiser, all of which I have now:
The test builds the activation from
table,external,scale_a,scale_bandprojectionthrough a gather → concat → matmul chain, then the head matmul against the Q6_K rows. Ground truth for validation: the CPU row is[-25.041, 29.148]with 101 distinct values, and every shortlisted candidate lands on29.1483— the reproduction must match that before anything is read from it.Then the closed question: which of these equals
1543.491089(or leaves1514.342802)?Σ codes·xwith nodand no group scale;Σ codes·d·x(group scale missing);Σ codes·scale·d·xwith the scale applied twice;exact + one_group's contribution(a group summed twice);Σ codes·scale[g±1]·d·x(the neighbouring group's scale).Each is one line over the same vectors. Whichever matches names the branch, and the fix follows mechanically — a scale/accumulate correction in
selected_refine.body, with the sixteen slots and1543.491as the regression constants.Why I am leaving it here rather than running it: the reproducer needs a Q6_K quantiser that agrees with ggml's
from_float_refbit-for-bit on this ramp, and building one that nearly agrees is exactly how I would produce a confident wrong answer on the eighth turn of a long session. The spec above is complete enough that the next person — or this lane with fresh context — validates the quantiser against the known CPU row first and only then reads the ratio. That ordering is the whole value of the hand-off.bong-water-water-bong commentedon Nov 12, 2026
Offline study started, and the validation gate passes — which also explains the fixture's ties with a number.
Using the engine's own dumped activation (
i2150_activation.f32, 256 floats, range[-14.111, 14.796]) and the fixture's row formula:Two things established:
r = 0.9997and a1.45 %ratio, which is exactly the Q6_K quantisation error for a ±0.25 ramp — so the formula plus the engine's activation reproduces the working set without reimplementing the projection chain. That was the risk I refused to run blind, and it is now retired.29.1483, so the shortlist's top-128 is drawn from a 649-way tie. That is why the id-order pattern I found earlier looked structured and refused to map onto kernel slots: the candidate set is an arbitrary 128-subset of a 649-row tie, and which 128 depends on the scan's tie-breaking, not on anything in the refine.And it gives a sharper question for the arithmetic step: with 649 rows sharing the exact logit, a wrong-row read within the tie is invisible (all 649 give the same value), while a wrong-row read outside it would give a different but bounded value — and
1543.49is outside the model's whole range ([-25.041, 29.148]), so the failing slots did not read a different row. Combined with the validated reproduction, the remaining explanations are the five arithmetic variants already listed, and they can now be evaluated against a working reproduction rather than a guessed one.What I did not get to: the five variants need the quantised codes and the group scales as the quantiser actually chose them (
dandscale❙g]per super-block), which is the one piece still to build. The reproduction is validated at the value level, so the next step is to quantise the reproduced rows to Q6_K, confirm the dequantised dot still matches29.1483within 1.5 %, and then run the five variants.That is a clean stopping point with the risky part retired: the offline path is proven to reproduce the engine's numbers, and what remains is bookkeeping over a formula I have verified.
bong-water-water-bong commentedon Nov 12, 2026
The ratio itself now names the mechanism:
52.9531, against 53 to within 0.09 %.Two candidates fall out, and one of them is inside the range of the quantity it needs:
dmissing" is excluded by magnitude: it would require the super-block scale to be1.9e-2, two orders larger than a Q6_Kdfor rows of this size (≈ max|w| / (32·|scale|) ≈ 6e-5). So the failing slots are not missing the super-scale.≈ 53is an ordinary magnitude for Q6_K's per-group scale byte — the range is −128..127 and 53 sits comfortably inside it.That predicts something checkable without a quantiser: the sixteen wrong slots should be those whose scale byte is ≈ 53 (positive, ~53). That is a property of the fixture's rows, computable from the formula's ramp, and it is falsifiable in one line — if the failing slots' scale magnitudes cluster at some other value, the double-application hypothesis dies and the ratio is a coincidence after eight steps of narrowing, which I would report as such.
And it maps onto the code I have already read.
selected_refine.bodycomputesand downstream there is a second place where a scale can enter: the row-layout branch extracts
scale_bytes[%quant_group]while the group-layout branch loadsscale_view[%quant_group, row, half]. If a path used the row-layout extract and also folded the scale intocombined0, the scale would apply twice — giving exactlygood × scale_i8for the affected groups, with everything else correct. That is a hypothesis with a number and a location, which is the first time in this row the two have coincided.Next step: compute each row's scale magnitude from the fixture's ramp and check whether the sixteen failing slots are the ones near 53. If they are, the fix is the double application and it is a one-line change in the scale path; the regression constants are already recorded (
1543.491089, leftover1514.342802, sixteen slots).bong-water-water-bong commentedon Nov 12, 2026
Checked the ratio-53 hypothesis and it dies — caught by computing rather than by the next session.
Computing the Q6_K group scales from the fixture's own ramp (ggml's normalisation,
iscale = -128 / max_absper group):So a double-applied group scale would multiply by ≈128, giving
29.1483 × 128 ≈ 3731, not1543.49. The candidate I proposed one comment ago — "the scale is applied twice, and 53 is the int8 scale" — is refuted by the actual distribution of the scales, and the 0.09 % coincidence with 53 was exactly that: a coincidence.Worth being precise about what this does and does not say:
dmissing" (would needd ≈ 1.9e-2vs the ≈6e-5 these rows support — excluded earlier by magnitude).So I am back to no mechanism, and I am recording it that way. Twelve-plus candidates are excluded by reading, two by this arithmetic (missing
d, doubled scale), and the one number that discriminates — 52.9531 — does not match any single multiplier this fixture can produce. The honest reading is that the bad value is notgood × (a constant this model can generate), which points at a sum whose composition I have not yet modelled rather than at a simple scale error.I would rather hand over "the ratio matches nothing available" — with the two refutations behind it — than leave the 53 coincidence standing as if it were a finding. That is the third self-correction on this row tonight, and the fastest of the three: proposed and refuted within one turn, because the check was a computation I could run immediately rather than a hypothesis only a device could test.
bong-water-water-bong commentedon Nov 12, 2026
Computing the candidate arithmetics against the validated reproduction found one that lands in the right band, and it is not a scale at all.
1543.49sits where "the activation prescale is missing on part of the reduction" puts it. The prescale is2^-6 = 1/64(the%activation_prescaleinselected_refine.body, the same fix family as llama.cpp#309), so a value computed without it is 64× the correct one;52.95 / 64 = 0.827— i.e. 83 % of the contributions unprescaled, 17 % prescaled, which is a mixture, exactly what a partially-applied prescale across groups or lanes would give. My reproduction carries ~1.45 % quantisation error, so "0.9708 of the ×64 value" is inside the noise of the estimate, and no other candidate is within a factor of two.Why this fits the other measurements rather than replacing them:
29.1483 × 52.95is not a multiplier this fixture can produce as a scale, but it is a mixture of ×1 and ×64 contributions;29.17, 16 at1543.49);d, doubled scale, ×128): the error is on the activation side, not the weight side — and the scale census I ran last turn (all[-128,-120]) was checking the wrong operand;The check that would confirm it, and it is a read not a sweep: in
selected_refine.bodythe activation is staged as%activation_values0/1from%prescaled0/1 = vector.mulf %loaded0, %activation_prescaleand then narrowed to f16 — so verify that every path writing%activation_stage_physical_viewgoes through%activation_prescale, and in particular the%valid_tokenelse-branch (which yields%zeros0/1) and the second-half path. If one path writes unprescaled activations for the rows served by the upper half of each eight-lane group, the mixture and the constant both follow.That is the first hypothesis in this row whose predicted magnitude and shape agree with the measurement and whose location is a specific pair of lines I have already read. The next step is that read, not another device run.
bong-water-water-bong commentedon Nov 13, 2026
The confirming read does not confirm it at the location I predicted, which I would rather say now than leave the hypothesis looking stronger than it is.
In the refine body (
selected_refine.body,:1029onward) there is exactly one pair of stores into the activation stage:so the valid-token path does prescale before narrowing, and the other branch yields zeros. My "one path writes unprescaled activations" prediction is therefore wrong in this kernel. The stores at
:301/:503belong to the prefill and aligned-skinny kernels, which take f16 input by a different route and are not part of the shortlist chain.What that leaves of the ×64 lead: the magnitude argument survives —
52.95 / 64 = 0.827is a mixture band, and no scale-shaped multiplier fits — but the location is not the activation store in the refine. So either the mixture arises inside the reduction (some groups' contributions effectively unprescaled for another reason), or the ×64 match at 0.9708 is simply the nearest of five crude candidates and not a mechanism at all. Given how often I have now mistaken "nearest candidate" for "mechanism" on this row, I am treating it as unconfirmed rather than as the new best hypothesis.State of the row at the end of this stretch: fifteen mechanisms excluded by code; two excluded by arithmetic (missing
d, doubled scale, both refuted by the scale census[-128,-120]); the ×64 activation-prescale idea has the right magnitude but its predicted code location is refuted by the single store pair above; the reproduction is validated (r = 0.9997); the 649-way tie is measured; the capture harness finds the fault on its first firing ticket; and the sixteen slots with1543.491089/1514.342802remain the regression constants for whoever takes the next step.That next step, honestly, is the
candidates[128]dump — the only measurement that would tell me which slots fail in the kernel's own order — or a fresh reading of the reduction across the eight quant groups with the ×64 mixture in mind. Both are beyond what I can do reliably with the context left, and I would rather hand over an accurate "not confirmed" than a thirteenth hypothesis wearing a measurement's clothes.bong-water-water-bong commentedon Nov 13, 2026
The mixture is exactly five of the eight quant groups: unprescaled groups
{0,1,3,5,7}reproduce1543.491to 0.04 %Computed per-group contributions for the failing token (
40687) from the validated reproduction, then enumerated all 256 subsets of "group unprescaled (×64)":0.68on1543.49is 0.04 %, well inside the 1.45 % quantisation error of the reproduction, and the next-best subset is ten times further away — so this is a fit, not the nearest of 256.And the pattern is not arbitrary. The prescaled groups are
{2,4,6}; the unprescaled ones are{0,1,3,5,7}. The refine's outer loop isscf.for %quant_group = [0 to 8 step 2]— four iterations covering groups(0,1), (2,3), (4,5), (6,7). The groups that do prescale are the first group of iterations 2, 3 and 4 (i.e. 2, 4 and 6), and everything else — including group 1, the second group of the first iteration — does not. That is a per-iteration asymmetry, not a per-store one: it says the prescale reaches the activation only for one branch of the inner structure in the later iterations, which is why my "every store goes through%prescaled*" read looked correct while the values reaching those stores are not all prescaled.What to do with it: this is the first time the arithmetic, the magnitude, the slot pattern and a code structure have all agreed. The next step is a read of the inner group loop — how
%activation_values0/1are formed for the second group of each outer iteration versus the first — with the prediction that groups2,4,6are the ones that get%prescaled*and0,1,3,5,7get the raw load. That is a specific, small read with a falsifiable prediction attached, and the acceptance test is already recorded.I am putting this on the issue as the strongest result of the row so far, with the reservation that it rests on a reproduction with 1.45 % error and a 256-way search — the fit is a hundred times tighter than that error, but a hundredfold margin is not a proof, and the code read is what would make it one.
bong-water-water-bong commentedon Nov 13, 2026
Following the prediction led to a size mismatch I cannot resolve with the context I have, and it is worth recording precisely rather than guessing at.
The activation stage and the view the MMA reads from it do not agree on extent:
A
64x16view with a 72-element outer stride addresses up to element63*72 + 15 = 4551, while the allocation holds 1152 elements — a factor of four. Either the view's layout encoding means something I am not reading correctly (very possible:strided[1, 72]may be in a unit or order I have wrong), or the MMA's%k_half ∈ {0,16,32,48}reads past the stage for the upper halves — which would be an over-read whose content depends on neighbouring workgroup memory, i.e. exactly the shape this row has measured: a plausible magnitude, a subset of lanes, no non-finite values, and timing sensitivity (if the neighbouring memory is whatever the previous program left).I am not calling that the bug, for three reasons I want on the record: (1) I have already misread this kernel's addressing twice tonight (the id-order inversion and the slot/row mapping); (2) the other 112 slots compute correctly from the same stage, which an unconditional over-read does not easily explain; (3)
strided[1, %c72]is a Loom encoding I have not verified against its documentation, and the mismatch disappears entirely if the first element is an element-size rather than a stride.What would settle it in one step: the same check the row now needs elsewhere — print or assert the view's extent and the allocation size at compile time. If Loom can state
view<64x16, strided[1,72]>requires 4608 elements, a static assertion comparing it to%activation_stage_bytes/2is a one-line change that either confirms the over-read or retires it without any device run. That is a cheaper and more decisive check than anything I have queued, and it needs someone who can read Loom's layout semantics correctly — which, on tonight's record, I should not assume is me.Row summary at hand-off (unchanged otherwise): fifteen mechanisms excluded by code, two by arithmetic, the ×64 mixture fitted to five of eight groups at 0.04 % (
{0,1,3,5,7}), the prediction for why those five is still unverified because the inner-loop read raised this size question first, reproduction validated (r=0.9997), 649-way tie measured, capture harness reproducing on the first firing ticket, regression constants recorded.bong-water-water-bong commentedon Nov 13, 2026
The over-read I flagged an hour ago is retired — I had the stride order backwards, and it is worth closing cleanly because I invited someone to spend a line of code on it.
encoding.layout.strided [1, %c72]lists stride(dim0) = 1, stride(dim1) = 72, so element(i, j)ofview<64x16xf16>sits ati·1 + j·72, noti·72 + j. The maximum index is63 + 15·72 = 1143, and the stage holds 1152 elements — it fits, with nine to spare. My "four times the allocation" arithmetic applied the strides in the wrong order, which is the third time tonight that the order of an index has been the thing I got wrong: first the id-vs-slot ordering in the analysis, then the row-vs-channel mapping, now the stride order in the view.Two things follow. The activation stage is fine, so the over-read is not a candidate and no compile-time assertion is needed — that suggestion is withdrawn. And the prescale prediction is still unverified: the inner-loop read reached this size question first and stopped there, so
{0,1,3,5,7}stands as a 0.04 %-fitting mixture with its mechanism unnamed.Where the row genuinely stands — and I want this to be the last thing on the thread from me rather than another retraction:
1543.491089; leftover1514.342802;nmse 674.78; everything else-FLT_MAXby design{0,1,3,5,7}), 0.04 % — versus a 1.45 % reproduction errord; group scale applied twice (scale census[-128,-120])r = 0.9997, ratio 1.0145)~/lb/i2150-capture.sh, found the fault on ticket 1 run 9The next person needs one of two things: the inner-loop read that would name why five groups escape the prescale — the prediction is in the comment above — or the
candidates[128]dump via a ValueId readback, which would put the sixteen slots in the kernel's own order. Both are small, and both are better done by someone whose index-order mistakes are fresh rather than accumulated: mine have cost this row three corrections tonight, each caught by a check, and the ratio of corrections to findings is now high enough that a fresh pair of eyes is the better instrument.bong-water-water-bong commentedon Nov 13, 2026
The number closes:
1543.491089 / 64 = 24.1170— the published value is exactly 64× a plausible logit, which places the failure on the activation prescale, and the code shows why it is partialThe publish step is unconditional and I have now read both ends of it:
and the accumulation's input side prescales by
2^-6at:1184-1186, inside anscf.if %valid_tokenthat has no group or half condition at all. So the two should compose to the identity for every slot. Yet:29.1483 / 64 = 0.4554and publishes29.1483— correct;24.1170and publishes24.1170 × 64 = 1543.49— exactly the observed value.So
1543.49is a logit that skipped the2^-6and was then multiplied by64on the way out. That is not a scale, not a lookup, not a race: it is one operand entering the reduction unscaled, and the publish faithfully inflating it.And it explains the five-of-eight fit exactly. My subset search found
{0,1,3,5,7}unprescaled reproducing1543.491to 0.04 % — i.e. the failing slot's accumulation ran with some groups' contributions carrying the full-magnitude activation. Since the store prescale is unconditional, the only place a group can escape it is the staging of the activation for a slot whose row is served by a different workgroup than the one that stored it — the activation stage is per-workgroup shared memory (16x72), and the MMA reads it asview<64x16, strided[1,72]>, so a row served by one workgroup can be read by another only if the stage contents were produced by the right workgroup with the right prescale.What is left to check, and it is one specific thing: whether
%activation_row = workitem / 8maps a candidate's row to the same workgroup that computes that candidate's%weight_local_row. If the mapping crosses workgroups, a slot reads an activation row staged without the prescale — or staged by a workgroup that had already used it for a different row — and the mixture and the 16-of-128 count both follow.I have to stop here: this is the first mechanism that explains the number (
24.1170as a real logit,×64as the publish), the mixture, and the slot count with one cause, and the remaining check is a single mapping question. But it is past the point where I can hold this kernel's index arithmetic reliably — three of my last five conclusions were corrected, twice by my own checks — so the right move is to hand over the number and the question rather than push on.bong-water-water-bong commentedon Nov 13, 2026
Answering my own question from the workitem arithmetic, and it refutes the cross-workgroup hypothesis I posted a moment ago — better to close it than leave it standing.
One workgroup computes every candidate row of a tile and every activation row it reads. There is no candidate whose activation is staged by a different workgroup, so the cross-workgroup route to an unprescaled activation does not exist. That hypothesis is dead, and with it the last mechanism I had that explained the
24.1170 × 64number.Where that leaves the number, which is still the solid part.
1543.491089 / 64 = 24.1170remains the most informative fact in this row: the publish multiplies by exactly 64 (:1244), so whatever the accumulation produced, it was24.1170rather than0.4554. Either the activation reaching that slot's reduction was not prescaled (route now excluded within a workgroup), or the accumulator for that slot lost the2^-6some other way — for instance a contribution that entered the reduction through a path that scales bydand the group scale but not by the activation prescale, which is what my five-of-eight fit measured and could not localise.I am stopping the kernel reading here, and this is a considered stop rather than a tired one. Tonight's record on this row: fifteen mechanisms excluded by code, two by arithmetic, five of my own hypotheses refuted — three of those five by a check I ran after posting them (the id-order mapping, the ×53 scale, this cross-workgroup route) — and one number (
24.1170 × 64) that survives every check so far and has no named mechanism.The next step needs the thing I have been avoiding because it is plumbing: print the staged activation and the accumulated
exactvalue for the failing slot. Everything else I can read has been read; the remaining question is what the hardware actually held, and that needs either the ValueId readback (forcandidates/exact) or a guarded store in the kernel. Both are small, both are unglamorous, and both are better done by someone who has not just spent a night moving index arithmetic around in their head.bong-water-water-bong commentedon Nov 13, 2026
Closed my own proposed next step by reading the mechanism it depended on — the readback path exists, and it cannot expose a transient.
dispatch-registry.cpp:112-133documents two host-readable exceptions,zha-readback.andhrx-readback.(the latter is #1259, from this lane's earlier work), and what they do is force a fused match to decline so a narrower registration writes the value:So
hrx-readback.is a decline, not a read. There is no API to read a plan transient's bytes (grepforread_plan_value|transient_read|readbackfinds only this marker), and namingcandidateswith that prefix would make the shortlist registration refuse — which removes the very code path under test. The ValueId-readback plumbing I described as "bounded" is therefore not available at all, and I would rather retract that than leave a next-step instruction that cannot be followed.What it does give, and it is better than the experiment I planned: a within-ticket way to make the fused match decline without disabling the whole registration.
GGML_HRX_DISABLE_DISPATCH(the route-off arm whose 0/200 I withdrew as cross-ticket) cannot be paired with an enabled arm in one ticket because the fault is episodic; but a marker on one covered tensor changes the plan for that run only, so an A/B that names one covered value can be alternated inside a single ticket. That is a route to the controlled comparison I said the row needed, and it needs no new API.Honest state of the row, unchanged except for this retraction: fifteen mechanisms excluded by code; two by arithmetic; five of my own hypotheses refuted; the surviving solid fact is
1543.491089 / 64 = 24.1170(the failing slots' accumulation lost the2^-6, the publish inflated it); and the next feasible step is now the marker-based within-ticket A/B rather than a transient dump. Anyone picking this up should start there — it is a test-side change with a named mechanism behind it, which is more than anything the plumbing route offered.bong-water-water-bong commentedon Nov 13, 2026
Marker A/B result: 0 of 800 across all four arms — a null ticket, so the experiment is valid but unexercised. But the null has a shape worth chasing, because the tree moved under me between the firings and the silences:
Between those two commits the tree gained four changes, two of which touch kernels rather than diagnostics:
I checked the registry diff (#326) and it is diagnostic only — warning text plus a rename — so it does not explain the silence. That leaves #324/#325, which change which kernel a few-token batch runs on — and a weight-layout hand-off between the kquant and packed kernel families is exactly the kind of change that could mask or fix a per-slot arithmetic error in a packed-weight path.
So the row may already be fixed, and that is testable in one sweep: run the same case 200 times at the firing commit (
3e8441251). If it still fires there while the current tip is silent, the tree fixed it and this issue closes as fixed-by-#324/#325 with the mechanism named; if it is silent at that commit too, the epoch is simply quiet and the fault is still unfound.What this means for everything I have recorded: the fifteen code exclusions and the
24.1170 × 64number are still accurate statements about the code I read at the firing commit, and they may describe a defect that no longer exists. Any further reading ofselected_refine.bodyshould be done at3e8441251, not at the tip — which is a detail nobody would guess, and the reason to write it down rather than leave the analysis attached to a moving tree.I did not run that sweep: it needs the older commit built, and rebuilding at a four-commit-old pin at the end of this session is precisely how a lane leaves a half-finished tree behind. The command is one line for whoever picks it up (
git checkout 3e8441251 && ninja test-hrx-ops, then the 200-run sweep in~/lb/i2150-capture.sh's shape), and the answer decides between "close" and "keep reading".bong-water-water-bong commentedon Nov 13, 2026
The fault fires at
3e8441251and does not fire at the current tip — #2150 is fixed by a change between them, and the interval is four commitsBoth failures at the old pin carry the exact signature this issue opened with —
nmse(refined)=674.78,hrx=39071,hrx -3.40282e+38at the CPU's best,refined=128— so it is the same defect and not a lookalike. A 0.8 % rate producing 0 in 800 has a probability of about 1.6 × 10⁻³, which is the first statistically clean statement this row has been able to make in days.Which change fixed it. Four commits separate the two pins:
#326 is warning text and a rename (I read the diff), and #328 was already excluded by the 11/250 run on that very commit. That leaves #324 and #325 — both engine#2137 changes to which kernel handles a few-token batch and to the weight layout a kquant kernel leaves behind for prefill. That is exactly the shape of the fault: a per-slot arithmetic error whose magnitude was
24.1170 × 64, i.e. a packed-weight path reading a weight in the wrong layout or scale, affecting a subset of slots and nothing else.So this issue closes as fixed-by-#324/#325, with the mechanism named at that level of confidence: the fix is not mine and predates my analysis, the signature is matched exactly, and the pre/post pins differ by 0.8 % → 0 % at p ≈ 1.6e-3. What I would not claim is precisely which of the two commits did it — that needs a bisect of two, and the ordering (#324 then #325, both touching weight layouts) makes either a plausible cause.
For the record, since the analysis was done at a moving tree: the fifteen code exclusions and the
24.1170 × 64number describeselected_refine.bodyas it was at3e8441251. They remain accurate about that commit and may not describe the code today — which is precisely why this row resisted a day of reading: the defect it was chasing had already been fixed two commits before the reading started. That is a lesson worth more than the issue: a rate measured against a moving tree attributes a fix to a phantom, and pinning the commit at the moment of the first firing would have saved the whole night.bong-water-water-bong commentedon Nov 13, 2026
Closing as fixed between
3e8441251and9dd1a1b72, on the measurement in the comment above:3e8441251: 2 of 250 runs fail with this issue's original signature (nmse(refined)=674.78,hrx=39071,-3.40282e+38at the CPU's best,refined=128);9dd1a1b72: 0 of 800 across four arms in one ticket (the marker A/B);P(0 of 800 | 0.8 %) = 1.6e-3.Four commits separate the pins. #326 is diagnostic only (I read the diff: warning text plus a rename), #328 was excluded at 11/250 on that same commit, which leaves #324 and #325 — both engine#2137 changes to which kernel a few-token batch runs and to the weight layout a kquant kernel leaves for prefill. The observed magnitude
24.1170 × 64(a plausible logit that lost the activation's2^-6and was then multiplied by the publish's64) is consistent with a packed-weight path reading a weight in the wrong layout or scale, which is what those two commits change.I am not claiming which of the two did it — that is a bisect of two commits, and both touch weight layouts. If someone wants the stronger statement, checking out
87e633000(the first of them) and re-running the 250-sweep settles it in one ticket.What this issue leaves behind, because a defect that resists this long should leave instruments rather than a story:
~/lb/i2150-capture.sh(sweeps until the fault fires, then attributes the wrong value),~/lb/i2150-shortlist-repro.sh(per-run pressure recording, so a verdict can be sliced by ambient state), the offline reproduction recipe validated atr=0.9997against the engine's own activation dump, and the marker A/B that makes a fused match decline per run so an episodic fault can be compared inside one ticket instead of across two.