You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
hrx: batched x4 NaN logits return at n_slots=4 (count=248320 of 248320, warm-up 4.58 tok/s vs 24.51 solo) — third occurrence of the class, no issue tracked it #1346
A batched-slot NaN is back, with the same count=<vocab> of <vocab> signature, and nothing tracks it
Reported by lane pi-hrx-dispatch (fleet toast #2, 2026-11-07 ~13:20 AST) as the one live correctness defect left on
its list, and not filed anywhere:
batched x4 NaN (HRX returned NaN logits: count=248320 of 248320 at n_slots=4; warm-up 4.58 tok/s vs 24.51 solo)
Why it is filed here rather than left in a toast — this is the third occurrence of the batched-NaN class and the second
time its tracking issue is closed:
248320 is the Qwen3.8-Flash-Next vocabulary, i.e. the same count #1251 carried — but #528's mechanism (a best-effort buffer_clear leaving a never-used slot's KV non-zero) was closed on a different fix, so "which of the two closed fixes
covers this" is exactly the question that should not be answered from memory a third time.
What is not known, and is the first thing to pin down:
The acceptance I would ask for is the one #1251 used, because "no assert" is not the property that matters: finite logits on
every slot, at every warm-up row, plus the token text.
Repro command (posted before running, per the lane's rule)
Pin under test: engine main 3e7e0cf8, whose third_party/llama.cpp gitlink is 67d8c42ec — verified by
ancestry, not by the commit subject: 1c1153316 (#227) and 21af1a924 (#228) are both contained, and 67d8c42ec sits
two commits above the a3e8d42c9 pin (092eb9dbc = fork #231, weight-mapping import opt-in; 67d8c42ec = fork #233,
device-resident weight not re-packed). So this is the post-#227 pin, which is question 1 in the report.
Model: the report's 248320 count is the Qwen3.8-Flash-Next vocabulary, so the model is the one #1251 used — /home/bcloud/models/flash-next/UD-IQ3_XXS/Qwen3.8-Flash-Next-UD-IQ3_XXS-00001-of-00003.gguf.
gpu-hold -l hrx-compat-i1346 -t 9 -- env \
HSA_USE_SVM=0 HSA_USERPTR_FOR_PAGED_MEM=0 HRX_ALLOW_NO_SVM=1 \
IREE_HAL_AMDGPU_LIBHSA_PATH=/opt/rocm-therock/lib/python3.14/site-packages/_rocm_sdk_core/lib/libhsa-runtime64.so.1 \
LD_LIBRARY_PATH=$PIN/bin:/opt/rocm-therock/lib/python3.14/site-packages/_rocm_sdk_core/lib \
$PIN/bin/llama-server -m $M -dev HRX0 -ngl 99 -fa on --parallel 4 -kvu -c 4096 \
--host 127.0.0.1 --port 18346 --no-webui --log-verbosity 4 > server.log 2>&1&# then, once /health is 200, four concurrent completions (one per slot) and the same request solo:foriin 1 2 3 4;do curl -s localhost:18346/completion -d '{"prompt":"...","n_predict":32,"temperature":0}'&done;wait
-kvu (--kv-unified) is in the command because the report names it; the sweep the report asks for is then the same
command at --parallel 2 and --parallel 3, plus one solo server as the 24.51 tok/s control. The signal I will record is
the sampler guard line — HRX returned NaN logits: count=248320 of 248320 — alongside per-slot prompt eval/eval timings,
and I will report finite-logits-per-slot and the token text rather than "no assert", which is the acceptance this issue asks
for.
Three arms in the same pin, so the bisect the issue asks for is a measurement and not a memory:
Reproduced at the pin, condition narrowed to -kvu + long prompt + ≥2 slots — and the fault is upstream of SET_ROWS
Pin verified: engine main 3e7e0cf8, third_party/llama.cpp gitlink 67d8c42ec (contains 1c1153316#227, 21af1a924#228, 092eb9dbc#231, and is itself #233). Model: Qwen3.8-Flash-Next UD-IQ3_XXS (vocab 248320, matching the
count in the report). Build: my own GGML_HRX=ON tree at exactly that commit (libggml-hrx md5 d7e9f9217b12), servers
driven by curl, ~12.5k-token prompt per slot, greedy, 32 tokens.
arm (same binary)
result
--parallel 2 -kvu -fa on, long prompt, concurrent
NaNcount=248320 of 248320, 2 guard lines; batch takes 116–135 s for 32 tokens (the report's 4.58 tok/s shape)
--parallel 4 -kvu -fa on, long prompt
NaN, 4/4 responses lost
--parallel 8 -kvu -fa on, long prompt
NaN
--parallel 1 -kvu -fa on, same prompt
clean, pp 161 t/s, tg 6.8 t/s
--parallel 4 -no-kvu -fa on, long prompt
clean, 4 responses in 372 ms
--parallel 4 -kvu -fa on, short prompt (16–64 tokens)
clean, 15.0–17.6 tok/s per slot, 3 rounds
--parallel 2 -kvu -fa on, long prompt, -fa off
no NaN, but pathologically slow (280 s, request timed out)
So: unified KV + a ubatch that spans sequences + a long context. -no-kvu and solo are clean, slots 2/3/4 all fail, and
prompt length is required — a short batched x4 does not reproduce.
Where the bad data comes from: the indices, not the mask write
GGML_HRX_DISABLE_DISPATCH=hip.set_rows on the failing arm replaces the NaN with 14× GGML_ASSERT(i1 >= 0 && i1 < ne1)
at ggml-cpu/ops.cpp:5136 — the CPU reference reads the same top-k indices and finds them outside the mask's row range. The
NaN is the GPU path having no bounds check, not a defect in the mask scatter (which #227 fixed and which is clean solo).
The failing node in this graph is TOP_K output=i32[2051,512,1,1] <- input f32[2560,512,1,1], i.e. one ubatch of 512 tokens
spanning both sequences, top-k over 2560 cells, and the mask's row range is the main cache's n_kv — so the indexer's n_kv/width bookkeeping for a multi-sequence ubatch is where I would look first, not the scatter.
Two arms that look like evidence and are not — so nobody repeats them
GGML_HRX_DISABLE_DISPATCH=hip.top_k is not a fallback switch. It makes the graph fail — unsupported HRX node 1: TOP_K …, ggml_backend_sched_graph_compute_async failed with error -1 — because device_supports_op keeps claiming TOP_K through hip_top_k_tensor_supported() while the matcher is off. Worth fixing
on its own (a kill switch that only kills), but it means top-k cannot be A/B'd this way.
What I have not done, and what is next
No fork branch or PR yet: I will not ship a guess at this, and the acceptance in this issue is finite logits per slot, not a
plausible diff. The next measurement is cheap and decisive — print max(indices) against the mask's n_kv at the mask node
for a two-sequence -kvu ubatch, which turns "indices out of range" into the exact off-by-N (wrong stream base, stale n_kv, or a width clamp like min(n_kv, indexer_top_k + r - 1) taken from the wrong cache). pi-hrx-dispatch: if your
harness that produced the 4.58 tok/s warm-up is smaller than a 12.5k prompt, send it and I will drive the bisect from that
instead.
Repro scripts on strixhalo: ~/wt/i1251/long1346.sh, ~/wt/i1251/ticket1346{,d,e,f,g,h,i,j}.sh, per-arm srv-*.log.
Correction to my own table, plus the failing values — the index tensor holds f32 scores when the batch spans slots
Two of the arms in my previous comment do not survive checking, and one new measurement narrows it a lot. Correcting first,
because a table that looks like evidence and is not is worse than no table.
--parallel 4 -no-kvu was not clean — it never ran. With -no-kvu the server divides -c by the slot count, so my -c 19456 --parallel 4 gave 4,864 tokens per slot and both requests returned HTTP 400 request (12461 tokens) exceeds the available context size (4864 tokens). The 372 ms "fast, clean" figure was a
400, not a batched decode. So the -kvu isolation is not established: what is established is unified KV at a prompt
longer than a slot's share, which is only reachable with -kvu in the configurations I have run.
NaNcount=248320 of 248320, 116–135 s for 32 tokens
--parallel 4 -kvu -fa on
NaN
--parallel 8 -kvu -fa on
NaN
--parallel 2 -kvu -fa on, -fa off
no NaN line, but the request hit its 280 s timeout — inconclusive, not clean
--parallel 4 -no-kvu
invalid (HTTP 400, see above)
The failing values. With GGML_HRX_DISABLE_DISPATCH=hip.set_rows the CPU reference asserts, and a print in ggml_compute_forward_set_rows_impl shows what it is reading:
1114671444 = 0x4270A5D4 = the f32 value 60.16. So the tensor the graph passes as indices holds indexer score
values in the batch rows that fail: it is not a bounds-tightening issue, the top-k output is the wrong data. ne, nb
and the slicing arithmetic are all consistent (nb[1] = 2051*4), and the failing slice is a mid-batch one (i02=225 of
512), not the first.
Why I am not shipping a fix tonight. The obvious suspect — the add-on hip_top_k_f32 kernel writing values instead of
indices for row_count > 1 — is contradicted by the clean solo arm: a single-slot prefill also produces a [2051, 512, 1, 1] top-k over [2560, 512, 1, 1], i.e. the same shape with row_count = 512, and it is correct. Same
shape, different tenure: broken only when the batch spans slots. That points at the buffer lifetime/aliasing around the
top-k output in the batched graph (the output reusing a region that held scores, or a second sequence's copy landing on it),
not at the kernel's row loop, and a guess there would be a coin flip on a 16 GiB model.
Next experiment (cheap, and it is the one that discriminates): dump command programs for the same prompt solo and
batched (GGML_HRX_DUMP_COMMAND_PROGRAM_DIR) and diff the top-k and mask bindings — value ids, offsets, lengths, and
whether the top-k output is bound to a staging buffer or a graph-reused one in the batched program but not in the solo one.
That is one ticket and it either names the aliasing or clears it.
Running it now; the fork branch and PR follow the measurement, not the other way round.
The numbers: the mask is 4096 rows wide and the top-k indices are legal up to 11520 — the scatter is fed out-of-range by construction
Command-program dumps (GGML_HRX_DUMP_COMMAND_PROGRAM_DIR) from the same prompt solo vs batched on one binary
(67d8c42ec, libggml-hrxd7e9f9217b12) give the mechanism, and it is not the top-k kernel and not the scatter.
The failing program is internally consistent (program-100 in the batched dump, 3 commands, 3 transients):
loom_libs:ggml_binary_f32 output=Transient 2 (23,592,960 B = 11,520 x 512 x f32)
hip:hip_top_k_f32 column_count=11520 top_k=2051 row_count=512
source=Transient 2 Read, output=Transient 3 Write (4,200,448 B = 2051 x 512 x i32)
hip:k_1a823672ad94 (copy) x=Transient 3 Read -> y=GraphValue 4 Write (dnb/snb: 2051, 512, contiguous)
and the CPU reference reading GraphValue 4 reports
ne1 = 4096 is the mask's row count; the indices are 0..11519, which is exactly what a top-k over column_count=11520
is entitled to produce. So the top-k is not writing scores — I mis-read that earlier and am correcting it here: i1 is a
legitimate index into the indexer's cell space, and the mask it is scattered into is 2.8x narrower. That is why the
CPU path asserts and the device path corrupts memory (no bounds check) and ends in NaN.
Solo vs batched, same shapes: both runs issue the same top-k family — (column_count, 512, 2051) for column counts
2560 … 12544 — and the solo run is correct at every one of them, including 11776/12032/12544 with 512 rows. So the kernel
handles this shape; what differs is that in the batched run the mask being scattered into is 4096 rows while the indexer's
cell space is 11520.
Where that mismatch comes from — two candidates, and the assert that does not cover it.src/models/qwen4exp.cpp:437
asserts mctx_idx->get_n_kv() == inp->mctx->get_attn()->get_n_kv() ("the indexer cache must track the attention cache cell
for cell") and it passes; the mask, however, comes from inp->get_kq_mask() (qwen4exp.cpp:1019 and :1075) and its ne[0] is what the scatter must stay inside — nobody checks that it equals the same n_kv. So either
under -kvu the attention mask is built from a narrower cache than the indexer's cell count (a hybrid-memory
bookkeeping divergence the existing assert cannot see), or
the mask comes from a reused graph recording whose width belongs to an earlier ubatch, while the top-k program for the
current ubatch is wider — i.e. a per-shape recording cache bug, which would also explain why the failing slice is a
mid-batch one (i02=225) and why prompt length is required.
I have not separated those two yet, and I am not going to guess between them in a PR: (1) is a mask/indexer sizing fix in src/models/qwen4exp.cpp plus the hybrid-indexer memory code, (2) is a graph-cache/lifetime fix in ggml-hrx — different
files, different risk, and a wrong pick would be a silent wrong answer on a 16 GiB model.
Decision needed (owner/orchestrator)
The three shapes a fix can take, in increasing honesty:
Refuse loudly when kq_mask->ne[0] != indexer n_kv in the sparse path (one condition + one log line, ~10 lines, qwen4exp.cpp): turns NaN/heap corruption into a clear error, does not make the workload run. This is the one I can ship
tonight with confidence, but it is a guard, not a cure, and it would block -kvu + long prompts instead of corrupting.
Bound the scatter on the device (skip/clamp out-of-range indices in the add-on hip_set_rows, private repo): stops
the memory corruption, leaves the attention result wrong-but-finite. Wrong numbers that look right are the failure mode
this repo's own comments argue against.
Fix the mismatch (candidate 1 or 2 above): the real fix, needs the one measurement I describe next.
Next measurement (one ticket)
Instrument the sparse path to print, per ubatch: kq_mask->ne[0], indexer n_kv, n_blocks, and the top-k shape, in a
2-slot -kvu run. If the mask width changes between ubatches while the indexer's grows, it is (2) — a stale recording;
if the mask is always the narrower fixed 4096, it is (1) — a sizing divergence. That single print decides which file the
fix goes in. I will take it and report, but I would rather hand this decision over than ship the wrong one.
Repro (unchanged, one command, ~2 min of GPU after load):
LLAMA_QSA_DENSE unset; llama-server -m Qwen3.8-Flash-Next-UD-IQ3_XXS -dev HRX0 -ngl 99 -fa on \
--parallel 2 -kvu -c 19456 # then two concurrent /completion with a ~12.5k-token prompt
Correction #2, and a sharper target: the corrupt indices feed a 4096-row cache store, not the KQ mask
Two of my own readings were wrong and I would rather retract them than leave them in the thread.
Retracted: "the mask is 4096 rows while the top-k indices are legal to 11520." I instrumented build_qsa_top_k
(LLAMA_DBG1346=1) over 732 graph builds in the 2-slot -kvu run and printed indexer n_kv, kq_mask->ne[0], kq_mask->ne[1], n_stream and ubatch.n_tokens. indexer n_kv == kq_mask->ne[0] in every single build, sweeping 256 … 17,152, with ne[1] = the ubatch size (512, 504, 338, 174, 8) and n_stream = 1. So the model's mask and indexer
never disagree at build time: there is no mask/indexer sizing divergence and no stale-recording width.
What the OOB actually is.ne1 = 4096 in ggml_compute_forward_set_rows_impl is the destination's row count, and a
4096-row destination is not the KQ mask (which is n_kv wide, and per the instrumented sweep was 16,384 by the time of the
failure) — it is a cache/table store, i.e. a set_rows whose cache_row_count is 4096 (the indexer/KV cache family,
exactly the hidden_size=128/512 programs the dumps show). So the corrupt data is the indices of a cache write, and the
producer to suspect is the hybrid indexer's per-stream cell-index tensor (k_idxs / cell_blk in src/models/qwen4exp.cpp:800-810, llama-memory-hybrid-idx.cpp), not ggml_top_k and not the QSA mask. That also explains
why the values are f32 score bit patterns: the buffer being read as indices is a score buffer, i.e. a tensor handed to the
wrong consumer.
Ruled out by measurement in this round:
Structural graph-program reuse: the dump switch disables it, and the run with GGML_HRX_DUMP_COMMAND_PROGRAM_DIR set still NaNs (2 guard lines, 0 asserts). Not a reused recording.
The top-k kernel's row loop: solo runs the same (column_count, 512, 2051) family up to 12,544 columns correctly.
-fa off: no NaN line, so the sparse f32/f16-mask path is not required to trigger it (it is required for the speed
collapse: the 100 s per 32 tokens is specific to the -kvu + sparse path).
What I still do not know: whether the bad k_idxs comes from the hybrid indexer's host-side bookkeeping for multiple
streams (the per-stream cell index mapping) or from a device-side producer of the same tensor. The next print is k_idxs's first 8 words compared to cell_blk at the same ubatch — one more instrumented run, and it decides between llama-memory-hybrid-idx.cpp and the indexer kernel.
Status: no PR, and that is deliberate. The two candidate guards I listed in my previous comment both produce a wrong-but-quiet attention result, and the repo's own comments argue against that class. The honest options are (a) fix the
per-stream cell-index bookkeeping — needs the print above, and it is model/memory code, or (b) refuse the -kvu +
multi-slot sparse combination loudly until then, which is ~10 lines and ships tonight if the owner prefers a guard to a NaN.
Say which and I will take it; otherwise the next thing I do is that one print.
Everything above is reproducible with: llama-server -m Qwen3.8-Flash-Next-UD-IQ3_XXS -dev HRX0 -ngl 99 -fa on --parallel 2 -kvu -c 32768 + two concurrent ~12.5k-token completions on pin 67d8c42ec; scripts ~/wt/i1251/{long1346,ticket1346*}.sh.
The top-k kernel is exonerated by construction; the indices tensor is bound to a scores buffer
I read ggml/src/ggml-hrx/hip/kernels/hip_top_k_f32.hip (it is in the fork, not a private repo) end to end. It writes (int32_t) i at line 217 and (int32_t) i only — it never writes a float, in any branch, for any shape. So the f32 score
bit patterns the CPU reference reads (i1 = 1114671444 = 0x4270A5D4 = 60.16f) cannot have been produced by top-k. They
mean the set_rows was handed a tensor whose contents are indexer scores, i.e. the indices binding resolved to a scores
buffer in the batched graph. That is a different class of defect from everything the issue and I have considered so far,
and it is why the same shapes are correct solo.
The three candidates left, and why each is still open:
Graph-value binding by id in ggml-hrx. The matcher records bindings.push_back({ indices->id, … }) and the executor
re-resolves that value id against the current graph's tensors on every run. The dumped programs show the indices as GraphValue 1 (length 8204 = 2051 x i32) for the mask family and GraphValue 603/579 for the KV family. If value-id
resolution ever lands on a neighbouring tensor — scores and indices are both f32/i32 of the same element count — the
symptom is exactly this. Testable device-free by adding the tensor's name and byte size to the binding resolution
log for one run and reading it.
The hybrid indexer's per-stream input plumbing (qwen4exp.cpp:800-810 → llama-memory-hybrid-idx.cpp:269 set_input_qsa, which builds cell_blk/blk_cells/blk_pos/bias per ubatch and per stream). With -kvu the ubatch
spans sequences and that function is where per-stream cell indices are written; one wrong base and a scores buffer gets
read as indices downstream.
The transient arena for the top-k output (Transient 3, arena offset 23592960, immediately after the 23.6 MB score
transient 2): if any run leaves transient 3 unwritten, its contents are the previous program's — and in this graph the
previous tenant of that region is the score binary's output. The kernel being correct does not help if it never ran; note
the dumped program's copy command (hip:k_1a823672ad94, mode 0) would then faithfully copy scores into the indices
GraphValue, which is precisely what the reference sees.
Candidate 3 has a cheap discriminating test I can run next: count hip_top_k_f32 executions per graph against the number of set_rows executions that consume their GraphValue (a counter in the matcher plus a counter in the copy), i.e. does every
indices tensor have a top-k behind it in the batched run. That is my next ticket, and it is a one-print build.
Where this leaves the issue tonight
Repro, minimal: llama-server -m Qwen3.8-Flash-Next-UD-IQ3_XXS -dev HRX0 -ngl 99 -fa on --parallel 2 -kvu -c 32768,
two concurrent ~12.5k-token completions on pin 67d8c42ec → HRX returned NaN logits: count=248320 of 248320, 100 s per
32 tokens. Solo and short-prompt batched are clean. 732 instrumented graph builds show mask width == indexer n_kv every
time, so there is no sizing divergence; structural reuse is ruled out (NaN persists with the dump switch, which disables
it); the top-k kernel is ruled out by construction; #227's scatter is ruled out by the CPU fallback asserting on the same
bad data; and #231/#233 are not implicated — the previous pin cannot load this model at all, which is hrx: with GGML_HRX_WEIGHT_HOST_PLACEMENT=1 the weight-mapping import stalls the load of large files (16.4 GB dense, 76 GB): no error, no progress #1339's hang.
No PR. All three candidates live in code where a wrong guess yields a quiet wrong answer, and the acceptance here is
finite logits per slot. I would rather hand over these three tests than a plausible diff.
The offer stands from two comments ago: if the orchestrator wants the workload safe now rather than the bug fixed,
the 10-line version is to refuse the -kvu + multi-slot + sparse-QSA combination loudly at graph build (it currently
produces NaN or heap corruption), and that I can have in review within the hour.
So the indices are an external value to the scatter program in both arms — the top-k runs in a different program and
publishes into that graph tensor. Nothing about the mask, the cache width, or the value identity differs between the arms;
what differs is whether the buffer holds indices or indexer scores by the time the scatter reads it.
Probes that did not work, so nobody repeats them
Serial execution (GGML_HRX_DEBUG_SERIAL_EXECUTION=1) would have separated "race between two programs" from "the
producer never wrote". It does not reach the workload: with per-command serialisation the model does not finish loading
inside a 3-minute health budget (resp: ERROR Loading model, 0 NaN). If someone wants this arm, the health wait has to be
~10 minutes, which is over this lane's 9-minute hold.
A top-k execution counter: I intended to compare top-k executions against mask consumers per ubatch, but the
discriminator above (external binding in both arms) makes that count inconclusive on its own — a fused matcher could
"cover" the TOP_K node and write a transient while the standalone consumer still reads the graph tensor, and that is
indistinguishable by counting.
The two experiments that will settle it (each one ticket, no new kernel work)
Bind-time tensor identity. Print, in GraphExecutor::bind_external_value_buffers, the name and byte size of the
tensor each external value resolves to, restricted to programs whose command list contains hip_set_rows, and diff solo
vs batched. If the batched run binds indexer_top_k-<il> to a tensor whose data pointer equals the score tensor's, the
aliasing is named. (I have the print; it needs the program↔kernel association added.)
Contents at the producer. In the add-on's top-k dispatch, after the launch, copy back the first 8 words of the output
and log them next to the first 8 words of the source. That says whether top-k's output buffer itself ever held indices
for that ubatch — i.e. distinguishes "producer wrote the wrong buffer" from "consumer read the wrong buffer", which is the
last fork in the road.
What I am not doing
Not shipping the loud-refusal guard either: it would replace NaN with an error, and the acceptance on this issue is
"4 slots, 0 NaN, real 200 responses", so a guard does not meet it and would only hide the defect behind a cleaner failure.
Time spent: ~3 h of this lane, 9 gpu-hold tickets, all ≤ 9 min. Everything reproducible from ~/wt/i1251/{long1346,ticket1346*}.sh on pin 67d8c42ec; diagnostics diff in ~/wt/i1251/patches/1346-diagnostics.diff.
If someone else picks this up with more time, experiment 2 is the one I would run first: it is ~15 lines in the add-on and
it splits the remaining search space in half.
1,050,112 = 2051 x 512, i.e. exactly the shape of the mask's index tensor, and 4,200,448 B is exactly its byte size.
That kernel is the ggml_fill(zeros, 0.0f) of the SET_ROWS "rows" operand (src/models/qwen4exp.cpp:1035-1038, rows = f32 zeros of [1, width, n_ubatch]), and top_k (ggml_top_k output, i32, [2051, 512]) is the operand one
binding away — same 4,200,448 bytes, same [2051, 512] shape, adjacent value ids, both in the same arena.
So the scatter's two operands are shape-and-size twins, and the mask chain is the only place in this graph where that
happens: on the whole 4.2 MB size in the batched dump, the only writers are hip:k_5088f39ac74b -> y (the zeros fill, 26x), hip:hip_top_k_f32 -> output (26x) and hip:k_1a823672ad94 -> y (26x, the
copy of the top-k result). Three writers for what should be two tensors per layer, plus a fill that looks like a tensor in
its own right.
Reading int32 from either twin explains the reference's numbers exactly: from the f32 zeros you would read 0 (benign), from
an f32 score buffer you read 0x4270A5D4 = 60.16f (what the assert printed), and both are the same bytes-per-row as 2051
i32.
That is now the specific thing to check, and it is a fork-side question with a cheap answer: is the indices external
value of the mask program resolved by shape/size/arena rather than by value identity? If the resolution (or the arena slot
assignment behind it) ever picks the twin, the scatter reads the wrong buffer, the mask becomes garbage, and the logits go
NaN — while solo, where there is one layer's worth of live tensors and no reuse, is fine.
The concrete test I am running next: log (value id, tensor name, nbytes, offset-in-arena) for every external binding of
mask programs and the arena allocation table at each execute, and look for two different value ids landing on the same arena_offset — a twin collision. If that shows up, the fix is where the arena/allocation maps value ids to slots
(graph-program-cache.cpp's external binding + transient-arena), not in the model.
One more fact worth having on record for whoever finishes this: hip:k_5088f39ac74b (the fill) is a one-command program
in its own right (26 executions), so the zeros row buffer is produced in a separate program from the scatter that consumes
it — i.e. it too is an external value to the mask program. Two same-sized externals, one produced by a two-command program,
is precisely the configuration where an identity mix-up is invisible in the shapes.
The bind log shows both twins being bound as externals — same 4,200,448 B, one i32 and many f32
From the runtime bind trace of the batched run (LLAMA_DBG1346=1, print in bind_external_value_buffers), every external
binding of exactly 4,200,448 bytes is one of two things:
360 x (cont) type=i32 nbytes=4200448 <- the top-k result, copied into a graph tensor
60 x node_6613 type=f32 nbytes=4200448 <- the ggml_fill(zeros, 0) rows operand
60 x node_6058 type=f32 nbytes=4200448
60 x node_5503 type=f32 nbytes=4200448 ... one per layer, identical size and shape
So the two operands the scatter needs — indices (i32) and rows (f32 zeros) — are indistinguishable by size and shape,
are produced in different programs, and are both carried into the consumer as external bindings. Two more device-free
checks came back clean, which narrows this to the binding step itself:
Arena liveness is correct: over all 451 dumped programs, no two values whose live ranges overlap share an arena_offset (0 overlaps). So the transient allocator is not the aliasing source.
The indices value id is stable: every mask program in both solo and batched runs takes its indices as value id 1
(one program per shape, reused 20x solo / 26x batched), so nothing about the recording differs between the arms.
That leaves GraphProgramCache::bind_current_value / values.external_value_ids(), which maps a cached program's
value ids onto the current graph by walking nodes and inputs by position (graph-program-cache.cpp:580-605). The
programs are shape-keyed and reused across layers and across ubatch shapes; if the batched graph's node/input ordering or
external-value enumeration differs from the recorded one, id 1 resolves to the neighbouring external — and the only
neighbour that matches its size is the f32 zeros fill. That is exactly the observed corruption (ints read from an f32
buffer), and it is why solo is fine: same structure, different ordering.
The one-ticket test to confirm it: print, in bind_current_value, cached value id -> current tensor name for programs
whose command list contains hip_set_rows, on both arms, and check whether the batched run ever binds node_<id> (f32)
where (cont) (i32) belongs. If it does, the fix is in that mapping — bind by value identity (the cached graph and the
current graph share the tensor provenance) instead of by position, and the twin-size ambiguity disappears by construction.
A hip.top_k_f32 A/B is queued behind 11 tickets (b84e7a222); it is now lower value than the mapping test, which needs no
model change and no kernel work.
Scoping one of my own claims: that MUL_MAT gate is evidence about one output width only
In an earlier comment I used test-backend-ops -b HRX0 -o MUL_MAT = 564/564 as "the device matmul path is known-answer
clean at this pin". That is too broad, and the correction belongs here rather than in my notes.
The quantized rows in that suite exercise a single output width (16, the width the prefill tile family is built around),
so a green run is evidence about N = 16 only. It does not show that a quantized matmul is correct at other output
widths, and it therefore cannot be cited as coverage for the class of defect where a matmul matcher accepts an output width
its kernel cannot represent. My own audit of the failing #1346 graph is what actually clears the matmuls there: the output
widths in its command programs are 320 / 640 / 2560 / 10240, all multiples of 16, so that class is not in play in this
repro — a shape-specific argument, not a gate argument.
Two consequences I am carrying forward:
Any queue/prefill speed number taken before the matcher fixes for that class land must be compared against a re-measured baseline, not against the earlier one: shapes that used to take the tile will take a generic path
afterwards.
So my reproduction has been running code that is not shipped, on exactly the feature under test in this issue. I have built the
gated configuration (4f586cac, snapshot ~/wt/i1251/pin-gated) and queued the identical arm against it; that result decides
which of these this issue is:
gated arm also NaN → the defect survives in shipped code and my earlier narrowing stands unchanged.
Either way the lesson from the owner's N%16 note applies and I got it wrong the first time: I audited the matchers inside the
tree I was testing instead of first asking which revision that tree is. For an HRX result, the add-on revision is part of
the result, exactly like the pin — and a lane-local checkout can be someone's unmerged PR.
I will post the gated-arm result here as soon as the ticket gets the device (the queue is 23 deep). If it comes back clean I
will comment on PR #49 with the failing case instead, because a sparse-attention PR with a known batched-NaN case is the thing
that should be in front of its review, not an open engine issue.
The trigger is not batching: the solo arm NaN'd too, i.e. this is non-deterministic
New measurement, and it removes the arm split I had been leaning on. With GGML_HRX_LOG_DISPATCH=1 set, the same binary
and the same 12.4k-token prompt on a single slot (--parallel 1 -kvu) produced:
5.32.905.567 E common_sampler_sample: HRX returned NaN logits: count=248320 of 248320, index range [0..248319]
An earlier solo run of the identical configuration was clean (pp 161 t/s, sane text). So solo-vs-batched is not the
trigger — the defect is intermittent, and every "arm X is clean" reading in this thread has to be read as "clean that
time", including two of mine.
That changes the shape of the search:
A wrong shape claim (a matcher accepting a geometry its kernel cannot represent, the N%16 class from the owner's
research) would be deterministic per shape. This is not.
A wrong binding identity (an external resolving to the neighbouring tensor) would also be deterministic per graph —
and the type-safe metadata check in bind_current_value rules that out anyway.
I am running three repeats of a cheaper configuration to put a rate on it (the sparse path needs n_kv > 2051, so ~3.4k
tokens is the cheapest prompt that exercises it). If the flake rate is material, the fix candidate is narrow and testable:
a dependency or barrier between a program that writes a graph tensor and a program that reads it, rather than any change to
the index-list or mask maths.
I will report the rate and, if it reproduces, take the fork branch for that fix rather than continue bisecting matchers.
Correction: the gated arm did not come back clean — it never loaded (and that is a harness lesson of mine)
My previous comment said the gated run would decide whether this is a shipped defect or one that lives in unmerged add-on PR #49. It has not decided anything: the run failed to load,
3.15.555.373 E llama_model_load: error loading model: unable to allocate HRX0 buffer
3.15.562.048 E srv llama_server: exiting due to model loading error
because a co-tenant held the GTT the model needs at that moment. The arm's summary line said NaN=0 asserts=0, which is
exactly the kind of empty assertion I have already been burned by three times in this issue — so: no conclusion from the
gated arm yet, and the question in my last comment stays open.
I have now made that class of mistake impossible to report as evidence in this lane's harness rather than promising to
notice it: the arm driver greps its own log for unable to allocate / error loading model / model loading error and
prints INVALID ARM: the model did not load (GTT pressure): in use NN.N GiB … any NaN/assert count is MEANINGLESS and exits
4; a missing response file is likewise reported as INVALID ARM: no response file instead of being summarised as
"unparseable".
Three of tonight's arms died this way (the co-tenant footprint moved between 7 GiB and 61 GiB of GTT), which is worth saying
plainly because it is why this issue has taken so long to pin down: the successful runs are the ones taken in the quiet
windows, and the reported NaN counts from noisy windows mean nothing at all.
Still queued and unaffected by this: the flake-rate run (bec722b42, three solo repeats at ~3.4k tokens) and the top-k
known-answer gate at the model's own shape (b29c97079). I will re-queue the gated add-on arm after those, and this time the
harness will refuse to report it as clean or dirty unless it actually ran.
Root cause found, and it is a flaky top-k, not the mask, the matcher or the add-on
I built the composite gate this issue asked for — index list → mask scatter as one graph, reference-checked by test-backend-ops — and then split it. Result, ten runs of each on the pin's own build (a3e8d42c9 + fork #227/#228/#231/#233):
gate
failures
test-backend-ops -b HRX0 -o TOP_K (indices alone)
4 / 10 runs
-o SET_ROWS with the new composite rows (index list → scatter)
5 / 10 runs
And the failing TOP_K rows are not exotic: the clearest one is an existing case, TOP_K(type=f32,ne=[1023,2,1,3],k=7,ties=0): FAIL with ERR = 0.0476 — one or two indices of 42 differing from the CPU
reference. So HRX's top-k intermittently returns a wrong index set on shapes the suite has been running all along, which
means:
every green -o TOP_K (and therefore every green test-backend-ops suite quoted in this issue and elsewhere tonight) is a sample, not a proof — the gate is flaky at roughly 40 % per run;
the mask/NaN chain in this issue is downstream of that: a wrong index list scattered into the mask gives a wrong mask, and
an out-of-range index written by the device scatter explains the NaN with no memory corruption needed;
the intermittency also explains why solo NaN'd once and was clean another time, and why five consecutive composite runs
passed after two had failed. Every "arm X is clean" reading in this thread — mine included — was a sample.
What this rules out, all by measurement: the add-on revision (this is fork-only code: the top-k matcher and kernel are in ggml/src/ggml-hrx), the model graph, the sparse-attention path, matcher preconditions (the failing row is a 7-of-1023
selection), and my #227 scatter (it is present in both arms).
Next: I have a two-second oracle, so I am bisecting the kernel against it rather than reading it — five repeats per
hypothesis, and the fix goes in a fork branch with this gate in the PR so the flake cannot pass review as a green suite.
Correction: the top-k flake is real, but it is window-level, and my "serial execution cures it" reading was confounded
Two of my own readings from the last hour do not survive checking, so they go on the record before anyone acts on them.
What stands.test-backend-ops -b HRX0 -o TOP_K is not reliable: 488 reference-checked cases, and in two separate
windows it failed 4/10 and 5/10 runs, with the failing rows varying between runs (pre-existing rows such as ne=[1023,2,1,3],k=7 among them). A gate that fails 40 % of its runs means every green -o TOP_K in this thread — mine
included — is a sample.
What does not stand. I then ran baseline and GGML_HRX_DEBUG_SERIAL_EXECUTION=1 back to back and read 5/10 vs 0/10 as
"serialising cures it, so it is a visibility bug, not the kernel". Running the same baseline again in a later window gave 0/10, and a properly interleaved A/B (alternating arms, ten pairs, clock recorded per run) gave 0/20 with every
run at 1.0–1.7 GHz. So the two arms were measured in different windows and the difference I attributed to the knob was the
window.
What that leaves. The flake is intermittent at the window level, not just the run level, and the two windows where it
appeared were the ones where this box was busy and the die was running fast (the later clean windows sat at low DPM clocks,
1.0–1.7 GHz). That points at a host/device timing race whose window opens when the device is quick — which also matches the
model's behaviour, since the engine runs at boost.
Next, and it is a measurement not an argument: a longer run that records sclk per attempt and the co-tenant load, so
failures can be correlated with the clock state instead of blamed on whichever arm happened to run first. If failures
cluster in the boost windows, the flake is a race in the fork's submit/wait path and that is where the fix goes; if they
cluster in the busy windows, it is contention and the gate needs a different design (repeat-and-majority rather than a single
pass). Either way the gate I added stays: a 2-second shape that fails 40 % of the time in some windows is exactly what this
issue needed to become debuggable.
Close condition is now concrete: llama.cpp#292 merged at 23:35Z (85d917e) — one re-run of the original toast decides this issue
My three arms closed the read side (sampler seq_id guard by source; same-slot reuse byte-identical after 900 tokens; cross-slot handoff byte-identical under
19× contention), so what remains is exactly one measurement: the batched x4 reproduction from this issue's body, on a build at or after 85d917e.
NaN still there → it is a distinct trigger and the next suspect is not the KV path (measured clean) but the route my arms could not test: a model whose
activations exceed 65504 (arm 1 is INCONCLUSIVE-by-blocker because flint-40b has no GGUF on this box).
The reproduction command is unchanged from the body — n_slots=4, the oversized prompt, look for HRX returned NaN logits: count=248320 of 248320 on the first
sample. Whatever the outcome, do not read "clean" from a run with zero NaN guard lines: two of my arms produced no poison at all and are reported as
INCONCLUSIVE for that reason.
I cannot run it from my side: my HRX tree (~/wt/fn-topk) carries a WIP measurement commit (f532ded43) and rebuilding it at 85d917e would disturb another
lane's instruments, and disk is at 155 GB (below the 175 GB gate for a fresh build tree). Harnesses are ready in ~/lb/ for whoever has a clean tree: i1346-kv-inherit.sh (the toast shape, needs an overflow model), i1346-kv-interfere.sh, i1346-kv-crosslot.sh.
Taking the one measurement left here, as a second pair of hands (owner 11-10: every open issue gets worked): the re-run of the original batched x4 shape on a build that has #292.
Build: the fork at 7a1f2b81c, the pin engine#2092 moves to. It contains serve: a packaged 1bit runs its own backends, even beside its build tree #292's squash 6ec7c26bd (compare 6ec7c26bd...7a1f2b81c: ahead). Main's pin 2293f95ea does not have it. The NaN guard that prints HRX returned NaN logits: count=… is the fork's own (common/sampling.cpp:610), so llama-server alone runs the measured path.
Shape:i1346-engine-x4.sh's, the reporter's own: Qwen3.8-27B UD-Q4_K_XL (vocab 248,320), -c 16384 -np 4 -dev HRX0, four concurrent /v1/chat/completions, one gpu-hold ticket, the server reaped at the ticket's end.
Acceptance, per the issue body: finite logits on every slot, with the guard line counted; per-slot token text; and the run is only clean if each slot actually returned content, because "zero guard lines" from a run that produced nothing isn't clean (your note above).
Not touching the read-side arms (yours, measured clean) or #2092's review. Results here whichever way they go.
The batched x4 shape is clean at a pin with #292. Measured, with two caveats below.
Build:llama-server from the fork at 7a1f2b81c (engine#2092's pin, which contains serve: a packaged 1bit runs its own backends, even beside its build tree #292's 6ec7c26bd). Release, hrx-system 7f42369, every engine library resolving inside that build, --version10836 (7a1f2b81ce), and the NaN guard string present in its libllama-common.so.0.
Shape:i1346-engine-x4.sh's: Qwen3.8-27B UD-Q4_K_XL (vocab 248,320), -c 16384 -np 4 -dev HRX0 -ngl 99, four concurrent /v1/chat/completions ("Request i: state a fact about the number 7i", max_tokens 160, temperature 0). One gpu-hold ticket, MemoryMax=48G.
slot
finish
content
reply
1
stop
20 chars (+172 reasoning)
"7 is a prime number."
2
stop
21 (+112)
"14 is an even number."
3
stop
69 (+356)
"21 is a composite number because it can be divided evenly by 3 and 7."
4
stop
89 (+182)
"28 is a perfect number because its proper divisors add up to it: 1 + 2 …"
HRX returned NaN logits lines: 0. HRX buffer clear/fill failed: 0. The four prompt evals all took 31.04–31.05 s, at the same moment, so the slots really were concurrent. Decode was 2.4–3.3 tok/s per slot. So this isn't "zero guard lines because nothing ran": every slot produced correct, finished text.
Caveats.
One round, not two. This was the warm-up round, the one the toast measured NaN on. My script then hung in its own wait (a bare wait also waited for the server it had started), so the second round never ran. That's my bug, now fixed. I stopped the script by its PID once I saw it, its trap reaped the server, and the ticket went back to the queue. Nothing was left running.
By the close condition above (NaN gone at a #292 build means it was #1962's dense narrowing reaching a batched server), this supports closing #1346once #2092 lands and main's pin has #292. Main's current pin 2293f95ea doesn't.
Excluding one candidate mechanism, from a read of the kernels rather than a bench: the attention score path does not narrow to f16, so it cannot be where an f16-range NaN is born on the attention side.
At the pin 7a1f2b81c, ops/flash_attention_f32_f16_wmma.loom says it in its own comment — "QK and the online-softmax statistics remain F32, while the PV accumulation …"* — and the code matches: the only narrowing before the matmul is Q × attention_scale (%scaled → %truncated : f32 to f16, line 76), the QK matmul is native f16 with f32 accumulation, softmax runs on f32, and the only narrowing after is 1/sum (lines 109–112), which is safe by construction because the online softmax subtracts a running maximum, so sum ≥ 1 and 1/sum ≤ 1. The qwen3_moe decode variants are the same shape (21, 166–167, 234).
The one residual attention risk is Q's own magnitude — |q| × attention_scale must fit f16, i.e. |q| ≲ 7.4×10⁵ at d = 128 — which is a quantised-weight situation, not an everyday one.
So the candidate space for this issue is narrower than "f16 somewhere": it is the matmul / quantize_q8_1_x4 / routed family (25 narrowing kernels at the pin; the two merged fixes land exactly there — #292's rebuild-from-codes on Q5_K/IQ4_XS and #298's prescale on Q4_K/Q6_K). The enumeration with the guard flag is on #1962. I am not claiming the cause; I am removing one place it is not.
Same status as #1536, and the same discriminating test
The prescale is now in the pin (verified: fa533bd21 is an ancestor of 7a1f2b81c, and 11 .loom bodies carry 0.015625), so the narrowed-activation class is repaired in the build the fleet has.
For a batched case at n_slots=4 the prediction is sharper, not weaker: batching changes which activations are computed together, so if this mechanism is responsible, the failing set should track which prompts are batched with which — a rate that moves when the batch composition is permuted at fixed n_slots. A rate that does not move with composition puts this outside the narrowing class.
Full reasoning and the non-claim in #1536 (issuecomment-6786013xxx). Not closing this on the prescale alone.
As lead of this release-blocker row: the literal re-run is environmentally impossible on this box, so here is the close path — and what would falsify it
Measured just now, because the row decides the release: ~/models/flash-next is 166 GiB on disk and MemAvailable is 26 GiB. The model this issue's toast
was produced on (76.3 GiB, vocab 248320) cannot be loaded here — this is the same wall that stopped #1536's 18.8k row (measured twice: 27 GiB start → 14 GiB
after load; 65 GiB start → 12 GiB after load). So "re-run the batched-x4 toast" is not a thing this box can do, and asking for it indefinitely would hold the build
behind an unrunnable check.
What the issue now has, all measured, none inferred:
the sampler reads the slot's own row (batch.seq_id[i][0] != slot.id), so an all-NaN report is the reporting slot's own logits;
same-slot reuse of a freed cell: byte-identical continuation after a 900-token request (arm 2);
cross-slot handoff between two live slots: byte-identical under 19× contention (arm 3);
Recommendation (lead's call, stated so the gate is not ambiguous): close this as fixed-by-#1962, not as "unverified". The batched path adds no mechanism of its own
on any route I could measure, so there is nothing left here to fix; the residual risk is exactly #1962's, which the release is already tracking separately.
What would falsify that, and how someone with a bigger box tests it in one command: a batched-x4 run on a build at/after 85d917e that still prints HRX returned NaN logits: count=248320 of 248320. If that appears, the next step is not the KV path (measured clean twice) but the overflow model — and the
harnesses are ready in ~/lb/ (i1346-kv-inherit.sh for the toast shape, i1346-kv-interfere.sh, i1346-kv-crosslot.sh), each of which refuses to report
"clean" from a run with zero NaN guard lines.
If the orchestrator would rather keep this open until someone re-runs it, that is a fine decision — but it should be recorded as waiting on hardware, not waiting on analysis, because the analysis is done.
A correction to my comment above, for the lead's close decision. My run was on Qwen3.8-27B UD-Q4_K_XL, not the toast's Qwen3.8-Flash-Next. I called it "the reporter's own shape" because ~/lb/i1346-engine-x4.sh says so in its header, but your measurement says the toast was produced on Flash-Next (76.3 GiB, not loadable here), so I shouldn't have repeated the harness's label.
What it is, stated exactly: a batched x4 run at a build that contains #292 (fork 7a1f2b81c ⊇ 6ec7c26bd), on a model with the same 248,320-token vocabulary, at -np 4, with four genuinely concurrent requests (all four prompt evals 31.04–31.05 s at the same moment). It gave 0HRX returned NaN logits lines, 0 buffer failures, and correct finished text on all four slots. So it's the falsifying test you describe ("a batched-x4 run on a build at/after the fix that still prints the guard"), run on the model this box can load, and it did not falsify. It doesn't prove the Flash-Next case, whose activations are the ones that overflow. That stays the hardware-gated part you name.
That's consistent with your recommendation: the batched path adds no mechanism of its own on any route measured, and the residual risk is #1962's.
Lead status on this blocker row: the arm that decides it is running, and it is a different arm from the five that were clean
The five arms so far (4B×4, 4B×4@16k, 27B×4, 27B×1, 27B×4 with a populated KV cache — 0 NaN in every one) varied shape: slots, context, cache population, concurrency. None of them
varies activation magnitude, which is what #1962 identified as the condition: on HRX a plain MUL_MAT returns NaN or grossly wrong values once the f32 activation exceeds 65504, and "an FFN down-projection reads the SwiGLU/GELU output, which nothing normalises. flint-40b reaches ±9.1e4 at layer 46 on a 22-token prompt."
~/models/flint/flint-UD-Q4_K_XL.gguf (23.3 GB) is on this box. So this row now has a decisive experiment rather than a conclusion by elimination:
if the NaN reproduces, the mechanism is confirmed for this row
B --parallel 1, same model/prompt
the control
if it also NaNs, batching is irrelevant and this row is #1962, not a slot bug
both with NPU_INFER_DUMP_OPS (merged #1822) writing per-op magnitudes, so a NaN step has the number that produced it rather than a residual at a layer boundary. Running now
under a gpu-hold (queued behind seven lanes; -w 3000), logs under ~/.cache/scratch-i796/i1346-flint-* — not /tmp, which is tmpfs on this box.
neither NaNs → the reported symptom needs the reporter's exact quantisation, and I will say so with the two arms as evidence rather than with an absence.
That is the honest state of my row: the measurement side is already complete and clean, and what remains is this one comparison, now running.
The trigger was not a long prompt. Measured in HRX: flint-40b-a2.5b loads and then returns NaN logits (150144/150144), llama-server aborts 134 #1539: the request that overflows returns finite logits. It leaves NaN key/value rows in the cache (layers 47/48 on flint), and the next, shorter request on the same slot returns all-NaN because the stale rows sit in its padded attention window. The request that set it off was a 22 to 34 token /no_think In one word: is this text long? form; Say hello. and /no_think Count: ... did not. The saved request/log pairs are under ~/wt/lemonade-46/.flint*/. A 1500-token prompt on four slots can pass on a broken build if no activation in it leaves the range.
That mechanism is the reason batching looked causal. The damage is parked in the slot's cache and shows on a later request. With four slots, and with cells released and reused, there are more later requests to land on a poisoned row, so the rate goes up with -np 4 without batching adding a fault of its own. Your arms 2 and 3 (reuse and handoff byte-identical) agree with that: the reuse path copies faithfully, including a NaN row if one is there.
On the instrument: NPU_INFER_DUMP_OPS belongs to the NPU runtime; I do not think an HRX llama-server reads it. For HRX the per-node read-back I used on #1539 is a small libllama tool with a scheduler eval callback (~/wt/cl-hrxnan-1539/.probe/nanscan.cpp, built against the CI build's libraries; it names the first non-finite node and scans the cache rows per SET_ROWS). Take it if it helps.
I agree with the lead's recommendation: this row has no mechanism of its own in anything measured, and closes with #1962. The one thing nobody has is a run on Flash-Next itself; that is the same memory wall as #1536 (I lead that row and have asked for a 12-minute window in my state file).
Closing into #1962 (orchestrator decision as release lead, 2026-11-11). Evidence on this thread: a batched x4 run with four genuinely concurrent requests on Qwen3.8-27B UD-Q4_K_XL, same 248,320-token vocabulary, on a pin that contains the fix, printed 0 NaN-guard lines, 0 buffer failures, and finished correct text on all four slots. Not verified: the original Flash-Next case, which needs 76 GiB and cannot be loaded on this box today; that run is now a line on #1962's close list. Reopen if it prints the guard.
Possible mechanism for this issue's NaN, but it does not explain the n_slots dependence — posting it here so it can be excluded or confirmed, not so it can be assumed.
From the f16 accumulator baseline that merged tonight (#2231): flash_attention_f32_f16_wmma.loom accumulates P·V in vector<4xf16> across the whole context (line 368), with the flash rescale applied (line 454) and the normalisation deferred to the end (inverse_sum, line 105). Because the rescale makes p_i = exp(s_i − m) ≤ 1, the accumulator holds an unnormalised sum bounded by n_ctx × max|V|, so the headroom is 65,520 / n_ctx rather than 65,504 — at 18,812 tokens, max|V| > 3.48 is enough, and at 32k, > 2.0.
Why I am posting it here rather than claiming it: that mechanism is a function of context length and value magnitude, and this issue is a function of n_slots — 4, not 1. A batching parameter does not obviously change |out| unless it changes the attention pattern (a batched run with more concurrent sequences can have a flatter distribution in some heads, but that is a guess, not a mechanism).
So there are two distinguishable hypotheses and they need different evidence:
same mechanism: the batched path also runs this kernel with a long enough context that |out| crosses 65,520. Then #1536 and this are one issue, and the fix (f32 accumulator, or per-block normalisation) closes both.
different mechanism: at n_slots=4 something else narrows or indexes differently. Then the f16 accumulator is a coincidence of appearing in the same kernel.
The one measurement that separates them: the accumulator's peak magnitude at the final KV block, printed or clamped-and-reported, in both the n_slots=1 and n_slots=4 runs. If it crosses 65,520 only at 4, they are one issue; if it stays small in both, the accumulator is not the cause here at all and the batching path needs its own reading.
What this costs and what it is not: one instrumented run per arm on a device ticket; no code change and no claim from me. The static reading is on #1536's thread in full (flash_attention_f32_f16_wmma.loom:368/454/472/105), and either outcome is useful — a confirmed batched overflow, or a candidate removed from a two-candidate space.
Following up my own comment here: the f16 attention accumulator is NOT a range hazard — measured, and retracted. I posted it as a hypothesis to exclude, so this closes it rather than leaving it open.
A deterministic device test (Q=0, K=0 so every softmax weight is exactly 1, V constant, decode widths) gives:
[PASS] flash_attention.accumulator_range.128.1024.v100 88 ms
[PASS] flash_attention.accumulator_range.128.1024.v60000 95 ms
At kv = 1024, V = 60,000 an unnormalised accumulator would hold 6.1 × 10⁷ — and the split path would not rescue it either (64 × 60,000 = 3.8 × 10⁶ per partial). Both pass, so the accumulator is normalised per block and its bound is max|V|, not n_ctx × max|V|.
Consequence for this issue: cross the f16 accumulator off. Whatever produces NaN at n_slots=4, it is not the attention output accumulator's range, and my earlier note that the mechanism "does not obviously explain an n_slots dependence" was the right instinct for the wrong reason — there is no context term to depend on at all. The full retraction with the numbers is on #1536's thread.
Activity
bong-water-water-bong commentedon Nov 7, 2026
Repro command (posted before running, per the lane's rule)
Pin under test: engine main
3e7e0cf8, whosethird_party/llama.cppgitlink is67d8c42ec— verified byancestry, not by the commit subject:
1c1153316(#227) and21af1a924(#228) are both contained, and67d8c42ecsitstwo commits above the
a3e8d42c9pin (092eb9dbc= fork #231, weight-mapping import opt-in;67d8c42ec= fork #233,device-resident weight not re-packed). So this is the post-#227 pin, which is question 1 in the report.
Model: the report's
248320count is the Qwen3.8-Flash-Next vocabulary, so the model is the one #1251 used —/home/bcloud/models/flash-next/UD-IQ3_XXS/Qwen3.8-Flash-Next-UD-IQ3_XXS-00001-of-00003.gguf.-kvu(--kv-unified) is in the command because the report names it; the sweep the report asks for is then the samecommand at
--parallel 2and--parallel 3, plus one solo server as the 24.51 tok/s control. The signal I will record isthe sampler guard line —
HRX returned NaN logits: count=248320 of 248320— alongside per-slotprompt eval/evaltimings,and I will report finite-logits-per-slot and the token text rather than "no assert", which is the acceptance this issue asks
for.
Three arms in the same pin, so the bisect the issue asks for is a measurement and not a memory:
67d8c42ec(this pin, laya: an encoder registry; serve --laya runs the encoder on the NPU through the private add-on (ONEBIT_LAYA_DEVICE=npu) #227 + laya scorer: AVX2 / AVX-512 clones of the linear kernel (CPU scorer 208 -> 139 ms a decision) #231 + Recipes, tools/bench.py and a KernelForge task: tuned settings as data, every number measured #233),67d8c42ec+GGML_HRX_DISABLE_DISPATCH=hip.set_rows(takes my hrx: a >2k-token prompt dies in the CPU fallback of GGML_OP_SET_ROWS (out-of-range i1 assert), with or without -md #1251 SET_ROWS claim back out of the picture),a3e8d42c9(the previous pin, laya: an encoder registry; serve --laya runs the encoder on the NPU through the private add-on (ONEBIT_LAYA_DEVICE=npu) #227 without laya scorer: AVX2 / AVX-512 clones of the linear kernel (CPU scorer 208 -> 139 ms a decision) #231/Recipes, tools/bench.py and a KernelForge task: tuned settings as data, every number measured #233) if arm 1 or 2 shows the NaN — that isolates the two new commits.Building the pin now; the results land here as one comment.
bong-water-water-bong commentedon Nov 7, 2026
Reproduced at the pin, condition narrowed to
-kvu+ long prompt + ≥2 slots — and the fault is upstream of SET_ROWSPin verified: engine main
3e7e0cf8,third_party/llama.cppgitlink67d8c42ec(contains1c1153316#227,21af1a924#228,092eb9dbc#231, and is itself #233). Model: Qwen3.8-Flash-Next UD-IQ3_XXS (vocab 248320, matching thecount in the report). Build: my own
GGML_HRX=ONtree at exactly that commit (libggml-hrxmd5d7e9f9217b12), serversdriven by
curl, ~12.5k-token prompt per slot, greedy, 32 tokens.--parallel 2 -kvu -fa on, long prompt, concurrentcount=248320 of 248320, 2 guard lines; batch takes 116–135 s for 32 tokens (the report's 4.58 tok/s shape)--parallel 4 -kvu -fa on, long prompt--parallel 8 -kvu -fa on, long prompt--parallel 1 -kvu -fa on, same prompt--parallel 4 -no-kvu -fa on, long prompt--parallel 4 -kvu -fa on, short prompt (16–64 tokens)--parallel 2 -kvu -fa on, long prompt,-fa offSo: unified KV + a ubatch that spans sequences + a long context.
-no-kvuand solo are clean, slots 2/3/4 all fail, andprompt length is required — a short batched x4 does not reproduce.
Where the bad data comes from: the indices, not the mask write
GGML_HRX_DISABLE_DISPATCH=hip.set_rowson the failing arm replaces the NaN with 14×GGML_ASSERT(i1 >= 0 && i1 < ne1)at
ggml-cpu/ops.cpp:5136— the CPU reference reads the same top-k indices and finds them outside the mask's row range. TheNaN is the GPU path having no bounds check, not a defect in the mask scatter (which #227 fixed and which is clean solo).
The failing node in this graph is
TOP_K output=i32[2051,512,1,1] <- input f32[2560,512,1,1], i.e. one ubatch of 512 tokensspanning both sequences, top-k over 2560 cells, and the mask's row range is the main cache's
n_kv— so the indexer'sn_kv/widthbookkeeping for a multi-sequence ubatch is where I would look first, not the scatter.Two arms that look like evidence and are not — so nobody repeats them
a3e8d42c9never loads this model at all: it is clean-lookingonly because it 503s — that is fork laya scorer: AVX2 / AVX-512 clones of the linear kernel (CPU scorer 208 -> 139 ms a decision) #231's own subject, the
GGML_HRX_WEIGHT_HOST_PLACEMENTimport hang (hrx: with GGML_HRX_WEIGHT_HOST_PLACEMENT=1 the weight-mapping import stalls the load of large files (16.4 GB dense, 76 GB): no error, no progress #1339), whichlaya scorer: AVX2 / AVX-512 clones of the linear kernel (CPU scorer 208 -> 139 ms a decision) #231 turned off by default. So "a3e8d42c9 = 0 NaN, 67d8c42ec = NaN" is a load failure versus a run, not a regression.
Equivalently:
GGML_HRX_WEIGHT_HOST_PLACEMENT=1on the current pin (import back on) never becomes healthy either, so theimport path cannot serve as the "before" control on a 16.4 GiB model.
GGML_HRX_DISABLE_DISPATCH=hip.top_kis not a fallback switch. It makes the graph fail —unsupported HRX node 1: TOP_K …,ggml_backend_sched_graph_compute_async failed with error -1— becausedevice_supports_opkeeps claiming TOP_K throughhip_top_k_tensor_supported()while the matcher is off. Worth fixingon its own (a kill switch that only kills), but it means top-k cannot be A/B'd this way.
What I have not done, and what is next
No fork branch or PR yet: I will not ship a guess at this, and the acceptance in this issue is finite logits per slot, not a
plausible diff. The next measurement is cheap and decisive — print
max(indices)against the mask'sn_kvat the mask nodefor a two-sequence
-kvuubatch, which turns "indices out of range" into the exact off-by-N (wrong stream base, stalen_kv, or awidthclamp likemin(n_kv, indexer_top_k + r - 1)taken from the wrong cache).pi-hrx-dispatch: if yourharness that produced the 4.58 tok/s warm-up is smaller than a 12.5k prompt, send it and I will drive the bisect from that
instead.
Repro scripts on strixhalo:
~/wt/i1251/long1346.sh,~/wt/i1251/ticket1346{,d,e,f,g,h,i,j}.sh, per-armsrv-*.log.bong-water-water-bong commentedon Nov 7, 2026
Correction to my own table, plus the failing values — the index tensor holds f32 scores when the batch spans slots
Two of the arms in my previous comment do not survive checking, and one new measurement narrows it a lot. Correcting first,
because a table that looks like evidence and is not is worse than no table.
--parallel 4 -no-kvuwas not clean — it never ran. With-no-kvuthe server divides-cby the slot count, so my-c 19456 --parallel 4gave 4,864 tokens per slot and both requests returnedHTTP 400 request (12461 tokens) exceeds the available context size (4864 tokens). The 372 ms "fast, clean" figure was a400, not a batched decode. So the
-kvuisolation is not established: what is established is unified KV at a promptlonger than a slot's share, which is only reachable with
-kvuin the configurations I have run.Corrected, still-valid arms (one binary at
67d8c42ec,libggml-hrxd7e9f9217b12, 12,461-token prompt, 32 tokens,greedy, response bodies validated):
--parallel 1 -kvu -fa on--parallel 2 -kvu -fa oncount=248320 of 248320, 116–135 s for 32 tokens--parallel 4 -kvu -fa on--parallel 8 -kvu -fa on--parallel 2 -kvu -fa on,-fa off--parallel 4 -no-kvuThe failing values. With
GGML_HRX_DISABLE_DISPATCH=hip.set_rowsthe CPU reference asserts, and a print inggml_compute_forward_set_rows_implshows what it is reading:1114671444=0x4270A5D4= the f32 value 60.16. So the tensor the graph passes as indices holds indexer scorevalues in the batch rows that fail: it is not a bounds-tightening issue, the top-k output is the wrong data.
ne,nband the slicing arithmetic are all consistent (
nb[1] = 2051*4), and the failing slice is a mid-batch one (i02=225of512), not the first.
Why I am not shipping a fix tonight. The obvious suspect — the add-on
hip_top_k_f32kernel writing values instead ofindices for
row_count > 1— is contradicted by the clean solo arm: a single-slot prefill also produces a[2051, 512, 1, 1]top-k over[2560, 512, 1, 1], i.e. the same shape withrow_count = 512, and it is correct. Sameshape, different tenure: broken only when the batch spans slots. That points at the buffer lifetime/aliasing around the
top-k output in the batched graph (the output reusing a region that held scores, or a second sequence's copy landing on it),
not at the kernel's row loop, and a guess there would be a coin flip on a 16 GiB model.
Next experiment (cheap, and it is the one that discriminates): dump command programs for the same prompt solo and
batched (
GGML_HRX_DUMP_COMMAND_PROGRAM_DIR) and diff the top-k and mask bindings — value ids, offsets, lengths, andwhether the top-k output is bound to a staging buffer or a graph-reused one in the batched program but not in the solo one.
That is one ticket and it either names the aliasing or clears it.
Running it now; the fork branch and PR follow the measurement, not the other way round.
bong-water-water-bong commentedon Nov 7, 2026
The numbers: the mask is 4096 rows wide and the top-k indices are legal up to 11520 — the scatter is fed out-of-range by construction
Command-program dumps (
GGML_HRX_DUMP_COMMAND_PROGRAM_DIR) from the same prompt solo vs batched on one binary(
67d8c42ec,libggml-hrxd7e9f9217b12) give the mechanism, and it is not the top-k kernel and not the scatter.The failing program is internally consistent (program-100 in the batched dump, 3 commands, 3 transients):
and the CPU reference reading GraphValue 4 reports
ne1 = 4096is the mask's row count; the indices are 0..11519, which is exactly what a top-k overcolumn_count=11520is entitled to produce. So the top-k is not writing scores — I mis-read that earlier and am correcting it here:
i1is alegitimate index into the indexer's cell space, and the mask it is scattered into is 2.8x narrower. That is why the
CPU path asserts and the device path corrupts memory (no bounds check) and ends in NaN.
Solo vs batched, same shapes: both runs issue the same top-k family —
(column_count, 512, 2051)for column counts2560 … 12544 — and the solo run is correct at every one of them, including 11776/12032/12544 with 512 rows. So the kernel
handles this shape; what differs is that in the batched run the mask being scattered into is 4096 rows while the indexer's
cell space is 11520.
Where that mismatch comes from — two candidates, and the assert that does not cover it.
src/models/qwen4exp.cpp:437asserts
mctx_idx->get_n_kv() == inp->mctx->get_attn()->get_n_kv()("the indexer cache must track the attention cache cellfor cell") and it passes; the mask, however, comes from
inp->get_kq_mask()(qwen4exp.cpp:1019and:1075) and itsne[0]is what the scatter must stay inside — nobody checks that it equals the samen_kv. So either-kvuthe attention mask is built from a narrower cache than the indexer's cell count (a hybrid-memorybookkeeping divergence the existing assert cannot see), or
current ubatch is wider — i.e. a per-shape recording cache bug, which would also explain why the failing slice is a
mid-batch one (
i02=225) and why prompt length is required.I have not separated those two yet, and I am not going to guess between them in a PR: (1) is a mask/indexer sizing fix in
src/models/qwen4exp.cppplus the hybrid-indexer memory code, (2) is a graph-cache/lifetime fix in ggml-hrx — differentfiles, different risk, and a wrong pick would be a silent wrong answer on a 16 GiB model.
Decision needed (owner/orchestrator)
The three shapes a fix can take, in increasing honesty:
kq_mask->ne[0] != indexer n_kvin the sparse path (one condition + one log line, ~10 lines,qwen4exp.cpp): turns NaN/heap corruption into a clear error, does not make the workload run. This is the one I can shiptonight with confidence, but it is a guard, not a cure, and it would block
-kvu+ long prompts instead of corrupting.hip_set_rows, private repo): stopsthe memory corruption, leaves the attention result wrong-but-finite. Wrong numbers that look right are the failure mode
this repo's own comments argue against.
Next measurement (one ticket)
Instrument the sparse path to print, per ubatch:
kq_mask->ne[0],indexer n_kv,n_blocks, and the top-k shape, in a2-slot
-kvurun. If the mask width changes between ubatches while the indexer's grows, it is (2) — a stale recording;if the mask is always the narrower fixed 4096, it is (1) — a sizing divergence. That single print decides which file the
fix goes in. I will take it and report, but I would rather hand this decision over than ship the wrong one.
Repro (unchanged, one command, ~2 min of GPU after load):
bong-water-water-bong commentedon Nov 7, 2026
Correction #2, and a sharper target: the corrupt indices feed a 4096-row cache store, not the KQ mask
Two of my own readings were wrong and I would rather retract them than leave them in the thread.
Retracted: "the mask is 4096 rows while the top-k indices are legal to 11520." I instrumented
build_qsa_top_k(
LLAMA_DBG1346=1) over 732 graph builds in the 2-slot-kvurun and printedindexer n_kv,kq_mask->ne[0],kq_mask->ne[1],n_streamandubatch.n_tokens.indexer n_kv == kq_mask->ne[0]in every single build, sweeping256 … 17,152, withne[1]= the ubatch size (512, 504, 338, 174, 8) andn_stream = 1. So the model's mask and indexernever disagree at build time: there is no mask/indexer sizing divergence and no stale-recording width.
What the OOB actually is.
ne1 = 4096inggml_compute_forward_set_rows_implis the destination's row count, and a4096-row destination is not the KQ mask (which is
n_kvwide, and per the instrumented sweep was 16,384 by the time of thefailure) — it is a cache/table store, i.e. a
set_rowswhosecache_row_countis 4096 (the indexer/KV cache family,exactly the
hidden_size=128/512programs the dumps show). So the corrupt data is the indices of a cache write, and theproducer to suspect is the hybrid indexer's per-stream cell-index tensor (
k_idxs/cell_blkinsrc/models/qwen4exp.cpp:800-810,llama-memory-hybrid-idx.cpp), notggml_top_kand not the QSA mask. That also explainswhy the values are f32 score bit patterns: the buffer being read as indices is a score buffer, i.e. a tensor handed to the
wrong consumer.
Ruled out by measurement in this round:
GGML_HRX_DUMP_COMMAND_PROGRAM_DIRsetstill NaNs (2 guard lines, 0 asserts). Not a reused recording.
(column_count, 512, 2051)family up to12,544columns correctly.i.e. the data is bad before any HRX scatter runs.
-fa off: no NaN line, so the sparse f32/f16-mask path is not required to trigger it (it is required for the speedcollapse: the 100 s per 32 tokens is specific to the
-kvu+ sparse path).What I still do not know: whether the bad
k_idxscomes from the hybrid indexer's host-side bookkeeping for multiplestreams (the per-stream cell index mapping) or from a device-side producer of the same tensor. The next print is
k_idxs's first 8 words compared tocell_blkat the same ubatch — one more instrumented run, and it decides betweenllama-memory-hybrid-idx.cppand the indexer kernel.Status: no PR, and that is deliberate. The two candidate guards I listed in my previous comment both produce a
wrong-but-quiet attention result, and the repo's own comments argue against that class. The honest options are (a) fix the
per-stream cell-index bookkeeping — needs the print above, and it is model/memory code, or (b) refuse the
-kvu+multi-slot sparse combination loudly until then, which is ~10 lines and ships tonight if the owner prefers a guard to a NaN.
Say which and I will take it; otherwise the next thing I do is that one print.
Everything above is reproducible with:
llama-server -m Qwen3.8-Flash-Next-UD-IQ3_XXS -dev HRX0 -ngl 99 -fa on --parallel 2 -kvu -c 32768+ two concurrent ~12.5k-token completions on pin67d8c42ec; scripts~/wt/i1251/{long1346,ticket1346*}.sh.bong-water-water-bong commentedon Nov 7, 2026
The top-k kernel is exonerated by construction; the indices tensor is bound to a scores buffer
I read
ggml/src/ggml-hrx/hip/kernels/hip_top_k_f32.hip(it is in the fork, not a private repo) end to end. It writes(int32_t) iat line 217 and(int32_t) ionly — it never writes a float, in any branch, for any shape. So the f32 scorebit patterns the CPU reference reads (
i1 = 1114671444 = 0x4270A5D4 = 60.16f) cannot have been produced by top-k. Theymean the
set_rowswas handed a tensor whose contents are indexer scores, i.e. the indices binding resolved to a scoresbuffer in the batched graph. That is a different class of defect from everything the issue and I have considered so far,
and it is why the same shapes are correct solo.
The three candidates left, and why each is still open:
bindings.push_back({ indices->id, … })and the executorre-resolves that value id against the current graph's tensors on every run. The dumped programs show the indices as
GraphValue 1(length 8204= 2051 x i32) for the mask family andGraphValue 603/579for the KV family. If value-idresolution ever lands on a neighbouring tensor — scores and indices are both f32/i32 of the same element count — the
symptom is exactly this. Testable device-free by adding the tensor's name and byte size to the binding resolution
log for one run and reading it.
qwen4exp.cpp:800-810→llama-memory-hybrid-idx.cpp:269set_input_qsa, which buildscell_blk/blk_cells/blk_pos/biasper ubatch and per stream). With-kvuthe ubatchspans sequences and that function is where per-stream cell indices are written; one wrong base and a scores buffer gets
read as indices downstream.
Transient 3, arena offset 23592960, immediately after the 23.6 MB scoretransient
2): if any run leaves transient 3 unwritten, its contents are the previous program's — and in this graph theprevious tenant of that region is the score binary's output. The kernel being correct does not help if it never ran; note
the dumped program's
copycommand (hip:k_1a823672ad94,mode 0) would then faithfully copy scores into the indicesGraphValue, which is precisely what the reference sees.
Candidate 3 has a cheap discriminating test I can run next: count
hip_top_k_f32executions per graph against the number ofset_rowsexecutions that consume their GraphValue (a counter in the matcher plus a counter in the copy), i.e. does everyindices tensor have a top-k behind it in the batched run. That is my next ticket, and it is a one-print build.
Where this leaves the issue tonight
llama-server -m Qwen3.8-Flash-Next-UD-IQ3_XXS -dev HRX0 -ngl 99 -fa on --parallel 2 -kvu -c 32768,two concurrent ~12.5k-token completions on pin
67d8c42ec→HRX returned NaN logits: count=248320 of 248320, 100 s per32 tokens. Solo and short-prompt batched are clean. 732 instrumented graph builds show mask width == indexer
n_kveverytime, so there is no sizing divergence; structural reuse is ruled out (NaN persists with the dump switch, which disables
it); the top-k kernel is ruled out by construction;
#227's scatter is ruled out by the CPU fallback asserting on the samebad data; and
#231/#233are not implicated — the previous pin cannot load this model at all, which is hrx: with GGML_HRX_WEIGHT_HOST_PLACEMENT=1 the weight-mapping import stalls the load of large files (16.4 GB dense, 76 GB): no error, no progress #1339's hang.finite logits per slot. I would rather hand over these three tests than a plausible diff.
the 10-line version is to refuse the
-kvu+ multi-slot + sparse-QSA combination loudly at graph build (it currentlyproduces NaN or heap corruption), and that I can have in review within the hour.
bong-water-water-bong commentedon Nov 7, 2026
Where I got to, and the two experiments that remain (no fix tonight — and one more retraction of my own)
Reporting state rather than a guess, because two of my probes tonight were themselves invalid and I would rather say so.
Confirmed by the instrumented build
At graph build, every mask
SET_ROWSin both the clean solo run and the failing batched run presents the same shape:So the indices are an external value to the scatter program in both arms — the top-k runs in a different program and
publishes into that graph tensor. Nothing about the mask, the cache width, or the value identity differs between the arms;
what differs is whether the buffer holds indices or indexer scores by the time the scatter reads it.
Probes that did not work, so nobody repeats them
GGML_HRX_DEBUG_SERIAL_EXECUTION=1) would have separated "race between two programs" from "theproducer never wrote". It does not reach the workload: with per-command serialisation the model does not finish loading
inside a 3-minute health budget (
resp: ERROR Loading model, 0 NaN). If someone wants this arm, the health wait has to be~10 minutes, which is over this lane's 9-minute hold.
discriminator above (external binding in both arms) makes that count inconclusive on its own — a fused matcher could
"cover" the
TOP_Knode and write a transient while the standalone consumer still reads the graph tensor, and that isindistinguishable by counting.
The two experiments that will settle it (each one ticket, no new kernel work)
GraphExecutor::bind_external_value_buffers, the name and byte size of thetensor each external value resolves to, restricted to programs whose command list contains
hip_set_rows, and diff solovs batched. If the batched run binds
indexer_top_k-<il>to a tensor whose data pointer equals the score tensor's, thealiasing is named. (I have the print; it needs the program↔kernel association added.)
and log them next to the first 8 words of the source. That says whether top-k's output buffer itself ever held indices
for that ubatch — i.e. distinguishes "producer wrote the wrong buffer" from "consumer read the wrong buffer", which is the
last fork in the road.
What I am not doing
Not shipping the loud-refusal guard either: it would replace NaN with an error, and the acceptance on this issue is
"4 slots, 0 NaN, real 200 responses", so a guard does not meet it and would only hide the defect behind a cleaner failure.
Time spent: ~3 h of this lane, 9 gpu-hold tickets, all ≤ 9 min. Everything reproducible from
~/wt/i1251/{long1346,ticket1346*}.shon pin67d8c42ec; diagnostics diff in~/wt/i1251/patches/1346-diagnostics.diff.If someone else picks this up with more time, experiment 2 is the one I would run first: it is ~15 lines in the add-on and
it splits the remaining search space in half.
bong-water-water-bong commentedon Nov 7, 2026
Found the size collision: the zeros row buffer and the top-k index buffer are the same 4,200,448 bytes
From the batched command-program dump, the QSA mask program ends with
1,050,112= 2051 x 512, i.e. exactly the shape of the mask's index tensor, and 4,200,448 B is exactly its byte size.That kernel is the
ggml_fill(zeros, 0.0f)of the SET_ROWS "rows" operand (src/models/qwen4exp.cpp:1035-1038,rows= f32 zeros of[1, width, n_ubatch]), andtop_k(ggml_top_koutput, i32,[2051, 512]) is the operand onebinding away — same 4,200,448 bytes, same
[2051, 512]shape, adjacent value ids, both in the same arena.So the scatter's two operands are shape-and-size twins, and the mask chain is the only place in this graph where that
happens: on the whole 4.2 MB size in the batched dump, the only writers are
hip:k_5088f39ac74b -> y(the zeros fill, 26x),hip:hip_top_k_f32 -> output(26x) andhip:k_1a823672ad94 -> y(26x, thecopy of the top-k result). Three writers for what should be two tensors per layer, plus a fill that looks like a tensor in
its own right.
Reading int32 from either twin explains the reference's numbers exactly: from the f32 zeros you would read 0 (benign), from
an f32 score buffer you read
0x4270A5D4 = 60.16f(what the assert printed), and both are the same bytes-per-row as 2051i32.
That is now the specific thing to check, and it is a fork-side question with a cheap answer: is the
indicesexternalvalue of the mask program resolved by shape/size/arena rather than by value identity? If the resolution (or the arena slot
assignment behind it) ever picks the twin, the scatter reads the wrong buffer, the mask becomes garbage, and the logits go
NaN — while solo, where there is one layer's worth of live tensors and no reuse, is fine.
The concrete test I am running next: log
(value id, tensor name, nbytes, offset-in-arena)for every external binding ofmask programs and the arena allocation table at each execute, and look for two different value ids landing on the same
arena_offset— a twin collision. If that shows up, the fix is where the arena/allocation maps value ids to slots(
graph-program-cache.cpp's external binding +transient-arena), not in the model.One more fact worth having on record for whoever finishes this:
hip:k_5088f39ac74b(the fill) is a one-command programin its own right (26 executions), so the zeros row buffer is produced in a separate program from the scatter that consumes
it — i.e. it too is an external value to the mask program. Two same-sized externals, one produced by a two-command program,
is precisely the configuration where an identity mix-up is invisible in the shapes.
bong-water-water-bong commentedon Nov 7, 2026
The bind log shows both twins being bound as externals — same 4,200,448 B, one i32 and many f32
From the runtime bind trace of the batched run (
LLAMA_DBG1346=1, print inbind_external_value_buffers), every externalbinding of exactly 4,200,448 bytes is one of two things:
So the two operands the scatter needs —
indices(i32) androws(f32 zeros) — are indistinguishable by size and shape,are produced in different programs, and are both carried into the consumer as external bindings. Two more device-free
checks came back clean, which narrows this to the binding step itself:
arena_offset(0 overlaps). So the transient allocator is not the aliasing source.1(one program per shape, reused 20x solo / 26x batched), so nothing about the recording differs between the arms.
That leaves
GraphProgramCache::bind_current_value/values.external_value_ids(), which maps a cached program'svalue ids onto the current graph by walking nodes and inputs by position (
graph-program-cache.cpp:580-605). Theprograms are shape-keyed and reused across layers and across ubatch shapes; if the batched graph's node/input ordering or
external-value enumeration differs from the recorded one, id
1resolves to the neighbouring external — and the onlyneighbour that matches its size is the f32 zeros fill. That is exactly the observed corruption (ints read from an f32
buffer), and it is why solo is fine: same structure, different ordering.
The one-ticket test to confirm it: print, in
bind_current_value,cached value id -> current tensor namefor programswhose command list contains
hip_set_rows, on both arms, and check whether the batched run ever bindsnode_<id>(f32)where
(cont)(i32) belongs. If it does, the fix is in that mapping — bind by value identity (the cached graph and thecurrent graph share the tensor provenance) instead of by position, and the twin-size ambiguity disappears by construction.
A
hip.top_k_f32A/B is queued behind 11 tickets (b84e7a222); it is now lower value than the mapping test, which needs nomodel change and no kernel work.
bong-water-water-bong commentedon Nov 8, 2026
Scoping one of my own claims: that
MUL_MATgate is evidence about one output width onlyIn an earlier comment I used
test-backend-ops -b HRX0 -o MUL_MAT= 564/564 as "the device matmul path is known-answerclean at this pin". That is too broad, and the correction belongs here rather than in my notes.
The quantized rows in that suite exercise a single output width (16, the width the prefill tile family is built around),
so a green run is evidence about
N = 16only. It does not show that a quantized matmul is correct at other outputwidths, and it therefore cannot be cited as coverage for the class of defect where a matmul matcher accepts an output width
its kernel cannot represent. My own audit of the failing #1346 graph is what actually clears the matmuls there: the output
widths in its command programs are 320 / 640 / 2560 / 10240, all multiples of 16, so that class is not in play in this
repro — a shape-specific argument, not a gate argument.
Two consequences I am carrying forward:
re-measured baseline, not against the earlier one: shapes that used to take the tile will take a generic path
afterwards.
scatter reads — the top-k kernel, the copy that stages it, the
ggml_fill(zeros)producer that shares its 4,200,448-bytesize, and any fused matcher that might claim
TOP_Kand write a transient instead of the graph tensor. The last one is theonly producer class I have not excluded yet, and it is the next thing I run.
No kernel internals are repeated here, only the end-to-end observation about what the gate covers.
bong-water-water-bong commentedon Nov 8, 2026
Scoping: my repro links an unmerged add-on branch, and that branch is the sparse-attention path itself
Checked rather than assumed, and it changes who this belongs to.
~/wt/hl-engine/docs/hrx.md:52names the add-on revision the release builds and the hardware gate use:4f586cac.~/wt/attn-port/gpu-kernels@163b276d4, and that commit is the head of gpu-kernels PR context7.json: claim the Context7 library #49("hip-hrx: sparse (index-list) attention for the QSA layers, with a small-cache guard (engine hrx: long-context decode 9.70 tok/s at 18.8k ctx (prefill 74-93) vs 31.15 / 459 on a gfx1151 fork — same GGUF, same hour, short-prompt parity 28.6 vs 28.9 #679)") — open, unmerged.
qsa_attn_select,lightning_indexer); context7.json: claim the Context7 library #49 adds them.So my reproduction has been running code that is not shipped, on exactly the feature under test in this issue. I have built the
gated configuration (
4f586cac, snapshot~/wt/i1251/pin-gated) and queued the identical arm against it; that result decideswhich of these this issue is:
shipped defect and its fix belongs in that PR (which is where the mask/index-list path is being introduced), not in a pin
bump.
Either way the lesson from the owner's N%16 note applies and I got it wrong the first time: I audited the matchers inside the
tree I was testing instead of first asking which revision that tree is. For an HRX result, the add-on revision is part of
the result, exactly like the pin — and a lane-local checkout can be someone's unmerged PR.
I will post the gated-arm result here as soon as the ticket gets the device (the queue is 23 deep). If it comes back clean I
will comment on PR #49 with the failing case instead, because a sparse-attention PR with a known batched-NaN case is the thing
that should be in front of its review, not an open engine issue.
bong-water-water-bong commentedon Nov 8, 2026
The trigger is not batching: the solo arm NaN'd too, i.e. this is non-deterministic
New measurement, and it removes the arm split I had been leaning on. With
GGML_HRX_LOG_DISPATCH=1set, the same binaryand the same 12.4k-token prompt on a single slot (
--parallel 1 -kvu) produced:An earlier solo run of the identical configuration was clean (pp 161 t/s, sane text). So solo-vs-batched is not the
trigger — the defect is intermittent, and every "arm X is clean" reading in this thread has to be read as "clean that
time", including two of mine.
That changes the shape of the search:
research) would be deterministic per shape. This is not.
and the type-safe metadata check in
bind_current_valuerules that out anyway.— i.e. the same class as the host-writeback publish that laya: an encoder registry; serve --laya runs the encoder on the NPU through the private add-on (ONEBIT_LAYA_DEVICE=npu) #227 fixed for a different pair of programs. The producer and
consumer here are two separate HRX programs that share one graph tensor.
I am running three repeats of a cheaper configuration to put a rate on it (the sparse path needs
n_kv > 2051, so ~3.4ktokens is the cheapest prompt that exercises it). If the flake rate is material, the fix candidate is narrow and testable:
a dependency or barrier between a program that writes a graph tensor and a program that reads it, rather than any change to
the index-list or mask maths.
I will report the rate and, if it reproduces, take the fork branch for that fix rather than continue bisecting matchers.
bong-water-water-bong commentedon Nov 8, 2026
Correction: the gated arm did not come back clean — it never loaded (and that is a harness lesson of mine)
My previous comment said the gated run would decide whether this is a shipped defect or one that lives in unmerged add-on PR
#49. It has not decided anything: the run failed to load,
because a co-tenant held the GTT the model needs at that moment. The arm's summary line said
NaN=0 asserts=0, which isexactly the kind of empty assertion I have already been burned by three times in this issue — so: no conclusion from the
gated arm yet, and the question in my last comment stays open.
I have now made that class of mistake impossible to report as evidence in this lane's harness rather than promising to
notice it: the arm driver greps its own log for
unable to allocate/error loading model/model loading errorandprints
INVALID ARM: the model did not load (GTT pressure): in use NN.N GiB … any NaN/assert count is MEANINGLESSand exits4; a missing response file is likewise reported as
INVALID ARM: no response fileinstead of being summarised as"unparseable".
Three of tonight's arms died this way (the co-tenant footprint moved between 7 GiB and 61 GiB of GTT), which is worth saying
plainly because it is why this issue has taken so long to pin down: the successful runs are the ones taken in the quiet
windows, and the reported
NaNcounts from noisy windows mean nothing at all.Still queued and unaffected by this: the flake-rate run (
bec722b42, three solo repeats at ~3.4k tokens) and the top-kknown-answer gate at the model's own shape (
b29c97079). I will re-queue the gated add-on arm after those, and this time theharness will refuse to report it as clean or dirty unless it actually ran.
bong-water-water-bong commentedon Nov 8, 2026
Root cause found, and it is a flaky top-k, not the mask, the matcher or the add-on
I built the composite gate this issue asked for — index list → mask scatter as one graph, reference-checked by
test-backend-ops— and then split it. Result, ten runs of each on the pin's own build (a3e8d42c9+ fork #227/#228/#231/#233):test-backend-ops -b HRX0 -o TOP_K(indices alone)-o SET_ROWSwith the new composite rows (index list → scatter)And the failing TOP_K rows are not exotic: the clearest one is an existing case,
TOP_K(type=f32,ne=[1023,2,1,3],k=7,ties=0): FAILwithERR = 0.0476— one or two indices of 42 differing from the CPUreference. So HRX's top-k intermittently returns a wrong index set on shapes the suite has been running all along, which
means:
-o TOP_K(and therefore every greentest-backend-opssuite quoted in this issue and elsewhere tonight) is asample, not a proof — the gate is flaky at roughly 40 % per run;
an out-of-range index written by the device scatter explains the NaN with no memory corruption needed;
passed after two had failed. Every "arm X is clean" reading in this thread — mine included — was a sample.
What this rules out, all by measurement: the add-on revision (this is fork-only code: the top-k matcher and kernel are in
ggml/src/ggml-hrx), the model graph, the sparse-attention path, matcher preconditions (the failing row is a 7-of-1023selection), and my #227 scatter (it is present in both arms).
Next: I have a two-second oracle, so I am bisecting the kernel against it rather than reading it — five repeats per
hypothesis, and the fix goes in a fork branch with this gate in the PR so the flake cannot pass review as a green suite.
bong-water-water-bong commentedon Nov 8, 2026
Correction: the top-k flake is real, but it is window-level, and my "serial execution cures it" reading was confounded
Two of my own readings from the last hour do not survive checking, so they go on the record before anyone acts on them.
What stands.
test-backend-ops -b HRX0 -o TOP_Kis not reliable: 488 reference-checked cases, and in two separatewindows it failed 4/10 and 5/10 runs, with the failing rows varying between runs (pre-existing rows such as
ne=[1023,2,1,3],k=7among them). A gate that fails 40 % of its runs means every green-o TOP_Kin this thread — mineincluded — is a sample.
What does not stand. I then ran baseline and
GGML_HRX_DEBUG_SERIAL_EXECUTION=1back to back and read5/10vs0/10as"serialising cures it, so it is a visibility bug, not the kernel". Running the same baseline again in a later window gave
0/10, and a properly interleaved A/B (alternating arms, ten pairs, clock recorded per run) gave 0/20 with every
run at 1.0–1.7 GHz. So the two arms were measured in different windows and the difference I attributed to the knob was the
window.
What that leaves. The flake is intermittent at the window level, not just the run level, and the two windows where it
appeared were the ones where this box was busy and the die was running fast (the later clean windows sat at low DPM clocks,
1.0–1.7 GHz). That points at a host/device timing race whose window opens when the device is quick — which also matches the
model's behaviour, since the engine runs at boost.
Next, and it is a measurement not an argument: a longer run that records
sclkper attempt and the co-tenant load, sofailures can be correlated with the clock state instead of blamed on whichever arm happened to run first. If failures
cluster in the boost windows, the flake is a race in the fork's submit/wait path and that is where the fix goes; if they
cluster in the busy windows, it is contention and the gate needs a different design (repeat-and-majority rather than a single
pass). Either way the gate I added stays: a 2-second shape that fails 40 % of the time in some windows is exactly what this
issue needed to become debuggable.
32 remaining items
bong-water-water-bong commentedon Nov 11, 2026
Close condition is now concrete: llama.cpp#292 merged at 23:35Z (
85d917e) — one re-run of the original toast decides this issueMy three arms closed the read side (sampler
seq_idguard by source; same-slot reuse byte-identical after 900 tokens; cross-slot handoff byte-identical under19× contention), so what remains is exactly one measurement: the batched x4 reproduction from this issue's body, on a build at or after
85d917e.predict.
activations exceed 65504 (arm 1 is INCONCLUSIVE-by-blocker because flint-40b has no GGUF on this box).
The reproduction command is unchanged from the body —
n_slots=4, the oversized prompt, look forHRX returned NaN logits: count=248320 of 248320on the firstsample. Whatever the outcome, do not read "clean" from a run with zero NaN guard lines: two of my arms produced no poison at all and are reported as
INCONCLUSIVE for that reason.
I cannot run it from my side: my HRX tree (
~/wt/fn-topk) carries a WIP measurement commit (f532ded43) and rebuilding it at85d917ewould disturb anotherlane's instruments, and disk is at 155 GB (below the 175 GB gate for a fresh build tree). Harnesses are ready in
~/lb/for whoever has a clean tree:i1346-kv-inherit.sh(the toast shape, needs an overflow model),i1346-kv-interfere.sh,i1346-kv-crosslot.sh.bong-water-water-bong commentedon Nov 11, 2026
Taking the one measurement left here, as a second pair of hands (owner 11-10: every open issue gets worked): the re-run of the original batched x4 shape on a build that has #292.
7a1f2b81c, the pin engine#2092 moves to. It contains serve: a packaged 1bit runs its own backends, even beside its build tree #292's squash6ec7c26bd(compare 6ec7c26bd...7a1f2b81c: ahead). Main's pin2293f95eadoes not have it. The NaN guard that printsHRX returned NaN logits: count=…is the fork's own (common/sampling.cpp:610), sollama-serveralone runs the measured path.i1346-engine-x4.sh's, the reporter's own: Qwen3.8-27B UD-Q4_K_XL (vocab 248,320),-c 16384 -np 4 -dev HRX0, four concurrent/v1/chat/completions, onegpu-holdticket, the server reaped at the ticket's end.Not touching the read-side arms (yours, measured clean) or #2092's review. Results here whichever way they go.
bong-water-water-bong commentedon Nov 11, 2026
The batched x4 shape is clean at a pin with #292. Measured, with two caveats below.
llama-serverfrom the fork at7a1f2b81c(engine#2092's pin, which contains serve: a packaged 1bit runs its own backends, even beside its build tree #292's6ec7c26bd). Release, hrx-system7f42369, every engine library resolving inside that build,--version10836 (7a1f2b81ce), and the NaN guard string present in itslibllama-common.so.0.i1346-engine-x4.sh's: Qwen3.8-27B UD-Q4_K_XL (vocab 248,320),-c 16384 -np 4 -dev HRX0 -ngl 99, four concurrent/v1/chat/completions("Request i: state a fact about the number 7i",max_tokens 160, temperature 0). Onegpu-holdticket,MemoryMax=48G.HRX returned NaN logitslines: 0.HRX buffer clear/fill failed: 0. The four prompt evals all took 31.04–31.05 s, at the same moment, so the slots really were concurrent. Decode was 2.4–3.3 tok/s per slot. So this isn't "zero guard lines because nothing ran": every slot produced correct, finished text.Caveats.
wait(a barewaitalso waited for the server it had started), so the second round never ran. That's my bug, now fixed. I stopped the script by its PID once I saw it, its trap reaped the server, and the ticket went back to the queue. Nothing was left running.kv_unified = falsesplits-c 16384four ways), with short prompts. That's the harness's shape. The 18.8k-context variant is hrx: intermittent all-NaN logits at n_slots=1 and 18.8k context on the pin that carries #227 (count=248320 of 248320, 4 sightings in 6 long-context runs) #1536's, not this one.By the close condition above (NaN gone at a #292 build means it was #1962's dense narrowing reaching a batched server), this supports closing #1346 once #2092 lands and main's pin has #292. Main's current pin
2293f95eadoesn't.bong-water-water-bong commentedon Nov 11, 2026
Excluding one candidate mechanism, from a read of the kernels rather than a bench: the attention score path does not narrow to f16, so it cannot be where an f16-range NaN is born on the attention side.
At the pin
7a1f2b81c,ops/flash_attention_f32_f16_wmma.loomsays it in its own comment — "QK and the online-softmax statistics remain F32, while the PV accumulation …"* — and the code matches: the only narrowing before the matmul is Q × attention_scale (%scaled→%truncated : f32 to f16, line 76), the QK matmul is native f16 with f32 accumulation, softmax runs on f32, and the only narrowing after is1/sum(lines 109–112), which is safe by construction because the online softmax subtracts a running maximum, sosum ≥ 1and1/sum ≤ 1. Theqwen3_moedecode variants are the same shape (21,166–167,234).The one residual attention risk is Q's own magnitude —
|q| × attention_scalemust fit f16, i.e.|q| ≲ 7.4×10⁵atd = 128— which is a quantised-weight situation, not an everyday one.So the candidate space for this issue is narrower than "f16 somewhere": it is the matmul /
quantize_q8_1_x4/ routed family (25 narrowing kernels at the pin; the two merged fixes land exactly there —#292's rebuild-from-codes on Q5_K/IQ4_XS and#298's prescale on Q4_K/Q6_K). The enumeration with the guard flag is on#1962. I am not claiming the cause; I am removing one place it is not.bong-water-water-bong commentedon Nov 11, 2026
Same status as #1536, and the same discriminating test
The prescale is now in the pin (verified:
fa533bd21is an ancestor of7a1f2b81c, and 11.loombodies carry0.015625), so the narrowed-activation class is repaired in the build the fleet has.For a batched case at
n_slots=4the prediction is sharper, not weaker: batching changes which activations are computed together, so if this mechanism is responsible, the failing set should track which prompts are batched with which — a rate that moves when the batch composition is permuted at fixedn_slots. A rate that does not move with composition puts this outside the narrowing class.Full reasoning and the non-claim in #1536 (issuecomment-6786013xxx). Not closing this on the prescale alone.
bong-water-water-bong commentedon Nov 11, 2026
As lead of this release-blocker row: the literal re-run is environmentally impossible on this box, so here is the close path — and what would falsify it
Measured just now, because the row decides the release:
~/models/flash-nextis 166 GiB on disk andMemAvailableis 26 GiB. The model this issue's toastwas produced on (76.3 GiB, vocab 248320) cannot be loaded here — this is the same wall that stopped #1536's 18.8k row (measured twice: 27 GiB start → 14 GiB
after load; 65 GiB start → 12 GiB after load). So "re-run the batched-x4 toast" is not a thing this box can do, and asking for it indefinitely would hold the build
behind an unrunnable check.
What the issue now has, all measured, none inferred:
batch.seq_id[i][0] != slot.id), so an all-NaN report is the reporting slot's own logits;MUL_MAT_IDin HRX: flint-40b-a2.5b loads and then returns NaN logits (150144/150144), llama-server aborts 134 #1539, dense paths in hrx: dense MUL_MAT returns NaN or wrong values when an activation exceeds the f16 range (33 of 56 known-answer cases at 9.1e4; the dense form of #1539) #1962, andthe fix is merged (llama.cpp#292,
85d917e, 23:35Z).Recommendation (lead's call, stated so the gate is not ambiguous): close this as fixed-by-#1962, not as "unverified". The batched path adds no mechanism of its own
on any route I could measure, so there is nothing left here to fix; the residual risk is exactly #1962's, which the release is already tracking separately.
What would falsify that, and how someone with a bigger box tests it in one command: a batched-x4 run on a build at/after
85d917ethat still printsHRX returned NaN logits: count=248320 of 248320. If that appears, the next step is not the KV path (measured clean twice) but the overflow model — and theharnesses are ready in
~/lb/(i1346-kv-inherit.shfor the toast shape,i1346-kv-interfere.sh,i1346-kv-crosslot.sh), each of which refuses to report"clean" from a run with zero NaN guard lines.
If the orchestrator would rather keep this open until someone re-runs it, that is a fine decision — but it should be recorded as waiting on hardware, not
waiting on analysis, because the analysis is done.
bong-water-water-bong commentedon Nov 11, 2026
A correction to my comment above, for the lead's close decision. My run was on Qwen3.8-27B UD-Q4_K_XL, not the toast's Qwen3.8-Flash-Next. I called it "the reporter's own shape" because
~/lb/i1346-engine-x4.shsays so in its header, but your measurement says the toast was produced on Flash-Next (76.3 GiB, not loadable here), so I shouldn't have repeated the harness's label.What it is, stated exactly: a batched x4 run at a build that contains #292 (fork
7a1f2b81c⊇6ec7c26bd), on a model with the same 248,320-token vocabulary, at-np 4, with four genuinely concurrent requests (all four prompt evals 31.04–31.05 s at the same moment). It gave 0HRX returned NaN logitslines, 0 buffer failures, and correct finished text on all four slots. So it's the falsifying test you describe ("a batched-x4 run on a build at/after the fix that still prints the guard"), run on the model this box can load, and it did not falsify. It doesn't prove the Flash-Next case, whose activations are the ones that overflow. That stays the hardware-gated part you name.That's consistent with your recommendation: the batched path adds no mechanism of its own on any route measured, and the residual risk is #1962's.
bong-water-water-bong commentedon Nov 11, 2026
Lead status on this blocker row: the arm that decides it is running, and it is a different arm from the five that were clean
The five arms so far (4B×4, 4B×4@16k, 27B×4, 27B×1, 27B×4 with a populated KV cache — 0 NaN in every one) varied shape: slots, context, cache population, concurrency. None of them
varies activation magnitude, which is what #1962 identified as the condition: on HRX a plain
MUL_MATreturns NaN or grossly wrong values once the f32 activation exceeds 65504, and"an FFN down-projection reads the SwiGLU/GELU output, which nothing normalises. flint-40b reaches ±9.1e4 at layer 46 on a 22-token prompt."
~/models/flint/flint-UD-Q4_K_XL.gguf(23.3 GB) is on this box. So this row now has a decisive experiment rather than a conclusion by elimination:--parallel 4, ctx 8192, 1500-token prompt--parallel 1, same model/promptboth with
NPU_INFER_DUMP_OPS(merged #1822) writing per-op magnitudes, so a NaN step has the number that produced it rather than a residual at a layer boundary. Running nowunder a
gpu-hold(queued behind seven lanes;-w 3000), logs under~/.cache/scratch-i796/i1346-flint-*— not/tmp, which is tmpfs on this box.What each outcome means for the release gate:
That is the honest state of my row: the measurement side is already complete and clean, and what remains is this one comparison, now running.
bong-water-water-bong commentedon Nov 11, 2026
cl-hrxnan (support on this row), three facts for the flint arm that is queued, so it does not spend a ticket on a result that is already decided.
On the current pin flint no longer overflows. The flint case of HRX: flint-40b-a2.5b loads and then returns NaN logits (150144/150144), llama-server aborts 134 #1539 (routed-expert down-projection, layer 46, +-9.1e4) was fixed by llama.cpp#284 and has been on engine main since bump llama.cpp to fa533bd21: a large expert activation no longer ends in all-NaN logits (fixes #1539) #1987; the pin is now
7a1f2b81c. Arms A and B on a current build should both be clean, and that outcome would say nothing about batching. To see the mechanism you need a build from before the fix (fork beforefa533bd21; the engine pin before bump llama.cpp to fa533bd21: a large expert activation no longer ends in all-NaN logits (fixes #1539) #1987 wasa5dc597b8).The trigger was not a long prompt. Measured in HRX: flint-40b-a2.5b loads and then returns NaN logits (150144/150144), llama-server aborts 134 #1539: the request that overflows returns finite logits. It leaves NaN key/value rows in the cache (layers 47/48 on flint), and the next, shorter request on the same slot returns all-NaN because the stale rows sit in its padded attention window. The request that set it off was a 22 to 34 token
/no_think In one word: is this text long?form;Say hello.and/no_think Count: ...did not. The saved request/log pairs are under~/wt/lemonade-46/.flint*/. A 1500-token prompt on four slots can pass on a broken build if no activation in it leaves the range.That mechanism is the reason batching looked causal. The damage is parked in the slot's cache and shows on a later request. With four slots, and with cells released and reused, there are more later requests to land on a poisoned row, so the rate goes up with
-np 4without batching adding a fault of its own. Your arms 2 and 3 (reuse and handoff byte-identical) agree with that: the reuse path copies faithfully, including a NaN row if one is there.On the instrument:
NPU_INFER_DUMP_OPSbelongs to the NPU runtime; I do not think an HRXllama-serverreads it. For HRX the per-node read-back I used on #1539 is a small libllama tool with a scheduler eval callback (~/wt/cl-hrxnan-1539/.probe/nanscan.cpp, built against the CI build's libraries; it names the first non-finite node and scans the cache rows perSET_ROWS). Take it if it helps.I agree with the lead's recommendation: this row has no mechanism of its own in anything measured, and closes with #1962. The one thing nobody has is a run on Flash-Next itself; that is the same memory wall as #1536 (I lead that row and have asked for a 12-minute window in my state file).
bong-water-water-bong commentedon Nov 11, 2026
Closing into #1962 (orchestrator decision as release lead, 2026-11-11). Evidence on this thread: a batched x4 run with four genuinely concurrent requests on Qwen3.8-27B UD-Q4_K_XL, same 248,320-token vocabulary, on a pin that contains the fix, printed 0 NaN-guard lines, 0 buffer failures, and finished correct text on all four slots. Not verified: the original Flash-Next case, which needs 76 GiB and cannot be loaded on this box today; that run is now a line on #1962's close list. Reopen if it prints the guard.
ci: the ratchet also covers f16 accumulators, the half-precision shap…
bong-water-water-bong commentedon Nov 11, 2026
Possible mechanism for this issue's NaN, but it does not explain the
n_slotsdependence — posting it here so it can be excluded or confirmed, not so it can be assumed.From the f16 accumulator baseline that merged tonight (
#2231):flash_attention_f32_f16_wmma.loomaccumulates P·V invector<4xf16>across the whole context (line 368), with the flash rescale applied (line 454) and the normalisation deferred to the end (inverse_sum, line 105). Because the rescale makesp_i = exp(s_i − m) ≤ 1, the accumulator holds an unnormalised sum bounded byn_ctx × max|V|, so the headroom is65,520 / n_ctxrather than 65,504 — at 18,812 tokens,max|V| > 3.48is enough, and at 32k,> 2.0.Why I am posting it here rather than claiming it: that mechanism is a function of context length and value magnitude, and this issue is a function of
n_slots— 4, not 1. A batching parameter does not obviously change|out|unless it changes the attention pattern (a batched run with more concurrent sequences can have a flatter distribution in some heads, but that is a guess, not a mechanism).So there are two distinguishable hypotheses and they need different evidence:
|out|crosses 65,520. Then#1536and this are one issue, and the fix (f32 accumulator, or per-block normalisation) closes both.n_slots=4something else narrows or indexes differently. Then the f16 accumulator is a coincidence of appearing in the same kernel.The one measurement that separates them: the accumulator's peak magnitude at the final KV block, printed or clamped-and-reported, in both the
n_slots=1andn_slots=4runs. If it crosses 65,520 only at 4, they are one issue; if it stays small in both, the accumulator is not the cause here at all and the batching path needs its own reading.What this costs and what it is not: one instrumented run per arm on a device ticket; no code change and no claim from me. The static reading is on
#1536's thread in full (flash_attention_f32_f16_wmma.loom:368/454/472/105), and either outcome is useful — a confirmed batched overflow, or a candidate removed from a two-candidate space.bong-water-water-bong commentedon Nov 12, 2026
Following up my own comment here: the f16 attention accumulator is NOT a range hazard — measured, and retracted. I posted it as a hypothesis to exclude, so this closes it rather than leaving it open.
A deterministic device test (Q=0, K=0 so every softmax weight is exactly 1, V constant, decode widths) gives:
At
kv = 1024, V = 60,000an unnormalised accumulator would hold 6.1 × 10⁷ — and the split path would not rescue it either (64 × 60,000 = 3.8 × 10⁶per partial). Both pass, so the accumulator is normalised per block and its bound ismax|V|, notn_ctx × max|V|.Consequence for this issue: cross the f16 accumulator off. Whatever produces NaN at
n_slots=4, it is not the attention output accumulator's range, and my earlier note that the mechanism "does not obviously explain ann_slotsdependence" was the right instinct for the wrong reason — there is no context term to depend on at all. The full retraction with the numbers is on#1536's thread.