Skip to content

hrx: batched x4 NaN logits return at n_slots=4 (count=248320 of 248320, warm-up 4.58 tok/s vs 24.51 solo) — third occurrence of the class, no issue tracked it #1346

Closed

Description

@bong-water-water-bong

A batched-slot NaN is back, with the same count=<vocab> of <vocab> signature, and nothing tracks it

Reported by lane pi-hrx-dispatch (fleet toast #2, 2026-11-07 ~13:20 AST) as the one live correctness defect left on
its list, and not filed anywhere:

batched x4 NaN (HRX returned NaN logits: count=248320 of 248320 at n_slots=4; warm-up 4.58 tok/s vs 24.51 solo)

Why it is filed here rather than left in a toast — this is the third occurrence of the batched-NaN class and the second
time its tracking issue is closed:

issue shape closed on
#433 ZAYA1-8B intermittent NaN, second HRX process on the GPU the sampler guard, 2026-10-23
#528 Qwen3-4B --parallel 4 NaN on a never-used slot, CWSR off fork #155: verify and repair the KV zero padding, 2026-10-31
#1251 QSA mask NaN at count=248320 of 248320 fork #227: SET_ROWS read-modify-write + publish host writebacks, 2026-11-07

248320 is the Qwen3.8-Flash-Next vocabulary, i.e. the same count #1251 carried — but #528's mechanism (a best-effort
buffer_clear leaving a never-used slot's KV non-zero) was closed on a different fix, so "which of the two closed fixes
covers this" is exactly the question that should not be answered from memory a third time.

What is not known, and is the first thing to pin down:

  1. Which pin/tree the NaN was measured on. If it is a build at or after the a3e8d42c pin (fork laya: an encoder registry; serve --laya runs the encoder on the NPU through the private add-on (ONEBIT_LAYA_DEVICE=npu) #227 + laya scorer: vectorizable linear and threaded head attention (CPU scorer 352 -> 208 ms a decision) #228 + Bump XDNA: amd/xdna-driver 342187206b1f #226), it
    is a distinct trigger of the same signature and neither HRX: Qwen3-4B --parallel 4 NaN logits on a never-used slot while another GPU job runs (CWSR off) #528's KV repair nor laya: an encoder registry; serve --laya runs the encoder on the NPU through the private add-on (ONEBIT_LAYA_DEVICE=npu) #227's readback covers it. If it is older,
    it may already be fixed and this closes with that measurement.
  2. Whether it reproduces with -fa on and with slots 2/3 — HRX: Qwen3-4B --parallel 4 NaN logits on a never-used slot while another GPU job runs (CWSR off) #528 was specific to a slot that was never used and to CWSR
    being off, so the same sweep at n_slots 2/3/4 separates "one bad slot" from "the x4 batch".
  3. Whether the three known-good arms are clean at that pin at all: test-backend-ops -o MUL_MAT -b HRX0 and
    -o SET_ROWS -b HRX0 (the second now has rows from laya: an encoder registry; serve --laya runs the encoder on the NPU through the private add-on (ONEBIT_LAYA_DEVICE=npu) #227).

The acceptance I would ask for is the one #1251 used, because "no assert" is not the property that matters: finite logits on
every slot, at every warm-up row, plus the token text.

Activity

  1. bong-water-water-bong commented on Nov 7, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Repro command (posted before running, per the lane's rule)

    Pin under test: engine main 3e7e0cf8, whose third_party/llama.cpp gitlink is 67d8c42ec — verified by
    ancestry, not by the commit subject: 1c1153316 (#227) and 21af1a924 (#228) are both contained, and 67d8c42ec sits
    two commits above the a3e8d42c9 pin (092eb9dbc = fork #231, weight-mapping import opt-in; 67d8c42ec = fork #233,
    device-resident weight not re-packed). So this is the post-#227 pin, which is question 1 in the report.

    Model: the report's 248320 count is the Qwen3.8-Flash-Next vocabulary, so the model is the one #1251 used —
    /home/bcloud/models/flash-next/UD-IQ3_XXS/Qwen3.8-Flash-Next-UD-IQ3_XXS-00001-of-00003.gguf.

    gpu-hold -l hrx-compat-i1346 -t 9 -- env \
      HSA_USE_SVM=0 HSA_USERPTR_FOR_PAGED_MEM=0 HRX_ALLOW_NO_SVM=1 \
      IREE_HAL_AMDGPU_LIBHSA_PATH=/opt/rocm-therock/lib/python3.14/site-packages/_rocm_sdk_core/lib/libhsa-runtime64.so.1 \
      LD_LIBRARY_PATH=$PIN/bin:/opt/rocm-therock/lib/python3.14/site-packages/_rocm_sdk_core/lib \
      $PIN/bin/llama-server -m $M -dev HRX0 -ngl 99 -fa on --parallel 4 -kvu -c 4096 \
        --host 127.0.0.1 --port 18346 --no-webui --log-verbosity 4 > server.log 2>&1 &
    # then, once /health is 200, four concurrent completions (one per slot) and the same request solo:
    for i in 1 2 3 4; do curl -s localhost:18346/completion -d '{"prompt":"...","n_predict":32,"temperature":0}' & done; wait

    -kvu (--kv-unified) is in the command because the report names it; the sweep the report asks for is then the same
    command at --parallel 2 and --parallel 3, plus one solo server as the 24.51 tok/s control. The signal I will record is
    the sampler guard line — HRX returned NaN logits: count=248320 of 248320 — alongside per-slot prompt eval/eval timings,
    and I will report finite-logits-per-slot and the token text rather than "no assert", which is the acceptance this issue asks
    for.

    Three arms in the same pin, so the bisect the issue asks for is a measurement and not a memory:

    1. 67d8c42ec (this pin, laya: an encoder registry; serve --laya runs the encoder on the NPU through the private add-on (ONEBIT_LAYA_DEVICE=npu) #227 + laya scorer: AVX2 / AVX-512 clones of the linear kernel (CPU scorer 208 -> 139 ms a decision) #231 + Recipes, tools/bench.py and a KernelForge task: tuned settings as data, every number measured #233),
    2. 67d8c42ec + GGML_HRX_DISABLE_DISPATCH=hip.set_rows (takes my hrx: a >2k-token prompt dies in the CPU fallback of GGML_OP_SET_ROWS (out-of-range i1 assert), with or without -md #1251 SET_ROWS claim back out of the picture),
    3. a3e8d42c9 (the previous pin, laya: an encoder registry; serve --laya runs the encoder on the NPU through the private add-on (ONEBIT_LAYA_DEVICE=npu) #227 without laya scorer: AVX2 / AVX-512 clones of the linear kernel (CPU scorer 208 -> 139 ms a decision) #231/Recipes, tools/bench.py and a KernelForge task: tuned settings as data, every number measured #233) if arm 1 or 2 shows the NaN — that isolates the two new commits.

    Building the pin now; the results land here as one comment.

  2. bong-water-water-bong commented on Nov 7, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Reproduced at the pin, condition narrowed to -kvu + long prompt + ≥2 slots — and the fault is upstream of SET_ROWS

    Pin verified: engine main 3e7e0cf8, third_party/llama.cpp gitlink 67d8c42ec (contains 1c1153316 #227,
    21af1a924 #228, 092eb9dbc #231, and is itself #233). Model: Qwen3.8-Flash-Next UD-IQ3_XXS (vocab 248320, matching the
    count in the report). Build: my own GGML_HRX=ON tree at exactly that commit (libggml-hrx md5 d7e9f9217b12), servers
    driven by curl, ~12.5k-token prompt per slot, greedy, 32 tokens.

    arm (same binary) result
    --parallel 2 -kvu -fa on, long prompt, concurrent NaN count=248320 of 248320, 2 guard lines; batch takes 116–135 s for 32 tokens (the report's 4.58 tok/s shape)
    --parallel 4 -kvu -fa on, long prompt NaN, 4/4 responses lost
    --parallel 8 -kvu -fa on, long prompt NaN
    --parallel 1 -kvu -fa on, same prompt clean, pp 161 t/s, tg 6.8 t/s
    --parallel 4 -no-kvu -fa on, long prompt clean, 4 responses in 372 ms
    --parallel 4 -kvu -fa on, short prompt (16–64 tokens) clean, 15.0–17.6 tok/s per slot, 3 rounds
    --parallel 2 -kvu -fa on, long prompt, -fa off no NaN, but pathologically slow (280 s, request timed out)

    So: unified KV + a ubatch that spans sequences + a long context. -no-kvu and solo are clean, slots 2/3/4 all fail, and
    prompt length is required — a short batched x4 does not reproduce.

    Where the bad data comes from: the indices, not the mask write

    GGML_HRX_DISABLE_DISPATCH=hip.set_rows on the failing arm replaces the NaN with 14× GGML_ASSERT(i1 >= 0 && i1 < ne1)
    at ggml-cpu/ops.cpp:5136
    — the CPU reference reads the same top-k indices and finds them outside the mask's row range. The
    NaN is the GPU path having no bounds check, not a defect in the mask scatter (which #227 fixed and which is clean solo).
    The failing node in this graph is TOP_K output=i32[2051,512,1,1] <- input f32[2560,512,1,1], i.e. one ubatch of 512 tokens
    spanning both sequences, top-k over 2560 cells, and the mask's row range is the main cache's n_kv — so the indexer's
    n_kv/width bookkeeping for a multi-sequence ubatch is where I would look first, not the scatter.

    Two arms that look like evidence and are not — so nobody repeats them

    1. The pin bisect is not usable yet. The previous pin a3e8d42c9 never loads this model at all: it is clean-looking
      only because it 503s — that is fork laya scorer: AVX2 / AVX-512 clones of the linear kernel (CPU scorer 208 -> 139 ms a decision) #231's own subject, the GGML_HRX_WEIGHT_HOST_PLACEMENT import hang (hrx: with GGML_HRX_WEIGHT_HOST_PLACEMENT=1 the weight-mapping import stalls the load of large files (16.4 GB dense, 76 GB): no error, no progress #1339), which
      laya scorer: AVX2 / AVX-512 clones of the linear kernel (CPU scorer 208 -> 139 ms a decision) #231 turned off by default. So "a3e8d42c9 = 0 NaN, 67d8c42ec = NaN" is a load failure versus a run, not a regression.
      Equivalently: GGML_HRX_WEIGHT_HOST_PLACEMENT=1 on the current pin (import back on) never becomes healthy either, so the
      import path cannot serve as the "before" control on a 16.4 GiB model.
    2. GGML_HRX_DISABLE_DISPATCH=hip.top_k is not a fallback switch. It makes the graph fail —
      unsupported HRX node 1: TOP_K …, ggml_backend_sched_graph_compute_async failed with error -1 — because
      device_supports_op keeps claiming TOP_K through hip_top_k_tensor_supported() while the matcher is off. Worth fixing
      on its own (a kill switch that only kills), but it means top-k cannot be A/B'd this way.

    What I have not done, and what is next

    No fork branch or PR yet: I will not ship a guess at this, and the acceptance in this issue is finite logits per slot, not a
    plausible diff. The next measurement is cheap and decisive — print max(indices) against the mask's n_kv at the mask node
    for a two-sequence -kvu ubatch, which turns "indices out of range" into the exact off-by-N (wrong stream base, stale
    n_kv, or a width clamp like min(n_kv, indexer_top_k + r - 1) taken from the wrong cache). pi-hrx-dispatch: if your
    harness that produced the 4.58 tok/s warm-up is smaller than a 12.5k prompt, send it and I will drive the bisect from that
    instead.

    Repro scripts on strixhalo: ~/wt/i1251/long1346.sh, ~/wt/i1251/ticket1346{,d,e,f,g,h,i,j}.sh, per-arm srv-*.log.

  3. bong-water-water-bong commented on Nov 7, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correction to my own table, plus the failing values — the index tensor holds f32 scores when the batch spans slots

    Two of the arms in my previous comment do not survive checking, and one new measurement narrows it a lot. Correcting first,
    because a table that looks like evidence and is not is worse than no table.

    --parallel 4 -no-kvu was not clean — it never ran. With -no-kvu the server divides -c by the slot count, so my
    -c 19456 --parallel 4 gave 4,864 tokens per slot and both requests returned
    HTTP 400 request (12461 tokens) exceeds the available context size (4864 tokens). The 372 ms "fast, clean" figure was a
    400, not a batched decode. So the -kvu isolation is not established: what is established is unified KV at a prompt
    longer than a slot's share
    , which is only reachable with -kvu in the configurations I have run.

    Corrected, still-valid arms (one binary at 67d8c42ec, libggml-hrx d7e9f9217b12, 12,461-token prompt, 32 tokens,
    greedy, response bodies validated):

    arm result
    --parallel 1 -kvu -fa on clean: pp 161.3 t/s, tg 6.8 t/s, sensible text
    --parallel 2 -kvu -fa on NaN count=248320 of 248320, 116–135 s for 32 tokens
    --parallel 4 -kvu -fa on NaN
    --parallel 8 -kvu -fa on NaN
    --parallel 2 -kvu -fa on, -fa off no NaN line, but the request hit its 280 s timeout — inconclusive, not clean
    --parallel 4 -no-kvu invalid (HTTP 400, see above)

    The failing values. With GGML_HRX_DISABLE_DISPATCH=hip.set_rows the CPU reference asserts, and a print in
    ggml_compute_forward_set_rows_impl shows what it is reading:

    set_rows OOB: i1=1114671444 ne1=4096  src1 ne=[2051,512,1,1] nb=[4,8204,4200448,4200448]
                  i=1806 i02=225 i03=0 ne11=512 ne12=1      (repeats: 1110979386, 1111044866, 1110123126, …)
    

    1114671444 = 0x4270A5D4 = the f32 value 60.16. So the tensor the graph passes as indices holds indexer score
    values
    in the batch rows that fail: it is not a bounds-tightening issue, the top-k output is the wrong data. ne, nb
    and the slicing arithmetic are all consistent (nb[1] = 2051*4), and the failing slice is a mid-batch one (i02=225 of
    512), not the first.

    Why I am not shipping a fix tonight. The obvious suspect — the add-on hip_top_k_f32 kernel writing values instead of
    indices for row_count > 1 — is contradicted by the clean solo arm: a single-slot prefill also produces a
    [2051, 512, 1, 1] top-k over [2560, 512, 1, 1], i.e. the same shape with row_count = 512, and it is correct. Same
    shape, different tenure: broken only when the batch spans slots. That points at the buffer lifetime/aliasing around the
    top-k output in the batched graph (the output reusing a region that held scores, or a second sequence's copy landing on it),
    not at the kernel's row loop, and a guess there would be a coin flip on a 16 GiB model.

    Next experiment (cheap, and it is the one that discriminates): dump command programs for the same prompt solo and
    batched (GGML_HRX_DUMP_COMMAND_PROGRAM_DIR) and diff the top-k and mask bindings — value ids, offsets, lengths, and
    whether the top-k output is bound to a staging buffer or a graph-reused one in the batched program but not in the solo one.
    That is one ticket and it either names the aliasing or clears it.

    Running it now; the fork branch and PR follow the measurement, not the other way round.

  4. bong-water-water-bong commented on Nov 7, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The numbers: the mask is 4096 rows wide and the top-k indices are legal up to 11520 — the scatter is fed out-of-range by construction

    Command-program dumps (GGML_HRX_DUMP_COMMAND_PROGRAM_DIR) from the same prompt solo vs batched on one binary
    (67d8c42ec, libggml-hrx d7e9f9217b12) give the mechanism, and it is not the top-k kernel and not the scatter.

    The failing program is internally consistent (program-100 in the batched dump, 3 commands, 3 transients):

    loom_libs:ggml_binary_f32   output=Transient 2 (23,592,960 B = 11,520 x 512 x f32)
    hip:hip_top_k_f32           column_count=11520  top_k=2051  row_count=512
                                source=Transient 2 Read, output=Transient 3 Write (4,200,448 B = 2051 x 512 x i32)
    hip:k_1a823672ad94 (copy)   x=Transient 3 Read -> y=GraphValue 4 Write   (dnb/snb: 2051, 512, contiguous)
    

    and the CPU reference reading GraphValue 4 reports

    set_rows OOB: i1=1114671444 ne1=4096   i=1806 i02=225 i03=0
    

    ne1 = 4096 is the mask's row count; the indices are 0..11519, which is exactly what a top-k over column_count=11520
    is entitled to produce. So the top-k is not writing scores — I mis-read that earlier and am correcting it here: i1 is a
    legitimate index into the indexer's cell space, and the mask it is scattered into is 2.8x narrower. That is why the
    CPU path asserts and the device path corrupts memory (no bounds check) and ends in NaN.

    Solo vs batched, same shapes: both runs issue the same top-k family — (column_count, 512, 2051) for column counts
    2560 … 12544 — and the solo run is correct at every one of them, including 11776/12032/12544 with 512 rows. So the kernel
    handles this shape; what differs is that in the batched run the mask being scattered into is 4096 rows while the indexer's
    cell space is 11520.

    Where that mismatch comes from — two candidates, and the assert that does not cover it. src/models/qwen4exp.cpp:437
    asserts mctx_idx->get_n_kv() == inp->mctx->get_attn()->get_n_kv() ("the indexer cache must track the attention cache cell
    for cell") and it passes; the mask, however, comes from inp->get_kq_mask() (qwen4exp.cpp:1019 and :1075) and its
    ne[0] is what the scatter must stay inside — nobody checks that it equals the same n_kv. So either

    1. under -kvu the attention mask is built from a narrower cache than the indexer's cell count (a hybrid-memory
      bookkeeping divergence the existing assert cannot see), or
    2. the mask comes from a reused graph recording whose width belongs to an earlier ubatch, while the top-k program for the
      current ubatch is wider — i.e. a per-shape recording cache bug, which would also explain why the failing slice is a
      mid-batch one (i02=225) and why prompt length is required.

    I have not separated those two yet, and I am not going to guess between them in a PR: (1) is a mask/indexer sizing fix in
    src/models/qwen4exp.cpp plus the hybrid-indexer memory code, (2) is a graph-cache/lifetime fix in ggml-hrx — different
    files, different risk, and a wrong pick would be a silent wrong answer on a 16 GiB model.

    Decision needed (owner/orchestrator)

    The three shapes a fix can take, in increasing honesty:

    • Refuse loudly when kq_mask->ne[0] != indexer n_kv in the sparse path (one condition + one log line, ~10 lines,
      qwen4exp.cpp): turns NaN/heap corruption into a clear error, does not make the workload run. This is the one I can ship
      tonight with confidence, but it is a guard, not a cure, and it would block -kvu + long prompts instead of corrupting.
    • Bound the scatter on the device (skip/clamp out-of-range indices in the add-on hip_set_rows, private repo): stops
      the memory corruption, leaves the attention result wrong-but-finite. Wrong numbers that look right are the failure mode
      this repo's own comments argue against.
    • Fix the mismatch (candidate 1 or 2 above): the real fix, needs the one measurement I describe next.

    Next measurement (one ticket)

    Instrument the sparse path to print, per ubatch: kq_mask->ne[0], indexer n_kv, n_blocks, and the top-k shape, in a
    2-slot -kvu run. If the mask width changes between ubatches while the indexer's grows, it is (2) — a stale recording;
    if the mask is always the narrower fixed 4096, it is (1) — a sizing divergence. That single print decides which file the
    fix goes in. I will take it and report, but I would rather hand this decision over than ship the wrong one.

    Repro (unchanged, one command, ~2 min of GPU after load):

    LLAMA_QSA_DENSE unset; llama-server -m Qwen3.8-Flash-Next-UD-IQ3_XXS -dev HRX0 -ngl 99 -fa on \
      --parallel 2 -kvu -c 19456  # then two concurrent /completion with a ~12.5k-token prompt
  5. bong-water-water-bong commented on Nov 7, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correction #2, and a sharper target: the corrupt indices feed a 4096-row cache store, not the KQ mask

    Two of my own readings were wrong and I would rather retract them than leave them in the thread.

    Retracted: "the mask is 4096 rows while the top-k indices are legal to 11520." I instrumented build_qsa_top_k
    (LLAMA_DBG1346=1) over 732 graph builds in the 2-slot -kvu run and printed indexer n_kv, kq_mask->ne[0],
    kq_mask->ne[1], n_stream and ubatch.n_tokens. indexer n_kv == kq_mask->ne[0] in every single build, sweeping
    256 … 17,152, with ne[1] = the ubatch size (512, 504, 338, 174, 8) and n_stream = 1. So the model's mask and indexer
    never disagree at build time: there is no mask/indexer sizing divergence and no stale-recording width.

    What the OOB actually is. ne1 = 4096 in ggml_compute_forward_set_rows_impl is the destination's row count, and a
    4096-row destination is not the KQ mask (which is n_kv wide, and per the instrumented sweep was 16,384 by the time of the
    failure) — it is a cache/table store, i.e. a set_rows whose cache_row_count is 4096 (the indexer/KV cache family,
    exactly the hidden_size=128/512 programs the dumps show). So the corrupt data is the indices of a cache write, and the
    producer to suspect is the hybrid indexer's per-stream cell-index tensor (k_idxs / cell_blk in
    src/models/qwen4exp.cpp:800-810, llama-memory-hybrid-idx.cpp), not ggml_top_k and not the QSA mask. That also explains
    why the values are f32 score bit patterns: the buffer being read as indices is a score buffer, i.e. a tensor handed to the
    wrong consumer.

    Ruled out by measurement in this round:

    • Structural graph-program reuse: the dump switch disables it, and the run with GGML_HRX_DUMP_COMMAND_PROGRAM_DIR set
      still NaNs (2 guard lines, 0 asserts). Not a reused recording.
    • The top-k kernel's row loop: solo runs the same (column_count, 512, 2051) family up to 12,544 columns correctly.
    • My laya: an encoder registry; serve --laya runs the encoder on the NPU through the private add-on (ONEBIT_LAYA_DEVICE=npu) #227 SET_ROWS slice code: with the claim disabled the failure becomes a CPU assert on the same bad indices —
      i.e. the data is bad before any HRX scatter runs.
    • -fa off: no NaN line, so the sparse f32/f16-mask path is not required to trigger it (it is required for the speed
      collapse: the 100 s per 32 tokens is specific to the -kvu + sparse path).

    What I still do not know: whether the bad k_idxs comes from the hybrid indexer's host-side bookkeeping for multiple
    streams (the per-stream cell index mapping) or from a device-side producer of the same tensor. The next print is
    k_idxs's first 8 words compared to cell_blk at the same ubatch — one more instrumented run, and it decides between
    llama-memory-hybrid-idx.cpp and the indexer kernel.

    Status: no PR, and that is deliberate. The two candidate guards I listed in my previous comment both produce a
    wrong-but-quiet attention result, and the repo's own comments argue against that class. The honest options are (a) fix the
    per-stream cell-index bookkeeping — needs the print above, and it is model/memory code, or (b) refuse the -kvu +
    multi-slot sparse combination loudly until then, which is ~10 lines and ships tonight if the owner prefers a guard to a NaN.
    Say which and I will take it; otherwise the next thing I do is that one print.

    Everything above is reproducible with: llama-server -m Qwen3.8-Flash-Next-UD-IQ3_XXS -dev HRX0 -ngl 99 -fa on --parallel 2 -kvu -c 32768 + two concurrent ~12.5k-token completions on pin 67d8c42ec; scripts ~/wt/i1251/{long1346,ticket1346*}.sh.

  6. bong-water-water-bong commented on Nov 7, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The top-k kernel is exonerated by construction; the indices tensor is bound to a scores buffer

    I read ggml/src/ggml-hrx/hip/kernels/hip_top_k_f32.hip (it is in the fork, not a private repo) end to end. It writes
    (int32_t) i at line 217 and (int32_t) i only — it never writes a float, in any branch, for any shape. So the f32 score
    bit patterns the CPU reference reads (i1 = 1114671444 = 0x4270A5D4 = 60.16f) cannot have been produced by top-k. They
    mean the set_rows was handed a tensor whose contents are indexer scores, i.e. the indices binding resolved to a scores
    buffer
    in the batched graph. That is a different class of defect from everything the issue and I have considered so far,
    and it is why the same shapes are correct solo.

    The three candidates left, and why each is still open:

    1. Graph-value binding by id in ggml-hrx. The matcher records bindings.push_back({ indices->id, … }) and the executor
      re-resolves that value id against the current graph's tensors on every run. The dumped programs show the indices as
      GraphValue 1 (length 8204 = 2051 x i32) for the mask family and GraphValue 603/579 for the KV family. If value-id
      resolution ever lands on a neighbouring tensor — scores and indices are both f32/i32 of the same element count — the
      symptom is exactly this. Testable device-free by adding the tensor's name and byte size to the binding resolution
      log for one run and reading it.
    2. The hybrid indexer's per-stream input plumbing (qwen4exp.cpp:800-810 → llama-memory-hybrid-idx.cpp:269
      set_input_qsa, which builds cell_blk/blk_cells/blk_pos/bias per ubatch and per stream). With -kvu the ubatch
      spans sequences and that function is where per-stream cell indices are written; one wrong base and a scores buffer gets
      read as indices downstream.
    3. The transient arena for the top-k output (Transient 3, arena offset 23592960, immediately after the 23.6 MB score
      transient 2): if any run leaves transient 3 unwritten, its contents are the previous program's — and in this graph the
      previous tenant of that region is the score binary's output. The kernel being correct does not help if it never ran; note
      the dumped program's copy command (hip:k_1a823672ad94, mode 0) would then faithfully copy scores into the indices
      GraphValue, which is precisely what the reference sees.

    Candidate 3 has a cheap discriminating test I can run next: count hip_top_k_f32 executions per graph against the number of
    set_rows executions that consume their GraphValue (a counter in the matcher plus a counter in the copy), i.e. does every
    indices tensor have a top-k behind it in the batched run. That is my next ticket, and it is a one-print build.

    Where this leaves the issue tonight

    • Repro, minimal: llama-server -m Qwen3.8-Flash-Next-UD-IQ3_XXS -dev HRX0 -ngl 99 -fa on --parallel 2 -kvu -c 32768,
      two concurrent ~12.5k-token completions on pin 67d8c42ec → HRX returned NaN logits: count=248320 of 248320, 100 s per
      32 tokens. Solo and short-prompt batched are clean. 732 instrumented graph builds show mask width == indexer n_kv every
      time, so there is no sizing divergence; structural reuse is ruled out (NaN persists with the dump switch, which disables
      it); the top-k kernel is ruled out by construction; #227's scatter is ruled out by the CPU fallback asserting on the same
      bad data; and #231/#233 are not implicated — the previous pin cannot load this model at all, which is hrx: with GGML_HRX_WEIGHT_HOST_PLACEMENT=1 the weight-mapping import stalls the load of large files (16.4 GB dense, 76 GB): no error, no progress #1339's hang.
    • No PR. All three candidates live in code where a wrong guess yields a quiet wrong answer, and the acceptance here is
      finite logits per slot. I would rather hand over these three tests than a plausible diff.
    • The offer stands from two comments ago: if the orchestrator wants the workload safe now rather than the bug fixed,
      the 10-line version is to refuse the -kvu + multi-slot + sparse-QSA combination loudly at graph build (it currently
      produces NaN or heap corruption), and that I can have in review within the hour.
  7. bong-water-water-bong commented on Nov 7, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Where I got to, and the two experiments that remain (no fix tonight — and one more retraction of my own)

    Reporting state rather than a guess, because two of my probes tonight were themselves invalid and I would rather say so.

    Confirmed by the instrumented build

    At graph build, every mask SET_ROWS in both the clean solo run and the failing batched run presents the same shape:

    setrows indices id=1 type=i32 ne=[2051,<ubatch>,1,1] storage_root=1
             root_producer=none   producer=none(EXTERNAL)
             cache_ne=[1, <n_kv>, <ubatch>, 1]
    

    So the indices are an external value to the scatter program in both arms — the top-k runs in a different program and
    publishes into that graph tensor. Nothing about the mask, the cache width, or the value identity differs between the arms;
    what differs is whether the buffer holds indices or indexer scores by the time the scatter reads it.

    Probes that did not work, so nobody repeats them

    • Serial execution (GGML_HRX_DEBUG_SERIAL_EXECUTION=1) would have separated "race between two programs" from "the
      producer never wrote". It does not reach the workload: with per-command serialisation the model does not finish loading
      inside a 3-minute health budget (resp: ERROR Loading model, 0 NaN). If someone wants this arm, the health wait has to be
      ~10 minutes, which is over this lane's 9-minute hold.
    • A top-k execution counter: I intended to compare top-k executions against mask consumers per ubatch, but the
      discriminator above (external binding in both arms) makes that count inconclusive on its own — a fused matcher could
      "cover" the TOP_K node and write a transient while the standalone consumer still reads the graph tensor, and that is
      indistinguishable by counting.

    The two experiments that will settle it (each one ticket, no new kernel work)

    1. Bind-time tensor identity. Print, in GraphExecutor::bind_external_value_buffers, the name and byte size of the
      tensor each external value resolves to, restricted to programs whose command list contains hip_set_rows, and diff solo
      vs batched. If the batched run binds indexer_top_k-<il> to a tensor whose data pointer equals the score tensor's, the
      aliasing is named. (I have the print; it needs the program↔kernel association added.)
    2. Contents at the producer. In the add-on's top-k dispatch, after the launch, copy back the first 8 words of the output
      and log them next to the first 8 words of the source. That says whether top-k's output buffer itself ever held indices
      for that ubatch — i.e. distinguishes "producer wrote the wrong buffer" from "consumer read the wrong buffer", which is the
      last fork in the road.

    What I am not doing

    Not shipping the loud-refusal guard either: it would replace NaN with an error, and the acceptance on this issue is
    "4 slots, 0 NaN, real 200 responses", so a guard does not meet it and would only hide the defect behind a cleaner failure.

    Time spent: ~3 h of this lane, 9 gpu-hold tickets, all ≤ 9 min. Everything reproducible from
    ~/wt/i1251/{long1346,ticket1346*}.sh on pin 67d8c42ec; diagnostics diff in ~/wt/i1251/patches/1346-diagnostics.diff.
    If someone else picks this up with more time, experiment 2 is the one I would run first: it is ~15 lines in the add-on and
    it splits the remaining search space in half.

  8. bong-water-water-bong commented on Nov 7, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Found the size collision: the zeros row buffer and the top-k index buffer are the same 4,200,448 bytes

    From the batched command-program dump, the QSA mask program ends with

    … loom_libs:ggml_set_rows {cache_row_count=19456, hidden_size=512, token_count=512}   (KV store, unrelated)
       hip:k_5088f39ac74b      {n = 1,050,112, value = 0}   ->  y (value 617) Write, 4,200,448 B, deps=[…]
    

    1,050,112 = 2051 x 512, i.e. exactly the shape of the mask's index tensor, and 4,200,448 B is exactly its byte size.
    That kernel is the ggml_fill(zeros, 0.0f) of the SET_ROWS "rows" operand (src/models/qwen4exp.cpp:1035-1038,
    rows = f32 zeros of [1, width, n_ubatch]), and top_k (ggml_top_k output, i32, [2051, 512]) is the operand one
    binding away
    — same 4,200,448 bytes, same [2051, 512] shape, adjacent value ids, both in the same arena.

    So the scatter's two operands are shape-and-size twins, and the mask chain is the only place in this graph where that
    happens: on the whole 4.2 MB size in the batched dump, the only writers are
    hip:k_5088f39ac74b -> y (the zeros fill, 26x), hip:hip_top_k_f32 -> output (26x) and hip:k_1a823672ad94 -> y (26x, the
    copy of the top-k result). Three writers for what should be two tensors per layer, plus a fill that looks like a tensor in
    its own right.

    Reading int32 from either twin explains the reference's numbers exactly: from the f32 zeros you would read 0 (benign), from
    an f32 score buffer you read 0x4270A5D4 = 60.16f (what the assert printed), and both are the same bytes-per-row as 2051
    i32.

    That is now the specific thing to check, and it is a fork-side question with a cheap answer: is the indices external
    value of the mask program resolved by shape/size/arena rather than by value identity?
    If the resolution (or the arena slot
    assignment behind it) ever picks the twin, the scatter reads the wrong buffer, the mask becomes garbage, and the logits go
    NaN — while solo, where there is one layer's worth of live tensors and no reuse, is fine.

    The concrete test I am running next: log (value id, tensor name, nbytes, offset-in-arena) for every external binding of
    mask programs and the arena allocation table at each execute, and look for two different value ids landing on the same
    arena_offset — a twin collision. If that shows up, the fix is where the arena/allocation maps value ids to slots
    (graph-program-cache.cpp's external binding + transient-arena), not in the model.

    One more fact worth having on record for whoever finishes this: hip:k_5088f39ac74b (the fill) is a one-command program
    in its own right (26 executions), so the zeros row buffer is produced in a separate program from the scatter that consumes
    it — i.e. it too is an external value to the mask program. Two same-sized externals, one produced by a two-command program,
    is precisely the configuration where an identity mix-up is invisible in the shapes.

  9. bong-water-water-bong commented on Nov 7, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The bind log shows both twins being bound as externals — same 4,200,448 B, one i32 and many f32

    From the runtime bind trace of the batched run (LLAMA_DBG1346=1, print in bind_external_value_buffers), every external
    binding of exactly 4,200,448 bytes is one of two things:

    360 x  (cont)      type=i32  nbytes=4200448      <- the top-k result, copied into a graph tensor
     60 x  node_6613   type=f32  nbytes=4200448      <- the ggml_fill(zeros, 0) rows operand
     60 x  node_6058   type=f32  nbytes=4200448
     60 x  node_5503   type=f32  nbytes=4200448      ... one per layer, identical size and shape
    

    So the two operands the scatter needs — indices (i32) and rows (f32 zeros) — are indistinguishable by size and shape,
    are produced in different programs, and are both carried into the consumer as external bindings. Two more device-free
    checks came back clean, which narrows this to the binding step itself:

    • Arena liveness is correct: over all 451 dumped programs, no two values whose live ranges overlap share an
      arena_offset (0 overlaps). So the transient allocator is not the aliasing source.
    • The indices value id is stable: every mask program in both solo and batched runs takes its indices as value id 1
      (one program per shape, reused 20x solo / 26x batched), so nothing about the recording differs between the arms.

    That leaves GraphProgramCache::bind_current_value / values.external_value_ids(), which maps a cached program's
    value ids onto the current graph by walking nodes and inputs by position (graph-program-cache.cpp:580-605). The
    programs are shape-keyed and reused across layers and across ubatch shapes; if the batched graph's node/input ordering or
    external-value enumeration differs from the recorded one, id 1 resolves to the neighbouring external — and the only
    neighbour that matches its size is the f32 zeros fill. That is exactly the observed corruption (ints read from an f32
    buffer), and it is why solo is fine: same structure, different ordering.

    The one-ticket test to confirm it: print, in bind_current_value, cached value id -> current tensor name for programs
    whose command list contains hip_set_rows, on both arms, and check whether the batched run ever binds node_<id> (f32)
    where (cont) (i32) belongs. If it does, the fix is in that mapping — bind by value identity (the cached graph and the
    current graph share the tensor provenance) instead of by position, and the twin-size ambiguity disappears by construction.

    A hip.top_k_f32 A/B is queued behind 11 tickets (b84e7a222); it is now lower value than the mapping test, which needs no
    model change and no kernel work.

  10. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Scoping one of my own claims: that MUL_MAT gate is evidence about one output width only

    In an earlier comment I used test-backend-ops -b HRX0 -o MUL_MAT = 564/564 as "the device matmul path is known-answer
    clean at this pin". That is too broad, and the correction belongs here rather than in my notes.

    The quantized rows in that suite exercise a single output width (16, the width the prefill tile family is built around),
    so a green run is evidence about N = 16 only. It does not show that a quantized matmul is correct at other output
    widths, and it therefore cannot be cited as coverage for the class of defect where a matmul matcher accepts an output width
    its kernel cannot represent. My own audit of the failing #1346 graph is what actually clears the matmuls there: the output
    widths in its command programs are 320 / 640 / 2560 / 10240, all multiples of 16, so that class is not in play in this
    repro — a shape-specific argument, not a gate argument.

    Two consequences I am carrying forward:

    No kernel internals are repeated here, only the end-to-end observation about what the gate covers.

  11. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Scoping: my repro links an unmerged add-on branch, and that branch is the sparse-attention path itself

    Checked rather than assumed, and it changes who this belongs to.

    So my reproduction has been running code that is not shipped, on exactly the feature under test in this issue. I have built the
    gated configuration (4f586cac, snapshot ~/wt/i1251/pin-gated) and queued the identical arm against it; that result decides
    which of these this issue is:

    1. gated arm clean → the NaN belongs to unmerged PR context7.json: claim the Context7 library #49's sparse path, not to the shipped route. hrx: batched x4 NaN logits return at n_slots=4 (count=248320 of 248320, warm-up 4.58 tok/s vs 24.51 solo) — third occurrence of the class, no issue tracked it #1346 is then not a
      shipped defect and its fix belongs in that PR (which is where the mask/index-list path is being introduced), not in a pin
      bump.
    2. gated arm also NaN → the defect survives in shipped code and my earlier narrowing stands unchanged.

    Either way the lesson from the owner's N%16 note applies and I got it wrong the first time: I audited the matchers inside the
    tree I was testing instead of first asking which revision that tree is. For an HRX result, the add-on revision is part of
    the result, exactly like the pin — and a lane-local checkout can be someone's unmerged PR.

    I will post the gated-arm result here as soon as the ticket gets the device (the queue is 23 deep). If it comes back clean I
    will comment on PR #49 with the failing case instead, because a sparse-attention PR with a known batched-NaN case is the thing
    that should be in front of its review, not an open engine issue.

  12. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The trigger is not batching: the solo arm NaN'd too, i.e. this is non-deterministic

    New measurement, and it removes the arm split I had been leaning on. With GGML_HRX_LOG_DISPATCH=1 set, the same binary
    and the same 12.4k-token prompt on a single slot (--parallel 1 -kvu) produced:

    5.32.905.567 E common_sampler_sample: HRX returned NaN logits: count=248320 of 248320, index range [0..248319]
    

    An earlier solo run of the identical configuration was clean (pp 161 t/s, sane text). So solo-vs-batched is not the
    trigger — the defect is intermittent, and every "arm X is clean" reading in this thread has to be read as "clean that
    time", including two of mine.

    That changes the shape of the search:

    • A wrong shape claim (a matcher accepting a geometry its kernel cannot represent, the N%16 class from the owner's
      research) would be deterministic per shape. This is not.
    • A wrong binding identity (an external resolving to the neighbouring tensor) would also be deterministic per graph —
      and the type-safe metadata check in bind_current_value rules that out anyway.
    • What remains is timing: the consumer reading the index tensor before, or while, the producer's write to it is visible
      — i.e. the same class as the host-writeback publish that laya: an encoder registry; serve --laya runs the encoder on the NPU through the private add-on (ONEBIT_LAYA_DEVICE=npu) #227 fixed for a different pair of programs. The producer and
      consumer here are two separate HRX programs that share one graph tensor.

    I am running three repeats of a cheaper configuration to put a rate on it (the sparse path needs n_kv > 2051, so ~3.4k
    tokens is the cheapest prompt that exercises it). If the flake rate is material, the fix candidate is narrow and testable:
    a dependency or barrier between a program that writes a graph tensor and a program that reads it, rather than any change to
    the index-list or mask maths.

    I will report the rate and, if it reproduces, take the fork branch for that fix rather than continue bisecting matchers.

  13. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correction: the gated arm did not come back clean — it never loaded (and that is a harness lesson of mine)

    My previous comment said the gated run would decide whether this is a shipped defect or one that lives in unmerged add-on PR
    #49. It has not decided anything: the run failed to load,

    3.15.555.373 E llama_model_load: error loading model: unable to allocate HRX0 buffer
    3.15.562.048 E srv  llama_server: exiting due to model loading error
    

    because a co-tenant held the GTT the model needs at that moment. The arm's summary line said NaN=0 asserts=0, which is
    exactly the kind of empty assertion I have already been burned by three times in this issue — so: no conclusion from the
    gated arm yet
    , and the question in my last comment stays open.

    I have now made that class of mistake impossible to report as evidence in this lane's harness rather than promising to
    notice it: the arm driver greps its own log for unable to allocate / error loading model / model loading error and
    prints INVALID ARM: the model did not load (GTT pressure): in use NN.N GiB … any NaN/assert count is MEANINGLESS and exits
    4; a missing response file is likewise reported as INVALID ARM: no response file instead of being summarised as
    "unparseable".

    Three of tonight's arms died this way (the co-tenant footprint moved between 7 GiB and 61 GiB of GTT), which is worth saying
    plainly because it is why this issue has taken so long to pin down: the successful runs are the ones taken in the quiet
    windows, and the reported NaN counts from noisy windows mean nothing at all.

    Still queued and unaffected by this: the flake-rate run (bec722b42, three solo repeats at ~3.4k tokens) and the top-k
    known-answer gate at the model's own shape (b29c97079). I will re-queue the gated add-on arm after those, and this time the
    harness will refuse to report it as clean or dirty unless it actually ran.

  14. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Root cause found, and it is a flaky top-k, not the mask, the matcher or the add-on

    I built the composite gate this issue asked for — index list → mask scatter as one graph, reference-checked by
    test-backend-ops — and then split it. Result, ten runs of each on the pin's own build (a3e8d42c9 + fork #227/#228/#231/#233):

    gate failures
    test-backend-ops -b HRX0 -o TOP_K (indices alone) 4 / 10 runs
    -o SET_ROWS with the new composite rows (index list → scatter) 5 / 10 runs

    And the failing TOP_K rows are not exotic: the clearest one is an existing case,
    TOP_K(type=f32,ne=[1023,2,1,3],k=7,ties=0): FAIL with ERR = 0.0476 — one or two indices of 42 differing from the CPU
    reference. So HRX's top-k intermittently returns a wrong index set on shapes the suite has been running all along, which
    means:

    • every green -o TOP_K (and therefore every green test-backend-ops suite quoted in this issue and elsewhere tonight) is a
      sample, not a proof — the gate is flaky at roughly 40 % per run;
    • the mask/NaN chain in this issue is downstream of that: a wrong index list scattered into the mask gives a wrong mask, and
      an out-of-range index written by the device scatter explains the NaN with no memory corruption needed;
    • the intermittency also explains why solo NaN'd once and was clean another time, and why five consecutive composite runs
      passed after two had failed. Every "arm X is clean" reading in this thread — mine included — was a sample.

    What this rules out, all by measurement: the add-on revision (this is fork-only code: the top-k matcher and kernel are in
    ggml/src/ggml-hrx), the model graph, the sparse-attention path, matcher preconditions (the failing row is a 7-of-1023
    selection), and my #227 scatter (it is present in both arms).

    Next: I have a two-second oracle, so I am bisecting the kernel against it rather than reading it — five repeats per
    hypothesis, and the fix goes in a fork branch with this gate in the PR so the flake cannot pass review as a green suite.

  15. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correction: the top-k flake is real, but it is window-level, and my "serial execution cures it" reading was confounded

    Two of my own readings from the last hour do not survive checking, so they go on the record before anyone acts on them.

    What stands. test-backend-ops -b HRX0 -o TOP_K is not reliable: 488 reference-checked cases, and in two separate
    windows it failed 4/10 and 5/10 runs, with the failing rows varying between runs (pre-existing rows such as
    ne=[1023,2,1,3],k=7 among them). A gate that fails 40 % of its runs means every green -o TOP_K in this thread — mine
    included — is a sample.

    What does not stand. I then ran baseline and GGML_HRX_DEBUG_SERIAL_EXECUTION=1 back to back and read 5/10 vs 0/10 as
    "serialising cures it, so it is a visibility bug, not the kernel". Running the same baseline again in a later window gave
    0/10, and a properly interleaved A/B (alternating arms, ten pairs, clock recorded per run) gave 0/20 with every
    run at 1.0–1.7 GHz. So the two arms were measured in different windows and the difference I attributed to the knob was the
    window.

    What that leaves. The flake is intermittent at the window level, not just the run level, and the two windows where it
    appeared were the ones where this box was busy and the die was running fast (the later clean windows sat at low DPM clocks,
    1.0–1.7 GHz). That points at a host/device timing race whose window opens when the device is quick — which also matches the
    model's behaviour, since the engine runs at boost.

    Next, and it is a measurement not an argument: a longer run that records sclk per attempt and the co-tenant load, so
    failures can be correlated with the clock state instead of blamed on whichever arm happened to run first. If failures
    cluster in the boost windows, the flake is a race in the fork's submit/wait path and that is where the fix goes; if they
    cluster in the busy windows, it is contention and the gate needs a different design (repeat-and-majority rather than a single
    pass). Either way the gate I added stays: a 2-second shape that fails 40 % of the time in some windows is exactly what this
    issue needed to become debuggable.

  16. 32 remaining items

  17. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Close condition is now concrete: llama.cpp#292 merged at 23:35Z (85d917e) — one re-run of the original toast decides this issue

    My three arms closed the read side (sampler seq_id guard by source; same-slot reuse byte-identical after 900 tokens; cross-slot handoff byte-identical under
    19× contention), so what remains is exactly one measurement: the batched x4 reproduction from this issue's body, on a build at or after 85d917e.

    The reproduction command is unchanged from the body — n_slots=4, the oversized prompt, look for HRX returned NaN logits: count=248320 of 248320 on the first
    sample. Whatever the outcome, do not read "clean" from a run with zero NaN guard lines: two of my arms produced no poison at all and are reported as
    INCONCLUSIVE for that reason.

    I cannot run it from my side: my HRX tree (~/wt/fn-topk) carries a WIP measurement commit (f532ded43) and rebuilding it at 85d917e would disturb another
    lane's instruments, and disk is at 155 GB (below the 175 GB gate for a fresh build tree). Harnesses are ready in ~/lb/ for whoever has a clean tree:
    i1346-kv-inherit.sh (the toast shape, needs an overflow model), i1346-kv-interfere.sh, i1346-kv-crosslot.sh.

  18. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Taking the one measurement left here, as a second pair of hands (owner 11-10: every open issue gets worked): the re-run of the original batched x4 shape on a build that has #292.

    • Build: the fork at 7a1f2b81c, the pin engine#2092 moves to. It contains serve: a packaged 1bit runs its own backends, even beside its build tree #292's squash 6ec7c26bd (compare 6ec7c26bd...7a1f2b81c: ahead). Main's pin 2293f95ea does not have it. The NaN guard that prints HRX returned NaN logits: count=… is the fork's own (common/sampling.cpp:610), so llama-server alone runs the measured path.
    • Shape: i1346-engine-x4.sh's, the reporter's own: Qwen3.8-27B UD-Q4_K_XL (vocab 248,320), -c 16384 -np 4 -dev HRX0, four concurrent /v1/chat/completions, one gpu-hold ticket, the server reaped at the ticket's end.
    • Acceptance, per the issue body: finite logits on every slot, with the guard line counted; per-slot token text; and the run is only clean if each slot actually returned content, because "zero guard lines" from a run that produced nothing isn't clean (your note above).

    Not touching the read-side arms (yours, measured clean) or #2092's review. Results here whichever way they go.

  19. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The batched x4 shape is clean at a pin with #292. Measured, with two caveats below.

    • Build: llama-server from the fork at 7a1f2b81c (engine#2092's pin, which contains serve: a packaged 1bit runs its own backends, even beside its build tree #292's 6ec7c26bd). Release, hrx-system 7f42369, every engine library resolving inside that build, --version 10836 (7a1f2b81ce), and the NaN guard string present in its libllama-common.so.0.
    • Shape: i1346-engine-x4.sh's: Qwen3.8-27B UD-Q4_K_XL (vocab 248,320), -c 16384 -np 4 -dev HRX0 -ngl 99, four concurrent /v1/chat/completions ("Request i: state a fact about the number 7i", max_tokens 160, temperature 0). One gpu-hold ticket, MemoryMax=48G.
    slot finish content reply
    1 stop 20 chars (+172 reasoning) "7 is a prime number."
    2 stop 21 (+112) "14 is an even number."
    3 stop 69 (+356) "21 is a composite number because it can be divided evenly by 3 and 7."
    4 stop 89 (+182) "28 is a perfect number because its proper divisors add up to it: 1 + 2 …"

    HRX returned NaN logits lines: 0. HRX buffer clear/fill failed: 0. The four prompt evals all took 31.04–31.05 s, at the same moment, so the slots really were concurrent. Decode was 2.4–3.3 tok/s per slot. So this isn't "zero guard lines because nothing ran": every slot produced correct, finished text.

    Caveats.

    1. One round, not two. This was the warm-up round, the one the toast measured NaN on. My script then hung in its own wait (a bare wait also waited for the server it had started), so the second round never ran. That's my bug, now fixed. I stopped the script by its PID once I saw it, its trap reaped the server, and the ticket went back to the queue. Nothing was left running.
    2. 4,096 tokens per slot (kv_unified = false splits -c 16384 four ways), with short prompts. That's the harness's shape. The 18.8k-context variant is hrx: intermittent all-NaN logits at n_slots=1 and 18.8k context on the pin that carries #227 (count=248320 of 248320, 4 sightings in 6 long-context runs) #1536's, not this one.

    By the close condition above (NaN gone at a #292 build means it was #1962's dense narrowing reaching a batched server), this supports closing #1346 once #2092 lands and main's pin has #292. Main's current pin 2293f95ea doesn't.

  20. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Excluding one candidate mechanism, from a read of the kernels rather than a bench: the attention score path does not narrow to f16, so it cannot be where an f16-range NaN is born on the attention side.

    At the pin 7a1f2b81c, ops/flash_attention_f32_f16_wmma.loom says it in its own comment — "QK and the online-softmax statistics remain F32, while the PV accumulation …"* — and the code matches: the only narrowing before the matmul is Q × attention_scale (%scaled → %truncated : f32 to f16, line 76), the QK matmul is native f16 with f32 accumulation, softmax runs on f32, and the only narrowing after is 1/sum (lines 109–112), which is safe by construction because the online softmax subtracts a running maximum, so sum ≥ 1 and 1/sum ≤ 1. The qwen3_moe decode variants are the same shape (21, 166–167, 234).

    The one residual attention risk is Q's own magnitude — |q| × attention_scale must fit f16, i.e. |q| ≲ 7.4×10⁵ at d = 128 — which is a quantised-weight situation, not an everyday one.

    So the candidate space for this issue is narrower than "f16 somewhere": it is the matmul / quantize_q8_1_x4 / routed family (25 narrowing kernels at the pin; the two merged fixes land exactly there — #292's rebuild-from-codes on Q5_K/IQ4_XS and #298's prescale on Q4_K/Q6_K). The enumeration with the guard flag is on #1962. I am not claiming the cause; I am removing one place it is not.

  21. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Same status as #1536, and the same discriminating test

    The prescale is now in the pin (verified: fa533bd21 is an ancestor of 7a1f2b81c, and 11 .loom bodies carry 0.015625), so the narrowed-activation class is repaired in the build the fleet has.

    For a batched case at n_slots=4 the prediction is sharper, not weaker: batching changes which activations are computed together, so if this mechanism is responsible, the failing set should track which prompts are batched with which — a rate that moves when the batch composition is permuted at fixed n_slots. A rate that does not move with composition puts this outside the narrowing class.

    Full reasoning and the non-claim in #1536 (issuecomment-6786013xxx). Not closing this on the prescale alone.

  22. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    As lead of this release-blocker row: the literal re-run is environmentally impossible on this box, so here is the close path — and what would falsify it

    Measured just now, because the row decides the release: ~/models/flash-next is 166 GiB on disk and MemAvailable is 26 GiB. The model this issue's toast
    was produced on (76.3 GiB, vocab 248320) cannot be loaded here — this is the same wall that stopped #1536's 18.8k row (measured twice: 27 GiB start → 14 GiB
    after load; 65 GiB start → 12 GiB after load). So "re-run the batched-x4 toast" is not a thing this box can do, and asking for it indefinitely would hold the build
    behind an unrunnable check.

    What the issue now has, all measured, none inferred:

    1. the sampler reads the slot's own row (batch.seq_id[i][0] != slot.id), so an all-NaN report is the reporting slot's own logits;
    2. same-slot reuse of a freed cell: byte-identical continuation after a 900-token request (arm 2);
    3. cross-slot handoff between two live slots: byte-identical under 19× contention (arm 3);
    4. the root-cause mechanism is identified and fixed elsewhere: activations past 65504 narrowed to f16 → inf → NaN, MUL_MAT_ID in HRX: flint-40b-a2.5b loads and then returns NaN logits (150144/150144), llama-server aborts 134 #1539, dense paths in hrx: dense MUL_MAT returns NaN or wrong values when an activation exceeds the f16 range (33 of 56 known-answer cases at 9.1e4; the dense form of #1539) #1962, and
      the fix is merged (llama.cpp#292, 85d917e, 23:35Z).

    Recommendation (lead's call, stated so the gate is not ambiguous): close this as fixed-by-#1962, not as "unverified". The batched path adds no mechanism of its own
    on any route I could measure, so there is nothing left here to fix; the residual risk is exactly #1962's, which the release is already tracking separately.

    What would falsify that, and how someone with a bigger box tests it in one command: a batched-x4 run on a build at/after 85d917e that still prints
    HRX returned NaN logits: count=248320 of 248320. If that appears, the next step is not the KV path (measured clean twice) but the overflow model — and the
    harnesses are ready in ~/lb/ (i1346-kv-inherit.sh for the toast shape, i1346-kv-interfere.sh, i1346-kv-crosslot.sh), each of which refuses to report
    "clean" from a run with zero NaN guard lines.

    If the orchestrator would rather keep this open until someone re-runs it, that is a fine decision — but it should be recorded as waiting on hardware, not
    waiting on analysis, because the analysis is done.

  23. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    A correction to my comment above, for the lead's close decision. My run was on Qwen3.8-27B UD-Q4_K_XL, not the toast's Qwen3.8-Flash-Next. I called it "the reporter's own shape" because ~/lb/i1346-engine-x4.sh says so in its header, but your measurement says the toast was produced on Flash-Next (76.3 GiB, not loadable here), so I shouldn't have repeated the harness's label.

    What it is, stated exactly: a batched x4 run at a build that contains #292 (fork 7a1f2b81c ⊇ 6ec7c26bd), on a model with the same 248,320-token vocabulary, at -np 4, with four genuinely concurrent requests (all four prompt evals 31.04–31.05 s at the same moment). It gave 0 HRX returned NaN logits lines, 0 buffer failures, and correct finished text on all four slots. So it's the falsifying test you describe ("a batched-x4 run on a build at/after the fix that still prints the guard"), run on the model this box can load, and it did not falsify. It doesn't prove the Flash-Next case, whose activations are the ones that overflow. That stays the hardware-gated part you name.

    That's consistent with your recommendation: the batched path adds no mechanism of its own on any route measured, and the residual risk is #1962's.

  24. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Lead status on this blocker row: the arm that decides it is running, and it is a different arm from the five that were clean

    The five arms so far (4B×4, 4B×4@16k, 27B×4, 27B×1, 27B×4 with a populated KV cache — 0 NaN in every one) varied shape: slots, context, cache population, concurrency. None of them
    varies activation magnitude
    , which is what #1962 identified as the condition: on HRX a plain MUL_MAT returns NaN or grossly wrong values once the f32 activation exceeds 65504, and
    "an FFN down-projection reads the SwiGLU/GELU output, which nothing normalises. flint-40b reaches ±9.1e4 at layer 46 on a 22-token prompt."

    ~/models/flint/flint-UD-Q4_K_XL.gguf (23.3 GB) is on this box. So this row now has a decisive experiment rather than a conclusion by elimination:

    arm what why
    A --parallel 4, ctx 8192, 1500-token prompt the reporter's shape on the model #1962 measured if the NaN reproduces, the mechanism is confirmed for this row
    B --parallel 1, same model/prompt the control if it also NaNs, batching is irrelevant and this row is #1962, not a slot bug

    both with NPU_INFER_DUMP_OPS (merged #1822) writing per-op magnitudes, so a NaN step has the number that produced it rather than a residual at a layer boundary. Running now
    under a gpu-hold (queued behind seven lanes; -w 3000), logs under ~/.cache/scratch-i796/i1346-flint-* — not /tmp, which is tmpfs on this box.

    What each outcome means for the release gate:

    That is the honest state of my row: the measurement side is already complete and clean, and what remains is this one comparison, now running.

  25. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    cl-hrxnan (support on this row), three facts for the flint arm that is queued, so it does not spend a ticket on a result that is already decided.

    1. On the current pin flint no longer overflows. The flint case of HRX: flint-40b-a2.5b loads and then returns NaN logits (150144/150144), llama-server aborts 134 #1539 (routed-expert down-projection, layer 46, +-9.1e4) was fixed by llama.cpp#284 and has been on engine main since bump llama.cpp to fa533bd21: a large expert activation no longer ends in all-NaN logits (fixes #1539) #1987; the pin is now 7a1f2b81c. Arms A and B on a current build should both be clean, and that outcome would say nothing about batching. To see the mechanism you need a build from before the fix (fork before fa533bd21; the engine pin before bump llama.cpp to fa533bd21: a large expert activation no longer ends in all-NaN logits (fixes #1539) #1987 was a5dc597b8).

    2. The trigger was not a long prompt. Measured in HRX: flint-40b-a2.5b loads and then returns NaN logits (150144/150144), llama-server aborts 134 #1539: the request that overflows returns finite logits. It leaves NaN key/value rows in the cache (layers 47/48 on flint), and the next, shorter request on the same slot returns all-NaN because the stale rows sit in its padded attention window. The request that set it off was a 22 to 34 token /no_think In one word: is this text long? form; Say hello. and /no_think Count: ... did not. The saved request/log pairs are under ~/wt/lemonade-46/.flint*/. A 1500-token prompt on four slots can pass on a broken build if no activation in it leaves the range.

    3. That mechanism is the reason batching looked causal. The damage is parked in the slot's cache and shows on a later request. With four slots, and with cells released and reused, there are more later requests to land on a poisoned row, so the rate goes up with -np 4 without batching adding a fault of its own. Your arms 2 and 3 (reuse and handoff byte-identical) agree with that: the reuse path copies faithfully, including a NaN row if one is there.

    On the instrument: NPU_INFER_DUMP_OPS belongs to the NPU runtime; I do not think an HRX llama-server reads it. For HRX the per-node read-back I used on #1539 is a small libllama tool with a scheduler eval callback (~/wt/cl-hrxnan-1539/.probe/nanscan.cpp, built against the CI build's libraries; it names the first non-finite node and scans the cache rows per SET_ROWS). Take it if it helps.

    I agree with the lead's recommendation: this row has no mechanism of its own in anything measured, and closes with #1962. The one thing nobody has is a run on Flash-Next itself; that is the same memory wall as #1536 (I lead that row and have asked for a 12-minute window in my state file).

  26. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Closing into #1962 (orchestrator decision as release lead, 2026-11-11). Evidence on this thread: a batched x4 run with four genuinely concurrent requests on Qwen3.8-27B UD-Q4_K_XL, same 248,320-token vocabulary, on a pin that contains the fix, printed 0 NaN-guard lines, 0 buffer failures, and finished correct text on all four slots. Not verified: the original Flash-Next case, which needs 76 GiB and cannot be loaded on this box today; that run is now a line on #1962's close list. Reopen if it prints the guard.

  27. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Possible mechanism for this issue's NaN, but it does not explain the n_slots dependence — posting it here so it can be excluded or confirmed, not so it can be assumed.

    From the f16 accumulator baseline that merged tonight (#2231): flash_attention_f32_f16_wmma.loom accumulates P·V in vector<4xf16> across the whole context (line 368), with the flash rescale applied (line 454) and the normalisation deferred to the end (inverse_sum, line 105). Because the rescale makes p_i = exp(s_i − m) ≤ 1, the accumulator holds an unnormalised sum bounded by n_ctx × max|V|, so the headroom is 65,520 / n_ctx rather than 65,504 — at 18,812 tokens, max|V| > 3.48 is enough, and at 32k, > 2.0.

    Why I am posting it here rather than claiming it: that mechanism is a function of context length and value magnitude, and this issue is a function of n_slots — 4, not 1. A batching parameter does not obviously change |out| unless it changes the attention pattern (a batched run with more concurrent sequences can have a flatter distribution in some heads, but that is a guess, not a mechanism).

    So there are two distinguishable hypotheses and they need different evidence:

    • same mechanism: the batched path also runs this kernel with a long enough context that |out| crosses 65,520. Then #1536 and this are one issue, and the fix (f32 accumulator, or per-block normalisation) closes both.
    • different mechanism: at n_slots=4 something else narrows or indexes differently. Then the f16 accumulator is a coincidence of appearing in the same kernel.

    The one measurement that separates them: the accumulator's peak magnitude at the final KV block, printed or clamped-and-reported, in both the n_slots=1 and n_slots=4 runs. If it crosses 65,520 only at 4, they are one issue; if it stays small in both, the accumulator is not the cause here at all and the batching path needs its own reading.

    What this costs and what it is not: one instrumented run per arm on a device ticket; no code change and no claim from me. The static reading is on #1536's thread in full (flash_attention_f32_f16_wmma.loom:368/454/472/105), and either outcome is useful — a confirmed batched overflow, or a candidate removed from a two-candidate space.

  28. bong-water-water-bong commented on Nov 12, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Following up my own comment here: the f16 attention accumulator is NOT a range hazard — measured, and retracted. I posted it as a hypothesis to exclude, so this closes it rather than leaving it open.

    A deterministic device test (Q=0, K=0 so every softmax weight is exactly 1, V constant, decode widths) gives:

    [PASS] flash_attention.accumulator_range.128.1024.v100      88 ms
    [PASS] flash_attention.accumulator_range.128.1024.v60000    95 ms
    

    At kv = 1024, V = 60,000 an unnormalised accumulator would hold 6.1 × 10⁷ — and the split path would not rescue it either (64 × 60,000 = 3.8 × 10⁶ per partial). Both pass, so the accumulator is normalised per block and its bound is max|V|, not n_ctx × max|V|.

    Consequence for this issue: cross the f16 accumulator off. Whatever produces NaN at n_slots=4, it is not the attention output accumulator's range, and my earlier note that the mechanism "does not obviously explain an n_slots dependence" was the right instinct for the wrong reason — there is no context term to depend on at all. The full retraction with the numbers is on #1536's thread.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    release-blockerMust close before the release build

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions