qwen4exp graph builder aborts (ggml_abort) on SM121 under sustained load

[aycibatuhan]

Bug: qwen4exp graph builder aborts (ggml_abort) on DGX Spark (SM121) under sustained load

Model: Qwen3.8-Flash-Next UD-IQ4_XS (unsloth GGUF, 93.7G)
Hardware: DGX Spark (GB10, SM121, 128GB unified)
Build: PR #27742 branch, -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=121 -DGGML_CUDA_FA=ON
Serve: llama-server -m <gguf> -ngl 99 --flash-attn on --ctx-size 32768

Symptom

The server aborts (SIGABRT) during sustained multi-request load — specifically during a tool-calling benchmark battery. It handles single/short requests fine (coherent output, tool calls, structured output, long prompts all work), but aborts deterministically at a specific scenario (~scenario 20 of the battery, reproduced 3×).

Backtrace (from the abort)

#0  __GI___wait4
#1  ggml_print_backtrace ()
#2  ggml_abort ()
#3  llama_model_qwen4exp::graph::graph(llama_model const&, llm_graph_params const&)
#4  llama_model_qwen4exp::build_arch_graph(llm_graph_params const&) const
#5  llama_model::build_graph(llm_graph_params const&) const
#6  llama_context::process_ubatch(...)
#7  llama_context::decode(llama_batch const&)
#8  llama_decode ()
#9  server_context_impl::decode(...)

Analysis

ggml_abort() is called from inside the qwen4exp graph constructor (llama_model_qwen4exp::graph::graph). This is a deterministic abort on a specific batch shape (a token count / batch size the benchmark sends), not a memory-pressure or OOM issue. The graph builder hits an unsupported path and hard-aborts.

Repro

Run a sustained multi-turn tool-calling benchmark against the server (e.g. tool-eval-bench). It aborts at a specific scenario, reproducibly. I could not isolate the exact shape with targeted single requests (tool calls, structured output, long prompts, multi-turn all work individually) — it only triggers under the benchmark's specific batch pattern.

Environment

• DGX Spark (GB10, SM121), 128GB unified
• llama.cpp PR model: add Qwen3.8-Flash-Next (qwen4exp) #27742 branch, build 10656, commit 035e227
• -ngl 99 --flash-attn on --ctx-size 32768

[vladimir-voinea]

Same abort here, same build (035e227, DGX Spark / GB10 SM121), and I can pin the exact assert plus two clean, isolated triggers.

Exact assert (Q4_K_XL, -c 262144 -ngl 48 -kvu --cache-ram -1):

src/models/qwen4exp.cpp:284: GGML_ASSERT(mctx_idx->get_n_kv() == inp->mctx->get_attn()->get_n_kv() && "the indexer cache must track the attention cache cell for cell") failed

So the root cause is the QSA indexer cache (llama_memory_hybrid_idx) not tracking the attention cache cell-for-cell, not a generic batch-shape OOM.

Trigger 1 - deep context: a single request with context > ~2048 tokens (the QSA budget) that then decodes long (thinking mode) aborts. Deep prefill with a short output (max_tokens 8) is fine; the abort fires during extended decode once the sparse path is active.

Trigger 2 - concurrency: two concurrent slots whose sequences have different KV lengths abort identically (unified KV, --parallel 2).

Single short requests never push past the sparse budget, so the server looks stable until real long-context or concurrent load arrives. Happy to attach the full backtrace/log if useful.

[hirparak]

Possible root cause, and a related symptom that doesn't abort.

llama_memory_hybrid_idx (added in #27742) overrides seq_rm, seq_cp,
seq_keep, seq_add, seq_div, clear and state_write/state_read so its
third mem_idx cache follows the attention cache. It has no override for the
attention-only variants:

• seq_rm_attn
• seq_rm_transient
• seq_rm_attn_transient

Those trim only mem_attn in the base class —
llama_memory_hybrid::seq_rm_attn is return mem_attn->seq_rm(seq_id, p0, p1);
(src/llama-memory-hybrid.cpp:162).

The server reuses a cached prompt through exactly that call —
llama_memory_seq_rm_attn(...) at tools/server/server-context.cpp:638. So the
attention cache drops the divergent suffix while the indexer cache keeps its
cells, and the two stop agreeing cell-for-cell. That is precisely the invariant
qwen4exp.cpp asserts on when building the graph, which matches the abort in
llama_model_qwen4exp::graph::graph() reported here.

It would also explain why this reproduces under sustained multi-request load but
not with isolated single requests: single requests never reuse a cached prefix.
Worth checking whether your benchmark has prompt caching / cache reuse active,
and whether it goes away with reuse disabled.

The non-aborting symptom. When the caches diverge in content but not in cell
count, there's no assert — the model just attends against stale indexer cells and
returns wrong answers. Needle-in-haystack over a ~31k document, same question
repeated, temperature 0:

cold      cached=0      "VIOLET-SEVEN-DELTA"   correct
cached 1  cached=31258  "VIOLET"
cached 2  cached=31258  "VSD"
cached 3  cached=31258  "VVD"
cached 4  cached=31258  "VUR"

Varying at temperature 0, so corrupted state rather than sampling. 1/5 correct.
Adding the three overrides — trimming mem_idx after the base call reports
success — gives 5/5 with no other change.

Caveat: measured on a downstream build carrying extra patches, so please confirm
on a clean tree before relying on the numbers. The mechanism is verifiable in the
PR source and I don't believe anything downstream is involved.

(Tried to post this on #27742 directly but comments there are locked.)

[vladimir-voinea]

Fixed. Root cause confirmed and a PR is up: unslothai#135 (targets the qwen4exp branch of PR #27742, since this code isn't on master yet).

Root cause: the QSA indexer cache (llama_memory_hybrid_idx::mem_idx) was a separate llama_kv_cache with its own cell metadata (positions / sequence ownership / used-set). Although init_batch hands it the attention cache's slot infos, the two caches' cell state drifts apart under server slot management (seq_rm on slot release) — the indexer drops cells the attention cache keeps. Because get_n_kv() is driven by used_max_p1, the two caches' n_kv diverge and the graph builder aborts on this exact assert.

Fix: construct the indexer cache with the attention cache as mem_other, so it shares the attention cache's cell metadata (v_cells_impl) via the existing other mechanism. The two caches then agree cell-for-cell by construction and can never drift. The indexer keeps its own K/V tensors, and the attn_rot/n_embd_head it inherits are harmless (the indexer path never uses them).

Verified on DGX Spark / GB10 (Q4_K_XL, -c 262144 -kvu --parallel 2): the assert no longer fires; deep-context 25k–175k generation and 2 concurrent slots with different KV lengths both complete; 32k-token prompts answer coherently.

[scooterblume]

Summary

qwen4exp (Qwen3.8-Flash-Next) hard-aborts whenever a ubatch contains tokens shared by
more than one sequence. This makes all of llama.cpp's multiple-choice evaluations
(HellaSwag / MMLU / ARC via llama-perplexity) unusable for this architecture, and I
believe it is also the root cause of #27780, which was reported as an SM121-specific
crash under multi-request server load.

The assert is in set_input (CPU-side index construction), so this is not
backend- or GPU-specific despite how #27780 was framed.

Deterministic reproduction

llama-perplexity -m <any-qwen4exp.gguf> -ngl 99 \
  --multiple-choice --multiple-choice-tasks 40 \
  -bf arc-challenge-validation.bin

Aborts within ~90s of load, every time. Batch size makes no difference —
reproduced with defaults and with -b 512 -ub 128.

src/models/qwen4exp.cpp:1008: GGML_ASSERT(ubatch->n_seq_id[i] == 1 &&
  "PLE n-gram embeddings do not support tokens shared by multiple sequences") failed

#2  ggml_abort ()
#3  llm_graph_input_ple::set_input(llama_ubatch const*)
#4  llm_graph_result::set_inputs(llama_ubatch const*)
#5  llama_context::process_ubatch(...)
#6  llama_context::decode(llama_batch const&)
#8  decode_helper(...)          libllama-perplexity-impl.so
#9  multiple_choice_score(...)  libllama-perplexity-impl.so

Cause

The two sides are in direct conflict:

• tools/perplexity/perplexity.cpp:889 — "the common prefix is shared among the 4
  sequences to save tokens" — assigns one set of prefix tokens to multiple seq_ids.
• src/models/qwen4exp.cpp:1008 — PLE n-gram rows are looked up via each token's
  predecessors, which are ambiguous when a token belongs to several sequences.

The assert is correct — the lookup genuinely is undefined here, and returning
arbitrary rows would be worse. But it means any caller that shares prefix tokens
across sequences cannot run this architecture.

Likely relationship to #27780

#27780 reports ggml_abort in llama_model_qwen4exp::graph::graph on DGX Spark
(SM121) under "sustained multi-request load", with single requests working fine.
That is consistent with this same assert: llama-server slots sharing a common
system-prompt prefix produce n_seq_id > 1 for the shared tokens, while a single
request never does.

Supporting evidence from my box: running llama-server with -np 1 is completely
stable — 63K-token prefills, long multi-turn sessions, hours of use, no aborts.
The crash only appears once multiple sequences are in play.

If that is the same bug, #27780 is not SM121-specific and the hardware framing there
may be sending people down the wrong path.

What works

Single-sequence paths are unaffected. Standard perplexity completes normally:

llama-perplexity -m <model> -f wikitext-2-raw/wiki.test.raw -c 512 -ngl 99
Final estimate: PPL = 4.7895 +/- 0.02940

Possible fixes

1. Give PLE per-sequence predecessor tracking so shared tokens resolve per seq_id.
2. Have multiple_choice_score / hellaswag_score fall back to duplicating the
   common prefix per sequence when the model reports it cannot share tokens
   (a model capability flag), trading tokens for correctness.
3. At minimum, fail with a clear message naming the unsupported combination rather
   than GGML_ASSERT, so users know evals are unavailable rather than that their
   build is broken.

Environment

llama.cpp   ca3d5a3 (2026-08-28), version 0.3.0-dev
built with GCC 13.3.0, Linux aarch64
cmake -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=121
GPU         NVIDIA GB10 (DGX Spark), compute capability 12.1, driver 580.173.02
CUDA        13.0.88
OS          Ubuntu 24.04.4 LTS, kernel 6.17.0-1031-nvidia
Model       Qwen3.8-Flash-Next, AtomicChat AD-5.00bpw-Q5_K_M-M64 GGUF (102.92 GiB)
Dataset     ikawrakow/validation-datasets-for-llama.cpp