Name and Version
b10653 (with some patches to support tensor parallelism on Vulkan, plus #27209 which is needed to make -ts work)
Operating systems
Linux
GGML backends
Vulkan
Hardware
3xIntel Arc Pro B65 + 2xIntel Arc Pro B60
Models
unsloth/Qwen3.8-Flash-Next-GGUF:Q6_K_XL
Problem description & steps to reproduce
Running llama-server (with -sm tensor -fa 1 --tensor-read-lazy --load-mode mmap --override-tensor per_layer_token_embd.weight=CPU -ub 512 -b 8192 -ts 4,4,4,3,3) with unified KV cache works great until there's a second request while the first one is running. As soon as the second request starts processing the first request's decode gets confused and starts recalling incorrectly from previous context.
Claude had this to say when asked why it was breaking:
Mechanism
set_input_qsa (src/llama-memory-hybrid-idx.cpp:383-417) is written per stream, but n_stream = unified ? 1 : n_seq_max (src/llama-kv-cache.cpp:85). Under unified KV it makes one pass over the single cell array holding every conversation's cells, and keys the block map by position:
const llama_pos p = cells.pos_get(j);
blk_of[j] = p/r;
cur_blk_cells[b*r + (p%r)] = j; // every conversation has a token at position p → collision, last writer wins
filled[b]++; // counts foreign cells too
Both conversations occupy positions 0..N, so they collide in every block slot. Each block's pooled indexer key ends up built from whichever conversation wrote last, so conversation A's top-2048 block selection is scored against B's content — A can no longer find its own context. filled[b] inflation also makes incomplete blocks look poolable. The attention mask still hides foreign cells, so there's no leakage of B's text into A — it presents as amnesia and hallucination about A's own history, which is why it reads like a quantization/quality problem.
Nothing forces non-unified for indexer models: cparams.kv_unified is taken straight from params (src/llama-context.cpp:273).
First Bad Commit
Code was added recently in #27742 so presumably that.
Relevant log output
N/A, all logs are trivial.
Name and Version
b10653 (with some patches to support tensor parallelism on Vulkan, plus #27209 which is needed to make
-tswork)Operating systems
Linux
GGML backends
Vulkan
Hardware
3xIntel Arc Pro B65 + 2xIntel Arc Pro B60
Models
unsloth/Qwen3.8-Flash-Next-GGUF:Q6_K_XL
Problem description & steps to reproduce
Running llama-server (with
-sm tensor -fa 1 --tensor-read-lazy --load-mode mmap --override-tensor per_layer_token_embd.weight=CPU -ub 512 -b 8192 -ts 4,4,4,3,3) with unified KV cache works great until there's a second request while the first one is running. As soon as the second request starts processing the first request's decode gets confused and starts recalling incorrectly from previous context.Claude had this to say when asked why it was breaking:
Mechanism
set_input_qsa (src/llama-memory-hybrid-idx.cpp:383-417) is written per stream, but n_stream = unified ? 1 : n_seq_max (src/llama-kv-cache.cpp:85). Under unified KV it makes one pass over the single cell array holding every conversation's cells, and keys the block map by position:
const llama_pos p = cells.pos_get(j);
blk_of[j] = p/r;
cur_blk_cells[b*r + (p%r)] = j; // every conversation has a token at position p → collision, last writer wins
filled[b]++; // counts foreign cells too
Both conversations occupy positions 0..N, so they collide in every block slot. Each block's pooled indexer key ends up built from whichever conversation wrote last, so conversation A's top-2048 block selection is scored against B's content — A can no longer find its own context. filled[b] inflation also makes incomplete blocks look poolable. The attention mask still hides foreign cells, so there's no leakage of B's text into A — it presents as amnesia and hallucination about A's own history, which is why it reads like a quantization/quality problem.
Nothing forces non-unified for indexer models: cparams.kv_unified is taken straight from params (src/llama-context.cpp:273).
First Bad Commit
Code was added recently in #27742 so presumably that.
Relevant log output
N/A, all logs are trivial.