Skip to content

Eval bug: Parallel Runs with Unified KV Cache Cuases Loss of Coherency on qwen4exp #27994

Description

@TheBlueMatt

Name and Version

b10653 (with some patches to support tensor parallelism on Vulkan, plus #27209 which is needed to make -ts work)

Operating systems

Linux

GGML backends

Vulkan

Hardware

3xIntel Arc Pro B65 + 2xIntel Arc Pro B60

Models

unsloth/Qwen3.8-Flash-Next-GGUF:Q6_K_XL

Problem description & steps to reproduce

Running llama-server (with -sm tensor -fa 1 --tensor-read-lazy --load-mode mmap --override-tensor per_layer_token_embd.weight=CPU -ub 512 -b 8192 -ts 4,4,4,3,3) with unified KV cache works great until there's a second request while the first one is running. As soon as the second request starts processing the first request's decode gets confused and starts recalling incorrectly from previous context.

Claude had this to say when asked why it was breaking:

Mechanism

set_input_qsa (src/llama-memory-hybrid-idx.cpp:383-417) is written per stream, but n_stream = unified ? 1 : n_seq_max (src/llama-kv-cache.cpp:85). Under unified KV it makes one pass over the single cell array holding every conversation's cells, and keys the block map by position:

const llama_pos p = cells.pos_get(j);
blk_of[j] = p/r;
cur_blk_cells[b*r + (p%r)] = j; // every conversation has a token at position p → collision, last writer wins
filled[b]++; // counts foreign cells too

Both conversations occupy positions 0..N, so they collide in every block slot. Each block's pooled indexer key ends up built from whichever conversation wrote last, so conversation A's top-2048 block selection is scored against B's content — A can no longer find its own context. filled[b] inflation also makes incomplete blocks look poolable. The attention mask still hides foreign cells, so there's no leakage of B's text into A — it presents as amnesia and hallucination about A's own history, which is why it reads like a quantization/quality problem.

Nothing forces non-unified for indexer models: cparams.kv_unified is taken straight from params (src/llama-context.cpp:273).

First Bad Commit

Code was added recently in #27742 so presumably that.

Relevant log output

N/A, all logs are trivial.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions