[Bug]: KVCacheManagerV2 + DSA: indexer (draft) cache pool is not part of admission; exhaustion under prefix-shared long-context load aborts the engine

[PierreLeGuen]

System Info

• GPU: 8x NVIDIA H200 NVL (SM90; two 4-GPU NVLink islands, cross-island over PCIe/UPI), driver 595.71.05
• Host: Ubuntu 24.04.4, 2 TiB RAM, Xeon 6747P
• Container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc25 (CUDA 13.2, Python 3.12.3, torch 2.12.0a0+nv26.5, transformers 5.5.4)
• TensorRT-LLM: 1.3.0rc25 (tag v1.3.0rc25, 785c948); the code paths cited below were also checked on main @ a6616d6 (2026-09-03)
• Backend: PyTorch (trtllm-serve --backend pytorch)

Who can help?

@yizhang-nv (KV cache manager V2 for DSA, #16060 / #18093)

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Model: GLM-5.3 (GlmMoeDsaForCausalLM -> DeepSeek-V3.2 path, DSA sparse attention), W4AFP8 checkpoint, pp_size 2 --tp_size 4 --ep_size 4, speculative_config: {decoding_type: MTP, max_draft_len: 1}, kv_cache_config: {enable_block_reuse: true, free_gpu_memory_fraction: 0.8, dtype: auto} (KV cache manager V2 auto-selected: Resolved use_kv_cache_manager_v2='auto' to True for GlmMoeDsaForCausalLM), enable_chunked_prefill: true, max_num_tokens 8192, max_seq_len 262144.

Traffic: replay of real agentic conversations at their original arrival pace, 16-32 requests in flight, prompt length p50 ~146k / p95 ~169k tokens, with long shared prefixes between turns of the same conversation (multi-turn tool use). Startup log:

Splitting KV cache max_gpu_total_bytes: total=60.51 GiB, target=... (49968 bytes/token), draft=... (1284 bytes/token)
KVCacheV2Scheduler: tokens_per_block=64, max_num_tokens=8192, max_batch_size=64, draft_mgr=DSACacheManagerV2, ...

After ~180 requests (max_batch_size 64):

[TRT-LLM] [E] [executor][RANK 0] submit request failed: Draft KV cache context resize failed for request 186: could not resize to 102991 tokens
  File ".../tensorrt_llm/_torch/pyexecutor/py_executor.py", line 2686, in _executor_loop_pp
    self.resource_manager.prepare_resources(scheduled_batch)
  File ".../tensorrt_llm/_torch/pyexecutor/kv_cache_manager_v2.py", line 2759, in _prepare_draft_resources
    raise RuntimeError(
RuntimeError: Draft KV cache context resize failed for request 186: could not resize to 102991 tokens
[TRT-LLM] [E] [_torch][RANK 3] HangDetector: propagating hard-kill to all ranks via MPI_Abort.

Same traffic with max_batch_size 8, after 26 requests:

[TRT-LLM] [E] [_torch][RANK 0] Error in event loop: Failed to resume draft KV cache for request 78
RuntimeError: Failed to resume draft KV cache for request 78
[TRT-LLM] [E] [executor][RANK 0] Broadcasting event-loop error to 16 pending request(s): Failed to resume draft KV cache for request 78

Both go away with kv_cache_config.enable_block_reuse: false (which is what the GLM-5 deployment guide ships), at the cost of losing prefix caching entirely.

Expected behavior

Admission accounts for the DSA indexer ("draft") cache so a context request is only scheduled when both pools can hold it, or the indexer mirror reuses blocks like the primary pool; and a failed resize/resume pauses or rejects the request instead of taking the engine down.

actual behavior

The docstring of _prepare_draft_resources says "The main V2 scheduler only manages the primary KV cache manager", and the mirror calls stop_committing() (no block reuse). Both pools are sized to the same token count, but with block reuse the primary pool holds far fewer blocks than the sum of in-flight prompt lengths on prefix-shared traffic, so admission proceeds while the indexer pool is already full; the resize/resume failure is then fatal for the whole engine (HangDetector MPI_Abort). The HTTP frontend keeps running and answers /health with 503 while every request fails (#18663).

additional notes

• [https://nvbugs/6621358][fix] Enable one-model draft KV reuse in cache manager V2 #18093 (one-model draft KV reuse in cache manager V2) looks related because it coordinates target/draft reuse, but the pool that fills here is the DSA indexer K-cache mirror (DSACacheManagerV2, 1,284 B/token), not the speculative draft-model cache; please deduplicate if you consider it the same root cause.
• Reproducible without agentic traffic by sending many long prompts that share a long prefix (so block reuse keeps the primary pool small) at 16+ concurrency.
• Related but different: [Bug]: KVCacheManagerV2: windowed pool min_slots underflow causes scheduler deadlock at high concurrency #15401 / [https://nvbugs/6330273][fix] Reserve KV cache slots for concurrent decode in V2 #15462 concern windowed (SWA) pools; this model has no sliding window.
• Sibling reports from the same runs: [Bug]: KVCacheManagerV2 raises 'History length cannot exceed capacity' in update_resources on a PP stage-2 rank (DSA model, block reuse off) #18661 (V2 History length cannot exceed capacity with reuse off), [Bug]: MTP accepts no draft tokens with use_kv_cache_manager_v2=false on DSA models (acceptance length 1.0) #18662 (MTP acceptance with V1), [Bug]: DSA indexer wk FP8 block scale is dropped at load on the PyTorch backend (DeepSeek-V3.2 / GLM-5 FP8 checkpoints) #18658 (indexer wk scale), [Bug]: MIXED_PRECISION quant config lacks the default kv_b_proj exclusion; GLM-5 (192/256 head dims) crashes in load_kv_b_proj_and_k_b_proj_trans #18659 (MIXED_PRECISION kv_b_proj).

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

[manu-kerrigan]

We reproduced the same fatal draft-cache resume error on a hybrid Gated-DeltaNet/attention model, and now have a synthetic reproducer containing no private prompts. This is a different model/path from the DSA indexer in the original report; the evidence below supports a related admission gap, not an assertion that the pools are identical.

Reproduction package (all reviewed files, with a Download ZIP option): https://gist.github.com/manu-kerrigan/6c4e9a80d6fdea795c70eedf2e8a34c6

Environment and reproducer

TensorRT-LLM 1.3.0rc27, PyTorch backend, Qwen/Qwen3.8-27B (Qwen3_5ForConditionalGeneration), TP=2 on 2× H100 80 GB. MTP (max_draft_len: 3), KV manager v2, block reuse and chunked prefill all enabled. Batch limit 32, token budget 8192, sequence limit 131072, GPU memory fraction 0.85, hybrid snapshots every 2048 tokens.

The attached package includes the full config and a standard-library Python client. It generates invented records, gives each of 12 clients a distinct prefix, warms each prefix, then sends an approximately 81k-token follow-up with a 32768-token output cap. It requires no evaluation fixture or cluster credentials. Run only against an isolated engine: it intentionally creates crash-inducing pressure.

python3 repro_public.py --base-url http://localhost:8000 --records 1420 --concurrency 12

Measured evidence and explanation

Three public synthetic runs reproduced Failed to resume draft KV cache: two earlier runs with client failure detected after 80.1s and 107.24s, plus the final packaged-launcher run. The first used diagnostic instrumentation; the second used the unpatched release image. In the second, both ranks failed on request 53, MPI_ABORT followed, all 12 requests failed, and post-run health failed. The final packaged stock run failed on request 52 on both ranks, followed by MPI_ABORT. An earlier interrupted attempt is excluded because its failure could not be classified.

At the first public failure, the draft attention pool had only 54/22848 blocks available (none evictable), while target attention had 22613/35936 available and target Mamba had 59/71 slots available. Draft IndexMapper still had 22 free slots. The rejected draft cache was empty and inactive. The draft pool was above its 95% resume-utilization ceiling; this was not a demonstrated loss of host-offloaded history.

Code inspection on rc27 found:

• Hybrid _supports_reuse_match_backoff=False disables joint target/draft reuse.
• Scheduler paired admission is gated on joint reuse, leaving this hybrid path without draft capacity admission accounting.
• The draft request view also sizes prefill from a near-full prompt cursor while the target is processing 2048-token chunks.

In a separate diagnostic replay, correcting cursor sizing alone still failed on a later draft resize (69634 tokens requested, only four draft blocks free). Both allocation sizing and admission accounting matter for the tested workload.

Scoped workaround and validation

The enclosed diff combines target-cursor sizing with conservative draft-capacity reservation before admission; requests that do not fit wait, and completion/cancellation releases reservations. It keeps MTP and block reuse enabled and does not force hybrid joint reuse.

Test	Result
Public synthetic, original allocator	Three confirmed fatal failures, including the packaged run below
Public synthetic, patched image	Four passes total: three consecutive earlier passes in 375/393/480 seconds, plus the packaged pass below; 12/12 completions each
Packaged launcher + client, stock vs patched	Stock rc27 failed on request 52; patched image passed 12/12 in 472 seconds, post-health 200
Patched image, 30-minute cancellation soak	2629 completions, 435 cancellations, zero reported errors; post-health 200
Patched sequential public cache probe	79872 cached tokens on identical repeat; 80992 on extended conversation

Captured patched replay/soak logs had no matching fatal draft-cache/event-loop errors. The soak ended with 11 requests still in flight; these are not counted as completions. The included reproducer covers the replay, not the separate soak harness.

An earlier equivalent-runtime-patch comparison on a non-crashing single-client load (10 fixed-seed rounds per variant) measured 85.66 vs 85.86 output tokens/s, identical draft-position acceptance (0.76367/0.56182/0.46257), 2.788 tokens/step, and a 94.1232% final cache-hit gauge. That workload was private and is not attached; it is not a saturated-throughput benchmark of the packaged image.

Limitations: the 12-client public replays report zero cached prompt tokens even with reuse enabled. A short-output 12-client control also reported zero hits; the sequential probe does show reuse. We have not established why concurrency loses the hits, and do not claim the public crash depends on a demonstrated cache hit. The workaround assumes one draft attention pool and beam width 1, reserves worst-case prompt-plus-output capacity, and budgets below the tested 95% resume threshold. It can reduce concurrency and is not a general per-request exception-containment fix.

Code review also found that a request whose reservation alone exceeds the draft budget can remain deferred indefinitely. This was not observed in the measured tests; the workaround has no explicit rejection for that case.

Could upstream admission account for draft capacity independently of support for joint prefix reuse? The enclosed workaround is evidence and a comparison target, not a claim of a general merge-ready implementation. A durable fix should also make recoverable capacity refusal defer or fail one request instead of escaping as an executor-fatal exception.