[Bug]: KVCacheManagerV2 raises 'History length cannot exceed capacity' in update_resources on a PP stage-2 rank (DSA model, block reuse off)

[PierreLeGuen]

System Info

• GPU: 8x NVIDIA H200 NVL (SM90; two 4-GPU NVLink islands, cross-island over PCIe/UPI), driver 595.71.05
• Host: Ubuntu 24.04.4, 2 TiB RAM, Xeon 6747P
• Container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc25 (CUDA 13.2, Python 3.12.3, torch 2.12.0a0+nv26.5, transformers 5.5.4)
• TensorRT-LLM: 1.3.0rc25 (tag v1.3.0rc25, 785c948); the code paths cited below were also checked on main @ a6616d6 (2026-09-03)
• Backend: PyTorch (trtllm-serve --backend pytorch)

Who can help?

@yizhang-nv (KV cache manager V2 for DSA, #16060)

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Same model, image and traffic as #18660 (GLM-5.3 W4AFP8 on the DeepSeek-V3.2 path, pp_size 2 --tp_size 4 --ep_size 4, MTP draft 1, KV cache manager V2, max_seq_len 262144, enable_chunked_prefill: true, max_num_tokens 8192, 16 requests in flight, prompts ~146k tokens), but with kv_cache_config.enable_block_reuse: false and --max_batch_size 16. After 115 requests the stage-2 rank (rank 4) raised in update_resources:

[TRT-LLM] [E] [_torch][RANK 4] Error in event loop: History length cannot exceed capacity
  File ".../tensorrt_llm/_torch/pyexecutor/py_executor.py", line 3246, in _handle_executed_batch
    self.resource_manager.update_resources(
  File ".../tensorrt_llm/_torch/pyexecutor/resource_manager.py", line 3018, in update_resources
    resource_manager.update_resources(scheduled_batch,
  File ".../tensorrt_llm/_torch/pyexecutor/kv_cache_manager_v2.py", line 3989, in update_resources
    success = kv_cache.resize(new_capacity, history_length)
ValueError: History length cannot exceed capacity
[TRT-LLM] [E] [_torch][RANK 7] Executor loop crashed on this rank; hard-killing all 8 ranks in 9.99947s (peers cannot make progress without this rank). Set TLLM_RANK_CRASH_HARD_KILL_GRACE=-1 to disable.

The request that triggered it was a long chunked-prefill context (146k-169k prompt tokens, max_completion_tokens 16384, non-streaming). Only rank 4 (first rank of pipeline stage 2, which also hosts the MTP layer) raised.

Expected behavior

resize(new_capacity, history_length) is called with a capacity that covers the history, or the executor pauses/rejects the request; either way the engine keeps serving.

actual behavior

ValueError in the executor loop on one rank, followed by the rank-crash hard kill of all ranks; the HTTP frontend stays alive answering 503 (#18663).

additional notes

• Not the SWA min-slot underflow of [Bug]: KVCacheManagerV2: windowed pool min_slots underflow causes scheduler deadlock at high concurrency #15401 / [https://nvbugs/6330273][fix] Reserve KV cache slots for concurrent decode in V2 #15462: this model has no sliding window and the failure is in update_resources, not in capacity planning.
• The same traffic on the V1 manager (use_kv_cache_manager_v2: false) ran 245 requests before a different failure (HangDetector, no Python traceback), so V1 is not a clean workaround either (and V1 loses MTP acceptance, [Bug]: MTP accepts no draft tokens with use_kv_cache_manager_v2=false on DSA models (acceptance length 1.0) #18662).
• Happy to run a candidate fix on the same replay.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

[dundysm]

i'll take this; plan is to fix the KVCacheManagerV2 resize/history contract on PP stage-2 (capacity must cover history_length, or reject/pause instead of throwing in the executor loop) under DSA + block reuse off, with a focused unit/harness around update_resources plus a full 8-gpu replay when i have the box.