[Bug]: Responses history trimming hangs on over-capacity input without a completed turn

[zupengwang]

With the default history capacity, storing 65 consecutive user messages can enter a non-yielding loop in ConversationHistoryStore. Harmony histories without a final message have the same failure mode.

System Info

• TensorRT-LLM: main, commit 9964d34d67add61764ff1047bc76992fde376518 (also the current main revision checked before filing).
• Local component validation: macOS arm64, Python 3.12.14, pytest 9.1.1, pytest-asyncio 1.4.0, openai-harmony 0.0.8.
• No TensorRT/CUDA container, model, or GPU was used. This is a source-level component reproduction, not an end-to-end serving result. Native TensorRT-LLM pytest remains unverified.

Information

• The official example scripts
• My own modified scripts: an isolated harness executing the unmodified upstream history-store class.

Tasks

• An officially supported task in the examples folder
• My own task or dataset: Responses API conversation-history capacity handling with synthetic user messages.

Reproduction

The corresponding minimal call in an environment where TensorRT-LLM is importable is:

import asyncio

from tensorrt_llm.serve.responses_utils import ConversationHistoryStore


async def main():
    store = ConversationHistoryStore()
    messages = [
        {"role": "user", "content": f"input {i}"}
        for i in range(store.conversation_capacity + 1)
    ]
    print("enter store_messages: 65 messages; capacity=64", flush=True)
    await store.store_messages("resp_probe", messages, prev_resp_id=None)
    print("returned", flush=True)


asyncio.run(main())

Run this in a separate process with an external timeout. asyncio.wait_for() cannot interrupt the synchronous loop because it never yields.

For the Harmony variant, construct the messages with Message.from_role_and_content("user", f"input {i}") from openai_harmony instead of dictionaries.

Locally, I loaded the exact upstream ConversationHistoryStore definition and its UUID/logging helpers via AST, postponing annotation evaluation and bypassing TensorRT-LLM package initialization. The Harmony checks used real openai_harmony.Message objects. The direct-import snippet above was not executed in that form locally.

Expected behavior

Trimming should terminate when a conversation exceeds capacity, including when no completed turn is available. The incomplete-turn case should have an explicit eviction or rejection policy.

Actual behavior

In the current trimming implementation, scanning 65 user messages leaves start_index == 64 and end_index == 0. Consequently, del conversation[64:1] removes nothing, and _trim_conversation() repeats indefinitely while holding conversations_lock.

Observed with the unmodified class:

• Three consecutive calls to the deletion helper leave the message counts at 65 -> 65 -> 65 -> 65.
• Separate processes executing store_messages() reach the call and exceed a two-second external timeout for both dictionary and Harmony user messages.
• An alternating user/assistant control returns and trims 80 messages to 64.

The ordinary input-conversion path also preserves these consecutive user messages before storage. No HTTP-server hang or GPU result is claimed here.

Additional notes

I have prepared a local fix that distinguishes a missing end index from index zero. When no completed turn exists, it removes the oldest non-system/developer message, retaining recent input and instructions where capacity permits. If only instructions remain above capacity, it removes the oldest instruction. Completed-turn trimming keeps its existing behavior.

The focused test file has 19 cases: the unmodified class produces 7 failed / 12 passed, and the local fix produces 19 passed in the isolated CPU runner. Coverage includes ordinary and Harmony messages, instruction prefixes, capacity boundaries, completed turns, instruction-only histories, and Harmony output without a final channel. Native TensorRT-LLM pytest and upstream CI are still pending.

Related work: merged PR #15043 fixed capacity enforcement after writes and trimming through unmapped response IDs. Its complete-turn tests do not cover this empty-deletion case. I also checked open PR #18497; its inspected patch changes Responses serving paths but does not modify ConversationHistoryStore.

Would the proposed incomplete-turn retention policy be acceptable for a focused PR? I am filing this issue first in accordance with the contribution process.

Before submitting a new issue...

• Searched existing and past issues/PRs and reviewed the Responses API documentation and examples for relevant guidance.