[Bug]: Multimodal metadata failures terminate the PyTorch executor instead of failing the request

[dibyo10]

Summary

Split out from #18849 (suggestion 3), so that the vocab_size propagation fix in #18970 stays a single concern.

When the PyTorch backend cannot build multimodal metadata for a request, the exception escapes into the executor event loop and every rank is hard-killed, taking the whole server down for a full engine start-up (~6 minutes for a large hybrid model). The right behaviour is to fail that one request with a 4xx and keep serving.

Two reported instances

1. Missing KV cache manager state. gen_multimodal_cache_key_tokens() raised a binding TypeError because the hybrid-Mamba KV cache manager was constructed without vocab_size:

   [E] [_torch][RANK 0] Error in event loop: gen_multimodal_cache_key_tokens():
       incompatible function arguments. ...
       Invoked with types: NoneType, bytes, int, kwargs = { token_offset: int }
   [E] Executor loop crashed on this rank; hard-killing all 2 ranks in 10 s
   -> MPI_ABORT -> container exit 137
   

   [#18849][fix] Pass vocab_size to hybrid-Mamba KV cache managers #18970 fixes the root cause and makes this specific failure raise a named RuntimeError instead, but the raise is still fatal to the executor.

2. disable_mm_encoder: true. An image_url request raises "Raw multimodal inputs require a local multimodal encoder" inside the event loop and is likewise fatal for all ranks. This one is purely a misrouted request against a deliberately text-only deployment — a 4xx is unambiguously the correct response, and no configuration change can prevent a client from sending it.

Impact

Any client that can reach the endpoint can terminate the server with a single well-formed-but-unsupported request. The original reporter saw two pods behind a session-affinity router die within two minutes of each other, because the client retried onto the second pod after the first died.

Suggested direction

Classify multimodal metadata/encoder failures as request-level errors in the PyExecutor error path (see tensorrt_llm/_torch/pyexecutor/error_classification.py) so they are returned to the requesting client as a 4xx and the executor loop survives, rather than propagating as fatal.

Context

• Original report: [Bug] Hybrid-Mamba KV cache manager is constructed without vocab_size; the first multimodal request hard-kills every rank #18849 (Qwen3.5 hybrid Gated-DeltaNet + attention, 2x H100, TP2, block reuse + use_kv_cache_manager_v2, vision encoder enabled)
• Root-cause fix for instance 1: [#18849][fix] Pass vocab_size to hybrid-Mamba KV cache managers #18970

I am not the original reporter; filing this so the remaining requirement from #18849 is tracked separately rather than left implicit in the PR description.