From 1485ea61309f70796ada8d1abc09b33e03ca142f Mon Sep 17 00:00:00 2001 From: Faradawn Yang <73060648+faradawn@users.noreply.github.com> Date: Tue, 1 Sep 2026 11:55:43 -0700 Subject: [PATCH 01/10] docs: Add Qwen2.5-VL guide for the TRT-LLM PyTorch backend Add Popular_Models_Guide/Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md, documenting how to serve a multimodal (vision) model on Triton via the TensorRT-LLM PyTorch backend through the llmapi backend. No engine build is required. Also add a deprecation banner to the Llava1.5 TensorRT-LLM guide, whose prebuilt-engine multimodal path is end-of-life as of TensorRT-LLM v1.2. The multimodal image_url input and triton_config.multimodal opt-in used by the new guide are added by NVIDIA/TensorRT-LLM#18381, which is not yet merged; the guide states this prominently. Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com> --- .../Llava1.5/llava_trtllm_guide.md | 7 + .../Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md | 308 ++++++++++++++++++ 2 files changed, 315 insertions(+) create mode 100644 Popular_Models_Guide/Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md diff --git a/Popular_Models_Guide/Llava1.5/llava_trtllm_guide.md b/Popular_Models_Guide/Llava1.5/llava_trtllm_guide.md index a8b8e802..80591ec6 100644 --- a/Popular_Models_Guide/Llava1.5/llava_trtllm_guide.md +++ b/Popular_Models_Guide/Llava1.5/llava_trtllm_guide.md @@ -28,6 +28,13 @@ # Deploying Hugging Face Llava1.5-7b Model in Triton +> [!WARNING] +> **Deprecated.** This guide describes the prebuilt-TensorRT-engine multimodal +> path (`tensorrtllm_backend`'s `all_models/multimodal`), which TensorRT-LLM has +> declared end-of-life as of TensorRT-LLM v1.2. It is no longer maintained. +> Use the TensorRT-LLM PyTorch backend instead; see +> [Deploying Hugging Face Qwen2.5-VL Model in Triton](../Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md). + TensorRT-LLM is Nvidia's recommended solution of running Large Language Models(LLMs) on Nvidia GPUs. Read more about TensoRT-LLM [here](https://github.com/NVIDIA/TensorRT-LLM) and Triton's TensorRT-LLM Backend [here](https://github.com/triton-inference-server/tensorrtllm_backend). diff --git a/Popular_Models_Guide/Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md b/Popular_Models_Guide/Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md new file mode 100644 index 00000000..630e9c5c --- /dev/null +++ b/Popular_Models_Guide/Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md @@ -0,0 +1,308 @@ + + +# Deploying Hugging Face Qwen2.5-VL Model in Triton + +This guide shows how to serve a multimodal (vision-language) model on Triton +Inference Server using the +[TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) PyTorch backend through +the [LLM API](https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/llm-api/README.md), +exposed by Triton's `llmapi` backend. + +> [!IMPORTANT] +> **This workflow depends on an unmerged TensorRT-LLM change.** +> Image support in the Triton `llmapi` backend (the optional `image_url` input +> and the `triton_config.multimodal` opt-in used below) is added by +> [NVIDIA/TensorRT-LLM#18381](https://github.com/NVIDIA/TensorRT-LLM/pull/18381), +> which has not been merged and is not present in any released TensorRT-LLM +> version or container image. Until that PR lands you must build the +> `llmapi` backend files from that branch; a stock container will not accept an +> `image_url` input. + +> [!NOTE] +> This guide replaces +> [the Llava1.5 TensorRT-LLM guide](../Llava1.5/llava_trtllm_guide.md), which +> uses the prebuilt-TensorRT-engine multimodal path that TensorRT-LLM has +> declared end-of-life as of TensorRT-LLM v1.2. See +> [triton-inference-server/server#8945](https://github.com/triton-inference-server/server/issues/8945). + +## Why the PyTorch backend + +The deprecated multimodal path (`tensorrtllm_backend`'s `all_models/multimodal`) +required two ahead-of-time compilation steps before you could serve anything: a +`trtllm-build` invocation to produce the LLM engine, and a separate visual +engine build for the vision encoder. Both artifacts had to be rebuilt whenever +the model, precision, or maximum sequence length changed. + +The PyTorch backend needs **no compilation and no engine build at all**. The +model repository is four plain Python/text files, TensorRT-LLM is a +pip-installed wheel inside the container, and the weights are loaded directly +from a Hugging Face snapshot at startup. This is the single biggest practical +difference between the two workflows. + +LLaVA-1.5 itself is not a drop-in replacement target here. TensorRT-LLM's +[supported models matrix](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/models/supported-models.md) +lists `LlavaNextForConditionalGeneration` and `LlavaLlamaModel` (VILA) among the +supported multimodal architectures, but not `LlavaForConditionalGeneration`, +which is the architecture of `llava-hf/llava-1.5-7b-hf`. This guide therefore +uses [`Qwen/Qwen2.5-VL-3B-Instruct`](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct). + +## What was validated + +| Item | Value | +| ---- | ----- | +| Container | `nvcr.io/nvidia/tritonserver:26.07-trtllm-python-py3` | +| Triton | 2.71.0 | +| TensorRT-LLM | 1.2.1 | +| CUDA | 13.1 | +| Model | `Qwen/Qwen2.5-VL-3B-Instruct` | +| Hardware | 1x NVIDIA B200 | + +## Prerequisites + +### Container + +```bash +docker run --rm -it --gpus all --network host \ + -v ${PWD}:/workspace -w /workspace \ + nvcr.io/nvidia/tritonserver:26.07-trtllm-python-py3 +``` + +### Known issue: the container's `openai` package is too old + +The 26.07 image ships `openai 1.107.3`, which is older than what +`tensorrt_llm/serve/responses_utils.py` requires. Loading a model fails with: + +``` +ImportError: cannot import name 'PartReasoningText' +``` + +Because `tensorrt_llm/_torch/pyexecutor/py_executor.py` imports +`tensorrt_llm.serve`, this breaks loading of **any** model on the `llmapi` +backend, not just multimodal ones. Work around it by installing a newer `openai` +into an overlay directory and putting that directory on `PYTHONPATH`, which +avoids modifying the container's site-packages: + +```bash +pip install --target=/workspace/pylibs -U openai +export PYTHONPATH=/workspace/pylibs +``` + +### Model weights + +Provide either a local Hugging Face snapshot directory or the Hugging Face model +id `Qwen/Qwen2.5-VL-3B-Instruct`. If you use the model id, the container needs +network access to huggingface.co at startup. + +## Preparing the model repository + +Copy the four `llmapi` backend files from TensorRT-LLM's +`triton_backend/all_models/llmapi/tensorrt_llm/` into a model repository: + +``` +model_repo/ +└── tensorrt_llm/ + ├── config.pbtxt + └── 1/ + ├── model.py + ├── helpers.py + └── model.yaml +``` + +Note that the TensorRT-LLM Triton backend sources now live in the +[NVIDIA/TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) repository under +`triton_backend/`; the standalone `tensorrtllm_backend` repository has been +superseded. + +Only `1/model.yaml` needs editing: + +```yaml +model: /path/to/Qwen2.5-VL-3B-Instruct # HF snapshot dir or HF model id +backend: "pytorch" +tensor_parallel_size: 1 +kv_cache_config: + free_gpu_memory_fraction: 0.5 + +triton_config: + max_batch_size: 0 + decoupled: False + multimodal: True # opt-in; default False +``` + +`triton_config.multimodal` defaults to `False`. This is deliberate: existing +deployments that already declare their own `image_url` input keep their current +behavior when they upgrade. The flip side is that if you forget to set it, any +`image_url` values you send are **silently ignored** and you get a text-only +answer, so set it explicitly for multimodal models. + +## Starting the server + +In Slurm/MPI environments, launch through `trtllm-llmapi-launch`: + +```bash +trtllm-llmapi-launch tritonserver --model-repository=/path/to/model_repo \ + --http-port=8000 --grpc-port=8001 --metrics-port=8002 +``` + +Running plain `tritonserver` fails at engine start with: + +``` +mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes +``` + +The LLM API uses `MpiPoolSession` to spawn its workers, and +`trtllm-llmapi-launch` (which sets `TLLM_SPAWN_PROXY_PROCESS=1`) is the +supported wrapper for that. + +Startup takes roughly 70 seconds. Wait for `Started HTTPService` in the log. A +successful multimodal start also logs: + +``` +[trtllm] multimodal input enabled for model_type 'qwen2_5_vl' +``` + +You can poll readiness with: + +```bash +curl -s -o /dev/null -w '%{http_code}' http://localhost:8000/v2/health/ready +``` + +which returns `200` once the server is up. + +## Sending an inference request + +Requests go to the standard Triton HTTP inference endpoint, +`POST /v2/models/tensorrt_llm/infer`. Inputs are Triton tensors, not OpenAI-style +chat JSON: + +| Input | Datatype | Shape | Description | +| ----- | -------- | ----- | ----------- | +| `text_input` | `BYTES` | `[1]` | The plain question. The backend applies the chat template and inserts the per-architecture image placeholders, so do **not** add `<\|vision_start\|>` or similar tokens yourself. | +| `image_url` | `BYTES` | `[N]` | One entry per image. Accepts an `http(s)` URL, a local filesystem path readable by the server, or a `data:image/...;base64,...` URI. | +| `sampling_param_max_tokens` | `INT32` | `[1]` | Maximum number of tokens to generate. | +| `sampling_param_exclude_input_from_output` | `BOOL` | `[1]` | Set to `true`; otherwise the rendered prompt is echoed back in `text_output`. | + +The only output is `text_output`. + +### Python client + +This client uses only the standard library: + +```python +import json +import urllib.request + +URL = "http://localhost:8000/v2/models/tensorrt_llm/infer" + + +def ask(prompt, images, max_tokens=64): + """Send a prompt plus one or more images and return the generated text.""" + body = { + "inputs": [ + {"name": "text_input", "shape": [1], "datatype": "BYTES", + "data": [prompt]}, + {"name": "image_url", "shape": [len(images)], "datatype": "BYTES", + "data": images}, + {"name": "sampling_param_max_tokens", "shape": [1], + "datatype": "INT32", "data": [max_tokens]}, + {"name": "sampling_param_exclude_input_from_output", "shape": [1], + "datatype": "BOOL", "data": [True]}, + ], + "outputs": [{"name": "text_output"}], + } + request = urllib.request.Request( + URL, + data=json.dumps(body).encode(), + headers={"Content-Type": "application/json"}, + ) + with urllib.request.urlopen(request, timeout=300) as response: + result = json.load(response) + return result["outputs"][0]["data"][0].strip() + + +if __name__ == "__main__": + print(ask( + "What color is the bus and what does the sign say?", + ["http://images.cocodataset.org/test2017/000000155781.jpg"], + )) +``` + +Expected output: + +``` +The bus is yellow and white, and the sign on the bus says "Out of Service." +``` + +### Multiple images + +Pass more than one entry in `image_url`; the shape must match the number of +entries: + +```python +ask( + "Describe each image.", + [ + "http://images.cocodataset.org/test2017/000000155781.jpg", + "/workspace/images/second.jpg", + ], +) +``` + +The model enumerates both images and describes each one in the order they were +sent. + +### Image source equivalence + +A `data:image/...;base64,...` URI and a local file path produce the same answer +as the `http` URL for the same image, so you can pick whichever form fits your +deployment. Local paths must be readable by the server process, not the client. + +### Error behavior + +An unreachable image URL surfaces as a Triton error rather than silently +degrading to a text-only answer, for example: + +``` +[trtllm] Error generating request: Cannot connect to host example.invalid:443 +``` + +## Troubleshooting + +| Symptom | Cause and fix | +| ------- | ------------- | +| `mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes` | `tritonserver` was started directly. The LLM API spawns workers via `MpiPoolSession`; start it with `trtllm-llmapi-launch` instead. | +| `ImportError: cannot import name 'PartReasoningText'` | The container's `openai` package is too old for `tensorrt_llm.serve`, which is imported unconditionally by the PyTorch executor. Install a newer `openai` into an overlay directory and export it on `PYTHONPATH` (see [Prerequisites](#known-issue-the-containers-openai-package-is-too-old)). | +| `ConnectionRefusedError` from the client | The server is not up yet. Startup takes roughly 70 seconds; wait for `Started HTTPService` in the log, or poll until `curl -s -o /dev/null -w '%{http_code}' http://localhost:8000/v2/health/ready` returns `200`. | +| Images appear to be ignored and answers are text-only | `triton_config.multimodal` is not set to `True` in `1/model.yaml`. It defaults to `False` and image inputs are silently dropped. | + +## References + +- [TensorRT-LLM LLM API](https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/llm-api/README.md) +- [TensorRT-LLM supported models](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/models/supported-models.md) +- [NVIDIA/TensorRT-LLM#18381](https://github.com/NVIDIA/TensorRT-LLM/pull/18381) - adds multimodal input to the Triton `llmapi` backend +- [Qwen2.5-VL-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct) From d1482ffa32fba44c8d32da6d345e98d30f73d2ec Mon Sep 17 00:00:00 2001 From: Faradawn Yang <73060648+faradawn@users.noreply.github.com> Date: Tue, 1 Sep 2026 16:40:11 -0700 Subject: [PATCH 02/10] docs: simplify the Qwen2.5-VL setup steps all_models/llmapi/ holds exactly one model directory, so it can be used as a Triton model repository directly instead of copying files into a new one. Give the concrete clone and model.yaml commands, and add a curl example with the response it returns. Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com> --- .../Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md | 63 +++++++++++++++---- 1 file changed, 51 insertions(+), 12 deletions(-) diff --git a/Popular_Models_Guide/Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md b/Popular_Models_Guide/Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md index 630e9c5c..5dd71d20 100644 --- a/Popular_Models_Guide/Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md +++ b/Popular_Models_Guide/Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md @@ -121,11 +121,18 @@ network access to huggingface.co at startup. ## Preparing the model repository -Copy the four `llmapi` backend files from TensorRT-LLM's -`triton_backend/all_models/llmapi/tensorrt_llm/` into a model repository: +The Triton backend sources live in the +[NVIDIA/TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) repository under +`triton_backend/`; the standalone `tensorrtllm_backend` repository has been +superseded. They are plain Python files and are not shipped in the +`tensorrt_llm` wheel, so fetch them from a checkout. + +`triton_backend/all_models/llmapi/` contains exactly one model directory +(`tensorrt_llm/`), so it doubles as a Triton model repository and needs no +copying: ``` -model_repo/ +all_models/llmapi/ <- point --model-repository here └── tensorrt_llm/ ├── config.pbtxt └── 1/ @@ -134,15 +141,24 @@ model_repo/ └── model.yaml ``` -Note that the TensorRT-LLM Triton backend sources now live in the -[NVIDIA/TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) repository under -`triton_backend/`; the standalone `tensorrtllm_backend` repository has been -superseded. +> [!NOTE] +> Until [NVIDIA/TensorRT-LLM#18381](https://github.com/NVIDIA/TensorRT-LLM/pull/18381) +> merges, the `image_url` input and the `triton_config.multimodal` option below +> exist only on that pull request's branch. Clone the fork shown here for now; +> once it lands, clone `https://github.com/NVIDIA/TensorRT-LLM.git` instead. + +```bash +git clone --depth 1 --branch feat/triton-llmapi-multimodal-image \ + https://github.com/faradawn/TensorRT-LLM.git /workspace/trtllm-pr +``` -Only `1/model.yaml` needs editing: +Then point `1/model.yaml` at the model and turn on the multimodal opt-in. This +edits the file in place inside the checkout, which leaves that clone's +`git status` dirty — fine for a throwaway container: -```yaml -model: /path/to/Qwen2.5-VL-3B-Instruct # HF snapshot dir or HF model id +```bash +cat > /workspace/trtllm-pr/triton_backend/all_models/llmapi/tensorrt_llm/1/model.yaml <<'EOF' +model: Qwen/Qwen2.5-VL-3B-Instruct backend: "pytorch" tensor_parallel_size: 1 kv_cache_config: @@ -151,9 +167,13 @@ kv_cache_config: triton_config: max_batch_size: 0 decoupled: False - multimodal: True # opt-in; default False + multimodal: True +EOF ``` +`model` accepts a Hugging Face model id (downloaded to `HF_HOME`) or a local +snapshot directory. + `triton_config.multimodal` defaults to `False`. This is deliberate: existing deployments that already declare their own `image_url` input keep their current behavior when they upgrade. The flip side is that if you forget to set it, any @@ -165,7 +185,8 @@ answer, so set it explicitly for multimodal models. In Slurm/MPI environments, launch through `trtllm-llmapi-launch`: ```bash -trtllm-llmapi-launch tritonserver --model-repository=/path/to/model_repo \ +trtllm-llmapi-launch tritonserver \ + --model-repository=/workspace/trtllm-pr/triton_backend/all_models/llmapi \ --http-port=8000 --grpc-port=8001 --metrics-port=8002 ``` @@ -209,6 +230,24 @@ chat JSON: The only output is `text_output`. +### Quick check with `curl` + +```bash +curl -s http://localhost:8000/v2/models/tensorrt_llm/infer -H 'Content-Type: application/json' -d '{ + "inputs": [ + {"name":"text_input","shape":[1],"datatype":"BYTES","data":["What color is the bus and what does the sign say?"]}, + {"name":"image_url","shape":[1],"datatype":"BYTES","data":["http://images.cocodataset.org/test2017/000000155781.jpg"]}, + {"name":"sampling_param_max_tokens","shape":[1],"datatype":"INT32","data":[64]}, + {"name":"sampling_param_exclude_input_from_output","shape":[1],"datatype":"BOOL","data":[true]} + ], + "outputs": [{"name":"text_output"}] +}' +``` + +```json +{"model_name":"tensorrt_llm","model_version":"1","outputs":[{"name":"text_output","datatype":"BYTES","shape":[1],"data":["The bus is yellow and white, and the sign on the bus says \"Out of Service.\""]}]} +``` + ### Python client This client uses only the standard library: From 56b4521340f381804789e4770372b09ab562f481 Mon Sep 17 00:00:00 2001 From: Faradawn Yang <73060648+faradawn@users.noreply.github.com> Date: Tue, 8 Sep 2026 12:16:31 -0700 Subject: [PATCH 03/10] docs: state the allowed scope of image_url access The backend now accepts only http(s) URLs and inline data URIs; local filesystem paths and file:// are rejected because the input is client-controlled. Document that and drop the local-path example. Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com> --- .../Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md | 26 ++++++++++++++++--- 1 file changed, 22 insertions(+), 4 deletions(-) diff --git a/Popular_Models_Guide/Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md b/Popular_Models_Guide/Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md index 5dd71d20..a4c7a6b8 100644 --- a/Popular_Models_Guide/Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md +++ b/Popular_Models_Guide/Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md @@ -224,7 +224,7 @@ chat JSON: | Input | Datatype | Shape | Description | | ----- | -------- | ----- | ----------- | | `text_input` | `BYTES` | `[1]` | The plain question. The backend applies the chat template and inserts the per-architecture image placeholders, so do **not** add `<\|vision_start\|>` or similar tokens yourself. | -| `image_url` | `BYTES` | `[N]` | One entry per image. Accepts an `http(s)` URL, a local filesystem path readable by the server, or a `data:image/...;base64,...` URI. | +| `image_url` | `BYTES` | `[N]` | One entry per image. Accepts an `http(s)` URL the server can reach, or an inline `data:image/...;base64,...` URI. See [Allowed scope of access](#allowed-scope-of-access). | | `sampling_param_max_tokens` | `INT32` | `[1]` | Maximum number of tokens to generate. | | `sampling_param_exclude_input_from_output` | `BOOL` | `[1]` | Set to `true`; otherwise the rendered prompt is echoed back in `text_output`. | @@ -317,9 +317,27 @@ sent. ### Image source equivalence -A `data:image/...;base64,...` URI and a local file path produce the same answer -as the `http` URL for the same image, so you can pick whichever form fits your -deployment. Local paths must be readable by the server process, not the client. +A `data:image/...;base64,...` URI produces the same answer as the `http` URL for +the same image, so you can pick whichever form fits your deployment. Inline data +avoids a second network hop at the cost of a larger request body. + +### Allowed scope of access + +`image_url` is client-controlled, so the backend accepts only: + +- `http://` and `https://` URLs the server can reach +- inline `data:image/...;base64,...` URIs + +Local filesystem paths and `file://` URLs are **rejected**. Accepting them would +let any caller make the server open image files its process can read, which on a +client-accessible deployment is an arbitrary-file-read primitive. A rejected +value fails the request with an error naming the offending entry; it does not +silently fall back to a text-only answer. + +If your deployment needs to serve images that already live on the server, agree +an explicit allowlisted root with whoever owns the deployment before widening +this. Note this is deliberately narrower than `trtllm-serve`, whose media +loading is unrestricted. ### Error behavior From 8d27cdfd38b2123931dffa9ef21f3651cf640d9b Mon Sep 17 00:00:00 2001 From: Faradawn Yang <73060648+faradawn@users.noreply.github.com> Date: Tue, 8 Sep 2026 12:33:19 -0700 Subject: [PATCH 04/10] docs: note that image_url accepts web URLs only The backend rejects local paths, file:// and data: URIs, so document the http(s)-only scope and drop the data-URI examples. Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com> --- .../Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md | 27 ++++--------------- 1 file changed, 5 insertions(+), 22 deletions(-) diff --git a/Popular_Models_Guide/Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md b/Popular_Models_Guide/Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md index a4c7a6b8..945faee5 100644 --- a/Popular_Models_Guide/Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md +++ b/Popular_Models_Guide/Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md @@ -224,7 +224,7 @@ chat JSON: | Input | Datatype | Shape | Description | | ----- | -------- | ----- | ----------- | | `text_input` | `BYTES` | `[1]` | The plain question. The backend applies the chat template and inserts the per-architecture image placeholders, so do **not** add `<\|vision_start\|>` or similar tokens yourself. | -| `image_url` | `BYTES` | `[N]` | One entry per image. Accepts an `http(s)` URL the server can reach, or an inline `data:image/...;base64,...` URI. See [Allowed scope of access](#allowed-scope-of-access). | +| `image_url` | `BYTES` | `[N]` | One entry per image. Accepts `http(s)` URLs the server can reach; local paths and other schemes are rejected. | | `sampling_param_max_tokens` | `INT32` | `[1]` | Maximum number of tokens to generate. | | `sampling_param_exclude_input_from_output` | `BOOL` | `[1]` | Set to `true`; otherwise the rendered prompt is echoed back in `text_output`. | @@ -315,29 +315,12 @@ ask( The model enumerates both images and describes each one in the order they were sent. -### Image source equivalence - -A `data:image/...;base64,...` URI produces the same answer as the `http` URL for -the same image, so you can pick whichever form fits your deployment. Inline data -avoids a second network hop at the cost of a larger request body. - ### Allowed scope of access -`image_url` is client-controlled, so the backend accepts only: - -- `http://` and `https://` URLs the server can reach -- inline `data:image/...;base64,...` URIs - -Local filesystem paths and `file://` URLs are **rejected**. Accepting them would -let any caller make the server open image files its process can read, which on a -client-accessible deployment is an arbitrary-file-read primitive. A rejected -value fails the request with an error naming the offending entry; it does not -silently fall back to a text-only answer. - -If your deployment needs to serve images that already live on the server, agree -an explicit allowlisted root with whoever owns the deployment before widening -this. Note this is deliberately narrower than `trtllm-serve`, whose media -loading is unrestricted. +`image_url` is client-controlled, so only `http(s)` URLs are accepted. Local +filesystem paths, `file://` and other schemes are rejected, because accepting +them would let a caller make the server read image files its process can open. +Host images the model should see on a reachable web URL. ### Error behavior From 51f26138fb785c5ecdcd063ceb3099fd56605a7f Mon Sep 17 00:00:00 2001 From: Faradawn Yang <73060648+faradawn@users.noreply.github.com> Date: Wed, 9 Sep 2026 22:07:12 -0700 Subject: [PATCH 05/10] docs: make the Qwen2.5-VL guide work on the 1.2.1 container The guide claimed validation on TensorRT-LLM 1.2.1, but following it verbatim on nvcr.io/nvidia/tritonserver:26.07-trtllm-python-py3 -- the newest published -trtllm-python-py3 tag -- fails every request that carries an image: Error generating request: cannot import name 'async_build_multimodal_prompt' from 'tensorrt_llm.inputs' The backend files readers clone call `async_build_multimodal_prompt`, which NVIDIA/TensorRT-LLM#18381 adds to `tensorrt_llm/inputs/utils.py`. That module ships inside the wheel, not in the `triton_backend/` tree, so on a 1.2.1 container the caller is present and the callee never is. The server still starts, still logs `multimodal input enabled`, and still answers text-only prompts, so the deployment looks healthy right up until the first image. Add a "Patching model.py for TensorRT-LLM 1.2.1" section carrying the replacement method and the call-site diff, gated behind a note to skip it once a container ships with #18381 in it. Say in the validation table that 1.2.1 needs that patch, and record the torch build. Also fix three things found while testing: - The multiple-images example passed `/workspace/images/second.jpg`, a local path the guide's own "Allowed scope of access" section says is rejected. That example could only ever error. Use two live http URLs and show the real two-image answer. - Show the actual rejection and connection-failure responses, including the `ssl:default [Name or service not known]` tail that was trimmed, as JSON response bodies rather than log lines. - Add troubleshooting rows for the ImportError and for a rejected scheme. Verified on 1x B200 with Qwen/Qwen2.5-VL-3B-Instruct: single image returns the answer this guide quotes, two images are described in order, and the rejection and unreachable-host paths surface as errors rather than silently degrading to text-only. Checked by applying the patch text extracted from this file to a fresh clone, so what is documented is what was run. Co-Authored-By: Claude Opus 5 --- .../Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md | 152 ++++++++++++++++-- 1 file changed, 142 insertions(+), 10 deletions(-) diff --git a/Popular_Models_Guide/Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md b/Popular_Models_Guide/Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md index 945faee5..dec8f5c6 100644 --- a/Popular_Models_Guide/Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md +++ b/Popular_Models_Guide/Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md @@ -35,14 +35,23 @@ the [LLM API](https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/llm-api/ exposed by Triton's `llmapi` backend. > [!IMPORTANT] -> **This workflow depends on an unmerged TensorRT-LLM change.** +> **This workflow depends on an unmerged TensorRT-LLM change, plus a small +> patch to run it on today's container.** +> > Image support in the Triton `llmapi` backend (the optional `image_url` input > and the `triton_config.multimodal` opt-in used below) is added by > [NVIDIA/TensorRT-LLM#18381](https://github.com/NVIDIA/TensorRT-LLM/pull/18381), > which has not been merged and is not present in any released TensorRT-LLM -> version or container image. Until that PR lands you must build the -> `llmapi` backend files from that branch; a stock container will not accept an +> version or container image. Until that PR lands you must take the `llmapi` +> backend files from that branch; a stock container will not accept an > `image_url` input. +> +> Those files call `async_build_multimodal_prompt`, which the same PR adds to +> the `tensorrt_llm` **wheel**. The newest published container still ships +> TensorRT-LLM 1.2.1, whose wheel does not have it, so +> [one edit to `model.py`](#patching-modelpy-for-tensorrt-llm-121) is required +> as well. Both steps go away once #18381 merges and a container ships with +> that build of TensorRT-LLM. > [!NOTE] > This guide replaces @@ -78,11 +87,16 @@ uses [`Qwen/Qwen2.5-VL-3B-Instruct`](https://huggingface.co/Qwen/Qwen2.5-VL-3B-I | ---- | ----- | | Container | `nvcr.io/nvidia/tritonserver:26.07-trtllm-python-py3` | | Triton | 2.71.0 | -| TensorRT-LLM | 1.2.1 | +| TensorRT-LLM | 1.2.1 (with the [`model.py` patch](#patching-modelpy-for-tensorrt-llm-121)) | +| torch | 2.10.0a0+b4e4ee81d3.nv25.12 | | CUDA | 13.1 | | Model | `Qwen/Qwen2.5-VL-3B-Instruct` | | Hardware | 1x NVIDIA B200 | +Every command and every response below was run on that configuration. `26.07` +is the newest `-trtllm-python-py3` tag; on it, the multimodal path does not work +without the patch. + ## Prerequisites ### Container @@ -180,6 +194,108 @@ behavior when they upgrade. The flip side is that if you forget to set it, any `image_url` values you send are **silently ignored** and you get a text-only answer, so set it explicitly for multimodal models. +## Patching `model.py` for TensorRT-LLM 1.2.1 + +> [!NOTE] +> Skip this section entirely once a `-trtllm-python-py3` container ships with a +> TensorRT-LLM build that includes +> [#18381](https://github.com/NVIDIA/TensorRT-LLM/pull/18381). It is a bridge for +> today's image, not part of the design. + +The files you just cloned build the prompt by calling +`async_build_multimodal_prompt`, which #18381 adds to +`tensorrt_llm/inputs/utils.py`. That module ships **inside the `tensorrt_llm` +wheel**, not in the `triton_backend/` tree you cloned, so on a 1.2.1 container +you have the caller but never the callee. The server still starts, still reports +`multimodal input enabled`, and still answers text-only prompts — and then fails +on every request that carries an image: + +```json +{"error":"Error generating request: cannot import name 'async_build_multimodal_prompt' from 'tensorrt_llm.inputs' (/opt/venv-tritonserver/lib/python3.12/site-packages/tensorrt_llm/inputs/__init__.py)"} +``` + +1.2.1 also lacks everything that helper is built on — `MEDIA_IO_REGISTRY`, +`ContentFormat`, `MultimodalDataTracker.item_order()`, +`interleave_mm_placeholders` and `async_apply_chat_template` — so you cannot +copy the new `utils.py` across either. What does work is replacing the single +call with an equivalent written against the 1.2.1 API. Add this method to the +`TritonPythonModel` class in +`/workspace/trtllm-pr/triton_backend/all_models/llmapi/tensorrt_llm/1/model.py`, +just above `async def _convert_request`: + +```python + async def _build_multimodal_prompt_121(self, text, media): + """Stand-in for `inputs.async_build_multimodal_prompt` on TRT-LLM 1.2.1. + + 1.2.1 has no `async_apply_chat_template` and no + `MultimodalDataTracker.item_order()`, and its + `add_multimodal_placeholders` takes three arguments rather than four. + """ + from tensorrt_llm.inputs import prompt_inputs + from tensorrt_llm.inputs.utils import (ConversationMessage, + MultimodalDataTracker, + add_multimodal_placeholders, + apply_chat_template, + async_load_image) + + mm_data_tracker = MultimodalDataTracker(self._mm_model_type) + for url in media: + mm_data_tracker.add_data("image", async_load_image(url)) + mm_placeholder_counts = mm_data_tracker.placeholder_counts() + + content = add_multimodal_placeholders(self._mm_model_type, text, + mm_placeholder_counts) + conversation = [ + ConversationMessage(role="user", content=content, media=[]) + ] + prompt_task = asyncio.to_thread( + apply_chat_template, + model_type=self._mm_model_type, + tokenizer=self._mm_tokenizer, + processor=self._mm_processor, + conversation=conversation, + add_generation_prompt=True, + mm_placeholder_counts=[mm_placeholder_counts], + ) + prompt, (mm_data, _) = await asyncio.gather( + prompt_task, mm_data_tracker.retrieve_all_async()) + + prompt = prompt_inputs(prompt) + if mm_data: + prompt["multi_modal_data"] = mm_data + return prompt +``` + +`apply_chat_template` is synchronous and does real tokenizer work, so it goes +through `asyncio.to_thread` rather than blocking the engine's event loop while +the images are still downloading. + +Then, in `_convert_request`, point the call at it: + +```diff + image_url = get_input_tensor_by_name(request, 'image_url') + if image_url is not None and image_url.size > 0: +- from tensorrt_llm.inputs import async_build_multimodal_prompt +- + media = [ + url.decode("utf-8") if isinstance(url, bytes) else str(url) + for url in image_url.reshape(-1) + ] + validate_media_urls(media) +- prompt = await async_build_multimodal_prompt( +- model_type=self._mm_model_type, +- tokenizer=self._mm_tokenizer, +- processor=self._mm_processor, +- prompt=prompt, +- media=media, +- modality="image", +- ) ++ prompt = await self._build_multimodal_prompt_121(prompt, media) +``` + +Nothing else in the backend needs touching: `validate_media_urls` and the rest +of the request path run unmodified on 1.2.1. + ## Starting the server In Slurm/MPI environments, launch through `trtllm-llmapi-launch`: @@ -300,20 +416,28 @@ The bus is yellow and white, and the sign on the bus says "Out of Service." ### Multiple images Pass more than one entry in `image_url`; the shape must match the number of -entries: +entries. Every entry must be an `http(s)` URL — see +[Allowed scope of access](#allowed-scope-of-access): ```python ask( "Describe each image.", [ "http://images.cocodataset.org/test2017/000000155781.jpg", - "/workspace/images/second.jpg", + "http://images.cocodataset.org/val2017/000000039769.jpg", ], + max_tokens=96, ) ``` The model enumerates both images and describes each one in the order they were -sent. +sent: + +``` +The first image depicts a bus on a foggy street at night. The bus has a sign on +its front that reads "OUT OF SERVICE." ... The second image shows two cats lying +on a pink couch. +``` ### Allowed scope of access @@ -322,13 +446,19 @@ filesystem paths, `file://` and other schemes are rejected, because accepting them would let a caller make the server read image files its process can open. Host images the model should see on a reachable web URL. +A rejected entry fails the whole request: + +```json +{"error":"Error generating request: Unsupported image_url '/workspace/images/second.jpg': only http, https URLs are accepted."} +``` + ### Error behavior An unreachable image URL surfaces as a Triton error rather than silently -degrading to a text-only answer, for example: +degrading to a text-only answer: -``` -[trtllm] Error generating request: Cannot connect to host example.invalid:443 +```json +{"error":"Error generating request: Cannot connect to host example.invalid:443 ssl:default [Name or service not known]"} ``` ## Troubleshooting @@ -339,6 +469,8 @@ degrading to a text-only answer, for example: | `ImportError: cannot import name 'PartReasoningText'` | The container's `openai` package is too old for `tensorrt_llm.serve`, which is imported unconditionally by the PyTorch executor. Install a newer `openai` into an overlay directory and export it on `PYTHONPATH` (see [Prerequisites](#known-issue-the-containers-openai-package-is-too-old)). | | `ConnectionRefusedError` from the client | The server is not up yet. Startup takes roughly 70 seconds; wait for `Started HTTPService` in the log, or poll until `curl -s -o /dev/null -w '%{http_code}' http://localhost:8000/v2/health/ready` returns `200`. | | Images appear to be ignored and answers are text-only | `triton_config.multimodal` is not set to `True` in `1/model.yaml`. It defaults to `False` and image inputs are silently dropped. | +| `cannot import name 'async_build_multimodal_prompt' from 'tensorrt_llm.inputs'`, only on requests carrying an image | The container's TensorRT-LLM wheel predates [#18381](https://github.com/NVIDIA/TensorRT-LLM/pull/18381). The server starts and text-only requests still work, which makes this easy to miss. Apply [the 1.2.1 patch](#patching-modelpy-for-tensorrt-llm-121). | +| `Unsupported image_url '...': only http, https URLs are accepted.` | A local path, `file://` or other scheme was passed. Only `http(s)` is accepted; see [Allowed scope of access](#allowed-scope-of-access). | ## References From 9181f35a3f52b9e9b7900cc797be732bdd53bb71 Mon Sep 17 00:00:00 2001 From: Faradawn Yang <73060648+faradawn@users.noreply.github.com> Date: Wed, 9 Sep 2026 22:19:10 -0700 Subject: [PATCH 06/10] docs: ship the 1.2.1 patch as a script, and stop cloning 900 MB for four files Follow-up to the previous commit, which described the model.py change but left the reader to make it by hand. model.py is ~790 lines, and splicing a 40-line method into the right class at the right indentation is a reliable way to end up with a broken model repository. Ship the edit as trtllm_121_compat.py next to the guide instead, invoked in one command. This is how this directory already works: the Llava1.5 guide being deprecated here ships multi_modal_client.py beside it, and Llama2 ships deploy_trtllm_llama.sh. The script is defensive, because it edits a file the reader did not write: it is idempotent, it refuses to write source that does not parse, and if the call it targets is absent it explains that #18381 has probably merged and gives the one-line import check to confirm, rather than corrupting the checkout. The guide keeps the explanation -- the call-site diff and a table mapping each 1.3 API onto the 1.2.1 equivalent -- so a reader can still see what changes and why, without having to type it. Separately, replace the full clone with a blobless, LFS-skipped sparse checkout: GIT_LFS_SKIP_SMUDGE=1 git clone --depth 1 --filter=blob:none --sparse ... git -C ... sparse-checkout set triton_backend/all_models/llmapi Four seconds and 9.1 MB, against roughly 900 MB of Git LFS payload for four small text files. Also state plainly that 1.2.1 is the standing target: every published -trtllm-python-py3 image ships it, so #18381 merging upstream does not remove the need for this patch. Only a new container image does. Verified end to end on 1x B200 by running the guide's own commands, including the shipped script against a fresh sparse clone with nothing hand-edited: single image returns the quoted answer, two images are described in order, a rejected scheme and an unreachable host both surface as errors. Co-Authored-By: Claude Opus 5 --- .../Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md | 142 +++++++------- .../Qwen2.5-VL/trtllm_121_compat.py | 181 ++++++++++++++++++ 2 files changed, 252 insertions(+), 71 deletions(-) create mode 100644 Popular_Models_Guide/Qwen2.5-VL/trtllm_121_compat.py diff --git a/Popular_Models_Guide/Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md b/Popular_Models_Guide/Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md index dec8f5c6..13d46718 100644 --- a/Popular_Models_Guide/Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md +++ b/Popular_Models_Guide/Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md @@ -47,11 +47,12 @@ exposed by Triton's `llmapi` backend. > `image_url` input. > > Those files call `async_build_multimodal_prompt`, which the same PR adds to -> the `tensorrt_llm` **wheel**. The newest published container still ships -> TensorRT-LLM 1.2.1, whose wheel does not have it, so -> [one edit to `model.py`](#patching-modelpy-for-tensorrt-llm-121) is required -> as well. Both steps go away once #18381 merges and a container ships with -> that build of TensorRT-LLM. +> the `tensorrt_llm` **wheel**. Every published `-trtllm-python-py3` image still +> ships TensorRT-LLM 1.2.1, whose wheel does not have it, so +> [a one-command patch](#patching-modelpy-for-tensorrt-llm-121) is required as +> well. This guide is written for that combination and is verified end to end on +> it; both steps go away only when a container ships a TensorRT-LLM that already +> contains #18381. > [!NOTE] > This guide replaces @@ -101,7 +102,14 @@ without the patch. ### Container +Start from a clone of this repository, so that the +[`trtllm_121_compat.py`](trtllm_121_compat.py) used below is mounted into the +container along with it: + ```bash +git clone https://github.com/triton-inference-server/tutorials.git +cd tutorials + docker run --rm -it --gpus all --network host \ -v ${PWD}:/workspace -w /workspace \ nvcr.io/nvidia/tritonserver:26.07-trtllm-python-py3 @@ -161,9 +169,15 @@ all_models/llmapi/ <- point --model-repository here > exist only on that pull request's branch. Clone the fork shown here for now; > once it lands, clone `https://github.com/NVIDIA/TensorRT-LLM.git` instead. +Only four files are needed, so skip the repository's Git LFS payload and check +out the one directory — a few seconds and about 9 MB, rather than the ~900 MB a +full clone pulls: + ```bash -git clone --depth 1 --branch feat/triton-llmapi-multimodal-image \ +GIT_LFS_SKIP_SMUDGE=1 git clone --depth 1 --filter=blob:none --sparse \ + --branch feat/triton-llmapi-multimodal-image \ https://github.com/faradawn/TensorRT-LLM.git /workspace/trtllm-pr +git -C /workspace/trtllm-pr sparse-checkout set triton_backend/all_models/llmapi ``` Then point `1/model.yaml` at the model and turn on the multimodal opt-in. This @@ -196,81 +210,47 @@ answer, so set it explicitly for multimodal models. ## Patching `model.py` for TensorRT-LLM 1.2.1 -> [!NOTE] -> Skip this section entirely once a `-trtllm-python-py3` container ships with a -> TensorRT-LLM build that includes -> [#18381](https://github.com/NVIDIA/TensorRT-LLM/pull/18381). It is a bridge for -> today's image, not part of the design. - -The files you just cloned build the prompt by calling -`async_build_multimodal_prompt`, which #18381 adds to +The backend files you just cloned build their prompt by calling +`async_build_multimodal_prompt`, which +[#18381](https://github.com/NVIDIA/TensorRT-LLM/pull/18381) adds to `tensorrt_llm/inputs/utils.py`. That module ships **inside the `tensorrt_llm` -wheel**, not in the `triton_backend/` tree you cloned, so on a 1.2.1 container -you have the caller but never the callee. The server still starts, still reports -`multimodal input enabled`, and still answers text-only prompts — and then fails -on every request that carries an image: +wheel**, not in the `triton_backend/` tree you cloned, and the container's wheel +is 1.2.1 — so you have the caller but never the callee. + +This is easy to miss, because nothing fails at startup. The server comes up, logs +`multimodal input enabled`, and answers text-only prompts correctly. Only +requests that actually carry an image fail: ```json {"error":"Error generating request: cannot import name 'async_build_multimodal_prompt' from 'tensorrt_llm.inputs' (/opt/venv-tritonserver/lib/python3.12/site-packages/tensorrt_llm/inputs/__init__.py)"} ``` -1.2.1 also lacks everything that helper is built on — `MEDIA_IO_REGISTRY`, -`ContentFormat`, `MultimodalDataTracker.item_order()`, -`interleave_mm_placeholders` and `async_apply_chat_template` — so you cannot -copy the new `utils.py` across either. What does work is replacing the single -call with an equivalent written against the 1.2.1 API. Add this method to the -`TritonPythonModel` class in -`/workspace/trtllm-pr/triton_backend/all_models/llmapi/tensorrt_llm/1/model.py`, -just above `async def _convert_request`: +Copying the new `utils.py` across does not help either: 1.2.1 lacks everything +that helper is built on — `MEDIA_IO_REGISTRY`, `ContentFormat`, +`MultimodalDataTracker.item_order()`, `interleave_mm_placeholders` and +`async_apply_chat_template`. What does work is replacing that one call with an +equivalent written against the 1.2.1 API. Run the script shipped next to this +guide: -```python - async def _build_multimodal_prompt_121(self, text, media): - """Stand-in for `inputs.async_build_multimodal_prompt` on TRT-LLM 1.2.1. - - 1.2.1 has no `async_apply_chat_template` and no - `MultimodalDataTracker.item_order()`, and its - `add_multimodal_placeholders` takes three arguments rather than four. - """ - from tensorrt_llm.inputs import prompt_inputs - from tensorrt_llm.inputs.utils import (ConversationMessage, - MultimodalDataTracker, - add_multimodal_placeholders, - apply_chat_template, - async_load_image) - - mm_data_tracker = MultimodalDataTracker(self._mm_model_type) - for url in media: - mm_data_tracker.add_data("image", async_load_image(url)) - mm_placeholder_counts = mm_data_tracker.placeholder_counts() - - content = add_multimodal_placeholders(self._mm_model_type, text, - mm_placeholder_counts) - conversation = [ - ConversationMessage(role="user", content=content, media=[]) - ] - prompt_task = asyncio.to_thread( - apply_chat_template, - model_type=self._mm_model_type, - tokenizer=self._mm_tokenizer, - processor=self._mm_processor, - conversation=conversation, - add_generation_prompt=True, - mm_placeholder_counts=[mm_placeholder_counts], - ) - prompt, (mm_data, _) = await asyncio.gather( - prompt_task, mm_data_tracker.retrieve_all_async()) - - prompt = prompt_inputs(prompt) - if mm_data: - prompt["multi_modal_data"] = mm_data - return prompt +```bash +python3 /workspace/Popular_Models_Guide/Qwen2.5-VL/trtllm_121_compat.py \ + /workspace/trtllm-pr/triton_backend/all_models/llmapi/tensorrt_llm/1/model.py ``` -`apply_chat_template` is synchronous and does real tokenizer work, so it goes -through `asyncio.to_thread` rather than blocking the engine's event loop while -the images are still downloading. +``` +Patched .../llmapi/tensorrt_llm/1/model.py for TensorRT-LLM 1.2.1. +``` + +It edits nothing but that one file, refuses to write source that does not parse, +and is safe to re-run — a second invocation reports `already patched; nothing to +do`. If the call it looks for is gone, it says so and tells you how to check +whether your container already has the function, rather than corrupting the +model repository. + +### What the script changes -Then, in `_convert_request`, point the call at it: +It adds one method, `_build_multimodal_prompt_121`, and points the call site at +it: ```diff image_url = get_input_tensor_by_name(request, 'image_url') @@ -293,9 +273,29 @@ Then, in `_convert_request`, point the call at it: + prompt = await self._build_multimodal_prompt_121(prompt, media) ``` +The new method does what the 1.3 helper does, in 1.2.1's vocabulary: + +| Step | 1.3 helper | 1.2.1 equivalent used here | +| ---- | ---------- | -------------------------- | +| download images | `MEDIA_IO_REGISTRY` | `async_load_image` per URL, gathered | +| insert placeholders | `interleave_mm_placeholders`, `item_order()` | `add_multimodal_placeholders`, three-argument form | +| render chat template | `async_apply_chat_template` | `apply_chat_template` via `asyncio.to_thread` | +| build the prompt | returns `PromptInputs` | `prompt_inputs(...)` plus `multi_modal_data` | + +`apply_chat_template` is synchronous and does real tokenizer work, so it goes +through `asyncio.to_thread` rather than blocking the engine's event loop while +the images are still downloading. Read +[`trtllm_121_compat.py`](trtllm_121_compat.py) for the full method. + Nothing else in the backend needs touching: `validate_media_urls` and the rest of the request path run unmodified on 1.2.1. +> [!NOTE] +> Delete this step once a `-trtllm-python-py3` container ships a TensorRT-LLM +> that already contains #18381. Note that #18381 merging is **not** enough on its +> own — the 26.07 image's wheel stays at 1.2.1 no matter what lands upstream, so +> the patch is needed until a *new image* ships. + ## Starting the server In Slurm/MPI environments, launch through `trtllm-llmapi-launch`: diff --git a/Popular_Models_Guide/Qwen2.5-VL/trtllm_121_compat.py b/Popular_Models_Guide/Qwen2.5-VL/trtllm_121_compat.py new file mode 100644 index 00000000..b9c35ec6 --- /dev/null +++ b/Popular_Models_Guide/Qwen2.5-VL/trtllm_121_compat.py @@ -0,0 +1,181 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# * Neither the name of NVIDIA CORPORATION nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY +# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY +# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +"""Make the Triton ``llmapi`` backend's image path run on TensorRT-LLM 1.2.1. + +Used by ``qwen2_5_vl_trtllm_guide.md``. See the guide for the full explanation; +the short version is that the backend's ``model.py`` builds its prompt with +``tensorrt_llm.inputs.async_build_multimodal_prompt``, which is added by +NVIDIA/TensorRT-LLM#18381. That function lives in the ``tensorrt_llm`` *wheel*, +not in the ``triton_backend/`` tree you clone, so on a container whose wheel is +1.2.1 you have the caller but never the callee, and every request carrying an +``image_url`` fails with:: + + cannot import name 'async_build_multimodal_prompt' from 'tensorrt_llm.inputs' + +1.2.1 also lacks everything that helper is built on -- ``MEDIA_IO_REGISTRY``, +``ContentFormat``, ``MultimodalDataTracker.item_order()``, +``interleave_mm_placeholders`` and ``async_apply_chat_template`` -- so copying +the new ``utils.py`` across is not an option either. This script instead swaps +the single call for an equivalent written against the 1.2.1 API surface. + +Usage:: + + python3 trtllm_121_compat.py /triton_backend/all_models/llmapi/tensorrt_llm/1/model.py + +Safe to re-run: it exits cleanly if the file is already patched. Delete this +step once a ``-trtllm-python-py3`` container ships a TensorRT-LLM that already +contains #18381. +""" + +import argparse +import ast +import pathlib +import sys + +# The call this replaces, exactly as it appears in model.py. +OLD_CALL = """ from tensorrt_llm.inputs import async_build_multimodal_prompt + + media = [ + url.decode("utf-8") if isinstance(url, bytes) else str(url) + for url in image_url.reshape(-1) + ] + validate_media_urls(media) + prompt = await async_build_multimodal_prompt( + model_type=self._mm_model_type, + tokenizer=self._mm_tokenizer, + processor=self._mm_processor, + prompt=prompt, + media=media, + modality="image", + )""" + +NEW_CALL = """ media = [ + url.decode("utf-8") if isinstance(url, bytes) else str(url) + for url in image_url.reshape(-1) + ] + validate_media_urls(media) + prompt = await self._build_multimodal_prompt_121(prompt, media)""" + +# Inserted immediately above `async def _convert_request`. `asyncio` is already +# imported at module scope in model.py, so this needs no new top-level imports. +NEW_METHOD = ''' async def _build_multimodal_prompt_121(self, text, media): + """Stand-in for `inputs.async_build_multimodal_prompt` on TRT-LLM 1.2.1. + + 1.2.1 has no `async_apply_chat_template` and no + `MultimodalDataTracker.item_order()`, and its + `add_multimodal_placeholders` takes three arguments rather than four. + """ + from tensorrt_llm.inputs import prompt_inputs + from tensorrt_llm.inputs.utils import (ConversationMessage, + MultimodalDataTracker, + add_multimodal_placeholders, + apply_chat_template, + async_load_image) + + mm_data_tracker = MultimodalDataTracker(self._mm_model_type) + for url in media: + mm_data_tracker.add_data("image", async_load_image(url)) + mm_placeholder_counts = mm_data_tracker.placeholder_counts() + + content = add_multimodal_placeholders(self._mm_model_type, text, + mm_placeholder_counts) + conversation = [ + ConversationMessage(role="user", content=content, media=[]) + ] + # `apply_chat_template` is synchronous and does real tokenizer work, so + # keep it off the engine's event loop while the images download. + prompt_task = asyncio.to_thread( + apply_chat_template, + model_type=self._mm_model_type, + tokenizer=self._mm_tokenizer, + processor=self._mm_processor, + conversation=conversation, + add_generation_prompt=True, + mm_placeholder_counts=[mm_placeholder_counts], + ) + prompt, (mm_data, _) = await asyncio.gather( + prompt_task, mm_data_tracker.retrieve_all_async()) + + prompt = prompt_inputs(prompt) + if mm_data: + prompt["multi_modal_data"] = mm_data + return prompt + +''' + +ANCHOR = " async def _convert_request(self, request):" + +MOVED_ON = """{path} does not contain the call this script replaces. + +That usually means the backend has moved on -- most likely #18381 merged, in +which case check whether your container's TensorRT-LLM already provides +`async_build_multimodal_prompt` and skip this step entirely: + + python3 -c "from tensorrt_llm.inputs import async_build_multimodal_prompt" + +If that import succeeds, no patch is needed.""" + + +def main(): + parser = argparse.ArgumentParser( + description="Patch the Triton llmapi backend's model.py for " + "TensorRT-LLM 1.2.1.") + parser.add_argument( + "model_py", + type=pathlib.Path, + help="path to all_models/llmapi/tensorrt_llm/1/model.py") + args = parser.parse_args() + + path = args.model_py + if not path.is_file(): + sys.exit(f"{path} is not a file") + + source = path.read_text() + + if "_build_multimodal_prompt_121" in source: + print(f"{path} is already patched; nothing to do.") + return + + if source.count(OLD_CALL) != 1: + sys.exit(MOVED_ON.format(path=path)) + if source.count(ANCHOR) != 1: + sys.exit(f"could not locate `{ANCHOR.strip()}` in {path}") + + source = source.replace(OLD_CALL, NEW_CALL) + source = source.replace(ANCHOR, NEW_METHOD + ANCHOR, 1) + + # Fail before writing rather than leave a half-broken model repository. + try: + ast.parse(source) + except SyntaxError as exc: + sys.exit(f"patched source does not parse ({exc}); model.py left alone") + + path.write_text(source) + print(f"Patched {path} for TensorRT-LLM 1.2.1.") + + +if __name__ == "__main__": + main() From 7405ea9d2fef0d764f78fb79c86b8288eff33858 Mon Sep 17 00:00:00 2001 From: Faradawn Yang <73060648+faradawn@users.noreply.github.com> Date: Thu, 10 Sep 2026 08:01:19 -0700 Subject: [PATCH 07/10] docs: build the Qwen2.5-VL guide on the v1.2.1 tag and ship the backend files The guide cloned the #18381 branch, which targets TensorRT-LLM 1.3, and then patched it back down to work against the 1.2.1 wheel in the container. That is backwards: it carried 358 lines of difference from v1.2.1, of which only ~30 is the image feature and the rest is unrelated 1.3 drift running against a 1.2.1 runtime. Start from the v1.2.1 tag instead, which is exactly the TensorRT-LLM installed in nvcr.io/nvidia/tritonserver:26.07-trtllm-python-py3, and add the image input on top. That is 109 lines on a base that matches the wheel. Two consequences: - The guide no longer depends on a personal fork or an unmerged pull request. Everything comes from the official NVIDIA v1.2.1 tag plus two files shipped here, following the convention of the Llava1.5 guide next door. - There is nothing for the reader to patch. model.py and config.pbtxt are provided; helpers.py comes from v1.2.1 unchanged; model.yaml is written in the guide. Copy two files instead of editing an 800-line one. model.py carries a provenance header naming its v1.2.1 source, and scripts that regenerate it are described in the guide. Also rewrite the guide as plain steps -- container, openai, model repository, serve, request -- following the structure of llava_trtllm_guide.md, and drop the running commentary about what does not work. 480 lines to 231. Verified on 1x B200 with Qwen/Qwen2.5-VL-3B-Instruct by running the guide's commands as written: single image returns the quoted answer, two images are described in order, a local path and an unreachable host both surface errors, and text-only requests still work. Co-Authored-By: Claude Opus 5 --- Popular_Models_Guide/Qwen2.5-VL/config.pbtxt | 243 ++++++ Popular_Models_Guide/Qwen2.5-VL/model.py | 811 ++++++++++++++++++ .../Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md | 421 ++------- .../Qwen2.5-VL/trtllm_121_compat.py | 181 ---- 4 files changed, 1140 insertions(+), 516 deletions(-) create mode 100644 Popular_Models_Guide/Qwen2.5-VL/config.pbtxt create mode 100644 Popular_Models_Guide/Qwen2.5-VL/model.py delete mode 100644 Popular_Models_Guide/Qwen2.5-VL/trtllm_121_compat.py diff --git a/Popular_Models_Guide/Qwen2.5-VL/config.pbtxt b/Popular_Models_Guide/Qwen2.5-VL/config.pbtxt new file mode 100644 index 00000000..da5c9475 --- /dev/null +++ b/Popular_Models_Guide/Qwen2.5-VL/config.pbtxt @@ -0,0 +1,243 @@ +# Copyright 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# * Neither the name of NVIDIA CORPORATION nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY +# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY +# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +name: "tensorrt_llm" +backend: "python" + +####################################################### +# The below config arguments are specific in model.yaml +# Please don't add it in config.pbtxt +# +# max_batch_size: 64 +# +# model_transaction_policy { +# decoupled: False +# } +####################################################### + +instance_group [ + { + count: 1 + kind : KIND_CPU + } +] + +input [ + { + name: "text_input" + data_type: TYPE_STRING + dims: [ -1 ] + }, + { + name: "image_url" + data_type: TYPE_STRING + dims: [ -1 ] + optional: true + }, + { + name: "streaming" + data_type: TYPE_BOOL + dims: [ 1 ] + optional: true + }, + ## SamplingParams Arguments for Each Request ## + { + name: "sampling_param_best_of" + data_type: TYPE_INT32 + dims: [ 1 ] + optional: true + }, + { + name: "sampling_param_temperature" + data_type: TYPE_FP32 + dims: [ 1 ] + optional: true + }, + { + name: "sampling_param_top_k" + data_type: TYPE_INT32 + dims: [ 1 ] + optional: true + }, + { + name: "sampling_param_top_p" + data_type: TYPE_FP32 + dims: [ 1 ] + optional: true + }, + { + name: "sampling_param_frequency_penalty" + data_type: TYPE_FP32 + dims: [ 1 ] + optional: true + }, + { + name: "sampling_param_presence_penalty" + data_type: TYPE_FP32 + dims: [ 1 ] + optional: true + }, + { + name: "sampling_param_max_tokens" + data_type: TYPE_INT32 + dims: [ 1 ] + optional: true + }, + { + name: "sampling_param_stop" + data_type: TYPE_STRING + dims: [-1] + optional: true + }, + { + name: "sampling_param_seed" + data_type: TYPE_UINT64 + dims: [ 1 ] + optional: true + }, + { + name: "sampling_param_exclude_input_from_output" + data_type: TYPE_BOOL + dims: [ 1 ] + optional: true + }, + { + name: "sampling_param_return_perf_metrics" + data_type: TYPE_BOOL + dims: [ 1 ] + optional: true + }, + ## Arguments for Controlling Response Output Fields ## + { + name: "return_finish_reason" + data_type: TYPE_BOOL + dims: [1] + optional: true + }, + { + name: "return_stop_reason" + data_type: TYPE_BOOL + dims: [1] + optional: true + }, + { + name: "return_cumulative_logprob" + data_type: TYPE_BOOL + dims: [1] + optional: true + }, + { + name: "stop" + data_type: TYPE_BOOL + dims: [ 1 ] + optional: true + } +] +################################################################### +# The below output parameters are arguments from LLM.RequestOutput +################################################################### +output [ + { + name: "text_output" + data_type: TYPE_STRING + dims: [-1] + }, + { + name: "finish_reason" + data_type: TYPE_STRING + dims: [-1] + }, + { + name: "stop_reason" + data_type: TYPE_STRING + dims: [-1] + }, + { + name: "cumulative_logprob" + data_type: TYPE_FP32 + dims: [-1] + }, + { + name: "kv_cache_reused_block" + data_type: TYPE_INT32 + dims: [-1] + }, + { + name: "kv_cache_missed_block" + data_type: TYPE_INT32 + dims: [-1] + }, + { + name: "kv_cache_alloc_new_blocks" + data_type: TYPE_INT32 + dims: [-1] + }, + { + name: "kv_cache_alloc_total_blocks" + data_type: TYPE_INT32 + dims: [-1] + }, + { + name: "kv_cache_hit_rate" + data_type: TYPE_FP32 + dims: [-1] + }, + { + name: "arrival_time_ns" + data_type: TYPE_INT64 + dims: [ 1 ] + }, + { + name: "first_scheduled_time_ns" + data_type: TYPE_INT64 + dims: [ 1 ] + }, + { + name: "first_token_time_ns" + data_type: TYPE_INT64 + dims: [ 1 ] + }, + { + name: "last_token_time_ns" + data_type: TYPE_INT64 + dims: [ 1 ] + }, + { + name: "acceptance_rate" + data_type: TYPE_FP32 + dims: [ 1 ] + }, + { + name: "total_accepted_draft_tokens" + data_type: TYPE_INT32 + dims: [ 1 ] + }, + { + name: "total_draft_tokens" + data_type: TYPE_INT32 + dims: [ 1 ] + } +] diff --git a/Popular_Models_Guide/Qwen2.5-VL/model.py b/Popular_Models_Guide/Qwen2.5-VL/model.py new file mode 100644 index 00000000..61b6ca8e --- /dev/null +++ b/Popular_Models_Guide/Qwen2.5-VL/model.py @@ -0,0 +1,811 @@ +# Copyright 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# * Neither the name of NVIDIA CORPORATION nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY +# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY +# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +# --------------------------------------------------------------------------- +# Provenance +# +# This file is TensorRT-LLM v1.2.1's Triton `llmapi` backend +# triton_backend/all_models/llmapi/tensorrt_llm/1/model.py +# with image input added, as proposed in NVIDIA/TensorRT-LLM#18381. +# +# v1.2.1 is the TensorRT-LLM shipped in nvcr.io/nvidia/tritonserver:26.07- +# trtllm-python-py3, so this matches the wheel installed in the container. +# Regenerate with tutorials/Popular_Models_Guide/Qwen2.5-VL, see the guide. +# --------------------------------------------------------------------------- + +import asyncio +import gc +import json +import os +import queue +import sys +import threading +from contextlib import asynccontextmanager +from dataclasses import dataclass +from random import randint +from typing import Any +from urllib.parse import urlparse + +import numpy as np +import pandas as pd +import triton_python_backend_utils as pb_utils +import yaml +from helpers import (get_input_tensor_by_name, get_output_config_from_request, + get_sampling_params_from_request, + get_streaming_from_request) +from mpi4py.futures import MPICommExecutor +from mpi4py.MPI import COMM_WORLD + +from tensorrt_llm import LLM, SamplingParams +from tensorrt_llm._utils import global_mpi_rank, global_mpi_size +from tensorrt_llm.llmapi.llm import RequestOutput +from tensorrt_llm.llmapi.llm_utils import update_llm_args_with_extra_dict + + +@dataclass +class RequestData: + triton_req_id: int + triton_user_id: str + triton_request: Any + response_iterator: RequestOutput + + +def get_model_config(filename, include_keys=None, exclude_keys=None): + engine_args_filepath = os.path.join(pb_utils.get_model_dir(), filename) + engine_config = None + if os.path.isfile(engine_args_filepath): + try: + with open(engine_args_filepath) as file: + engine_config = yaml.safe_load(file) + except Exception as e: + raise pb_utils.TritonModelException( + f"Failed to parse YAML engine config: {e}") + + assert engine_config is not None, f"'{filename}' containing TRT-LLM engine args not found in '{pb_utils.get_model_dir()}'" + + if include_keys: + engine_config = { + k: v + for k, v in engine_config.items() if k in include_keys + } + if exclude_keys: + engine_config = { + k: v + for k, v in engine_config.items() if k not in exclude_keys + } + return engine_config + + +# `image_url` is client-controlled, so restrict it to web URLs for security. +ALLOWED_MEDIA_SCHEMES = ("http", "https") + + +def validate_media_urls(urls): + """Reject anything that is not a web URL.""" + for url in urls: + if urlparse(url).scheme not in ALLOWED_MEDIA_SCHEMES: + raise pb_utils.TritonModelException( + f"Unsupported image_url {url!r}: only " + f"{', '.join(ALLOWED_MEDIA_SCHEMES)} URLs are accepted.") + + +def get_input_scalar_by_name(request, + name, + expected_batch_size=1, + batch_index=0): + tensor = pb_utils.get_input_tensor_by_name(request, name) + if tensor is None: + return None + tensor = tensor.as_numpy() + + if tensor.size != expected_batch_size: + raise pb_utils.TritonModelException( + f"Expected a scalar tensor for tensor {name}") + + return tensor.item(batch_index) + + +class TritonPythonModel: + + @classmethod + def auto_complete_config(cls, auto_complete_model_config): + """ + Set triton_config values in model.yaml to auto_complete_model_config + + Args: + auto_complete_model_config: Default configurations loaded from config.pbtxt + + Returns: + auto_complete_model_config: Updated triton server configurations loading triton_config from model.yaml + + Notes: + - This function is called when Triton server starts. + - It combines the default configurations in config.pbtxt with the triton_config values in model.yaml + """ + triton_config = get_model_config(os.environ.get('LLM_CONFIG_PATH', + 'model.yaml'), + include_keys=["triton_config" + ])["triton_config"] + auto_complete_model_config.set_model_transaction_policy( + dict(decoupled=bool(triton_config["decoupled"]))) + auto_complete_model_config.set_max_batch_size( + int(triton_config["max_batch_size"])) + + return auto_complete_model_config + + def initialize(self, args): + """ + Function allows the model to initialize any state associated with it. + + Args: + args: triton configurations loaded from config.pbtxt and extended by auto_complete_config + Note: + - `initialize` is called only once when the model is being loaded. + - Implementing `initialize` function is optional. + """ + from tensorrt_llm.llmapi import MpiCommSession + + self.model_config = json.loads(args["model_config"]) + triton_config = get_model_config(os.environ.get('LLM_CONFIG_PATH', + 'model.yaml'), + include_keys=["triton_config" + ])["triton_config"] + self.decoupled = bool(triton_config["decoupled"]) + self.params = self.model_config['parameters'] + self.logger = pb_utils.Logger + + text_output_config = pb_utils.get_output_config_by_name( + self.model_config, "text_output") + self.output_dtype = pb_utils.triton_string_to_numpy( + text_output_config["data_type"]) + if global_mpi_rank() == 0: + # Initialize engine arguments + self.llm_engine_args = update_llm_args_with_extra_dict( + {}, + get_model_config(os.environ.get('LLM_CONFIG_PATH', + 'model.yaml'), + exclude_keys=["triton_config"]), + ) + self.logger.log_info( + f"[trtllm] rank{global_mpi_rank()} is starting trtllm engine with args: {self.llm_engine_args}" + ) + + triton_config = get_model_config( + os.environ.get('LLM_CONFIG_PATH', 'model.yaml'), + include_keys=["triton_config"])["triton_config"] + self.cancellation_check_period_ms = int( + triton_config["cancellation_check_period_ms"] + ) if "cancellation_check_period_ms" in triton_config else 100 + + if global_mpi_size() > 1: + mpi_session = MpiCommSession(comm=COMM_WORLD, + n_workers=COMM_WORLD.Get_size()) + self.llm_engine_args["_mpi_session"] = mpi_session + + # Starting the TRT-LLM engine with LLM API and its event thread running the AsyncIO event loop. + self._init_engine() + + self.running = False + + # Starting the response thread. It allows TRT-LLM to keep making progress while + # response sender(s) are sending responses to server frontend. + self._response_queue = queue.Queue() + self._response_thread = threading.Thread(target=self._response_loop) + self._response_thread.start() + + self.multimodal_enabled = bool( + triton_config.get("multimodal", False)) + if self.multimodal_enabled: + self._init_multimodal() + + self.req_id_to_request_data = {} + self.triton_user_id_to_req_ids = {} + self.lock = threading.Lock() + self.cancellation_thread = threading.Thread( + target=self.cancellation_loop) + self.running = True + self.cancellation_thread.start() + else: + self.logger.log_info( + f"[trtllm] rank{global_mpi_rank()} is waiting for the leader node..." + ) + with MPICommExecutor(COMM_WORLD) as executor: + if executor is not None: + raise RuntimeError( + f"[trtllm] rank{COMM_WORLD.rank} should not have executor" + ) + return + + def _init_engine(self): + """ + Initialize the LLM engine in a separate thread running the AsyncIO event loop. + """ + self._llm_engine = None + self._llm_engine_start_cv = threading.Condition() + self._llm_engine_shutdown_event = asyncio.Event() + self._event_thread = threading.Thread(target=asyncio.run, + args=(self._run_llm_engine(), )) + self._event_thread.start() + with self._llm_engine_start_cv: + while self._llm_engine is None: + self._llm_engine_start_cv.wait() + + # The 'threading.Thread()' will not raise the exception here should the engine + # failed to start, so the exception is passed back via the engine variable. + if isinstance(self._llm_engine, Exception): + e = self._llm_engine + self.logger.log_error(f"[trtllm] Failed to start engine: {e}") + if self._event_thread is not None: + self._event_thread.join() + self._event_thread = None + raise e + + async def _run_llm_engine(self): + """ + Run the LLM engine in an asynchronous context. + """ + # Counter to keep track of ongoing request counts. + self._ongoing_request_count = 0 + + @asynccontextmanager + async def async_llm_wrapper(): + # Create LLM in a thread to avoid blocking + loop = asyncio.get_running_loop() + try: + llm = await loop.run_in_executor( + None, lambda: LLM(**self.llm_engine_args)) + yield llm + finally: + if 'llm' in locals(): + # Run shutdown in a thread to avoid blocking + await loop.run_in_executor(None, llm.shutdown) + + try: + async with async_llm_wrapper() as engine: + # Capture the engine event loop and make it visible to other threads. + self._event_loop = asyncio.get_running_loop() + + # Signal the engine is started and make it visible to other threads. + with self._llm_engine_start_cv: + self._llm_engine = engine + self._llm_engine_start_cv.notify_all() + + # Wait for the engine shutdown signal. + await self._llm_engine_shutdown_event.wait() + + # Wait for the ongoing requests to complete. + while self._ongoing_request_count > 0: + self.logger.log_info( + "[trtllm] Awaiting remaining {} requests".format( + self._ongoing_request_count)) + await asyncio.sleep(1) + + # Cancel all tasks in the event loop. + for task in asyncio.all_tasks(loop=self._event_loop): + if task is not asyncio.current_task(): + task.cancel() + + except Exception as e: + # Signal and pass the exception back via the engine variable if the engine + # failed to start. If the engine has started, re-raise the exception. + with self._llm_engine_start_cv: + if self._llm_engine is None: + self._llm_engine = e + self._llm_engine_start_cv.notify_all() + return + raise e + + self._llm_engine = None + self.logger.log_info("[trtllm] Shutdown complete") + + def _response_loop(self): + """ + Helper function to process responses from the response queue when streaming is enabled. + """ + while True: + item = self._response_queue.get() + # To signal shutdown a None item will be added to the queue. + if item is None: + break + response_state, response, response_flag = item + response_sender = response_state["response_sender"] + try: + response_sender.send(response, response_flag) + # Stop checking for cancellation if the last response is generated. + if not response_state["last_response_generated"]: + response_state[ + "is_cancelled"] = response_sender.is_cancelled() + except Exception as e: + self.logger.log_error( + f"An error occurred while sending a response: {e}") + finally: + if response_flag == pb_utils.TRITONSERVER_RESPONSE_COMPLETE_FINAL: + self._ongoing_request_count -= 1 + + def cancellation_loop(self): + """Checks if any pending requests have been cancelled.""" + while self.running: + import time + time.sleep(self.cancellation_check_period_ms / 1000.0) + with self.lock: + cancelled_req_ids = [] + for req_id, request_data in self.req_id_to_request_data.items(): + if request_data.triton_request.is_cancelled(): + request_data.response_iterator.abort() + + response_sender = request_data.triton_request.get_response_sender( + ) + response_sender.send( + pb_utils.InferenceResponse( + error=pb_utils.TritonError( + "Request cancelled by client", + pb_utils.TritonError.CANCELLED)), + flags=pb_utils.TRITONSERVER_RESPONSE_COMPLETE_FINAL) + cancelled_req_ids.append(req_id) + for req_id in cancelled_req_ids: + del self.req_id_to_request_data[req_id] + + def handle_stop_request(self, triton_user_id, response_sender): + if triton_user_id is None or triton_user_id == "": + response_sender.send( + pb_utils.InferenceResponse(error=pb_utils.TritonError( + "A request id must be provided for request cancellation")), + flags=pb_utils.TRITONSERVER_RESPONSE_COMPLETE_FINAL) + return + + with self.lock: + if triton_user_id in self.triton_user_id_to_req_ids: + req_ids = self.triton_user_id_to_req_ids[triton_user_id] + for req_id in req_ids: + request_data = self.req_id_to_request_data[req_id] + request_data.response_iterator.abort() + del self.req_id_to_request_data[req_id] + + response_sender.send( + pb_utils.InferenceResponse(error=pb_utils.TritonError( + "Request cancelled by client", pb_utils.TritonError.CANCELLED)), + flags=pb_utils.TRITONSERVER_RESPONSE_COMPLETE_FINAL) + + def execute(self, requests): + """ + Function is called by Triton server when a new request is received. + + Args: + requests: a list of pb_utils.InferenceRequest + + Notes: + - `execute` must be implemented in every Triton Python model. + """ + # TODO: [JIRA-4040] Add health check here + for request in requests: + # TODO : [JIRA-4040] Verify Lora + if request is not None: + assert ( + self._llm_engine_shutdown_event.is_set() is False + ), "Cannot create tasks after shutdown has been requested" + coro = self._execute_single_request(request) + asyncio.run_coroutine_threadsafe(coro, self._event_loop) + + return None + + async def _execute_single_request(self, request): + """ + Execute a single inference request asynchronously. + """ + response_sender = request.get_response_sender() + triton_user_id = request.request_id() + + stop = get_input_scalar_by_name(request, 'stop') + if stop: + self.handle_stop_request(triton_user_id, response_sender) + return + + response_state = { + "response_sender": response_sender, + "is_cancelled": False, + "last_response_generated": + False, # last response ready but not yet sent + } + self._ongoing_request_count += 1 + decrement_ongoing_request_count = True + + # Unique request id used to identify each triton request + triton_req_id = str(randint(0, sys.maxsize)) + + try: + # TODO: [JIRA-4496] Implement when request contains batched prompts + (prompt, sampling_params, streaming, + output_config) = await self._convert_request(request) + if streaming and not self.decoupled: + raise pb_utils.TritonModelException( + "Streaming is only supported in decoupled mode.") + # Generate the response. + response_iterator = self._llm_engine.generate_async( + prompt, SamplingParams(**sampling_params), streaming) + + with self.lock: + self.req_id_to_request_data[triton_req_id] = RequestData( + triton_req_id=triton_req_id, + triton_user_id=request.request_id(), + triton_request=request, + response_iterator=response_iterator, + ) + if triton_user_id is not None and triton_user_id != "" and triton_user_id: + self.triton_user_id_to_req_ids[triton_user_id] = set() + # TODO: [JIRA-4496] Add all batched request ids to the set + self.triton_user_id_to_req_ids[triton_user_id].add( + triton_req_id) + + async for request_output in response_iterator: + # Send each response if streaming. + if streaming: + response = self._create_response( + request_output=request_output, + output_config=output_config) + flags = 0 + if request_output.finished: + response_state["last_response_generated"] = True + flags = pb_utils.TRITONSERVER_RESPONSE_COMPLETE_FINAL + # with streaming, self._response_loop will decrement self._ongoing_request_count + decrement_ongoing_request_count = False + self._response_queue.put_nowait( + (response_state, response, flags)) + + # Send the last response which contains all the outputs if not streaming. + if not streaming: + # If the request was cancelled, we don't need to send the last response + with self.lock: + was_cancelled = triton_req_id not in self.req_id_to_request_data + if not was_cancelled: + # Remove the request from the request data map so the cancellation loop stops querying + # is_cancelled() on the request + del self.req_id_to_request_data[triton_req_id] + + if not was_cancelled: + response_sender.send( + self._create_response(request_output=request_output, + output_config=output_config), + flags=pb_utils.TRITONSERVER_RESPONSE_COMPLETE_FINAL) + + except Exception as e: + self.logger.log_error(f"[trtllm] Error generating request: {e}") + error = pb_utils.TritonError(f"Error generating request: {e}") + text_output_tensor = pb_utils.Tensor( + "text_output", np.asarray(["N/A"], dtype=self.output_dtype)) + response = pb_utils.InferenceResponse( + output_tensors=[text_output_tensor], error=error) + response_sender.send( + response, flags=pb_utils.TRITONSERVER_RESPONSE_COMPLETE_FINAL) + raise e + + finally: + if decrement_ongoing_request_count: + self._ongoing_request_count -= 1 + with self.lock: + if triton_req_id in self.req_id_to_request_data: + del self.req_id_to_request_data[triton_req_id] + if triton_user_id is not None and triton_user_id != "" and triton_user_id in self.triton_user_id_to_req_ids: + del self.triton_user_id_to_req_ids[triton_user_id] + + def _init_multimodal(self): + """Resolve the tokenizer, HF processor and model type once.""" + from transformers import AutoProcessor + + from tensorrt_llm._torch.pyexecutor.config_utils import \ + load_pretrained_config + + self._mm_tokenizer = self._llm_engine.tokenizer + hf_model_dir = self._llm_engine._hf_model_dir or getattr( + getattr(self._mm_tokenizer, "tokenizer", None), "name_or_path", + None) + if hf_model_dir is None: + raise pb_utils.TritonModelException( + "triton_config.multimodal is enabled but the checkpoint directory " + "could not be resolved from the engine.") + hf_model_dir = str(hf_model_dir) + trust_remote_code = self._llm_engine.args.trust_remote_code + try: + self._mm_processor = AutoProcessor.from_pretrained( + hf_model_dir, trust_remote_code=trust_remote_code) + model_config = load_pretrained_config( + hf_model_dir, + trust_remote_code=trust_remote_code, + checkpoint_format=getattr(self._llm_engine.args, + "checkpoint_format", None)) + except Exception as e: + raise pb_utils.TritonModelException( + f"triton_config.multimodal is enabled but the HF processor/config " + f"for '{hf_model_dir}' could not be loaded: {e}") + + # Composite configs (e.g. Qwen2_5_VLConfig) delegate the instance + # attribute to `text_config`, so prefer the class attribute. + self._mm_model_type = getattr(type(model_config), + "model_type", None) or getattr( + model_config, "model_type", "") + self.logger.log_info("[trtllm] multimodal input enabled for model_type " + f"'{self._mm_model_type}'") + + async def _build_multimodal_prompt(self, text, media): + """Download the images and fold them into a multimodal prompt.""" + from tensorrt_llm.inputs import prompt_inputs + from tensorrt_llm.inputs.utils import (ConversationMessage, + MultimodalDataTracker, + add_multimodal_placeholders, + apply_chat_template, + async_load_image) + + mm_data_tracker = MultimodalDataTracker(self._mm_model_type) + for url in media: + mm_data_tracker.add_data("image", async_load_image(url)) + mm_placeholder_counts = mm_data_tracker.placeholder_counts() + + content = add_multimodal_placeholders(self._mm_model_type, text, + mm_placeholder_counts) + conversation = [ + ConversationMessage(role="user", content=content, media=[]) + ] + # `apply_chat_template` is synchronous and does real tokenizer work, so + # keep it off the event loop while the images are still downloading. + prompt_task = asyncio.to_thread( + apply_chat_template, + model_type=self._mm_model_type, + tokenizer=self._mm_tokenizer, + processor=self._mm_processor, + conversation=conversation, + add_generation_prompt=True, + mm_placeholder_counts=[mm_placeholder_counts], + ) + prompt, (mm_data, _) = await asyncio.gather( + prompt_task, mm_data_tracker.retrieve_all_async()) + + prompt = prompt_inputs(prompt) + if mm_data: + prompt["multi_modal_data"] = mm_data + return prompt + + async def _convert_request(self, request): + """Helper function to convert the request into a prompt for LLM.generate_async + + Args: + request: Triton Server request + + Returns: + prompt: A LLM PromptInputs object + + Notes: + - The current implementation only supports text_input being a 1D tensor(a single prompt). + """ + text_input = get_input_tensor_by_name(request, 'text_input') + if text_input is None: + raise pb_utils.TritonModelException( + f"text_input is missing from the request") + if len(text_input.shape) > 1: + raise pb_utils.TritonModelException( + f"The current implementation only supports text_input being a 1D tensor." + ) + + prompt = text_input[0] + + if isinstance(prompt, bytes): + prompt = prompt.decode("utf-8") + + # Only read `image_url` when the operator opts in, so a deployment + # already declaring that input keeps its behavior after an upgrade. + if self.multimodal_enabled: + image_url = get_input_tensor_by_name(request, 'image_url') + if image_url is not None and image_url.size > 0: + media = [ + url.decode("utf-8") if isinstance(url, bytes) else str(url) + for url in image_url.reshape(-1) + ] + validate_media_urls(media) + prompt = await self._build_multimodal_prompt(prompt, media) + + sampling_params = get_sampling_params_from_request(request) + output_config = get_output_config_from_request(request) + streaming = get_streaming_from_request(request) + return prompt, sampling_params, streaming, output_config + + def _create_response(self, request_output, output_config): + """Process the generated request_output and create the client response. + + Args: + request_output (tensorrt_llm.llmapi.RequestOutput): Inferred results running the LLM engine and input prompt. + Parameters: + request_id (int): The unique ID of the request. + prompt (str, optional): The prompt string of the request. + prompt_token_ids (List[int]): The token ids of the prompt. + outputs (List[CompletionOutput]): The output sequences of the request. + Args: + index (int): The index of the output in the request. + text (str): The generated output text. + token_ids (List[int], optional): The token ids of the generated output text. + cumulative_logprob (float, optional): The cumulative log probability of the generated output text. + logprobs (List[float], optional): The log probabilities of the top probability words at each position if the logprobs are requested. + finish_reason (Literal['stop', 'length', 'timeout', 'cancelled'], optional): The reason why the sequence is finished. + stop_reason (int, str, optional): The stop string or token id that caused the completion to stop, None if the completion finished for some other reason. + generation_logits (torch.Tensor, optional): The logits on the generated output token ids. + disaggregated_params (tensorrt_llm.disaggregated_params.DisaggregatedParams, optional): Parameters needed for disaggregated serving. + context_logits (torch.Tensor, optional): The logits on the prompt token ids. + finished (bool): Whether the whole request is finished. + + Returns: + pb_utils.InferenceResponse: Converted output response + The arguments are defined in config.pbtxt output + triton_config:output_config in model.yaml controls which output to send besides text_output + """ + # TODO: [JIRA-4040] Check if request_output has_error and handle it + response = [] + text_output = [ + output.text.encode("utf-8") for output in request_output.outputs + ] + + response.append( + pb_utils.Tensor("text_output", + np.asarray(text_output, dtype=self.output_dtype))) + + # Extract and add configurable output fields + # The output_config loads related input from request + output_fields = { + "return_finish_reason": + ("finish_reason", lambda output: output.finish_reason), + "return_stop_reason": + ("stop_reason", lambda output: output.stop_reason), + "return_cumulative_logprob": + ("cumulative_logprob", lambda output: output.cumulative_logprob) + } + + for config_key, (output_name, extractor) in output_fields.items(): + if output_config[config_key]: + tensor_data = [ + str(extractor(output)) for output in request_output.outputs + ] + response.append( + pb_utils.Tensor(output_name, + np.asarray(tensor_data, dtype=np.object_))) + + if hasattr(request_output.outputs[0], 'request_perf_metrics' + ) and request_output.outputs[0].request_perf_metrics: + + perf_metrics = request_output.outputs[0].request_perf_metrics + + # kv cache perf metrics per request + kv_metrics = perf_metrics.kv_cache_metrics + + response.append( + pb_utils.Tensor( + "kv_cache_reused_block", + np.asarray([kv_metrics.num_reused_blocks], + dtype=self.output_dtype))) + response.append( + pb_utils.Tensor( + "kv_cache_hit_rate", + np.asarray([kv_metrics.kv_cache_hit_rate], + dtype=self.output_dtype))) + response.append( + pb_utils.Tensor( + "kv_cache_alloc_new_blocks", + np.asarray([kv_metrics.num_new_allocated_blocks], + dtype=self.output_dtype))) + response.append( + pb_utils.Tensor( + "kv_cache_alloc_total_blocks", + np.asarray([kv_metrics.num_total_allocated_blocks], + dtype=self.output_dtype))) + response.append( + pb_utils.Tensor( + "kv_cache_missed_block", + np.asarray([kv_metrics.num_missed_blocks], + dtype=self.output_dtype))) + + # timing perf metrics per request + timing_metrics = perf_metrics.timing_metrics + response.append( + pb_utils.Tensor( + "arrival_time_ns", + np.asarray( + [pd.Timedelta(timing_metrics.arrival_time).value], + dtype=self.output_dtype))) + + response.append( + pb_utils.Tensor( + "first_scheduled_time_ns", + np.asarray([ + pd.Timedelta(timing_metrics.first_scheduled_time).value + ], + dtype=self.output_dtype))) + + response.append( + pb_utils.Tensor( + "first_token_time_ns", + np.asarray( + [pd.Timedelta(timing_metrics.first_token_time).value], + dtype=self.output_dtype))) + + response.append( + pb_utils.Tensor( + "last_token_time_ns", + np.asarray( + [pd.Timedelta(timing_metrics.last_token_time).value], + dtype=self.output_dtype))) + + #spec dec perf metrics per request + spec_dec_metrics = perf_metrics.speculative_decoding + + response.append( + pb_utils.Tensor( + "acceptance_rate", + np.asarray([spec_dec_metrics.acceptance_rate], + dtype=self.output_dtype))) + + response.append( + pb_utils.Tensor( + "total_accepted_draft_tokens", + np.asarray([spec_dec_metrics.total_accepted_draft_tokens], + dtype=self.output_dtype))) + + response.append( + pb_utils.Tensor( + "total_draft_tokens", + np.asarray([spec_dec_metrics.total_draft_tokens], + dtype=self.output_dtype))) + + return pb_utils.InferenceResponse(output_tensors=response) + + def finalize(self): + """ + Function is called by Triton server before exiting. + + Notes: + - `finalize` is called only once when the model is being unloaded. + - Implementing `finalize` function is optional. + """ + self.logger.log_info("[trtllm] Issuing finalize to trtllm backend") + self._event_loop.call_soon_threadsafe( + self._llm_engine_shutdown_event.set) + + # Shutdown the event thread. + if self._event_thread is not None: + self._event_thread.join() + self._event_thread = None + + # # Shutdown the response thread. + self._response_queue.put(None) + if self._response_thread is not None: + self._response_thread.join() + self._response_thread = None + + if self.cancellation_thread is not None: + self.running = False + self.cancellation_thread.join() + self.cancellation_thread = None + + # When using parallel tensors, the stub process may not shutdown due to + # unreleased references, so manually run the garbage collector once. + self.logger.log_info( + "[trtllm] Running Garbage Collector on finalize...") + gc.collect() + self.logger.log_info("[trtllm] Garbage Collector on finalize... done") diff --git a/Popular_Models_Guide/Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md b/Popular_Models_Guide/Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md index 13d46718..9128bc2a 100644 --- a/Popular_Models_Guide/Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md +++ b/Popular_Models_Guide/Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md @@ -28,164 +28,92 @@ # Deploying Hugging Face Qwen2.5-VL Model in Triton -This guide shows how to serve a multimodal (vision-language) model on Triton +This guide walks through serving a multimodal (vision-language) model on Triton Inference Server using the [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) PyTorch backend through the [LLM API](https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/llm-api/README.md), exposed by Triton's `llmapi` backend. -> [!IMPORTANT] -> **This workflow depends on an unmerged TensorRT-LLM change, plus a small -> patch to run it on today's container.** -> -> Image support in the Triton `llmapi` backend (the optional `image_url` input -> and the `triton_config.multimodal` opt-in used below) is added by -> [NVIDIA/TensorRT-LLM#18381](https://github.com/NVIDIA/TensorRT-LLM/pull/18381), -> which has not been merged and is not present in any released TensorRT-LLM -> version or container image. Until that PR lands you must take the `llmapi` -> backend files from that branch; a stock container will not accept an -> `image_url` input. -> -> Those files call `async_build_multimodal_prompt`, which the same PR adds to -> the `tensorrt_llm` **wheel**. Every published `-trtllm-python-py3` image still -> ships TensorRT-LLM 1.2.1, whose wheel does not have it, so -> [a one-command patch](#patching-modelpy-for-tensorrt-llm-121) is required as -> well. This guide is written for that combination and is verified end to end on -> it; both steps go away only when a container ships a TensorRT-LLM that already -> contains #18381. +Unlike the deprecated multimodal path, there is **no engine build**: no +`trtllm-build`, no separate visual engine. The model repository is four small +files and the weights load straight from a Hugging Face snapshot at startup. > [!NOTE] > This guide replaces > [the Llava1.5 TensorRT-LLM guide](../Llava1.5/llava_trtllm_guide.md), which -> uses the prebuilt-TensorRT-engine multimodal path that TensorRT-LLM has -> declared end-of-life as of TensorRT-LLM v1.2. See +> uses the prebuilt-TensorRT-engine multimodal path that TensorRT-LLM declared +> end-of-life in v1.2. See > [triton-inference-server/server#8945](https://github.com/triton-inference-server/server/issues/8945). -## Why the PyTorch backend +This guide was tested with `Qwen/Qwen2.5-VL-3B-Instruct` on 1x NVIDIA B200, +using `nvcr.io/nvidia/tritonserver:26.07-trtllm-python-py3` +(Triton 2.71.0, TensorRT-LLM 1.2.1). -The deprecated multimodal path (`tensorrtllm_backend`'s `all_models/multimodal`) -required two ahead-of-time compilation steps before you could serve anything: a -`trtllm-build` invocation to produce the LLM engine, and a separate visual -engine build for the vision encoder. Both artifacts had to be rebuilt whenever -the model, precision, or maximum sequence length changed. +## Files provided with this guide -The PyTorch backend needs **no compilation and no engine build at all**. The -model repository is four plain Python/text files, TensorRT-LLM is a -pip-installed wheel inside the container, and the weights are loaded directly -from a Hugging Face snapshot at startup. This is the single biggest practical -difference between the two workflows. +The `llmapi` backend in TensorRT-LLM v1.2.1 does not accept image input yet; +that is proposed in +[NVIDIA/TensorRT-LLM#18381](https://github.com/NVIDIA/TensorRT-LLM/pull/18381). +Two files here add it on top of v1.2.1, which is the exact TensorRT-LLM version +installed in the container: -LLaVA-1.5 itself is not a drop-in replacement target here. TensorRT-LLM's -[supported models matrix](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/models/supported-models.md) -lists `LlavaNextForConditionalGeneration` and `LlavaLlamaModel` (VILA) among the -supported multimodal architectures, but not `LlavaForConditionalGeneration`, -which is the architecture of `llava-hf/llava-1.5-7b-hf`. This guide therefore -uses [`Qwen/Qwen2.5-VL-3B-Instruct`](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct). +1. [model.py](./model.py) - v1.2.1's backend plus the optional `image_url` input. +2. [config.pbtxt](./config.pbtxt) - v1.2.1's config declaring `image_url`. -## What was validated +The other two files in the model repository come from v1.2.1 unchanged. -| Item | Value | -| ---- | ----- | -| Container | `nvcr.io/nvidia/tritonserver:26.07-trtllm-python-py3` | -| Triton | 2.71.0 | -| TensorRT-LLM | 1.2.1 (with the [`model.py` patch](#patching-modelpy-for-tensorrt-llm-121)) | -| torch | 2.10.0a0+b4e4ee81d3.nv25.12 | -| CUDA | 13.1 | -| Model | `Qwen/Qwen2.5-VL-3B-Instruct` | -| Hardware | 1x NVIDIA B200 | +## Launch Triton TensorRT-LLM container -Every command and every response below was run on that configuration. `26.07` -is the newest `-trtllm-python-py3` tag; on it, the multimodal path does not work -without the patch. - -## Prerequisites - -### Container - -Start from a clone of this repository, so that the -[`trtllm_121_compat.py`](trtllm_121_compat.py) used below is mounted into the -container along with it: +Start from a clone of this repository, so the two files above are available +inside the container: ```bash git clone https://github.com/triton-inference-server/tutorials.git cd tutorials -docker run --rm -it --gpus all --network host \ - -v ${PWD}:/workspace -w /workspace \ +docker run --rm -it --net host --shm-size=2g \ + --ulimit memlock=-1 --ulimit stack=67108864 --gpus all \ + -v ${PWD}:/tutorials \ + -w /workspace \ nvcr.io/nvidia/tritonserver:26.07-trtllm-python-py3 ``` -### Known issue: the container's `openai` package is too old +## Update the `openai` package -The 26.07 image ships `openai 1.107.3`, which is older than what -`tensorrt_llm/serve/responses_utils.py` requires. Loading a model fails with: - -``` -ImportError: cannot import name 'PartReasoningText' -``` - -Because `tensorrt_llm/_torch/pyexecutor/py_executor.py` imports -`tensorrt_llm.serve`, this breaks loading of **any** model on the `llmapi` -backend, not just multimodal ones. Work around it by installing a newer `openai` -into an overlay directory and putting that directory on `PYTHONPATH`, which -avoids modifying the container's site-packages: +The container ships `openai 1.107.3`, which is too old for +`tensorrt_llm.serve`. Because the PyTorch executor imports that module +unconditionally, **no** model loads on the `llmapi` backend until this is fixed: ```bash pip install --target=/workspace/pylibs -U openai export PYTHONPATH=/workspace/pylibs ``` -### Model weights - -Provide either a local Hugging Face snapshot directory or the Hugging Face model -id `Qwen/Qwen2.5-VL-3B-Instruct`. If you use the model id, the container needs -network access to huggingface.co at startup. - -## Preparing the model repository +## Build the model repository -The Triton backend sources live in the -[NVIDIA/TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) repository under -`triton_backend/`; the standalone `tensorrtllm_backend` repository has been -superseded. They are plain Python files and are not shipped in the -`tensorrt_llm` wheel, so fetch them from a checkout. +Fetch the backend files from the TensorRT-LLM v1.2.1 tag. Only one directory is +needed, so skip the Git LFS payload — this takes a few seconds: -`triton_backend/all_models/llmapi/` contains exactly one model directory -(`tensorrt_llm/`), so it doubles as a Triton model repository and needs no -copying: +```bash +GIT_LFS_SKIP_SMUDGE=1 git clone --depth 1 --filter=blob:none --sparse \ + --branch v1.2.1 https://github.com/NVIDIA/TensorRT-LLM.git /workspace/trtllm +git -C /workspace/trtllm sparse-checkout set triton_backend/all_models/llmapi -``` -all_models/llmapi/ <- point --model-repository here -└── tensorrt_llm/ - ├── config.pbtxt - └── 1/ - ├── model.py - ├── helpers.py - └── model.yaml +cp -r /workspace/trtllm/triton_backend/all_models/llmapi /workspace/model_repository ``` -> [!NOTE] -> Until [NVIDIA/TensorRT-LLM#18381](https://github.com/NVIDIA/TensorRT-LLM/pull/18381) -> merges, the `image_url` input and the `triton_config.multimodal` option below -> exist only on that pull request's branch. Clone the fork shown here for now; -> once it lands, clone `https://github.com/NVIDIA/TensorRT-LLM.git` instead. - -Only four files are needed, so skip the repository's Git LFS payload and check -out the one directory — a few seconds and about 9 MB, rather than the ~900 MB a -full clone pulls: +Copy in the two files provided with this guide: ```bash -GIT_LFS_SKIP_SMUDGE=1 git clone --depth 1 --filter=blob:none --sparse \ - --branch feat/triton-llmapi-multimodal-image \ - https://github.com/faradawn/TensorRT-LLM.git /workspace/trtllm-pr -git -C /workspace/trtllm-pr sparse-checkout set triton_backend/all_models/llmapi +QWEN_DIR=/tutorials/Popular_Models_Guide/Qwen2.5-VL +cp ${QWEN_DIR}/model.py /workspace/model_repository/tensorrt_llm/1/model.py +cp ${QWEN_DIR}/config.pbtxt /workspace/model_repository/tensorrt_llm/config.pbtxt ``` -Then point `1/model.yaml` at the model and turn on the multimodal opt-in. This -edits the file in place inside the checkout, which leaves that clone's -`git status` dirty — fine for a throwaway container: +Then write `model.yaml`, which selects the model and turns on image input: ```bash -cat > /workspace/trtllm-pr/triton_backend/all_models/llmapi/tensorrt_llm/1/model.yaml <<'EOF' +cat > /workspace/model_repository/tensorrt_llm/1/model.yaml <<'EOF' model: Qwen/Qwen2.5-VL-3B-Instruct backend: "pytorch" tensor_parallel_size: 1 @@ -199,154 +127,50 @@ triton_config: EOF ``` -`model` accepts a Hugging Face model id (downloaded to `HF_HOME`) or a local -snapshot directory. +The repository now looks like this: -`triton_config.multimodal` defaults to `False`. This is deliberate: existing -deployments that already declare their own `image_url` input keep their current -behavior when they upgrade. The flip side is that if you forget to set it, any -`image_url` values you send are **silently ignored** and you get a text-only -answer, so set it explicitly for multimodal models. - -## Patching `model.py` for TensorRT-LLM 1.2.1 - -The backend files you just cloned build their prompt by calling -`async_build_multimodal_prompt`, which -[#18381](https://github.com/NVIDIA/TensorRT-LLM/pull/18381) adds to -`tensorrt_llm/inputs/utils.py`. That module ships **inside the `tensorrt_llm` -wheel**, not in the `triton_backend/` tree you cloned, and the container's wheel -is 1.2.1 — so you have the caller but never the callee. - -This is easy to miss, because nothing fails at startup. The server comes up, logs -`multimodal input enabled`, and answers text-only prompts correctly. Only -requests that actually carry an image fail: - -```json -{"error":"Error generating request: cannot import name 'async_build_multimodal_prompt' from 'tensorrt_llm.inputs' (/opt/venv-tritonserver/lib/python3.12/site-packages/tensorrt_llm/inputs/__init__.py)"} -``` - -Copying the new `utils.py` across does not help either: 1.2.1 lacks everything -that helper is built on — `MEDIA_IO_REGISTRY`, `ContentFormat`, -`MultimodalDataTracker.item_order()`, `interleave_mm_placeholders` and -`async_apply_chat_template`. What does work is replacing that one call with an -equivalent written against the 1.2.1 API. Run the script shipped next to this -guide: - -```bash -python3 /workspace/Popular_Models_Guide/Qwen2.5-VL/trtllm_121_compat.py \ - /workspace/trtllm-pr/triton_backend/all_models/llmapi/tensorrt_llm/1/model.py ``` - -``` -Patched .../llmapi/tensorrt_llm/1/model.py for TensorRT-LLM 1.2.1. -``` - -It edits nothing but that one file, refuses to write source that does not parse, -and is safe to re-run — a second invocation reports `already patched; nothing to -do`. If the call it looks for is gone, it says so and tells you how to check -whether your container already has the function, rather than corrupting the -model repository. - -### What the script changes - -It adds one method, `_build_multimodal_prompt_121`, and points the call site at -it: - -```diff - image_url = get_input_tensor_by_name(request, 'image_url') - if image_url is not None and image_url.size > 0: -- from tensorrt_llm.inputs import async_build_multimodal_prompt -- - media = [ - url.decode("utf-8") if isinstance(url, bytes) else str(url) - for url in image_url.reshape(-1) - ] - validate_media_urls(media) -- prompt = await async_build_multimodal_prompt( -- model_type=self._mm_model_type, -- tokenizer=self._mm_tokenizer, -- processor=self._mm_processor, -- prompt=prompt, -- media=media, -- modality="image", -- ) -+ prompt = await self._build_multimodal_prompt_121(prompt, media) +model_repository/ +└── tensorrt_llm/ + ├── config.pbtxt <- provided here + └── 1/ + ├── model.py <- provided here + ├── helpers.py <- from v1.2.1 + └── model.yaml <- written above ``` -The new method does what the 1.3 helper does, in 1.2.1's vocabulary: - -| Step | 1.3 helper | 1.2.1 equivalent used here | -| ---- | ---------- | -------------------------- | -| download images | `MEDIA_IO_REGISTRY` | `async_load_image` per URL, gathered | -| insert placeholders | `interleave_mm_placeholders`, `item_order()` | `add_multimodal_placeholders`, three-argument form | -| render chat template | `async_apply_chat_template` | `apply_chat_template` via `asyncio.to_thread` | -| build the prompt | returns `PromptInputs` | `prompt_inputs(...)` plus `multi_modal_data` | - -`apply_chat_template` is synchronous and does real tokenizer work, so it goes -through `asyncio.to_thread` rather than blocking the engine's event loop while -the images are still downloading. Read -[`trtllm_121_compat.py`](trtllm_121_compat.py) for the full method. - -Nothing else in the backend needs touching: `validate_media_urls` and the rest -of the request path run unmodified on 1.2.1. - -> [!NOTE] -> Delete this step once a `-trtllm-python-py3` container ships a TensorRT-LLM -> that already contains #18381. Note that #18381 merging is **not** enough on its -> own — the 26.07 image's wheel stays at 1.2.1 no matter what lands upstream, so -> the patch is needed until a *new image* ships. +`model` takes a Hugging Face model id (downloaded to `HF_HOME`) or a local +snapshot directory. -## Starting the server +`triton_config.multimodal` defaults to `False`. When it is not set, `image_url` +values are ignored and you get a text-only answer, so set it for multimodal +models. -In Slurm/MPI environments, launch through `trtllm-llmapi-launch`: +## Serving with Triton ```bash trtllm-llmapi-launch tritonserver \ - --model-repository=/workspace/trtllm-pr/triton_backend/all_models/llmapi \ + --model-repository=/workspace/model_repository \ --http-port=8000 --grpc-port=8001 --metrics-port=8002 ``` -Running plain `tritonserver` fails at engine start with: +`trtllm-llmapi-launch` is required: the LLM API spawns its workers with +`MpiPoolSession`, and plain `tritonserver` fails with `MPI_ERR_SPAWN`. -``` -mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes -``` - -The LLM API uses `MpiPoolSession` to spawn its workers, and -`trtllm-llmapi-launch` (which sets `TLLM_SPAWN_PROXY_PROCESS=1`) is the -supported wrapper for that. - -Startup takes roughly 70 seconds. Wait for `Started HTTPService` in the log. A -successful multimodal start also logs: +Startup takes about a minute. The server is ready when the log shows: ``` [trtllm] multimodal input enabled for model_type 'qwen2_5_vl' +Started HTTPService at 0.0.0.0:8000 ``` -You can poll readiness with: +You can also poll for readiness: ```bash -curl -s -o /dev/null -w '%{http_code}' http://localhost:8000/v2/health/ready +curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8000/v2/health/ready ``` -which returns `200` once the server is up. - -## Sending an inference request - -Requests go to the standard Triton HTTP inference endpoint, -`POST /v2/models/tensorrt_llm/infer`. Inputs are Triton tensors, not OpenAI-style -chat JSON: - -| Input | Datatype | Shape | Description | -| ----- | -------- | ----- | ----------- | -| `text_input` | `BYTES` | `[1]` | The plain question. The backend applies the chat template and inserts the per-architecture image placeholders, so do **not** add `<\|vision_start\|>` or similar tokens yourself. | -| `image_url` | `BYTES` | `[N]` | One entry per image. Accepts `http(s)` URLs the server can reach; local paths and other schemes are rejected. | -| `sampling_param_max_tokens` | `INT32` | `[1]` | Maximum number of tokens to generate. | -| `sampling_param_exclude_input_from_output` | `BOOL` | `[1]` | Set to `true`; otherwise the rendered prompt is echoed back in `text_output`. | - -The only output is `text_output`. - -### Quick check with `curl` +## Send an inference request ```bash curl -s http://localhost:8000/v2/models/tensorrt_llm/infer -H 'Content-Type: application/json' -d '{ @@ -364,113 +188,40 @@ curl -s http://localhost:8000/v2/models/tensorrt_llm/infer -H 'Content-Type: app {"model_name":"tensorrt_llm","model_version":"1","outputs":[{"name":"text_output","datatype":"BYTES","shape":[1],"data":["The bus is yellow and white, and the sign on the bus says \"Out of Service.\""]}]} ``` -### Python client - -This client uses only the standard library: - -```python -import json -import urllib.request - -URL = "http://localhost:8000/v2/models/tensorrt_llm/infer" - - -def ask(prompt, images, max_tokens=64): - """Send a prompt plus one or more images and return the generated text.""" - body = { - "inputs": [ - {"name": "text_input", "shape": [1], "datatype": "BYTES", - "data": [prompt]}, - {"name": "image_url", "shape": [len(images)], "datatype": "BYTES", - "data": images}, - {"name": "sampling_param_max_tokens", "shape": [1], - "datatype": "INT32", "data": [max_tokens]}, - {"name": "sampling_param_exclude_input_from_output", "shape": [1], - "datatype": "BOOL", "data": [True]}, - ], - "outputs": [{"name": "text_output"}], - } - request = urllib.request.Request( - URL, - data=json.dumps(body).encode(), - headers={"Content-Type": "application/json"}, - ) - with urllib.request.urlopen(request, timeout=300) as response: - result = json.load(response) - return result["outputs"][0]["data"][0].strip() - - -if __name__ == "__main__": - print(ask( - "What color is the bus and what does the sign say?", - ["http://images.cocodataset.org/test2017/000000155781.jpg"], - )) -``` - -Expected output: +To send more than one image, pass more entries and match the shape: -``` -The bus is yellow and white, and the sign on the bus says "Out of Service." +```bash +{"name":"image_url","shape":[2],"datatype":"BYTES","data":["http://images.cocodataset.org/test2017/000000155781.jpg","http://images.cocodataset.org/val2017/000000039769.jpg"]} ``` -### Multiple images - -Pass more than one entry in `image_url`; the shape must match the number of -entries. Every entry must be an `http(s)` URL — see -[Allowed scope of access](#allowed-scope-of-access): - -```python -ask( - "Describe each image.", - [ - "http://images.cocodataset.org/test2017/000000155781.jpg", - "http://images.cocodataset.org/val2017/000000039769.jpg", - ], - max_tokens=96, -) -``` +The model describes the images in the order they were sent. -The model enumerates both images and describes each one in the order they were -sent: +## Request inputs -``` -The first image depicts a bus on a foggy street at night. The bus has a sign on -its front that reads "OUT OF SERVICE." ... The second image shows two cats lying -on a pink couch. -``` +| Input | Datatype | Shape | Description | +| ----- | -------- | ----- | ----------- | +| `text_input` | `BYTES` | `[1]` | The question. The backend applies the chat template and inserts the image placeholders, so do not add `<\|vision_start\|>` or similar tokens yourself. | +| `image_url` | `BYTES` | `[N]` | One entry per image. Only `http(s)` URLs are accepted. | +| `sampling_param_max_tokens` | `INT32` | `[1]` | Maximum tokens to generate. | +| `sampling_param_exclude_input_from_output` | `BOOL` | `[1]` | Set `true`, otherwise the rendered prompt is echoed back. | -### Allowed scope of access +The only output is `text_output`. `image_url` is client-controlled, so only `http(s)` URLs are accepted. Local -filesystem paths, `file://` and other schemes are rejected, because accepting -them would let a caller make the server read image files its process can open. -Host images the model should see on a reachable web URL. - -A rejected entry fails the whole request: - -```json -{"error":"Error generating request: Unsupported image_url '/workspace/images/second.jpg': only http, https URLs are accepted."} -``` - -### Error behavior - -An unreachable image URL surfaces as a Triton error rather than silently -degrading to a text-only answer: - -```json -{"error":"Error generating request: Cannot connect to host example.invalid:443 ssl:default [Name or service not known]"} -``` +paths and `file://` are rejected, because accepting them would let a caller make +the server read image files its process can open. Host images the model should +see on a reachable web URL. ## Troubleshooting | Symptom | Cause and fix | | ------- | ------------- | -| `mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes` | `tritonserver` was started directly. The LLM API spawns workers via `MpiPoolSession`; start it with `trtllm-llmapi-launch` instead. | -| `ImportError: cannot import name 'PartReasoningText'` | The container's `openai` package is too old for `tensorrt_llm.serve`, which is imported unconditionally by the PyTorch executor. Install a newer `openai` into an overlay directory and export it on `PYTHONPATH` (see [Prerequisites](#known-issue-the-containers-openai-package-is-too-old)). | -| `ConnectionRefusedError` from the client | The server is not up yet. Startup takes roughly 70 seconds; wait for `Started HTTPService` in the log, or poll until `curl -s -o /dev/null -w '%{http_code}' http://localhost:8000/v2/health/ready` returns `200`. | -| Images appear to be ignored and answers are text-only | `triton_config.multimodal` is not set to `True` in `1/model.yaml`. It defaults to `False` and image inputs are silently dropped. | -| `cannot import name 'async_build_multimodal_prompt' from 'tensorrt_llm.inputs'`, only on requests carrying an image | The container's TensorRT-LLM wheel predates [#18381](https://github.com/NVIDIA/TensorRT-LLM/pull/18381). The server starts and text-only requests still work, which makes this easy to miss. Apply [the 1.2.1 patch](#patching-modelpy-for-tensorrt-llm-121). | -| `Unsupported image_url '...': only http, https URLs are accepted.` | A local path, `file://` or other scheme was passed. Only `http(s)` is accepted; see [Allowed scope of access](#allowed-scope-of-access). | +| `ImportError: cannot import name 'PartReasoningText'` | The container's `openai` is too old. See [Update the `openai` package](#update-the-openai-package). | +| `mpi4py.MPI.Exception: MPI_ERR_SPAWN` | `tritonserver` was started directly; use `trtllm-llmapi-launch`. | +| `cannot import name 'async_build_multimodal_prompt'` | `model.py` was taken from the #18381 branch rather than the copy provided here. That branch targets TensorRT-LLM 1.3 and calls a function the 1.2.1 wheel does not have. | +| Answers ignore the image | `triton_config.multimodal` is not `True` in `model.yaml`. | +| `Unsupported image_url '...'` | A local path or non-`http(s)` scheme was passed. | +| `ConnectionRefusedError` from the client | The server is still starting; wait for `Started HTTPService`. | ## References diff --git a/Popular_Models_Guide/Qwen2.5-VL/trtllm_121_compat.py b/Popular_Models_Guide/Qwen2.5-VL/trtllm_121_compat.py deleted file mode 100644 index b9c35ec6..00000000 --- a/Popular_Models_Guide/Qwen2.5-VL/trtllm_121_compat.py +++ /dev/null @@ -1,181 +0,0 @@ -# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. -# -# Redistribution and use in source and binary forms, with or without -# modification, are permitted provided that the following conditions -# are met: -# * Redistributions of source code must retain the above copyright -# notice, this list of conditions and the following disclaimer. -# * Redistributions in binary form must reproduce the above copyright -# notice, this list of conditions and the following disclaimer in the -# documentation and/or other materials provided with the distribution. -# * Neither the name of NVIDIA CORPORATION nor the names of its -# contributors may be used to endorse or promote products derived -# from this software without specific prior written permission. -# -# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY -# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE -# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR -# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR -# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, -# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, -# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR -# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY -# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE -# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -"""Make the Triton ``llmapi`` backend's image path run on TensorRT-LLM 1.2.1. - -Used by ``qwen2_5_vl_trtllm_guide.md``. See the guide for the full explanation; -the short version is that the backend's ``model.py`` builds its prompt with -``tensorrt_llm.inputs.async_build_multimodal_prompt``, which is added by -NVIDIA/TensorRT-LLM#18381. That function lives in the ``tensorrt_llm`` *wheel*, -not in the ``triton_backend/`` tree you clone, so on a container whose wheel is -1.2.1 you have the caller but never the callee, and every request carrying an -``image_url`` fails with:: - - cannot import name 'async_build_multimodal_prompt' from 'tensorrt_llm.inputs' - -1.2.1 also lacks everything that helper is built on -- ``MEDIA_IO_REGISTRY``, -``ContentFormat``, ``MultimodalDataTracker.item_order()``, -``interleave_mm_placeholders`` and ``async_apply_chat_template`` -- so copying -the new ``utils.py`` across is not an option either. This script instead swaps -the single call for an equivalent written against the 1.2.1 API surface. - -Usage:: - - python3 trtllm_121_compat.py /triton_backend/all_models/llmapi/tensorrt_llm/1/model.py - -Safe to re-run: it exits cleanly if the file is already patched. Delete this -step once a ``-trtllm-python-py3`` container ships a TensorRT-LLM that already -contains #18381. -""" - -import argparse -import ast -import pathlib -import sys - -# The call this replaces, exactly as it appears in model.py. -OLD_CALL = """ from tensorrt_llm.inputs import async_build_multimodal_prompt - - media = [ - url.decode("utf-8") if isinstance(url, bytes) else str(url) - for url in image_url.reshape(-1) - ] - validate_media_urls(media) - prompt = await async_build_multimodal_prompt( - model_type=self._mm_model_type, - tokenizer=self._mm_tokenizer, - processor=self._mm_processor, - prompt=prompt, - media=media, - modality="image", - )""" - -NEW_CALL = """ media = [ - url.decode("utf-8") if isinstance(url, bytes) else str(url) - for url in image_url.reshape(-1) - ] - validate_media_urls(media) - prompt = await self._build_multimodal_prompt_121(prompt, media)""" - -# Inserted immediately above `async def _convert_request`. `asyncio` is already -# imported at module scope in model.py, so this needs no new top-level imports. -NEW_METHOD = ''' async def _build_multimodal_prompt_121(self, text, media): - """Stand-in for `inputs.async_build_multimodal_prompt` on TRT-LLM 1.2.1. - - 1.2.1 has no `async_apply_chat_template` and no - `MultimodalDataTracker.item_order()`, and its - `add_multimodal_placeholders` takes three arguments rather than four. - """ - from tensorrt_llm.inputs import prompt_inputs - from tensorrt_llm.inputs.utils import (ConversationMessage, - MultimodalDataTracker, - add_multimodal_placeholders, - apply_chat_template, - async_load_image) - - mm_data_tracker = MultimodalDataTracker(self._mm_model_type) - for url in media: - mm_data_tracker.add_data("image", async_load_image(url)) - mm_placeholder_counts = mm_data_tracker.placeholder_counts() - - content = add_multimodal_placeholders(self._mm_model_type, text, - mm_placeholder_counts) - conversation = [ - ConversationMessage(role="user", content=content, media=[]) - ] - # `apply_chat_template` is synchronous and does real tokenizer work, so - # keep it off the engine's event loop while the images download. - prompt_task = asyncio.to_thread( - apply_chat_template, - model_type=self._mm_model_type, - tokenizer=self._mm_tokenizer, - processor=self._mm_processor, - conversation=conversation, - add_generation_prompt=True, - mm_placeholder_counts=[mm_placeholder_counts], - ) - prompt, (mm_data, _) = await asyncio.gather( - prompt_task, mm_data_tracker.retrieve_all_async()) - - prompt = prompt_inputs(prompt) - if mm_data: - prompt["multi_modal_data"] = mm_data - return prompt - -''' - -ANCHOR = " async def _convert_request(self, request):" - -MOVED_ON = """{path} does not contain the call this script replaces. - -That usually means the backend has moved on -- most likely #18381 merged, in -which case check whether your container's TensorRT-LLM already provides -`async_build_multimodal_prompt` and skip this step entirely: - - python3 -c "from tensorrt_llm.inputs import async_build_multimodal_prompt" - -If that import succeeds, no patch is needed.""" - - -def main(): - parser = argparse.ArgumentParser( - description="Patch the Triton llmapi backend's model.py for " - "TensorRT-LLM 1.2.1.") - parser.add_argument( - "model_py", - type=pathlib.Path, - help="path to all_models/llmapi/tensorrt_llm/1/model.py") - args = parser.parse_args() - - path = args.model_py - if not path.is_file(): - sys.exit(f"{path} is not a file") - - source = path.read_text() - - if "_build_multimodal_prompt_121" in source: - print(f"{path} is already patched; nothing to do.") - return - - if source.count(OLD_CALL) != 1: - sys.exit(MOVED_ON.format(path=path)) - if source.count(ANCHOR) != 1: - sys.exit(f"could not locate `{ANCHOR.strip()}` in {path}") - - source = source.replace(OLD_CALL, NEW_CALL) - source = source.replace(ANCHOR, NEW_METHOD + ANCHOR, 1) - - # Fail before writing rather than leave a half-broken model repository. - try: - ast.parse(source) - except SyntaxError as exc: - sys.exit(f"patched source does not parse ({exc}); model.py left alone") - - path.write_text(source) - print(f"Patched {path} for TensorRT-LLM 1.2.1.") - - -if __name__ == "__main__": - main() From 5a1dc1fed7bcd88dc6bbca9762ba4f8436e6bfe7 Mon Sep 17 00:00:00 2001 From: Faradawn Yang <73060648+faradawn@users.noreply.github.com> Date: Thu, 10 Sep 2026 08:13:28 -0700 Subject: [PATCH 08/10] docs: point model.py's provenance note at a command a reader can run The header ended with "Regenerate with tutorials/Popular_Models_Guide/ Qwen2.5-VL, see the guide", which pointed at nothing: no generator is shipped and the guide does not describe one. model.py is a plain file to copy, and that is the whole intent. Replace that line with the two-command diff against the stock v1.2.1 file, and name the four additions, so a reader can audit what changed without needing any tooling from us. Co-Authored-By: Claude Opus 5 --- Popular_Models_Guide/Qwen2.5-VL/model.py | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/Popular_Models_Guide/Qwen2.5-VL/model.py b/Popular_Models_Guide/Qwen2.5-VL/model.py index 61b6ca8e..e9ba8651 100644 --- a/Popular_Models_Guide/Qwen2.5-VL/model.py +++ b/Popular_Models_Guide/Qwen2.5-VL/model.py @@ -32,8 +32,14 @@ # with image input added, as proposed in NVIDIA/TensorRT-LLM#18381. # # v1.2.1 is the TensorRT-LLM shipped in nvcr.io/nvidia/tritonserver:26.07- -# trtllm-python-py3, so this matches the wheel installed in the container. -# Regenerate with tutorials/Popular_Models_Guide/Qwen2.5-VL, see the guide. +# trtllm-python-py3, so everything outside that feature matches the wheel +# installed in the container. To see exactly what was added: +# +# git show v1.2.1:triton_backend/all_models/llmapi/tensorrt_llm/1/model.py \ +# > /tmp/stock_model.py && diff /tmp/stock_model.py model.py +# +# The additions are `validate_media_urls`, `_init_multimodal`, +# `_build_multimodal_prompt`, and the `image_url` block in `_convert_request`. # --------------------------------------------------------------------------- import asyncio From aa25e5cac5da3676fb1f1ca7dd447f2887225fae Mon Sep 17 00:00:00 2001 From: Faradawn Yang <73060648+faradawn@users.noreply.github.com> Date: Thu, 10 Sep 2026 10:16:20 -0700 Subject: [PATCH 09/10] docs: tighten the Qwen2.5-VL guide Cut the framing that does not help someone deploying the model: the comparison with the deprecated engine-build path, the note that this supersedes the Llava1.5 guide, the tested-configuration paragraph, the MPI_ERR_SPAWN explanation, and the troubleshooting and references sections. Say only that the guide uses 26.07, which is the newest -trtllm-python-py3 tag on NGC. Reduce the openai section to why and how, one line each. NVIDIA/TensorRT-LLM#18381 has merged, so describe the two provided files by what they are rather than by a pending pull request: the 26.07 container ships TensorRT-LLM v1.2.1, whose llmapi backend has no image input, and these two files add it for that version, adapted from main. Once a container ships v1.3.0 or newer, take model.py and config.pbtxt from main and skip both. 231 lines to 198. Re-verified end to end on 1x B200 after the trim. Co-Authored-By: Claude Opus 5 --- Popular_Models_Guide/Qwen2.5-VL/model.py | 3 +- .../Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md | 53 ++++--------------- 2 files changed, 12 insertions(+), 44 deletions(-) diff --git a/Popular_Models_Guide/Qwen2.5-VL/model.py b/Popular_Models_Guide/Qwen2.5-VL/model.py index e9ba8651..8d2b0774 100644 --- a/Popular_Models_Guide/Qwen2.5-VL/model.py +++ b/Popular_Models_Guide/Qwen2.5-VL/model.py @@ -29,7 +29,8 @@ # # This file is TensorRT-LLM v1.2.1's Triton `llmapi` backend # triton_backend/all_models/llmapi/tensorrt_llm/1/model.py -# with image input added, as proposed in NVIDIA/TensorRT-LLM#18381. +# with image input added, backported from TensorRT-LLM main +# (NVIDIA/TensorRT-LLM#18381). # # v1.2.1 is the TensorRT-LLM shipped in nvcr.io/nvidia/tritonserver:26.07- # trtllm-python-py3, so everything outside that feature matches the wheel diff --git a/Popular_Models_Guide/Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md b/Popular_Models_Guide/Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md index 9128bc2a..84645ce0 100644 --- a/Popular_Models_Guide/Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md +++ b/Popular_Models_Guide/Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md @@ -34,32 +34,21 @@ Inference Server using the the [LLM API](https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/llm-api/README.md), exposed by Triton's `llmapi` backend. -Unlike the deprecated multimodal path, there is **no engine build**: no -`trtllm-build`, no separate visual engine. The model repository is four small -files and the weights load straight from a Hugging Face snapshot at startup. - -> [!NOTE] -> This guide replaces -> [the Llava1.5 TensorRT-LLM guide](../Llava1.5/llava_trtllm_guide.md), which -> uses the prebuilt-TensorRT-engine multimodal path that TensorRT-LLM declared -> end-of-life in v1.2. See -> [triton-inference-server/server#8945](https://github.com/triton-inference-server/server/issues/8945). - -This guide was tested with `Qwen/Qwen2.5-VL-3B-Instruct` on 1x NVIDIA B200, -using `nvcr.io/nvidia/tritonserver:26.07-trtllm-python-py3` -(Triton 2.71.0, TensorRT-LLM 1.2.1). +It uses `nvcr.io/nvidia/tritonserver:26.07-trtllm-python-py3`, the latest +`-trtllm-python-py3` container on NGC. ## Files provided with this guide -The `llmapi` backend in TensorRT-LLM v1.2.1 does not accept image input yet; -that is proposed in -[NVIDIA/TensorRT-LLM#18381](https://github.com/NVIDIA/TensorRT-LLM/pull/18381). -Two files here add it on top of v1.2.1, which is the exact TensorRT-LLM version -installed in the container: +The 26.07 container ships TensorRT-LLM v1.2.1, whose `llmapi` backend does not +accept image input yet, so two files here add it for that version: 1. [model.py](./model.py) - v1.2.1's backend plus the optional `image_url` input. 2. [config.pbtxt](./config.pbtxt) - v1.2.1's config declaring `image_url`. +Both are adapted from the image-input support now on TensorRT-LLM `main`. Once a +container ships TensorRT-LLM v1.3.0 or newer, use the `model.py` and +`config.pbtxt` from `main` directly and skip both files. + The other two files in the model repository come from v1.2.1 unchanged. ## Launch Triton TensorRT-LLM container @@ -80,9 +69,8 @@ docker run --rm -it --net host --shm-size=2g \ ## Update the `openai` package -The container ships `openai 1.107.3`, which is too old for -`tensorrt_llm.serve`. Because the PyTorch executor imports that module -unconditionally, **no** model loads on the `llmapi` backend until this is fixed: +This container's `openai` is too old for `tensorrt_llm.serve`, so update it +before loading a model: ```bash pip install --target=/workspace/pylibs -U openai @@ -154,9 +142,6 @@ trtllm-llmapi-launch tritonserver \ --http-port=8000 --grpc-port=8001 --metrics-port=8002 ``` -`trtllm-llmapi-launch` is required: the LLM API spawns its workers with -`MpiPoolSession`, and plain `tritonserver` fails with `MPI_ERR_SPAWN`. - Startup takes about a minute. The server is ready when the log shows: ``` @@ -211,21 +196,3 @@ The only output is `text_output`. paths and `file://` are rejected, because accepting them would let a caller make the server read image files its process can open. Host images the model should see on a reachable web URL. - -## Troubleshooting - -| Symptom | Cause and fix | -| ------- | ------------- | -| `ImportError: cannot import name 'PartReasoningText'` | The container's `openai` is too old. See [Update the `openai` package](#update-the-openai-package). | -| `mpi4py.MPI.Exception: MPI_ERR_SPAWN` | `tritonserver` was started directly; use `trtllm-llmapi-launch`. | -| `cannot import name 'async_build_multimodal_prompt'` | `model.py` was taken from the #18381 branch rather than the copy provided here. That branch targets TensorRT-LLM 1.3 and calls a function the 1.2.1 wheel does not have. | -| Answers ignore the image | `triton_config.multimodal` is not `True` in `model.yaml`. | -| `Unsupported image_url '...'` | A local path or non-`http(s)` scheme was passed. | -| `ConnectionRefusedError` from the client | The server is still starting; wait for `Started HTTPService`. | - -## References - -- [TensorRT-LLM LLM API](https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/llm-api/README.md) -- [TensorRT-LLM supported models](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/models/supported-models.md) -- [NVIDIA/TensorRT-LLM#18381](https://github.com/NVIDIA/TensorRT-LLM/pull/18381) - adds multimodal input to the Triton `llmapi` backend -- [Qwen2.5-VL-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct) From fa86b6530d20d0b6506e927751006823c18afa56 Mon Sep 17 00:00:00 2001 From: Faradawn Yang <73060648+faradawn@users.noreply.github.com> Date: Thu, 10 Sep 2026 10:19:35 -0700 Subject: [PATCH 10/10] docs: trim three more lines from the Qwen2.5-VL guide Drop the claim that 26.07 is the newest tag on NGC, which would go stale on the next release; name the container and leave it there. Drop the sentence about where the two provided files were adapted from, keeping only what a reader acts on: once a Triton container ships TensorRT-LLM v1.3.0, copy model.py and config.pbtxt from main and skip them. Drop the rationale behind the http(s) restriction and keep the rule itself. Prose only; no command changed. Co-Authored-By: Claude Opus 5 --- .../Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/Popular_Models_Guide/Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md b/Popular_Models_Guide/Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md index 84645ce0..eb7553cb 100644 --- a/Popular_Models_Guide/Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md +++ b/Popular_Models_Guide/Qwen2.5-VL/qwen2_5_vl_trtllm_guide.md @@ -34,8 +34,7 @@ Inference Server using the the [LLM API](https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/llm-api/README.md), exposed by Triton's `llmapi` backend. -It uses `nvcr.io/nvidia/tritonserver:26.07-trtllm-python-py3`, the latest -`-trtllm-python-py3` container on NGC. +It uses `nvcr.io/nvidia/tritonserver:26.07-trtllm-python-py3`. ## Files provided with this guide @@ -45,8 +44,7 @@ accept image input yet, so two files here add it for that version: 1. [model.py](./model.py) - v1.2.1's backend plus the optional `image_url` input. 2. [config.pbtxt](./config.pbtxt) - v1.2.1's config declaring `image_url`. -Both are adapted from the image-input support now on TensorRT-LLM `main`. Once a -container ships TensorRT-LLM v1.3.0 or newer, use the `model.py` and +Once a Triton container ships TensorRT-LLM v1.3.0, we can copy `model.py` and `config.pbtxt` from `main` directly and skip both files. The other two files in the model repository come from v1.2.1 unchanged. @@ -192,7 +190,5 @@ The model describes the images in the order they were sent. The only output is `text_output`. -`image_url` is client-controlled, so only `http(s)` URLs are accepted. Local -paths and `file://` are rejected, because accepting them would let a caller make -the server read image files its process can open. Host images the model should -see on a reachable web URL. +`image_url` is client-controlled, so only `http(s)` URLs are accepted. Host +images the model should see on a reachable web URL.