Add TensorRT-LLM Transfer and Action guides

README.md

@@ -1164,12 +1164,15 @@ We are building examples that show Cosmos 3 Super/Nano/Edge capabilities end to
 | Generator (audiovisual) with NIM | Generator | Text2Video and Image2Video only, against the prebuilt `Cosmos3-Generator` NIM; requests use `POST /v1/infer` and decode JSON `b64_video` responses. | [Notebook](cookbooks/cosmos3/generator/audiovisual/run_with_nim.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/audiovisual/run_with_nim.ipynb) |
 | Generator (audiovisual) with SGLang | Generator | Text-to-image, plus text-to-video and image-to-video each with sound on or off, against an OpenAI-compatible SGLang server. | [Notebook](cookbooks/cosmos3/generator/audiovisual/run_with_sglang.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/audiovisual/run_with_sglang.ipynb) |
 | Forward dynamics with Cosmos Framework | Generator | Forward dynamics: action-conditioned future-observation prediction for AV, DROID, UMI, and human hand pose, through the `cosmos_framework.scripts.inference` entrypoint. | [Notebook](cookbooks/cosmos3/generator/action/run_fd_with_cosmos_framework.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/action/run_fd_with_cosmos_framework.ipynb) |
+| Forward dynamics with TensorRT-LLM | Generator | Forward dynamics from a checked-in AV image and action trajectory, returning rollout and action tensors through the VisualGen API. | [Notebook](cookbooks/cosmos3/generator/action/run_fd_with_trt_llm.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/action/run_fd_with_trt_llm.ipynb) |
 | Forward dynamics with vLLM-Omni | Generator | Forward dynamics: action-conditioned future-observation prediction for AV, DROID, UMI, and human hand pose, against an OpenAI-compatible vLLM-Omni server. | [Notebook](cookbooks/cosmos3/generator/action/run_fd_with_vllm_omni.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/action/run_fd_with_vllm_omni.ipynb) |
 | Forward dynamics with SGLang | Generator | Forward dynamics: action-conditioned future-observation prediction for AV, DROID, and UMI, against an OpenAI-compatible SGLang server. | [Notebook](cookbooks/cosmos3/generator/action/run_fd_with_sglang.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/action/run_fd_with_sglang.ipynb) |
 | Inverse dynamics with Cosmos Framework | Generator | Inverse dynamics: ego-motion trajectory prediction from input AV video, through the `cosmos_framework.scripts.inference` entrypoint. | [Notebook](cookbooks/cosmos3/generator/action/run_id_with_cosmos_framework.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/action/run_id_with_cosmos_framework.ipynb) |
+| Inverse dynamics with TensorRT-LLM | Generator | Inverse dynamics from a checked-in AV observation clip, returning reconstructed video and the predicted trajectory in a tensor payload. | [Notebook](cookbooks/cosmos3/generator/action/run_id_with_trt_llm.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/action/run_id_with_trt_llm.ipynb) |
 | Inverse dynamics with vLLM-Omni | Generator | Inverse dynamics: ego-motion trajectory prediction from input AV video, against an OpenAI-compatible vLLM-Omni server. | [Notebook](cookbooks/cosmos3/generator/action/run_id_with_vllm_omni.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/action/run_id_with_vllm_omni.ipynb) |
 | Inverse dynamics with SGLang | Generator | Inverse dynamics: ego-motion trajectory prediction from input AV video, against an OpenAI-compatible SGLang server. | [Notebook](cookbooks/cosmos3/generator/action/run_id_with_sglang.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/action/run_id_with_sglang.ipynb) |
 | Transfer with Cosmos Framework | Generator | Video transfer: edge, blur, depth, segmentation, and world-scenario controls with captions, through the `cosmos_framework.scripts.inference` entrypoint. | [Notebook](cookbooks/cosmos3/generator/transfer/run_video_transfer_with_cosmos_framework.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/transfer/run_video_transfer_with_cosmos_framework.ipynb) |
+| Transfer with TensorRT-LLM | Generator | Edge, blur, depth, segmentation, and WSM transfer using inline precomputed controls and synchronous encoded-video responses; raw uploads are used only for server-derived edge/blur. | [Notebook](cookbooks/cosmos3/generator/transfer/run_video_transfer_with_trt_llm.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/transfer/run_video_transfer_with_trt_llm.ipynb) |
 | Transfer with vLLM-Omni | Generator | Video transfer: edge, blur, depth, segmentation, and world-scenario controls with captions, against an OpenAI-compatible vLLM-Omni server. | [Notebook](cookbooks/cosmos3/generator/transfer/run_video_transfer_with_vllm_omni.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/transfer/run_video_transfer_with_vllm_omni.ipynb) |
 | Reasoner with Cosmos Framework | Reasoner | Text and image reasoning: detailed captioning, robot task planning, 2D grounding, describe-anything, and action-trajectory prompts, through the `cosmos_framework.scripts.inference` entrypoint. | [Notebook](cookbooks/cosmos3/reasoner/run_with_cosmos_framework.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/reasoner/run_with_cosmos_framework.ipynb) |
 | Reasoner with vLLM | Reasoner | Image and video reasoning: captioning, temporal localization, embodied reasoning, common-sense reasoning, 2D grounding, describe-anything, action CoT, driving scenes, physical-plausibility, and situation understanding, against an OpenAI-compatible vLLM server (Cosmos3-Super on 4 GPUs by default; switch to Nano per the cookbook README). | [Notebook](cookbooks/cosmos3/reasoner/run_with_vllm.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/reasoner/run_with_vllm.ipynb) |

cookbooks/cosmos3/README.md

@@ -8,7 +8,7 @@ backend you want to run and follow that one section.
 | --- | --- | --- |
 | [Cosmos Framework](#cosmos-framework) | Native PyTorch inference, launched with `torchrun` | Reasoner, Generator (Audiovisual, Action, **Transfer**) |
 | [Diffusers](#diffusers) | Direct generation with `Cosmos3OmniPipeline` | Generator (Audiovisual) |
-| [TensorRT-LLM Generator](#tensorrt-llm-generator) | OpenAI-compatible VisualGen server (image/video/audio generation) | Generator (Audiovisual) |
+| [TensorRT-LLM Generator](#tensorrt-llm-generator) | OpenAI-compatible VisualGen server (image/video/audio/action/transfer generation) | Generator (Audiovisual, Action, **Transfer**) |
 | [TensorRT-LLM Reasoner](#tensorrt-llm-reasoner) | OpenAI-compatible image/video reasoning server | Reasoner |
 | [Transformers](#transformers) | Hugging Face Transformers inference | Reasoner |
 | [vLLM](#vllm) | OpenAI-compatible reasoning server (image/video understanding) | Reasoner |
@@ -170,15 +170,27 @@ uv pip install --torch-backend=cu130 \
 ## TensorRT-LLM Generator
 
 OpenAI-compatible **VisualGen** server for Generator audiovisual text-to-image,
-text-to-video, image-to-video, video-to-video, and synchronized audio examples.
+text-to-video, image-to-video, video-to-video, synchronized audio, Transfer, and
+Action examples.
 Initial Cosmos3 support was added in TensorRT-LLM PR
 [#14824](https://github.com/NVIDIA/TensorRT-LLM/pull/14824), synchronized audio
 in [#14827](https://github.com/NVIDIA/TensorRT-LLM/pull/14827), and
-video-to-video in [#16155](https://github.com/NVIDIA/TensorRT-LLM/pull/16155). The
-DMD2-distilled four-step checkpoints were added in
+video-to-video in [#16155](https://github.com/NVIDIA/TensorRT-LLM/pull/16155),
+Transfer in [#16394](https://github.com/NVIDIA/TensorRT-LLM/pull/16394), and
+Action in [#17325](https://github.com/NVIDIA/TensorRT-LLM/pull/17325).
+The DMD2-distilled four-step checkpoints were added in
 [#16563](https://github.com/NVIDIA/TensorRT-LLM/pull/16563) (text-to-image) and
 [#16690](https://github.com/NVIDIA/TensorRT-LLM/pull/16690) (image-to-video).
-Use a TensorRT-LLM checkout or package that includes those changes.
+These changes are merged on TensorRT-LLM `main`. The Action and Transfer
+notebooks were executed against source revision
+[`bca6761ab84fbcd58fc7f914eade7de48b32e35e`](https://github.com/NVIDIA/TensorRT-LLM/commit/bca6761ab84fbcd58fc7f914eade7de48b32e35e).
+Use that revision to reproduce their request contract, or a newer build with
+the same API. The older `799d7d42` validation used `input_reference` and does
+not validate these notebooks' `image_reference` / `video_reference` fields.
+See the [Action and Transfer validation record](generator/trtllm-validation.md)
+for the executed cases, output dimensions, and remaining runtime limitations.
+The source revision is significant: a package version of `1.3.0rc26` alone
+does not establish compatibility with the action image-decoding path.
 
 Install TensorRT-LLM following its upstream documentation.
 
@@ -189,11 +201,13 @@ Cosmos3 VisualGen change before it is available in your installed package or
 release image.
 
 ```bash
-apt-get update && apt-get -y install ffmpeg git git-lfs
+apt-get update && apt-get -y install git git-lfs
 git lfs install
 
 git clone https://github.com/NVIDIA/TensorRT-LLM.git
 cd TensorRT-LLM
+# Source revision used for the Action/Transfer notebook validation below.
+git checkout bca6761ab84fbcd58fc7f914eade7de48b32e35e
 git submodule update --init --recursive
 git lfs pull
 
@@ -208,6 +222,7 @@ docker run --rm -it \
   nvcr.io/nvidia/tensorrt-llm/devel:<tag>
 
 # Inside the container:
+apt-get update && apt-get -y install ffmpeg
 python3 scripts/build_wheel.py --use_ccache --skip_building_wheel --linking_install_binary
 pip install -e .
 ```
@@ -221,10 +236,58 @@ explicitly disable guardrails before starting the server:
 
 ```bash
 pip install cosmos_guardrail==0.3.0
-# If needed by your OpenCV stack:
-# pip uninstall opencv-python
+# On headless servers without libGL.so.1, replace the OpenCV wheel pulled in by
+# cosmos_guardrail with the matching headless build:
+pip uninstall -y opencv-python
+pip install opencv-python-headless==5.0.0.93
+```
+
+#### Server-side NLTK data setup
+
+With `cosmos_guardrail==0.3.0`, NLTK's path checks can reject tokenizer or
+dictionary files that are symlinks from a Hugging Face snapshot into its
+`blobs/` directory. Prepare a separate copy of the NLTK data as regular files
+**on the server, in the same container and shell used to launch TensorRT-LLM**:
+
+```bash
+export NLTK_DATA="$(mktemp -d "${TMPDIR:-/tmp}/cosmos3-nltk.XXXXXX")"
+python3 - <<'PY'
+import os
+import shutil
+from pathlib import Path
+
+import nltk
+from huggingface_hub import snapshot_download
+
+snapshot = snapshot_download(
+    "nvidia/Cosmos-1.0-Guardrail",
+    revision="cf03c0395fac8c4de386c0bdab12cc4fc8d66362",
+    allow_patterns=["blocklist/**"],
+)
+source = Path(snapshot) / "blocklist" / "nltk_data"
+destination = Path(os.environ["NLTK_DATA"]).resolve()
+shutil.copytree(source, destination, symlinks=False, dirs_exist_ok=True)
+assert not any(path.is_symlink() for path in destination.rglob("*"))
+
+# Check both resource lookups used by the text blocklist before starting a GPU server.
+nltk.data.path[:] = [str(destination)]
+tokens = nltk.word_tokenize("You are an autonomous vehicle planning system.")
+assert nltk.WordNetLemmatizer().lemmatize("vehicles") == "vehicle"
+print("Guardrail NLTK data ready:", destination, tokens)
+PY
 ```
 
+Keep `NLTK_DATA` exported when starting the server below. Repeat this setup
+after recreating the container or removing the temporary directory. Running it
+only in the client notebook's environment does not configure a remote server.
+This workaround does not rewrite cached files or symlinks, and keeps NLTK
+path security and `use_guardrails=True` enabled.
+
+The separate `No safety models found, returning safe` warning in guardrail
+0.3.0 refers to its intentionally empty video-content classifier list. Text
+checks and face blurring remain configured; the NLTK workaround does not enable
+video-content classification or suppress that warning.
+
 Set the TensorRT-LLM source root for the shared VisualGen config YAMLs. Run this
 from inside the TensorRT-LLM checkout — the directory the `git clone` above
 created, which is where `examples/` lives — or point `TRTLLM_ROOT` at that
@@ -285,22 +348,42 @@ runs both against a running server. These students cover text-to-image and
 image-to-video only; use the base checkpoints for text-to-video, video-to-video,
 and synchronized audio.
 
-The server exposes `/health`, `/v1/videos/generations`, `/v1/videos`, and
-`/v1/images/generations`. The audiovisual notebook uses the validated video
-generation endpoint for text-to-image, text-to-video, image-to-video,
-video-to-video, and synchronized audio. Cosmos3 text-to-image is sent as a
-one-frame video request, matching the TensorRT-LLM Cosmos3 pipeline; the notebook
-sends it as `num_frames=1`, `seconds=1`, and `fps=8` to satisfy the video request
-schema while preserving a single generated frame. Image-to-video and
-video-to-video upload their reference media as multipart `input_reference`;
-TensorRT-LLM classifies the reference by content. Synchronized audio is enabled
-with `enable_audio: true` in `extra_params` and is muxed into the output video.
-Keep `ffmpeg` on the server `PATH`: without it, TensorRT-LLM falls back to a
-video-only AVI encoder and cannot preserve generated audio. Requests send
+The server exposes `/health`, the blocking `/v1/videos/sync`, the asynchronous
+`/v1/videos`, and `/v1/images/generations`. The older
+`/v1/videos/generations` spelling is a deprecated alias of `/v1/videos/sync`.
+The base audiovisual notebook uses `/v1/images/generations` for text-to-image
+and `/v1/videos/sync` for text-to-video, image-to-video, video-to-video, and
+synchronized audio. Text-to-image sets `extra_params.output_type="image"` and
+returns a base64-encoded PNG. Image-to-video uploads multipart
+`image_reference`; video-to-video uses `video_reference`. Synchronized audio is
+enabled with `enable_audio: true` in `extra_params` and is muxed into the output
+video.
+Every video request explicitly selects MP4. Keep `ffmpeg` on the server `PATH`:
+without it, the request fails early instead of returning browser-incompatible
+AVI or dropping generated audio. Requests send
 Cosmos3 controls through `extra_params`, so use a TensorRT-LLM build that includes
 the Cosmos3 VisualGen API schema. The notebook sets request-level
 `max_sequence_length=4096` for longer structured JSON prompts.
 
+Transfer uses the synchronous `/v1/videos/sync` route. For server-derived edge
+or blur, upload the raw source video as multipart `video_reference` and set the
+corresponding `extra_params` hint to `true`. For a precomputed edge, blur,
+depth, segmentation, or WSM control, base64-encode the control inside its hint;
+no top-level reference is needed. The server decodes inline media to bytes at the
+HTTP boundary. TensorRT-LLM uses `use_guardrails` for its per-request safety
+switch; `guardrails`, `control_path`, and other vLLM-Omni-only names are not
+interchangeable.
+
+Action requests use the same synchronous route and upload an image as
+`image_reference` or a video as `video_reference`. For the checked-in AV
+examples, use the Cosmos Framework reference prompt shown in the Action
+cookbook; current TensorRT-LLM ignores the legacy `view_point` field.
+Because an action trajectory cannot be represented in MP4 or
+AVI, `format=auto` resolves to `safetensors`; the payload contains named `video`,
+`action`, and `frame_rate` tensors. The asynchronous `/v1/videos` route also
+supports this payload: poll `GET /v1/videos/{id}`, then download it from
+`GET /v1/videos/{id}/content`.
+
 ## TensorRT-LLM Reasoner
 
 OpenAI-compatible **reasoning** server for image and video understanding. Run