Add TensorRT-LLM Transfer and Action guides by ishovkun · Pull Request #310 · NVIDIA/cosmos

ishovkun · 2026-08-07T22:19:53Z

Summary

Add Cosmos3 Action and Transfer cookbooks for TensorRT-LLM's VisualGen serving API.

Action

Forward dynamics from an AV conditioning image and action trajectory.
Inverse dynamics from an AV observation video.
Multipart image_reference and video_reference requests, safetensors decoding, and MP4 previews.

Transfer

Edge, blur, depth, segmentation, and world-scenario (WSM) examples using the checked-in controls and captions.
Inline control inputs through extra_params and synchronous MP4 responses from /v1/videos/sync.
Per-control resolution, frame count, and guidance settings, with saved outputs and notebook previews.

Setup and documentation

Shared TensorRT-LLM source setup and Nano/Super VisualGen launch commands.
Server-side guardrail setup that prepares NLTK resources as regular files and exports NLTK_DATA, with path security enabled.
Updated audiovisual API examples, explicit MP4 output, and links from the cookbook indexes.
A checked-in validation record with runtime revisions, notebook hashes, and output checks.

Testing

Executed all code cells in the two Action notebooks and the Transfer notebook on one H200 using Cosmos3-Nano and a native build of TensorRT-LLM bca6761. The run used the documented NLTK setup and server command, with cosmos-guardrail==0.3.0 and use_guardrails=True.

Example	Result	Video output
Forward dynamics	PASS	832×480, 61 frames, 10 fps
Inverse dynamics	PASS	832×480, 61 frames, 10 fps
Transfer edge	PASS	1280×720, 121 frames, 30 fps
Transfer blur	PASS	1104×832, 121 frames, 30 fps
Transfer depth	PASS	1280×720, 121 frames, 30 fps
Transfer segmentation	PASS	1280×720, 121 frames, 30 fps
Transfer WSM	PASS	1280×720, 100 frames, 10 fps

All seven MP4s were decoded frame by frame. Both Action tensor payloads passed shape, dtype, and finite-value checks. All 35 tracked notebooks passed JSON/schema validation and the CI lint selection; the three added notebooks have cleared outputs and pass strict schema validation.

Guardrail coverage follows the package's default configuration: Blocklist and Qwen3Guard text checks plus RetinaFace face blurring; video-content classification is not enabled in 0.3.0.

The GPU results above cover Nano Action and Transfer. See the validation record for the exact environment, execution records, and artifact metadata.

Signed-off-by: Igor Shovkun <ishovkun@nvidia.com>

Signed-off-by: Igor Shovkun <igshov@gmail.com>

ConstBob

Thanks for these TensorRT-LLM Transfer / Action / distilled guides. The use_guardrails vs vLLM guardrails split, distilled omit-num_inference_steps/guidance_scale, and Action safetensors contract are clear.

Static checks passed (notebook JSON, Python compile, assets, local links). I did not run GPU inference as part of the review. Transfer #16394 and Action #17325 are already on TensorRT-LLM main.

Two places where the write-up and the notebooks drifted apart (inline).

Nit, non-blocking: new distilled T2I/I2V cells in run_with_trt_llm.ipynb still go through the existing /v1/videos/generations helper, while this PR documents /v1/videos/sync as canonical. Fine if the alias stays.

Signed-off-by: Igor Shovkun <igshov@gmail.com>

ConstBob

Thanks @ishovkun for updating the PR! LGTM now - I've approved this PR.

adsridhar

LGTM

MaciejBalaNV · 2026-09-07T13:00:19Z

When testing this PR there were a few issues that showed up. Shared it offline with Igor. Let's wait before merging this PR.

Signed-off-by: Igor Shovkun <igshov@gmail.com>

ConstBob · 2026-09-17T22:31:50Z

Follow-up GPU check of Action forward dynamics (FD) on current HEAD.

1x H200 + nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc27.dev202609150000
1x H100 + same rc27
1x H100 + TRT-LLM main HEAD 757fcf3d (Python overlay of visual_gen/serve on the rc27 nightly; not a from-source C++ rebuild)
1x H100 + nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc26.dev202609060001

All four: POST /v1/videos/sync → 200, 73085300 bytes, video=(61,480,832,3) uint8, action=(60,9) float32, 10 fps. Rollout is coherent (some mid-clip vehicle artifacts). Did not see a collapse in these experiments.

Note: notebook default use_guardrails=true returns HTTP 500 here (pathsec on Guardrail HF blob symlinks). Succeeds with use_guardrails=false. Non-blocking.

Did not re-run Transfer / inverse dynamics, and did not retest GB200.

LGTM on the Action FD path.

skasetty-Siri · 2026-09-17T22:46:37Z

The Action notebooks now use image_reference and video_reference, but the TensorRT-LLM revision referenced in the validation (799d7d42) appears to accept only input_reference. Could you document a compatible runtime revision and update the validation to cover the latest notebook request format?
Please reconcile the blur resolution as well: the notebook requests 1104×832, while the validation summary reports 1280×720.
Could you also confirm whether the runtime issues called out in the September 7 merge-hold comment have been resolved?

skasetty-Siri · 2026-09-17T22:49:58Z

The new t2i_edge example sends a one-frame video request without extra_params.output_type="image". At the referenced TensorRT-LLM revision (8cfa341332), this selects video mode and uses its default negative prompt, which differs from native image mode.
Could you switch this to image mode, using the image endpoint and response decoder, while keeping Edge’s 640×640 resolution, 50 steps, and guidance 4.0?
If single-frame video generation is intentional, could you clarify in the guide how this differs from native T2I

Signed-off-by: Igor Shovkun <igshov@gmail.com>

ishovkun · 2026-09-18T15:33:38Z

00 + same rc27

1x H100 + TRT-LLM main HEAD 757fcf3d (Python overlay of visual_gen/serve on the rc27 nightly; not a from-source C++ rebuild)

1x H100 + nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc26.dev202609060001

All four: POST /v1/videos/sync → 200, 73085300 bytes, video=(61,480,832,3) uint8, action=(60,9) float32, 10 fps. Rollout is coherent (some mid-clip vehicle artifacts). Did not see a collapse in these experiments.

Note: notebook default use_guardrails=true returns HTTP 500 here (pathsec on Guardrail HF blob symlinks). Succeeds with use_guardrails=false. Non-blocking.

Did not re-run Transfer / inverse dynamics, and did not retest GB200.

LGTM on the Action FD path.

Updated docs: a9029df, e4d59b6.

Runtime: 799d7d42 validation was stale. Both Action notebooks passed on bca6761ab84fbcd58fc7f914eade7de48b32e35e, using image_reference / video_reference. Setup now pins that revision; validation includes run IDs.
Blur: Later notebook run produced 1104×832, 121 frames, 30 fps. 1280×720 belonged to older request. Validation corrected.
Merge hold: Runtime issue concerned distilled models. Examples removed in 7ff4a86; issue no longer applies to this PR.

ishovkun · 2026-09-18T15:58:44Z

The new t2i_edge example sends a one-frame video request without extra_params.output_type="image". At the referenced TensorRT-LLM revision (8cfa341332), this selects video mode and uses its default negative prompt, which differs from native image mode.

Could you switch this to image mode, using the image endpoint and response decoder, while keeping Edge’s 640×640 resolution, 50 steps, and guidance 4.0?

If single-frame video generation is intentional, could you clarify in the guide how this differs from native T2I

Could this comment be intended for another PR? #354 has no t2i_edge example. Its Nano/Super T2I examples already use image mode, /v1/images/generations, and PNG decoding.

skasetty-Siri

Thanks—the original runtime-schema and blur-resolution comments are addressed. Before I approve for public release, please clarify the remaining guardrail limitation: the validation record reports missing safety models, and the September 17 follow-up reports failure with guardrails enabled.

Please provide validation with guardrails actually loaded, or obtain explicit release-owner acceptance of this limitation. Please also update the PR description to match the corrected validation record.

ishovkun · 2026-09-18T22:17:12Z

Thanks—the original runtime-schema and blur-resolution comments are addressed. Before I approve for public release, please clarify the remaining guardrail limitation: the validation record reports missing safety models, and the September 17 follow-up reports failure with guardrails enabled.

Please provide validation with guardrails actually loaded, or obtain explicit release-owner acceptance of this limitation. Please also update the PR description to match the corrected validation record.

Updated the PR description with the corrected runtime revision, blur resolution, and guardrail clarification.
I also need to correct my validation record: “missing safety models” was a misleading interpretation. cosmos-guardrail==0.3.0 initializes text guardrails and face blurring, but intentionally disables its video-content classifier. The warning refers to that empty classifier list—not failure to load all guardrails.
The September 17 pathsec failure remains a separate unresolved issue. This update clarifies existing evidence; it is not new guardrail validation or release-owner acceptance.

skasetty-Siri · 2026-09-18T22:57:26Z

Thanks for clarifying the warning and updating the description. Understood that it does not mean all guardrails failed. The remaining question is the separate pathsec failure with guardrails enabled. Please provide validation of a working configuration or explicit release-owner acceptance before we close that item.

ishovkun · 2026-09-19T05:56:18Z

@skassety I added server-side NLTK setup in 1595ad3: it copies the required tokenizer/dictionary resources into a private directory as regular files and exports NLTK_DATA before launching TensorRT-LLM. This avoids the cache-symlink pathsec error without disabling path security or guardrails. The READMEs and notebooks now include this setup.

To clarify exactly which guardrail components are used:

Component	Status in our tested configuration
`aegis/` — LlamaGuard-7B + Aegis adapter	Not instantiated by `cosmos-guardrail`
`video_content_safety_filter/` — SigLIP-based classifier	Disabled in `cosmos-guardrail` itself, with the comment “Too many false positives”
`face_blur_filter/Resnet50_Final.pth` — RetinaFace	Used for face detection and blurring
`blocklist/`	Used for text filtering
Qwen3Guard — downloaded separately from `Qwen/Qwen3Guard-Gen-0.6B`	Used for text-safety classification

We use every component enabled by the package’s default configuration; we have not disabled any additional checks. The 0.3.0 warning refers to its empty video-classifier list, not missing active models.

I rebuilt the documented TensorRT-LLM revision and ran both Action notebooks and all five Transfer cases on one H200 with use_guardrails=True. All seven passed, including video decoding and Action tensor checks. The PR description and validation record now reflect this configuration.

skasetty-Siri

Thanks for the fix and detailed clarification. The documented NLTK workaround, updated guardrail-enabled validation, and revised PR description address my remaining comments. Rechecked 1595ad3; no remaining blocking findings from my review.

## Summary - add a TensorRT-LLM walkthrough for the two published DMD2-distilled Cosmos3-Super students, `nvidia/Cosmos3-Super-Text2Image-4Step` (text-to-image) and `nvidia/Cosmos3-Super-Image2Video-4Step` (image-to-video), each on a single GPU - document the distilled request contract: the checkpoint's scheduler config locks the schedule to four stochastic (SDE) steps and classifier-free guidance is baked into the weights, so requests omit `num_inference_steps` and `guidance_scale` entirely and the server fills both in; a conflicting value is rejected, not clamped - leave `use_system_prompt` unset in every request so the image-to-video student's checkpoint-declared `default_use_system_prompt: true` applies - add both `trtllm-serve` launch commands to the shared setup guide, and index the notebook from the root README - **separately**: reword two guardrail blocklist false positives in the shared audiovisual negative prompts, without which no image-to-video request can run at all (see below) This is the follow-up #310 deferred ("keep the four-step distilled T2I/I2V examples out of this PR pending separate output-quality follow-up"). The examples match the assets every other backend already uses for its distilled sections — `robot_draping` for text-to-image and `car_driving` for image-to-video — so the cookbook surface is consistent across Cosmos Framework, Diffusers, vLLM-Omni, SGLang, and now TensorRT-LLM. The shipped scope is text-to-image and image-to-video only. These students are task-specialized and no other backend advertises text-to-video, video-to-video, or synchronized audio for them either. ## The guardrail fix, and why it is in this PR The shared audiovisual negative prompts contained two substrings that the `cosmos_guardrail` 0.3.0 `Blocklist` censors. Any request carrying them fails with HTTP 500 and `Text guardrail blocked prompt` **before any denoising runs**: | Text | Censored word | Replacement | |---|---|---| | `color bleeding between elements` | `bleeding` | `color smearing between elements` | | `The scene feels lifeless and sterile` | `lifeless` | `The scene feels inert and sterile` | Neither describes anything the blocklist is meant to catch; both are ordinary image-artifact vocabulary. The replacement wording is not invented here. TensorRT-LLM already ships a corrected copy of this exact prompt as `examples/visual_gen/models/cosmos3/cosmos3_negative_prompt.json`, and that is the default every Cosmos3 request falls back to when it sends no negative prompt of its own. Upstream's copy and the cookbook's differ in **exactly four leaves, and all four are these two substrings** — so the cookbook asset is simply stale. Both files are now identical to the TensorRT-LLM default. Kept as separate commits so they can be split out if preferred. Text-to-image was unaffected because that path sends no negative prompt — which is why only the image-to-video example hit it. Three related issues found while diagnosing this, **not changed here**: 1. The two files are shared by the Diffusers, vLLM-Omni, SGLang and base TensorRT-LLM notebooks, so this also unblocks their image-to-video / text-to-video / video-to-video examples. 2. `assets/prompts/image2video/humanoid_robot.json` is blocked by a *different* rule (partial match, not the word blocklist). That is the prompt the merged `run_with_trt_llm.ipynb` uses for `i2v_nano` and `i2v_super`. Left alone deliberately: rewording a *positive* prompt changes generated content. 3. `transfer/assets/negative_prompt.json` carries the same two substrings. That file belongs to #310, which now rewords both, so it is covered there rather than duplicated here. ## Validation ### GPU inference Both notebook sections were executed **as written** with `jupyter nbconvert --to notebook --execute` against real `trtllm-serve` VisualGen servers — the shipped notebook's own cells, not a re-implementation. TensorRT-LLM at `505a02ce04`, whose `tensorrt_llm/_torch/visual_gen`, `tensorrt_llm/visual_gen` and `examples/visual_gen` trees are byte-identical to `main` (`2dbab44d4f`); the only delta is test files. Hardware: 1 x NVIDIA B300. **Text-to-image** — `Cosmos3-Super-Text2Image-4Step` with `configs/cosmos3-t2i-1gpu.yaml`: ``` Distilled Cosmos3 checkpoint: fixed 4-step schedule [1.0, 0.9375, 0.8333333333333334, 0.625], classifier-free guidance baked in. Running warmup for Cosmos3OmniMoTPipeline: 1 shape(s) [1024x1024x1], 4 steps Denoising: 4 steps, guidance=1.0 POST /v1/videos/generations HTTP/1.1" 200 OK ``` Artifact: 57,617 bytes, decoded with PyAV as H.264 1024x1024, 1 frame. **Image-to-video** — `Cosmos3-Super-Image2Video-4Step`, no config file: ``` Running warmup for Cosmos3OmniMoTPipeline: 1 shape(s) [720x1280x189], 4 steps Denoising: 4 steps, guidance=1.0 POST /v1/videos/generations HTTP/1.1" 200 OK ``` Artifact: 4,890,310 bytes, decoded as H.264 1280x720, 24 fps, 189 frames. The warmup shape confirms the claim that this checkpoint deploys at the default omni shape and needs no config file. **Negative control** (both servers): a request that sets `guidance_scale=6.0` is rejected rather than clamped — ``` HTTP 400: This is a distilled Cosmos3 checkpoint; classifier-free guidance is baked into the weights. guidance_scale must be 1.0 or left unset (got 6.0). ``` **Guardrail fix verified directly** against `cosmos_guardrail` 0.3.0's `Blocklist`: both reworded negative prompts and both positive prompts used by the notebook now report `safe=True`. The trigger words were located by diffing the censored output against the original, not guessed. Re-verified after adopting upstream's wording: `bleeding` and `lifeless` are present in NVIDIA's `blocklist`/`exact_match` word sets (383 and 1430 entries) while `smearing` and `inert` are not. **Visual review**: the text-to-image frame renders the prompt's robot-arm-draping-satin scene coherently. The image-to-video clip follows its `temporal_caption` — coastal dashcam view, rockfall beginning a few seconds in, boulders and dust building until the road is partially blocked — with no flicker or geometry snapping across the 189 frames. ### Local validation - parsed the new notebook as JSON - validated it against the nbformat schema with `nbformat==5.11.1` (the CI gate) - linted it with `ruff==0.16.4 check --select=E9,F63,F7,F82` (the CI gate) - verified unique cell IDs, empty outputs, null execution counts, and that every code cell parses - re-validated both edited JSON assets with `json.tool` - ran `git diff --check` ### Source contract audit The request contract is read from TensorRT-LLM `main` (`tensorrt_llm/_torch/visual_gen/models/cosmos3/sampling.py`): - `load_scheduler` selects `FlowMatchEulerDiscreteScheduler` for distilled checkpoints and `UniPCMultistepScheduler` for base ones; an unknown declaration is a load-time error - `Cosmos3SamplingPolicy.validate_request` rejects any `num_inference_steps` other than `len(t_list)` and any `guidance_scale` other than `1.0` - `generation_default_overrides` supplies both values when the request leaves them unset - `set_flow_shift` is a structural no-op for distilled checkpoints, so `flow_shift` does not apply - both checkpoints declare `fixed_step_sampler_config.t_list = [1.0, 0.9375, 0.8333333333333334, 0.625]` with `stochastic_sampling: true` ## Dependencies and review state - TensorRT-LLM support is already merged: NVIDIA/TensorRT-LLM#16563 (distilled text-to-image) and NVIDIA/TensorRT-LLM#16690 (distilled image-to-video). - No dependency on #310; the two PRs touch different notebooks. If #310 merges first there is no expected conflict. --------- Signed-off-by: Igor Shovkun <ishovkun@nvidia.com>

## Summary Add a Cosmos3-Edge section to the TensorRT-LLM audiovisual cookbook, covering exactly what TensorRT-LLM `main` supports for that checkpoint. - add Edge text-to-image, text-to-video, and image-to-video walkthroughs to `run_with_trt_llm.ipynb`, with the Edge endpoint, launch command, and per-case request shape - send Edge's 480p-native contract explicitly: 832x480 with 121 frames, 50 UniPC steps, guidance 5.0 for video, and Edge's native 640x640 with guidance 4.0 for text-to-image. Flow shift (3.0) rides the checkpoint-declared native flow schedule, so requests do not send it - run Edge text-to-image in native image mode against `/v1/images/generations` (see below) - document the Edge launch and request contract in the audiovisual README and the shared Cosmos3 environment setup guide, and note the new section in the root notebook index - realign the shared audiovisual negative prompt with the TensorRT-LLM default (separate commit, details below) ## What TensorRT-LLM head supports for Edge Checked against `main` at `8cfa341332`. Edge support landed in [TensorRT-LLM #16773](NVIDIA/TensorRT-LLM#16773). | Mode | Edge on TensorRT-LLM | Basis | | --- | --- | --- | | Text-to-image | supported | `COSMOS3_EDGE_T2I_PARAMS` (640x640, 50 steps, guidance 4.0), via `/v1/images/generations` with `output_type="image"` | | Text-to-video | supported | `COSMOS3_EDGE_VIDEO_PARAMS` (832x480, 121 frames, 50 steps, guidance 5.0, shift 3.0) | | Image-to-video | supported | same video defaults; reference image as multipart `input_reference` | | Synchronized audio | not supported | Edge has no audio tower | | Video-to-video | not supported | V2V is validated for Nano/Super only | | Action (forward/inverse dynamics, policy) | not supported | the VisualGen pipeline has no action path on `main`, and the Edge checkpoint's action weights are explicitly unsupported | | Transfer controls | not supported | no transfer/control path exists in TensorRT-LLM `main` | | Reasoning | not supported | measured: `POST /v1/chat/completions` returns 404 against a live Edge server, and the served OpenAPI exposes only `/health`, `/metrics`, `/v1/images/*`, `/v1/videos/*`, `/v1/models`, `/version` | Edge is served with no config override — its generation defaults come from the checkpoint. That is also the invocation the upstream `test_cosmos3_edge_i2v_example` integration test exercises. For comparison, Edge coverage across the backends already in this repo: Cosmos Framework, Diffusers, and vLLM-Omni ship image-to-video; SGLang ships text-to-image, text-to-video, and image-to-video. This PR brings TensorRT-LLM to the same three modes. ## Text-to-image runs in native image mode Raised in review. The notebook's existing pattern expresses text-to-image as a one-frame video request. At `8cfa341332` that selects **video** mode, because `extra_params.output_type` defaults to `"video"`, and the pipeline keys its default negative prompt off that: ```python def default_negative_prompt(output_type: str) -> str: return (COSMOS3_DEFAULT_NEGATIVE_PROMPT if output_type == "image" else default_video_negative_prompt()) ``` The example sends no negative prompt of its own, so a still image was being steered by Cosmos3's **video** negative prompt — "Incoherent motion", "visible frame-to-frame discontinuities", "Movement appears as a slideshow" — none of which mean anything for a single frame. The Edge case now posts to `/v1/images/generations` with `output_type="image"` and decodes the base64 PNG, keeping 640x640, 50 steps, and guidance 4.0. Both halves are required: `parse_visual_gen_params` does not set `output_type` for image requests either, so hitting the image endpoint without the flag runs video mode, leaves `output.image` as `None`, and returns 500. For Edge this is the only generation-semantics change — `guidance_interval` is `None` in both `COSMOS3_EDGE_VIDEO_PARAMS` and `COSMOS3_EDGE_T2I_PARAMS`. **`t2i_nano` and `t2i_super` hit the same default and are deliberately left alone here.** They are the `qwen3` family, whose image-mode table also flips `guidance_interval` from `None` to `(400, 1000)`, so switching them changes their output in a way this PR cannot validate (Super needs four GPUs). Worth a follow-up. ## The negative-prompt commit **Update after merging `main`:** #353 landed the byte-identical rewording independently, so this change no longer appears in the diff against `main`. Commit `0407123` remains in the branch history and the explanation below still describes why it was needed at the time. A prerequisite for the Edge examples to run, but not Edge-specific — it fixes the Nano and Super video cases in the same notebook. TensorRT-LLM runs the text guardrail over the **negative** prompt as well as the positive one. The checked-in audiovisual negative prompt had drifted from the reference default in two places — `color bleeding between elements` and `the scene feels lifeless and sterile` — and the Cosmos guardrail blocklist matches `bleeding` and `lifeless`. Every video request carrying this asset therefore failed with HTTP 500 (`Text guardrail blocked prompt`) under the documented default of guardrails enabled. Text-to-image was unaffected only because the payload builder attaches no negative prompt for that mode. [TensorRT-LLM #17523](NVIDIA/TensorRT-LLM#17523) already reworded those exact phrases to `color smearing` and `feels inert` in TensorRT-LLM's own default. The cookbook ships its own copy and sends it explicitly as `negative_prompt`, so the corrected default never applied. This commit adopts the upstream wording; the serialized request body each file produces is now byte-for-byte identical to `COSMOS3_VIDEO_NEGATIVE_PROMPT` (14674 bytes, key order included). `cookbooks/cosmos3/generator/transfer/assets/negative_prompt.json` carries the same drift and is left alone — #310 already covers it. ## Validation Real Edge server, real generations, guardrails enabled throughout. **Environment.** 1x NVIDIA B200, x86_64. TensorRT-LLM built from source at `8cfa341332` (reports `1.3.0rc25+8cfa341332`). `cosmos_guardrail==0.3.0` with the pinned `nvidia/Cosmos-1.0-Guardrail` revision `cf03c0395fac8c4de386c0bdab12cc4fc8d66362`. `ffmpeg` 7.0.2 on `PATH`, so video responses take the MP4 encoder rather than the AVI fallback. The checkpoint is a local copy of `nvidia/Cosmos3-Edge`, served by path. **Server.** `trtllm-serve <Cosmos3-Edge> --port 8000`, no `--visual_gen_args`, exactly as the new section documents. **Execution.** The notebook's Edge cells were run verbatim through `jupyter nbconvert --to notebook --execute` from a checkout of this branch, so `find_repo_root`, the repo-relative asset paths, `create_payload`, `check_trtllm_server`, `run_trtllm_payload`, and `view_run` all ran as a reader hits them. `nbconvert` exited 0 with every cell executed. **Guardrail preflight.** All five prompt/negative assets the Edge cases send pass `CosmosSafetyChecker.check_text_safety`. **Endpoints actually served**, from the server log: ``` 1 "POST /v1/images/generations HTTP/1.1" 200 OK <- t2i_edge 2 "POST /v1/videos/generations HTTP/1.1" 200 OK <- t2v_edge, i2v_edge ``` **Generation parameters the server resolved**, from the server log (warmup line omitted): ``` Cosmos3 generation dims: 640x640 (WxH), num_frames=1, num_inference_steps=50, guidance_scale=4.00 Cosmos3 generation dims: 832x480 (WxH), num_frames=121, num_inference_steps=50, guidance_scale=5.00 Cosmos3 generation dims: 832x480 (WxH), num_frames=121, num_inference_steps=50, guidance_scale=5.00 ``` **Results**, decoded from the returned payloads: | Case | Decoded shape | FPS | Size | | --- | --- | --- | --- | | `t2i_edge` | `(640, 640, 3)` PNG | n/a | 514 KB | | `t2v_edge` | `(121, 480, 832, 3)` | 24 | 510 KB | | `i2v_edge` | `(121, 480, 832, 3)` | 24 | 1952 KB | Every case matches the documented contract. The run logged zero model-card envelope advisories; the one previously emitted came from the one-frame video text-to-image request and disappears in native image mode. Outputs were inspected visually as well as by shape. As a cross-check the same three cases were also run through cosmos-framework (`3845f4c`) at matched settings — same prompts, negative prompts, seed, sampler, steps and guidance. Shapes agree on all three, and after the image-mode fix the text-to-image results agree in composition as well, which they did not before: cosmos-framework already sends an empty negative prompt for text-to-image, so it was the TensorRT-LLM video-mode default that diverged. --------- Signed-off-by: Igor Shovkun <ishovkun@nvidia.com>

ConstBob

GB200 Action FD visual quality is a merge blocker. Maciej’s observation reproduces; H100/H200 does not.

Setup: GB200 (aarch64 Grace, driver 580.173.02), 1× GB200 (CUDA_VISIBLE_DEVICES=0), Cosmos3-Nano, POST /v1/videos/sync, use_guardrails=false (cookbook default still 500s here on Guardrail HF blobs). Two TRT-LLM stacks: main HEAD Python overlay, and the public NGC image nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc26.dev202609060001. Both produced byte-identical safetensors.

What fails: frames 0–35 match H100/H200. From t≈40 the upper half smears and never recovers.

H100/H200 (top) vs GB200 (bottom), same input.

t=50

t=60

Full clip

fd_gb200_av.mp4

Numbers (Laplacian-variance sharpness; t=0 is essentially identical across platforms — mean 97.65 vs 97.66, sharpness 182.7 vs 182.6 — so this is rollout divergence, not a bad start frame):

                        H100/H200    GB200
sharpness, first 5        172.3      181.1
sharpness, last 5         144.0      114.5
retention ratio           0.836      0.632

On GB200, sharpness falls between t=40 (135.6) and t=44 (103.8), then flatlines at ~104.

This does not look like a cookbook diff. On GB200, main HEAD and rc26 were byte-identical (deec0969…); on H100/H200, main HEAD matched the earlier run byte-for-byte (31206e4d…). Same code, same inputs, only the platform differs — points at a Blackwell/SM100 kernel or precision path in TRT-LLM.

Please don’t merge on the previous Action FD LGTM. Happy to re-approve once either GB200 FD quality is fixed or explained as acceptable with a cookbook caveat that Maciej/Igor sign off on.

ishovkun and others added 4 commits August 7, 2026 14:09

Add TensorRT-LLM audio and V2V guide

f5deb1c

Signed-off-by: Igor Shovkun <ishovkun@nvidia.com>

Add TensorRT-LLM Transfer, Action, and Distilled guides

7a07a15

Signed-off-by: Igor Shovkun <ishovkun@nvidia.com>

Update TensorRT-LLM Action and Transfer recipes

a454afd

Signed-off-by: Igor Shovkun <igshov@gmail.com>

Fix TensorRT-LLM guardrail cookbook setup

9ceef65

Signed-off-by: Igor Shovkun <igshov@gmail.com>

ishovkun force-pushed the docs/cosmos3-trt-transfer-action-distilled branch from c52332d to 9ceef65 Compare August 29, 2026 06:12

ishovkun marked this pull request as ready for review August 29, 2026 17:39

Merge NVIDIA/cosmos main into TensorRT-LLM cookbook branch

2bd4c31

Signed-off-by: Igor Shovkun <igshov@gmail.com>

ConstBob reviewed Aug 31, 2026

View reviewed changes

Comment thread cookbooks/cosmos3/generator/transfer/README.md Outdated

Comment thread cookbooks/cosmos3/generator/action/run_policy_with_trt_llm.ipynb Outdated

ishovkun added 2 commits August 31, 2026 15:21

docs: align TRT-LLM cookbooks with runtime validation

bf2c652

Signed-off-by: Igor Shovkun <igshov@gmail.com>

docs: align TRT-LLM Transfer input description

82ac400

Signed-off-by: Igor Shovkun <igshov@gmail.com>

ConstBob approved these changes Aug 31, 2026

View reviewed changes

adsridhar approved these changes Sep 6, 2026

View reviewed changes

ishovkun changed the title ~~Add TensorRT-LLM Transfer, Action, and Distilled guides~~ Add TensorRT-LLM Transfer and Action guides Sep 8, 2026

ishovkun added 3 commits September 8, 2026 10:36

docs: use canonical TensorRT-LLM video endpoint

b367242

docs: validate TensorRT-LLM Cosmos3 recipes

394af6c

docs: remove TensorRT-LLM distilled examples

7ff4a86

Signed-off-by: Igor Shovkun <igshov@gmail.com>

ishovkun force-pushed the docs/cosmos3-trt-transfer-action-distilled branch from 5a51304 to 7ff4a86 Compare September 8, 2026 17:37

docs: align TRT-LLM action and transfer recipes

7f973ea

Signed-off-by: Igor Shovkun <igshov@gmail.com>

This was referenced Sep 15, 2026

Add TensorRT-LLM distilled 4-step guide #353

Merged

Add a TensorRT-LLM Cosmos3-Edge audiovisual guide #354

Merged

ishovkun added 2 commits September 18, 2026 08:19

docs: reconcile TRT-LLM Action and Transfer validation

a9029df

Signed-off-by: Igor Shovkun <igshov@gmail.com>

docs: scope September merge hold to removed distilled models

e4d59b6

Signed-off-by: Igor Shovkun <igshov@gmail.com>

skasetty-Siri reviewed Sep 18, 2026

View reviewed changes

docs(cosmos3): validate server-side NLTK guardrail workaround

1595ad3

skasetty-Siri approved these changes Sep 19, 2026

View reviewed changes

Merge upstream main into TensorRT-LLM cookbook guides

6c7f378

ConstBob requested changes Sep 21, 2026

View reviewed changes

ishovkun mentioned this pull request Sep 22, 2026

[None][fix] Compute the Cosmos3 rotary table with a broadcast multiply, not a K=1 matmul NVIDIA/TensorRT-LLM#19532

Merged

1 task

Conversation

ishovkun commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Action

Transfer

Setup and documentation

Testing

Uh oh!

ConstBob left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ConstBob left a comment

Choose a reason for hiding this comment

Uh oh!

adsridhar left a comment

Choose a reason for hiding this comment

Uh oh!

MaciejBalaNV commented Sep 7, 2026

Uh oh!

ConstBob commented Sep 17, 2026

Uh oh!

skasetty-Siri commented Sep 17, 2026

Uh oh!

skasetty-Siri commented Sep 17, 2026

Uh oh!

ishovkun commented Sep 18, 2026

Uh oh!

ishovkun commented Sep 18, 2026

Uh oh!

skasetty-Siri left a comment

Choose a reason for hiding this comment

Uh oh!

ishovkun commented Sep 18, 2026

Uh oh!

skasetty-Siri commented Sep 18, 2026

Uh oh!

ishovkun commented Sep 19, 2026

Uh oh!

skasetty-Siri left a comment

Choose a reason for hiding this comment

Uh oh!

ConstBob left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ishovkun commented Aug 7, 2026 •

edited

Loading