Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 (LLM part) failed to convert to TensorRT Engine

[BitCircuit]

Describe the bug

During the model conversion of Nemotron Omni from ONNX to TensorRT Engine on the target platform, the error message shows:

...
[07:48:03.497] [INFO] [TensorRT] Successfully created plugin: Nvfp4MoePlugin
[07:48:03.516] [INFO] [llmBuilder.cpp:118:build] Created directory Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4/omni_engine_nvfp4.trtedgellm070/llm for saving LLM engine.
[07:48:03.546] [ERROR] [TensorRT] IBuilder::buildSerializedNetwork: Error Code 9: API Usage Error (INT8 and FP8 mixed precision is allowed only when building network with kSTRONGLY_TYPED mode on Blackwell+ platforms.)
[07:48:03.546] [ERROR] [builderUtils.cpp:313:buildAndSerializeEngine] Failed to build serialized engine
[07:48:04.430] [ERROR] [llm_build.cpp:242:main] Failed to build LLM engine.

Steps/Code to reproduce bug

1. Download model from https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4
2. Change model_type in config.json from NemotronH_Nano_Omni_Reasoning_V3 to NemotronH_Nano_VL_V2
3. Convert model to ONNX by using experimental llm_loader
4. Send model & cross-compiled TensorRT Edge-LLM program to Nvidia Drive Thor U (DriveOS 7.0.3)
5. Set /proc/sys/vm/nr_hugepages to 16384
6. Run ./build/examples/llm/llm_build --onnxDir Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4/omni_onnx_nvfp4.trtedgellm070/llm/ --engineDir Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4/omni_engine_nvfp4.trtedgellm070/llm

Build configuration:

cmake .. \
    -DCMAKE_BUILD_TYPE=Release \
    -DTRT_PACKAGE_DIR=/usr \
    -DCMAKE_TOOLCHAIN_FILE=cmake/aarch64_linux_toolchain.cmake \
    -DEMBEDDED_TARGET=auto-thor \
    -DCUDA_CTK_VERSION=12.8

Expected behavior

The LLM build should be completed. And I can run this omni model with llm_inference

System information (Edge Device)

• Platform (e.g., NVIDIA Jetson Thor): Drive Thor U
• Software release (e.g., JetPack 7.1): DriveOS 7.0.3
• CPU architecture: aarch64
• GPU compute capability (e.g., SM110 for Jetson Thor): SM101
• Total device memory: 59045MB
• Build type (e.g., Release, Debug): Release
• Library versions:
  • TensorRT Edge-LLM version or commit hash: 0.7.0
  • CUDA: 12.8
  • TensorRT: 10.10
  • C++ compiler (e.g., GCC 11.4): GCC 13.3 (x86 docker dev env)

[lix19937]

Drive thor-u is Blackwell arch, but INT8 and FP8 mixed precision is allowed only when building network with kSTRONGLY_TYPED mode on Blackwell+ platforms.

[BitCircuit]

@lix19937 Thanks for replying. Are you saying "Blackwell+ platform" does not include "Blackwell platform"? But doc states that this omni model can be run on Jetson Thor, which is also Blackwell platform, isn't it?

[nvluxiaoz]

DOS 7.0.3 is not what we officially claimed support and massively tested against, since it was released much earlier than TensorRT Edge-LLM. However we are interested to see why it has such failures. Do you see the same failure for some other smaller models, or only this model? In addition, can you share the build log with --debug, so we have better insights on what happened? Thanks

[BitCircuit]

@nvluxiaoz Hi, thank you for replying. I have captured the debug log while converting the same model with TensorRT-Edge-LLM version 0.7.1. Please check the attachment. As you may see, only llm part is not able to make it.
I have not gotten any chance to try any other smaller models with TensorRT-Edge-LLM with version beyond 0.6.1. But the models I have tried with older versions work just fine.

nemotron-omni-moe-audio-build.trtedgellm071.log
nemotron-omni-moe-llm-build.trtedgellm071.log
nemotron-omni-moe-visual-build.trtedgellm071.log

[lix19937]

From https://github.com/NVIDIA/TensorRT-Edge-LLM/blob/release/0.7.1/cpp/builder/builderUtils.cpp#L231-L245 , can see network has been built with kSTRONGLY_TYPED.

[nvamberl]

Hi @BitCircuit , I can reproduce this issue on DOS7.0.3.0 + TRT10.10. Is it possible for you to upgrade to newer DOS + TRT versions? DOS7.0.5.0 + TRT10.14 or newer versions should work.

[fans-nv]

Seems coming from the fact we pack FP4 weights and pass as int8 dtype which doesn't work well in this TRT version. Should be resolvable by changing the weights type representation.

[lix19937]

In Drive OS 7030/THOR-U, if onnx include int8 + fp8 quant nodes, it reports error when build with trt, user can split the onnx int8 part and fp8 part, then separate build and infer.