Can not reproduce decode rate for Qwen3-4B-Instruct NVFP4 on Jetson Thor

[yuchen-he]

Hi, I am trying to reproduce the numbers from the public table for Qwen3-4B-Instruct NVFP4.
With using llm_bench, I can obtain much better prefill performance than llm_inference. But can not get decode rate as good as 90.2 tok/s (I got about 55 tok/s).
Could you please help check my command or share your commands that generated the numbers?

My environment:

• TensorRT-Edge-LLM version: v.0.70
• Model name: Qwen3-4B-Instruct-2507
• Precision: NVFP4 (including lm_head_quantization)
• --maxInputLen: 1024
• --maxKVCacheCapacity: 4096
• --maxBatchSize: 1
• --inputLen for llm_bench command: 364 (align with your table)

[nvluxiaoz]

Thanks for your question! Prefill performance is heavily sequence length dependent and is not linear with respect to number of tokens, so actually the llm_inference way of average tokens + average time is not the best solution for prefill measurement. It needs to be a strict num_token -> prefill runtime match (and that is why we have llm_bench). For decode, 90.2 tok/s should be a reasonable time. Have you locked your Jetson Thor clock to max using sudo nvpmodel -m 0 and sudo jetson_clocks?

[hsulin0806]

Hi,

I also get around 59 tokens/s when running the benchmark on Thor, which is still quite far from 90.2 tok/s.

May I ask which TensorRT-Edge-LLM version, parameters, or command you used to achieve 90.2 tok/s?

I followed the original poster’s format. The only difference is that my version is v0.71; everything else is the same.

TensorRT-Edge-LLM version: v0.71
Model name: Qwen3-4B-Instruct-2507
Precision: NVFP4 (including lm_head_quantization)
--maxInputLen: 1024
--maxKVCacheCapacity: 4096
--maxBatchSize: 1
--inputLen for llm_bench command: 364 (aligned with your table)

I used llm_bench for the benchmark.
The system was also configured with sudo nvpmodel -m 0 and sudo jetson_clocks.

Thank you very much for your help.

[nvluxiaoz]

I think I understood why. llm_bench, when the profiler is attached, will always contain a heavier profiling overhead. Can you use llm_inference with --dumpProfile instead and use default engine build options? I double checked our data and it could achieve 90 t/s max with llm_inference. Using warmup will also help.