Performance / TTFT & TPS comparison with vLLM for single VLM inference on Jetson Orin 32GB

[Syd-Q]

Hi, I'm planning to use this tool for VLM inference on a Jetson Orin 32GB platform. Have you conducted any performance testing on this platform? I'm mainly interested in metrics like TTFT and TPS .

Additionally, do you have any comparison data with vLLM for single inferenceon similar hardware?
I'm trying to understand whether, without considering parallel throughput, your tool would be faster than vLLM for my use case.

Thanks!

[nvluxiaoz]

Do you have any models and precisions in mind? Jetpack 6.2 are compatible platforms but we are confident that our FP16 and INT4 bs = 1 decoding performance is great. We usually do not publish such results since both softwares belong to Jetson community. We could provide performance data for Jetson Orin in future releases for important models, but I want to understand your workload. Thanks.