[Bug]: Llama-3-70B FP16/BF16 inference fails while FP8 works on TensorRT-LLM 0.21.c-rc0 with 8x RTX 6000D

[Dazui-Wang]

Description

I am testing Llama-3-70B inference performance with TensorRT-LLM 0.21.c-rc0 on an 8-GPU NVIDIA RTX 6000D system.

The target workload is:

• Input length: 128 tokens
• Output length: 2048 tokens
• Concurrency: 1 and 4
• Tensor Parallelism (TP): 8
• Pipeline Parallelism (PP): 1

During testing, I found that the behavior is different between FP8 quantization and non-quantized inference.

---

Environment

Hardware

• GPU:
  • 8 x NVIDIA RTX 6000D

Model

• Model:
  • Meta-Llama-3-70B

TensorRT-LLM Version

• TensorRT-LLM:
  • 0.21.c-rc0

Parallel Configuration

• Tensor Parallelism (TP): 8
• Pipeline Parallelism (PP): 1

Quantization

Two configurations were tested:

1. FP8 quantization
2. Without quantization (FP16/BF16)

Software Environment

• OS:debian 12
• CUDA version:13.0
• NVIDIA Driver version:580.82.07
• TensorRT version:0.21.0rc0

---

Issue Description

Case 1: FP8 quantization

The inference runs successfully.

Configuration:

• Model: Llama-3-70B
• GPU: 8 x NVIDIA RTX 6000D
• TP=8
• PP=1
• FP8 quantization

Test results:

Concurrency	Result
1	Successful
4	Successful

---

Case 2: Without quantization

Using the same hardware, model, and parallel configuration, but without quantization, the inference fails.

Configuration:

• Model: Llama-3-70B
• GPU: 8 x NVIDIA RTX 6000D
• TP=8
• PP=1
• FP16/BF16

Test results:

Concurrency	Result
1	Successful
4	Failed

The failure occurs during runtime execution.

Please see the attached logs:

• Attachment 1:
  • Non-quantized inference failure log

non-quantized.log

• Attachment 2:
  • FP8 quantization successful execution log

fp8-quantization.log

---

Benchmark Configuration

input_len=128
output_len=2048

concurrency=1   # or 4

TP=8
PP=1

[bo-nv]

@Dazui-Wang please try the latest trtllm, thanks!

[Dazui-Wang]

@bo-nv We upgraded to the latest v1.3.0rc22 as you suggested, but the issue persists — and is now worse. BF16 with TP>1 fails at engine initialization (not just at concurrency=4), with a different error path.

Environment (updated):

• TRT-LLM: 1.3.0rc22 (latest)
• Same hardware: 16x RTX 6000D (85GB), Driver 580.82.07, Debian 12, CUDA 13.2
• Container: PyTorch 26.05 image, --privileged --ipc=host --gpus all

Reproduction:

trtllm-bench --model meta-llama/Meta-Llama-3-70B \
  --model_path /model/Llama-3-70B \
  throughput \
  --dataset /tmp/synthetic_128_128.txt \
  --backend pytorch \
  --tp 8

Error:

RuntimeError: CUDA Runtime API error: <cudaError_t.cudaErrorInvalidDevice: 101>

Root cause analysis:

The failure occurs in the custom allreduce initialization path:

AllReduce.__init__
  → get_allreduce_workspace
    → CustomAllReduceHelper.allocate_allreduce_fusion_workspace
      → IpcMemory.__init__
        → IpcMemory.open_ipc_memory
          → cudaIpcOpenMemHandle  ← returns error 101 (invalid device ordinal)

Key findings:

1. can_access_peer() in _ipc_utils.py returns True (because cudaDeviceCanAccessPeer succeeds for all GPU pairs), so the code proceeds to open IPC memory handles.
2. cudaIpcOpenMemHandle fails with error 101 inside MPI worker processes spawned by MpiPoolSession.
3. In a single-process test, cudaIpcGetMemHandle on GPU 0 succeeds, but cudaIpcOpenMemHandle on GPU 1 returns error code 1 (cudaErrorInvalidValue), not 101. Error 101 only appears in the multi-process MPI worker context.
4. Container startup log shows: NVIDIA peer memory driver not detected.
5. TP=1 fails with insufficient memory (model needs 131GB, single GPU has 83GB), so there is no working TP configuration for BF16 on this hardware.

All attempts failed:

Attempt	Result
TP=4, TP=2, TP=8 (default container)	Same error 101
--privileged --ipc=host container restart	Same error 101
mpirun -n 8 -x CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7	Same error 101
TRTLLM_DISABLE_CUSTOM_ALLREDUCE=1	Same error 101 (env var appears ineffective in v1.3.0rc22)

Questions:

1. Is there a way to disable custom allreduce in v1.3.0rc22 and fall back to NCCL allreduce?
2. Could the missing nvidia-peermem kernel module on the host cause cudaIpcOpenMemHandle to fail?
3. FP8 works because it uses less memory per GPU — does it take a different allreduce code path that avoids IPC memory?

[Dazui-Wang]

@pjdurden We validated your PR #17034 on a different GPU platform. Good news: the fix works.

Test Environment:

• GPU: 8x RTX 5090 (32GB each, PCIe, no NVLink)
• OS: RHEL 9.5, Driver: 580.82.09
• TRT-LLM: v1.3.0rc22, Model: Llama-3-70B BF16

Before patch (original v1.3.0rc22): Immediate crash at engine init with cudaErrorInvalidDevice: 101 from cudaIpcOpenMemHandle. All 8 ranks failed. Same code path as the RTX 6000D report. This confirms the issue affects all PCIe-only NVIDIA GPUs.

After applying PR #17034 patch: The patch worked as designed:

[WARN] CUDA IPC is not usable on this system: cudaErrorInvalidDevice: 101. Custom all-reduce kernels are disabled, falling back to NCCL.
[WARN] CUDA IPC is unavailable, falling back from AUTO to NCCL allreduce.

No crash. Engine initialized, warmup completed, benchmark ran to completion.

Benchmark results (100 requests, ISL=128, OSL=2048, concurrency=1):

Metric	Value
Total Output Throughput	46.06 tokens/sec
Total Token Throughput	48.93 tokens/sec
Per GPU Output Throughput	5.76 tokens/sec/gpu
Average Request Latency	44467 ms

Throughput is low due to concurrency=1, NCCL_P2P_DISABLE=1, and KV cache 30%. This was a validation run.

Summary:

• PR [None][fix] Fall back to NCCL when CUDA IPC handles cannot be exchanged #17034 correctly degrades from custom allreduce to NCCL. Verified on both RTX 6000D and RTX 5090.
• Additional finding: on RTX 5090, NCCL P2P/CUMEM also fails, requiring NCCL_P2P_DISABLE=1. Separate NCCL issue.

Next steps: We plan to validate FP8 precision on this hardware soon. Keeping this issue open.