Fix BAR1 P2P on display-attached GPUs

[xycjscs]

Summary

Force static BAR1 by default so BAR1-based PCIe P2P remains functional when one of the GPUs is also driving the display.

The existing AUTO policy may reject static BAR1 on a display-attached 32 GB GPU even though the client-visible framebuffer mapping actually fits inside BAR1. Once static BAR1 is
disabled on one GPU, P2P connections involving that GPU fall through to the mailbox path, which is not initialized when the driver is configured to use BAR1 P2P.

When the problem occurs

The issue was reproduced on the following system:

• 3× NVIDIA GeForce RTX 5090
• 32 GB BAR1 per GPU
• 256 GB system RAM
• GPU0 attached to the display and used by Xorg/GNOME
• GPU1 and GPU2 used as headless compute GPUs
• Linux 7.0.0-28-generic
• NVIDIA driver 610.43.03
• IOMMU passthrough enabled with iommu=pt

All three GPU pairs supported working P2P before the system memory was upgraded to 256 GB.

After the memory upgrade, the platform PCIe/MMIO resource layout changed. P2P continued to work normally between the two headless GPUs, but communication between the display GPU
and either compute GPU stopped working correctly:

• GPU0 ↔ GPU1: broken
• GPU0 ↔ GPU2: broken
• GPU1 ↔ GPU2: working

CUDA still reported that all devices could access their peers, so the connectivity matrix alone did not expose the failure.

Observed behavior

The CUDA p2pBandwidthLatencyTest produced implausible results for pairs involving GPU0:

• GPU0 ↔ GPU1: approximately 128 GB/s
• GPU0 ↔ GPU2: approximately 128 GB/s
• GPU1 ↔ GPU2: approximately 14.3 GB/s

The approximately 128 GB/s values were not real PCIe bandwidth. The affected mappings did not point to a valid peer framebuffer region.

The kernel log also contained repeated assertions:

Assertion failed: ((base & RM_PAGE_MASK) == 0) @ kern_bus.c:298
Assertion failed: remoteWMBoxLocalAddr != ~0ULL @ kern_bus_gm200.c:89

Diagnostic logging showed:

GPU0 static BAR1: disabled
GPU1 static BAR1: enabled
GPU2 static BAR1: enabled

For the broken pairs, the driver attempted to construct a mailbox mapping using:

writeMailboxBar1Addr = 0xffffffffffffffff

Adding that sentinel value to the peer BAR address produced an invalid, unaligned address immediately below the BAR aperture. This explains both the assertions and the
meaningless bandwidth result.

The mailbox was not allocated because this branch configures PCIe P2P to use BAR1. Therefore, falling back to the mailbox path is invalid in this configuration.

Root cause

GPU0 was the only GPU driving a display and consequently had a console reservation at the beginning of BAR1.

The AUTO static-BAR1 sizing policy includes:

• The client-visible framebuffer mapping
• Worst-case UserD space
• Doorbell space
• MMIO overhead
• Console reservation
• BAR1 alignment requirements

This conservative calculation rejected static BAR1 on GPU0.

However, the actual client-visible mapping was approximately:

0x7d9400000 bytes

GPU0 used a 0x20000000 (512 MiB) BAR1 offset for its console reservation. The resulting static mapping still fit inside its 32 GB BAR1 aperture.

The ENABLE path already performs the relevant check against the actual client-visible mapping before creating the static mapping. It therefore allows the valid GPU0 mapping
without removing the existing capacity validation.

Change

Change the default RMForceStaticBar1 policy from AUTO to ENABLE:

pKernelBus->staticBar1ForceType =
NV_REG_STR_RM_FORCE_STATIC_BAR1_ENABLE;

An explicit registry override is still honored. Only the default behavior changes.

Result

After the change, static BAR1 was enabled on all three GPUs. All mappings used valid peer BAR1 addresses, and no mailbox assertions occurred.

Measured unidirectional P2P write bandwidth:

Pair Before After
━━━━━━━━━━━━━ ━━━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━━
GPU0 → GPU1 Invalid ~128 GB/s 14.32 GB/s
───────────── ─────────────────── ────────────
GPU1 → GPU0 Invalid ~128 GB/s 14.33 GB/s
───────────── ─────────────────── ────────────
GPU0 → GPU2 Invalid ~128 GB/s 28.63 GB/s
───────────── ─────────────────── ────────────
GPU2 → GPU0 Invalid ~122 GB/s 28.07 GB/s
───────────── ─────────────────── ────────────
GPU1 → GPU2 14.33 GB/s 14.33 GB/s
───────────── ─────────────────── ────────────
GPU2 → GPU1 14.32 GB/s 14.32 GB/s

The different bandwidth between GPU pairs is consistent with the motherboard PCIe topology. The important result is that every pair now reports credible PCIe bandwidth and
performs real peer communication.

The test also confirmed:

• All three GPUs have static BAR1 enabled.
• DMA addresses correspond to valid peer BAR1 mappings.
• No invalid mailbox path is used.
• No BAR alignment or uninitialized-mailbox assertions are emitted.
• P2P latency remains below 0.5 μs for peer writes.

[ducphuc]

Thanks for documenting the GB202/display-attached reproduction and the invalid mailbox address; that confirms this failure class is not GB206-specific.

I opened #35 with an alternative that retains the existing AUTO policy and uses runtime BAR1 geometry to select display-aware placement when the remaining aperture covers all aligned client framebuffer memory. It preserves the hardware-tested GB206 partial-window exception without generalizing partial coverage.

The concern with changing the global default to NV_REG_STR_RM_FORCE_STATIC_BAR1_ENABLE is that the force path checks whether the client-visible FB mapping fits, but intentionally bypasses AUTO's UserD, doorbell, MMIO, and dynamic-BAR1 capacity policy. That may solve this 32 GiB GB202 geometry while creating BAR1 pressure on other property/device configurations.

#35 also makes the uninitialized mailbox path transactional, validates external BAR1 addresses against the DMA window, documents that uncovered GB206 allocations reject rather than transparently fall back, and includes natural-boundary plus 100-cycle rejection/recovery testing. Its GB202 full-coverage runtime predicate should cover the geometry reported here without globally forcing static BAR1.

[xycjscs]

Sounds good. I will test your PR next week to see if it works for this issue.

[xycjscs]

Superseded by this PR.

The original global RMForceStaticBar1=ENABLE change was useful for confirming the root cause,
but the new implementation retains AUTO, requires complete runtime coverage, and preserves
dynamic BAR1 capacity accounting. It has been validated on the same 3× RTX 5090 system.