Enable display-aware BAR1 P2P using runtime coverage

[ducphuc]

Important

Depends on #33. The commits through cc19db21 are the prerequisite safety fixes from that PR; please review the follow-up commits after that point. This branch will be rebased onto 610.43.03-p2p after #33 merges.

Summary

Enable display-aware static BAR1 placement using runtime aperture geometry rather than globally forcing static BAR1 or maintaining a device exception:

• every property-enabled GPU is eligible when its aligned client framebuffer size and usable static BAR1 window are both nonzero;
• the static map covers min(clientFbSizeAligned, maxStaticMapSize), so both full and partial runtime geometries use the same policy;
• only allocations wholly inside the selected static window are supported; spanning and outside allocations fail closed with NV_ERR_NOT_SUPPORTED;
• external BAR1 P2P addresses are validated against the DMA window;
• PCIe mailbox setup is transactional and rejects an uninitialized mailbox address instead of leaving partial peer state;
• UVM reports actual system-memory coherence while preventing non-coherent Blackwell static BAR1 from being registered as PCI P2PDMA memory.

Existing generated HAL routing, PDB_PROP_KBUS_SUPPORT_BAR1_P2P_BY_DEFAULT, the global pcieP2PType policy, registry precedence, explicit static-BAR1 overrides, BAR-disabled checks, and forced static-BAR behavior remain unchanged. No chip, architecture, marketing-family, or device-ID allowlist is added.

Motivation and relationship to other PRs

This supersedes closed PR #31 with corrected boundary semantics, broader runtime-geometry policy, and complete validation.

PR #34 reproduces the same display-attached failure class on GB202, but changes the global default from AUTO to forced ENABLE. This PR retains AUTO and uses measured runtime geometry to select display-aware placement. It also addresses the invalid-mailbox failure path and external-address bounds that a global policy change does not cover.

Runtime-scoped partial coverage

Partial coverage is a runtime geometry condition, not a GB206 capability. When the usable static window is smaller than aligned client framebuffer memory, allocations wholly inside that window can be represented safely; spanning or outside allocations cannot and are rejected by the shared static-aperture and DMA-window bounds checks.

GB206 is the hardware-validated partial-coverage example, not an eligibility exception. Its matrix covers both GPU directions and peer-enable orderings, inside, spanning, and fully outside placements, 100 rejection-and-recovery cycles, cleanup, driver reload, suspend/resume, and reboot. The generalized predicate was subsequently regressed on the same hardware with unchanged inside success and fail-closed boundary behavior.

Other property-enabled devices with partial runtime coverage use the same contract. Hardware validation on those implementations remains follow-up work and should repeat the boundary, recovery, cross-GPU, and lifecycle matrix; it is not a merge gate for this runtime-only generalization.

Related upstream issues

• Resolves #11 — display-attached RTX 5090 P2P failure by enabling display-aware static BAR1 from runtime geometry.
• Resolves the invalid-mailbox failure class reported in #6 — RTX 5090 corruption and low P2P bandwidth by selecting a valid static mapping when eligible and otherwise rejecting and rolling back uninitialized mailbox setup.
• Addresses #13 — P2P among RTX 5060 Ti, 5070 Ti, 5080, and 5090 by applying the same runtime policy to full and partial geometries across all property-enabled implementations. Cross-die GB203/GB206 validation remains outstanding, so this PR does not claim to close the entire issue.
• Addresses #26 — mixed Ada/Blackwell CUDA initialization failure by restoring UVM's actual system-memory coherence classification. Mixed-generation hardware validation remains outstanding.
• Related to #7 — invalid BAR1 mailbox assertions: this PR prevents unsafe mailbox continuation and partial peer state, but it cannot enable BAR1 P2P when the platform exposes only a 256 MiB BAR.
• Related to #9 — RTX 5090 Resizable BAR and display-attached failures: this PR fixes the display/static-placement subset but cannot fix firmware or PCI resource assignment failures that prevent a usable BAR from being exposed.
• Related to #32 — gibberish output and crashes under P2P workloads: the address bounds and deterministic rejection protect against unsafe BAR1 mappings, but that report still needs reproduction on this branch before a fix can be claimed.

Validation

Validated on two RTX 5060 Ti 16 GB GPUs with natural BAR1 geometry and enrolled-MOK-signed modules:

• complete five-module build and git diff --check passed;
• the policy regression passed with make -C tests clean check, covering disabled policy, zero-sized geometry, partial coverage, exact coverage, and a static window larger than client FB;
• source inspection found no remaining GB206 eligibility check and confirmed generated HAL and registry sources were unchanged;
• inside mappings passed in both GPU directions and both peer-enable orderings, including peer kernels, cudaMemcpyPeer, and beginning/middle/end probes;
• the original full-coverage run completed 1,969 peer tests before normal allocation exhaustion;
• the original partial-coverage run completed 491 mappings before a classified 64 MiB spanning allocation was rejected;
• a prepared fully outside mapping was rejected with local data intact and immediate three-iteration inside recovery;
• 100 independent spanning/outside rejection-and-recovery cycles completed with zero data errors and zero retained GPU memory at each checkpoint;
• after the policy generalization, a bounded regression produced 492 inside passes, rejected the established 64 MiB spanning and prepared 4 MiB outside candidates, and passed immediate recovery with no data errors;
• simpleP2P passed before and after the generalized boundary run at 13.05 and 13.03 GB/s;
• p2pBandwidthLatencyTest measured about 14.09 GB/s per direction and 27.79 GB/s bidirectional;
• driver unload/reload, modeset/DRM loading, supported deep suspend/resume, updated-initramfs reboot, and post-resume/post-boot P2P passed in the original lifecycle run;
• no Xids, assertions, IOMMU/MMU faults, AER errors, oopses, panics, or hung tasks appeared after loading the generalized modules. Expected fail-closed boundary diagnostics were present.

P2P atomics report NS, not registry-disabled DR, confirming that the branch default no longer disables atomics.

The complete durable evidence is in the GB206 BAR1 boundary validation record. That record is evidence for GB206 behavior under the generalized contract; it does not claim other partial-window implementations were hardware-tested.

Separately, the shared full-coverage AUTO-selection logic was independently validated on three RTX 5090 GPUs in PR #36.

Residual hardware scope

The runtime policy is additive and source-audited across existing HAL coverage. Runtime partial-window testing was performed on GB206. Other property-enabled implementations are supported by the same runtime predicate and fail-closed bounds, but their partial-window and mixed-implementation hardware matrices remain explicit follow-up validation work.

[Copilot]

Pull request overview

Enable display-aware static BAR1 placement for BAR1-based PCIe P2P using runtime aperture coverage (full coverage required except for the existing GB206 partial-window exception), while tightening BAR1 P2P address validation and hardening PCIe mailbox P2P setup/rollback behavior. This extends the BAR1 P2P enablement logic without changing generated HAL routing or the global BAR1 protocol selection policy.

Changes:

• Add a runtime-coverage BAR1 policy predicate and use it to drive display-aware static BAR1 placement/clipping, including predictable rejection of spanning/outside mappings.
• Validate BAR1 P2P encoded addresses against the DMA window and refactor external PTE/phys-addr construction to match the selected aperture semantics.
• Make PCIe mailbox mapping setup transactional with cleanup/rollback, and add a small source-level regression test harness for the policy truth table.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
validation/gb206-bar1-boundary-2026-08-04.md	Adds durable GB206 boundary validation record for the new runtime-coverage behavior.
tests/README.md	Documents how to run the BAR1 policy regression test.
tests/Makefile	Adds a minimal build/run harness for the BAR1 policy test.
tests/bar1_p2p_policy_test.c	Introduces a truth-table regression test for the display-aware BAR1 policy predicate.
tests/.gitignore	Ignores the policy test binary.
src/nvidia/src/kernel/rmapi/nv_gpu_ops.c	Adds DMA-window bounds checks for BAR1 P2P address encoding and simplifies aperture handling for external mappings.
src/nvidia/src/kernel/mem_mgr/io_vaspace.c	Adds warning diagnostics for dangling IOVAS/mappings during teardown.
src/nvidia/src/kernel/gpu/bus/arch/turing/kern_bus_tu102.c	Implements runtime-coverage display-aware static BAR1 placement logic and updates boundary behavior semantics.
src/nvidia/src/kernel/gpu/bus/arch/turing/bar1_p2p_policy.h	Adds the shared BAR1 policy predicate macro.
src/nvidia/src/kernel/gpu/bus/arch/maxwell/kern_bus_gm200.c	Makes PCIe mailbox P2P setup transactional, with failure-path teardown and peer-mask rollback helpers.
src/nvidia/src/kernel/gpu/bif/kernel_bif.c	Updates default P2P override encoding to avoid overriding atomic capability defaults.
src/nvidia/arch/nvalloc/unix/src/osmemdesc.c	Adjusts OS page-count logic for hugepages and dynamic granularity descriptors.
src/nvidia/arch/nvalloc/unix/include/nv-reg.h	Modifies the default value for the resizable BAR registry key.
README.md	Updates user-facing documentation for runtime-coverage display-aware BAR1 and operational constraints.
kernel-open/nvidia/os-mlock.c	Refactors and tightens eligibility checks for the hugetlb pinning fast path.
kernel-open/nvidia-uvm/uvm_gpu.h	Removes unconditional coherence reporting for Blackwell+ and uses actual system-bus window reporting.
kernel-open/nvidia-uvm/uvm_devmem.c	Skips PCI P2PDMA static BAR1 registration on non-coherent Blackwell to avoid conflicts with BAR1 peer access.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.