Land immediately verifiable P2P safety fixes - #33
Open
ducphuc wants to merge 7 commits into
Open
Conversation
BAR1 mappings already select SYS_COH or SYS_NONCOH before PTE construction, so the PEER-aperture rewrites and bar1BusAddr base assignment were unreachable. Keep the live BAR1 DMA address encoder unchanged.
Keep BAR1 reads and writes enabled, but encode the atomics field as DEFAULT instead of implicitly disabling it with 0x11. Preserve the branch-wide BAR1 protocol policy, including qualifying PCIe-only datacenter GPUs, and retain registry-key precedence.
Use the reduced page-array path only for non-empty, overflow-safe ranges that are hugepage aligned, exactly sized in hugepages, and contained in one hugetlb VMA. Fall back to the existing base-page path for every other layout.
Report leftover mappings and mappings that outlive their I/O VA space without triggering a debug breakpoint. Use the portable NvU64 format for mappingCount and retain null-safe teardown.
Dynamic-granularity page arrays already use the OS page size, so pass PageCount directly when unregistering them. The RM-to-OS conversion is only correct for ordinary RM-granularity arrays on 64K-page systems.
Match the inactive nv-reg.h mirror to the kernel-open definition. Only os-registry.c instantiates the active table, so this is source hygiene and does not change runtime behavior.
Keep IOMMU passthrough required while translated-IOMMU scatterlist merging remains unsafe, document the hugetlb eligibility fallback, and replace the broad ACS override recommendation with scoped upstream controls and an isolation warning.
There was a problem hiding this comment.
Pull request overview
This PR tightens and clarifies P2P safety behavior in the kernel driver branch by removing unused BAR1 peer-address plumbing, hardening the hugetlb registration fast-path eligibility checks, restoring/adding diagnostics around IOVA/IOVA-lifetime issues, and aligning registry defaults/documentation with the intended safe operating constraints.
Changes:
- Remove dead BAR1 P2P address-plumbing from external alloc PTE/physaddr builders while keeping the live DMA encoder path intact.
- Add stricter hugetlb fast-path eligibility validation (alignment, single-VMA containment, overflow bounds) and fall back to the existing per-page pinning path when ineligible.
- Re-enable IOVA lifetime/mapping diagnostics, sync Resizable BAR global defaults, and update README documentation on current IOMMU/ACS/hugetlb constraints.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| src/nvidia/src/kernel/rmapi/nv_gpu_ops.c | Removes unused BAR1 bus-address threading and redundant aperture overrides in external mapping builders. |
| src/nvidia/src/kernel/mem_mgr/io_vaspace.c | Restores/widens diagnostics for leftover mappings and dangling mapping lifetimes. |
| src/nvidia/src/kernel/gpu/bif/kernel_bif.c | Adjusts default P2P registry override to enable reads/writes while leaving atomics at platform default. |
| src/nvidia/arch/nvalloc/unix/src/osmemdesc.c | Fixes page-count accounting selection for dynamic granularity paths during page-array teardown. |
| src/nvidia/arch/nvalloc/unix/include/nv-reg.h | Changes global default for resizable BAR enablement to match the mirrored registry header. |
| README.md | Documents current safety constraints (IOMMU translated mode limitations for hugetlb path; safer ACS guidance). |
| kernel-open/nvidia/os-mlock.c | Adds hugetlb fast-path eligibility helper with stronger validation and safe fallback behavior. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This was referenced Aug 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The retained global
pcieP2PType = BAR1default also selects BAR1 on qualifying PCIe-only datacenter GPUs; the existing static-BAR1 support checks still gate use of that protocol.Validation
git diff --check upstream/610.43.03-p2p..HEADmake modules -j64build on Linux 7.0.0-28-genericinstall.share unchangednvidia-smi topo -p2p: read, write, and PCIe areOK; atomics areNS, notDR, confirming this branch regkey no longer disables atomicssimpleP2P: bidirectional correctness passed at about 13.09 GB/sp2pBandwidthLatencyTest: 14.09 GB/s per direction and about 27.8 GB/s aggregate bidirectionalThe GB206 copy validation includes PR #31 because that PR contains the display-aware BAR1 and transactional mailbox fixes and is intended to rebase immediately after this prerequisite merges.