Skip to content

Restore UVM coherence without registering Blackwell BAR1 as P2PDMA - #1

Closed
ducphuc wants to merge 1 commit into
fix/610-p2p-safety-prerequisitefrom
fix/uvm-blackwell-p2pdma-guard
Closed

Restore UVM coherence without registering Blackwell BAR1 as P2PDMA#1
ducphuc wants to merge 1 commit into
fix/610-p2p-safety-prerequisitefrom
fix/uvm-blackwell-p2pdma-guard

Conversation

@ducphuc

@ducphuc ducphuc commented Aug 4, 2026

Copy link
Copy Markdown
Owner

Summary

  • report UVM coherence from the actual system-memory window again
  • keep device_p2p_initialised false for non-coherent Blackwell
  • return before static BAR1 can be registered with PCI P2PDMA and have its pagemap operations replaced
  • add a debug message exposing static BAR1 size and write-combined state for hardware validation

Static BAR1 is also the GPU peer aperture on this path. Device P2PDMA therefore remains intentionally unavailable on non-coherent Blackwell datacenter GPUs. This matches the current branch effective behavior and does not remove GPU-to-GPU P2P support.

This draft is stacked on the corrected prerequisite branch. Its hardware gate is satisfied, but it should remain draft until aikitoria#33 merges and an upstream one-commit base can be selected.

Validation

  • git diff --check fix/610-p2p-safety-prerequisite..HEAD
  • complete make modules -j64 build on Linux 7.0.0-28-generic
  • test modules signed with the already-enrolled system MOK; the installed signed driver was restored afterward

Stage 1: plain coherence revert

With uvm_debug_prints=1, both GB206 GPUs reached the non-coherent path with static_bar1_write_combined=1, confirming that the existing WC check returns before pci_p2pdma_add_resource(). A prerequisite-only peer-copy run also reproduced the known mailbox assertions fixed by PR aikitoria#31, demonstrating why the final copy validation must include that PR.

Stage 2: final guard plus PR aikitoria#31 integration

  • the defensive skip diagnostic fired for both non-coherent GB206 GPUs
  • simpleP2P passed bidirectional data verification at about 13.09 GB/s
  • p2pBandwidthLatencyTest reported 14.09 GB/s per direction and about 27.8 GB/s aggregate bidirectional
  • nvidia-smi topo -p2p: read, write, and PCIe are OK; atomics are NS, not DR
  • no Xids, assertions, BAR1 failures, IOMMU faults, or P2PDMA conflicts appeared in the final test log

Report coherence from the actual system-memory window again, but keep non-coherent Blackwell devices out of the static-BAR1 PCI P2PDMA registration path. Static BAR1 is also the GPU peer aperture there, and P2PDMA registration replaces its pagemap operations.

This intentionally leaves device P2PDMA unavailable on non-coherent Blackwell datacenter GPUs. It matches this branch’s effective behavior and does not remove GPU-to-GPU P2P support.
@ducphuc

ducphuc commented Aug 5, 2026

Copy link
Copy Markdown
Owner Author

Closing as superseded by production/runtime-bar1-coverage. No further review is needed; the head branch is retained temporarily for historical references.

@ducphuc ducphuc closed this Aug 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant