Restore UVM coherence without registering Blackwell BAR1 as P2PDMA - #1
Closed
ducphuc wants to merge 1 commit into
Closed
Restore UVM coherence without registering Blackwell BAR1 as P2PDMA#1ducphuc wants to merge 1 commit into
ducphuc wants to merge 1 commit into
Conversation
Report coherence from the actual system-memory window again, but keep non-coherent Blackwell devices out of the static-BAR1 PCI P2PDMA registration path. Static BAR1 is also the GPU peer aperture there, and P2PDMA registration replaces its pagemap operations. This intentionally leaves device P2PDMA unavailable on non-coherent Blackwell datacenter GPUs. It matches this branch’s effective behavior and does not remove GPU-to-GPU P2P support.
Owner
Author
|
Closing as superseded by production/runtime-bar1-coverage. No further review is needed; the head branch is retained temporarily for historical references. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
device_p2p_initialisedfalse for non-coherent BlackwellStatic BAR1 is also the GPU peer aperture on this path. Device P2PDMA therefore remains intentionally unavailable on non-coherent Blackwell datacenter GPUs. This matches the current branch effective behavior and does not remove GPU-to-GPU P2P support.
This draft is stacked on the corrected prerequisite branch. Its hardware gate is satisfied, but it should remain draft until aikitoria#33 merges and an upstream one-commit base can be selected.
Validation
git diff --check fix/610-p2p-safety-prerequisite..HEADmake modules -j64build on Linux 7.0.0-28-genericStage 1: plain coherence revert
With
uvm_debug_prints=1, both GB206 GPUs reached the non-coherent path withstatic_bar1_write_combined=1, confirming that the existing WC check returns beforepci_p2pdma_add_resource(). A prerequisite-only peer-copy run also reproduced the known mailbox assertions fixed by PR aikitoria#31, demonstrating why the final copy validation must include that PR.Stage 2: final guard plus PR aikitoria#31 integration
simpleP2Ppassed bidirectional data verification at about 13.09 GB/sp2pBandwidthLatencyTestreported 14.09 GB/s per direction and about 27.8 GB/s aggregate bidirectionalnvidia-smi topo -p2p: read, write, and PCIe areOK; atomics areNS, notDR