This allows using P2P on 4090 and 5090 GPUs with the 580.95.05 driver version.
See https://github.com/tinygrad/open-gpu-kernel-modules (various branches) for more info.
This modifies the kernel driver to force enable BAR1 P2P mode on GPUs not intended to use it.
Then, the transfers are done by directly writing to the other GPU physical addresses over DMA.
IOMMU virtualization must be disabled to use the patch, or transfers will fail.
Note that this is very dangerous if you run untrusted software or devices.
Proper IOMMU support is possible but would require registering IOVA mappings.
That would be a lot more work and seems unneeded given the intended use.
- Enable DMA passthrough mode for IOMMU:
- Edit
/etc/default/grub - Add
amd_iommu=on iommu=pttoGRUB_CMDLINE_LINUX_DEFAULT - Run
sudo update-grub
- Edit
- Install https://www.nvidia.com/en-us/drivers/details/254665/
- Run
./install.shin this repo - Reboot the server