diff --git a/README.md b/README.md index 2999d9671c..f9d6f060d0 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ # NVIDIA driver 610.43.03 with P2P for RTX 3090, RTX 4090, and RTX 5090 -This enables P2P on consumer GPUs with the 610.43.03 driver version. No kernel parameters -are needed for the default behavior, just build, install, and go. +This enables P2P on consumer GPUs with the 610.43.03 driver version. The current branch +requires the IOMMU passthrough configuration described below. See the [tinygrad 550.54.15-p2p README](https://github.com/tinygrad/open-gpu-kernel-modules/blob/550.54.15-p2p/README.md) for the original description of the approach. @@ -13,6 +13,7 @@ for the original description of the approach. | RTX 3090 | Pairwise NVLink where available, PCIe BAR1 otherwise | | RTX 4090 | PCIe BAR1 | | RTX 5090 | PCIe BAR1 | +| RTX 5060 Ti / 5060 (GB206) | PCIe BAR1, including with a display attached | P2P also works between different devices of the same generation, for example RTX 5090 to RTX PRO 6000 Blackwell. @@ -23,10 +24,18 @@ This enables BAR1 P2P on consumer GPUs where NVLink isn't available, and falls b NVLink where it is. For PCIe pairs, transfers write directly to the other GPU's physical address over DMA. +On property-enabled GPUs, display-aware static BAR1 placement is used whenever runtime +geometry leaves a non-empty aligned static window after fixed console and mailbox +reservations. Partial windows support allocations wholly inside that window; allocations +spanning or outside it are rejected by the CUDA API because there is currently no +transparent dynamic-mapping fallback. GB206 cards (RTX 5060 Ti / 5060) are the +hardware-validated partial-coverage example, not an implementation allowlist. + > [!WARNING] -> IOMMU must be in passthrough mode (`iommu=pt`), not translating, or DMA will go through -> IOMMU page tables and transfers will fail. This is very dangerous if you run untrusted -> software or devices. +> IOMMU must currently be in passthrough mode (`iommu=pt`), not translating. In particular, +> the experimental hugetlb registration path does not yet handle scatterlist entries merged +> by a translated IOMMU. Do not use translated mode until that path is fixed and validated. +> Passthrough mode weakens DMA isolation and is unsafe with untrusted software or devices. ## How to use @@ -52,17 +61,19 @@ options nvidia NVreg_RegistryDwords="RMForceP2PType=1" This branch also includes an experimental path that accelerates `cudaHostRegister` by several orders of magnitude when the registered buffer is backed by 1G hugepages, and -shrinks the device page tables used for such mappings. It is enabled automatically. This -path skips some of the per-4K-page bookkeeping the stock driver performs, so it may -misbehave in edge cases the stock driver handles correctly. +shrinks the device page tables used for such mappings. It is enabled automatically for a +non-empty registration that is hugepage-aligned, is an exact multiple of the hugepage +size, and stays within one hugetlb VMA. Other layouts use the normal per-page array path. +The fast path still skips some base-page bookkeeping and remains experimental. ## Potential issues If P2P transfers are slow, make sure your IOMMU is in passthrough (`pt`) mode and that ACS -is disabled. ACS on root ports forces all GPU-to-GPU traffic through the CPU root complex, -killing P2P bandwidth. ACS can be disabled in BIOS, with the -`pcie_acs_override=downstream,multifunction` kernel parameter (if your kernel supports it), -or with an ACS override patch applied to the kernel. +redirect is not forcing GPU-to-GPU traffic through the root complex. Prefer a firmware ACS +control. If the kernel supports the upstream per-device option, use a narrowly scoped +`pci=disable_acs_redir=[;...]` setting and verify the resulting IOMMU groups. +Disabling ACS redirect weakens device isolation; do not use the broad `pcie_acs_override` +patch or kernel parameter. ## Sample `p2pBandwidthLatencyTest` output diff --git a/kernel-open/nvidia-uvm/uvm_devmem.c b/kernel-open/nvidia-uvm/uvm_devmem.c index 38ca9713f5..bfa5b5f006 100644 --- a/kernel-open/nvidia-uvm/uvm_devmem.c +++ b/kernel-open/nvidia-uvm/uvm_devmem.c @@ -613,6 +613,18 @@ void uvm_devmem_device_p2p_init(uvm_parent_gpu_t *parent_gpu) parent_gpu->device_p2p_initialised = false; + if (parent_gpu->rm_info.gpuArch >= NV2080_CTRL_MC_ARCH_INFO_ARCHITECTURE_GB100) { + // Static BAR1 is also the GPU peer aperture on non-coherent Blackwell. + // Registering it as P2PDMA memory would replace its pagemap operations + // and conflict with the BAR1-as-sysmem PTEs used for GPU peer access. + UVM_DBG_PRINT("Skipping PCI P2PDMA static BAR1 registration on non-coherent GPU %s " + "(size 0x%llx, write-combined %u)\n", + uvm_parent_gpu_name(parent_gpu), + parent_gpu->static_bar1_size, + parent_gpu->static_bar1_write_combined); + return; + } + // RM sets static_bar1_size when it has created a contiguous BAR mapping // large enough to cover all of GPU memory that will be allocated to // userspace buffers. This is required to support the P2PDMA feature to diff --git a/kernel-open/nvidia-uvm/uvm_gpu.h b/kernel-open/nvidia-uvm/uvm_gpu.h index 7761f569cb..dedbfcce51 100644 --- a/kernel-open/nvidia-uvm/uvm_gpu.h +++ b/kernel-open/nvidia-uvm/uvm_gpu.h @@ -1825,13 +1825,6 @@ NvU64 uvm_parent_gpu_canonical_address(uvm_parent_gpu_t *parent_gpu, NvU64 addr) static bool uvm_parent_gpu_is_coherent(const uvm_parent_gpu_t *parent_gpu) { - // Blackwell+ consumer GPUs (e.g. 5090) use BAR1 P2P via the SYS_COH - // aperture rewrite in nvGpuOpsBuildExternalAllocPtes. UVM's P2P - // registration path must take the coherent route to match, otherwise - // the ZONE_DEVICE peer DMA setup conflicts with BAR1-as-sysmem PTEs. - if (parent_gpu->rm_info.gpuArch >= NV2080_CTRL_MC_ARCH_INFO_ARCHITECTURE_GB100) - return true; - return parent_gpu->system_bus.memory_window_end > parent_gpu->system_bus.memory_window_start; } diff --git a/kernel-open/nvidia/os-mlock.c b/kernel-open/nvidia/os-mlock.c index 994c326328..e038aa736f 100644 --- a/kernel-open/nvidia/os-mlock.c +++ b/kernel-open/nvidia/os-mlock.c @@ -264,6 +264,54 @@ static void nv_free_page_array(struct page **pages) os_free_mem((NvU8 *)pages - NV_PAGE_ARRAY_HEADER_SIZE); } +static NvBool nv_hugetlb_fast_path_eligible( + struct vm_area_struct *vma, + unsigned long start, + NvU64 page_count, + unsigned long *hpage_size_out, + unsigned int *compound_order_out +) +{ + NvU64 range_size; + NvU64 pages_per_hugepage; + unsigned long end; + unsigned long hpage_size; + + if (!vma || !is_vm_hugetlb_page(vma) || page_count == 0) + return NV_FALSE; + + if (page_count > ((NvU64)ULONG_MAX / PAGE_SIZE)) + return NV_FALSE; + + range_size = page_count * PAGE_SIZE; + if (range_size > ULONG_MAX || start > (ULONG_MAX - (unsigned long)range_size)) + return NV_FALSE; + + end = start + (unsigned long)range_size; + hpage_size = vma_kernel_pagesize(vma); + + if (hpage_size < PAGE_SIZE || hpage_size > NV_U32_MAX || + (hpage_size & (hpage_size - 1)) != 0) + { + return NV_FALSE; + } + + if ((start & (hpage_size - 1)) != 0 || + (range_size & (hpage_size - 1)) != 0 || + end > vma->vm_end) + { + return NV_FALSE; + } + + pages_per_hugepage = hpage_size / PAGE_SIZE; + if ((pages_per_hugepage & (pages_per_hugepage - 1)) != 0) + return NV_FALSE; + + *hpage_size_out = hpage_size; + *compound_order_out = ilog2(pages_per_hugepage); + return NV_TRUE; +} + NV_STATUS NV_API_CALL os_lock_user_pages( void *address, NvU64 page_count, @@ -303,47 +351,48 @@ NV_STATUS NV_API_CALL os_lock_user_pages( * unfaulted 1GB hugepages (~37ms per fault). */ { - struct vm_area_struct *vma = vma_lookup(mm, (unsigned long)address); - - if (vma && is_vm_hugetlb_page(vma)) + unsigned long start = (unsigned long)address; + struct vm_area_struct *vma = vma_lookup(mm, start); + unsigned long hpage_size; + unsigned int order; + + if (nv_hugetlb_fast_path_eligible(vma, + start, + page_count, + &hpage_size, + &order)) { - unsigned long hpage_size = vma_kernel_pagesize(vma); - unsigned int order = ilog2(hpage_size / PAGE_SIZE); - NvU64 pages_per_hp = 1ULL << order; + NvU64 num_hugepages = page_count >> order; - if ((page_count & (pages_per_hp - 1)) == 0) + rmStatus = nv_alloc_page_array(num_hugepages, order, &user_pages); + if (rmStatus != NV_OK) { - NvU64 num_hugepages = page_count >> order; + nv_mmap_read_unlock(mm); + nv_printf(NV_DBG_ERRORS, + "NVRM: failed to allocate hugepage table!\n"); + return rmStatus; + } - rmStatus = nv_alloc_page_array(num_hugepages, order, &user_pages); - if (rmStatus != NV_OK) + for (i = 0; i < num_hugepages; i++) + { + ret = NV_PIN_USER_PAGES(start + i * hpage_size, + 1, + gup_flags, + &user_pages[i]); + if (ret != 1) { + for (j = 0; j < i; j++) + NV_UNPIN_USER_PAGE(user_pages[j]); + nv_free_page_array(user_pages); nv_mmap_read_unlock(mm); - nv_printf(NV_DBG_ERRORS, - "NVRM: failed to allocate hugepage table!\n"); - return rmStatus; - } - - for (i = 0; i < num_hugepages; i++) - { - ret = NV_PIN_USER_PAGES( - (unsigned long)address + i * hpage_size, - 1, gup_flags, &user_pages[i]); - if (ret != 1) - { - for (j = 0; j < i; j++) - NV_UNPIN_USER_PAGE(user_pages[j]); - nv_free_page_array(user_pages); - nv_mmap_read_unlock(mm); - return NV_ERR_INVALID_ADDRESS; - } + return NV_ERR_INVALID_ADDRESS; } + } - nv_mmap_read_unlock(mm); + nv_mmap_read_unlock(mm); - *page_array = user_pages; - return NV_OK; - } + *page_array = user_pages; + return NV_OK; } } diff --git a/src/nvidia/arch/nvalloc/unix/include/nv-reg.h b/src/nvidia/arch/nvalloc/unix/include/nv-reg.h index 93999d4f3c..decf092752 100644 --- a/src/nvidia/arch/nvalloc/unix/include/nv-reg.h +++ b/src/nvidia/arch/nvalloc/unix/include/nv-reg.h @@ -1051,7 +1051,7 @@ NV_DEFINE_REG_ENTRY_GLOBAL(__NV_NVLINK_DISABLE, 0); NV_DEFINE_REG_ENTRY_GLOBAL(__NV_ENABLE_PCIE_RELAXED_ORDERING_MODE, 0); NV_DEFINE_REG_ENTRY_GLOBAL(__NV_REGISTER_PCI_DRIVER, 1); NV_DEFINE_REG_ENTRY_GLOBAL(__NV_REGISTER_PLATFORM_DEVICE_DRIVER, 1); -NV_DEFINE_REG_ENTRY_GLOBAL(__NV_ENABLE_RESIZABLE_BAR, 0); +NV_DEFINE_REG_ENTRY_GLOBAL(__NV_ENABLE_RESIZABLE_BAR, 1); NV_DEFINE_REG_ENTRY_GLOBAL(__NV_ENABLE_DBG_BREAKPOINT, 0); NV_DEFINE_REG_ENTRY_GLOBAL(__NV_TEGRA_GPU_PG_MASK, 0); NV_DEFINE_REG_ENTRY_GLOBAL(__NV_ENABLE_NONBLOCKING_OPEN, 1); diff --git a/src/nvidia/arch/nvalloc/unix/src/osmemdesc.c b/src/nvidia/arch/nvalloc/unix/src/osmemdesc.c index 89ee179c4d..6b7cb20f84 100644 --- a/src/nvidia/arch/nvalloc/unix/src/osmemdesc.c +++ b/src/nvidia/arch/nvalloc/unix/src/osmemdesc.c @@ -1243,7 +1243,7 @@ osDestroyOsDescriptorPageArray // Read before nv_unregister_user_pages frees the nv_alloc_t. compoundOrder = nv_get_compound_order(pPrivate); - if (compoundOrder > 0) + if (compoundOrder > 0 || IS_DISCONTIG_AND_DYNGRAN_ENABLED(pMemDesc)) osPageCount = pMemDesc->PageCount; else osPageCount = NV_RM_PAGES_TO_OS_PAGES(pMemDesc->PageCount); diff --git a/src/nvidia/src/kernel/gpu/bif/kernel_bif.c b/src/nvidia/src/kernel/gpu/bif/kernel_bif.c index 322bbc9730..269b8fea70 100644 --- a/src/nvidia/src/kernel/gpu/bif/kernel_bif.c +++ b/src/nvidia/src/kernel/gpu/bif/kernel_bif.c @@ -1127,8 +1127,11 @@ _kbifInitRegistryOverrides { NvU32 data32; - // P2P Override: default to both reads+writes enabled so BAR1 P2P works out of the box - pKernelBif->p2pOverride = 0x11; + // Enable BAR1 P2P reads and writes without overriding platform atomic capabilities. + pKernelBif->p2pOverride = + DRF_DEF(_REG_STR, _CL_FORCE_P2P, _READ, _ENABLE) | + DRF_DEF(_REG_STR, _CL_FORCE_P2P, _WRITE, _ENABLE) | + DRF_DEF(_REG_STR, _CL_FORCE_P2P, _ATOMICS, _DEFAULT); if (osReadRegistryDword(pGpu, NV_REG_STR_CL_FORCE_P2P, &data32) == NV_OK) { pKernelBif->p2pOverride = data32; @@ -2069,4 +2072,3 @@ kbifWaitForConfigAccessAfterReset_IMPL return NV_ERR_GENERIC; } - diff --git a/src/nvidia/src/kernel/gpu/bus/arch/maxwell/kern_bus_gm200.c b/src/nvidia/src/kernel/gpu/bus/arch/maxwell/kern_bus_gm200.c index 1d56500465..6711be5e07 100644 --- a/src/nvidia/src/kernel/gpu/bus/arch/maxwell/kern_bus_gm200.c +++ b/src/nvidia/src/kernel/gpu/bus/arch/maxwell/kern_bus_gm200.c @@ -37,6 +37,59 @@ ((PCIE_P2P_WRITE_MAILBOX_SIZE << DRF_SIZE(NV_P2P_WMBOX_ADDR_ADDR)) - \ PCIE_P2P_WRITE_MAILBOX_SIZE) +static NV_STATUS +_kbusSetupMailboxes_GM200 +( + OBJGPU *pGpu0, + KernelBus *pKernelBus0, + OBJGPU *pGpu1, + KernelBus *pKernelBus1, + NvU32 local2Remote, + NvU32 remote2Local, + NvBool *pbLocalMailboxTeardownAttempted, + NvBool *pbRemoteMailboxTeardownAttempted +); + +static NV_STATUS +_kbusProgramPciePeerMask_GM200 +( + OBJGPU *pGpu, + NvU32 peerMask +) +{ + RM_API *pRmApi = GPU_GET_PHYSICAL_RMAPI(pGpu); + NV2080_CTRL_INTERNAL_HSHUB_PEER_CONN_CONFIG_PARAMS params = {0}; + + params.programPciePeerMask = peerMask; + + return pRmApi->Control(pRmApi, + pGpu->hInternalClient, + pGpu->hInternalSubdevice, + NV2080_CTRL_CMD_INTERNAL_HSHUB_PEER_CONN_CONFIG, + ¶ms, + sizeof(params)); +} + +static NV_STATUS +_kbusInvalidatePeerMask_GM200 +( + OBJGPU *pGpu, + NvU32 peerMask +) +{ + RM_API *pRmApi = GPU_GET_PHYSICAL_RMAPI(pGpu); + NV2080_CTRL_INTERNAL_HSHUB_PEER_CONN_CONFIG_PARAMS params = {0}; + + params.invalidatePeerMask = peerMask; + + return pRmApi->Control(pRmApi, + pGpu->hInternalClient, + pGpu->hInternalSubdevice, + NV2080_CTRL_CMD_INTERNAL_HSHUB_PEER_CONN_CONFIG, + ¶ms, + sizeof(params)); +} + /*! * @brief Setup the mailboxes of 2 GPUs so that the local GPU can access remote GPU. * @@ -59,6 +112,27 @@ kbusSetupMailboxes_GM200 NvU32 local2Remote, NvU32 remote2Local ) +{ + NV_STATUS status = _kbusSetupMailboxes_GM200(pGpu0, pKernelBus0, + pGpu1, pKernelBus1, + local2Remote, remote2Local, + NULL, NULL); + + NV_ASSERT_OK(status); +} + +static NV_STATUS +_kbusSetupMailboxes_GM200 +( + OBJGPU *pGpu0, + KernelBus *pKernelBus0, + OBJGPU *pGpu1, + KernelBus *pKernelBus1, + NvU32 local2Remote, + NvU32 remote2Local, + NvBool *pbLocalMailboxTeardownAttempted, + NvBool *pbRemoteMailboxTeardownAttempted +) { PMEMORY_DESCRIPTOR *ppMemDesc = NULL; RmPhysAddr localP2PDomainRemoteAddr; @@ -72,35 +146,65 @@ kbusSetupMailboxes_GM200 NV2080_CTRL_CMD_INTERNAL_BUS_SETUP_P2P_MAILBOX_LOCAL_PARAMS params0 = {0}; NV2080_CTRL_CMD_INTERNAL_BUS_SETUP_P2P_MAILBOX_REMOTE_PARAMS params1 = {0}; NV_STATUS status; + NvBool bRemoteWMBoxMapped = NV_FALSE; + NvBool bLocalP2PDomainMapped = NV_FALSE; + NvBool bRemoteP2PDomainMapped = NV_FALSE; + NvBool bLocalMailboxControl = NV_FALSE; + NvBool bRemoteMailboxControl = NV_FALSE; + NvBool bMailboxTagWritten = NV_FALSE; + + if (pbLocalMailboxTeardownAttempted != NULL) + { + *pbLocalMailboxTeardownAttempted = NV_FALSE; + } + if (pbRemoteMailboxTeardownAttempted != NULL) + { + *pbRemoteMailboxTeardownAttempted = NV_FALSE; + } - NV_ASSERT_OR_RETURN_VOID(local2Remote < P2P_MAX_NUM_PEERS); - NV_ASSERT_OR_RETURN_VOID(remote2Local < P2P_MAX_NUM_PEERS); + NV_ASSERT_OR_RETURN(local2Remote < P2P_MAX_NUM_PEERS, NV_ERR_INVALID_ARGUMENT); + NV_ASSERT_OR_RETURN(remote2Local < P2P_MAX_NUM_PEERS, NV_ERR_INVALID_ARGUMENT); // Ensure we have the correct bidirectional peer mapping - NV_ASSERT_OR_RETURN_VOID(pKernelBus1->p2pPcie.busPeer[remote2Local].remotePeerId == - local2Remote); - NV_ASSERT_OR_RETURN_VOID(pKernelBus0->p2pPcie.busPeer[local2Remote].remotePeerId == - remote2Local); + NV_ASSERT_OR_RETURN(pKernelBus1->p2pPcie.busPeer[remote2Local].remotePeerId == + local2Remote, NV_ERR_INVALID_STATE); + NV_ASSERT_OR_RETURN(pKernelBus0->p2pPcie.busPeer[local2Remote].remotePeerId == + remote2Local, NV_ERR_INVALID_STATE); ppMemDesc = &pKernelBus0->p2pPcie.busPeer[local2Remote].pRemoteWMBoxMemDesc; remoteWMBoxLocalAddr = kbusSetupMailboxAccess_HAL(pGpu1, pKernelBus1, pGpu0, remote2Local, ppMemDesc); - NV_ASSERT_OR_RETURN_VOID(remoteWMBoxLocalAddr != ~0ULL); + if (remoteWMBoxLocalAddr == ~0ULL) + { + status = NV_ERR_INVALID_ADDRESS; + goto kbusSetupMailboxes_cleanup; + } + bRemoteWMBoxMapped = NV_TRUE; ppMemDesc = &pKernelBus1->p2pPcie.busPeer[remote2Local].pRemoteP2PDomMemDesc; localP2PDomainRemoteAddr = kbusSetupP2PDomainAccess_HAL(pGpu0, pKernelBus0, pGpu1, ppMemDesc); - NV_ASSERT_OR_RETURN_VOID(localP2PDomainRemoteAddr != ~0ULL); + if (localP2PDomainRemoteAddr == ~0ULL) + { + status = NV_ERR_INVALID_ADDRESS; + goto kbusSetupMailboxes_cleanup; + } + bLocalP2PDomainMapped = NV_TRUE; ppMemDesc = &pKernelBus0->p2pPcie.busPeer[local2Remote].pRemoteP2PDomMemDesc; remoteP2PDomainLocalAddr = kbusSetupP2PDomainAccess_HAL(pGpu1, pKernelBus1, pGpu0, ppMemDesc); - NV_ASSERT_OR_RETURN_VOID(remoteP2PDomainLocalAddr != ~0ULL); + if (remoteP2PDomainLocalAddr == ~0ULL) + { + status = NV_ERR_INVALID_ADDRESS; + goto kbusSetupMailboxes_cleanup; + } + bRemoteP2PDomainMapped = NV_TRUE; // Setup the local GPU to access remote GPU's FB. @@ -110,7 +214,11 @@ kbusSetupMailboxes_GM200 PCIE_P2P_WRITE_MAILBOX_SIZE * remote2Local; // Write mailbox data window needs to be 64KB aligned. - NV_ASSERT((remoteWMBoxAddrU64 & 0xFFFF) == 0); + if ((remoteWMBoxAddrU64 & 0xFFFF) != 0) + { + status = NV_ERR_INVALID_ADDRESS; + goto kbusSetupMailboxes_cleanup; + } // Setup PCIE P2P Mailbox on local GPU params0.local2Remote = local2Remote; @@ -127,7 +235,11 @@ kbusSetupMailboxes_GM200 NV2080_CTRL_CMD_INTERNAL_BUS_SETUP_P2P_MAILBOX_LOCAL, ¶ms0, sizeof(NV2080_CTRL_CMD_INTERNAL_BUS_SETUP_P2P_MAILBOX_LOCAL_PARAMS)); - NV_ASSERT(status == NV_OK); + if (status != NV_OK) + { + goto kbusSetupMailboxes_cleanup; + } + bLocalMailboxControl = NV_TRUE; // Setup PCIE P2P Mailbox on remote GPU params1.local2Remote = local2Remote; @@ -143,9 +255,62 @@ kbusSetupMailboxes_GM200 NV2080_CTRL_CMD_INTERNAL_BUS_SETUP_P2P_MAILBOX_REMOTE, ¶ms1, sizeof(NV2080_CTRL_CMD_INTERNAL_BUS_SETUP_P2P_MAILBOX_REMOTE_PARAMS)); - NV_ASSERT(status == NV_OK); + if (status != NV_OK) + { + goto kbusSetupMailboxes_cleanup; + } + bRemoteMailboxControl = NV_TRUE; kbusWriteP2PWmbTag_HAL(pGpu1, pKernelBus1, remote2Local, params0.p2pWmbTag); + bMailboxTagWritten = NV_TRUE; + + return NV_OK; + +kbusSetupMailboxes_cleanup: + NV_PRINTF(LEVEL_ERROR, + "P2P_MAILBOX_SETUP_FAIL localGpu=%u remoteGpu=%u localPeer=%u remotePeer=%u " + "status=0x%x wmboxMapped=%u localDomainMapped=%u remoteDomainMapped=%u " + "localCtrl=%u remoteCtrl=%u tagWritten=%u tag=0x%llx\n", + gpuGetInstance(pGpu0), + gpuGetInstance(pGpu1), + local2Remote, + remote2Local, + status, + bRemoteWMBoxMapped, + bLocalP2PDomainMapped, + bRemoteP2PDomainMapped, + bLocalMailboxControl, + bRemoteMailboxControl, + bMailboxTagWritten, + (NvU64)params0.p2pWmbTag); + + if (bLocalMailboxControl) + { + kbusDestroyMailbox(pGpu0, pKernelBus0, pGpu1, local2Remote); + if (pbLocalMailboxTeardownAttempted != NULL) + { + *pbLocalMailboxTeardownAttempted = NV_TRUE; + } + } + else if (bRemoteWMBoxMapped || bRemoteP2PDomainMapped) + { + kbusDestroyPeerAccess_HAL(pGpu0, pKernelBus0, local2Remote); + } + + if (bRemoteMailboxControl || bMailboxTagWritten) + { + kbusDestroyMailbox(pGpu1, pKernelBus1, pGpu0, remote2Local); + if (pbRemoteMailboxTeardownAttempted != NULL) + { + *pbRemoteMailboxTeardownAttempted = NV_TRUE; + } + } + else if (bLocalP2PDomainMapped) + { + kbusDestroyPeerAccess_HAL(pGpu1, pKernelBus1, remote2Local); + } + + return status; } void @@ -198,11 +363,28 @@ kbusSetupMailboxAccess_GM200 PMEMORY_DESCRIPTOR *ppWMBoxMemDesc ) { - return kbusSetupPeerBarAccess(pGpu0, pGpu1, - gpumgrGetGpuPhysFbAddr(pGpu0) + - pKernelBus0->p2pPcie.writeMailboxBar1Addr + - PCIE_P2P_WRITE_MAILBOX_SIZE * local2Remote, - PCIE_P2P_WRITE_MAILBOX_SIZE, ppWMBoxMemDesc); + RmPhysAddr fbBase = gpumgrGetGpuPhysFbAddr(pGpu0); + NvU64 mailboxOffset = pKernelBus0->p2pPcie.writeMailboxBar1Addr; + NvU64 peerOffset = PCIE_P2P_WRITE_MAILBOX_SIZE * local2Remote; + RmPhysAddr base; + + if (pKernelBus0->p2pPcie.writeMailboxBar1Addr == + PCIE_P2P_INVALID_WRITE_MAILBOX_ADDR) + { + NV_PRINTF(LEVEL_ERROR, + "PCIe mailbox P2P requested without an allocated mailbox area " + "ownerGpu=%u accessorGpu=%u peer=%u writeMailboxBar1Addr=0x%llx\n", + gpuGetInstance(pGpu0), + gpuGetInstance(pGpu1), + local2Remote, + pKernelBus0->p2pPcie.writeMailboxBar1Addr); + return ~0ULL; + } + + base = fbBase + mailboxOffset + peerOffset; + + return kbusSetupPeerBarAccess(pGpu0, pGpu1, base, + PCIE_P2P_WRITE_MAILBOX_SIZE, ppWMBoxMemDesc); } void @@ -353,9 +535,20 @@ kbusCreateP2PMappingForMailbox_GM200 NvU32 attributes ) { - RM_API *pRmApi; - NV2080_CTRL_INTERNAL_HSHUB_PEER_CONN_CONFIG_PARAMS params; NvU32 gpuInst0, gpuInst1; + NvBool bPeer0HshubProgrammed = NV_FALSE; + NvBool bPeer1HshubProgrammed = NV_FALSE; + NvBool bPeer0MailboxTeardownNeeded = NV_FALSE; + NvBool bPeer1MailboxTeardownNeeded = NV_FALSE; + NvBool bPeer0MailboxTeardownAttempted = NV_FALSE; + NvBool bPeer1MailboxTeardownAttempted = NV_FALSE; + NvU32 oldPeer0RemotePeerId; + NvU32 oldPeer1RemotePeerId; + NvU32 oldPeer0RefCount; + NvU32 oldPeer1RefCount; + NvU32 oldPeerMask0; + NvU32 oldPeerMask1; + NV_STATUS status; if (IS_VIRTUAL(pGpu0) || IS_VIRTUAL(pGpu1)) { @@ -397,25 +590,26 @@ kbusCreateP2PMappingForMailbox_GM200 NV_ASSERT(pKernelBus0->p2pPcie.busPeer[*peer0].remotePeerId == *peer1); NV_ASSERT(pKernelBus1->p2pPcie.busPeer[*peer1].remotePeerId == *peer0); - pRmApi = GPU_GET_PHYSICAL_RMAPI(pGpu0); - portMemSet(¶ms, 0, sizeof(params)); - params.programPciePeerMask = NVBIT32(*peer0); - NV_ASSERT_OK_OR_RETURN(pRmApi->Control(pRmApi, - pGpu0->hInternalClient, - pGpu0->hInternalSubdevice, - NV2080_CTRL_CMD_INTERNAL_HSHUB_PEER_CONN_CONFIG, - ¶ms, - sizeof(params))); - - pRmApi = GPU_GET_PHYSICAL_RMAPI(pGpu1); - portMemSet(¶ms, 0, sizeof(params)); - params.programPciePeerMask = NVBIT32(*peer1); - NV_ASSERT_OK_OR_RETURN(pRmApi->Control(pRmApi, - pGpu1->hInternalClient, - pGpu1->hInternalSubdevice, - NV2080_CTRL_CMD_INTERNAL_HSHUB_PEER_CONN_CONFIG, - ¶ms, - sizeof(params))); + status = _kbusProgramPciePeerMask_GM200(pGpu0, NVBIT32(*peer0)); + if (status != NV_OK) + { + pKernelBus0->p2pPcie.busPeer[*peer0].refCount--; + pKernelBus1->p2pPcie.busPeer[*peer1].refCount--; + return status; + } + + status = _kbusProgramPciePeerMask_GM200(pGpu1, NVBIT32(*peer1)); + if (status != NV_OK) + { + // + // The mapping pre-exists and its HSHUB peer masks are + // still needed by the existing references, so only drop + // the references taken above. + // + pKernelBus0->p2pPcie.busPeer[*peer0].refCount--; + pKernelBus1->p2pPcie.busPeer[*peer1].refCount--; + return status; + } return NV_OK; } @@ -449,25 +643,26 @@ kbusCreateP2PMappingForMailbox_GM200 NV_ASSERT(!pKernelBus0->p2pPcie.busPeer[*peer0].bReserved); NV_ASSERT(!pKernelBus1->p2pPcie.busPeer[*peer1].bReserved); - pRmApi = GPU_GET_PHYSICAL_RMAPI(pGpu0); - portMemSet(¶ms, 0, sizeof(params)); - params.programPciePeerMask = NVBIT32(*peer0); - NV_ASSERT_OK_OR_RETURN(pRmApi->Control(pRmApi, - pGpu0->hInternalClient, - pGpu0->hInternalSubdevice, - NV2080_CTRL_CMD_INTERNAL_HSHUB_PEER_CONN_CONFIG, - ¶ms, - sizeof(params))); - - pRmApi = GPU_GET_PHYSICAL_RMAPI(pGpu1); - portMemSet(¶ms, 0, sizeof(params)); - params.programPciePeerMask = NVBIT32(*peer1); - NV_ASSERT_OK_OR_RETURN(pRmApi->Control(pRmApi, - pGpu1->hInternalClient, - pGpu1->hInternalSubdevice, - NV2080_CTRL_CMD_INTERNAL_HSHUB_PEER_CONN_CONFIG, - ¶ms, - sizeof(params))); + status = _kbusProgramPciePeerMask_GM200(pGpu0, NVBIT32(*peer0)); + if (status != NV_OK) + { + pKernelBus0->p2pPcie.busPeer[*peer0].refCount--; + pKernelBus1->p2pPcie.busPeer[*peer1].refCount--; + return status; + } + + status = _kbusProgramPciePeerMask_GM200(pGpu1, NVBIT32(*peer1)); + if (status != NV_OK) + { + // + // The mapping pre-exists and its HSHUB peer masks are still + // needed by the existing references, so only drop the + // references taken above. + // + pKernelBus0->p2pPcie.busPeer[*peer0].refCount--; + pKernelBus1->p2pPcie.busPeer[*peer1].refCount--; + return status; + } return NV_OK; } @@ -521,6 +716,13 @@ kbusCreateP2PMappingForMailbox_GM200 } busCreateP2PMapping_setupMapping: + oldPeer0RemotePeerId = pKernelBus0->p2pPcie.busPeer[*peer0].remotePeerId; + oldPeer1RemotePeerId = pKernelBus1->p2pPcie.busPeer[*peer1].remotePeerId; + oldPeer0RefCount = pKernelBus0->p2pPcie.busPeer[*peer0].refCount; + oldPeer1RefCount = pKernelBus1->p2pPcie.busPeer[*peer1].refCount; + oldPeerMask0 = pKernelBus0->p2pPcie.peerNumberMask[gpuInst1]; + oldPeerMask1 = pKernelBus1->p2pPcie.peerNumberMask[gpuInst0]; + pKernelBus0->p2pPcie.busPeer[*peer0].remotePeerId = *peer1; pKernelBus0->p2pPcie.peerNumberMask[gpuInst1] |= NVBIT(*peer0); pKernelBus1->p2pPcie.busPeer[*peer1].remotePeerId = *peer0; @@ -538,34 +740,80 @@ kbusCreateP2PMappingForMailbox_GM200 pKernelBus0->p2pPcie.busPeer[*peer0].refCount++; pKernelBus1->p2pPcie.busPeer[*peer1].refCount++; - pRmApi = GPU_GET_PHYSICAL_RMAPI(pGpu0); - portMemSet(¶ms, 0, sizeof(params)); - params.programPciePeerMask = NVBIT32(*peer0); - NV_ASSERT_OK_OR_RETURN(pRmApi->Control(pRmApi, - pGpu0->hInternalClient, - pGpu0->hInternalSubdevice, - NV2080_CTRL_CMD_INTERNAL_HSHUB_PEER_CONN_CONFIG, - ¶ms, - sizeof(params))); - - pRmApi = GPU_GET_PHYSICAL_RMAPI(pGpu1); - portMemSet(¶ms, 0, sizeof(params)); - params.programPciePeerMask = NVBIT32(*peer1); - NV_ASSERT_OK_OR_RETURN(pRmApi->Control(pRmApi, - pGpu1->hInternalClient, - pGpu1->hInternalSubdevice, - NV2080_CTRL_CMD_INTERNAL_HSHUB_PEER_CONN_CONFIG, - ¶ms, - sizeof(params))); + status = _kbusProgramPciePeerMask_GM200(pGpu0, NVBIT32(*peer0)); + if (status != NV_OK) + { + goto busCreateP2PMapping_rollback; + } + bPeer0HshubProgrammed = NV_TRUE; + + status = _kbusProgramPciePeerMask_GM200(pGpu1, NVBIT32(*peer1)); + if (status != NV_OK) + { + goto busCreateP2PMapping_rollback; + } + bPeer1HshubProgrammed = NV_TRUE; + + status = _kbusSetupMailboxes_GM200(pGpu0, pKernelBus0, pGpu1, pKernelBus1, + *peer0, *peer1, + &bPeer0MailboxTeardownAttempted, + &bPeer1MailboxTeardownAttempted); + if (status != NV_OK) + { + goto busCreateP2PMapping_rollback; + } + bPeer0MailboxTeardownNeeded = NV_TRUE; + bPeer1MailboxTeardownNeeded = NV_TRUE; + + status = _kbusSetupMailboxes_GM200(pGpu1, pKernelBus1, pGpu0, pKernelBus0, + *peer1, *peer0, + &bPeer1MailboxTeardownAttempted, + &bPeer0MailboxTeardownAttempted); + if (status != NV_OK) + { + goto busCreateP2PMapping_rollback; + } NV_PRINTF(LEVEL_INFO, "added PCIe P2P mapping between GPU%u (peer %u) and GPU%u (peer %u)\n", gpuInst0, *peer0, gpuInst1, *peer1); - kbusSetupMailboxes_HAL(pGpu0, pKernelBus0, pGpu1, pKernelBus1, *peer0, *peer1); - kbusSetupMailboxes_HAL(pGpu1, pKernelBus1, pGpu0, pKernelBus0, *peer1, *peer0); - return NV_OK; + +busCreateP2PMapping_rollback: + if (bPeer0MailboxTeardownNeeded && !bPeer0MailboxTeardownAttempted) + { + kbusDestroyMailbox(pGpu0, pKernelBus0, pGpu1, *peer0); + bPeer0MailboxTeardownAttempted = NV_TRUE; + } + + if (bPeer1MailboxTeardownNeeded && !bPeer1MailboxTeardownAttempted) + { + kbusDestroyMailbox(pGpu1, pKernelBus1, pGpu0, *peer1); + bPeer1MailboxTeardownAttempted = NV_TRUE; + } + + if (bPeer0HshubProgrammed && !bPeer0MailboxTeardownAttempted) + { + NV_ASSERT_OK(_kbusInvalidatePeerMask_GM200(pGpu0, NVBIT32(*peer0))); + } + + if (bPeer1HshubProgrammed && !bPeer1MailboxTeardownAttempted) + { + NV_ASSERT_OK(_kbusInvalidatePeerMask_GM200(pGpu1, NVBIT32(*peer1))); + } + + pKernelBus0->p2pPcie.busPeer[*peer0].remotePeerId = oldPeer0RemotePeerId; + pKernelBus1->p2pPcie.busPeer[*peer1].remotePeerId = oldPeer1RemotePeerId; + pKernelBus0->p2pPcie.busPeer[*peer0].refCount = oldPeer0RefCount; + pKernelBus1->p2pPcie.busPeer[*peer1].refCount = oldPeer1RefCount; + pKernelBus0->p2pPcie.peerNumberMask[gpuInst1] = oldPeerMask0; + pKernelBus1->p2pPcie.peerNumberMask[gpuInst0] = oldPeerMask1; + + *peer0 = BUS_INVALID_PEER; + *peer1 = BUS_INVALID_PEER; + + return status; } /*! @@ -797,6 +1045,15 @@ kbusSetP2PMailboxBar1Area_GM200 if (!kbusIsP2pMailboxClientAllocated(pKernelBus)) { + if (pKernelBus->p2pPcie.writeMailboxBar1Addr == + PCIE_P2P_INVALID_WRITE_MAILBOX_ADDR) + { + NV_PRINTF(LEVEL_ERROR, + "P2P mailbox area expected from RM but no valid address is installed gpu=%u\n", + gpuGetInstance(pGpu)); + return NV_ERR_INVALID_STATE; + } + // P2P mailbox area already allocated by RM. Nothing to do. return NV_OK; } diff --git a/src/nvidia/src/kernel/gpu/bus/arch/turing/bar1_p2p_policy.h b/src/nvidia/src/kernel/gpu/bus/arch/turing/bar1_p2p_policy.h new file mode 100644 index 0000000000..4008b21262 --- /dev/null +++ b/src/nvidia/src/kernel/gpu/bus/arch/turing/bar1_p2p_policy.h @@ -0,0 +1,36 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 Duc P. Tran + * SPDX-License-Identifier: MIT + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER + * DEALINGS IN THE SOFTWARE. + */ + +#ifndef KERN_BUS_BAR1_P2P_POLICY_H +#define KERN_BUS_BAR1_P2P_POLICY_H + +/* + * Display-aware placement is additive for default-enabled devices with a + * non-empty aligned client FB range and a non-empty aligned static BAR1 + * window. Partial windows are safe because each external mapping is checked + * against the selected static BAR1 DMA window before its addresses are used. + */ +#define KBUS_USE_DISPLAY_AWARE_STATIC_BAR1(defaultEnabled, clientFbSize, maxStaticMapSize) \ + ((defaultEnabled) && ((clientFbSize) != 0) && ((maxStaticMapSize) != 0)) + +#endif // KERN_BUS_BAR1_P2P_POLICY_H diff --git a/src/nvidia/src/kernel/gpu/bus/arch/turing/kern_bus_tu102.c b/src/nvidia/src/kernel/gpu/bus/arch/turing/kern_bus_tu102.c index f06fb0953f..74e28c0c50 100644 --- a/src/nvidia/src/kernel/gpu/bus/arch/turing/kern_bus_tu102.c +++ b/src/nvidia/src/kernel/gpu/bus/arch/turing/kern_bus_tu102.c @@ -31,6 +31,7 @@ #include "gpu/mem_mgr/virt_mem_allocator.h" #include "nvrm_registry.h" #include "kernel/virtualization/hypervisor/hypervisor.h" +#include "bar1_p2p_policy.h" #include "published/turing/tu102/dev_bus.h" #include "published/turing/tu102/dev_vm.h" @@ -385,8 +386,30 @@ kbusIsStaticBar1Supported_TU102 // NvU64 fbSize = pMemoryManager->Ram.fbAddrSpaceSizeMb << 20; NvU64 fbSizeAligned = RM_ALIGN_UP(fbSize, RM_PAGE_SIZE_2M); + NvU64 clientFbSize = memmgrGetClientFbAddrSpaceSize(pGpu, pMemoryManager); + NvU64 clientFbSizeAligned = RM_ALIGN_DOWN(clientFbSize, RM_PAGE_SIZE_2M); NvU64 bar1VASize = pKernelBus->bar1[gfid].mappableLength; NvU64 bar1VASizeAligned = RM_ALIGN_DOWN(bar1VASize, RM_PAGE_SIZE_2M); + NvU64 staticBar1Offset = NV_ALIGN_UP(consoleSize + mailboxSize, RM_PAGE_SIZE_512M); + NvBool bBar1P2PDefault = + pKernelBus->getProperty(pKernelBus, PDB_PROP_KBUS_SUPPORT_BAR1_P2P_BY_DEFAULT); + NvU64 maxStaticMapSize = + (bar1VASizeAligned > staticBar1Offset) ? + RM_ALIGN_DOWN(bar1VASizeAligned - staticBar1Offset, RM_PAGE_SIZE_2M) : 0; + NvBool bUseDisplayAwareStaticBar1 = + KBUS_USE_DISPLAY_AWARE_STATIC_BAR1(bBar1P2PDefault, + clientFbSizeAligned, + maxStaticMapSize); + // + // Default-enabled GPUs may place a complete or partial static mapping + // after fixed console/mailbox mappings whenever runtime geometry leaves a + // non-empty aligned window. External mappings are checked against the + // resulting DMA window, so spanning and outside allocations fail safely. + // + NvU64 autoStaticMapSize = bUseDisplayAwareStaticBar1 ? + ((clientFbSizeAligned < maxStaticMapSize) ? + clientFbSizeAligned : maxStaticMapSize) : + fbSizeAligned; if (gfid != 0) { @@ -427,14 +450,13 @@ kbusIsStaticBar1Supported_TU102 // really wants to enable static BAR1 regardless of the auto checks // NvU64 bar1MapSize = - RM_ALIGN_DOWN(memmgrGetClientFbAddrSpaceSize(pGpu, pMemoryManager), - RM_PAGE_SIZE_2M); + clientFbSizeAligned; - if (bar1VASizeAligned < bar1MapSize) + if (bar1VASizeAligned < (staticBar1Offset + bar1MapSize)) { - NV_PRINTF(LEVEL_ERROR, "BAR1 size %lld is not large enough to map FB size" - "%lld to force static BAR1\n", - bar1VASizeAligned, bar1MapSize); + NV_PRINTF(LEVEL_ERROR, "BAR1 size %" NvU64_fmtu " is not large enough to map FB size " + "%" NvU64_fmtu " at offset %" NvU64_fmtu " to force static BAR1\n", + bar1VASizeAligned, bar1MapSize, staticBar1Offset); DBG_BREAKPOINT(); return NV_ERR_INVALID_REGISTRY_KEY; @@ -469,19 +491,17 @@ kbusIsStaticBar1Supported_TU102 // NvU32 userdSize = 0; NvU32 numChannels = kfifoGetMaxChannelsInSystem(pGpu, pKernelFifo); - NvU64 requiredAutoBar1Size = fbSizeAligned; + NvU64 requiredAutoBar1Size = autoStaticMapSize; NvU64 mmioPrivSize = 16 * RM_PAGE_SIZE; NvU64 doorbellSize = 16 * RM_PAGE_SIZE; + NvU64 alignmentPadding = staticBar1Offset - (consoleSize + mailboxSize); + NvU64 dynamicBar1Size; kfifoGetUserdSizeAlign_HAL(pKernelFifo, &userdSize, NULL); userdSize *= numChannels; - requiredAutoBar1Size += userdSize; - requiredAutoBar1Size += mmioPrivSize; - requiredAutoBar1Size += doorbellSize; - requiredAutoBar1Size += consoleSize; - requiredAutoBar1Size += mailboxSize; + dynamicBar1Size = userdSize + mmioPrivSize + doorbellSize; // // Console mappings are already mapped from the bottom of the BAR1 VASpace, @@ -493,10 +513,30 @@ kbusIsStaticBar1Supported_TU102 // if ((consoleSize != 0) || (mailboxSize != 0)) { - requiredAutoBar1Size += RM_PAGE_SIZE_512M - ((consoleSize + mailboxSize) % RM_PAGE_SIZE_512M); + if (bUseDisplayAwareStaticBar1) + { + requiredAutoBar1Size += staticBar1Offset; + + if (dynamicBar1Size > alignmentPadding) + { + requiredAutoBar1Size += dynamicBar1Size - alignmentPadding; + } + } + else + { + requiredAutoBar1Size += dynamicBar1Size; + requiredAutoBar1Size += consoleSize; + requiredAutoBar1Size += mailboxSize; + requiredAutoBar1Size += alignmentPadding; + } + } + else + { + requiredAutoBar1Size += dynamicBar1Size; } - if (bar1VASizeAligned >= requiredAutoBar1Size) + if ((autoStaticMapSize != 0) && + (bar1VASizeAligned >= requiredAutoBar1Size)) { NV_PRINTF(LEVEL_INFO, "Enabling static BAR1 automatically!\n"); return NV_OK; @@ -535,8 +575,12 @@ kbusEnableStaticBar1Mapping_TU102 MEMORY_DESCRIPTOR *pDmaMemDesc = NULL; NV_STATUS status = NV_OK; NvU64 bar1MapSize; + NvU64 clientFbSizeAligned; + NvU64 bar1VASizeAligned; NvU64 bar1BusAddr; NvU32 mapFlags = BUS_MAP_FB_FLAGS_MAP_UNICAST | BUS_MAP_FB_FLAGS_MAP_OFFSET_FIXED; + NvBool bBar1P2PDefault = + pKernelBus->getProperty(pKernelBus, PDB_PROP_KBUS_SUPPORT_BAR1_P2P_BY_DEFAULT); // // But use memmgrGetClientFbAddrSpaceSize @@ -548,8 +592,28 @@ kbusEnableStaticBar1Mapping_TU102 // The last client FB addresses not aligned to 2MB will // not be mappable to a 2MB mapping. // - bar1MapSize = RM_ALIGN_DOWN(memmgrGetClientFbAddrSpaceSize(pGpu, pMemoryManager), - RM_PAGE_SIZE_2M); + clientFbSizeAligned = RM_ALIGN_DOWN(memmgrGetClientFbAddrSpaceSize(pGpu, pMemoryManager), + RM_PAGE_SIZE_2M); + bar1VASizeAligned = RM_ALIGN_DOWN(pKernelBus->bar1[gfid].mappableLength, + RM_PAGE_SIZE_2M); + bar1MapSize = clientFbSizeAligned; + + { + NvU64 maxStaticMapSize = + (bar1Offset < bar1VASizeAligned) ? + RM_ALIGN_DOWN(bar1VASizeAligned - bar1Offset, RM_PAGE_SIZE_2M) : 0; + NvBool bUseDisplayAwareStaticBar1 = + KBUS_USE_DISPLAY_AWARE_STATIC_BAR1(bBar1P2PDefault, + clientFbSizeAligned, + maxStaticMapSize); + + if (bUseDisplayAwareStaticBar1 && (bar1MapSize > maxStaticMapSize)) + { + bar1MapSize = maxStaticMapSize; + } + } + + NV_ASSERT_OR_RETURN(bar1MapSize != 0, NV_ERR_NOT_SUPPORTED); // // The static mapping is not backed by an allocated physical FB. @@ -574,7 +638,7 @@ kbusEnableStaticBar1Mapping_TU102 // Deploy the static mapping. The RUSD statistics will read incorrectly // until the subsequent call to kbusUpdateRusdStatistics at the end of // kbusStatePostLoad_GM107 with bStaticBar1Enabled set - // + // NV_ASSERT_OK_OR_GOTO(status, kbusMapFbApertureSingle(pGpu, pKernelBus, pMemDesc, 0, &bar1Offset, &bar1MapSize, @@ -1043,7 +1107,6 @@ kbusGetStaticFbAperture_TU102 NvBool bDiscontigAllowed = !!(busMapFlags & BUS_MAP_FB_FLAGS_ALLOW_DISCONTIG); NvBool bInStaticRegion = NV_FALSE; NvBool bInDynamicRegion = NV_FALSE; - NvBool bInLastPage = NV_TRUE; NV_CHECK_OR_RETURN(LEVEL_SILENT, kbusIsStaticBar1Enabled(pGpu, pKernelBus), NV_ERR_NOT_SUPPORTED); @@ -1076,7 +1139,6 @@ kbusGetStaticFbAperture_TU102 if (curLimit > staticBar1Size) { bInDynamicRegion = NV_TRUE; - bInLastPage = bInLastPage && ((curLimit - staticBar1Size) < RM_PAGE_SIZE_2M); } else { @@ -1090,25 +1152,14 @@ kbusGetStaticFbAperture_TU102 if (bInDynamicRegion && bInStaticRegion) { // - // With rounding down the static region to 2MB, - // we can allocate the last non-2MB aligned region - // but not have a mapping for it + // The static region may not cover all of client FB: it is rounded + // down to 2MB and may be clipped to the BAR1 VA left after the + // console/mailbox reservation. The static BAR1 path cannot represent + // a range spanning that boundary; current CUDA P2P callers receive a + // predictable API rejection rather than a transparent dynamic-mapping + // fallback. // - if (bInLastPage) - { - return NV_ERR_NOT_SUPPORTED; - } - - NV_PRINTF(LEVEL_ERROR, "MemDesc spans both static and dynamic region," - "which is unsupported.\n"); - NV_PRINTF(LEVEL_ERROR, "static Bar1 map [0, 0x%llx]\n", - pKernelBus->bar1[gfid].staticBar1.size); - NV_PRINTF(LEVEL_ERROR, "Requested map range 0x%llx to 0x%llx, mapGranularity 0x%llx\n", - mapRange.start, mrangeLimit(mapRange) - 1llu, mapRange.size); - - memdescPrintMemdesc(pMemDesc, NV_TRUE, MAKE_NV_PRINTF_STR("Dumping memdesc:")); - - return NV_ERR_INVALID_ARGUMENT; + return NV_ERR_NOT_SUPPORTED; } if (bInDynamicRegion) diff --git a/src/nvidia/src/kernel/mem_mgr/io_vaspace.c b/src/nvidia/src/kernel/mem_mgr/io_vaspace.c index eaa8cecab1..d3256379c8 100644 --- a/src/nvidia/src/kernel/mem_mgr/io_vaspace.c +++ b/src/nvidia/src/kernel/mem_mgr/io_vaspace.c @@ -59,15 +59,15 @@ iovaspaceConstruct__IMPL void iovaspaceDestruct_IMPL(OBJIOVASPACE *pIOVAS) { - // OBJVASPACE *pVAS = staticCast(pIOVAS, OBJVASPACE); - - // TODO: might keep p2p mappings... - // if (pIOVAS->mappingCount != 0) - // { - // NV_PRINTF(LEVEL_ERROR, "%lld left-over mappings in IOVAS 0x%x\n", - // pIOVAS->mappingCount, pVAS->vaspaceId); - // DBG_BREAKPOINT(); - // } + OBJVASPACE *pVAS = staticCast(pIOVAS, OBJVASPACE); + + if (pIOVAS->mappingCount != 0) + { + NV_PRINTF(LEVEL_WARNING, + "%" NvU64_fmtu " left-over mappings in IOVAS 0x%x\n", + pIOVAS->mappingCount, + pVAS->vaspaceId); + } } NV_STATUS @@ -600,12 +600,9 @@ OBJIOVASPACE *iovaspaceFromMapping(PIOVAMAPPING pIovaMapping) OBJIOVASPACE *pIOVAS = iovaspaceFromId(pIovaMapping->iovaspaceId); // - // The IOVASPACE has to be there as the mapping is referencing it. If it's - // not, the mapping has been left dangling outlasting the IOVAS it was - // under. + // A missing IOVAS means the mapping was left dangling and outlived the + // address space it belonged to. The destroy path reports that condition. // - // NV_ASSERT(pIOVAS != NULL); - return pIOVAS; } @@ -613,7 +610,14 @@ void iovaMappingDestroy(PIOVAMAPPING pIovaMapping) { OBJIOVASPACE *pIOVAS = iovaspaceFromMapping(pIovaMapping); - if (pIOVAS == NULL) return; + if (pIOVAS == NULL) + { + NV_PRINTF(LEVEL_WARNING, + "IOVA mapping outlived IOVAS 0x%x\n", + pIovaMapping->iovaspaceId); + return; + } + iovaspaceDestroyMapping(pIOVAS, pIovaMapping); } diff --git a/src/nvidia/src/kernel/rmapi/nv_gpu_ops.c b/src/nvidia/src/kernel/rmapi/nv_gpu_ops.c index 3de77d045a..1db38cbbfd 100644 --- a/src/nvidia/src/kernel/rmapi/nv_gpu_ops.c +++ b/src/nvidia/src/kernel/rmapi/nv_gpu_ops.c @@ -3934,22 +3934,42 @@ nvGpuOpsMemGetPageSize * * @param[in] pAddresses : Array of physical addresses to be encoded. * @param[in] dmaBaseAddress : IOVA base address. + * @param[in] dmaSize : IOVA window size. + * @param[in] pageSize : Size covered by each physical address. * @param[in] count : Count of physical addresses. */ -static void +static NV_STATUS _nvGpuOpsEncodeBar1P2PAddrs ( NvU64 *pAddresses, NvU64 dmaBaseAddress, + NvU64 dmaSize, + NvU64 pageSize, NvU64 count ) { - NvU32 i; + NvU64 i; for (i = 0; i < count; i++) { - pAddresses[i] = dmaBaseAddress + pAddresses[i]; + NvU64 offset = pAddresses[i]; + NvU64 encodedAddress; + + if ((offset >= dmaSize) || + (pageSize > (dmaSize - offset)) || + !portSafeAddU64(dmaBaseAddress, offset, &encodedAddress)) + { + NV_PRINTF(LEVEL_ERROR, + "BAR1 P2P address range exceeds DMA window: " + "offset=0x%llx pageSize=0x%llx dmaBase=0x%llx dmaSize=0x%llx\n", + offset, pageSize, dmaBaseAddress, dmaSize); + return NV_ERR_INVALID_ADDRESS; + } + + pAddresses[i] = encodedAddress; } + + return NV_OK; } static @@ -3967,8 +3987,7 @@ nvGpuOpsBuildExternalAllocPtes NvBool isPeerSupported, NvBool isBar1P2PSupported, NvU32 peerId, - gpuExternalMappingInfo *pGpuExternalMappingInfo, - RmPhysAddr bar1BusAddr + gpuExternalMappingInfo *pGpuExternalMappingInfo ) { NV_STATUS status = NV_OK; @@ -4125,14 +4144,7 @@ nvGpuOpsBuildExternalAllocPtes NvU32 ptePcfHw = 0; nvFieldSetBool(&pPteFmt->fldValid, NV_TRUE, pte.v8); - if ((aperture == GMMU_APERTURE_PEER) && isBar1P2PSupported) - { - gmmuFieldSetAperture(&pPteFmt->fldAperture, GMMU_APERTURE_SYS_COH, pte.v8); - } - else - { - gmmuFieldSetAperture(&pPteFmt->fldAperture, aperture, pte.v8); - } + gmmuFieldSetAperture(&pPteFmt->fldAperture, aperture, pte.v8); nvFieldSet32(&pPteFmt->fldKind, kind, pte.v8); ptePcfSw |= vol ? (1 << SW_MMU_PCF_UNCACHED_IDX) : 0; @@ -4176,14 +4188,7 @@ nvGpuOpsBuildExternalAllocPtes if (nvFieldIsValid32(&pPteFmt->fldAtomicDisable.desc)) nvFieldSetBool(&pPteFmt->fldAtomicDisable, !atomic, pte.v8); - if ((aperture == GMMU_APERTURE_PEER) && isBar1P2PSupported) - { - gmmuFieldSetAperture(&pPteFmt->fldAperture, GMMU_APERTURE_SYS_NONCOH, pte.v8); - } - else - { - gmmuFieldSetAperture(&pPteFmt->fldAperture, aperture, pte.v8); - } + gmmuFieldSetAperture(&pPteFmt->fldAperture, aperture, pte.v8); if (!isCompressedKind) { @@ -4194,11 +4199,6 @@ nvGpuOpsBuildExternalAllocPtes } } - if ((aperture == GMMU_APERTURE_PEER) && isBar1P2PSupported) - { - fabricBaseAddress = bar1BusAddr; - } - if ((aperture == GMMU_APERTURE_PEER) && !isBar1P2PSupported) { nvFieldSet32(&pPteFmt->fldPeerIndex, peerId, pte.v8); @@ -4309,7 +4309,13 @@ nvGpuOpsBuildExternalAllocPtes status = NV_ERR_INVALID_STATE; goto done; } - _nvGpuOpsEncodeBar1P2PAddrs(physicalAddresses, dmaBaseAddress, pteCount); + NV_CHECK_OK_OR_GOTO(status, LEVEL_ERROR, + _nvGpuOpsEncodeBar1P2PAddrs(physicalAddresses, + dmaBaseAddress, + dmaSize, + mappingPageSize, + pteCount), + done); } else { @@ -4458,8 +4464,7 @@ nvGpuOpsBuildExternalAllocPhysAddrs NvBool isPeerSupported, NvBool isBar1P2PSupported, NvU32 peerId, - UvmGpuExternalPhysAddrInfo *pGpuExternalPhysAddrInfo, - RmPhysAddr bar1BusAddr + UvmGpuExternalPhysAddrInfo *pGpuExternalPhysAddrInfo ) { NV_STATUS status = NV_OK; @@ -4537,11 +4542,6 @@ nvGpuOpsBuildExternalAllocPhysAddrs return NV_ERR_BUFFER_TOO_SMALL; - if ((aperture == GMMU_APERTURE_PEER) && isBar1P2PSupported) - { - fabricBaseAddress = bar1BusAddr; - } - if ((aperture == GMMU_APERTURE_PEER) && !isBar1P2PSupported) { // @@ -4649,7 +4649,13 @@ nvGpuOpsBuildExternalAllocPhysAddrs status = NV_ERR_INVALID_STATE; goto done; } - _nvGpuOpsEncodeBar1P2PAddrs(physicalAddresses, dmaBaseAddress, physAddrCount); + NV_CHECK_OK_OR_GOTO(status, LEVEL_ERROR, + _nvGpuOpsEncodeBar1P2PAddrs(physicalAddresses, + dmaBaseAddress, + dmaSize, + mappingPageSize, + physAddrCount), + done); } else { @@ -4684,7 +4690,6 @@ NV_STATUS nvGpuOpsGetExternalAllocPtesOrPhysAddrs(struct gpuAddressSpace *vaSpac Memory *pMemory = NULL; PMEMORY_DESCRIPTOR pMemDesc = NULL; OBJGPU *pMappingGpu = NULL; - RmPhysAddr bar1BusAddr = 0; NvU32 peerId = 0; NvBool isSliSupported = NV_FALSE; NvBool isPeerSupported = NV_FALSE; @@ -4824,8 +4829,6 @@ NV_STATUS nvGpuOpsGetExternalAllocPtesOrPhysAddrs(struct gpuAddressSpace *vaSpac &peerId); if (status != NV_OK) goto freeGpaMemdesc; - - bar1BusAddr = gpumgrGetGpuPhysFbAddr(pAdjustedMemDesc->pGpu); } // @@ -4914,15 +4917,14 @@ NV_STATUS nvGpuOpsGetExternalAllocPtesOrPhysAddrs(struct gpuAddressSpace *vaSpac isPeerSupported, isBar1P2PSupported, peerId, - pGpuExternalMappingInfo, - bar1BusAddr); + pGpuExternalMappingInfo); } if (pGpuExternalPhysAddrInfo != NULL) { status = nvGpuOpsBuildExternalAllocPhysAddrs(pVAS, vaSpace->device->session, pMappingGpu, pAdjustedMemDesc, pMemory, offset, size, isIndirectPeerSupported, isPeerSupported, - isBar1P2PSupported, peerId, pGpuExternalPhysAddrInfo, bar1BusAddr); + isBar1P2PSupported, peerId, pGpuExternalPhysAddrInfo); } freeGpaMemdesc: @@ -11076,7 +11078,7 @@ NV_STATUS nvGpuOpsGetChannelResourcePtes(struct gpuAddressSpace *vaSpace, status = nvGpuOpsBuildExternalAllocPtes(pVAS, vaSpace->device->session, pMappingGpu, pMemDesc, NULL, offset, size, NV_FALSE, NV_FALSE, - NV_FALSE, 0, pGpuExternalMappingInfo, 0); + NV_FALSE, 0, pGpuExternalMappingInfo); _nvGpuOpsLocksRelease(&acquiredLocks); threadStateFree(&threadState, THREAD_STATE_FLAGS_NONE); diff --git a/tests/.gitignore b/tests/.gitignore new file mode 100644 index 0000000000..95e69555a7 --- /dev/null +++ b/tests/.gitignore @@ -0,0 +1 @@ +/bar1_p2p_policy_test diff --git a/tests/Makefile b/tests/Makefile new file mode 100644 index 0000000000..1589c5525d --- /dev/null +++ b/tests/Makefile @@ -0,0 +1,18 @@ +CC ?= cc + +POLICY_TEST := bar1_p2p_policy_test +POLICY_HEADER := ../src/nvidia/src/kernel/gpu/bus/arch/turing/bar1_p2p_policy.h +POLICY_TEST_CFLAGS := -std=c11 -Wall -Wextra -Werror + +.PHONY: all check clean + +all: $(POLICY_TEST) + +$(POLICY_TEST): $(POLICY_TEST).c $(POLICY_HEADER) + $(CC) $(CPPFLAGS) $(CFLAGS) $(POLICY_TEST_CFLAGS) -o $@ $< + +check: $(POLICY_TEST) + ./$(POLICY_TEST) + +clean: + $(RM) $(POLICY_TEST) diff --git a/tests/README.md b/tests/README.md new file mode 100644 index 0000000000..d991be436d --- /dev/null +++ b/tests/README.md @@ -0,0 +1,13 @@ +# BAR1 policy tests + +Run the source-level BAR1 P2P policy regression test with: + +```sh +make -C tests check +``` + +The test verifies the runtime-coverage truth table. Display-aware placement is +available only when BAR1 P2P is enabled by the existing device property and +both the aligned client framebuffer and available static BAR1 window are +non-empty. Partial, exact, and larger-than-client coverage are accepted without +an implementation-specific exception. diff --git a/tests/bar1_p2p_policy_test.c b/tests/bar1_p2p_policy_test.c new file mode 100644 index 0000000000..fda9280762 --- /dev/null +++ b/tests/bar1_p2p_policy_test.c @@ -0,0 +1,26 @@ +/* SPDX-License-Identifier: MIT */ + +#include + +#include "../src/nvidia/src/kernel/gpu/bus/arch/turing/bar1_p2p_policy.h" + +int main(void) +{ + const unsigned long long clientFbSize = 16ULL << 30; + const unsigned long long partialStaticSize = 15ULL << 30; + + assert(!KBUS_USE_DISPLAY_AWARE_STATIC_BAR1(0, clientFbSize, + partialStaticSize)); + assert(!KBUS_USE_DISPLAY_AWARE_STATIC_BAR1(1, 0, + partialStaticSize)); + assert(!KBUS_USE_DISPLAY_AWARE_STATIC_BAR1(1, clientFbSize, 0)); + + assert(KBUS_USE_DISPLAY_AWARE_STATIC_BAR1(1, clientFbSize, + partialStaticSize)); + assert(KBUS_USE_DISPLAY_AWARE_STATIC_BAR1(1, clientFbSize, + clientFbSize)); + assert(KBUS_USE_DISPLAY_AWARE_STATIC_BAR1(1, clientFbSize, + clientFbSize + (1ULL << 30))); + + return 0; +} diff --git a/validation/gb206-bar1-boundary-2026-08-04.md b/validation/gb206-bar1-boundary-2026-08-04.md new file mode 100644 index 0000000000..ffbff3419a --- /dev/null +++ b/validation/gb206-bar1-boundary-2026-08-04.md @@ -0,0 +1,145 @@ +# GB206 BAR1 boundary validation — 2026-08-04 + +## Revisions + +- Diagnostic branch: `test/gb206-bar1-boundary-pressure` +- Diagnostic tag/commit: `test/gb206-bar1-boundary-d0676cfc` / `d0676cfc` +- Production branch: `production/runtime-bar1-coverage` +- Production module commit: `d682adc5` +- Driver/kernel: `610.43.03` / `7.0.0-28-generic` +- Secure Boot signer: `sugardaddy Secure Boot Module Signature key` + +The production branch does not contain the diagnostic logging, CUDA boundary +harness source, or generated harness binary. + +## Diagnostic module identity + +| Module | SHA-256 | srcversion | +|---|---|---| +| `nvidia.ko` | `04925f26b64b5095705cf98f2e4fefd405ec300220fe340e2c0f07220553d346` | `9020898A6D608A1C767CC72` | +| `nvidia-uvm.ko` | `58ec9925164d3dc39d5a6b54076792fe3b80548c2b0fb3ef93501d9fd01bf4f9` | `1BD7F0E70C0717835738BDB` | + +Both modules used vermagic `7.0.0-28-generic SMP preempt mod_unload modversions`. + +## Natural geometry and deterministic matrix + +The test used natural BAR1 geometry only. The invalidating 4 GiB override was not +used. + +- Full aligned coverage: `selectedStatic=0x3e1000000`, aligned client FB + `0x3e1000000`. +- Display/console partial coverage: `selectedStatic=0x3dfe00000`, raw client FB + `0x3e10a0000`, console reservation `0x260000`, static offset `0x20000000`. +- Inside tests passed in both GPU directions and both peer-enable orderings, + including peer kernels, `cudaMemcpyPeer`, and beginning/middle/end probes. +- Full-coverage pressure reached normal allocation exhaustion after 1,969 passing + tests with no unsafe classification or data error. +- The partial side produced the same 64 MiB spanning range in both orderings: + `minOffset=0x3dcc00000`, `maxEnd=0x3e0c00000`, + `dmaSize=0x3dfe00000`. Peer-before rejected during allocation; peer-after + rejected during peer enablement. Each had 491 prior passing tests. +- Prepared boundary mode reported an outside range + `0x3e0c00000..0x3e1000000`, rejected it with `localHealthy=1`, and passed the + three-iteration recovery. + +## Rejection/recovery cycles + +From `2026-08-04T14:53:04-07:00` through +`2026-08-04T15:02:57-07:00`, 100 independent prepared-boundary cycles completed. +Every cycle required: + +- one spanning candidate rejection; +- one outside rejection with `localHealthy=1`; +- `rejected=2 apiErrors=2 dataErrors=0`; and +- a three-iteration inside recovery. + +Both GPUs returned to zero MiB used at every ten-cycle checkpoint. The cycle +window contained no assertion, Xid, mailbox setup failure, IOMMU/AER fault, +stale state, invalid state, or cleanup warning. + +The individual logs and their checksum manifest were written under `/tmp` and +were cleared by the required reboot. The pass count, checkpoints, and journal +result were captured before reboot; this file records the durable summary. + +## Production policy and routing audit + +At the commit validated by this record, the production predicate was additive: +property-enabled GPUs used display-aware placement when runtime geometry +covered all aligned client FB, while GB206 retained the tested partial-window +exception. `make -C tests check` ran `tests/bar1_p2p_policy_test.c` to verify +that policy truth table. + +The policy was subsequently generalized to use only the existing BAR1 P2P +device property and runtime geometry. Any property-enabled GPU with a non-empty +aligned client FB range and non-empty aligned static BAR1 window may use the +same partial-window behavior. This record remains hardware evidence for GB206; +it does not claim that other partial-window implementations were tested. + +### Post-generalization GB206 regression + +On 2026-08-05, the generalized runtime policy was built as all five kernel +modules, signed with the enrolled Secure Boot key, installed on kernel +`7.0.0-29-generic`, and exercised on the same two RTX 5060 Ti GPUs. The bounded +boundary run completed with 492 inside passes and no data errors. On the +partial-coverage GPU, the established 64 MiB spanning candidate and a prepared +4 MiB outside candidate were rejected by the retained static-aperture bounds +checks. Local access remained healthy and the immediate three-iteration inside +recovery passed. The other GPU reached ordinary allocation exhaustion without +an unsafe mapping or data error. + +`simpleP2P` passed before and after the boundary run at 13.05 and 13.03 GB/s. +`p2pBandwidthLatencyTest` measured 14.09 GB/s in each unidirectional direction +and 27.79 GB/s bidirectionally. The post-load kernel log contained the expected +fail-closed boundary diagnostics and no Xid, assertion, IOMMU/MMU fault, AER +error, oops, panic, or hung-task report. This regression confirms unchanged +GB206 behavior; other partial-window GPU implementations remain hardware +validation follow-ups. + +Generated HAL dispatch, the global `pcieP2PType` default, registry precedence, +and the GH100 BAR1 routing source were unchanged. Their pre/post hashes matched. +The live BAR1 encoder bounds checks remain present. + +## Signed production modules + +Installed path: `/lib/modules/7.0.0-28-generic/updates/local/`. + +| Module | SHA-256 | srcversion | +|---|---|---| +| `nvidia.ko` | `9bf54a3665eea55e839b93feb44f4c92c994db88baf7e31331341ab1d712a626` | `9020898A6D608A1C767CC72` | +| `nvidia-modeset.ko` | `8e3217b97a4432cc9b84f5d1eb475d8f76f0fc7985565bd4e20a986f1a3a65ef` | `0BCB09E2E1D4422BB162693` | +| `nvidia-drm.ko` | `7402ab6be81127e636d5bd29b9920a9aaa204c6bc5991848781add2a7fa75b9f` | `65769FC23A53EFDFC4A2DB5` | +| `nvidia-uvm.ko` | `b5c2d10652954ff40c566b596ebbf2d56b094d6be722f5123de37bf29da3dfea` | `1BD7F0E70C0717835738BDB` | +| `nvidia-peermem.ko` | `61363a4d58f1f0fccf29b944b833e35502e0a584a64fd5202190b18858b4f9ba` | `05E8CF2F419E46C7D3D974E` | + +## Production validation + +- Full module build and source policy regression: passed. +- `nvidia-smi`: both RTX 5060 Ti GPUs healthy. +- P2P read/write capability: `OK` both directions. +- P2P atomics: `NS`, not `DR` (not disabled by the registry default). +- `simpleP2P`: passed before reload, after reload, after resume, and after reboot; + 13.09 GB/s. +- `p2pBandwidthLatencyTest`: 14.09 GB/s each unidirectional P2P direction and + 27.79–27.80 GB/s bidirectional. +- Both peer-enable orderings passed bidirectional inside correctness. +- The known partial-boundary sequence rejected in both orderings and recovered + immediately. +- Modeset/DRM stack load and full driver unload/reload: passed. +- Deep suspend via the enabled NVIDIA systemd suspend/resume hooks: entered at + 15:17:13 and exited at 15:17:39; post-resume P2P passed. +- Boot from the updated initramfs: candidate hashes/signer matched and post-boot + P2P passed. No GPU-specific boot journal fault was present. + +## Suspend configuration notes + +`NVreg_PreserveVideoMemoryAllocations=1` correctly requires the NVIDIA procfs +suspend hook; a raw `rtcwake -m mem` attempt was rejected, while the supported +systemd path succeeded. `NVreg_TemporaryFilePath=/var` selects the root filesystem +on `/dev/nvme6n1p5` for preservation files, but the journal I/O errors referenced +the separate `/dev/nvme6n1p3` partition. That partition is intentionally inaccessible +while OPAL-locked for BitLocker, so those messages are expected and are unrelated to +the NVIDIA validation. + +The restricted Codex mount namespace exposes `/` with a read-only VFS mount flag +while the ext4 filesystem reports `rw`; this does not indicate that the host root +filesystem was remounted read-only. No storage failure is inferred from this test.