Add details on IPC cache#1081
Conversation
ashesh2512
changed the title
|
@trey-ornl , @GeorgiadouAntigoni , @hagertnl - I'd like to talk about this at our next GPU MPI meeting (I'll forward the invite to @ashesh2512 ). The hope is that in the next MPI + ROCM release all the edge case memory leaks will be resolved and we can just strop back to the "here's how to enable the IPC signal cache and tune the IPC cache size" documentation. |
|
|
||
| Registering and unregistering GPU buffers for inter-process communication takes a relatively long time to do. The MPI library caches remote buffer attachments in order to reuse them, instead of detaching after every transfer. One limitation has been that when a process stops using a GPU buffer, there was no easy way to let the processes that are caching attachments know about this, and so they continued to cache the buffer. This consumes some of the limited GPU memory. A new feature since ROCM 6.4 enables the MPI library to alert the other processes through a signalling mechanism. If this environment variable is set, then this feature will be used to help reduce the amount of memory that the cache is consuming. This may help prevent out of memory situations. | ||
|
|
||
| ``HSA_ENABLE_IPC_MODE_LEGACY`` |
There was a problem hiding this comment.
I think you need to document both versions of this variable, including the Rocm version dependence.
|
|
||
| Cray MPICH uses IPC by default for intra-node inter-process MPI data movement operations that involve GPU buffers, i.e., enabling GPU-aware MPI (``MPICH_GPU_SUPPORT_ENABLED=1``) also sets ``MPICH_GPU_IPC_ENABLED=1``. Disabling IPC is known to noticeably impact intra-node MPI performance. | ||
|
|
||
| Prior to Cray MPICH 8.1.33, a memory leak was observed in edge cases involving repeated creation and freeing of GPU-buffers used in MPI communication. This issue was prevelant when GPU buffers continuously increased in size or occupied new memory regions (e.g., due to defragmentation). The root cause was traced to IPC cache handles not being properly released. AMD addressed this by introducing an API that allows remote processes to free IPC cache handles. For Cray MPICH 9.0.1 and prior, the updated IPC mechanism is enabled by setting ``GTL_ENABLE_HSA_IPC_SIGNAL_CACHE=1`` and ``HSA_DISABLE_IPC_MODE_LEGACY=0``. Starting with Cray MPICH 9.1.0, the latter variable was renamed to ``HSA_ENABLE_IPC_MODE_LEGACY=1`` for improved clarity. |
There was a problem hiding this comment.
Doesn't HSA_ENABLE/DISABLE_IPC_MODE_LEGACY depend on Rocm version, not Cray MPI version?
There was a problem hiding this comment.
Aha! Yes, it's RocR, not Cray MPICH.
https://github.com/ROCm/rocm-systems/blame/bece9cc90b397fc420ffe699ee812298a95e99db/projects/rocr-runtime/runtime/hsa-runtime/core/util/flag.h#L265
There was a problem hiding this comment.
I can find no evidence of HSA_DISABLE_IPC_MODE_LEGACY in any RocR release. It looks like HSA_ENABLE_IPC_MODE_LEGACY appeared in tag rocm-6.1.0, with no option, neither DISABLE or ENABLE, in the previous tag, rocm-6.0.2.
https://github.com/ROCm/ROCR-Runtime/blob/a4ab29c549573e8f02067cdd2a8e21f82d1a6958/src/core/util/flag.h#L228
Are you sure the DISABLE version exists anywhere on Frontier? Maybe it was only in a debug version HPE got from AMD, which they don't appear to have made public?
There was a problem hiding this comment.
You are right in that there is no HSA_DISABLE_IPC_MODE_LEGACY. I have unfortunately mistyped that throughout the PR when I was trying to distinguish between GTL_ENABLE_HSA_IPC_SIGNAL_CACHE and GTL_DISABLE_HSA_IPC_SIGNAL_CACHE. I need to clean that up.
There was a problem hiding this comment.
I'm just glad I never make mitsakes.
| Prior to Cray MPICH 8.1.33, a memory leak was observed in edge cases involving repeated creation and freeing of GPU-buffers used in MPI communication. This issue was prevelant when GPU buffers continuously increased in size or occupied new memory regions (e.g., due to defragmentation). The root cause was traced to IPC cache handles not being properly released. AMD addressed this by introducing an API that allows remote processes to free IPC cache handles. For Cray MPICH 9.0.1 and prior, the updated IPC mechanism is enabled by setting ``GTL_ENABLE_HSA_IPC_SIGNAL_CACHE=1`` and ``HSA_DISABLE_IPC_MODE_LEGACY=0``. Starting with Cray MPICH 9.1.0, the latter variable was renamed to ``HSA_ENABLE_IPC_MODE_LEGACY=1`` for improved clarity. | ||
|
|
||
| A separate issue remains where enabling IPC can still lead to memory leaks for message sizes less than or equal to 1 MB. A workaround is to set ``HSA_ENABLE_IPC_MODE_LEGACY=0`` and ``GTL_HSA_VSMSG_CUTOFF_SIZE=0``. When message sizes fall at or below ``GTL_HSA_VSMSG_CUTOFF_SIZE``, the GPU Transport Layer (GTL) uses a blocking memory copy driven by the CPU instead of an HSA memcpy. For context, disabling legacy IPC mode (``HSA_ENABLE_IPC_MODE_LEGACY=0``) switches to AMD's newer IPC implementation based on the Linux dma_buf mechanism. | ||
|
|
There was a problem hiding this comment.
Is it true that HSA_ENABLE_IPC_MODE_LEGACY=0 is only supported by Cray MPI iff GTL_HSA_VSMSG_CUTOFF_SIZE=0?
3 participants
This PR adds information on the combination of environment variables required to enable stable operation of IPC. Added best practices. Added a summary on known issues associated with IPC cache, and related workarounds. Also added a description on relevant environment variables.