Add AWS EFA networking guide for running DGXC benchmarks on P5en (H200) and P6-B200 (B200) - #39
Add AWS EFA networking guide for running DGXC benchmarks on P5en (H200) and P6-B200 (B200)#39paragao wants to merge 1 commit into
Conversation
…0) and P6-B200 (B200) AWS GPU instances use Elastic Fabric Adapter (EFA) instead of InfiniBand. The stock NeMo container has a critical NCCL plugin naming mismatch that causes silent fallback to TCP sockets (~23% performance degradation). This adds: - EFA-upgraded container Dockerfile (EFA installer 1.47.0, NCCL 2.29.3, GDRCopy v2.5.1, plugin symlink fix, /etc/environment dump for Pyxis env var propagation) - Four code patches: EFA env vars and NCCL tuning (executors.py), container image reference (launch.sh), cuDNN fused RMSNorm preservation (perf_plugins.py), os.fork() OOM fix for 70B models (llama3_llm_pretrain.py) - Comprehensive guide covering prerequisites, build, enroot import, installer setup, patching, verification, cluster configuration examples, and troubleshooting Validated end-to-end on H200 P5en cluster: 8B bf16 pretrain on 64 GPUs (8 nodes) completed successfully with EFA active.
sudostock
left a comment
sudostock
left a comment
There was a problem hiding this comment.
@paragao Thanks for the PR!
After reviewing, I think the biggest win from this is the instructions on updating EFA libraries in the container if needed for perf reasons or container age.
I'll start paring down the docs to focus on that in the next few days.
| ### The NCCL Plugin Naming Bug | ||
|
|
||
| NCCL 2.29+ uses `shinit_v2`, which sets `NCCL_NET_PLUGIN=aws-ofi`. This tells NCCL to search for a shared library named `libnccl-net-aws-ofi.so`. However, the AWS EFA installer packages the plugin as `libnccl-net-ofi.so` (without the `aws-` prefix). | ||
|
|
||
| **If NCCL cannot find the plugin by the expected name, it silently falls back to TCP sockets.** There is no error message -- only degraded performance. The fix is to create symlinks in the container image (see [Section 3](#3-build-the-efa-upgraded-container)). |
There was a problem hiding this comment.
Not a bug, the earlier container versions renamed the aws ofi library to 'aws-ofi' and adjusted paths and such accordingly.
Newer containers don't do this anymore.
There was a problem hiding this comment.
In general we don't allow model config changes for our benchmarks to still be considered apples to apples.
The changes here and in llama3_llm_pretrain will be tested internally to see if they broadly improve performance.
There was a problem hiding this comment.
These should be set in the environment var section of cluster_config.yaml file instead of editing source.
There was a problem hiding this comment.
Same comment as executors
| export FW_VERSION=25.11.01 | ||
|
|
||
| -export IMAGE=${RUN_CONF_IMAGE:-$LLMB_INSTALL/images/nvidia+nemo+$FW_VERSION.sqsh} | ||
| +export IMAGE=${RUN_CONF_IMAGE:-$LLMB_INSTALL/images/nvidia+nemo+${FW_VERSION}-efa-nccl29.sqsh} |
There was a problem hiding this comment.
Symlink the updated image to the expected name would be better and less error prone than updating all the launch.sh scripts.
|
@sudostock thank you for reviewing this. Please, find updated instructions on how to create a container with the proper software stack here: https://github.com/awslabs/awsome-distributed-ai/blob/main/micro-benchmarks/nccl-tests/nccl-tests.Dockerfile (this is maintained and kept updated by my team) Those are the recommendations we are using with customers and are providing the best performance possible:
CUDA can be 12.9+ depending on your GPU. Important to include that with the newer EFA Installer, the old environment variables for EFA (such as FI_USE_HUGE_PAGE=0 or FI_EFA_DEVICE_RDMA=1) shouldn't be set anymore. They are picked up by the OFI Tuner plugin, when required. |
2 participants
Summary
libnccl-net-aws-ofi.sobut EFA installer names itlibnccl-net-ofi.so) that causes silent fallback to TCP sockets with ~23% performance degradationFiles Added
alternative_recipes/aws/README.mdalternative_recipes/aws/container/nemo-efa-upgraded.Dockerfilealternative_recipes/aws/patches/executors.py.patchalternative_recipes/aws/patches/launch.sh.patchalternative_recipes/aws/patches/perf_plugins.py.patchalternative_recipes/aws/patches/llama3_llm_pretrain.py.patchTesting
Tested end-to-end 3 times on an 8-node P5en (H200) cluster:
Scope