From 3dd6565285a17df2871873865432cbd59283bdf3 Mon Sep 17 00:00:00 2001 From: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com> Date: Thu, 6 Aug 2026 10:01:37 -0700 Subject: [PATCH] [nvbugs/6561777][fix] Fence all Slurm ranks before launching pytest TestDeepSeekR1::test_nvfp4_multi_gpus[latency_adp_lmtp] is the fourth test row attributed to a single multi-node launch defect, alongside nvbugs/6541343, 6561775 and 6561778 -- all from build 2885 on the same GB200 8-GPU 2-node stage family, all reporting the same "Test terminated unexpectedly" with no traceback. That string is synthesized, not a crash: generate_timeout_xml.py fabricates a testcase with it for every nodeid left in unfinished_test.txt, so it only means the srun step was killed while the test was in flight. The only install lock lives under $resourcePathNode (/tmp) in slurm_install.sh. In a pyxis container /tmp is a per-step tmpfs, so that lock is node-local: its wait loop fences just the $SLURM_LOCALID peers on the same node, and a node can never observe another node's lock. Nothing then stops slurm_run.sh from reaching `eval $pytestCommand` on one node while another is still installing, and the per-rank work in between skews the ranks further: non-zero ranks cover rank 0's coverage-config write with a blind `sleep 30`, and slurm_setup_runtime_env shells out to pip3. Pytest's first action is `import tensorrt_llm`, whose module-scope MPI collective must be entered by every rank. Under --mpi=pmix, which is added exactly when nodeCount > 1, that collective has a 300s fence timeout, so a node whose pip3 install stalls (up to the 2700s retry budget) makes the collective abort every rank rather than merely run late. The ranks then die between pytest setup and teardown, which is what leaves the nodeid in unfinished_test.txt. Note that PMIX_MCA_gds=hash does not mitigate this: a fence that times out never exchanges the modex regardless of GDS mode, and the pml_ucx errors seen alongside it are downstream of the same missing exchange. Add a marker barrier on the shared $jobWorkspace immediately before `eval $pytestCommand`. It counts SLURM_NTASKS rank markers rather than nodes, so the fenced set is exactly the set that enters the aborting collective, and placing it after the block that wipes SLURM_* keeps it a no-op for single-node and disaggregated benchmark/server runs, which reach it with SLURM_NTASKS unset. The marker directory is keyed per job and per step because $jobWorkspace outlives a step, so a later step must not be released by an earlier one's markers. The wait is bounded above the 2700s pip3 budget so a genuinely dead rank fails the stage with a clear message instead of hanging until the partition walltime. This is the same change already proposed for nvbugs/6541343; it is carried here byte-identically so whichever lands first makes the other a no-op. This variant's ISOLATION marker in l0_gb200_multi_nodes.yml does not change any of the above: runIsolatedTests is only reachable from runLLMTestlistOnPlatformImpl, the Docker path, so on the Slurm path the marker is merely stripped and the test shares one slurm_run.sh pytest with its shard. Verified by asserting elapsed seconds, since a no-op barrier also returns 0: single rank, unset workspace and unset SLURM_NTASKS return at 0s; a full marker set releases at 0s; markers from another job, from an earlier step of the same job, and a dead rank each fail bounded with rc=1 through the ERR trap. On a live 2-node 8-rank pmix launch with rank 7 delayed 20s, ranks on both hosts blocked 20-30s and every rank agreed on the same SLURM_STEP_ID, showing the fence crosses the node boundary; with no delay all 8 released within one poll interval. Also un-waive the test, whose body was already healthy. Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com> --- jenkins/scripts/slurm_run.sh | 53 +++++++++++++++++++++++++ tests/integration/test_lists/waives.txt | 1 - 2 files changed, 53 insertions(+), 1 deletion(-) diff --git a/jenkins/scripts/slurm_run.sh b/jenkins/scripts/slurm_run.sh index 25622daff391..2d247e0c310d 100755 --- a/jenkins/scripts/slurm_run.sh +++ b/jenkins/scripts/slurm_run.sh @@ -72,6 +72,59 @@ if [ "${SLURM_JOB_NUM_NODES:-1}" -eq 1 ] || \ done fi +# The only install lock (slurm_install.sh) lives under $resourcePathNode, a per-step +# tmpfs in the pyxis container, so it fences just the $SLURM_LOCALID peers on one node +# -- nothing stops one node from reaching `eval $pytestCommand` below while another is +# still installing. Pytest's first action is `import tensorrt_llm`, whose module-scope +# MPI collective must be entered by every rank, and under --mpi=pmix (added exactly +# when nodeCount > 1) that collective has a 300s fence timeout -- so the skew aborts +# every rank instead of merely running late. Fence on the shared $jobWorkspace so all +# ranks enter pytest together. Must stay below the SLURM_* unset block above, which is +# what makes this a no-op for single-node and disaggregated runs. +slurm_wait_all_ranks() { + # SLURM_NTASKS rather than a node count: this is exactly the set of ranks that + # enters the aborting collective. + local numRanks="${SLURM_NTASKS:-1}" + if [ "$numRanks" -le 1 ] || [ -z "${jobWorkspace:-}" ]; then + return 0 + fi + + # Keyed per job *and* per step: $jobWorkspace outlives a single step, so markers + # from another job or from an earlier step must not satisfy the count. Slurm gives + # every rank of a step the same step id, so all of them agree on this path. + local readyDir="$jobWorkspace/run_ready_job_${SLURM_JOB_ID:-local}_step_${SLURM_STEP_ID:-0}" + mkdir -p "$readyDir" + touch "$readyDir/rank_${SLURM_PROCID}.ready" + + # Bounded above the 2700s pip3 retry budget in slurm_install.sh: a merely slow rank + # still releases the barrier, while a dead one fails the stage loudly instead of + # hanging until the partition walltime kills it. + local timeoutSecs=3600 + local deadline=$((SECONDS + timeoutSecs)) + local markers ready + while true; do + # Globbed, not `ls | wc -l`: under `set -Eeuo pipefail` a failing `ls` fires the + # ERR trap. The touch above guarantees a match, so no nullglob is needed. + markers=("$readyDir"/*.ready) + ready=${#markers[@]} + if [ "$ready" -ge "$numRanks" ]; then + return 0 + fi + if [ "$SECONDS" -ge "$deadline" ]; then + echo "ERROR: rank ${SLURM_PROCID} timed out after ${timeoutSecs}s waiting for" \ + "all $numRanks ranks to be ready; ready: $ready/$numRanks" + return 1 + fi + # One rank reports progress; all of them would spam the log every 10s. + if [ "$SLURM_PROCID" -eq 0 ]; then + echo "(Waiting for all $numRanks ranks to be ready) ready: $ready/$numRanks" + fi + sleep 10 + done +} + +slurm_wait_all_ranks + # Turn off "exit on error" so the following lines always run set +e diff --git a/tests/integration/test_lists/waives.txt b/tests/integration/test_lists/waives.txt index 71b5595714c3..34b8d4c040ab 100644 --- a/tests/integration/test_lists/waives.txt +++ b/tests/integration/test_lists/waives.txt @@ -22,7 +22,6 @@ accuracy/test_llm_api_pytorch.py::TestDeepSeekR1::test_fp8_blockscale[throughput accuracy/test_llm_api_pytorch.py::TestDeepSeekR1::test_fp8_blockscale[throughput_mtp] SKIP (https://nvbugs/6428101) accuracy/test_llm_api_pytorch.py::TestDeepSeekR1::test_fp8_blockscale[throughput_mtp_trtllm] SKIP (https://nvbugs/6426868) accuracy/test_llm_api_pytorch.py::TestDeepSeekR1::test_nvfp4_multi_gpus[latency] SKIP (https://nvbugs/6561778) -accuracy/test_llm_api_pytorch.py::TestDeepSeekR1::test_nvfp4_multi_gpus[latency_adp_lmtp] SKIP (https://nvbugs/6561777) accuracy/test_llm_api_pytorch.py::TestDeepSeekR1::test_nvfp4_multi_gpus[throughput_pp4_mtp] SKIP (https://nvbugs/6481323) accuracy/test_llm_api_pytorch.py::TestDeepSeekV32::test_dsa_host_cache_offload[host_cache_offload_mtp1] SKIP (https://nvbugs/6384357) accuracy/test_llm_api_pytorch.py::TestDeepSeekV32::test_dsa_host_cache_offload[host_cache_offload_mtp3_no_adp] SKIP (https://nvbugs/6384357)