[https://nvbugs/6649384][test] Support full B200 stress A/B at pinned current head

[chienchunhung]

Description

Investigation-only draft for NVBUG 6649384. Do not merge. This PR preserves the historical #16687 revert and supplies a shared harness for a full eight-B200 stress comparison. It does not establish that #16687 caused the failure.

The active experiment compares current upstream main with and without the candidate behavior. It rebuilds the current control once and shares that wheel, compiled artifacts, dependency environment, and image digest between arms. The treatment changes only two Python files. The public PyTorch base matches current Dockerfile.multi; this is a current-environment comparison, not an exact recreation of August CI.

Active experiment identity	Pinned revision
Control runtime, upstream main resolved September 18	a8f436b2e9b85c0cedaa1eeceb2834597d7f7dd1
Treatment runtime, adapted semantic #16687 revert	1d2a6a4f27fc4c9be13b8981e08a19e6128eab66
Common current test checkout, with opt-in log retention	27af8c456fcd4c3ce70f29d6c445701f756e6a05
Orchestration code in this PR	12ddc952a8db03496b960b583f8b2f7f719df270

The current runtime/test commits are prepared in isolated checkouts and a verified Git bundle retained with cluster artifacts; they are distinct from this historical PR branch. Subsequent refactors moved part of #16687's behavior into disagg_adapter.py, so the current treatment changes that file together with py_executor.py, while preserving later admission/coordinator refactors. The shared current test checkout retains worker logs through TLLM_DISAGG_STRESS_KEEP_LOGS=1.

Experiment and validity

examples/disaggregated/slurm/stress_ab/ contains the baseline builder, Slurm launcher, trial driver, reconciliation, tests, and setup instructions. An explicit profile pins both runtime commits, the common test checkout, allowed Python deltas, request count, and dependency versions. The historical comparison remains the default; current runs must supply their explicit profile and separate runner root.

• Run control, treatment, treatment, control sequentially in one exclusive node allocation: eight B200s, 224 CPUs, approximately 2 TB host memory, and 16 hours. Each trial gets a fresh Slurm step/container and a 3.5-hour pytest limit. Build separately on CPUs.
• Invoke disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-gpt_oss_120b_eagle_trtllm_stress] directly, without a waive file.
• Use TP4 context plus TP4 generation, attention DP, GPT-OSS120B/Eagle3, Python NIXL, 8K/1K, concurrency 512, 60,000 requests and the test's cancellation settings. Preserve the 5% non-cancellation error and 0.42 GSM8K gates.
• Both arms use the same current test/YAML and dependencies: AIPerf 0.8.0, lm_eval 0.4.10 and NIXL 1.4.0. Current YAML/dependencies differ from the historical environment.
• Verify exact source and wheel hashes, identical compiled artifacts, imported runtime paths, persistent venv/dependency inventory, inputs, GPU identity and cleanup. Retain initiating context/generation exceptions as well as downstream shutdown errors.
• Reject skips, missing tests, partial request accounting, missing logs, setup failures, dependency drift, timeout and residual processes. Preserve failed test results. No automatic retries or requeue.

The builder creates a persistent venv before running the source build, keeps large CMake output on node-local storage, and records provenance. CPU builds may skip GPU-dependent stub generation without skipping native compilation.

Current status and validation

As of September 18, 2026, 19:28 UTC:

• 27 GPU-free harness tests passed, including real temporary-venv checks and shell launcher tests using scheduler stand-ins. All applicable pre-commit, shell syntax, dry-plan and whitespace checks passed.
• Seven CPU behavioral checks passed for the adapted current treatment. These validate the intended diagnostic delta, not GPU behavior.
• Pinned source/submodules and shared GPT-OSS120B/Eagle3/GSM8K inputs are prepared on computelab-sc-01.
• Native prerequisite checks passed, but the current control build failed during CMake configuration in CPU allocation 4394822 (step .6, exit1; September18 19:13:57UTC). CMake installs cutlass_library using develop --user, while the builder sets PYTHONNOUSERSITE=1; kernel generation then raises ModuleNotFoundError. No wheel or GPU experiment exists. A private build-only user-site correction is prepared but has not been applied or tested on the cluster. No automatic retry was launched.
• Common build/runtime base: nvcr.io/nvidia/pytorch:26.08-py3, amd64 manifest sha256:33ef5fc15e8937602d64022209cdb2777b32dadf742f41332023d946041b3c14. Exact current CI devel-image access returned HTTP403; this public base is recorded as an environment difference.
• Persistent run records: /home/scratch.chienchunh_coreai/trtllm-agent/pr18327-current-4394822. GPU submission is gated on a successful build, verified provenance and runtime/input checks.

The historical default remains control 0f2c3a95f9415045bdf06a7230759475692483b6 versus treatment 3245fc3ecd76e2fb610f42f2422102e2430c28fe; it requires its own matching runtime and image.

Coverage correction: the earlier B200-filtered PR pipeline ran zero mapped tests. Its success does not establish stress coverage. The earlier eight-request TP2 rc25 smoke run also does not reproduce this stress shape.

A failing control and passing treatment supports an effect under these recorded conditions; two trials per arm do not establish flake causality. Both passing means no reproduction under current conditions. Compare initiating worker exceptions and complete request/accuracy evidence, not just the downstream LLM is shutting down symptom.

PR Checklist

• Current and historical profiles keep runtime, test checkout and runner identities separate.
• Full stress shape, validity gates, log retention and bounded execution documented.
• Local harness and current-treatment checks passed.
• Current baseline build and GPU runtime verification completed.
• Full same-node A/B/B/A experiment executed and analyzed.

[chienchunhung]

/bot run --disable-fail-fast --gpu-type "B200"

[tensorrt-cicd]

PR_Github #69828 [ run ] triggered by Bot. Commit: 3245fc3 Link to invocation

[tensorrt-cicd]

PR_Github #69828 [ run ] completed with state SUCCESS. Commit: 3245fc3
/LLM/main/L0_MergeRequest_PR pipeline #57119 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation