[Klaud Cold] minimaxm3-fp4-b200-vllm-agentic-mtp: MiniMax-M3 NVFP4 B200 AgentX with EAGLE3-GQA MTP and golden-AL synthetic acceptance / 新增 MiniMax-M3 NVFP4 B200 vLLM EAGLE3-GQA MTP AgentX 配方（黄金 AL 合成接受率） #2511
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,214 @@ | ||
| #!/usr/bin/env bash | ||
| set -euo pipefail | ||
| set -x | ||
|
|
||
| # MiniMax-M3 NVFP4 B200 AgentX with EAGLE3-GQA and synthetic acceptance. | ||
| # DRAM KV offload uses vLLM's SimpleCPUOffloadConnector in lazy mode. | ||
| # | ||
| # Port of the validated agentic/minimaxm3_fp4_b300_mtp.sh. Spec-decode only, | ||
| # per the AgentX policy that agentic recipes are run and published with | ||
| # speculative decoding enabled rather than as an STP/MTP A/B (MODELS.md: | ||
| # MiniMax-M3 agentic non-EAGLE3 is deprecated after 2026-08-03, and the 8k1k | ||
| # scenario that carried the B200 MiniMax-M3 curves was removed in #2493 -- | ||
| # without this recipe MiniMax-M3 has no active B200 config at all). | ||
| # | ||
| # The only B200 deltas are the two blocks marked "B200:" below -- the | ||
| # checkpoint-resolution guard and the draft staging path. Every engine flag is | ||
| # the B300 script unchanged so the two SKU curves stay directly comparable. | ||
| # | ||
| # Required env vars: | ||
| # MODEL, TP, CONC, KV_OFFLOADING, TOTAL_CPU_DRAM_GB, RESULT_DIR, DURATION | ||
|
|
||
| source "$(dirname "$0")/../../benchmark_lib.sh" | ||
|
|
||
| export EVAL_FRAMEWORK="lm-eval" | ||
|
|
||
| check_env_vars MODEL TP CONC KV_OFFLOADING TOTAL_CPU_DRAM_GB RESULT_DIR DURATION | ||
|
|
||
| DRAFT_MODEL="Inferact/MiniMax-M3-EAGLE3-GQA" | ||
| NUM_SPEC_TOKENS=3 | ||
| # Golden AL for the GQA draft head: golden_al_distribution/minimaxm3_eagle3_gqa.yaml | ||
| # minimax-m3.thinking_on[3]. The non-GQA curve (minimaxm3_eagle3.yaml) reads 2.83 | ||
| # at the same level -- that head is not what this script runs. The curve is a | ||
| # property of the target/draft pair, not of the SKU it was measured on, so B200 | ||
| # pins the same 2.78 the B300 sibling does. | ||
| SYNTHETIC_ACCEPT_LEN=2.78 | ||
|
|
||
| if [[ -n "${SLURM_JOB_ID:-}" ]]; then | ||
| echo "JOB $SLURM_JOB_ID running on ${SLURMD_NODENAME:-unknown}" | ||
| fi | ||
|
|
||
| # A non-empty directory is NOT a staged checkpoint: an aborted pull leaves | ||
| # config.json and friends behind with no weights, and an `ls -A` guard accepts | ||
| # it, so the cell skips the download and dies later inside the loader. Check | ||
| # that every shard the index names is actually present -- and accept the | ||
| # single-file layout the EAGLE3 draft head ships in, which has no index. | ||
| checkpoint_is_complete() { | ||
| local dir="$1" | ||
| [[ -d "$dir" && -f "$dir/config.json" ]] || return 1 | ||
| CKPT_DIR="$dir" python3 - <<'PYEOF' | ||
| import glob, json, os, sys | ||
|
|
||
| d = os.environ["CKPT_DIR"] | ||
| index = os.path.join(d, "model.safetensors.index.json") | ||
| if os.path.isfile(index): | ||
| with open(index) as fh: | ||
| shards = sorted(set(json.load(fh)["weight_map"].values())) | ||
| missing = [s for s in shards if not os.path.isfile(os.path.join(d, s))] | ||
| if missing: | ||
| print( | ||
| f"{len(missing)}/{len(shards)} shards missing, e.g. {missing[:3]}", | ||
| file=sys.stderr, | ||
| ) | ||
| sys.exit(1) | ||
| elif not glob.glob(os.path.join(d, "*.safetensors")): | ||
| print("no shard index and no .safetensors present", file=sys.stderr) | ||
| sys.exit(1) | ||
| PYEOF | ||
| } | ||
|
|
||
| # B200: runners/launch_b200-dgxc.sh resolves the checkpoint to a cluster-local | ||
| # path (/scratch/fsw/models/MiniMax-M3-NVFP4) and then rewrites MODEL to that | ||
| # path before handing off to this script, so `hf download "$MODEL"` cannot work | ||
| # on this runner the way it does on b300-nv, where MODEL stays the HF repo id. | ||
| # Keep the repo id separate for the case where the checkpoint is not staged. | ||
| HF_MODEL_ID="${HF_MODEL_ID:-nvidia/MiniMax-M3-NVFP4}" | ||
|
|
||
| if [[ -n "${MODEL_PATH:-}" ]]; then | ||
| if ! checkpoint_is_complete "$MODEL_PATH"; then | ||
| # Every concurrency of this sweep runs as its own allocation against | ||
| # the same shared path, so serialize: one cell pulls the ~250 GB | ||
| # checkpoint and the rest wait on it rather than racing as writers. | ||
| # `hf download` resumes into a partially-populated --local-dir. | ||
| mkdir -p "$MODEL_PATH" | ||
| MODEL_DOWNLOAD_LOCK="${MODEL_PATH%/}.download.lock" | ||
| echo "Checkpoint at $MODEL_PATH is incomplete; acquiring $MODEL_DOWNLOAD_LOCK" | ||
| exec 9>"$MODEL_DOWNLOAD_LOCK" | ||
| flock -w "${MODEL_DOWNLOAD_LOCK_TIMEOUT:-21600}" 9 || { | ||
| echo "Error: timed out waiting for another cell to stage $MODEL_PATH" >&2 | ||
| exit 1 | ||
| } | ||
| if checkpoint_is_complete "$MODEL_PATH"; then | ||
| echo "Another cell staged $MODEL_PATH while we waited" | ||
| else | ||
| hf download "$HF_MODEL_ID" --local-dir "$MODEL_PATH" | ||
| fi | ||
| flock -u 9 | ||
| exec 9>&- | ||
| checkpoint_is_complete "$MODEL_PATH" || { | ||
| echo "Error: $MODEL_PATH is still incomplete after hf download $HF_MODEL_ID." >&2 | ||
| exit 1 | ||
| } | ||
| fi | ||
| else | ||
| hf download "$HF_MODEL_ID" | ||
| export MODEL_PATH="$HF_MODEL_ID" | ||
| fi | ||
|
|
||
| # B200: the B300 sibling stages the draft under /data/models, which does not | ||
| # exist on b200-dgxc. The launcher bind-mounts only $MODEL_PATH itself | ||
| # (--container-mounts=...,$MODEL_PATH:$MODEL_PATH,...), so its parent exists | ||
| # solely inside the container overlay -- writable, but per-job. Stage the draft | ||
| # there rather than inside $MODEL_PATH, which is the shared read-mostly | ||
| # checkpoint directory and must not be polluted. The EAGLE3-GQA head is small | ||
| # next to the target, so the per-job pull is cheap; this is what the | ||
| # now-deprecated 8k1k B200 MiniMax-M3 MTP recipe did as well. | ||
| DRAFT_MODEL_PATH="${DRAFT_MODEL_PATH:-$(dirname "${MODEL_PATH%/}")/${DRAFT_MODEL##*/}}" | ||
|
Comment on lines
+104
to
+116
Contributor
There was a problem hiding this comment. 🟡 In the else-branch where Extended reasoning...The bug: else
hf download "$HF_MODEL_ID"
export MODEL_PATH="$HF_MODEL_ID"
fi
DRAFT_MODEL_PATH="${DRAFT_MODEL_PATH:-$(dirname "${MODEL_PATH%/}")/${DRAFT_MODEL##*/}}"Step-by-step proof:
Why this formula exists at all: it's correct in the Why the B300 sibling doesn't have this issue: Impact: This is dead code in the actual b200-dgxc sweep — Fix: In the else-branch, set |
||
| if ! checkpoint_is_complete "$DRAFT_MODEL_PATH"; then | ||
| hf download "$DRAFT_MODEL" --local-dir "$DRAFT_MODEL_PATH" | ||
| checkpoint_is_complete "$DRAFT_MODEL_PATH" || { | ||
| echo "Error: $DRAFT_MODEL_PATH is incomplete after hf download $DRAFT_MODEL." >&2 | ||
| exit 1 | ||
| } | ||
| fi | ||
|
|
||
| nvidia-smi | ||
| resolve_trace_source | ||
| install_agentic_deps | ||
|
|
||
| OFFLOAD_ARGS=() | ||
| if require_agentic_kv_offload_backend vllm-simple; then | ||
| CPU_OFFLOAD_BYTES=$((TOTAL_CPU_DRAM_GB * 1024 * 1024 * 1024)) | ||
| export VLLM_USE_SIMPLE_KV_OFFLOAD=1 | ||
| OFFLOAD_CONFIG=$(printf \ | ||
| '{"kv_connector":"SimpleCPUOffloadConnector","kv_role":"kv_both","kv_connector_extra_config":{"cpu_bytes_to_use":%d,"lazy_offload":true}}' \ | ||
| "$CPU_OFFLOAD_BYTES") | ||
| OFFLOAD_ARGS=(--kv-transfer-config "$OFFLOAD_CONFIG") | ||
| fi | ||
|
|
||
| export PYTHONNOUSERSITE=1 | ||
| export VLLM_ENGINE_READY_TIMEOUT_S=3600 | ||
| export VLLM_FLOAT32_MATMUL_PRECISION=high | ||
| export VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm | ||
|
|
||
| # Same 0.9 the B300 sibling and the deprecated 8k1k B200 MiniMax-M3 NVFP4 MTP | ||
| # recipe ran. Exposed as an override because B200's 180 GB leaves little beyond | ||
| # the ~250 GB checkpoint: TP2 could not host the 1M-context KV for even one | ||
| # request at this value and was dropped from the search space, so TP4 is the | ||
| # smallest topology this script is expected to serve. | ||
| GPU_MEMORY_UTILIZATION="${GPU_MEMORY_UTILIZATION:-0.9}" | ||
|
|
||
| SERVER_LOG="$RESULT_DIR/server.log" | ||
| mkdir -p "$RESULT_DIR" | ||
|
|
||
| SERVER_PID="" | ||
| cleanup_agentic_services() { | ||
| local exit_code=$? | ||
| trap - EXIT INT TERM | ||
| set +e | ||
| stop_background_process_tree "$SERVER_PID" "vLLM server" 60 | ||
| exit "$exit_code" | ||
| } | ||
| trap cleanup_agentic_services EXIT | ||
| trap 'exit 130' INT | ||
| trap 'exit 143' TERM | ||
|
|
||
| if [ "${EVAL_ONLY:-}" = "true" ]; then | ||
| SPEC_CONFIG=$(printf \ | ||
| '{"method":"eagle3","model":"%s","num_speculative_tokens":%d,"attention_backend":"FLASH_ATTN"}' \ | ||
| "$DRAFT_MODEL_PATH" "$NUM_SPEC_TOKENS") | ||
| else | ||
| SPEC_CONFIG=$(printf \ | ||
| '{"method":"eagle3","model":"%s","num_speculative_tokens":%d,"attention_backend":"FLASH_ATTN","rejection_sample_method":"synthetic","synthetic_acceptance_length":%.2f}' \ | ||
| "$DRAFT_MODEL_PATH" "$NUM_SPEC_TOKENS" "$SYNTHETIC_ACCEPT_LEN") | ||
| fi | ||
|
|
||
| { set +x; } 2>/dev/null | ||
| VLLM_CMD=( | ||
| vllm serve "$MODEL_PATH" | ||
| --served-model-name "$MODEL" | ||
| --host 0.0.0.0 | ||
| --port "$PORT" | ||
| --tensor-parallel-size "$TP" | ||
| --gpu-memory-utilization "$GPU_MEMORY_UTILIZATION" | ||
| --block-size 128 | ||
| --language-model-only | ||
| --enable-prefix-caching | ||
| --no-enable-flashinfer-autotune | ||
| --reasoning-parser minimax_m3 | ||
| --tool-call-parser minimax_m3 | ||
| --enable-auto-tool-choice | ||
| --default-chat-template-kwargs '{"thinking_mode":"enabled"}' | ||
| --attention-config '{"backend":"FLASHINFER","use_trtllm_attention":true,"indexer_kv_dtype":"fp8"}' | ||
| --kv-cache-dtype fp8 | ||
| --max-cudagraph-capture-size 512 | ||
| --max-num-batched-tokens 16384 | ||
| --stream-interval 20 | ||
| --trust-remote-code | ||
| --speculative-config "$SPEC_CONFIG" | ||
| "${OFFLOAD_ARGS[@]}" | ||
| ) | ||
| printf '%q ' "${VLLM_CMD[@]}" | tee "$RESULT_DIR/vllm_command.txt" | ||
| printf '\n' | tee -a "$RESULT_DIR/vllm_command.txt" | ||
| "${VLLM_CMD[@]}" > "$SERVER_LOG" 2>&1 & | ||
| SERVER_PID=$! | ||
| echo "Server PID: $SERVER_PID" | ||
| set -x | ||
|
|
||
| wait_for_server_ready --port "$PORT" --server-log "$SERVER_LOG" --server-pid "$SERVER_PID" | ||
| if [ "${EVAL_ONLY}" = "true" ]; then | ||
| run_eval --port "$PORT" | ||
| else | ||
| build_replay_cmd "$RESULT_DIR" | ||
| run_agentic_replay_and_write_outputs "$RESULT_DIR" | ||
| fi | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔴 The checkpoint-download flock's lock file is placed at
${MODEL_PATH%/}.download.lock, a sibling of $MODEL_PATH in its parent directory — butlaunch_b200-dgxc.sh(line 509) bind-mounts only $MODEL_PATH itself into each cell's container, so that parent (and the lock file inside it) is private per-container overlay, not shared storage. On a cold start where the checkpoint isn't pre-staged, every concurrent cell in the sweep gets its own uncontested lock and simultaneously runshf downloadagainst the same shared $MODEL_PATH — exactly the multi-writer race the flock was added to prevent. Fix by placing the lock file inside $MODEL_PATH (e.g.$MODEL_PATH/.download.lock), which is actually bind-mounted and shared across cells.Extended reasoning...
The bug: In
minimaxm3_fp4_b200_mtp.sh(lines 83-101), when the checkpoint at$MODEL_PATHis incomplete, the script does:For the minimaxm3/fp4 case,
MODEL_PATH=/scratch/fsw/models/MiniMax-M3-NVFP4(launch_b200-dgxc.sh:73), soMODEL_DOWNLOAD_LOCKresolves to/scratch/fsw/models/MiniMax-M3-NVFP4.download.lock— a sibling file in the parent directory/scratch/fsw/models/, not a path inside$MODEL_PATHitself.Why the lock never crosses containers:
launch_b200-dgxc.sh:509bind-mounts only:The parent directory
/scratch/fsw/models/is never bind-mounted from the host. Every cell in the sweep runs as its ownsalloc --exclusive/srunallocation with its own container (lines 484-513), so each container's view of/scratch/fsw/models/above the$MODEL_PATHmount point is its own private overlay filesystem. The PR's own comment on the draft-staging block two paragraphs below confirms this exact model of the parent directory: "the launcher bind-mounts only $MODEL_PATH itself ... so its parent exists solely inside the container overlay — writable, but per-job." That statement applies equally to the lock file, which lives at that same per-job parent path.Concrete walk-through: Suppose the checkpoint is not pre-staged (a fresh cluster, or a purged cache) and the sweep launches the 8-point TP4 GPU-resident arm (conc = 1, 2, 5, 8, 10, 12, 15, 20) concurrently, each as its own srun/container:
checkpoint_is_complete "$MODEL_PATH"→ false. It runsexec 9>/scratch/fsw/models/MiniMax-M3-NVFP4.download.lockinside its own container overlay, creating a private lock file no other container can see.flockacquires instantly since there is no real contention.hf download nvidia/MiniMax-M3-NVFP4 --local-dir "$MODEL_PATH"at the same time, all writing into the one real, bind-mounted, shared$MODEL_PATHdirectory on/scratch/fsw.checkpoint_is_completecheck sees "all shards present" while a sibling cell is still mid-write on some of those same shard files, loading a truncated/corrupt checkpoint.This is precisely the multi-writer race the flock block's own comment says it exists to prevent ("serialize: one cell pulls the ~220 GB checkpoint and the rest wait on it rather than racing as writers").
Why nothing else catches this:
checkpoint_is_completeonly checks for shard presence, not size/hash, so a checkpoint mid-write by another cell can appear complete once all shard filenames exist, even if content is still being written.hf download's internal locking (if any) is scoped to a single process/host and does not span separate containers on separate nodes.Fix: Put the lock file inside the shared mount, e.g.
MODEL_DOWNLOAD_LOCK="$MODEL_PATH/.download.lock"(or${MODEL_PATH%/}/.download.lock), since$MODEL_PATHitself is the one path actually bind-mounted and shared across all cells. This is a one-line fix and doesn't require any other structural change to the block.Scope note: In steady state, the checkpoint is normally pre-staged on
/scratch/fsw/models(per the launcher's own documented pre-provisioned paths), socheckpoint_is_completeshort-circuits and this download+flock block never executes. The bug only bites the cold-start/day-zero staging path — but it fully defeats the safety mechanism the PR explicitly added for exactly that scenario, and the failure mode when triggered is concurrent corruption-prone writers on a real shared 220 GB checkpoint directory.