Validate bounded vLLM cache token sources on H100 #2796
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2,7 +2,7 @@ | |
| set -eo pipefail | ||
| set -x | ||
|
|
||
| # H100 MiniMax-M3 MXFP8 AgentX with EAGLE3 and optional DRAM or NVMe KV offload. | ||
|
|
||
| source "$(dirname "$0")/../../benchmark_lib.sh" | ||
|
|
||
|
|
@@ -97,13 +97,37 @@ EOF | |
| --kv-transfer-config | ||
| '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_both","kv_connector_extra_config":{"load_async":true}}' | ||
| ) | ||
| elif [ "$KV_OFFLOADING" = "nvme" ]; then | ||
| require_agentic_kv_offload_backend vllm-simple | ||
| : "${NVME_OFFLOAD_DIR:?NVME_OFFLOAD_DIR must be mounted by the H100 launcher}" | ||
| NVME_OFFLOAD_TOTAL_BYTES=8000000000000 | ||
| NVME_OFFLOAD_PER_RANK_BYTES=$((NVME_OFFLOAD_TOTAL_BYTES / TP)) | ||
| # vLLM appends .rank_<CUDA device index> to give each TP rank its own file. | ||
| OFFLOAD_ARGS=( | ||
| --kv-transfer-config | ||
| "{\"kv_connector\":\"SimpleCPUOffloadConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"kv_offload_backend\":\"disk\",\"disk_path\":\"$NVME_OFFLOAD_DIR/cache.bin\",\"disk_capacity_bytes\":$NVME_OFFLOAD_PER_RANK_BYTES,\"disk_buffer_slots\":4,\"lazy_offload\":false}}" | ||
| ) | ||
| elif [ "$KV_OFFLOADING" = "dram+nvme" ]; then | ||
| require_agentic_kv_offload_backend vllm-native | ||
| : "${NVME_OFFLOAD_DIR:?NVME_OFFLOAD_DIR must be mounted by the H100 launcher}" | ||
| TOTAL_CPU_DRAM_GIB=$((TOTAL_CPU_DRAM_GB * 1000000000 / 1073741824)) | ||
| PER_RANK_GIB=$(((TOTAL_CPU_DRAM_GIB - MODEL_CHECKPOINT_PAGE_CACHE_GIB) / TP - MODEL_CPU_OFFLOAD_GB - MOONCAKE_LOCAL_BUFFER_GIB)) | ||
| if (( PER_RANK_GIB <= 0 )); then | ||
| echo "Error: CPU DRAM budget is too small for checkpoint cache, model, and DRAM+NVMe KV offload" >&2 | ||
| exit 1 | ||
| fi | ||
| CPU_OFFLOAD_TOTAL_BYTES=$((PER_RANK_GIB * TP * 1073741824)) | ||
|
There was a problem hiding this comment. Native tier budget subtracts Mooncake bufferLow Severity The   Reviewed by Cursor Bugbot for commit fd88716. Configure here. |
||
| OFFLOAD_ARGS=( | ||
| --kv-transfer-config | ||
| "{\"kv_connector\":\"OffloadingConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"spec_name\":\"TieringOffloadingSpec\",\"cpu_bytes_to_use\":$CPU_OFFLOAD_TOTAL_BYTES,\"eviction_policy\":\"lru\",\"secondary_tiers\":[{\"type\":\"fs\",\"root_dir\":\"$NVME_OFFLOAD_DIR\",\"n_read_threads\":32,\"n_write_threads\":16,\"locality\":\"LOCAL\"}]}}" | ||
| ) | ||
| else | ||
| echo "Error: unsupported KV_OFFLOADING='$KV_OFFLOADING'" >&2 | ||
| exit 1 | ||
| fi | ||
|
|
||
| export AIPERF_SERVER_METRICS_URLS="http://localhost:${PORT}/metrics" | ||
| export AIPERF_REQUIRED_SERVER_METRIC_PREFIX="vllm:prompt_tokens_cached_by_source" | ||
|
|
||
| NUM_SPEC_TOKENS=3 | ||
| TOKENS_PER_SEQ=$((1 + NUM_SPEC_TOKENS)) | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -298,13 +298,40 @@ else | |
|
|
||
| export GPU_COUNT="${GPU_COUNT:-${TP:?TP must be set}}" | ||
|
|
||
| # These NVMe AgentX points can need >4 hours of warmup before a 1-hour profile. | ||
| if [[ "${MODEL_PREFIX:-}" == "minimaxm3" && "${SCENARIO_TYPE:-}" == "agentic-coding" && ( "${KV_OFFLOADING:-}" == "nvme" || "${KV_OFFLOADING:-}" == "dram+nvme" ) ]]; then | ||
| SALLOC_TIME_LIMIT="${SALLOC_TIME_LIMIT:-420}" | ||
| fi | ||
| SALLOC_TIME_LIMIT="${SALLOC_TIME_LIMIT:-300}" | ||
| salloc --partition="$SLURM_PARTITION" --account="$SLURM_ACCOUNT" \ | ||
| --gres="gpu:$GPU_COUNT" --exclusive --time="$SALLOC_TIME_LIMIT" \ | ||
| --no-shell --job-name="$RUNNER_NAME" | ||
| JOB_ID=$(squeue --name="$RUNNER_NAME" -u "$USER" -h -o %A | head -n1) | ||
| if [[ -z "$JOB_ID" ]]; then | ||
| echo "ERROR: failed to resolve H100 Slurm allocation" >&2 | ||
| exit 1 | ||
| fi | ||
| cleanup_allocation() { | ||
| local rc=$? | ||
| trap - EXIT INT TERM | ||
| scancel "$JOB_ID" 2>/dev/null || true | ||
| exit "$rc" | ||
| } | ||
| trap cleanup_allocation EXIT INT TERM | ||
|
|
||
| NVME_CONTAINER_MOUNT="" | ||
| if [[ "${KV_OFFLOADING:-none}" == "nvme" || "${KV_OFFLOADING:-none}" == "dram+nvme" ]]; then | ||
| NVME_HOST_ROOT="/mnt/numa0/enroot/cache/group-$(id -g)" | ||
| NVME_HOST_DIR="$NVME_HOST_ROOT/inferencex-kv-$JOB_ID" | ||
| srun --jobid="$JOB_ID" bash -c " | ||
| set -e | ||
| test -w '$NVME_HOST_ROOT' | ||
| mkdir -m 700 '$NVME_HOST_DIR' | ||
| findmnt -T '$NVME_HOST_DIR' | ||
| " | ||
| NVME_CONTAINER_MOUNT=",$NVME_HOST_DIR:/kv-offload" | ||
| export NVME_OFFLOAD_DIR=/kv-offload | ||
|
Comment on lines
+322
to
+333
Contributor
There was a problem hiding this comment. 🔴 The new nvme/dram+nvme branch creates a job-scoped NVMe cache directory ( Extended reasoning...Every sweep run using kv-offloading=nvme or dram+nvme allocates a unique per-JOB_ID directory on the shared host NVMe and writes disk_capacity_bytes-sized cache.bin/tiering files into it (up to ~1TB per TP rank), but cleanup_allocation() at lines 296-303 only calls scancel on EXIT/INT/TERM and never touches NVME_HOST_DIR. Unlike the pre-existing dram/none modes which never wrote to local NVMe, repeated CI sweep invocations now leave orphaned multi-hundred-GB directories under /mnt/numa0/enroot/cache/group-*/ that accumulate indefinitely across runs, eventually exhausting shared NVMe capacity on the H100 cluster. Verification: Severity: normal — a resource leak this change introduces that the base branch does not have. The nvme/dram+nvme path creates a per-job directory on the compute node's local NVMe and mounts it into the container as the KV-offload target, but nothing ever deletes it: - runners/launch_h100-dgxc-slurm.sh:311-318 (in the diff) creates it via |
||
| fi | ||
|
There was a problem hiding this comment. NVMe job cache is never cleaned upHigh Severity The launcher creates a job-scoped NVMe directory under the node's enroot cache and mounts it for Reviewed by Cursor Bugbot for commit fd88716. Configure here. |
||
|
|
||
| # Check the shared cache before opening its lock. A valid squash file is | ||
| # immutable, so readers do not need to touch a lock owned by another user. | ||
|
|
@@ -327,12 +354,10 @@ else | |
|
|
||
| srun --jobid=$JOB_ID \ | ||
| --container-image=$SQUASH_FILE \ | ||
| --container-mounts=$GITHUB_WORKSPACE:/workspace/,$HF_HUB_CACHE_MOUNT:$HF_HUB_CACHE,$AIPERF_MMAP_CACHE_HOST_PATH:/aiperf_mmap_cache$NVME_CONTAINER_MOUNT \ | ||
| --no-container-mount-home \ | ||
| --container-workdir=/workspace/ \ | ||
| --no-container-entrypoint --export=ALL,PORT=8888,AIPERF_DATASET_MMAP_CACHE_DIR=/aiperf_mmap_cache \ | ||
| bash benchmarks/single_node/${SCENARIO_SUBDIR}${EXP_NAME%%_*}_${PRECISION}_h100${SPEC_SUFFIX}.sh | ||
|
|
||
| fi | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🟡 (optional) The new dram+nvme branch copy-pastes the Mooncake DRAM budget formula and still subtracts MOONCAKE_LOCAL_BUFFER_GIB (4 GiB/rank) even though this path uses OffloadingConnector/TieringOffloadingSpec, which never starts Mooncake or allocates that buffer, so the derived per-rank DRAM tier is needlessly smaller than the base branch would give a Mooncake-only run, and can trip the 'CPU DRAM budget is too small' exit at configs the actual budget could otherwise support. Fix: only subtract MOONCAKE_LOCAL_BUFFER_GIB when the connector actually uses Mooncake; for the dram+nvme/OffloadingConnector path compute PER_RANK_GIB from TOTAL_CPU_DRAM_GIB, MODEL_CHECKPOINT_PAGE_CACHE_GIB, and MODEL_CPU_OFFLOAD_GB only.
Extended reasoning...
Line 114 reuses the exact expression from the dram/Mooncake branch (line 65) verbatim, including '- MOONCAKE_LOCAL_BUFFER_GIB', inside the elif KV_OFFLOADING=dram+nvme block that instead builds an OffloadingConnector kv-transfer-config (lines 120-123) which has no Mooncake component and never sets MOONCAKE_CONFIG_PATH or starts mooncake_master. The 4 GiB/rank/TP is therefore withheld from the DRAM tier for no reason tied to this connector, and at the lower end of the DRAM budget this extra deduction can push PER_RANK_GIB to <=0, aborting the run (line 115-118) at a TOTAL_CPU_DRAM_GB value that would otherwise succeed.
Verification: nit. Factually real but negligible impact. Line 114 (
PER_RANK_GIB=$(((TOTAL_CPU_DRAM_GIB - MODEL_CHECKPOINT_PAGE_CACHE_GIB) / TP - MODEL_CPU_OFFLOAD_GB - MOONCAKE_LOCAL_BUFFER_GIB))) in the newdram+nvmebranch copies line 65's Mooncake formula verbatim, still subtractingMOONCAKE_LOCAL_BUFFER_GIB=4(set line 59). That branch builds anOffloadingConnector/TieringOffloadingSpecconfig…