diff --git a/.agents/skills/deploy-afd-k8s/SKILL.md b/.agents/skills/deploy-afd-k8s/SKILL.md index 8a528f701..a7ef9b68e 100644 --- a/.agents/skills/deploy-afd-k8s/SKILL.md +++ b/.agents/skills/deploy-afd-k8s/SKILL.md @@ -1,6 +1,6 @@ --- name: deploy-afd-k8s -description: Use when the user asks to deploy, serve, or stand up an AFD GPU recipe on Kubernetes/OpenShift from a recipe .sh script - creating the model PVC, recipe ConfigMap, serve pod(s), and Service(s), and waiting until the endpoint answers. Placement across pods is never asked for or assumed -- it is read directly off the recipe's own top-of-file comment block (however that block describes it, strict or prose): a recipe whose comment describes a per-pod placement deploys exactly that many pods with exactly that per-pod setup; a recipe without one deploys as a single pod. Do not use for local (non-k8s) serving, NPU recipes, prefill-decode disaggregation recipes, or E2E correctness testing (see run-e2e). +description: Use when the user asks to deploy, serve, or stand up an AFD GPU recipe on Kubernetes/OpenShift from a recipe .sh script - creating the model PVC, recipe ConfigMap, serve pod(s), and Service(s), and waiting until the endpoint answers. Placement across pods is never asked for or assumed -- it is the recipe's LAYOUT (the caller's setting, else the script's default): one pod per LAYOUT entry, a one-entry LAYOUT being a single pod. Do not use for local (non-k8s) serving, NPU recipes, prefill-decode disaggregation recipes, or E2E correctness testing (see run-e2e). --- # Deploy an AFD GPU recipe on Kubernetes @@ -15,13 +15,13 @@ On success this skill guarantees, in the target namespace: | | | |---|---| -| Endpoint | `http://vllm-service:${CLIENT_PORT}` (OpenAI-compatible, in-namespace; `CLIENT_PORT` defaults to `18305`) | +| Endpoint | `http://vllm-service:${CLIENT_PORT}` (OpenAI-compatible, in-namespace; `CLIENT_PORT` is the launcher's Attention port, `18000`) | | Serving | the model named by `MODEL_ID` | -| Pod(s) / Service | `vllm-pod`, or one pod per pod the recipe's header comment describes (named `vllm-pod-`); Service `vllm-service`, labels `app=afd-recipe,role=serve` | +| Pod(s) / Service | `vllm-pod`, or one pod per `LAYOUT` entry (named `vllm-pod-afd-`); Service `vllm-service`, labels `app=afd-recipe,role=serve` | | Model cache | `PVC_NAME`, mounted at `/models`, `HF_HOME=/models/.hf_home` | | Left running | yes, deliberately -- so weights stay warm for follow-up runs; step 5 asks the caller when to delete the pods | -Whenever a recipe's header spans more than one pod, two more internal-only +Whenever a recipe's `LAYOUT` spans more than one pod, two more internal-only Services may also exist (`vllm-ffn-p2p-service` always; `vllm-attn-dp-service` and/or `vllm-ffn-dp-service` only if that role is itself split across more than one pod) -- callers never talk to these directly. See @@ -38,15 +38,15 @@ than one pod) -- callers never talk to these directly. See NIXL-enabled image plus a proxy, and resolve a `SCRIPT_DIR`-relative path that this skill's flat script mount (step 4b) breaks. Ask the user for a different workflow for disaggregation. -- Recipes are hand-written shell scripts, not JSON configs -- there is no - generator step and no manifest to read. -- A recipe is **placement-driven** (more than one pod) exactly when its - top-of-file comment block describes a per-pod placement -- read that - block, don't infer shape from flags. Today's recipes spell it as one - `# For pod N: POD=...` line per pod, but the comment isn't guaranteed to - use that exact grammar going forward. See - [resolve-recipe.md](resources/resolve-recipe.md). -- Not for `tools/benchmarks/decode_bench_server.sh` (local, non-k8s). +- Recipes are short shell scripts on the shared launcher + `recipe/gpu/common/afd_colocation.sh` (see `recipe/gpu/common/README.md`); + the recipe ConfigMap ships both files (step 4b). +- Placement is `LAYOUT` (one pod per `AF` entry) and run-specific + settings are env (`ATTN_TP`, `FFN_TP`, `VLLM_ARGS`, `DECODE_BENCH`, ...); + see [resolve-recipe.md](resources/resolve-recipe.md). This skill only + serves; load for the decode-only benchmark + (`tools/benchmarks/request_generator.sh`) is run separately against the + endpoint. ## Requirements @@ -63,9 +63,9 @@ than one pod) -- callers never talk to these directly. See docker push "$IMAGE" ``` - The image supplies only the `afd-plugin` install. The recipe script is - mounted fresh from local disk on every run (step 4b), so editing or adding - a recipe never requires a rebuild. + The image supplies only the `afd-plugin` install. The recipe script and + the shared launcher are mounted fresh from local disk on every run (step + 4b), so editing or adding a recipe never requires a rebuild. - An `hf-token-secret` Secret with a `token` key, needed when `MODEL_ID` is a HF Hub repo id (not needed when weights are already staged on the PVC): ```bash @@ -81,7 +81,7 @@ exist **before** any Pod is deployed -- see | Resource | Covers | |---|---| -| [resolve-recipe.md](resources/resolve-recipe.md) | Step 1 (the only step that parses the recipe): read the recipe's serve-block fields and model id; read its placement (or lack of one) from its header comment -- never ask; record `GPU_COUNT` per pod. | +| [resolve-recipe.md](resources/resolve-recipe.md) | Step 1 (the only step that parses the recipe): model id, the `LAYOUT` plan (never ask), `GPU_COUNT` per pod, and the launcher's ports. | | [deploy.md](resources/deploy.md) | Steps 2-4f: take step 1's `GPU_COUNT`, confirm, PVC, ConfigMap, fsGroup, Services (step 4d -- which of the four, see [services.md](resources/services.md)), pod(s) via `templates/pod.yaml`, wait for Running and then for each pod's `Ready` condition. | | [report-and-teardown.md](resources/report-and-teardown.md) | Step 5: report the endpoint/model/nodes and ask when to delete the running pods. Step 6: teardown when the caller says so (pods/Services only -- never the PVC unless asked). | diff --git a/.agents/skills/deploy-afd-k8s/resources/deploy.md b/.agents/skills/deploy-afd-k8s/resources/deploy.md index 4bffc08c5..0a971ec66 100644 --- a/.agents/skills/deploy-afd-k8s/resources/deploy.md +++ b/.agents/skills/deploy-afd-k8s/resources/deploy.md @@ -3,8 +3,14 @@ ## 2. GPU_COUNT Use the `GPU_COUNT` recorded by [resolve-recipe.md](resolve-recipe.md) -- -one value for a plain recipe, one per plan entry. Don't re-read the script -here; all recipe parsing lives in step 1. +`a + f` per `LAYOUT` entry. Don't re-read the script here; all recipe +parsing lives in step 1: + +```bash +LAYOUT=${LAYOUT:-$(grep -oE 'LAYOUT:-[0-9AF,]+' "$RECIPE_SCRIPT_PATH" | cut -d- -f2)} +IFS=, read -ra ENTRIES <<<"$LAYOUT" +for e in "${ENTRIES[@]}"; do f=${e#*A}; echo "$e -> $(( ${e%%A*} + ${f%F} )) GPUs"; done +``` ## 3. Confirm before touching the cluster @@ -19,7 +25,7 @@ existing Pod(s)** with the same name(s) about to be deployed. AFD_PLUGIN_IMAGE= MODEL_ID= PVC_NAME= -CLIENT_PORT= # from the script's --port; default 18305 +CLIENT_PORT= # the launcher's ATTN_PORT (18000), see resolve-recipe.md RECIPE_SCRIPT_PATH= ``` @@ -72,11 +78,14 @@ rather than re-downloading. ## 4b. Recipe ConfigMap Lets the pod run any local recipe -- edited, uncommitted, or new -- without -rebuilding the image. Run from wherever `RECIPE_SCRIPT_PATH` resolves: +rebuilding the image. The recipe sources the shared launcher, so ship both; +`templates/pod.yaml` points the recipe at `/recipe/afd_colocation.sh` +(`AFD_RECIPE_LIB`). Run from the repository root: ```bash kubectl create configmap afd-recipe-script \ --from-file=recipe.sh="${RECIPE_SCRIPT_PATH}" \ + --from-file=afd_colocation.sh=recipe/gpu/common/afd_colocation.sh \ --dry-run=client -o yaml | kubectl apply -f - ``` @@ -108,17 +117,19 @@ recipe) against the one shared `templates/pod.yaml`: ```bash deploy_pod() { local pod_name="$1" gpu="$2" attn_role="$3" ffn_role="$4" extra_env_pairs="$5" - # extra_env_pairs: space-separated VAR=value tokens straight off this - # plan entry's header line (including POD=... itself), empty for a plain - # recipe. Build the spliced env-entries block for templates/pod.yaml. - # Leading (not trailing) newlines keep the `# ${TEMPLATE_EXTRA_ENV}` - # placeholder line a harmless comment when this is empty, and valid - # sibling list items when it isn't. - local extra_env="" - for pair in $extra_env_pairs; do + # extra_env_pairs: one VAR=value per line -- this plan entry's overrides + # (including POD=... itself) plus the caller's settings -- empty for a plain + # recipe. Values may hold spaces and JSON (VLLM_ARGS, DECODE_BENCH); they + # become single-quoted YAML strings. Leading (not trailing) newlines keep + # the `# ${TEMPLATE_EXTRA_ENV}` placeholder line a harmless comment when + # this is empty, and valid sibling list items when it isn't. + local extra_env="" pair val q="'" + while IFS= read -r pair; do + [ -n "$pair" ] || continue + val=${pair#*=} extra_env="$extra_env - - {name: ${pair%%=*}, value: \"${pair#*=}\"}" - done + - {name: ${pair%%=*}, value: '${val//$q/$q$q}'}" + done <<<"$extra_env_pairs" kubectl delete pod "$pod_name" --ignore-not-found @@ -131,26 +142,30 @@ deploy_pod() { } ``` -Plain recipe, one pod, both roles head, no overrides: +One-entry `LAYOUT` (e.g. the default `2A2F`), one pod, both roles head, +with or without the caller's settings: ```bash deploy_pod vllm-pod "$GPU_COUNT" head head "" +deploy_pod vllm-pod 4 head head $'VLLM_ARGS=--max-num-seqs 32' ``` -Placement-driven recipe, one call per entry parsed in -[resolve-recipe.md](resolve-recipe.md), e.g. the qwen3.5 3-pod plan: +Several entries, one call per entry parsed in +[resolve-recipe.md](resolve-recipe.md), e.g. `LAYOUT=2A0F,0A2F` (2 GPUs per +pod) with the decode-only benchmark, `LAYOUT=4A1F,4A1F` (5 GPUs per pod), +or the Qwen3.5 recipe's default `2A0F,2A0F,0A2F`: ```bash -deploy_pod vllm-pod-attention-0 2 head none "POD=ATTENTION_0" -deploy_pod vllm-pod-attention-1 2 worker none "POD=ATTENTION_1 ATTENTION_HEADLESS=1 ATTENTION_DP_START_RANK=2" -deploy_pod vllm-pod-ffn-0 2 none head "POD=FFN_0" -``` +SETTINGS=$'LAYOUT=2A0F,0A2F\nDECODE_BENCH={}\nAFD_REPO=/opt/afd-plugin' +deploy_pod vllm-pod-afd-0 2 head none $'POD=AFD_0\n'"$SETTINGS" +deploy_pod vllm-pod-afd-1 2 none head $'POD=AFD_1\n'"$SETTINGS" -`POD=AFD_` recipe, e.g. the DeepEP M2N 2 x 4A1F plan (5 GPUs per pod): +deploy_pod vllm-pod-afd-0 5 head head $'POD=AFD_0\nLAYOUT=4A1F,4A1F' +deploy_pod vllm-pod-afd-1 5 worker worker $'POD=AFD_1\nLAYOUT=4A1F,4A1F' -```bash -deploy_pod vllm-pod-afd-0 5 head head "POD=AFD_0" -deploy_pod vllm-pod-afd-1 5 worker worker "POD=AFD_1" +deploy_pod vllm-pod-afd-0 2 head none 'POD=AFD_0' +deploy_pod vllm-pod-afd-1 2 worker none 'POD=AFD_1' +deploy_pod vllm-pod-afd-2 2 none head 'POD=AFD_2' ``` ## 4f. Wait for Running, then for each pod's Ready condition @@ -198,6 +213,6 @@ done Only report the endpoint ready once every pod in the plan is `Ready` -- one pod becoming ready doesn't imply the others did. -Startup markers are keyed on the fixed log names every colocation recipe -uses today -- see `templates/pod.yaml`'s `ready_marker()` for the exact -lines. A recipe using different log names needs that template updated. +Startup markers are keyed on the launcher's fixed log names (`attn.log`, +`ffn.log`) -- see `templates/pod.yaml`'s `ready_marker()` for the exact +lines. diff --git a/.agents/skills/deploy-afd-k8s/resources/resolve-recipe.md b/.agents/skills/deploy-afd-k8s/resources/resolve-recipe.md index 1a7f5f0bb..3c244867e 100644 --- a/.agents/skills/deploy-afd-k8s/resources/resolve-recipe.md +++ b/.agents/skills/deploy-afd-k8s/resources/resolve-recipe.md @@ -6,130 +6,74 @@ under `recipe/gpu/P2pNcclAFDConnector/**/prefill_decode_colocation/` or only; prefill/decode disaggregation recipes are out of scope -- see `SKILL.md` Scope). -Read the script and note, per `vllm serve` block (each backgrounded process -ending `> .log 2>&1 &`): +Every such recipe is a short script on the shared launcher +`recipe/gpu/common/afd_colocation.sh` (see `recipe/gpu/common/README.md`): +the recipe sets `MODEL_PATH`, `AFD_CONNECTOR`, `LAYOUT`, `MODE` and its +`SERVE_ARGS`; the launcher derives devices, DP flags and ports. Never parse +the `vllm serve` command out of the script -- print it: -- `CUDA_VISIBLE_DEVICES`, `--data-parallel-size`, `--tensor-parallel-size` -- the `"afd": {"role": ...}` block in `--additional-config` -- eager vs graph, and every other flag (`--enable-expert-parallel`, - `--max-num-seqs`, `--max-num-batched-tokens`, `--max-model-len`, - `--trust-remote-code`, host/port) - -Record `--max-num-batched-tokens` and report it to the caller later -- it -caps how much prefill work a single step can absorb. - -Resolve `MODEL_ID`: read the script's `MODEL_PATH` default. It is either a -HF Hub id (map the recipe's model directory name to one, e.g. -`deepseek_v2_lite` -> `deepseek-ai/DeepSeek-V2-Lite`; ask if the mapping -isn't obvious) or a **placeholder** path of the form -`/path/model_weights/` -- every colocation recipe uses exactly this -literal `/path/` stand-in today, regardless of what's actually staged on any -given cluster's PVC. Never pass this through verbatim: it names only the -model directory (``), not a real in-container path. Derive the real -`MODEL_ID` as `/`, where `` is the PVC mount -point fixed by `templates/pod.yaml` (`/models`) -- e.g. -`/path/model_weights/Qwen3.5-122B-A10B-FP8` resolves to -`/models/Qwen3.5-122B-A10B-FP8`. Before deploying, confirm that directory -actually exists on the target `PVC_NAME` (if it already exists, `kubectl -exec` into any pod already mounting it, e.g. `ls /models`, or spin up a -short-lived busybox pod mounting the PVC) -- don't rely on the recipe's -claim; a mismatch here fails fast inside the pod with a confusing -HF-repo-id validation error instead of a clear message. - -Tell the caller which form it is, and the resolved value if derived. - -Set `CLIENT_PORT` to the `--port` value on the attention (or sole, for a -single-pod recipe) `vllm serve` block -- `templates/pod.yaml` rebinds this -port off loopback and `vllm-service` (step 4d, see -[services.md](services.md)) exposes it, whatever value is actually there. - -## Read the placement plan -- never ask - -Look at the top-of-file comment block (above the first `vllm serve`/env-var -line). If it says nothing about per-pod placement, this is a plain -single-pod recipe: one pod, `POD` unset, both roles run unconditionally -- -continue to [deploy.md](deploy.md) with no plan. - -If it does describe a placement across multiple pods, read and interpret -that description directly. - -Example of a placement description: -``` -# For pod 1: POD=ATTENTION_0 -# For pod 2: POD=ATTENTION_1, ATTENTION_HEADLESS=1, ATTENTION_DP_START_RANK=2 -# For pod 3: POD=FFN_0 -``` - -Or -``` -"runs -on 3 pods: the first two run ATTENTION_0 and ATTENTION_1 -- the second one -headless, starting at data-parallel rank 2 -- and the third runs FFN_0" +```bash +RECIPE_LIB_PATH=recipe/gpu/common/afd_colocation.sh +AFD_DRY_RUN=1 LAYOUT= POD=AFD_ bash "$RECIPE_SCRIPT_PATH" ``` -Whatever the phrasing, read it and extract, per pod, in the order the -comment presents them: - -- that pod's `POD=` identity -- any further container-env overrides for that pod only (e.g. - `ATTENTION_HEADLESS=1`, `ATTENTION_DP_START_RANK=2`) - -The recipe already states ranks/ordering per pod -- there is nothing to -derive or ask about, regardless of how it's phrased. (A quick -`grep -E '^# For pod [0-9]+: '` is a fine way to spot today's convention at -a glance, but its absence doesn't mean "no plan" -- check the comment block -itself before concluding this is a plain single-pod recipe.) - -For each entry, derive (used by [deploy.md](deploy.md) and -[services.md](services.md)): - -- **k8s Pod name**: `vllm-pod-$(echo "$POD" | tr 'A-Z_' 'a-z-')`, e.g. - `vllm-pod-attention-0`. -- **`afd-attn-node-role`**: `none` unless `POD` starts with `ATTENTION_`, in - which case `worker` if that line's `ATTENTION_HEADLESS` override is `1`, - else `head`. -- **`afd-ffn-node-role`**: symmetric, `none` unless `POD` starts with `FFN_`. -- **`GPU_COUNT`**: the number of devices in the literal - `CUDA_VISIBLE_DEVICES=` of the `vllm serve` block this pod runs -- the one - inside the recipe's branch that tests `$POD` for this pod's role - (`ATTENTION` or `FFN`). Every pod sharing a role shares that literal, so - read it off the script; don't derive it from ranks or DP sizes. - -### Pods that run both roles: `POD=AFD_` - -DeepEP M2N multi-pod recipes (e.g. -`DeepEPM2NAFDConnector/qwen3_235b_a22b_fp8/prefill_decode_colocation/multipod_4a1f_4a1f_graph.sh`) -list `POD=AFD_0`, `POD=AFD_1`, ... with no other per-pod overrides. Every -such pod runs one FFN and one Attention server and derives its DP flags -(headless, start rank) from its own index, so: - -- **`afd-attn-node-role`** and **`afd-ffn-node-role`**: both `head` for - `AFD_0`, both `worker` for every other `AFD_`. -- The pod shape comes from the script's `NUM_PODS`, `ATTN_PER_POD` and - `FFN_PER_POD` defaults (`NUM_PODS` must equal the number of entries). If - the user overrides them, pass the same values to every pod as container - env. -- **`GPU_COUNT`**: `ATTN_PER_POD + FFN_PER_POD`, the same for every pod. - These recipes build `CUDA_VISIBLE_DEVICES` with `$(seq ...)`, so the - literal-device rules above and below don't apply: - - ```bash - default() { grep -oE "$1:-[0-9]+" "$RECIPE_SCRIPT_PATH" | grep -oE '[0-9]+$'; } - GPU_COUNT=$(( ${ATTN_PER_POD:-$(default ATTN_PER_POD)} + ${FFN_PER_POD:-$(default FFN_PER_POD)} )) - ``` -- Ports are literals in the script, not `*_PORT:-N` defaults: the AFD - rendezvous port is the `"port": N` of the `afd` JSON, and the FFN and - Attention DP RPC ports are the last argument of the `role_dp "$F" ...` and - `role_dp "$A" ...` lines. - -## GPU_COUNT for a plain single-pod recipe - -The union of every literal `CUDA_VISIBLE_DEVICES=` in the script: +prints, per role this pod runs, the `CUDA_VISIBLE_DEVICES=... vllm serve ...` +line. Record `--max-num-batched-tokens` and report it to the caller later -- +it caps how much prefill work a single step can absorb. + +## Model + +Resolve `MODEL_ID` from the script's `MODEL_PATH` default. It is either a +HF Hub id or a **placeholder** path of the form `/path/model_weights/` +-- every colocation recipe uses exactly this literal `/path/` stand-in. +Never pass this through verbatim: derive the real `MODEL_ID` as +`/models/` (`/models` is the PVC mount point fixed by +`templates/pod.yaml`), e.g. `/path/model_weights/Qwen3.5-122B-A10B-FP8` +resolves to `/models/Qwen3.5-122B-A10B-FP8`. Before deploying, confirm that +directory exists on the target `PVC_NAME` (`kubectl exec` into any pod +already mounting it, e.g. `ls /models`, or a short-lived busybox pod +mounting the PVC); a mismatch fails inside the pod with a confusing +HF-repo-id validation error. Tell the caller which form it is, and the +resolved value if derived. + +## Placement plan -- never ask + +The placement is `LAYOUT`: one pod per comma-separated `AF` entry, in +order, pod `k` running `POD=AFD_`, every pod with the same `LAYOUT`. Take +`LAYOUT` from the caller's settings, else from the script's +`LAYOUT=${LAYOUT:-...}` default; never ask. For entry `k` with `a` +Attention and `f` FFN GPUs: + +- **k8s Pod name**: `vllm-pod-afd-`. A one-entry `LAYOUT` (e.g. `2A2F`) + is a plain single pod: `vllm-pod`, no `POD` needed. +- **`GPU_COUNT`**: `a + f` (FFN on the pod's first `f` GPUs). +- **`afd-attn-node-role`**: `none` when `a` is 0, `head` on the first entry + with `a > 0`, `worker` on later ones. **`afd-ffn-node-role`**: the same + with `f`. For example `2A0F,0A2F` is `head none` and `none head`, + `1A1F,1A1F` is `head head` and `worker worker`, and `2A0F,2A0F,0A2F` is + `head none`, `worker none` and `none head`. +- **Container env**: `POD=AFD_` and `LAYOUT=`, plus every setting + the caller gives (`MODEL_PATH`, `ATTN_TP`, `FFN_TP`, `VLLM_ARGS`, + `ATTN_ARGS`, `FFN_ARGS`, `DECODE_BENCH`, `M2N_EXTRA_CONFIG`, NCCL + variables), the same on every pod. Leave `HEAD_HOST`, `ATTN_HEAD_HOST` and + `FFN_HEAD_HOST` unset: the defaults name the Services from + [services.md](services.md). With `DECODE_BENCH`, also set + `AFD_REPO=/opt/afd-plugin` (the repository in `docker/Dockerfile.ci` + images), since the connector lives in `tools/benchmarks` and the recipe + runs from `/work`. +- Whether the connector accepts the placement is checked at startup, not + here; see the connector guide's supported layouts. + +## Ports + +The launcher fixes them on one line (`ATTN_PORT=18000 FFN_PORT=18001 +AFD_PORT=6239 ATTN_DP_RPC_PORT=13345 FFN_DP_RPC_PORT=13355`); read them from +it rather than assuming: ```bash -GPU_COUNT="$(grep -oE 'CUDA_VISIBLE_DEVICES=[0-9,]+' "$RECIPE_SCRIPT_PATH" \ - | cut -d= -f2 | tr ',' '\n' | sort -u | wc -l)" +lib_port() { grep -oE "\b$1=[0-9]+" "$RECIPE_LIB_PATH" | head -1 | cut -d= -f2; } +CLIENT_PORT=$(lib_port ATTN_PORT) # the Attention API, exposed by vllm-service ``` This step is the only place that parses the recipe: -[deploy.md](deploy.md) uses the `GPU_COUNT` (and plan) recorded here as-is. +[deploy.md](deploy.md) uses the `GPU_COUNT` and plan recorded here as-is. diff --git a/.agents/skills/deploy-afd-k8s/resources/services.md b/.agents/skills/deploy-afd-k8s/resources/services.md index 761d23e65..5f5302dfc 100644 --- a/.agents/skills/deploy-afd-k8s/resources/services.md +++ b/.agents/skills/deploy-afd-k8s/resources/services.md @@ -24,32 +24,28 @@ CLIENT_PORT="$CLIENT_PORT" envsubst '${CLIENT_PORT}' \ Internal only, carries AFD rendezvous traffic to whichever pod holds `afd-ffn-node-role: head`. Must exist before any multi-pod deploy, since the -recipe's own `AFD_CONNECTOR_HOST` default now points at this name directly. -Traffic to a headless Service goes straight to the pod IP, and there's one -rendezvous port, so a single static port entry is enough -- read -`AFD_CONNECTOR_PORT` off the recipe's own shell default: +launcher's default rendezvous host points at this name directly. Traffic to +a headless Service goes straight to the pod IP, and there's one rendezvous +port, so a single static port entry is enough (`lib_port` from +[resolve-recipe.md](resolve-recipe.md#ports)): ```bash -AFD_CONNECTOR_PORT="$(grep -oE 'AFD_CONNECTOR_PORT:-[0-9]+' "$RECIPE_SCRIPT_PATH" | grep -oE '[0-9]+')" -AFD_CONNECTOR_PORT="$AFD_CONNECTOR_PORT" envsubst '${AFD_CONNECTOR_PORT}' \ +AFD_CONNECTOR_PORT="$(lib_port AFD_PORT)" envsubst '${AFD_CONNECTOR_PORT}' \ < templates/service-ffn-p2p.yaml | kubectl apply -f - ``` ## `vllm-attn-dp-service` / `vllm-ffn-dp-service` -- iff that role is split -Only when **more than one** plan entry's `POD` carries the same role (i.e. -that role is split across pods, not just placed in its own single pod). -Every `POD=AFD_` entry carries both roles, so a plan of two or more -such entries needs both Services; read their ports (and the rendezvous port) -as [resolve-recipe.md](resolve-recipe.md) describes for those recipes rather -than with the `*_PORT:-N` greps below. -Carries that role's DP-RPC coordination traffic to its head pod. One per -split role, generic over role via `templates/service-dp-coord.yaml`: +Only when **more than one** `LAYOUT` entry has GPUs of that role (both +roles in `1A1F,1A1F`, Attention in `2A0F,2A0F,0A2F`, neither in +`2A0F,0A2F`). Carries that role's DP-RPC coordination traffic to its head +pod. One per split role, generic over role via +`templates/service-dp-coord.yaml`: ```bash -TEMPLATE_ROLE=attn -TEMPLATE_PORT="$(grep -oE 'ATTENTION_DP_RPC_PORT:-[0-9]+' "$RECIPE_SCRIPT_PATH" | grep -oE '[0-9]+')" -TEMPLATE_ROLE="$TEMPLATE_ROLE" TEMPLATE_PORT="$TEMPLATE_PORT" \ +TEMPLATE_ROLE=attn TEMPLATE_PORT="$(lib_port ATTN_DP_RPC_PORT)" \ + envsubst '${TEMPLATE_ROLE} ${TEMPLATE_PORT}' < templates/service-dp-coord.yaml | kubectl apply -f - +TEMPLATE_ROLE=ffn TEMPLATE_PORT="$(lib_port FFN_DP_RPC_PORT)" \ envsubst '${TEMPLATE_ROLE} ${TEMPLATE_PORT}' < templates/service-dp-coord.yaml | kubectl apply -f - ``` diff --git a/.agents/skills/deploy-afd-k8s/templates/pod.yaml b/.agents/skills/deploy-afd-k8s/templates/pod.yaml index 561f16f18..f43a0694e 100644 --- a/.agents/skills/deploy-afd-k8s/templates/pod.yaml +++ b/.agents/skills/deploy-afd-k8s/templates/pod.yaml @@ -33,33 +33,19 @@ spec: - -c - | set -euo pipefail - RECIPE_SCRIPT=/recipe/recipe.sh - mkdir -p /work/bin - # Recipes only ever call `uv run `; drop the `run` and exec - # the rest directly instead of invoking uv at all. - printf '#!/bin/bash\nshift\nexec "$@"\n' > /work/bin/uv - chmod +x /work/bin/uv - export PATH="/work/bin:$PATH" - - echo "=== launching $(basename "$RECIPE_SCRIPT") (POD=${POD:-} attn=${AFD_ATTN_NODE_ROLE} ffn=${AFD_FFN_NODE_ROLE}) ===" - - # Rebind the client-facing port off loopback: every `--host - # 127.0.0.1` on the branch is an API-server flag (the recipe binds - # :${TEMPLATE_CLIENT_PORT} on 127.0.0.1), but the Service needs - # 0.0.0.0. - RECIPE_BASENAME="$(basename "$RECIPE_SCRIPT" .sh)" - PATCHED_SCRIPT="/work/${RECIPE_BASENAME}.patched.sh" - sed 's/--host 127\.0\.0\.1/--host 0.0.0.0/' "$RECIPE_SCRIPT" > "$PATCHED_SCRIPT" + echo "=== launching recipe.sh (POD=${POD:-} attn=${AFD_ATTN_NODE_ROLE} ffn=${AFD_FFN_NODE_ROLE}) ===" + # The recipe sources the launcher from AFD_RECIPE_LIB and binds its + # API server to SERVE_HOST (0.0.0.0, so the Service can reach it). cd /work - bash "$PATCHED_SCRIPT" & + bash /recipe/recipe.sh & RECIPE_PID=$! echo "$RECIPE_PID" > /work/recipe.pid # Which log(s) this pod will actually produce, and this pod's own # readiness marker per log -- both keyed on this pod's node-role # labels, not re-derived from $POD. Fixed to the attn.log/ffn.log - # names every colocation recipe uses today. + # names of the colocation launcher. LOGS="" [ "$AFD_ATTN_NODE_ROLE" != "none" ] && LOGS="$LOGS attn.log" [ "$AFD_FFN_NODE_ROLE" != "none" ] && LOGS="$LOGS ffn.log" @@ -148,6 +134,10 @@ spec: value: "${TEMPLATE_ATTN_ROLE}" - name: AFD_FFN_NODE_ROLE value: "${TEMPLATE_FFN_ROLE}" + - name: AFD_RECIPE_LIB + value: "/recipe/afd_colocation.sh" + - name: SERVE_HOST + value: "0.0.0.0" # ${TEMPLATE_EXTRA_ENV} - name: HF_TOKEN valueFrom: diff --git a/.agents/skills/review-pr/references/features/disaggregation-topologies.md b/.agents/skills/review-pr/references/features/disaggregation-topologies.md index b5d920306..c23a93b38 100644 --- a/.agents/skills/review-pr/references/features/disaggregation-topologies.md +++ b/.agents/skills/review-pr/references/features/disaggregation-topologies.md @@ -14,10 +14,11 @@ recipes under `recipe/gpu/` and `recipe/npu/`. | Topology | Where it lives | | --- | --- | | Baseline (native DP/TP, no AFD) | `baseline-graph` E2E scenarios; native DP4/TP1/EP4 | -| CUDA NCCL P2P, colocation | `recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/` (2A2F and 4A4F, eager/graph, DBO variants, plus `multipod_2a_2f_graph_dbo_dp1tp2.sh`) | +| CUDA NCCL P2P, colocation | `recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/` (`dbo.sh`, `eager_dbo.sh`; any `LAYOUT`, `ATTN_TP`, `FFN_TP`, e.g. 2A2F, 4A4F, `2A0F,0A2F`) | | CUDA NCCL P2P, disaggregation | `recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_disaggregation/` (2P1A1F eager/graph DBO) | -| CUDA NCCL P2P, Qwen3.5 multi-pod | `recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/multipod_2a_2a_2f_graph.sh` | -| CUDA DeepEP M2N (routed experts) | `recipe/gpu/DeepEPM2NAFDConnector/{deepseek_v2_lite,qwen3_30b_a3b,qwen3_235b_a22b_fp8}/` (2A2F, and 4A2F for the 235B FP8 case) | +| CUDA NCCL P2P, Qwen3.5 multi-pod | `recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/graph.sh` (default `LAYOUT=2A0F,2A0F,0A2F`) | +| CUDA DeepEP M2N (routed experts) | `recipe/gpu/DeepEPM2NAFDConnector/{deepseek_v2_lite,qwen3_30b_a3b,qwen3_235b_a22b_fp8}/` (graph/eager/dbo, any `LAYOUT`) | +| GPU colocation launcher | `recipe/gpu/common/afd_colocation.sh`, shared by every GPU colocation recipe | | NPU CAM P2P disaggregation | `recipe/npu/CAMP2pAFDConnector/deepseek_v3_2/` (attention/FFN/baseline scripts) | | NPU async CAM | `recipe/npu/CAMAsyncAFDConnector/deepseek_v3_2/v0_26_accuracy/` and `deepseek_v4_flash/` (DSpark A/F) | | Gate matrices | `tests/e2e/models/*/` scenario lists; `docs/design/module/e2e_testing.md` | diff --git a/.agents/skills/run-vllm-bench-k8s/SKILL.md b/.agents/skills/run-vllm-bench-k8s/SKILL.md index e418a05fa..d0a3a66fc 100644 --- a/.agents/skills/run-vllm-bench-k8s/SKILL.md +++ b/.agents/skills/run-vllm-bench-k8s/SKILL.md @@ -16,7 +16,7 @@ One invocation = one run = one `RUN_ID` = one report. - GPU colocation recipes only -- whatever `deploy-afd-k8s` supports. - Not for local benchmarking (run `request_generator.sh` directly against - `decode_bench_server.sh` for that) or correctness E2E (`run-e2e`). + a local recipe for that) or correctness E2E (`run-e2e`). - Requires everything `deploy-afd-k8s` requires. No extra image: the bench pod runs `AFD_PLUGIN_IMAGE`, which already carries the repo at `/opt/afd-plugin` and vLLM's `bench` CLI. @@ -42,10 +42,10 @@ folder as the list of possible recipes. Follow **`deploy-afd-k8s`** with `RECIPE_SCRIPT_PATH`. It returns `MODEL_ID`, `PVC_NAME`, `GPU_COUNT`, the serving node, the recipe's -`--max-num-batched-tokens`, and an endpoint at `http://vllm-service:18305`. +`--max-num-batched-tokens`, and an endpoint at `http://vllm-service:${CLIENT_PORT}` (18000). `vllm bench serve` is pointed at that endpoint with `HOST=vllm-service` -`PORT=18305`, and `MODEL_PATH=${MODEL_ID}` -- it must name exactly what the +`PORT=18000` (the launcher's Attention port), and `MODEL_PATH=${MODEL_ID}` -- it must name exactly what the pod serves, since the model id is sent in each request and used to load the tokenizer. @@ -195,7 +195,7 @@ spec: conc="${point##*:}" echo "=== sweep point rate=${rate} concurrency=${conc} ===" MODEL_PATH="${MODEL_ID}" \ - HOST=vllm-service PORT=18305 \ + HOST=vllm-service PORT=18000 \ DATASET_NAME="${DATASET_NAME}" \ INPUT_LEN="${INPUT_LEN}" OUTPUT_LEN="${OUTPUT_LEN}" \ NUM_PROMPTS="${NUM_PROMPTS}" \ diff --git a/docs/gpu/DEEPEP_M2N_CONNECTOR_USER_GUIDE.md b/docs/gpu/DEEPEP_M2N_CONNECTOR_USER_GUIDE.md index cc9aad6c3..76a2333bb 100644 --- a/docs/gpu/DEEPEP_M2N_CONNECTOR_USER_GUIDE.md +++ b/docs/gpu/DEEPEP_M2N_CONNECTOR_USER_GUIDE.md @@ -291,10 +291,9 @@ the headless pods. The FFN DP rank is the FFN's EP rank, so it decides which experts a pod loads; `--enable-ep-weight-filter` keeps each FFN from reading the other experts' weights. -The recipe -[`multipod_4a1f_4a1f_graph.sh`](../../recipe/gpu/DeepEPM2NAFDConnector/qwen3_235b_a22b_fp8/README.md#multi-pod-2-x-4a1f) -derives all of these flags for symmetric placements `N x AF` from the -pod index (`POD=AFD_`). +The [DeepEP M2N recipes](../../recipe/gpu/DeepEPM2NAFDConnector/README.md#layout) +derive all of these flags from a `LAYOUT` such as `2A0F,0A2F` or `4A1F,4A1F` +and the pod index (`POD=AFD_`). ## Minimal launch shape diff --git a/recipe/README.md b/recipe/README.md index 364b3be37..c98bea1cf 100644 --- a/recipe/README.md +++ b/recipe/README.md @@ -23,8 +23,9 @@ Recipes are organized by hardware backend, connector, and model: | --- | --- | --- | --- | --- | --- | | GPU | `P2pNcclAFDConnector` | DeepSeek-V2-Lite | Decode | Deployment examples; see the linked README for validation scope | [Launch examples](gpu/P2pNcclAFDConnector/deepseek_v2_lite/README.md) | | GPU (SM90+) | `DeepEPM2NAFDConnector` | DeepSeek-V2-Lite | Decode | Deployment examples matching the checked-in E2E layouts; hardware qualification is separate | [Launch examples](gpu/DeepEPM2NAFDConnector/deepseek_v2_lite/README.md) | -| GPU (SM90+) | `DeepEPM2NAFDConnector` | Qwen3-30B-A3B | Decode | Same 2A2F layouts as the Qwen3 MoE DeepEP M2N E2E cases | [Launch examples](gpu/DeepEPM2NAFDConnector/qwen3_30b_a3b/README.md) | -| GPU (SM90+) | `DeepEPM2NAFDConnector` | Qwen3-235B-A22B-FP8 | Decode | 4A2F FP8-payload configuration run on H200 | [Launch examples](gpu/DeepEPM2NAFDConnector/qwen3_235b_a22b_fp8/README.md) | +| GPU (SM90+ for `DeepEPM2NAFDConnector`) | `DeepEPM2NAFDConnector` or `P2pNcclAFDConnector` | Qwen3-30B-A3B | Prefill / decode; decode-only benchmark | Graph, eager and DBO scripts for any `LAYOUT` (default 2A2F; multi-pod such as `2A0F,0A2F` or `1A1F,1A1F`), the layouts of the Qwen3 MoE DeepEP M2N E2E cases | [Graph, eager, DBO](gpu/DeepEPM2NAFDConnector/qwen3_30b_a3b/README.md) | +| GPU (SM90+ for `DeepEPM2NAFDConnector`) | `DeepEPM2NAFDConnector` or `P2pNcclAFDConnector` | Qwen3-235B-A22B-FP8 | Prefill / decode; decode-only benchmark | FP8-payload graph, eager and DBO scripts for any `LAYOUT` (2A2F, 4A2F, 2 x 4A1F, ...); H200-class FFN GPUs | [Graph, eager, DBO](gpu/DeepEPM2NAFDConnector/qwen3_235b_a22b_fp8/README.md) | +| GPU | `P2pNcclAFDConnector` | Qwen3.5-122B-A10B-FP8 | Decode | Graph script for any `LAYOUT` (default 3 pods, `2A0F,2A0F,0A2F`) | [Graph](gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/graph.sh) | | Ascend NPU | `CAMP2pAFDConnector` | DeepSeek-V3.2 | Decode | Historical v0.19.1rc1 recipe; not qualified for the 0.32 target | [Synchronous decode](npu/CAMP2pAFDConnector/deepseek_v3_2/README.md) | | Ascend NPU | `CAMAsyncAFDConnector` | DeepSeek-V3.2 | Prefill / decode | Historical v0.19.1rc1/v0.26 recipes; standard V3.2 is unsupported by the current Async CAM path | [Async CAM](npu/CAMAsyncAFDConnector/deepseek_v3_2/README.md) | | Ascend NPU | `CAMAsyncAFDConnector` | DeepSeek-V4-Flash W4A8/W8A8 DSpark | Separate P / decode, or colocated | v0.32; eager A TP1, F layered eager/FULL. W4A8 PD: native full-draft P TP4, A DP2, F EP4. W8A8 A/F: A DP2, F EP8; PD not qualified | [DSpark A/F](npu/CAMAsyncAFDConnector/deepseek_v4_flash/README.md) | @@ -41,3 +42,8 @@ directory should contain a `README.md` describing prerequisites, topology, launch commands, validation steps, and limitations. Keep model-specific launch scripts, configuration files, and result images in the same directory so the recipe can be moved or linked as one unit. + +GPU `prefill_decode_colocation` recipes are short scripts on the shared +[colocation launcher](gpu/common/README.md): a recipe sets its model, +connector, default `LAYOUT` and `vllm serve` flags, and the launcher handles +placement across GPUs and pods. Start a new one from an existing recipe. diff --git a/recipe/gpu/DeepEPM2NAFDConnector/README.md b/recipe/gpu/DeepEPM2NAFDConnector/README.md index bfbb024dd..01b411de6 100644 --- a/recipe/gpu/DeepEPM2NAFDConnector/README.md +++ b/recipe/gpu/DeepEPM2NAFDConnector/README.md @@ -2,34 +2,40 @@ Launch scripts for running Attention-FFN Disaggregation (AFD) with `DeepEPM2NAFDConnector`, the expert-routed M2N connector for Hopper (SM90) and -newer GPUs. See +newer GPUs, with vLLM `v0.32.0`. See [`docs/gpu/DEEPEP_M2N_CONNECTOR_USER_GUIDE.md`](../../../docs/gpu/DEEPEP_M2N_CONNECTOR_USER_GUIDE.md) for installation, the configuration contract, the required vLLM settings, and -supported layouts. +the placements the connector accepts. -## Directory layout +This README covers what every recipe here shares; each model's `README.md` +lists only its defaults and model-specific notes. + +## Recipes + +Every model directory holds the same three scripts in +`prefill_decode_colocation/`: + +| Script | Mode | +| --- | --- | +| `graph.sh` | `FULL_DECODE_ONLY` CUDA graphs; prefill runs eager | +| `eager.sh` | Eager (`--enforce-eager`) | +| `dbo.sh` | `FULL_DECODE_ONLY` CUDA graphs and dual-batch overlap on both roles (`--enable-dbo`, decode/prefill token thresholds 2 and 8) | ```text . -├── deepseek_v2_lite/ -│ ├── README.md -│ └── prefill_decode_colocation/ # 2A2F: eager, graph, graph + DBO -├── qwen3_30b_a3b/ -│ ├── README.md -│ └── prefill_decode_colocation/ # 2A2F: eager, graph, graph + DBO, Attention TP2 -└── qwen3_235b_a22b_fp8/ - ├── README.md - └── prefill_decode_colocation/ # 4A2F FP8 payload: graph, graph + DBO; - # 2 pods x 4A1F (multipod_4a1f_4a1f_graph.sh) +├── deepseek_v2_lite/ # BF16 payload +├── qwen3_30b_a3b/ # BF16 payload +└── qwen3_235b_a22b_fp8/ # FP8 payload (fp8_block128_fp32) ``` -Each model directory holds its own scripts; open its `README.md` before -running anything in it. +Each script sets the model, its tuning and mode flags, and runs the shared +[colocation launcher](../common/README.md), which documents the layout, +every setting, and how to run a recipe directly or in a pod. ## Prerequisites -- SM90 or newer GPUs on one node: 4 for the DeepSeek-V2-Lite and - Qwen3-30B-A3B recipes, 6 for the Qwen3-235B-A22B-FP8 recipes. +- SM90 or newer GPUs. The default `2A2F` layout uses 4 GPUs on one host; see + the model README for memory needs. - vLLM `v0.32.0`, the plugin, afdEP-comm (the `afdep` extra) and DeepGEMM installed in one environment, with the elfutils headers available to DeepGEMM's JIT. See @@ -42,37 +48,99 @@ running anything in it. export AFDEP_NCCL_ROOT=$NCCL LD_LIBRARY_PATH=$NCCL/lib:${LD_LIBRARY_PATH:-} ``` +- Free ports `6239` (AFD rendezvous), `18000` (Attention API) and `18001` + (FFN, serves no requests); across pods also `13345` (Attention DP RPC) and + `13355` (FFN DP RPC). + The scripts call `vllm serve` directly rather than through `uv run`, so they use the environment that holds afdEP-comm and DeepGEMM. -## Running on a local host +## Layout + +`LAYOUT` (default `2A2F`, one pod of 4 GPUs) lists the pods, one `AF` +entry each; see [Layout](../common/README.md#layout) and +[Settings](../common/README.md#settings). Examples for this connector: +`4A2F` (one 6-GPU node), `2A0F,0A2F` and `1A1F,1A1F` (two pods), `4A1F,4A1F` +(two 5-GPU pods). `ATTN_TP=2` runs `2A2F`'s Attention as one DP1 x TP2 engine; +the FFN runs TP 1, and its DP rank (its EP rank) decides which experts it +loads. The connector checks the placement at startup and names the pods' +shapes when it rejects one; the +[supported layouts](../../../docs/gpu/DEEPEP_M2N_CONNECTOR_USER_GUIDE.md#supported-layouts) +table lists which placements it accepts (for example `4A0F,0A2F` is not, its +pods differ in size). -Run any script from the repository root, for example: +## Running ```bash export MODEL_PATH=/path/model_weights/Qwen3-30B-A3B -bash recipe/gpu/DeepEPM2NAFDConnector/qwen3_30b_a3b/prefill_decode_colocation/2a2f_graph_dp2tp1.sh +bash recipe/gpu/DeepEPM2NAFDConnector/qwen3_30b_a3b/prefill_decode_colocation/graph.sh ``` -Each script starts the FFN server and then the Attention server in the -background and writes `ffn.log` and `attn.log` into the current directory. -Wait for `Application startup complete` in both logs, then send traffic to the -Attention server on port `18000`. The AFD rendezvous uses port `6239`; the FFN -server listens on `18001` but serves no requests. - -The first start on a node compiles transport and DeepGEMM kernels, so it is -slower than later starts. +Send traffic to the Attention server on port `18000` once `attn.log` prints +`Application startup complete`. The first start on a host compiles transport +and DeepGEMM kernels, so it is slower than later starts. Raise +`CUDAGRAPH_CAPTURE_SIZES` together with `--max-num-seqs`, so decode batches +stay in a captured size. Across pods, pass the fabric's NCCL settings (for +example `NCCL_IB_HCA`) through the environment; see +[Running directly](../common/README.md#running-directly) for hosts without +Kubernetes. ## Deploying on Kubernetes -The single-node recipes have no placement header, so the -**`deploy-afd-k8s`** skill (`.agents/skills/deploy-afd-k8s/SKILL.md`) deploys -each one as a single pod. The image must contain afdEP-comm and DeepGEMM. - -`multipod_4a1f_4a1f_graph.sh` carries a `POD=AFD_` header, so the skill -deploys it as one pod per entry, each running both roles. Every pod gets the -same `NUM_PODS`, `ATTN_PER_POD` and `FFN_PER_POD` (2, 4 and 1 by default), -which select any symmetric layout `N x AF`; the FFN uses each pod's -first `f` GPUs. See -[Multi-pod deployment](../../../docs/gpu/DEEPEP_M2N_CONNECTOR_USER_GUIDE.md#multi-pod-deployment) -for the placements the connector accepts. +The **`deploy-afd-k8s`** skill (`.agents/skills/deploy-afd-k8s/SKILL.md`) +deploys a recipe as one pod per `LAYOUT` entry; see +[Running in a pod](../common/README.md#running-in-a-pod). The image must +contain afdEP-comm and DeepGEMM. + +## Checking a deployment + +```bash +curl -s http://127.0.0.1:18000/v1/completions \ + -H 'Content-Type: application/json' \ + -d '{"model":"'"$MODEL_PATH"'","prompt":"The capital of France is","max_tokens":12,"temperature":0}' +``` + +The answer should name Paris. For an accuracy check, run the matching E2E case +(`afd-deepep-m2n-*`); see +[`tests/e2e`](../../../tests/e2e/README.md#deepep-m2n-scenarios). + +## Decode-only benchmark + +`DECODE_BENCH` attaches the decode-only benchmark's KV connector +(`AFDDecodeBenchConnector`) to Attention and takes its +`kv_connector_extra_config` as JSON (`{}` is enough). The connector reports +every prompt token but the last as externally computed, so requests skip +prefill and decode at a context of the full prompt length over unwritten KV. +The connector lives in `tools/benchmarks`, which is not in the wheel: run from +the repository root or set `AFD_REPO`. + +```bash +export MODEL_PATH=/path/model_weights/Qwen3-30B-A3B +DECODE_BENCH='{}' \ + bash recipe/gpu/DeepEPM2NAFDConnector/qwen3_30b_a3b/prefill_decode_colocation/graph.sh + +INPUT_LEN=1024 OUTPUT_LEN=256 NUM_PROMPTS=2000 \ + MAX_CONCURRENCY=64 REQUEST_RATE=inf RESULT_DIR=/tmp/results/decode_bench \ + tools/benchmarks/request_generator.sh +``` + +Drive the Attention port (18000). The benchmark works in every layout and +mode; see [`tools/benchmarks/README.md`](../../../tools/benchmarks/README.md). + +## Configuration notes + +- Both roles set `compute_gate_on_attention: true` and run with + `--enable-expert-parallel` and `--enable-ep-weight-filter` (in each + recipe's `SERVE_ARGS`), so each FFN rank reads only its own experts. +- The FFN keeps vLLM's default `--moe-backend auto`, which the plugin pins to + `triton` so the expert weights stay in the layout the connector reads. +- The connector derives its dispatch buckets from the CUDA graph capture + sizes and compiles one transport kernel set per bucket at startup. The + dispatch capacity is the larger of `--max-num-batched-tokens` and the + largest capture size, so lowering `--max-num-batched-tokens` shrinks the + transport buffers. +- With DBO the connector uses two lanes; DBO must be on for both roles. +- `expert_num_sms` and `num_sms` split SMs between the DeepGEMM experts and + the transport kernels; pass them with `M2N_EXTRA_CONFIG`. See + [`connector_extra_config`](../../../docs/gpu/DEEPEP_M2N_CONNECTOR_USER_GUIDE.md#connector_extra_config). +- `FULL_DECODE_ONLY` is the only CUDA graph mode. diff --git a/recipe/gpu/DeepEPM2NAFDConnector/deepseek_v2_lite/README.md b/recipe/gpu/DeepEPM2NAFDConnector/deepseek_v2_lite/README.md index f2ff98ed8..4e014b0d9 100644 --- a/recipe/gpu/DeepEPM2NAFDConnector/deepseek_v2_lite/README.md +++ b/recipe/gpu/DeepEPM2NAFDConnector/deepseek_v2_lite/README.md @@ -1,46 +1,41 @@ # DeepSeek-V2-Lite with DeepEPM2NAFDConnector -Launch scripts for DeepSeek-V2-Lite (BF16) on `DeepEPM2NAFDConnector` with -vLLM `v0.32.0`. They use the same layouts as the DeepSeek-V2-Lite DeepEP M2N -E2E cases (`test_deepseek_v2_lite_deepep_m2n` in -[`tests/e2e`](../../../../tests/e2e/README.md#deepep-m2n-scenarios)). - -## Prerequisites - -- 4 SM90 or newer GPUs on one node. -- The environment described in the - [connector recipes README](../README.md#prerequisites). -- DeepSeek-V2-Lite weights on disk. The scripts default to - `/path/model_weights/DeepSeek-V2-Lite`; override with `MODEL_PATH=...`. -- Free ports `6239` (AFD rendezvous), `18000` (Attention API) and `18001` - (FFN). - -## Layout - -`2A2F`: Attention on GPUs 0,1 (DP2, or DP1 x TP2) and FFN DP2 on GPUs 2,3. -FFN rank 0 owns routed experts 0-31 and FFN rank 1 owns routed experts 32-63. +DeepSeek-V2-Lite (BF16) recipes. The scripts, layouts, settings, multi-pod +runs and the decode-only benchmark are described once in the +[connector recipes README](../README.md); this page lists what is specific to +the model. ```text prefill_decode_colocation/ -├── 2a2f_eager_dp2tp1.sh # Attention DP2 x TP1, eager -├── 2a2f_graph_dp2tp1.sh # Attention DP2 x TP1, FULL_DECODE_ONLY graph -├── 2a2f_graph_dbo_dp2tp1.sh # Attention DP2 x TP1, graph + DBO -└── 2a2f_graph_dp1tp2.sh # Attention DP1 x TP2, FULL_DECODE_ONLY graph +├── graph.sh # FULL_DECODE_ONLY CUDA graphs +├── eager.sh # eager +└── dbo.sh # graph + DBO ``` -## Configuration notes +## Defaults + +| Setting | Value | +| --- | --- | +| `LAYOUT` | `2A2F`: FFN DP2 on GPUs 0,1, Attention DP2 on GPUs 2,3 | +| `--max-num-seqs` | 64 per Attention DP rank | +| `--max-num-batched-tokens` | 4096 | +| `--max-model-len` | 4096 | +| `CUDAGRAPH_CAPTURE_SIZES` | 8,16,32,64 | +| Payload | BF16 (no `connector_extra_config`); the FFN runs DeepGEMM experts | + +These match the DeepSeek-V2-Lite DeepEP M2N E2E cases +(`test_deepseek_v2_lite_deepep_m2n` in +[`tests/e2e`](../../../../tests/e2e/README.md#deepep-m2n-scenarios)); +`ATTN_TP=2` matches the Attention TP2 case. + +## Notes -- Both roles set `compute_gate_on_attention: true`, use the same rank counts, - and run with `--enable-expert-parallel`. The FFN runs TP 1. - Attention runs the gate, the router, the dense first layer and the two - shared experts; the FFN runs only the routed experts. -- With Attention TP 2 each TP rank routes and dispatches its slice of the + shared experts; the FFN runs only the routed experts. With two FFN ranks, + FFN rank 0 owns routed experts 0-31 and FFN rank 1 owns 32-63. +- With Attention TP each TP rank routes and dispatches its slice of the tokens and runs the shared experts on that slice with replicated weights; the TP group all-gathers the result. -- `connector_extra_config` is left empty, so the payload is BF16 and the FFN - runs DeepGEMM experts. -- The graph and DBO settings match the - [Qwen3-30B-A3B recipes](../qwen3_30b_a3b/README.md#configuration-notes). ## Limitations diff --git a/recipe/gpu/DeepEPM2NAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_eager_dp2tp1.sh b/recipe/gpu/DeepEPM2NAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_eager_dp2tp1.sh deleted file mode 100644 index 3da8f033f..000000000 --- a/recipe/gpu/DeepEPM2NAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_eager_dp2tp1.sh +++ /dev/null @@ -1,55 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project - -# 2A2F eager: Attention DP2 x TP1 on GPUs 0,1; FFN DP2 on GPUs 2,3. - -MODEL_PATH=${MODEL_PATH:-/path/model_weights/DeepSeek-V2-Lite} -export VLLM_USE_V2_MODEL_RUNNER=1 - -CUDA_VISIBLE_DEVICES=2,3 vllm serve "$MODEL_PATH" \ - --data-parallel-size 2 \ - --tensor-parallel-size 1 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "ffn", - "connector": "DeepEPM2NAFDConnector", - "host": "127.0.0.1", - "port": 6239, - "num_attention_ranks": 2, - "num_ffn_ranks": 2, - "compute_gate_on_attention": true - } - }' \ - --max-num-seqs 64 \ - --max-num-batched-tokens 4096 \ - --max-model-len 4096 \ - --enforce-eager \ - --host 127.0.0.1 \ - --port 18001 \ - --trust-remote-code > ffn.log 2>&1 & - -CUDA_VISIBLE_DEVICES=0,1 vllm serve "$MODEL_PATH" \ - --data-parallel-size 2 \ - --tensor-parallel-size 1 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "attention", - "connector": "DeepEPM2NAFDConnector", - "host": "127.0.0.1", - "port": 6239, - "num_attention_ranks": 2, - "num_ffn_ranks": 2, - "compute_gate_on_attention": true - } - }' \ - --max-num-seqs 64 \ - --max-num-batched-tokens 4096 \ - --max-model-len 4096 \ - --enforce-eager \ - --host 127.0.0.1 \ - --port 18000 \ - --trust-remote-code > attn.log 2>&1 & - -wait diff --git a/recipe/gpu/DeepEPM2NAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp2tp1.sh b/recipe/gpu/DeepEPM2NAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp2tp1.sh deleted file mode 100644 index 0ea5a336e..000000000 --- a/recipe/gpu/DeepEPM2NAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp2tp1.sh +++ /dev/null @@ -1,64 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project - -# 2A2F graph + DBO: Attention DP2 x TP1 on GPUs 0,1; FFN DP2 on GPUs 2,3. -# DBO must be enabled on both roles. - -MODEL_PATH=${MODEL_PATH:-/path/model_weights/DeepSeek-V2-Lite} -export VLLM_USE_V2_MODEL_RUNNER=1 - -CUDA_VISIBLE_DEVICES=2,3 vllm serve "$MODEL_PATH" \ - --data-parallel-size 2 \ - --tensor-parallel-size 1 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "ffn", - "connector": "DeepEPM2NAFDConnector", - "host": "127.0.0.1", - "port": 6239, - "num_attention_ranks": 2, - "num_ffn_ranks": 2, - "compute_gate_on_attention": true - } - }' \ - --max-num-seqs 64 \ - --max-num-batched-tokens 4096 \ - --max-model-len 4096 \ - --max-cudagraph-capture-size 64 \ - --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes": [8, 16, 32, 64]}' \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 8 \ - --host 127.0.0.1 \ - --port 18001 \ - --trust-remote-code > ffn.log 2>&1 & - -CUDA_VISIBLE_DEVICES=0,1 vllm serve "$MODEL_PATH" \ - --data-parallel-size 2 \ - --tensor-parallel-size 1 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "attention", - "connector": "DeepEPM2NAFDConnector", - "host": "127.0.0.1", - "port": 6239, - "num_attention_ranks": 2, - "num_ffn_ranks": 2, - "compute_gate_on_attention": true - } - }' \ - --max-num-seqs 64 \ - --max-num-batched-tokens 4096 \ - --max-model-len 4096 \ - --max-cudagraph-capture-size 64 \ - --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes": [8, 16, 32, 64]}' \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 8 \ - --host 127.0.0.1 \ - --port 18000 \ - --trust-remote-code > attn.log 2>&1 & - -wait diff --git a/recipe/gpu/DeepEPM2NAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dp1tp2.sh b/recipe/gpu/DeepEPM2NAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dp1tp2.sh deleted file mode 100644 index e08e1d695..000000000 --- a/recipe/gpu/DeepEPM2NAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dp1tp2.sh +++ /dev/null @@ -1,58 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project - -# 2A2F graph with Attention TP: Attention DP1 x TP2 on GPUs 0,1; FFN DP2 x TP1 on -# GPUs 2,3. Each Attention TP rank dispatches its slice of the tokens. - -MODEL_PATH=${MODEL_PATH:-/path/model_weights/DeepSeek-V2-Lite} -export VLLM_USE_V2_MODEL_RUNNER=1 - -CUDA_VISIBLE_DEVICES=2,3 vllm serve "$MODEL_PATH" \ - --data-parallel-size 2 \ - --tensor-parallel-size 1 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "ffn", - "connector": "DeepEPM2NAFDConnector", - "host": "127.0.0.1", - "port": 6239, - "num_attention_ranks": 2, - "num_ffn_ranks": 2, - "compute_gate_on_attention": true - } - }' \ - --max-num-seqs 64 \ - --max-num-batched-tokens 4096 \ - --max-model-len 4096 \ - --max-cudagraph-capture-size 64 \ - --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes": [8, 16, 32, 64]}' \ - --host 127.0.0.1 \ - --port 18001 \ - --trust-remote-code > ffn.log 2>&1 & - -CUDA_VISIBLE_DEVICES=0,1 vllm serve "$MODEL_PATH" \ - --data-parallel-size 1 \ - --tensor-parallel-size 2 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "attention", - "connector": "DeepEPM2NAFDConnector", - "host": "127.0.0.1", - "port": 6239, - "num_attention_ranks": 2, - "num_ffn_ranks": 2, - "compute_gate_on_attention": true - } - }' \ - --max-num-seqs 64 \ - --max-num-batched-tokens 4096 \ - --max-model-len 4096 \ - --max-cudagraph-capture-size 64 \ - --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes": [8, 16, 32, 64]}' \ - --host 127.0.0.1 \ - --port 18000 \ - --trust-remote-code > attn.log 2>&1 & - -wait diff --git a/recipe/gpu/DeepEPM2NAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dp2tp1.sh b/recipe/gpu/DeepEPM2NAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dp2tp1.sh deleted file mode 100644 index c7f1801ad..000000000 --- a/recipe/gpu/DeepEPM2NAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dp2tp1.sh +++ /dev/null @@ -1,57 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project - -# 2A2F FULL_DECODE_ONLY graph: Attention DP2 x TP1 on GPUs 0,1; FFN DP2 on GPUs 2,3. - -MODEL_PATH=${MODEL_PATH:-/path/model_weights/DeepSeek-V2-Lite} -export VLLM_USE_V2_MODEL_RUNNER=1 - -CUDA_VISIBLE_DEVICES=2,3 vllm serve "$MODEL_PATH" \ - --data-parallel-size 2 \ - --tensor-parallel-size 1 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "ffn", - "connector": "DeepEPM2NAFDConnector", - "host": "127.0.0.1", - "port": 6239, - "num_attention_ranks": 2, - "num_ffn_ranks": 2, - "compute_gate_on_attention": true - } - }' \ - --max-num-seqs 64 \ - --max-num-batched-tokens 4096 \ - --max-model-len 4096 \ - --max-cudagraph-capture-size 64 \ - --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes": [8, 16, 32, 64]}' \ - --host 127.0.0.1 \ - --port 18001 \ - --trust-remote-code > ffn.log 2>&1 & - -CUDA_VISIBLE_DEVICES=0,1 vllm serve "$MODEL_PATH" \ - --data-parallel-size 2 \ - --tensor-parallel-size 1 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "attention", - "connector": "DeepEPM2NAFDConnector", - "host": "127.0.0.1", - "port": 6239, - "num_attention_ranks": 2, - "num_ffn_ranks": 2, - "compute_gate_on_attention": true - } - }' \ - --max-num-seqs 64 \ - --max-num-batched-tokens 4096 \ - --max-model-len 4096 \ - --max-cudagraph-capture-size 64 \ - --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes": [8, 16, 32, 64]}' \ - --host 127.0.0.1 \ - --port 18000 \ - --trust-remote-code > attn.log 2>&1 & - -wait diff --git a/recipe/gpu/DeepEPM2NAFDConnector/deepseek_v2_lite/prefill_decode_colocation/dbo.sh b/recipe/gpu/DeepEPM2NAFDConnector/deepseek_v2_lite/prefill_decode_colocation/dbo.sh new file mode 100644 index 000000000..dbb24388f --- /dev/null +++ b/recipe/gpu/DeepEPM2NAFDConnector/deepseek_v2_lite/prefill_decode_colocation/dbo.sh @@ -0,0 +1,19 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project +# +# DeepSeek-V2-Lite (BF16) on DeepEPM2NAFDConnector, FULL_DECODE_ONLY graphs + DBO. +# Placement: one pod per comma-separated LAYOUT entry (default 2A2F), pod k +# running POD=AFD_ with the same LAYOUT; FFN on each pod's first GPUs. +# Settings and running in a pod: recipe/gpu/common/README.md +# shellcheck disable=SC1090,SC2034 # the sourced launcher reads these variables + +AFD_CONNECTOR=${AFD_CONNECTOR:-DeepEPM2NAFDConnector} +LAYOUT=${LAYOUT:-2A2F} +MODEL_PATH=${MODEL_PATH:-/path/model_weights/DeepSeek-V2-Lite} +MODE=graph CUDAGRAPH_CAPTURE_SIZES=${CUDAGRAPH_CAPTURE_SIZES:-8,16,32,64} # ascending +SERVE_ARGS=(--enable-ep-weight-filter --max-num-seqs 64 --max-num-batched-tokens 4096 --max-model-len 4096 + --enable-dbo --dbo-decode-token-threshold 2 --dbo-prefill-token-threshold 8) + +source "${AFD_RECIPE_LIB:-$(dirname "${BASH_SOURCE[0]}")/../../../common/afd_colocation.sh}" +afd_colocation_serve "$@" diff --git a/recipe/gpu/DeepEPM2NAFDConnector/deepseek_v2_lite/prefill_decode_colocation/eager.sh b/recipe/gpu/DeepEPM2NAFDConnector/deepseek_v2_lite/prefill_decode_colocation/eager.sh new file mode 100644 index 000000000..53c3bc605 --- /dev/null +++ b/recipe/gpu/DeepEPM2NAFDConnector/deepseek_v2_lite/prefill_decode_colocation/eager.sh @@ -0,0 +1,18 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project +# +# DeepSeek-V2-Lite (BF16) on DeepEPM2NAFDConnector, eager mode. +# Placement: one pod per comma-separated LAYOUT entry (default 2A2F), pod k +# running POD=AFD_ with the same LAYOUT; FFN on each pod's first GPUs. +# Settings and running in a pod: recipe/gpu/common/README.md +# shellcheck disable=SC1090,SC2034 # the sourced launcher reads these variables + +AFD_CONNECTOR=${AFD_CONNECTOR:-DeepEPM2NAFDConnector} +LAYOUT=${LAYOUT:-2A2F} +MODEL_PATH=${MODEL_PATH:-/path/model_weights/DeepSeek-V2-Lite} +MODE=eager +SERVE_ARGS=(--enable-ep-weight-filter --max-num-seqs 64 --max-num-batched-tokens 4096 --max-model-len 4096) + +source "${AFD_RECIPE_LIB:-$(dirname "${BASH_SOURCE[0]}")/../../../common/afd_colocation.sh}" +afd_colocation_serve "$@" diff --git a/recipe/gpu/DeepEPM2NAFDConnector/deepseek_v2_lite/prefill_decode_colocation/graph.sh b/recipe/gpu/DeepEPM2NAFDConnector/deepseek_v2_lite/prefill_decode_colocation/graph.sh new file mode 100644 index 000000000..1e267ee28 --- /dev/null +++ b/recipe/gpu/DeepEPM2NAFDConnector/deepseek_v2_lite/prefill_decode_colocation/graph.sh @@ -0,0 +1,18 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project +# +# DeepSeek-V2-Lite (BF16) on DeepEPM2NAFDConnector, FULL_DECODE_ONLY CUDA graphs. +# Placement: one pod per comma-separated LAYOUT entry (default 2A2F), pod k +# running POD=AFD_ with the same LAYOUT; FFN on each pod's first GPUs. +# Settings and running in a pod: recipe/gpu/common/README.md +# shellcheck disable=SC1090,SC2034 # the sourced launcher reads these variables + +AFD_CONNECTOR=${AFD_CONNECTOR:-DeepEPM2NAFDConnector} +LAYOUT=${LAYOUT:-2A2F} +MODEL_PATH=${MODEL_PATH:-/path/model_weights/DeepSeek-V2-Lite} +MODE=graph CUDAGRAPH_CAPTURE_SIZES=${CUDAGRAPH_CAPTURE_SIZES:-8,16,32,64} # ascending +SERVE_ARGS=(--enable-ep-weight-filter --max-num-seqs 64 --max-num-batched-tokens 4096 --max-model-len 4096) + +source "${AFD_RECIPE_LIB:-$(dirname "${BASH_SOURCE[0]}")/../../../common/afd_colocation.sh}" +afd_colocation_serve "$@" diff --git a/recipe/gpu/DeepEPM2NAFDConnector/qwen3_235b_a22b_fp8/README.md b/recipe/gpu/DeepEPM2NAFDConnector/qwen3_235b_a22b_fp8/README.md index c9bd1a761..4c94ac5ac 100644 --- a/recipe/gpu/DeepEPM2NAFDConnector/qwen3_235b_a22b_fp8/README.md +++ b/recipe/gpu/DeepEPM2NAFDConnector/qwen3_235b_a22b_fp8/README.md @@ -1,90 +1,52 @@ # Qwen3-235B-A22B-FP8 with DeepEPM2NAFDConnector -Launch scripts for Qwen3-235B-A22B-FP8 on `DeepEPM2NAFDConnector` with vLLM -`v0.30.0`, using the FP8 dispatch payload. - -## Prerequisites - -- 6 SM90 or newer GPUs on one node with enough memory for the model, or two - pods of 5 such GPUs on an RDMA fabric for the multi-pod recipe. These - configurations were run on H200. -- The environment described in the - [connector recipes README](../README.md#prerequisites). -- Qwen3-235B-A22B-FP8 weights on disk (block-128 FP8). The scripts default to - `/path/model_weights/Qwen3-235B-A22B-FP8`; override with `MODEL_PATH=...`. -- Free ports `6239` (AFD rendezvous), `18000` (Attention API) and `18001` - (FFN). - -## Layout - -`4A2F`: Attention DP4 x TP1 on GPUs 0-3 and FFN DP2 on GPUs 4,5. Each FFN rank -owns half of the 128 experts. +Qwen3-235B-A22B-FP8 recipes with the FP8 dispatch payload. The scripts, +layouts, settings, multi-pod runs and the decode-only benchmark are described +once in the [connector recipes README](../README.md); this page lists what is +specific to the model. ```text prefill_decode_colocation/ -├── 4a2f_graph_dp4tp1.sh # 256 sequences per Attention rank, FULL_DECODE_ONLY graph -├── 4a2f_graph_dbo_dp4tp1.sh # 512 sequences per Attention rank, graph + DBO -└── multipod_4a1f_4a1f_graph.sh # 2 pods x 4A1F, FULL_DECODE_ONLY graph (DBO optional) -``` - -### Multi-pod: 2 x 4A1F - -`multipod_4a1f_4a1f_graph.sh` runs on two pods of 5 GPUs. Each pod runs one -FFN rank on GPU 0 and four Attention ranks on GPUs 1-4 (Attention DP8 x TP1, -FFN DP2/EP2 in total), so every FFN GPU holds half of the experts, as in 4A2F. -Pod 0 leads both DP groups and hosts the AFD rendezvous; pod 1 starts both -servers headless and serves no requests. Send traffic to pod 0's port `18000`. - -Only `POD` differs between pods: - -```bash -# Pod 0 -POD=AFD_0 bash recipe/gpu/DeepEPM2NAFDConnector/qwen3_235b_a22b_fp8/prefill_decode_colocation/multipod_4a1f_4a1f_graph.sh -# Pod 1 -POD=AFD_1 bash recipe/gpu/DeepEPM2NAFDConnector/qwen3_235b_a22b_fp8/prefill_decode_colocation/multipod_4a1f_4a1f_graph.sh -``` - -On Kubernetes the script reaches pod 0 through the Services that the -`deploy-afd-k8s` skill creates. Without Kubernetes, set `HEAD_HOST` to pod 0's -address on both pods; pod 0 must accept ports `6239`, `13345` (Attention DP) -and `13355` (FFN DP). Pass the fabric's NCCL settings (for example -`NCCL_IB_HCA`) through the environment. - -`NUM_PODS`, `ATTN_PER_POD` and `FFN_PER_POD` select another symmetric layout -(`2 x 6A2F`, ...); the tuning values (`MAX_NUM_SEQS`, `MAX_NUM_BATCHED_TOKENS`, -`MAX_MODEL_LEN`, `CAPTURE_SIZES`, `PAYLOAD_FORMAT`) and DBO (`ENABLE_DBO=1`, -`DBO_DECODE_TOKEN_THRESHOLD`, `DBO_PREFILL_TOKEN_THRESHOLD`) are environment -variables with the 4A2F values as defaults. A placement the connector cannot -carry fails at startup with the pods' shapes in the error. - -## Configuration notes - -- `connector_extra_config` sets `payload_format: "fp8_block128_fp32"`. The - FFN must load the block-128 FP8 checkpoint; the startup handshake rejects a - mismatch between the payload format and the loaded expert weights. -- The FFN keeps vLLM's default `--moe-backend auto`, which the plugin pins to - `triton` so the expert weights stay in the layout the connector reads. -- `afd_process_group_timeout_s` and `--cpu-distributed-timeout-seconds` are - raised to 3600 s because loading the weights can take longer than the - default 120 s join timeout, especially from cold storage. -- The capture sizes cover `--max-num-seqs`. The connector derives its dispatch - buckets from them and compiles one transport kernel set per bucket at - startup. -- `expert_num_sms` (default `0`, all SMs) and `num_sms` tune how SMs are split - between DeepGEMM experts and the transport kernels; see the connector guide. - -## Checking a deployment - -```bash -curl -s http://127.0.0.1:18000/v1/completions \ - -H 'Content-Type: application/json' \ - -d '{"model":"'"$MODEL_PATH"'","prompt":"The capital of France is","max_tokens":12,"temperature":0}' +├── graph.sh # FULL_DECODE_ONLY CUDA graphs, 256 sequences per Attention rank +├── eager.sh # eager, 256 sequences per Attention rank +└── dbo.sh # graph + DBO, 512 sequences per Attention rank ``` -The answer should name Paris. An FFN log line `Using TRITON Fp8 MoE backend` -confirms the expected expert-weight layout. +## Defaults + +| Setting | `graph.sh`, `eager.sh` | `dbo.sh` | +| --- | --- | --- | +| `LAYOUT` | `2A2F` | `2A2F` | +| `--max-num-seqs` | 256 per Attention DP rank | 512 | +| `--max-num-batched-tokens` | 3072 | 3072 | +| `--max-model-len` | 4096 | 4096 | +| `CUDAGRAPH_CAPTURE_SIZES` | 8,12,16,24,32,48,64,96,128,192,256 | 8,16,32,64,96,128,192,256,384,512 | +| `connector_extra_config` | `"payload_format": "fp8_block128_fp32"` | same | + +Common layouts: + +| `LAYOUT` | GPUs | Notes | +| --- | --- | --- | +| `2A2F` | 4 on one host | each FFN GPU holds half of the 128 experts | +| `4A2F` | 6 on one host | Attention DP4 on GPUs 2-5 | +| `4A1F,4A1F` | 2 pods of 5 | Attention DP8, FFN DP2/EP2 | + +## Notes + +- Every layout with two FFN ranks puts about 110 GB of FP8 experts on each + FFN GPU, so it needs H200-class GPUs (about 140 GB); four or more FFN ranks + fit smaller GPUs. Attention ranks hold no experts + (`--enable-ep-weight-filter`). +- The FFN must load the block-128 FP8 checkpoint; the startup handshake + rejects a mismatch between the payload format and the loaded expert weights. +- `ATTN_TP=4` with `LAYOUT=4A2F` runs Attention as one DP1/TP4 engine; raise + `--max-num-seqs` and the capture sizes with it, since one engine then holds + every running request. +- The baseline without AFD is + [`recipe/gpu/baseline/qwen3_235b_a22b_fp8/vanilla_tp4.sh`](../../baseline/qwen3_235b_a22b_fp8/vanilla_tp4.sh). +- Qwen3-235B holds 188 KiB of BF16 KV per token (94 layers, 4 KV heads of + 128); an H200 Attention rank has room for about 640K tokens. ## Limitations - Qwen3 MoE only; see the connector guide for the full list of requirements. -- `FULL_DECODE_ONLY` is the only CUDA graph mode. diff --git a/recipe/gpu/DeepEPM2NAFDConnector/qwen3_235b_a22b_fp8/prefill_decode_colocation/4a2f_graph_dbo_dp4tp1.sh b/recipe/gpu/DeepEPM2NAFDConnector/qwen3_235b_a22b_fp8/prefill_decode_colocation/4a2f_graph_dbo_dp4tp1.sh deleted file mode 100644 index 37aef9314..000000000 --- a/recipe/gpu/DeepEPM2NAFDConnector/qwen3_235b_a22b_fp8/prefill_decode_colocation/4a2f_graph_dbo_dp4tp1.sh +++ /dev/null @@ -1,68 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project - -# 4A2F graph + DBO, FP8 payload: Attention DP4 x TP1 on GPUs 0-3; FFN DP2 on -# GPUs 4,5. DBO must be enabled on both roles. - -MODEL_PATH=${MODEL_PATH:-/path/model_weights/Qwen3-235B-A22B-FP8} -export VLLM_USE_V2_MODEL_RUNNER=1 - -CUDA_VISIBLE_DEVICES=4,5 vllm serve "$MODEL_PATH" \ - --data-parallel-size 2 \ - --tensor-parallel-size 1 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "ffn", - "connector": "DeepEPM2NAFDConnector", - "host": "127.0.0.1", - "port": 6239, - "num_attention_ranks": 4, - "num_ffn_ranks": 2, - "compute_gate_on_attention": true, - "afd_process_group_timeout_s": 3600, - "connector_extra_config": {"payload_format": "fp8_block128_fp32"} - } - }' \ - --max-num-seqs 512 \ - --max-num-batched-tokens 3072 \ - --max-model-len 4096 \ - --max-cudagraph-capture-size 512 \ - --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes": [8, 16, 32, 64, 96, 128, 192, 256, 384, 512]}' \ - --cpu-distributed-timeout-seconds 3600 \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 8 \ - --host 127.0.0.1 \ - --port 18001 > ffn.log 2>&1 & - -CUDA_VISIBLE_DEVICES=0,1,2,3 vllm serve "$MODEL_PATH" \ - --data-parallel-size 4 \ - --tensor-parallel-size 1 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "attention", - "connector": "DeepEPM2NAFDConnector", - "host": "127.0.0.1", - "port": 6239, - "num_attention_ranks": 4, - "num_ffn_ranks": 2, - "compute_gate_on_attention": true, - "afd_process_group_timeout_s": 3600, - "connector_extra_config": {"payload_format": "fp8_block128_fp32"} - } - }' \ - --max-num-seqs 512 \ - --max-num-batched-tokens 3072 \ - --max-model-len 4096 \ - --max-cudagraph-capture-size 512 \ - --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes": [8, 16, 32, 64, 96, 128, 192, 256, 384, 512]}' \ - --cpu-distributed-timeout-seconds 3600 \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 8 \ - --host 127.0.0.1 \ - --port 18000 > attn.log 2>&1 & - -wait diff --git a/recipe/gpu/DeepEPM2NAFDConnector/qwen3_235b_a22b_fp8/prefill_decode_colocation/4a2f_graph_dp4tp1.sh b/recipe/gpu/DeepEPM2NAFDConnector/qwen3_235b_a22b_fp8/prefill_decode_colocation/4a2f_graph_dp4tp1.sh deleted file mode 100644 index 04491ad2a..000000000 --- a/recipe/gpu/DeepEPM2NAFDConnector/qwen3_235b_a22b_fp8/prefill_decode_colocation/4a2f_graph_dp4tp1.sh +++ /dev/null @@ -1,61 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project - -# 4A2F graph, FP8 payload: Attention DP4 x TP1 on GPUs 0-3; FFN DP2 on GPUs 4,5. - -MODEL_PATH=${MODEL_PATH:-/path/model_weights/Qwen3-235B-A22B-FP8} -export VLLM_USE_V2_MODEL_RUNNER=1 - -CUDA_VISIBLE_DEVICES=4,5 vllm serve "$MODEL_PATH" \ - --data-parallel-size 2 \ - --tensor-parallel-size 1 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "ffn", - "connector": "DeepEPM2NAFDConnector", - "host": "127.0.0.1", - "port": 6239, - "num_attention_ranks": 4, - "num_ffn_ranks": 2, - "compute_gate_on_attention": true, - "afd_process_group_timeout_s": 3600, - "connector_extra_config": {"payload_format": "fp8_block128_fp32"} - } - }' \ - --max-num-seqs 256 \ - --max-num-batched-tokens 3072 \ - --max-model-len 4096 \ - --max-cudagraph-capture-size 256 \ - --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes": [8, 16, 32, 64, 96, 128, 192, 256]}' \ - --cpu-distributed-timeout-seconds 3600 \ - --host 127.0.0.1 \ - --port 18001 > ffn.log 2>&1 & - -CUDA_VISIBLE_DEVICES=0,1,2,3 vllm serve "$MODEL_PATH" \ - --data-parallel-size 4 \ - --tensor-parallel-size 1 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "attention", - "connector": "DeepEPM2NAFDConnector", - "host": "127.0.0.1", - "port": 6239, - "num_attention_ranks": 4, - "num_ffn_ranks": 2, - "compute_gate_on_attention": true, - "afd_process_group_timeout_s": 3600, - "connector_extra_config": {"payload_format": "fp8_block128_fp32"} - } - }' \ - --max-num-seqs 256 \ - --max-num-batched-tokens 3072 \ - --max-model-len 4096 \ - --max-cudagraph-capture-size 256 \ - --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes": [8, 16, 32, 64, 96, 128, 192, 256]}' \ - --cpu-distributed-timeout-seconds 3600 \ - --host 127.0.0.1 \ - --port 18000 > attn.log 2>&1 & - -wait diff --git a/recipe/gpu/DeepEPM2NAFDConnector/qwen3_235b_a22b_fp8/prefill_decode_colocation/dbo.sh b/recipe/gpu/DeepEPM2NAFDConnector/qwen3_235b_a22b_fp8/prefill_decode_colocation/dbo.sh new file mode 100644 index 000000000..bbbbb0cc3 --- /dev/null +++ b/recipe/gpu/DeepEPM2NAFDConnector/qwen3_235b_a22b_fp8/prefill_decode_colocation/dbo.sh @@ -0,0 +1,20 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project +# +# Qwen3-235B-A22B-FP8 on DeepEPM2NAFDConnector, FULL_DECODE_ONLY graphs + DBO. +# Placement: one pod per comma-separated LAYOUT entry (default 2A2F), pod k +# running POD=AFD_ with the same LAYOUT; FFN on each pod's first GPUs. +# Settings and running in a pod: recipe/gpu/common/README.md +# shellcheck disable=SC1090,SC2034 # the sourced launcher reads these variables + +AFD_CONNECTOR=${AFD_CONNECTOR:-DeepEPM2NAFDConnector} +LAYOUT=${LAYOUT:-2A2F} +MODEL_PATH=${MODEL_PATH:-/path/model_weights/Qwen3-235B-A22B-FP8} +CONNECTOR_CONFIG='"payload_format": "fp8_block128_fp32"' # FP8 payload +MODE=graph CUDAGRAPH_CAPTURE_SIZES=${CUDAGRAPH_CAPTURE_SIZES:-8,16,32,64,96,128,192,256,384,512} # ascending +SERVE_ARGS=(--enable-ep-weight-filter --max-num-seqs 512 --max-num-batched-tokens 3072 --max-model-len 4096 + --enable-dbo --dbo-decode-token-threshold 2 --dbo-prefill-token-threshold 8) + +source "${AFD_RECIPE_LIB:-$(dirname "${BASH_SOURCE[0]}")/../../../common/afd_colocation.sh}" +afd_colocation_serve "$@" diff --git a/recipe/gpu/DeepEPM2NAFDConnector/qwen3_235b_a22b_fp8/prefill_decode_colocation/eager.sh b/recipe/gpu/DeepEPM2NAFDConnector/qwen3_235b_a22b_fp8/prefill_decode_colocation/eager.sh new file mode 100644 index 000000000..96d1b0cc6 --- /dev/null +++ b/recipe/gpu/DeepEPM2NAFDConnector/qwen3_235b_a22b_fp8/prefill_decode_colocation/eager.sh @@ -0,0 +1,19 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project +# +# Qwen3-235B-A22B-FP8 on DeepEPM2NAFDConnector, eager mode. +# Placement: one pod per comma-separated LAYOUT entry (default 2A2F), pod k +# running POD=AFD_ with the same LAYOUT; FFN on each pod's first GPUs. +# Settings and running in a pod: recipe/gpu/common/README.md +# shellcheck disable=SC1090,SC2034 # the sourced launcher reads these variables + +AFD_CONNECTOR=${AFD_CONNECTOR:-DeepEPM2NAFDConnector} +LAYOUT=${LAYOUT:-2A2F} +MODEL_PATH=${MODEL_PATH:-/path/model_weights/Qwen3-235B-A22B-FP8} +CONNECTOR_CONFIG='"payload_format": "fp8_block128_fp32"' # FP8 payload +MODE=eager +SERVE_ARGS=(--enable-ep-weight-filter --max-num-seqs 256 --max-num-batched-tokens 3072 --max-model-len 4096) + +source "${AFD_RECIPE_LIB:-$(dirname "${BASH_SOURCE[0]}")/../../../common/afd_colocation.sh}" +afd_colocation_serve "$@" diff --git a/recipe/gpu/DeepEPM2NAFDConnector/qwen3_235b_a22b_fp8/prefill_decode_colocation/graph.sh b/recipe/gpu/DeepEPM2NAFDConnector/qwen3_235b_a22b_fp8/prefill_decode_colocation/graph.sh new file mode 100644 index 000000000..354fbe141 --- /dev/null +++ b/recipe/gpu/DeepEPM2NAFDConnector/qwen3_235b_a22b_fp8/prefill_decode_colocation/graph.sh @@ -0,0 +1,19 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project +# +# Qwen3-235B-A22B-FP8 on DeepEPM2NAFDConnector, FULL_DECODE_ONLY CUDA graphs. +# Placement: one pod per comma-separated LAYOUT entry (default 2A2F), pod k +# running POD=AFD_ with the same LAYOUT; FFN on each pod's first GPUs. +# Settings and running in a pod: recipe/gpu/common/README.md +# shellcheck disable=SC1090,SC2034 # the sourced launcher reads these variables + +AFD_CONNECTOR=${AFD_CONNECTOR:-DeepEPM2NAFDConnector} +LAYOUT=${LAYOUT:-2A2F} +MODEL_PATH=${MODEL_PATH:-/path/model_weights/Qwen3-235B-A22B-FP8} +CONNECTOR_CONFIG='"payload_format": "fp8_block128_fp32"' # FP8 payload +MODE=graph CUDAGRAPH_CAPTURE_SIZES=${CUDAGRAPH_CAPTURE_SIZES:-8,12,16,24,32,48,64,96,128,192,256} # ascending +SERVE_ARGS=(--enable-ep-weight-filter --max-num-seqs 256 --max-num-batched-tokens 3072 --max-model-len 4096) + +source "${AFD_RECIPE_LIB:-$(dirname "${BASH_SOURCE[0]}")/../../../common/afd_colocation.sh}" +afd_colocation_serve "$@" diff --git a/recipe/gpu/DeepEPM2NAFDConnector/qwen3_235b_a22b_fp8/prefill_decode_colocation/multipod_4a1f_4a1f_graph.sh b/recipe/gpu/DeepEPM2NAFDConnector/qwen3_235b_a22b_fp8/prefill_decode_colocation/multipod_4a1f_4a1f_graph.sh deleted file mode 100644 index ed48f9b2a..000000000 --- a/recipe/gpu/DeepEPM2NAFDConnector/qwen3_235b_a22b_fp8/prefill_decode_colocation/multipod_4a1f_4a1f_graph.sh +++ /dev/null @@ -1,82 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project - -# Qwen3-235B-A22B-FP8 with DeepEPM2NAFDConnector, 2 x 4A1F, FULL_DECODE_ONLY -# CUDA graphs: 2 pods of 5 GPUs, each with FFN on GPU 0 and Attention DP4 x TP1 -# on GPUs 1-4 (Attention DP8, FFN DP2/EP2 in total). Send traffic to port -# 18000 of the POD=AFD_0 pod. -# -# Use .agents/skills/deploy-afd-k8s/SKILL.md to deploy this recipe -# on 2 pods using the setup below. -# For pod 1: POD=AFD_0 -# For pod 2: POD=AFD_1 -# -# NUM_PODS x (ATTN_PER_POD Attention + FFN_PER_POD FFN) sets any symmetric -# layout (2 x 1A1F, 3 x 6A2F, ...); give every pod the same values. HEAD_HOST -# points the AFD rendezvous and both DP leaders at pod 0 (runs without k8s). - -MODEL_PATH=${MODEL_PATH:-/path/model_weights/Qwen3-235B-A22B-FP8} -NUM_PODS=${NUM_PODS:-2} -ATTN_PER_POD=${ATTN_PER_POD:-4} -FFN_PER_POD=${FFN_PER_POD:-1} -PAYLOAD_FORMAT=${PAYLOAD_FORMAT:-fp8_block128_fp32} -MAX_NUM_SEQS=${MAX_NUM_SEQS:-256} # per Attention DP rank -MAX_NUM_BATCHED_TOKENS=${MAX_NUM_BATCHED_TOKENS:-3072} -MAX_MODEL_LEN=${MAX_MODEL_LEN:-4096} -CAPTURE_SIZES=${CAPTURE_SIZES:-8,16,32,64,96,128,192,256} # ascending -ENABLE_DBO=${ENABLE_DBO:-0} -DBO_DECODE_TOKEN_THRESHOLD=${DBO_DECODE_TOKEN_THRESHOLD:-2} -DBO_PREFILL_TOKEN_THRESHOLD=${DBO_PREFILL_TOKEN_THRESHOLD:-2} -export VLLM_USE_V2_MODEL_RUNNER=1 - -K=${POD#AFD_} A=$ATTN_PER_POD F=$FFN_PER_POD - -# role_dp PER_POD ADDRESS RPC_PORT -> DP: this pod's DP flags for one role. -# Pod 0 leads and omits its start rank: vLLM reads an explicit start rank on a -# non-headless engine as hybrid load balancing and rejects the headless peers. -role_dp() { - DP=(--data-parallel-size $((NUM_PODS * $1)) --data-parallel-size-local "$1" - --data-parallel-address "$2" --data-parallel-rpc-port "$3") - if ((K > 0)); then DP+=(--headless --data-parallel-start-rank $((K * $1))); fi -} - -afd() { - cat < ffn.log 2>&1 & - -role_dp "$A" "${HEAD_HOST:-vllm-attn-dp-service}" 13345 -CUDA_VISIBLE_DEVICES=$(seq "$F" $((F + A - 1)) | paste -sd, -) vllm serve "$MODEL_PATH" \ - --additional-config "$(afd attention)" "${COMMON[@]}" "${DP[@]}" \ - --port 18000 > attn.log 2>&1 & - -wait diff --git a/recipe/gpu/DeepEPM2NAFDConnector/qwen3_30b_a3b/README.md b/recipe/gpu/DeepEPM2NAFDConnector/qwen3_30b_a3b/README.md index 184806b5e..9e9c7bbec 100644 --- a/recipe/gpu/DeepEPM2NAFDConnector/qwen3_30b_a3b/README.md +++ b/recipe/gpu/DeepEPM2NAFDConnector/qwen3_30b_a3b/README.md @@ -1,59 +1,42 @@ # Qwen3-30B-A3B with DeepEPM2NAFDConnector -Launch scripts for Qwen3-30B-A3B (BF16) on `DeepEPM2NAFDConnector` with vLLM -`v0.30.0`. They use the same layouts as the Qwen3 MoE DeepEP M2N E2E cases -(`afd-deepep-m2n-*-2a2f` in [`tests/e2e`](../../../../tests/e2e/README.md#deepep-m2n-scenarios)). - -## Prerequisites - -- 4 SM90 or newer GPUs on one node. -- The environment described in the - [connector recipes README](../README.md#prerequisites). -- Qwen3-30B-A3B weights on disk. The scripts default to - `/path/model_weights/Qwen3-30B-A3B`; override with `MODEL_PATH=...`. -- Free ports `6239` (AFD rendezvous), `18000` (Attention API) and `18001` - (FFN). - -## Layout - -`2A2F`: Attention on GPUs 0,1 and FFN DP2 on GPUs 2,3. FFN rank 0 owns experts -0-63 and FFN rank 1 owns experts 64-127. +Qwen3-30B-A3B (BF16) recipes. The scripts, layouts, settings, multi-pod runs +and the decode-only benchmark are described once in the +[connector recipes README](../README.md); this page lists what is specific to +the model. ```text prefill_decode_colocation/ -├── 2a2f_eager_dp2tp1.sh # Attention DP2 x TP1, eager -├── 2a2f_graph_dp2tp1.sh # Attention DP2 x TP1, FULL_DECODE_ONLY graph -├── 2a2f_graph_dbo_dp2tp1.sh # Attention DP2 x TP1, graph + DBO -└── 2a2f_graph_dp1tp2.sh # Attention DP1 x TP2, FULL_DECODE_ONLY graph +├── graph.sh # FULL_DECODE_ONLY CUDA graphs +├── eager.sh # eager +└── dbo.sh # graph + DBO ``` -## Configuration notes - -- Both roles set `compute_gate_on_attention: true`, use the same rank counts, - and run with `--enable-expert-parallel`. The FFN runs TP 1. -- `connector_extra_config` is left empty, so the payload is BF16 and the FFN - runs DeepGEMM experts. -- The graph scripts capture batch sizes 8, 16, 32 and 64. The connector - derives its dispatch buckets from the capture sizes and compiles one - transport kernel set per bucket at startup. -- In the DBO script, both roles pass `--enable-dbo` with the same thresholds; - the connector then uses two lanes. -- The dispatch capacity is derived from `--max-num-batched-tokens` and the - largest capture size, so lowering `--max-num-batched-tokens` shrinks the - transport buffers. - -## Checking a deployment - -```bash -curl -s http://127.0.0.1:18000/v1/completions \ - -H 'Content-Type: application/json' \ - -d '{"model":"'"$MODEL_PATH"'","prompt":"The capital of France is","max_tokens":12,"temperature":0}' -``` +## Defaults + +| Setting | Value | +| --- | --- | +| `LAYOUT` | `2A2F`: FFN DP2 on GPUs 0,1, Attention DP2 on GPUs 2,3 | +| `--max-num-seqs` | 64 per Attention DP rank | +| `--max-num-batched-tokens` | 4096 | +| `--max-model-len` | 4096 | +| `CUDAGRAPH_CAPTURE_SIZES` | 8,16,32,64 | +| Payload | BF16 (no `connector_extra_config`); the FFN runs DeepGEMM experts | + +These match the Qwen3 MoE DeepEP M2N E2E cases (`afd-deepep-m2n-*-2a2f` and +the two-pod `2A0F,0A2F` and `1A1F,1A1F` cases in +[`tests/e2e`](../../../../tests/e2e/README.md#deepep-m2n-scenarios)); +`ATTN_TP=2` matches the Attention TP2 cases. + +## Notes -For an accuracy check, run the matching E2E case, for example -`afd-deepep-m2n-graph-2a2f`; see [`tests/e2e`](../../../../tests/e2e/README.md#deepep-m2n-scenarios). +- 4 GPUs of 80 GB (H100) or more fit `2A2F`. With two FFN ranks, FFN rank 0 + owns experts 0-63 and FFN rank 1 owns experts 64-127; the FFN count must + divide the 128 experts. +- Decode-only benchmark pool sizing: Qwen3-30B-A3B holds about 96 KiB of BF16 + KV per token, so 500 prompts of 1025 tokens need about 47 GiB per Attention + rank and fit an H100; 1000 prompts need about 94 GiB and need an H200. ## Limitations - Qwen3 MoE only; see the connector guide for the full list of requirements. -- `FULL_DECODE_ONLY` is the only CUDA graph mode. diff --git a/recipe/gpu/DeepEPM2NAFDConnector/qwen3_30b_a3b/prefill_decode_colocation/2a2f_eager_dp2tp1.sh b/recipe/gpu/DeepEPM2NAFDConnector/qwen3_30b_a3b/prefill_decode_colocation/2a2f_eager_dp2tp1.sh deleted file mode 100644 index 18b37e29a..000000000 --- a/recipe/gpu/DeepEPM2NAFDConnector/qwen3_30b_a3b/prefill_decode_colocation/2a2f_eager_dp2tp1.sh +++ /dev/null @@ -1,53 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project - -# 2A2F eager: Attention DP2 x TP1 on GPUs 0,1; FFN DP2 on GPUs 2,3. - -MODEL_PATH=${MODEL_PATH:-/path/model_weights/Qwen3-30B-A3B} -export VLLM_USE_V2_MODEL_RUNNER=1 - -CUDA_VISIBLE_DEVICES=2,3 vllm serve "$MODEL_PATH" \ - --data-parallel-size 2 \ - --tensor-parallel-size 1 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "ffn", - "connector": "DeepEPM2NAFDConnector", - "host": "127.0.0.1", - "port": 6239, - "num_attention_ranks": 2, - "num_ffn_ranks": 2, - "compute_gate_on_attention": true - } - }' \ - --max-num-seqs 64 \ - --max-num-batched-tokens 4096 \ - --max-model-len 4096 \ - --enforce-eager \ - --host 127.0.0.1 \ - --port 18001 > ffn.log 2>&1 & - -CUDA_VISIBLE_DEVICES=0,1 vllm serve "$MODEL_PATH" \ - --data-parallel-size 2 \ - --tensor-parallel-size 1 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "attention", - "connector": "DeepEPM2NAFDConnector", - "host": "127.0.0.1", - "port": 6239, - "num_attention_ranks": 2, - "num_ffn_ranks": 2, - "compute_gate_on_attention": true - } - }' \ - --max-num-seqs 64 \ - --max-num-batched-tokens 4096 \ - --max-model-len 4096 \ - --enforce-eager \ - --host 127.0.0.1 \ - --port 18000 > attn.log 2>&1 & - -wait diff --git a/recipe/gpu/DeepEPM2NAFDConnector/qwen3_30b_a3b/prefill_decode_colocation/2a2f_graph_dbo_dp2tp1.sh b/recipe/gpu/DeepEPM2NAFDConnector/qwen3_30b_a3b/prefill_decode_colocation/2a2f_graph_dbo_dp2tp1.sh deleted file mode 100644 index 1736b681c..000000000 --- a/recipe/gpu/DeepEPM2NAFDConnector/qwen3_30b_a3b/prefill_decode_colocation/2a2f_graph_dbo_dp2tp1.sh +++ /dev/null @@ -1,62 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project - -# 2A2F graph + DBO: Attention DP2 x TP1 on GPUs 0,1; FFN DP2 on GPUs 2,3. -# DBO must be enabled on both roles. - -MODEL_PATH=${MODEL_PATH:-/path/model_weights/Qwen3-30B-A3B} -export VLLM_USE_V2_MODEL_RUNNER=1 - -CUDA_VISIBLE_DEVICES=2,3 vllm serve "$MODEL_PATH" \ - --data-parallel-size 2 \ - --tensor-parallel-size 1 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "ffn", - "connector": "DeepEPM2NAFDConnector", - "host": "127.0.0.1", - "port": 6239, - "num_attention_ranks": 2, - "num_ffn_ranks": 2, - "compute_gate_on_attention": true - } - }' \ - --max-num-seqs 64 \ - --max-num-batched-tokens 4096 \ - --max-model-len 4096 \ - --max-cudagraph-capture-size 64 \ - --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes": [8, 16, 32, 64]}' \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 8 \ - --host 127.0.0.1 \ - --port 18001 > ffn.log 2>&1 & - -CUDA_VISIBLE_DEVICES=0,1 vllm serve "$MODEL_PATH" \ - --data-parallel-size 2 \ - --tensor-parallel-size 1 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "attention", - "connector": "DeepEPM2NAFDConnector", - "host": "127.0.0.1", - "port": 6239, - "num_attention_ranks": 2, - "num_ffn_ranks": 2, - "compute_gate_on_attention": true - } - }' \ - --max-num-seqs 64 \ - --max-num-batched-tokens 4096 \ - --max-model-len 4096 \ - --max-cudagraph-capture-size 64 \ - --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes": [8, 16, 32, 64]}' \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 8 \ - --host 127.0.0.1 \ - --port 18000 > attn.log 2>&1 & - -wait diff --git a/recipe/gpu/DeepEPM2NAFDConnector/qwen3_30b_a3b/prefill_decode_colocation/2a2f_graph_dp1tp2.sh b/recipe/gpu/DeepEPM2NAFDConnector/qwen3_30b_a3b/prefill_decode_colocation/2a2f_graph_dp1tp2.sh deleted file mode 100644 index 447ace10f..000000000 --- a/recipe/gpu/DeepEPM2NAFDConnector/qwen3_30b_a3b/prefill_decode_colocation/2a2f_graph_dp1tp2.sh +++ /dev/null @@ -1,56 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project - -# 2A2F graph with Attention TP: Attention DP1 x TP2 on GPUs 0,1; FFN DP2 x TP1 on -# GPUs 2,3. Each Attention TP rank dispatches its slice of the tokens. - -MODEL_PATH=${MODEL_PATH:-/path/model_weights/Qwen3-30B-A3B} -export VLLM_USE_V2_MODEL_RUNNER=1 - -CUDA_VISIBLE_DEVICES=2,3 vllm serve "$MODEL_PATH" \ - --data-parallel-size 2 \ - --tensor-parallel-size 1 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "ffn", - "connector": "DeepEPM2NAFDConnector", - "host": "127.0.0.1", - "port": 6239, - "num_attention_ranks": 2, - "num_ffn_ranks": 2, - "compute_gate_on_attention": true - } - }' \ - --max-num-seqs 64 \ - --max-num-batched-tokens 4096 \ - --max-model-len 4096 \ - --max-cudagraph-capture-size 64 \ - --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes": [8, 16, 32, 64]}' \ - --host 127.0.0.1 \ - --port 18001 > ffn.log 2>&1 & - -CUDA_VISIBLE_DEVICES=0,1 vllm serve "$MODEL_PATH" \ - --data-parallel-size 1 \ - --tensor-parallel-size 2 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "attention", - "connector": "DeepEPM2NAFDConnector", - "host": "127.0.0.1", - "port": 6239, - "num_attention_ranks": 2, - "num_ffn_ranks": 2, - "compute_gate_on_attention": true - } - }' \ - --max-num-seqs 64 \ - --max-num-batched-tokens 4096 \ - --max-model-len 4096 \ - --max-cudagraph-capture-size 64 \ - --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes": [8, 16, 32, 64]}' \ - --host 127.0.0.1 \ - --port 18000 > attn.log 2>&1 & - -wait diff --git a/recipe/gpu/DeepEPM2NAFDConnector/qwen3_30b_a3b/prefill_decode_colocation/2a2f_graph_dp2tp1.sh b/recipe/gpu/DeepEPM2NAFDConnector/qwen3_30b_a3b/prefill_decode_colocation/2a2f_graph_dp2tp1.sh deleted file mode 100644 index 2660ec88d..000000000 --- a/recipe/gpu/DeepEPM2NAFDConnector/qwen3_30b_a3b/prefill_decode_colocation/2a2f_graph_dp2tp1.sh +++ /dev/null @@ -1,55 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project - -# 2A2F FULL_DECODE_ONLY graph: Attention DP2 x TP1 on GPUs 0,1; FFN DP2 on GPUs 2,3. - -MODEL_PATH=${MODEL_PATH:-/path/model_weights/Qwen3-30B-A3B} -export VLLM_USE_V2_MODEL_RUNNER=1 - -CUDA_VISIBLE_DEVICES=2,3 vllm serve "$MODEL_PATH" \ - --data-parallel-size 2 \ - --tensor-parallel-size 1 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "ffn", - "connector": "DeepEPM2NAFDConnector", - "host": "127.0.0.1", - "port": 6239, - "num_attention_ranks": 2, - "num_ffn_ranks": 2, - "compute_gate_on_attention": true - } - }' \ - --max-num-seqs 64 \ - --max-num-batched-tokens 4096 \ - --max-model-len 4096 \ - --max-cudagraph-capture-size 64 \ - --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes": [8, 16, 32, 64]}' \ - --host 127.0.0.1 \ - --port 18001 > ffn.log 2>&1 & - -CUDA_VISIBLE_DEVICES=0,1 vllm serve "$MODEL_PATH" \ - --data-parallel-size 2 \ - --tensor-parallel-size 1 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "attention", - "connector": "DeepEPM2NAFDConnector", - "host": "127.0.0.1", - "port": 6239, - "num_attention_ranks": 2, - "num_ffn_ranks": 2, - "compute_gate_on_attention": true - } - }' \ - --max-num-seqs 64 \ - --max-num-batched-tokens 4096 \ - --max-model-len 4096 \ - --max-cudagraph-capture-size 64 \ - --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes": [8, 16, 32, 64]}' \ - --host 127.0.0.1 \ - --port 18000 > attn.log 2>&1 & - -wait diff --git a/recipe/gpu/DeepEPM2NAFDConnector/qwen3_30b_a3b/prefill_decode_colocation/dbo.sh b/recipe/gpu/DeepEPM2NAFDConnector/qwen3_30b_a3b/prefill_decode_colocation/dbo.sh new file mode 100644 index 000000000..845b912d8 --- /dev/null +++ b/recipe/gpu/DeepEPM2NAFDConnector/qwen3_30b_a3b/prefill_decode_colocation/dbo.sh @@ -0,0 +1,19 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project +# +# Qwen3-30B-A3B (BF16) on DeepEPM2NAFDConnector, FULL_DECODE_ONLY graphs + DBO. +# Placement: one pod per comma-separated LAYOUT entry (default 2A2F), pod k +# running POD=AFD_ with the same LAYOUT; FFN on each pod's first GPUs. +# Settings and running in a pod: recipe/gpu/common/README.md +# shellcheck disable=SC1090,SC2034 # the sourced launcher reads these variables + +AFD_CONNECTOR=${AFD_CONNECTOR:-DeepEPM2NAFDConnector} +LAYOUT=${LAYOUT:-2A2F} +MODEL_PATH=${MODEL_PATH:-/path/model_weights/Qwen3-30B-A3B} +MODE=graph CUDAGRAPH_CAPTURE_SIZES=${CUDAGRAPH_CAPTURE_SIZES:-8,16,32,64} # ascending +SERVE_ARGS=(--enable-ep-weight-filter --max-num-seqs 64 --max-num-batched-tokens 4096 --max-model-len 4096 + --enable-dbo --dbo-decode-token-threshold 2 --dbo-prefill-token-threshold 8) + +source "${AFD_RECIPE_LIB:-$(dirname "${BASH_SOURCE[0]}")/../../../common/afd_colocation.sh}" +afd_colocation_serve "$@" diff --git a/recipe/gpu/DeepEPM2NAFDConnector/qwen3_30b_a3b/prefill_decode_colocation/eager.sh b/recipe/gpu/DeepEPM2NAFDConnector/qwen3_30b_a3b/prefill_decode_colocation/eager.sh new file mode 100644 index 000000000..1df79d3cf --- /dev/null +++ b/recipe/gpu/DeepEPM2NAFDConnector/qwen3_30b_a3b/prefill_decode_colocation/eager.sh @@ -0,0 +1,18 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project +# +# Qwen3-30B-A3B (BF16) on DeepEPM2NAFDConnector, eager mode. +# Placement: one pod per comma-separated LAYOUT entry (default 2A2F), pod k +# running POD=AFD_ with the same LAYOUT; FFN on each pod's first GPUs. +# Settings and running in a pod: recipe/gpu/common/README.md +# shellcheck disable=SC1090,SC2034 # the sourced launcher reads these variables + +AFD_CONNECTOR=${AFD_CONNECTOR:-DeepEPM2NAFDConnector} +LAYOUT=${LAYOUT:-2A2F} +MODEL_PATH=${MODEL_PATH:-/path/model_weights/Qwen3-30B-A3B} +MODE=eager +SERVE_ARGS=(--enable-ep-weight-filter --max-num-seqs 64 --max-num-batched-tokens 4096 --max-model-len 4096) + +source "${AFD_RECIPE_LIB:-$(dirname "${BASH_SOURCE[0]}")/../../../common/afd_colocation.sh}" +afd_colocation_serve "$@" diff --git a/recipe/gpu/DeepEPM2NAFDConnector/qwen3_30b_a3b/prefill_decode_colocation/graph.sh b/recipe/gpu/DeepEPM2NAFDConnector/qwen3_30b_a3b/prefill_decode_colocation/graph.sh new file mode 100644 index 000000000..9859a3c61 --- /dev/null +++ b/recipe/gpu/DeepEPM2NAFDConnector/qwen3_30b_a3b/prefill_decode_colocation/graph.sh @@ -0,0 +1,18 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project +# +# Qwen3-30B-A3B (BF16) on DeepEPM2NAFDConnector, FULL_DECODE_ONLY CUDA graphs. +# Placement: one pod per comma-separated LAYOUT entry (default 2A2F), pod k +# running POD=AFD_ with the same LAYOUT; FFN on each pod's first GPUs. +# Settings and running in a pod: recipe/gpu/common/README.md +# shellcheck disable=SC1090,SC2034 # the sourced launcher reads these variables + +AFD_CONNECTOR=${AFD_CONNECTOR:-DeepEPM2NAFDConnector} +LAYOUT=${LAYOUT:-2A2F} +MODEL_PATH=${MODEL_PATH:-/path/model_weights/Qwen3-30B-A3B} +MODE=graph CUDAGRAPH_CAPTURE_SIZES=${CUDAGRAPH_CAPTURE_SIZES:-8,16,32,64} # ascending +SERVE_ARGS=(--enable-ep-weight-filter --max-num-seqs 64 --max-num-batched-tokens 4096 --max-model-len 4096) + +source "${AFD_RECIPE_LIB:-$(dirname "${BASH_SOURCE[0]}")/../../../common/afd_colocation.sh}" +afd_colocation_serve "$@" diff --git a/recipe/gpu/P2pNcclAFDConnector/README.md b/recipe/gpu/P2pNcclAFDConnector/README.md index 58862a6f0..de503ede9 100644 --- a/recipe/gpu/P2pNcclAFDConnector/README.md +++ b/recipe/gpu/P2pNcclAFDConnector/README.md @@ -11,36 +11,33 @@ for the connector's rank mapping and configuration contract. ```text . ├── deepseek_v2_lite/ -│ ├── README.md -│ ├── prefill_decode_colocation/ # 2A2F, DP/TP variants, eager/graph +│ ├── prefill_decode_colocation/ # dbo.sh, eager_dbo.sh (any LAYOUT, ATTN_TP, FFN_TP) │ └── prefill_decode_disaggregation/ # 2P1A1F -└── qwen3_5_122b_a10b_fp8/ - └── prefill_decode_colocation/ # multipod_2a_2a_2f_graph.sh: 4A2F split across 2 attention pods, graph +├── qwen3_30b_a3b/prefill_decode_colocation/ # graph.sh, default 2A2F +├── qwen3_235b_a22b_fp8/prefill_decode_colocation/ # graph.sh, default 4A2F +└── qwen3_5_122b_a10b_fp8/prefill_decode_colocation/ # graph.sh, default 2A0F,2A0F,0A2F (3 pods) ``` Each model directory holds its own recipe scripts; open its `README.md` -(where present) before running anything in it -- it documents prerequisites, -topology, ports, and known limitations for that model. +(where present) before running anything in it. -Recipes under `prefill_decode_colocation/` background an attention worker and -an FFN worker (no prefill split). Recipes under `prefill_decode_disaggregation/` -add a separate prefill stage and a proxy in front of it -- see the caveat in -"Deploying on Kubernetes" below. +The `prefill_decode_colocation/` recipes run an Attention server and an FFN +server (no prefill split) through the shared +[colocation launcher](../common/README.md), which documents the layout, +settings, ports, and how to run a recipe directly or in a pod. Recipes under +`prefill_decode_disaggregation/` add a separate prefill stage and a proxy in +front of it -- see the caveat in "Deploying on Kubernetes" below. ## Running on a local host -Run any script directly from the repository root, e.g.: - ```bash export MODEL_PATH=/path/model_weights/DeepSeek-V2-Lite -bash recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp1tp2.sh +bash recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/dbo.sh ``` -Each script backgrounds its workers and writes per-worker logs (`attn.log`, -`ffn.log`, and for disaggregation `afd_prefill*.log`) into the current -directory. Wait for `Application startup complete` in each log before -sending traffic. See the model README for exact prerequisites (GPU count, -weights, ports) and the benchmark command for that model. +A colocation recipe writes `attn.log` and `ffn.log` into the current +directory; send traffic to port `18000` once `attn.log` prints +`Application startup complete`. ## Deploying on Kubernetes @@ -49,15 +46,11 @@ to stand up a recipe from this folder on a Kubernetes/OpenShift cluster -- see that skill for the full contract (PVC/ConfigMap/pod/Service lifecycle, scope, and requirements). -The skill requires the recipe path (e.g. -`recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp1tp2.sh`) -plus the image, model id, and PVC name it needs. Placement is never asked -for: a recipe without a top-of-file placement comment deploys as a single -pod; a recipe whose header describes a per-pod placement (e.g. -`qwen3_5_122b_a10b_fp8/prefill_decode_colocation/multipod_2a_2a_2f_graph.sh`, -`deepseek_v2_lite/prefill_decode_colocation/multipod_2a_2f_graph_dbo_dp1tp2.sh`) -deploys exactly that many pods with exactly that per-pod setup -- see the -skill's `resolve-recipe.md` for the header convention. +The skill takes the recipe path (e.g. +`recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/dbo.sh`) +plus the image, model id, and PVC name, and deploys one pod per `LAYOUT` +entry (for example `LAYOUT=2A0F,0A2F ATTN_TP=2 FFN_TP=2` on the DeepSeek +recipe, or the Qwen3.5 recipe's default three pods). `prefill_decode_disaggregation` recipes are out of scope for the skill; run those locally instead. diff --git a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/README.md b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/README.md index 2f18e18a9..6e8d2acf8 100644 --- a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/README.md +++ b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/README.md @@ -14,18 +14,18 @@ for tested E2E configurations. ## Prerequisites - Install [NIXL](https://github.com/ai-dynamo/nixl). -- Four GPUs for the `2a2f_*` and `2p1a1f_*` examples; the `4a4f_*` - examples require eight GPUs. The multipod example uses two GPUs per pod - across two pods. +- Four GPUs for the default `2A2F` colocation layout and the `2p1a1f_*` + examples; `LAYOUT=4A4F` needs eight GPUs, and `LAYOUT=2A0F,0A2F` two GPUs + on each of two pods. - vLLM `v0.32.0` and the `afd-plugin` package installed in the same environment (see repository root `AGENTS.md`). - DeepSeek-V2-Lite weights on disk. Colocation scripts default to `/path/model_weights/DeepSeek-V2-Lite`; disaggregation scripts default to `/home/models/hub/models--deepseek-ai--DeepSeek-V2-Lite/snapshots/604d5664dddd88a0433dbae533b7fe9472482de0`. Set `MODEL_PATH=...` to the local weights when launching. -- A free TCP port `6269` on `127.0.0.1` for the AFD p2p connector, and - HTTP ports `18305` and `18306` for local colocation, or - `18301`–`18305` for prefill/decode disaggregation. +- Free ports `6239` (AFD rendezvous), `18000` and `18001` (Attention and + FFN HTTP) for colocation, or `18301`–`18305` and `6269` for + prefill/decode disaggregation. ## Directory layout @@ -34,14 +34,9 @@ for tested E2E configurations. ├── prefill_decode_disaggregation/ # prefill_decode_disaggregation, 2P1A1F topology │ ├── 2p1a1f_eager_dbo.sh │ └── 2p1a1f_graph_dbo.sh -└── prefill_decode_colocation/ # prefill_decode_colocation, 2A2F/4A4F topology - ├── 2a2f_eager_dbo_dp1tp2.sh - ├── 2a2f_eager_dbo_dp2tp1.sh - ├── 2a2f_graph_dbo_dp1tp2.sh - ├── 2a2f_graph_dbo_dp2tp1.sh - ├── 4a4f_eager_dbo_dp2tp2.sh - ├── 4a4f_graph_dbo_dp2tp2.sh - └── multipod_2a_2f_graph_dbo_dp1tp2.sh # 2a2f_graph_dbo_dp1tp2, split across 2 k8s pods +└── prefill_decode_colocation/ # any LAYOUT, ATTN_TP, FFN_TP + ├── dbo.sh # FULL_DECODE_ONLY graphs + DBO + └── eager_dbo.sh # eager + DBO ``` ### 1. Prefill/Decode Disaggregation — `1a1f` @@ -58,31 +53,17 @@ for tested E2E configurations. ### 2. Prefill/Decode Colocation — `2a2f` -2 processes, two GPUs each: - -| GPUs | Role | Port | -|------|-----------|-------| -| 0, 1 | Attention | 18305 | -| 2, 3 | FFN | 18306 | - -The four `2a2f_*` variants cover the TP/DP cross product; `4a4f_*` doubles -both attention and FFN ranks (DP=2 on each side); `multipod_2a_2f_graph_dbo_dp1tp2.sh` -is the same `2a2f_graph_dbo_dp1tp2` config, but gated on a `POD` env var -(`ATTENTION_0` or `FFN_0`) for the `deploy-afd-k8s` skill to deploy it -across 2 pods (see repo-root `recipe/gpu/P2pNcclAFDConnector/README.md`); -its Attention and FFN HTTP ports can both be 18305 because they are in -separate pods -- -unlike the scripts above, it is **not** meant to be run directly: with -`POD` unset, neither `vllm serve` block executes, and its -`AFD_CONNECTOR_HOST` default (`vllm-ffn-p2p-service`) only resolves inside -the cluster the skill deploys it to: - -| File | DP | TP | -|---------------------------------------|----|----| -| `2a2f_*_dp1tp2.sh` | 1 | 2 | -| `2a2f_*_dp2tp1.sh` | 2 | 1 | -| `4a4f_*_dp2tp2.sh` | 2 | 2 | -| `multipod_2a_2f_graph_dbo_dp1tp2.sh` | 1 | 2 | +The colocation recipes run on the shared +[colocation launcher](../../common/README.md): `LAYOUT` places the ranks +(default `2A2F`: FFN DP2 on GPUs 0,1, Attention DP2 on GPUs 2,3, Attention +HTTP port 18000), and `ATTN_TP`/`FFN_TP` set each role's TP size: + +| Settings | Attention / FFN | GPUs | +| ---------- | ----------------- | ------ | +| (defaults) | DP2 x TP1 each | 4 | +| `ATTN_TP=2 FFN_TP=2` | DP1 x TP2 each | 4 | +| `LAYOUT=4A4F ATTN_TP=2 FFN_TP=2` | DP2 x TP2 each | 8 | +| `LAYOUT=2A0F,0A2F ATTN_TP=2 FFN_TP=2` | DP1 x TP2 each, one pod per role (`POD=AFD_0`, `POD=AFD_1`) | 2 + 2 | ## Running @@ -96,8 +77,8 @@ before sending traffic. ```bash export MODEL_PATH=/path/model_weights/DeepSeek-V2-Lite -export VLLM_USE_V2_MODEL_RUNNER=1 -bash recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp1tp2.sh +ATTN_TP=2 FFN_TP=2 \ + bash recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/dbo.sh ``` ### prefill_decode_disaggregation @@ -115,14 +96,15 @@ Confirm the serving stack is ready before running: ```bash export MODEL_PATH=/path/model_weights/DeepSeek-V2-Lite export RESULT_DIR=/tmp/results -export RESULT_FILENAME=2a2f_graph_dbo_dp1tp2.json +export RESULT_FILENAME=2a2f_dbo_dp1tp2.json bash tools/benchmarks/request_generator.sh ``` The script runs `vllm bench serve`; it does not wait for readiness or send a separate smoke request. By default it sends 1024 random requests (1024 input tokens / 128 output tokens) at request rate 5 with `--max-concurrency 32` -against `127.0.0.1:18305`, and saves the JSON result to +against `127.0.0.1:18000` (the colocation Attention port; set `PORT=18305` +for the disaggregation proxy), and saves the JSON result to `$RESULT_DIR/$RESULT_FILENAME`. Override `HOST`, `PORT`, `MODEL_PATH`, `NUM_PROMPTS`, `REQUEST_RATE`, `MAX_CONCURRENCY`, `INPUT_LEN`, and `OUTPUT_LEN` for smaller smoke runs or larger throughput sweeps. @@ -130,7 +112,9 @@ for smaller smoke runs or larger throughput sweeps. ## Common AFD configuration Every AFD worker is wired through `--additional-config` with the same -shape; `role` differs between attention and FFN: +shape; `role` differs between attention and FFN. The colocation launcher +builds it from `LAYOUT` (the disaggregation scripts write it inline, on port +6269); print it with `AFD_DRY_RUN=1`: ```jsonc { @@ -138,9 +122,11 @@ shape; `role` differs between attention and FFN: "role": "attention", // or "ffn" "connector": "P2pNcclAFDConnector", "host": "127.0.0.1", - "port": 6269, - "num_attention_ranks": 1, // 2 in 2A2F - "num_ffn_ranks": 1 // 2 in 2A2F + "port": 6239, + "num_attention_ranks": 2, // Attention GPUs in LAYOUT + "num_ffn_ranks": 2, // FFN GPUs in LAYOUT + "compute_gate_on_attention": false, + "afd_process_group_timeout_s": 3600 } } ``` @@ -150,16 +136,6 @@ DBO (Dual Batch Overlap) is turned on for all examples with These threshold values are example defaults; tune them for the actual workload and deployment configuration. -### Switching eager → graph - -Graph mode replaces `--enforce-eager` with: - -```bash ---max-cudagraph-capture-size 64 ---compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", - "cudagraph_capture_sizes":[64]}' -``` - -The capture size of `64` is an example value; adjust -`--max-cudagraph-capture-size` and `cudagraph_capture_sizes` together based on -the actual batch sizes and available GPU memory. +`dbo.sh` captures `FULL_DECODE_ONLY` CUDA graphs of size `64` +(`CUDAGRAPH_CAPTURE_SIZES=64`); `eager_dbo.sh` runs `--enforce-eager`. Adjust +`CUDAGRAPH_CAPTURE_SIZES` to the actual batch sizes and available GPU memory. diff --git a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_eager_dbo_dp1tp2.sh b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_eager_dbo_dp1tp2.sh deleted file mode 100644 index 1cac638bc..000000000 --- a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_eager_dbo_dp1tp2.sh +++ /dev/null @@ -1,55 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project - -MODEL_PATH=${MODEL_PATH:-/path/model_weights/DeepSeek-V2-Lite} -export VLLM_USE_V2_MODEL_RUNNER=1 - -CUDA_VISIBLE_DEVICES=0,1 uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size 1 \ - --tensor-parallel-size 2 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "attention", - "connector": "P2pNcclAFDConnector", - "host": "127.0.0.1", - "port": 6269, - "num_attention_ranks": 2, - "num_ffn_ranks": 2 - } - }' \ - --max-num-seqs 64 \ - --max-num-batched-tokens 64 \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 12 \ - --enforce-eager \ - --host 127.0.0.1 \ - --port 18305 \ - --trust-remote-code > attn.log 2>&1 & - -CUDA_VISIBLE_DEVICES=2,3 uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size 1 \ - --tensor-parallel-size 2 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "ffn", - "connector": "P2pNcclAFDConnector", - "host": "127.0.0.1", - "port": 6269, - "num_attention_ranks": 2, - "num_ffn_ranks": 2 - } - }' \ - --max-num-seqs 64 \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 12 \ - --max-num-batched-tokens 64 \ - --enforce-eager \ - --host 127.0.0.1 \ - --port 18306 \ - --trust-remote-code > ffn.log 2>&1 & - -wait diff --git a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_eager_dbo_dp2tp1.sh b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_eager_dbo_dp2tp1.sh deleted file mode 100644 index edbab3812..000000000 --- a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_eager_dbo_dp2tp1.sh +++ /dev/null @@ -1,55 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project - -MODEL_PATH=${MODEL_PATH:-/path/model_weights/DeepSeek-V2-Lite} -export VLLM_USE_V2_MODEL_RUNNER=1 - -CUDA_VISIBLE_DEVICES=0,1 uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size 2 \ - --tensor-parallel-size 1 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "attention", - "connector": "P2pNcclAFDConnector", - "host": "127.0.0.1", - "port": 6269, - "num_attention_ranks": 2, - "num_ffn_ranks": 2 - } - }' \ - --max-num-seqs 64 \ - --max-num-batched-tokens 64 \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 12 \ - --enforce-eager \ - --host 127.0.0.1 \ - --port 18305 \ - --trust-remote-code > attn.log 2>&1 & - -CUDA_VISIBLE_DEVICES=2,3 uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size 2 \ - --tensor-parallel-size 1 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "ffn", - "connector": "P2pNcclAFDConnector", - "host": "127.0.0.1", - "port": 6269, - "num_attention_ranks": 2, - "num_ffn_ranks": 2 - } - }' \ - --max-num-seqs 64 \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 12 \ - --max-num-batched-tokens 64 \ - --enforce-eager \ - --host 127.0.0.1 \ - --port 18306 \ - --trust-remote-code > ffn.log 2>&1 & - -wait diff --git a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp1tp2.sh b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp1tp2.sh deleted file mode 100644 index e72893d22..000000000 --- a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp1tp2.sh +++ /dev/null @@ -1,61 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project - -MODEL_PATH=${MODEL_PATH:-/path/model_weights/DeepSeek-V2-Lite} -export VLLM_USE_V2_MODEL_RUNNER=1 - -CUDA_VISIBLE_DEVICES=0,1 uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size 1 \ - --tensor-parallel-size 2 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "attention", - "connector": "P2pNcclAFDConnector", - "host": "127.0.0.1", - "port": 6269, - "num_attention_ranks": 2, - "num_ffn_ranks": 2 - } - }' \ - --max-num-seqs 64 \ - --max-num-batched-tokens 64 \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 12 \ - --max-cudagraph-capture-size 64 \ - --compilation-config '{ - "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[64] - }' \ - --host 127.0.0.1 \ - --port 18305 \ - --trust-remote-code > attn.log 2>&1 & - -CUDA_VISIBLE_DEVICES=2,3 uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size 1 \ - --tensor-parallel-size 2 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "ffn", - "connector": "P2pNcclAFDConnector", - "host": "127.0.0.1", - "port": 6269, - "num_attention_ranks": 2, - "num_ffn_ranks": 2 - } - }' \ - --max-num-seqs 64 \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 12 \ - --max-num-batched-tokens 64 \ - --max-cudagraph-capture-size 64 \ - --compilation-config '{ - "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[64] - }' \ - --host 127.0.0.1 \ - --port 18306 \ - --trust-remote-code > ffn.log 2>&1 & - -wait diff --git a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp2tp1.sh b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp2tp1.sh deleted file mode 100644 index c775780db..000000000 --- a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp2tp1.sh +++ /dev/null @@ -1,61 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project - -MODEL_PATH=${MODEL_PATH:-/path/model_weights/DeepSeek-V2-Lite} -export VLLM_USE_V2_MODEL_RUNNER=1 - -CUDA_VISIBLE_DEVICES=0,1 uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size 2 \ - --tensor-parallel-size 1 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "attention", - "connector": "P2pNcclAFDConnector", - "host": "127.0.0.1", - "port": 6269, - "num_attention_ranks": 2, - "num_ffn_ranks": 2 - } - }' \ - --max-num-seqs 64 \ - --max-num-batched-tokens 64 \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 12 \ - --max-cudagraph-capture-size 64 \ - --compilation-config '{ - "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[64] - }' \ - --host 127.0.0.1 \ - --port 18305 \ - --trust-remote-code > attn.log 2>&1 & - -CUDA_VISIBLE_DEVICES=2,3 uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size 2 \ - --tensor-parallel-size 1 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "ffn", - "connector": "P2pNcclAFDConnector", - "host": "127.0.0.1", - "port": 6269, - "num_attention_ranks": 2, - "num_ffn_ranks": 2 - } - }' \ - --max-num-seqs 64 \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 12 \ - --max-num-batched-tokens 64 \ - --max-cudagraph-capture-size 64 \ - --compilation-config '{ - "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[64] - }' \ - --host 127.0.0.1 \ - --port 18306 \ - --trust-remote-code > ffn.log 2>&1 & - -wait diff --git a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/4a4f_eager_dbo_dp2tp2.sh b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/4a4f_eager_dbo_dp2tp2.sh deleted file mode 100644 index e2cbf9e71..000000000 --- a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/4a4f_eager_dbo_dp2tp2.sh +++ /dev/null @@ -1,55 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project - -MODEL_PATH=${MODEL_PATH:-/path/model_weights/DeepSeek-V2-Lite} -export VLLM_USE_V2_MODEL_RUNNER=1 - -CUDA_VISIBLE_DEVICES=0,1,2,3 uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size 2 \ - --tensor-parallel-size 2 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "attention", - "connector": "P2pNcclAFDConnector", - "host": "127.0.0.1", - "port": 6269, - "num_attention_ranks": 4, - "num_ffn_ranks": 4 - } - }' \ - --max-num-seqs 64 \ - --max-num-batched-tokens 64 \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 12 \ - --enforce-eager \ - --host 127.0.0.1 \ - --port 18305 \ - --trust-remote-code > attn.log 2>&1 & - -CUDA_VISIBLE_DEVICES=4,5,6,7 uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size 2 \ - --tensor-parallel-size 2 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "ffn", - "connector": "P2pNcclAFDConnector", - "host": "127.0.0.1", - "port": 6269, - "num_attention_ranks": 4, - "num_ffn_ranks": 4 - } - }' \ - --max-num-seqs 64 \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 12 \ - --max-num-batched-tokens 64 \ - --enforce-eager \ - --host 127.0.0.1 \ - --port 18306 \ - --trust-remote-code > ffn.log 2>&1 & - -wait diff --git a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/4a4f_graph_dbo_dp2tp2.sh b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/4a4f_graph_dbo_dp2tp2.sh deleted file mode 100644 index cd87ccbb4..000000000 --- a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/4a4f_graph_dbo_dp2tp2.sh +++ /dev/null @@ -1,61 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project - -MODEL_PATH=${MODEL_PATH:-/path/model_weights/DeepSeek-V2-Lite} -export VLLM_USE_V2_MODEL_RUNNER=1 - -CUDA_VISIBLE_DEVICES=0,1,2,3 uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size 2 \ - --tensor-parallel-size 2 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "attention", - "connector": "P2pNcclAFDConnector", - "host": "127.0.0.1", - "port": 6269, - "num_attention_ranks": 4, - "num_ffn_ranks": 4 - } - }' \ - --max-num-seqs 64 \ - --max-num-batched-tokens 64 \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 12 \ - --max-cudagraph-capture-size 64 \ - --compilation-config '{ - "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[64] - }' \ - --host 127.0.0.1 \ - --port 18305 \ - --trust-remote-code > attn.log 2>&1 & - -CUDA_VISIBLE_DEVICES=4,5,6,7 uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size 2 \ - --tensor-parallel-size 2 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "ffn", - "connector": "P2pNcclAFDConnector", - "host": "127.0.0.1", - "port": 6269, - "num_attention_ranks": 4, - "num_ffn_ranks": 4 - } - }' \ - --max-num-seqs 64 \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 12 \ - --max-num-batched-tokens 64 \ - --max-cudagraph-capture-size 64 \ - --compilation-config '{ - "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[64] - }' \ - --host 127.0.0.1 \ - --port 18306 \ - --trust-remote-code > ffn.log 2>&1 & - -wait diff --git a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/dbo.sh b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/dbo.sh new file mode 100644 index 000000000..f2af47086 --- /dev/null +++ b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/dbo.sh @@ -0,0 +1,21 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project +# +# DeepSeek-V2-Lite (BF16) on P2pNcclAFDConnector, FULL_DECODE_ONLY graphs + DBO. +# Placement: one pod per comma-separated LAYOUT entry (default 2A2F), pod k +# running POD=AFD_ with the same LAYOUT; ATTN_TP/FFN_TP set the TP size +# of each role, e.g. ATTN_TP=2 FFN_TP=2 for DP1 x TP2, LAYOUT=4A4F ATTN_TP=2 +# FFN_TP=2 on 8 GPUs, or LAYOUT=2A0F,0A2F ATTN_TP=2 FFN_TP=2 on two pods. +# Settings and running in a pod: recipe/gpu/common/README.md +# shellcheck disable=SC1090,SC2034 # the sourced launcher reads these variables + +AFD_CONNECTOR=${AFD_CONNECTOR:-P2pNcclAFDConnector} +LAYOUT=${LAYOUT:-2A2F} +MODEL_PATH=${MODEL_PATH:-/path/model_weights/DeepSeek-V2-Lite} +MODE=graph CUDAGRAPH_CAPTURE_SIZES=${CUDAGRAPH_CAPTURE_SIZES:-64} # ascending +SERVE_ARGS=(--max-num-seqs 64 --max-num-batched-tokens 64 + --enable-dbo --dbo-decode-token-threshold 2 --dbo-prefill-token-threshold 12) + +source "${AFD_RECIPE_LIB:-$(dirname "${BASH_SOURCE[0]}")/../../../common/afd_colocation.sh}" +afd_colocation_serve "$@" diff --git a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/eager_dbo.sh b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/eager_dbo.sh new file mode 100644 index 000000000..5d803ec70 --- /dev/null +++ b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/eager_dbo.sh @@ -0,0 +1,21 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project +# +# DeepSeek-V2-Lite (BF16) on P2pNcclAFDConnector, eager mode + DBO. +# Placement: one pod per comma-separated LAYOUT entry (default 2A2F), pod k +# running POD=AFD_ with the same LAYOUT; ATTN_TP/FFN_TP set the TP size +# of each role, e.g. ATTN_TP=2 FFN_TP=2 for DP1 x TP2, LAYOUT=4A4F ATTN_TP=2 +# FFN_TP=2 on 8 GPUs, or LAYOUT=2A0F,0A2F ATTN_TP=2 FFN_TP=2 on two pods. +# Settings and running in a pod: recipe/gpu/common/README.md +# shellcheck disable=SC1090,SC2034 # the sourced launcher reads these variables + +AFD_CONNECTOR=${AFD_CONNECTOR:-P2pNcclAFDConnector} +LAYOUT=${LAYOUT:-2A2F} +MODEL_PATH=${MODEL_PATH:-/path/model_weights/DeepSeek-V2-Lite} +MODE=eager +SERVE_ARGS=(--max-num-seqs 64 --max-num-batched-tokens 64 + --enable-dbo --dbo-decode-token-threshold 2 --dbo-prefill-token-threshold 12) + +source "${AFD_RECIPE_LIB:-$(dirname "${BASH_SOURCE[0]}")/../../../common/afd_colocation.sh}" +afd_colocation_serve "$@" diff --git a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/multipod_2a_2f_graph_dbo_dp1tp2.sh b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/multipod_2a_2f_graph_dbo_dp1tp2.sh deleted file mode 100644 index b1a1f0b14..000000000 --- a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/multipod_2a_2f_graph_dbo_dp1tp2.sh +++ /dev/null @@ -1,73 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project - -# Use .agents/skills/deploy-afd-k8s/SKILL.md to deploy this recipe -# on 2 pods using the setup below. -# For pod 1: POD=ATTENTION_0 -# For pod 2: POD=FFN_0 - -MODEL_PATH=${MODEL_PATH:-/path/model_weights/DeepSeek-V2-Lite} -export VLLM_USE_V2_MODEL_RUNNER=1 - -AFD_CONNECTOR_HOST=${AFD_CONNECTOR_HOST:-vllm-ffn-p2p-service} -AFD_CONNECTOR_PORT=${AFD_CONNECTOR_PORT:-6269} - -if [ "$POD" == "ATTENTION_0" ]; then - CUDA_VISIBLE_DEVICES=0,1 uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size 1 \ - --tensor-parallel-size 2 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "attention", - "connector": "P2pNcclAFDConnector", - "host": "'"${AFD_CONNECTOR_HOST}"'", - "port": '"${AFD_CONNECTOR_PORT}"', - "num_attention_ranks": 2, - "num_ffn_ranks": 2 - } - }' \ - --max-num-seqs 64 \ - --max-num-batched-tokens 64 \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 12 \ - --max-cudagraph-capture-size 64 \ - --compilation-config '{ - "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[64] - }' \ - --host 127.0.0.1 \ - --port 18305 \ - --trust-remote-code > attn.log 2>&1 & -fi - -if [ "$POD" == "FFN_0" ]; then - CUDA_VISIBLE_DEVICES=0,1 uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size 1 \ - --tensor-parallel-size 2 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "ffn", - "connector": "P2pNcclAFDConnector", - "host": "'"${AFD_CONNECTOR_HOST}"'", - "port": '"${AFD_CONNECTOR_PORT}"', - "num_attention_ranks": 2, - "num_ffn_ranks": 2 - } - }' \ - --max-num-seqs 64 \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 12 \ - --max-num-batched-tokens 64 \ - --max-cudagraph-capture-size 64 \ - --compilation-config '{ - "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[64] - }' \ - --host 127.0.0.1 \ - --port 18305 \ - --trust-remote-code > ffn.log 2>&1 & -fi - -wait diff --git a/recipe/gpu/P2pNcclAFDConnector/qwen3_235b_a22b_fp8/README.md b/recipe/gpu/P2pNcclAFDConnector/qwen3_235b_a22b_fp8/README.md new file mode 100644 index 000000000..9cc63e56b --- /dev/null +++ b/recipe/gpu/P2pNcclAFDConnector/qwen3_235b_a22b_fp8/README.md @@ -0,0 +1,17 @@ +# Qwen3-235B-A22B-FP8 AFD Recipe (P2pNcclAFDConnector) + +`prefill_decode_colocation/graph.sh` runs Qwen3-235B-A22B-FP8 with +`FULL_DECODE_ONLY` CUDA graphs on `P2pNcclAFDConnector`, with the same +settings as the +[`DeepEPM2NAFDConnector` graph recipe](../../DeepEPM2NAFDConnector/qwen3_235b_a22b_fp8/README.md), +in the 4A2F layout by default (one 6-GPU node). The layout, settings and +running in a pod are documented in the +[colocation launcher README](../../common/README.md). + +```bash +export MODEL_PATH=/path/model_weights/Qwen3-235B-A22B-FP8 +bash recipe/gpu/P2pNcclAFDConnector/qwen3_235b_a22b_fp8/prefill_decode_colocation/graph.sh +``` + +Do not set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` for this +connector's FFN: the FFN hangs on its first step. diff --git a/recipe/gpu/P2pNcclAFDConnector/qwen3_235b_a22b_fp8/prefill_decode_colocation/graph.sh b/recipe/gpu/P2pNcclAFDConnector/qwen3_235b_a22b_fp8/prefill_decode_colocation/graph.sh new file mode 100644 index 000000000..cd24d2468 --- /dev/null +++ b/recipe/gpu/P2pNcclAFDConnector/qwen3_235b_a22b_fp8/prefill_decode_colocation/graph.sh @@ -0,0 +1,20 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project +# +# Qwen3-235B-A22B-FP8 on P2pNcclAFDConnector, FULL_DECODE_ONLY CUDA graphs. +# Placement: one pod per comma-separated LAYOUT entry (default 4A2F), pod k +# running POD=AFD_ with the same LAYOUT; the default is one +# 6-GPU node. Do not set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True: the +# P2P FFN hangs on its first step. +# Settings and running in a pod: recipe/gpu/common/README.md +# shellcheck disable=SC1090,SC2034 # the sourced launcher reads these variables + +AFD_CONNECTOR=${AFD_CONNECTOR:-P2pNcclAFDConnector} +LAYOUT=${LAYOUT:-4A2F} +MODEL_PATH=${MODEL_PATH:-/path/model_weights/Qwen3-235B-A22B-FP8} +MODE=graph CUDAGRAPH_CAPTURE_SIZES=${CUDAGRAPH_CAPTURE_SIZES:-8,12,16,24,32,48,64,96,128,192,256} # ascending +SERVE_ARGS=(--enable-ep-weight-filter --max-num-seqs 256 --max-num-batched-tokens 3072 --max-model-len 4096) + +source "${AFD_RECIPE_LIB:-$(dirname "${BASH_SOURCE[0]}")/../../../common/afd_colocation.sh}" +afd_colocation_serve "$@" diff --git a/recipe/gpu/P2pNcclAFDConnector/qwen3_30b_a3b/README.md b/recipe/gpu/P2pNcclAFDConnector/qwen3_30b_a3b/README.md new file mode 100644 index 000000000..d3692de3a --- /dev/null +++ b/recipe/gpu/P2pNcclAFDConnector/qwen3_30b_a3b/README.md @@ -0,0 +1,13 @@ +# Qwen3-30B-A3B AFD Recipe (P2pNcclAFDConnector) + +`prefill_decode_colocation/graph.sh` runs Qwen3-30B-A3B with +`FULL_DECODE_ONLY` CUDA graphs on `P2pNcclAFDConnector`, with the same +settings as the +[`DeepEPM2NAFDConnector` graph recipe](../../DeepEPM2NAFDConnector/qwen3_30b_a3b/README.md) +(default layout 2A2F). The layout, settings and running in a pod are +documented in the [colocation launcher README](../../common/README.md). + +```bash +export MODEL_PATH=/path/model_weights/Qwen3-30B-A3B +bash recipe/gpu/P2pNcclAFDConnector/qwen3_30b_a3b/prefill_decode_colocation/graph.sh +``` diff --git a/recipe/gpu/P2pNcclAFDConnector/qwen3_30b_a3b/prefill_decode_colocation/graph.sh b/recipe/gpu/P2pNcclAFDConnector/qwen3_30b_a3b/prefill_decode_colocation/graph.sh new file mode 100644 index 000000000..9b1dd03c5 --- /dev/null +++ b/recipe/gpu/P2pNcclAFDConnector/qwen3_30b_a3b/prefill_decode_colocation/graph.sh @@ -0,0 +1,18 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project +# +# Qwen3-30B-A3B (BF16) on P2pNcclAFDConnector, FULL_DECODE_ONLY CUDA graphs. +# Placement: one pod per comma-separated LAYOUT entry (default 2A2F), pod k +# running POD=AFD_ with the same LAYOUT; FFN on each pod's first GPUs. +# Settings and running in a pod: recipe/gpu/common/README.md +# shellcheck disable=SC1090,SC2034 # the sourced launcher reads these variables + +AFD_CONNECTOR=${AFD_CONNECTOR:-P2pNcclAFDConnector} +LAYOUT=${LAYOUT:-2A2F} +MODEL_PATH=${MODEL_PATH:-/path/model_weights/Qwen3-30B-A3B} +MODE=graph CUDAGRAPH_CAPTURE_SIZES=${CUDAGRAPH_CAPTURE_SIZES:-8,16,32,64} # ascending +SERVE_ARGS=(--enable-ep-weight-filter --max-num-seqs 64 --max-num-batched-tokens 4096 --max-model-len 4096) + +source "${AFD_RECIPE_LIB:-$(dirname "${BASH_SOURCE[0]}")/../../../common/afd_colocation.sh}" +afd_colocation_serve "$@" diff --git a/recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/graph.sh b/recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/graph.sh new file mode 100644 index 000000000..5582c73ee --- /dev/null +++ b/recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/graph.sh @@ -0,0 +1,21 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project +# +# Qwen3.5-122B-A10B-FP8 on P2pNcclAFDConnector, FULL_DECODE_ONLY CUDA graphs. +# Placement: one pod per comma-separated LAYOUT entry (default 2A0F,2A0F,0A2F), pod k +# running POD=AFD_ with the same LAYOUT; the default runs +# 4A2F on three 2-GPU pods (Attention DP4 over pods 0-1, FFN DP2 on pod 2); +# LAYOUT=4A2F runs it on one 6-GPU node. +# Settings and running in a pod: recipe/gpu/common/README.md +# shellcheck disable=SC1090,SC2034 # the sourced launcher reads these variables + +AFD_CONNECTOR=${AFD_CONNECTOR:-P2pNcclAFDConnector} +LAYOUT=${LAYOUT:-2A0F,2A0F,0A2F} +MODEL_PATH=${MODEL_PATH:-/path/model_weights/Qwen3.5-122B-A10B-FP8} +MODE=graph CUDAGRAPH_CAPTURE_SIZES=${CUDAGRAPH_CAPTURE_SIZES:-32} # ascending +SERVE_ARGS=(--dtype bfloat16 --language-model-only --max-model-len 114688 --mamba-cache-mode align + --all2all-backend allgather_reducescatter --seed 0 --max-num-seqs 32 --max-num-batched-tokens 8192) + +source "${AFD_RECIPE_LIB:-$(dirname "${BASH_SOURCE[0]}")/../../../common/afd_colocation.sh}" +afd_colocation_serve "$@" diff --git a/recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/multipod_2a_2a_2f_graph.sh b/recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/multipod_2a_2a_2f_graph.sh deleted file mode 100644 index ec63badbd..000000000 --- a/recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/multipod_2a_2a_2f_graph.sh +++ /dev/null @@ -1,100 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project - -# Use .agents/skills/deploy-afd-k8s/SKILL.md to deploy this recipe -# on 3 pods using the setup below. -# For pod 1: POD=ATTENTION_0 -# For pod 2: POD=ATTENTION_1, ATTENTION_HEADLESS=1, ATTENTION_DP_START_RANK=2 -# For pod 3: POD=FFN_0 - -MODEL_PATH=${MODEL_PATH:-/path/model_weights/Qwen3.5-122B-A10B-FP8} -export VLLM_USE_V2_MODEL_RUNNER=1 - -AFD_CONNECTOR_HOST=${AFD_CONNECTOR_HOST:-vllm-ffn-p2p-service} -AFD_CONNECTOR_PORT=${AFD_CONNECTOR_PORT:-1239} - -ATTENTION_DP_START_RANK=${ATTENTION_DP_START_RANK:-0} -ATTENTION_HEADLESS=${ATTENTION_HEADLESS:-0} -ATTENTION_DP_ADDRESS=${ATTENTION_DP_ADDRESS:-vllm-attn-dp-service} -ATTENTION_DP_RPC_PORT=${ATTENTION_DP_RPC_PORT:-13345} - -HEADLESS_ARGS=() -if [[ "$ATTENTION_HEADLESS" == "1" ]]; then - HEADLESS_ARGS=(--headless) -fi - -DP_START_RANK_ARGS=() -if [[ "$ATTENTION_DP_START_RANK" != "0" ]]; then - DP_START_RANK_ARGS=(--data-parallel-start-rank "$ATTENTION_DP_START_RANK") -fi - -if [[ "$POD" == ATTENTION_* ]]; then - CUDA_VISIBLE_DEVICES=0,1 uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size 4 \ - --data-parallel-size-local 2 \ - "${DP_START_RANK_ARGS[@]}" \ - --data-parallel-address "$ATTENTION_DP_ADDRESS" \ - --data-parallel-rpc-port "$ATTENTION_DP_RPC_PORT" \ - --tensor-parallel-size 1 \ - --enable-expert-parallel \ - --dtype bfloat16 \ - --language-model-only \ - --max-model-len 114688 \ - --mamba-cache-mode align \ - --all2all-backend allgather_reducescatter \ - --seed 0 \ - --additional-config '{ - "afd": { - "role": "attention", - "connector": "P2pNcclAFDConnector", - "afd_process_group_timeout_s": 120, - "host": "'"${AFD_CONNECTOR_HOST}"'", - "port": '"${AFD_CONNECTOR_PORT}"', - "num_attention_ranks": 4, - "num_ffn_ranks": 2 - } - }' \ - --max-num-seqs 32 \ - --max-num-batched-tokens 8192 \ - --max-cudagraph-capture-size 32 \ - --compilation-config '{ - "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[32] - }' \ - "${HEADLESS_ARGS[@]}" \ - --host 127.0.0.1 \ - --port 18305 > attn.log 2>&1 & -fi - -if [[ "$POD" == FFN_* ]]; then - CUDA_VISIBLE_DEVICES=0,1 uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size 2 \ - --tensor-parallel-size 1 \ - --enable-expert-parallel \ - --dtype bfloat16 \ - --language-model-only \ - --max-model-len 114688 \ - --mamba-cache-mode align \ - --all2all-backend allgather_reducescatter \ - --seed 0 \ - --additional-config '{ - "afd": { - "role": "ffn", - "connector": "P2pNcclAFDConnector", - "afd_process_group_timeout_s": 120, - "host": "'"${AFD_CONNECTOR_HOST}"'", - "port": '"${AFD_CONNECTOR_PORT}"', - "num_attention_ranks": 4, - "num_ffn_ranks": 2 - } - }' \ - --max-num-seqs 32 \ - --max-num-batched-tokens 8192 \ - --max-cudagraph-capture-size 32 \ - --compilation-config '{ - "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[32] - }' \ - --host 127.0.0.1 \ - --port 18305 > ffn.log 2>&1 & -fi - -wait diff --git a/recipe/gpu/baseline/qwen3_235b_a22b_fp8/vanilla_tp4.sh b/recipe/gpu/baseline/qwen3_235b_a22b_fp8/vanilla_tp4.sh new file mode 100755 index 000000000..deda834a9 --- /dev/null +++ b/recipe/gpu/baseline/qwen3_235b_a22b_fp8/vanilla_tp4.sh @@ -0,0 +1,93 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project +# +# Qwen3-235B-A22B-FP8 vanilla vLLM baseline on one 4-GPU node. +# +# This is the no-AFD counterpart to +# recipe/gpu/DeepEPM2NAFDConnector/qwen3_235b_a22b_fp8/prefill_decode_colocation/graph.sh. +# It launches a single vLLM process with tensor parallelism 4 and no AFD +# connector, AFD additional config, or separate Attention/FFN roles. +# +# Every knob is an environment variable: +# +# MODEL_PATH weights dir or HF id (/path/model_weights/Qwen3-235B-A22B-FP8) +# TENSOR_PARALLEL_SIZE tensor-parallel size (4) +# CUDA_VISIBLE_DEVICES GPUs to use (0,1,2,3) +# HOST / PORT HTTP bind address and port (127.0.0.1 / 18000) +# MAX_NUM_SEQS maximum sequences (128) +# MAX_NUM_BATCHED_TOKENS prefill chunk per step (3072) +# MAX_MODEL_LEN maximum model length (34816) +# CUDAGRAPH_CAPTURE_SIZES comma-separated decode graph sizes +# (8,16,32,48,64,96,112,128) +# ENABLE_PREFIX_CACHING 1 | 0 (1) +# ENABLE_DBO 1 | 0 (0) +# DBO_DECODE_TOKEN_THRESHOLD / DBO_PREFILL_TOKEN_THRESHOLD (2 / 2) +# ALL2ALL_BACKEND MoE all-to-all backend (allgather_reducescatter); +# DBO requires DeepEP or NIXL instead +# GPU_MEMORY_UTILIZATION optional GPU memory utilization fraction (unset) +# EXTRA_ARGS extra `vllm serve` arguments (empty) + +set -euo pipefail + +MODEL_PATH=${MODEL_PATH:-/path/model_weights/Qwen3-235B-A22B-FP8} +TENSOR_PARALLEL_SIZE=${TENSOR_PARALLEL_SIZE:-4} +CUDA_VISIBLE_DEVICES=${CUDA_VISIBLE_DEVICES:-0,1,2,3} +HOST=${HOST:-127.0.0.1} +PORT=${PORT:-18000} +MAX_NUM_SEQS=${MAX_NUM_SEQS:-128} +MAX_NUM_BATCHED_TOKENS=${MAX_NUM_BATCHED_TOKENS:-3072} +MAX_MODEL_LEN=${MAX_MODEL_LEN:-34816} +CUDAGRAPH_CAPTURE_SIZES=${CUDAGRAPH_CAPTURE_SIZES:-8,16,32,48,64,96,112,128} +ENABLE_PREFIX_CACHING=${ENABLE_PREFIX_CACHING:-1} +ENABLE_DBO=${ENABLE_DBO:-0} +DBO_DECODE_TOKEN_THRESHOLD=${DBO_DECODE_TOKEN_THRESHOLD:-2} +DBO_PREFILL_TOKEN_THRESHOLD=${DBO_PREFILL_TOKEN_THRESHOLD:-2} +ALL2ALL_BACKEND=${ALL2ALL_BACKEND:-allgather_reducescatter} +GPU_MEMORY_UTILIZATION=${GPU_MEMORY_UTILIZATION:-} +EXTRA_ARGS=${EXTRA_ARGS:-} + +export VLLM_USE_V2_MODEL_RUNNER=1 +export CUDA_VISIBLE_DEVICES + +MAX_CUDAGRAPH_CAPTURE_SIZE=$(tr ',' '\n' <<<"$CUDAGRAPH_CAPTURE_SIZES" | sort -n | tail -n 1) + +ARGS=( + --tensor-parallel-size "$TENSOR_PARALLEL_SIZE" + --enable-expert-parallel + --enable-ep-weight-filter + --all2all-backend "$ALL2ALL_BACKEND" + --max-num-seqs "$MAX_NUM_SEQS" + --max-num-batched-tokens "$MAX_NUM_BATCHED_TOKENS" + --max-model-len "$MAX_MODEL_LEN" + --max-cudagraph-capture-size "$MAX_CUDAGRAPH_CAPTURE_SIZE" + --compilation-config "{\"cudagraph_mode\": \"FULL_DECODE_ONLY\", \"cudagraph_capture_sizes\": [${CUDAGRAPH_CAPTURE_SIZES}]}" + --cpu-distributed-timeout-seconds 3600 + --trust-remote-code +) + +if [[ "$ENABLE_PREFIX_CACHING" == 1 ]]; then + ARGS+=(--enable-prefix-caching) +else + ARGS+=(--no-enable-prefix-caching) +fi + +if [[ "$ENABLE_DBO" == 1 ]]; then + ARGS+=( + --enable-dbo + --dbo-decode-token-threshold "$DBO_DECODE_TOKEN_THRESHOLD" + --dbo-prefill-token-threshold "$DBO_PREFILL_TOKEN_THRESHOLD" + ) +fi + +if [[ -n "$GPU_MEMORY_UTILIZATION" ]]; then + ARGS+=(--gpu-memory-utilization "$GPU_MEMORY_UTILIZATION") +fi + +# shellcheck disable=SC2206 +ARGS+=($EXTRA_ARGS) + +exec vllm serve "$MODEL_PATH" \ + "${ARGS[@]}" \ + --host "$HOST" \ + --port "$PORT" diff --git a/recipe/gpu/common/README.md b/recipe/gpu/common/README.md new file mode 100644 index 000000000..f0c3cf5c1 --- /dev/null +++ b/recipe/gpu/common/README.md @@ -0,0 +1,156 @@ +# Shared launcher for the GPU colocation recipes + +Every GPU `prefill_decode_colocation` recipe (`DeepEPM2NAFDConnector` and +`P2pNcclAFDConnector`) is a short script that states a model, a connector, a +default layout and its `vllm serve` flags, and then hands over to +[`afd_colocation.sh`](afd_colocation.sh). The launcher owns everything the +recipes share: the placement (`LAYOUT` and `POD` to GPUs and data-parallel +flags), the `afd` additional config, the decode-only benchmark hook, the +override variables, and the two `vllm serve` processes. + +## Writing a recipe + +```bash +#!/usr/bin/env bash +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project +# +# Qwen3-30B-A3B (BF16) on DeepEPM2NAFDConnector, FULL_DECODE_ONLY graphs + DBO. +# Placement: one pod per comma-separated LAYOUT entry (default 2A2F), pod k +# running POD=AFD_ with the same LAYOUT; FFN on each pod's first GPUs. +# Settings and running in a pod: recipe/gpu/common/README.md +# shellcheck disable=SC1090,SC2034 # the sourced launcher reads these variables + +AFD_CONNECTOR=${AFD_CONNECTOR:-DeepEPM2NAFDConnector} +LAYOUT=${LAYOUT:-2A2F} +MODEL_PATH=${MODEL_PATH:-/path/model_weights/Qwen3-30B-A3B} +MODE=graph CUDAGRAPH_CAPTURE_SIZES=${CUDAGRAPH_CAPTURE_SIZES:-8,16,32,64} # ascending +SERVE_ARGS=(--enable-ep-weight-filter --max-num-seqs 64 --max-num-batched-tokens 4096 --max-model-len 4096 + --enable-dbo --dbo-decode-token-threshold 2 --dbo-prefill-token-threshold 8) + +source "${AFD_RECIPE_LIB:-$(dirname "${BASH_SOURCE[0]}")/../../../common/afd_colocation.sh}" +afd_colocation_serve "$@" +``` + +| Recipe variable | Meaning | +| --- | --- | +| `AFD_CONNECTOR` | Connector class. `P2pNcclAFDConnector` routes on the FFN and drops `CONNECTOR_CONFIG`; any other connector computes the gate on Attention. | +| `LAYOUT` | Default placement; see [Layout](#layout). | +| `MODEL_PATH` | Default weights directory or HF id. | +| `MODE` | `graph` (`FULL_DECODE_ONLY` CUDA graphs of `CUDAGRAPH_CAPTURE_SIZES`, the largest being the maximum capture size) or `eager` (`--enforce-eager`). | +| `SERVE_ARGS` | Array of model and tuning flags passed to both roles (DBO, `--enable-ep-weight-filter`, batch limits, dtype, ...). | +| `CONNECTOR_CONFIG` | Optional `connector_extra_config` members, for example `'"payload_format": "fp8_block128_fp32"'`. | + +The launcher always adds `--enable-expert-parallel`, `--trust-remote-code`, +`--cpu-distributed-timeout-seconds 3600` and `afd_process_group_timeout_s: +3600` (the roles can finish loading minutes apart). Its ports are fixed: +`18000` Attention API, `18001` FFN (serves no requests), `6239` AFD +rendezvous, `13345` and `13355` the Attention and FFN DP RPC ports across +pods. The default launcher path assumes the recipe sits at +`recipe/gpu////`. + +## Layout + +`LAYOUT` lists the pods in order, one `AF` entry each: `a` Attention +GPUs and `f` FFN GPUs. Inside a pod, the FFN takes the first `f` GPUs and the +Attention the next `a`; every pod runs the same script with the same +`LAYOUT` and its own `POD=AFD_` (`k` counts from 0; one pod needs no +`POD`). + +| `LAYOUT` | Pods | Per pod | +| --- | --- | --- | +| `2A2F` | 1 | FFN DP2 on GPUs 0,1, Attention DP2 on GPUs 2,3 | +| `4A2F` | 1 | FFN on GPUs 0,1, Attention DP4 on GPUs 2-5 | +| `2A0F,0A2F` | 2 | pod 0: Attention DP2; pod 1: FFN DP2 | +| `1A1F,1A1F` | 2 | FFN on GPU 0, Attention on GPU 1 | +| `2A0F,2A0F,0A2F` | 3 | pods 0-1: Attention DP4 (pod 1 headless); pod 2: FFN DP2 | + +`ATTN_TP` and `FFN_TP` split each pod's GPUs of that role into TP groups (they +must divide each pod's `a` or `f`); `num_attention_ranks` and `num_ffn_ranks` +still count GPUs. A role held by one pod gets only `--data-parallel-size`; a +role spread over pods is led by its first pod, and the others start headless +from their start rank. Each connector checks the placement at startup; see +its user guide for the layouts it accepts. + +## Settings + +Every setting is an environment variable; the defaults reproduce the recipe. + +| Variable | Default | Meaning | +| --- | --- | --- | +| `MODEL_PATH` | recipe | Weights directory or HF id. | +| `LAYOUT` | recipe | Pods and their GPUs; see [Layout](#layout). | +| `POD` | `AFD_0` | This pod's entry in `LAYOUT`, `AFD_`. | +| `ATTN_TP` / `FFN_TP` | `1` | TP size of the Attention / FFN role. | +| `CUDAGRAPH_CAPTURE_SIZES` | recipe | Ascending decode graph sizes (graph recipes). | +| `AFD_CONNECTOR` | recipe | Connector class, for example `P2pNcclAFDConnector` to run an M2N recipe on NCCL P2P. | +| `VLLM_ARGS` | empty | Extra `vllm serve` flags for both roles. | +| `ATTN_ARGS` / `FFN_ARGS` | empty | Extra `vllm serve` flags for one role. | +| `M2N_EXTRA_CONFIG` | empty | Extra `connector_extra_config` members, for example `'"expert_num_sms": 130'`. | +| `DECODE_BENCH` | empty | `kv_connector_extra_config` JSON of the decode-only benchmark connector on Attention (`{}` is enough); see [`tools/benchmarks`](../../../tools/benchmarks/). | +| `AFD_REPO` | current directory | Repository root that holds `tools/benchmarks` (with `DECODE_BENCH`). | +| `HEAD_HOST` | `127.0.0.1` on one pod | Address of pod 0 when it leads both roles (several pods, no Kubernetes). | +| `ATTN_HEAD_HOST` / `FFN_HEAD_HOST` | `HEAD_HOST`, else the Kubernetes Service names | Address of the first pod with Attention / FFN GPUs. The FFN head also hosts the AFD rendezvous. | +| `SERVE_HOST` | `127.0.0.1` | API server bind address (`0.0.0.0` to serve outside the pod). | +| `AFD_RECIPE_LIB` | next to the recipe in the repository | Path of `afd_colocation.sh`, for a recipe copied out of the repository. | +| `AFD_DRY_RUN` | empty | Print this pod's `vllm serve` commands instead of running them. | + +`VLLM_ARGS`, `ATTN_ARGS`, `FFN_ARGS` and any arguments after the script name +come after the recipe's flags, so a repeated flag overrides its default. They +are split on whitespace without shell quoting, so pass JSON without spaces: + +```bash +VLLM_ARGS="--max-num-seqs 512" ATTN_ARGS="--no-enable-prefix-caching" \ + bash recipe/gpu/DeepEPM2NAFDConnector/qwen3_30b_a3b/prefill_decode_colocation/graph.sh +``` + +## Running directly + +From any directory (logs go to the current directory): + +```bash +export MODEL_PATH=/path/model_weights/Qwen3-30B-A3B +bash recipe/gpu/DeepEPM2NAFDConnector/qwen3_30b_a3b/prefill_decode_colocation/graph.sh +``` + +The script starts the FFN and Attention servers in the background, writes +`ffn.log` and `attn.log`, and waits. Send traffic to port `18000` once +`attn.log` prints `Application startup complete`. To see what a pod would +run without starting anything: + +```bash +AFD_DRY_RUN=1 LAYOUT=2A0F,0A2F POD=AFD_1 bash .../graph.sh +``` + +Several hosts without Kubernetes run the same script with their own `POD` +and the leaders' addresses: + +```bash +LAYOUT=2A0F,0A2F POD=AFD_0 ATTN_HEAD_HOST= FFN_HEAD_HOST= bash .../graph.sh +LAYOUT=2A0F,0A2F POD=AFD_1 ATTN_HEAD_HOST= FFN_HEAD_HOST= bash .../graph.sh +``` + +## Running in a pod + +A recipe needs `afd_colocation.sh` next to it. Ship both files and point +`AFD_RECIPE_LIB` at the launcher: + +- **`deploy-afd-k8s` skill** (`.agents/skills/deploy-afd-k8s/`): puts both + files into the `afd-recipe-script` ConfigMap (`recipe.sh` and + `afd_colocation.sh`), mounts it at `/recipe`, and sets + `AFD_RECIPE_LIB=/recipe/afd_colocation.sh` and `SERVE_HOST=0.0.0.0`. It + creates one pod per `LAYOUT` entry and the Services that the default head + host names resolve to. Edited recipes need no image rebuild. +- **A pod you manage**: copy both files and run the recipe with the launcher + path: + + ```bash + kubectl cp recipe/gpu/common/afd_colocation.sh :/work/afd_colocation.sh + kubectl cp .sh :/work/recipe.sh + kubectl exec -- bash -c 'cd /work && SERVE_HOST=0.0.0.0 \ + AFD_RECIPE_LIB=/work/afd_colocation.sh MODEL_PATH=/models/ bash recipe.sh' + ``` + +- **An image with this repository** (`docker/Dockerfile.ci` puts it in + `/opt/afd-plugin`): run the recipe in place, as the image's copy finds its + launcher: `bash /opt/afd-plugin/recipe/gpu///prefill_decode_colocation/.sh`. diff --git a/recipe/gpu/common/afd_colocation.sh b/recipe/gpu/common/afd_colocation.sh new file mode 100644 index 000000000..2abf78226 --- /dev/null +++ b/recipe/gpu/common/afd_colocation.sh @@ -0,0 +1,116 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project +# +# Shared launcher of the GPU prefill_decode_colocation recipes: placement +# (LAYOUT/POD -> devices and DP flags), the AFD config and the two `vllm serve` +# processes. A recipe sets the variables below, sources this file and calls +# afd_colocation_serve "$@". See recipe/gpu/common/README.md. +# +# Set by the recipe: +# MODEL_PATH, AFD_CONNECTOR, LAYOUT model, connector class, default layout +# MODE graph (FULL_DECODE_ONLY CUDA graphs of +# CUDAGRAPH_CAPTURE_SIZES) or eager +# SERVE_ARGS array of model/tuning flags, both roles +# CONNECTOR_CONFIG optional connector_extra_config members +# Run settings (environment): POD, ATTN_TP, FFN_TP, VLLM_ARGS, ATTN_ARGS, +# FFN_ARGS, M2N_EXTRA_CONFIG, DECODE_BENCH, AFD_REPO, HEAD_HOST, +# ATTN_HEAD_HOST, FFN_HEAD_HOST, SERVE_HOST, AFD_DRY_RUN. + +ATTN_PORT=18000 FFN_PORT=18001 AFD_PORT=6239 ATTN_DP_RPC_PORT=13345 FFN_DP_RPC_PORT=13355 + +# role_dp TP DP_HOST DP_RPC_PORT ... -> TOTAL and this pod's DP +# flags in DP. A role spread over pods is led by its first pod; the others run +# headless from their start rank (the leader must omit it, or vLLM selects +# hybrid load balancing and rejects the headless peers). +role_dp() { + local tp=$1 host=$2 port=$3 i pods=0 start=0 lead=-1 + shift 3 + local n=("$@") + TOTAL=0 + for i in "${!n[@]}"; do + ((n[i])) || continue + ((n[i] % tp)) && { echo "TP $tp does not divide ${n[i]} ranks on pod $i" >&2; exit 1; } + ((lead < 0)) && lead=$i + ((i < K)) && start=$((start + n[i])) + pods=$((pods + 1)) TOTAL=$((TOTAL + n[i])) + done + DP=(--tensor-parallel-size "$tp" --data-parallel-size $((TOTAL / tp))) + ((pods > 1)) && DP+=(--data-parallel-size-local $((n[K] / tp)) + --data-parallel-address "$host" --data-parallel-rpc-port "$port") + ((pods > 1 && K != lead)) && DP+=(--headless --data-parallel-start-rank $((start / tp))) + return 0 +} + +afd_json() { # The AFD rendezvous runs on the pod that holds FFN rank 0. + cat <... +launch() { + local log=$1 devices + devices=$(seq "$2" $(($2 + $3 - 1)) | paste -sd, -) + shift 3 + if [[ -n ${AFD_DRY_RUN:-} ]]; then + echo "CUDA_VISIBLE_DEVICES=$devices vllm serve $(printf '%q ' "$@")> $log" + else + CUDA_VISIBLE_DEVICES=$devices vllm serve "$@" > "$log" 2>&1 & + fi +} + +afd_colocation_serve() { + export VLLM_USE_V2_MODEL_RUNNER=1 + local p + case ${MODE:-} in + graph) MODE_ARGS=(--max-cudagraph-capture-size "${CUDAGRAPH_CAPTURE_SIZES##*,}" + --compilation-config "{\"cudagraph_mode\": \"FULL_DECODE_ONLY\", \"cudagraph_capture_sizes\": [${CUDAGRAPH_CAPTURE_SIZES}]}") ;; + eager) MODE_ARGS=(--enforce-eager) ;; + *) echo "MODE must be graph or eager, got '${MODE:-}'" >&2; exit 1 ;; + esac + + GATE_ON_ATTENTION=true + CONNECTOR_CONFIG=${CONNECTOR_CONFIG:-}${M2N_EXTRA_CONFIG:+${CONNECTOR_CONFIG:+, }$M2N_EXTRA_CONFIG} + # P2P routes on the FFN and takes no connector_extra_config. + [[ $AFD_CONNECTOR == P2pNcclAFDConnector ]] && GATE_ON_ATTENTION=false CONNECTOR_CONFIG= + [[ -n $CONNECTOR_CONFIG ]] && CONNECTOR_CONFIG=", \"connector_extra_config\": {${CONNECTOR_CONFIG}}" + + IFS=, read -ra PODS <<<"$LAYOUT" + K=${POD:-AFD_0} && K=${K#AFD_} + if [[ ! $LAYOUT =~ ^[0-9]+A[0-9]+F(,[0-9]+A[0-9]+F)*$ || ! $K =~ ^[0-9]+$ ]] || ((K >= ${#PODS[@]})); then + echo "need LAYOUT=AF[,...] and POD=AFD_ in it: LAYOUT=$LAYOUT POD=${POD:-}" >&2 + exit 1 + fi + NUM_A=() NUM_F=() + for p in "${PODS[@]}"; do NUM_A+=("${p%%A*}") && p=${p#*A} && NUM_F+=("${p%F}"); done + ((${#PODS[@]} > 1)) || HEAD_HOST=${HEAD_HOST:-127.0.0.1} + + # Extra `vllm serve` flags go last, so they override the defaults. + read -ra ARGS <<<"${VLLM_ARGS:-}" && ARGS+=("$@") + read -ra ATTN_EXTRA <<<"${ATTN_ARGS:-}" + read -ra FFN_EXTRA <<<"${FFN_ARGS:-}" + if [[ -n ${DECODE_BENCH:-} ]]; then # tools/benchmarks is imported from the repository + ATTN_EXTRA=(--kv-transfer-config "{\"kv_connector\": \"AFDDecodeBenchConnector\", \"kv_connector_module_path\": \"tools.benchmarks.decode_bench\", \"kv_role\": \"kv_both\", \"kv_connector_extra_config\": ${DECODE_BENCH}}" "${ATTN_EXTRA[@]}") + export PYTHONPATH=${AFD_REPO:-$PWD}${PYTHONPATH:+:$PYTHONPATH} + fi + + role_dp "${FFN_TP:-1}" "${FFN_HEAD_HOST:-${HEAD_HOST:-vllm-ffn-dp-service}}" "$FFN_DP_RPC_PORT" "${NUM_F[@]}" + FFN_DP=("${DP[@]}") NUM_FFN=$TOTAL + role_dp "${ATTN_TP:-1}" "${ATTN_HEAD_HOST:-${HEAD_HOST:-vllm-attn-dp-service}}" "$ATTN_DP_RPC_PORT" "${NUM_A[@]}" + ATTN_DP=("${DP[@]}") NUM_ATTN=$TOTAL + + # The roles can finish loading minutes apart, hence the 3600 s timeouts. + COMMON=(--enable-expert-parallel --trust-remote-code --cpu-distributed-timeout-seconds 3600 + "${SERVE_ARGS[@]}" "${MODE_ARGS[@]}" --host "${SERVE_HOST:-127.0.0.1}") + + # FFN on this pod's GPUs [0, F), Attention on [F, F + A). + local F=${NUM_F[K]} A=${NUM_A[K]} + ((F)) && launch ffn.log 0 "$F" "$MODEL_PATH" --additional-config "$(afd_json ffn)" \ + "${COMMON[@]}" "${FFN_DP[@]}" "${ARGS[@]}" "${FFN_EXTRA[@]}" --port "$FFN_PORT" + ((A)) && launch attn.log "$F" "$A" "$MODEL_PATH" --additional-config "$(afd_json attention)" \ + "${COMMON[@]}" "${ATTN_DP[@]}" "${ARGS[@]}" "${ATTN_EXTRA[@]}" --port "$ATTN_PORT" + wait +} diff --git a/tests/unit/connectors/test_deepep_m2n_connector.py b/tests/unit/connectors/test_deepep_m2n_connector.py index eb9e9fbea..b5bf27c9d 100644 --- a/tests/unit/connectors/test_deepep_m2n_connector.py +++ b/tests/unit/connectors/test_deepep_m2n_connector.py @@ -527,6 +527,8 @@ def test_handshake_resolves_attention_transport_settings(attn_size, tp_size): assert result.expert_alignment == 128 assert not result.network_plan.use_gin + assert result.transport.is_identity + assert result.transport.ffn_ranks_per_domain == 0 @pytest.mark.parametrize( diff --git a/tools/benchmarks/README.md b/tools/benchmarks/README.md new file mode 100644 index 000000000..c5f8d5627 --- /dev/null +++ b/tools/benchmarks/README.md @@ -0,0 +1,82 @@ +# Decode-only benchmark + +This directory contains the decode-only benchmark, which measures decode +throughput and latency at a chosen input length without prefill work. + +- `decode_bench.py` implements the server-side KV connector, + `AFDDecodeBenchConnector`. +- `request_generator.sh` wraps `vllm bench serve` to drive load against the + server. + +## How it works + +The connector emulates the decode side of prefill-decode disaggregation with +the prefill instance faked out. For every new request, the scheduler side +reports every prompt token except the last as externally computed, so the +request prefills one token and goes straight to decode at a context of the +full prompt length. No prefill instance, KV producer or proxy is needed. + +The connector writes no KV values: the blocks allocated for the "external" +tokens keep whatever the KV cache held before. Throughput and latency are +meaningful; generated text is not. A preempted request that restarts from +scratch is reported as external again. + +Attach the connector to Attention only; the FFN needs no +`--kv-transfer-config`. The connector lives in `tools/benchmarks`, which is +not in the wheel, so it is loaded through `kv_connector_module_path` and needs +the repository root on `PYTHONPATH`. + +## Running with an AFD recipe + +The `DeepEPM2NAFDConnector` recipes, and the `P2pNcclAFDConnector` entry +points that reuse them, attach the connector to Attention when `DECODE_BENCH` +holds its `kv_connector_extra_config` as JSON (`{}` is enough) and add the +repository root (`AFD_REPO`, default the current directory) to `PYTHONPATH`; +see the +[recipes README](../../recipe/gpu/DeepEPM2NAFDConnector/README.md#decode-only-benchmark). +Every layout works, including multi-pod ones. From the repository root: + +```bash +export MODEL_PATH=/path/to/Qwen3-30B-A3B +DECODE_BENCH='{}' \ + bash recipe/gpu/DeepEPM2NAFDConnector/qwen3_30b_a3b/prefill_decode_colocation/graph.sh + +# once Attention answers on port 18000: +INPUT_LEN=1024 OUTPUT_LEN=256 NUM_PROMPTS=2000 MAX_CONCURRENCY=64 \ + REQUEST_RATE=inf RESULT_DIR=/tmp/results/decode_bench \ + tools/benchmarks/request_generator.sh +``` + +Use the Attention port, not the FFN port. `INPUT_LEN` plus `OUTPUT_LEN` must +fit the recipe's `--max-model-len`. + +## Running with vanilla vLLM + +The connector is not AFD-specific. For a single vanilla vLLM process, pass +the same `--kv-transfer-config`: + +```bash +export PYTHONPATH=$PWD +vllm serve "$MODEL_PATH" \ + --tensor-parallel-size 4 \ + --kv-transfer-config '{"kv_connector":"AFDDecodeBenchConnector","kv_connector_module_path":"tools.benchmarks.decode_bench","kv_role":"kv_both","kv_connector_extra_config":{}}' +``` + +and run `request_generator.sh` with `PORT=8000`. + +## Checking that a run was decode-only + +The server's `/metrics` split prompt tokens by source. Over a decode-only +run, `vllm:prompt_tokens_by_source{source="local_compute"}` grows by exactly +one per finished request (`vllm:request_success`). The rest of each prompt is +counted under `source="external_kv_transfer"`, or `source="local_cache_hit"` +for the few prompt blocks the prefix cache already holds. + +## Limitations + +- KV contents are uninitialized, so attention reads arbitrary values and the + generated text is meaningless. +- `kv_connector_extra_config` accepts `fill_mean` and `fill_std`, but the + worker never writes them into the KV cache (`start_load_kv` is a no-op). +- The connector's name includes `AFD`, but it does not require the AFD plugin; + it uses vLLM's standard KV-transfer interface. diff --git a/tools/benchmarks/decode_bench.py b/tools/benchmarks/decode_bench.py index 6ddd2a1be..f2e18c91f 100644 --- a/tools/benchmarks/decode_bench.py +++ b/tools/benchmarks/decode_bench.py @@ -5,7 +5,7 @@ from __future__ import annotations from dataclasses import dataclass -from typing import TYPE_CHECKING, Any +from typing import TYPE_CHECKING, Any, TypeAlias import torch from vllm.distributed.kv_transfer.kv_connector.v1 import ( @@ -28,7 +28,7 @@ logger = init_logger(__name__) -KVCacheLayer = torch.Tensor | tuple[torch.Tensor, ...] +KVCacheLayer: TypeAlias = torch.Tensor | tuple[torch.Tensor, ...] @dataclass diff --git a/tools/benchmarks/decode_bench_server.sh b/tools/benchmarks/decode_bench_server.sh deleted file mode 100755 index b39a7b8d8..000000000 --- a/tools/benchmarks/decode_bench_server.sh +++ /dev/null @@ -1,104 +0,0 @@ -#!/usr/bin/env bash -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project -# -# AF-disaggregated (2A2F) decode-only benchmark. -# -# This is the DECODE side of prefill-decode (PD) disaggregation with the prefill -# instance faked out. Compare recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/ -# prefill_decode_disaggregation/2p1a1f_graph_dbo.sh, where the decode side -# (attention + ffn) carries an LMCacheConnectorV1 kv_consumer and pulls real KV -# from a separate 1P prefill instance via a proxy. -# -# Here the AFDDecodeBenchConnector REPLACES that consumer connector on the -# ATTENTION instance and fabricates the KV: every prompt token except the last -# is reported as externally computed, and the KV cache is filled with dummy -# values. So requests skip prefill and go straight into decode -- letting you -# stress the decode path with arbitrary ISL, with NO prefill instance, no -# LMCache producer, and no proxy. Throughput/latency are meaningful; generated -# text is garbage. The connector is attached to attention only (the ffn side -# needs no kv-transfer-config), matching tests/e2e/runner.py. -# -# The connector lives in tools/benchmarks/ and is NOT shipped in the wheel, so -# it is loaded purely via kv_connector_module_path and needs the repo root on -# PYTHONPATH (inherited by the vLLM scheduler/worker subprocesses). -# -# Usage: -# MODEL_PATH=/path/to/DeepSeek-V2-Lite tools/benchmarks/decode_bench_server.sh -# then, once both instances are ready, drive load against the attention port: -# MODEL_PATH=/path/to/DeepSeek-V2-Lite tools/benchmarks/request_generator.sh -set -euo pipefail - -# --- make tools.benchmarks.decode_bench importable in vLLM subprocesses ------- -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -REPO_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)" -export PYTHONPATH="${REPO_ROOT}${PYTHONPATH:+:${PYTHONPATH}}" - -MODEL_PATH=${MODEL_PATH:-/path/model_weights/DeepSeek-V2-Lite} - -# dummy-KV fill params for the decode-bench connector -FILL_MEAN=${FILL_MEAN:-0.015} -FILL_STD=${FILL_STD:-0.0} -DECODE_BENCH_KV_CONFIG=$(cat < attn.log 2>&1 & - -# --- ffn instance (no decode-bench connector) --------------------------------- -CUDA_VISIBLE_DEVICES=2,3 uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size 2 \ - --tensor-parallel-size 1 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "ffn", - "connector": "P2pNcclAFDConnector", - "host": "127.0.0.1", - "port": 6269, - "num_attention_ranks": 2, - "num_ffn_ranks": 2 - } - }' \ - --max-num-seqs 64 \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 12 \ - --max-num-batched-tokens 64 \ - --max-cudagraph-capture-size 64 \ - --compilation-config '{ - "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[64] - }' \ - --host 127.0.0.1 \ - --port 18306 \ - --trust-remote-code > ffn.log 2>&1 & - -wait diff --git a/tools/benchmarks/request_generator.sh b/tools/benchmarks/request_generator.sh index a60dbbf7b..c5e3bf091 100755 --- a/tools/benchmarks/request_generator.sh +++ b/tools/benchmarks/request_generator.sh @@ -2,19 +2,20 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project # -# Drive load against a running AFD instance (e.g. the attention endpoint started -# by decode_bench_server.sh). Thin wrapper around `vllm bench serve`; every knob -# is overridable via an environment variable. +# Drive load against a running AFD instance (e.g. the Attention endpoint of a +# recipe under recipe/gpu/). Thin wrapper around `vllm bench serve`; every knob +# is overridable via an environment variable. For a decode-only run, start the +# recipe with DECODE_BENCH set; see tools/benchmarks/README.md. # # Usage: # MODEL_PATH=/path/to/weights tools/benchmarks/request_generator.sh -# HOST=127.0.0.1 PORT=18305 INPUT_LEN=8192 OUTPUT_LEN=256 MAX_CONCURRENCY=64 \ +# HOST=127.0.0.1 PORT=18000 INPUT_LEN=8192 OUTPUT_LEN=256 MAX_CONCURRENCY=64 \ # MODEL_PATH=/path/to/weights tools/benchmarks/request_generator.sh # # Variables (default): # MODEL_PATH model weights dir/name (required) # HOST server host (127.0.0.1) -# PORT server port (18305) +# PORT server port (18000) # DATASET_NAME vllm bench dataset (random) # INPUT_LEN random input length (ISL) (1024) # OUTPUT_LEN random output length (OSL) (128) @@ -28,7 +29,7 @@ set -euo pipefail MODEL_PATH=${MODEL_PATH:?set MODEL_PATH to the model weights dir/name} HOST=${HOST:-127.0.0.1} -PORT=${PORT:-18305} +PORT=${PORT:-18000} DATASET_NAME=${DATASET_NAME:-random} INPUT_LEN=${INPUT_LEN:-1024} OUTPUT_LEN=${OUTPUT_LEN:-128}