Decode bench topology

.agents/skills/deploy-afd-k8s/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: deploy-afd-k8s
-description: Use when the user asks to deploy, serve, or stand up an AFD GPU recipe on Kubernetes/OpenShift from a recipe .sh script - creating the model PVC, recipe ConfigMap, serve pod(s), and Service(s), and waiting until the endpoint answers. Placement across pods is never asked for or assumed -- it is read directly off the recipe's own top-of-file comment block (however that block describes it, strict or prose): a recipe whose comment describes a per-pod placement deploys exactly that many pods with exactly that per-pod setup; a recipe without one deploys as a single pod. Do not use for local (non-k8s) serving, NPU recipes, prefill-decode disaggregation recipes, or E2E correctness testing (see run-e2e).
+description: Use when the user asks to deploy, serve, or stand up an AFD GPU recipe on Kubernetes/OpenShift from a recipe .sh script - creating the model PVC, recipe ConfigMap, serve pod(s), and Service(s), and waiting until the endpoint answers. Placement across pods is never asked for or assumed -- it is the recipe's LAYOUT (the caller's setting, else the script's default): one pod per LAYOUT entry, a one-entry LAYOUT being a single pod. Do not use for local (non-k8s) serving, NPU recipes, prefill-decode disaggregation recipes, or E2E correctness testing (see run-e2e).
 ---
 
 # Deploy an AFD GPU recipe on Kubernetes
@@ -15,13 +15,13 @@ On success this skill guarantees, in the target namespace:
 
 | | |
 |---|---|
-| Endpoint | `http://vllm-service:${CLIENT_PORT}` (OpenAI-compatible, in-namespace; `CLIENT_PORT` defaults to `18305`) |
+| Endpoint | `http://vllm-service:${CLIENT_PORT}` (OpenAI-compatible, in-namespace; `CLIENT_PORT` is the launcher's Attention port, `18000`) |
 | Serving | the model named by `MODEL_ID` |
-| Pod(s) / Service | `vllm-pod`, or one pod per pod the recipe's header comment describes (named `vllm-pod-<pod>`); Service `vllm-service`, labels `app=afd-recipe,role=serve` |
+| Pod(s) / Service | `vllm-pod`, or one pod per `LAYOUT` entry (named `vllm-pod-afd-<k>`); Service `vllm-service`, labels `app=afd-recipe,role=serve` |
 | Model cache | `PVC_NAME`, mounted at `/models`, `HF_HOME=/models/.hf_home` |
 | Left running | yes, deliberately -- so weights stay warm for follow-up runs; step 5 asks the caller when to delete the pods |
 
-Whenever a recipe's header spans more than one pod, two more internal-only
+Whenever a recipe's `LAYOUT` spans more than one pod, two more internal-only
 Services may also exist (`vllm-ffn-p2p-service` always; `vllm-attn-dp-service`
 and/or `vllm-ffn-dp-service` only if that role is itself split across more
 than one pod) -- callers never talk to these directly. See
@@ -38,15 +38,15 @@ than one pod) -- callers never talk to these directly. See
   NIXL-enabled image plus a proxy, and resolve a `SCRIPT_DIR`-relative path
   that this skill's flat script mount (step 4b) breaks. Ask the user for a
   different workflow for disaggregation.
-- Recipes are hand-written shell scripts, not JSON configs -- there is no
-  generator step and no manifest to read.
-- A recipe is **placement-driven** (more than one pod) exactly when its
-  top-of-file comment block describes a per-pod placement -- read that
-  block, don't infer shape from flags. Today's recipes spell it as one
-  `# For pod N: POD=...` line per pod, but the comment isn't guaranteed to
-  use that exact grammar going forward. See
-  [resolve-recipe.md](resources/resolve-recipe.md).
-- Not for `tools/benchmarks/decode_bench_server.sh` (local, non-k8s).
+- Recipes are short shell scripts on the shared launcher
+  `recipe/gpu/common/afd_colocation.sh` (see `recipe/gpu/common/README.md`);
+  the recipe ConfigMap ships both files (step 4b).
+- Placement is `LAYOUT` (one pod per `<a>A<f>F` entry) and run-specific
+  settings are env (`ATTN_TP`, `FFN_TP`, `VLLM_ARGS`, `DECODE_BENCH`, ...);
+  see [resolve-recipe.md](resources/resolve-recipe.md). This skill only
+  serves; load for the decode-only benchmark
+  (`tools/benchmarks/request_generator.sh`) is run separately against the
+  endpoint.
 
 ## Requirements
 
@@ -63,9 +63,9 @@ than one pod) -- callers never talk to these directly. See
   docker push "$IMAGE"
   ```
 
-  The image supplies only the `afd-plugin` install. The recipe script is
-  mounted fresh from local disk on every run (step 4b), so editing or adding
-  a recipe never requires a rebuild.
+  The image supplies only the `afd-plugin` install. The recipe script and
+  the shared launcher are mounted fresh from local disk on every run (step
+  4b), so editing or adding a recipe never requires a rebuild.
 - An `hf-token-secret` Secret with a `token` key, needed when `MODEL_ID` is
   a HF Hub repo id (not needed when weights are already staged on the PVC):
   ```bash
@@ -81,7 +81,7 @@ exist **before** any Pod is deployed -- see
 
 | Resource | Covers |
 |---|---|
-| [resolve-recipe.md](resources/resolve-recipe.md) | Step 1 (the only step that parses the recipe): read the recipe's serve-block fields and model id; read its placement (or lack of one) from its header comment -- never ask; record `GPU_COUNT` per pod. |
+| [resolve-recipe.md](resources/resolve-recipe.md) | Step 1 (the only step that parses the recipe): model id, the `LAYOUT` plan (never ask), `GPU_COUNT` per pod, and the launcher's ports. |
 | [deploy.md](resources/deploy.md) | Steps 2-4f: take step 1's `GPU_COUNT`, confirm, PVC, ConfigMap, fsGroup, Services (step 4d -- which of the four, see [services.md](resources/services.md)), pod(s) via `templates/pod.yaml`, wait for Running and then for each pod's `Ready` condition. |
 | [report-and-teardown.md](resources/report-and-teardown.md) | Step 5: report the endpoint/model/nodes and ask when to delete the running pods. Step 6: teardown when the caller says so (pods/Services only -- never the PVC unless asked). |
 

.agents/skills/deploy-afd-k8s/resources/deploy.md

@@ -3,8 +3,14 @@
 ## 2. GPU_COUNT
 
 Use the `GPU_COUNT` recorded by [resolve-recipe.md](resolve-recipe.md) --
-one value for a plain recipe, one per plan entry. Don't re-read the script
-here; all recipe parsing lives in step 1.
+`a + f` per `LAYOUT` entry. Don't re-read the script here; all recipe
+parsing lives in step 1:
+
+```bash
+LAYOUT=${LAYOUT:-$(grep -oE 'LAYOUT:-[0-9AF,]+' "$RECIPE_SCRIPT_PATH" | cut -d- -f2)}
+IFS=, read -ra ENTRIES <<<"$LAYOUT"
+for e in "${ENTRIES[@]}"; do f=${e#*A}; echo "$e -> $(( ${e%%A*} + ${f%F} )) GPUs"; done
+```
 
 ## 3. Confirm before touching the cluster
 
@@ -19,7 +25,7 @@ existing Pod(s)** with the same name(s) about to be deployed.
 AFD_PLUGIN_IMAGE=<image>
 MODEL_ID=<model-id>
 PVC_NAME=<pvc-name>
-CLIENT_PORT=<n>   # from the script's --port; default 18305
+CLIENT_PORT=<n>   # the launcher's ATTN_PORT (18000), see resolve-recipe.md
 RECIPE_SCRIPT_PATH=<local-recipe-script-path>
 ```
 
@@ -72,11 +78,14 @@ rather than re-downloading.
 ## 4b. Recipe ConfigMap
 
 Lets the pod run any local recipe -- edited, uncommitted, or new -- without
-rebuilding the image. Run from wherever `RECIPE_SCRIPT_PATH` resolves:
+rebuilding the image. The recipe sources the shared launcher, so ship both;
+`templates/pod.yaml` points the recipe at `/recipe/afd_colocation.sh`
+(`AFD_RECIPE_LIB`). Run from the repository root:
 
 ```bash
 kubectl create configmap afd-recipe-script \
   --from-file=recipe.sh="${RECIPE_SCRIPT_PATH}" \
+  --from-file=afd_colocation.sh=recipe/gpu/common/afd_colocation.sh \
   --dry-run=client -o yaml | kubectl apply -f -
 ```
 
@@ -108,17 +117,19 @@ recipe) against the one shared `templates/pod.yaml`:
 ```bash
 deploy_pod() {
   local pod_name="$1" gpu="$2" attn_role="$3" ffn_role="$4" extra_env_pairs="$5"
-  # extra_env_pairs: space-separated VAR=value tokens straight off this
-  # plan entry's header line (including POD=... itself), empty for a plain
-  # recipe. Build the spliced env-entries block for templates/pod.yaml.
-  # Leading (not trailing) newlines keep the `# ${TEMPLATE_EXTRA_ENV}`
-  # placeholder line a harmless comment when this is empty, and valid
-  # sibling list items when it isn't.
-  local extra_env=""
-  for pair in $extra_env_pairs; do
+  # extra_env_pairs: one VAR=value per line -- this plan entry's overrides
+  # (including POD=... itself) plus the caller's settings -- empty for a plain
+  # recipe. Values may hold spaces and JSON (VLLM_ARGS, DECODE_BENCH); they
+  # become single-quoted YAML strings. Leading (not trailing) newlines keep
+  # the `# ${TEMPLATE_EXTRA_ENV}` placeholder line a harmless comment when
+  # this is empty, and valid sibling list items when it isn't.
+  local extra_env="" pair val q="'"
+  while IFS= read -r pair; do
+    [ -n "$pair" ] || continue
+    val=${pair#*=}
     extra_env="$extra_env
-        - {name: ${pair%%=*}, value: \"${pair#*=}\"}"
-  done
+        - {name: ${pair%%=*}, value: '${val//$q/$q$q}'}"
+  done <<<"$extra_env_pairs"
 
   kubectl delete pod "$pod_name" --ignore-not-found
 
@@ -131,26 +142,30 @@ deploy_pod() {
 }
 ```
 
-Plain recipe, one pod, both roles head, no overrides:
+One-entry `LAYOUT` (e.g. the default `2A2F`), one pod, both roles head,
+with or without the caller's settings:
 
 ```bash
 deploy_pod vllm-pod "$GPU_COUNT" head head ""
+deploy_pod vllm-pod 4 head head $'VLLM_ARGS=--max-num-seqs 32'
 ```
 
-Placement-driven recipe, one call per entry parsed in
-[resolve-recipe.md](resolve-recipe.md), e.g. the qwen3.5 3-pod plan:
+Several entries, one call per entry parsed in
+[resolve-recipe.md](resolve-recipe.md), e.g. `LAYOUT=2A0F,0A2F` (2 GPUs per
+pod) with the decode-only benchmark, `LAYOUT=4A1F,4A1F` (5 GPUs per pod),
+or the Qwen3.5 recipe's default `2A0F,2A0F,0A2F`:
 
 ```bash
-deploy_pod vllm-pod-attention-0 2 head none "POD=ATTENTION_0"
-deploy_pod vllm-pod-attention-1 2 worker none "POD=ATTENTION_1 ATTENTION_HEADLESS=1 ATTENTION_DP_START_RANK=2"
-deploy_pod vllm-pod-ffn-0 2 none head "POD=FFN_0"
-```
+SETTINGS=$'LAYOUT=2A0F,0A2F\nDECODE_BENCH={}\nAFD_REPO=/opt/afd-plugin'
+deploy_pod vllm-pod-afd-0 2 head none $'POD=AFD_0\n'"$SETTINGS"
+deploy_pod vllm-pod-afd-1 2 none head $'POD=AFD_1\n'"$SETTINGS"
 
-`POD=AFD_<index>` recipe, e.g. the DeepEP M2N 2 x 4A1F plan (5 GPUs per pod):
+deploy_pod vllm-pod-afd-0 5 head head $'POD=AFD_0\nLAYOUT=4A1F,4A1F'
+deploy_pod vllm-pod-afd-1 5 worker worker $'POD=AFD_1\nLAYOUT=4A1F,4A1F'
 
-```bash
-deploy_pod vllm-pod-afd-0 5 head head "POD=AFD_0"
-deploy_pod vllm-pod-afd-1 5 worker worker "POD=AFD_1"
+deploy_pod vllm-pod-afd-0 2 head none 'POD=AFD_0'
+deploy_pod vllm-pod-afd-1 2 worker none 'POD=AFD_1'
+deploy_pod vllm-pod-afd-2 2 none head 'POD=AFD_2'
 ```
 
 ## 4f. Wait for Running, then for each pod's Ready condition
@@ -198,6 +213,6 @@ done
 Only report the endpoint ready once every pod in the plan is `Ready` -- one
 pod becoming ready doesn't imply the others did.
 
-Startup markers are keyed on the fixed log names every colocation recipe
-uses today -- see `templates/pod.yaml`'s `ready_marker()` for the exact
-lines. A recipe using different log names needs that template updated.
+Startup markers are keyed on the launcher's fixed log names (`attn.log`,
+`ffn.log`) -- see `templates/pod.yaml`'s `ready_marker()` for the exact
+lines.