examples(glm5.2 1P1D): accuracy evaluation, Mooncake trace replay, per-role torch profiling

[ChengYao-amd]

examples(glm5.2 1P1D): accuracy evaluation, Mooncake trace replay, per-role torch profiling

Branch dev/yaoc/profile → main · Base 96bfe72 (0 behind) · 18 files, +1975 / −25

Summary

Three opt-in capabilities for the GLM-5.2 1P1D kit, plus one sglang DSA patch
that fixes a warm-up crash on the v0.5.17 engine base.

what	entry point	answers
Accuracy	engine/lm_eval.sh	is the model served correctly, not just quickly
Trace replay	engine/trace_replay.sh	how does it behave under a load with a real shared prefix
Profiling	engine/capture.sh	which operators dominate each PD role
DSA patch 05	patches/sglang_dsa/dsa_indexer_idle_metadata.diff	EAGLE eager IDLE killing the scheduler during warm-up

Serving defaults are unchanged. With PROFILE_* and AIPERF_TRACE unset,
up → smoke → bench → down behaves exactly as on main.

1. Accuracy

smoke proves the features are on and bench proves the stack is fast; neither
reads an answer. MXFP4 weights, fp8_e4m3 KV, ROCm sparse attention, mooncake
KV transfer, EAGLE speculation and a shared radix cache are six independent ways
to serve fast, fluent, wrong tokens, all green in the existing checks.

• lm_eval.sh drives sglang.test.run_eval, already inside the engine
  image, over OpenAI /v1/chat/completions. Nothing is installed into $CTR.
  probe (~40 s) → quick (GSM8K 200, ~2 min) → full (gsm8k +
  mixed_prefix_gsm8k, ~13 min). --thinking-mode glm-45 is load-bearing: the
  leg runs --reasoning-parser glm45, and without the matching
  chat_template_kwargs every eval scores 0.00 on a healthy deployment.
• tools/probe_accuracy.py gates the rest, because the failures it catches
  produce a number rather than an error. ~30 requests covering: both legs
  registered, content actually arriving, the same question giving the same
  answer (not the same bytes — at temperature=0 completion lengths ranged
  214–289 tokens across ten identical repeats), and a shared prefix not changing
  the answer behind it. That last check is the only one in the kit that can see a
  radix-cache / kvd / kv-aware lookup return the wrong block.
• tools/summarise_eval.py puts a 95% Wilson interval on every score, and in
  compare mode estimates the interval of the difference directly (Newcombe)
  rather than checking whether two intervals overlap — overlap is too
  conservative and hides real regressions. --gate exits non-zero on a drop.

Calibration measured on rocm/infera:sglang-v0.2.10 / MI355X: GSM8K 200 spans
0.940–0.965 over six unchanged runs; GSM8K full 0.945, 95% CI [0.932, 0.956],
1314 scored; mixed_prefix_gsm8k 0.945 over 1299; prefix-reuse delta +0.000.

2. Trace replay

bench.sh sweeps --dataset-name random, so it has no shared prefix by
construction. trace_replay.sh replays a Mooncake trace through AIPerf at the
recorded timestamps; hash_ids expand to real token blocks, so the radix cache,
kvd and kv-aware routing are all on the path.

• prepare is read-only: image, trace, tokenizer, AIPERF_OUT and router all
  checked from the node that generates load, then an analyze-trace shape
  report. run sends with --fixed-schedule.
• The client runs in its own container (aiperf:0.12.0, 255 MB) — the engine
  image ships Python 3.10 and AIPerf needs ≥ 3.11.
• ignore_eos:true by default: the mooncake_trace loader sets max_tokens
  from output_length but no min_tokens, so a reasoning model emitting EOS
  early would replay short. This makes decode-side numbers incomparable to
  bench, which the README states.
• aiperf_compat/sitecustomize.py (commit e43ae95) works around AIPerf
  0.12 sending existing local directories through snapshot_download, which
  HF_HUB_OFFLINE=1 turns into a startup failure. Injected via PYTHONPATH
  into the AIPerf container only.

3. Profiling

Prefill and decode overlap in wall-clock time, so one trace cannot be split by
role afterwards — the router's role selector is what keeps them apart.

• up.sh gains PROFILE_PREFILL / PROFILE_DECODE (both 0), which append
  --enable-profiling to the new ROUTER_EXTRA_ARGS.
• start_router() flips the backend rust → python on that flag, because
  launch_rust.py raises SystemExit rather than degrading. Throughput from a
  profiling run is therefore not comparable to a normal one.
• PROFILE_DECODE=1 also sets CUDA_GRAPH=0 on the decode leg. A replayed
  graph is one opaque launch: measured 134 decode steps in a window with 1
  carrying GPU operators. The cost is real (TPOT 35.6 → 137.2 ms), hence not a
  default; DECODE_CUDA_GRAPH=1 overrides.
• start_container() gains two optional mounts: TRACE_OUT (same absolute path
  inside and out, so traces need no docker cp/SSH fetch) and INFERA_SRC (dev
  overlay at /opt/infera:ro, asserted live by importing infera after start).
• capture.sh verifies the bind mount with docker inspect before profiling,
  probes the control plane with an invalid role (400 = enabled, 403 = not),
  starts both roles from one in-container script so their windows are sub-ms
  apart, pins with_stack=false (SGLang defaults to True: 122 MB vs 14 MB per
  rank), stops with a single call, and polls du -sb until the files stop
  growing. LOAD_KIND=bench|trace_replay picks where the load probe looks.

4. DSA patch 05

EAGLE eager IDLE skips draft_attn_backend.init_forward_metadata(), then
draft_forward() selects a per-step backend the generic IDLE runner never
updated, so its first IDLE use raises:

AttributeError: 'DeepseekSparseAttnBackend' object has no attribute 'forward_metadata'

The existing empty-seq_lens guard returns the right tensor but sits hundreds of
lines after get_indexer_metadata() on v0.5.17. The patch returns the same
all-invalid top-k before the metadata lookup, keeping
_broadcast_indexer_topk_from_rank0 since IDLE ranks still need the DP
synchronization primitive.

v0.5.17 / mi35x only — joins the full arm, marker count 7 → 8. The gfx942
v0.5.16 arm is untouched. Upstream #32209
carries the same strategy for CUDA; #31683
widens the later guard, which is too late on this base. Both OPEN.

Commits

commit	subject	files	net
156e3e2	add glm5.2 profile script	11	+482 / −14
07063e1	add mooncake trace replay scripts	5	+389 / −12
e43ae95	fix trace replay local tokenlizer bugs	3	+40 / −6
b1f7b20	add glm5.2 1p1d example lm_eval	6	+931 / −9

Test plan

bash -n clean on all eight shell scripts; ruff format/ruff check clean on
the three Python files; file modes match engine/.

# regression — must behave as on main
bash cluster/<wrapper>.sh up && bash cluster/<wrapper>.sh smoke
bash cluster/<wrapper>.sh bench 64 && bash cluster/<wrapper>.sh down

# accuracy
bash cluster/<wrapper>.sh lm_eval probe     # PROBE PASSED
bash cluster/<wrapper>.sh lm_eval quick     # 0.940-0.965
bash cluster/<wrapper>.sh lm_eval full      # prefix-reuse delta contains 0

# trace replay
AIPERF_TRACE=<...>/conversation_trace.jsonl bash cluster/<wrapper>.sh trace_replay prepare
AIPERF_TRACE=<...>/conversation_trace.jsonl START_MS=0 END_MS=120000 \
  bash cluster/<wrapper>.sh trace_replay run

# profiling, both roles from one window
PROFILE_PREFILL=1 PROFILE_DECODE=1 bash cluster/<wrapper>.sh up
nohup bash cluster/<wrapper>.sh bench 64 &
PROFILE_PREFILL=1 PROFILE_DECODE=1 bash cluster/<wrapper>.sh capture

# image
docker build -f deploy/docker/Dockerfile.sglang .   # 5 patches, 8 markers

Profiling run should show the router logging the rust → python swap, the
decode leg logging cuda_graph=0, both $TRACE_OUT/<ts>/{prefill,decode}/
filling, and the flush stage reporting a stable byte count per role.