Repository navigation
tools: hrx_decode_bw.py, effective decode bandwidth of a model as one table row (CL-220GBS) - #2372
Conversation
… one table row (CL-220GBS) bytes one decode step reads, from the GGUF itself (every split file): every tensor at its stored size, token-embedding tables at one row, routed experts at used/total, the MTP block left out unless --mtp, KV for the full-attention layers at the mean depth the generated tokens read, and the linear-attention state read and written. ms/step is the median of llama-bench's per-repetition t/s; GB/s is the quotient. 'table' joins rows. Self-test registered (hrx_decode_bw_self). Refs CL-220GBS (owner goal 220 GB/s) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PR Reviewer Guide 🔍(Review updated until commit 9b4c524)Here are some key observations to aid the review process:
✅ Resolved findingstools/hrx_decode_bw.py:102-103MTP Block Exclusion Logic The logic for excluding MTP blocks when |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 6206d60
The formula is exact in its self-test and reproduces a number from October, which is the cross-check that makes a bytes model trustworthy.
want = {"weights_dense": 1000 + 40 + 40 + 64 + 64, "weights_experts": 200, "embedding_rows": 10,
"kv": 1 * 10 * 2 * 8 * 2, "ssm_state": 1 * 2 * 2 * 4 * 4 * 4}
assert terms == wantA term-by-term expectation, not "it runs" -- and it pins all three exceptions the body names: routed experts at 2/8 (800 x 2/8 = 200), the MTP block left out (weights_dense + 999 and kv doubled only with --mtp), and the tied head (embedding_rows == 0 with no output.weight). Plus ssm_state and kv as their own keys, so a formula that folded them into weights would fail on the dict, not on a total.
And the body's cross-check reproduces:
1.970 GB/step ÷ 11.20 ms = 175.9 GB/s (body says 176; 178.7 was the October record)
That is the difference between a model and a fit: the tool is asked to explain a measurement taken before it existed, and lands within 1.6 % of it.
One thing the body oversells, and it is the same shape as the census list: it says step 1 is "one re-baseline table over six models" and then shows three rows (Qwen3-Coder-30B-A3B, Qwen3.8-27B, Qwen3.8-Flash-Next). Three of six is a fine state to publish -- the tool is the deliverable -- but the sentence should say so, or a reader concludes the other three were dropped.
One convention note, not a defect: the self-test is assert-based, so the first failure aborts and a reviewer sees one arm per run. Every other tool in tools/ci prints ok/FAIL per arm and counts them (24 arms on moving_ref_check, 10 on notice_covers_deps), which is what makes "the arms bite" checkable in one run. For a ctest row the exit code is enough; for a reviewer it is one assertion at a time.
Worth keeping exactly as it is: the three exceptions are the ones that make a bytes model wrong rather than approximate -- a token-embedding read per step (not the whole table), experts at their used share, and MTP excluded unless asked. Each is a quantity that would silently inflate the bandwidth by a factor, and each has its own dict key.
Ran: --self-test (passes) and the arithmetic of the body's cross-check. Not run: run mode (needs llama-bench and a model) and the six-model table itself.
…rom the HRX dispatch profile (CL-220GBS) rocprofv3 sees no HRX dispatch, so the step-1 table's top-10 column reads the backend's own profile (HRX_PROFILE_MODE=dispatch, then iree-profile dispatch --dispatch_events): duration_ns summed per kernel key over the decode steps, add-on ids named from the build's KERNEL-IDS.map. Self-test extended. Refs CL-220GBS (owner goal 220 GB/s) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
VERDICT: READY at I ran it on a real MoE on this boxThe term-by-term decomposition is exactly right: bytes from the GGUF, not from a parameter count; the MoE term scaled by The one thing that must be settled before this prints a row: what a step reads
Your tool models flint's step as 6 of 256 experts (2.213 GB). That sentence can be read as every stored weight is read per step (≈25 GB). The two readings differ by 10×, and they bracket the 220 GB/s goal differently: The second reading would make a 40B model hit the owner's 220 GB/s target at ten tokens a second, which cannot be what the goal means; the first makes 220 GB/s ≈ 100 tok/s on flint, which is a real and demanding target. So: one line in the body saying which one the hardware does, and if the doc's sentence is about copies in memory (its paragraph is about duplicates) rather than bytes per step, say that too and fix the sentence — as written it is the contradiction. This is the definition of the metric, not a detail: every row this tool prints inherits it. Two asks
Why this is the right first step for the goalBecause "220 GB/s" has been a target without a denominator — GB/s of what, read by which step, at which depth. This tool answers all three from the artifact, and its self-tests make it re-runnable. Once (1) and (2) are in, the six-model re-baseline is a table someone can argue with instead of a claim. |
|
VERDICT: READY at
|
|
Persistent review updated to latest commit 435d18b |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 9b4c524
The new top subcommand is armed with the four quantities it computes, and both mutants I tried die on the numbers rather than on a missing key.
$ hrx_decode_bw.py --self-test -> self-test passed
mutant A: divide by `steps + 1` -> AssertionError: [('gemv_small', 2.0, 0.6, 0.667), ('loom_x', 1.333, 0.4, 0.333)]
mutant B: ignore the `--map` (no id resolution)
-> AssertionError: [('k_aaaaaaaaaaaa', 3.0, 0.6, 1.0), ('loom_x', 2.0, 0.4, 0.5)]
The fixture is what makes those two separable: two dispatches of k_aaaaaaaaaaaa at 3000 ns plus one loom_x at 4000 ns over steps=2, and a map naming only the first. So the arm pins all four outputs at once --
gemv_small (the opaque id was resolved), 3.0 µs/step (2 x 3000 / 2), 0.6 share (6 of 10 µs), 1.0 dispatch/step, and dev == 5.0 µs per step for the total. A steps+1 off-by-one, which is exactly what the body warns about
("llama-bench's warm-up token counts as a step: give n + 1"), fails on the first two numbers; dropping the map fails on the name while leaving the arithmetic right. Two different defects, two different visible symptoms.
And the caveat about what the number means is stated in the tool, not only in the body: "Profiling slows a step down, so read the shares and the device-time total, not the step time it implies." That is the sentence that keeps top from being quoted as a
step-time measurement -- the same distinction as the device_gemm column including the weight fill (#2361).
One thing it explains that no other instrument here does: "top reads HRX's own dispatch profile (rocprofv3 does not see HRX dispatches)" -- with the two-command recipe (HRX_PROFILE_MODE=dispatch … then iree-profile dispatch --format=jsonl …). That is the missing link for anyone who has tried rocprofv3 on an HRX run and got
an empty timeline.
Unchanged from 6206d60: the bytes formula and its term-by-term arm; the three exceptions; and the cross-check that reproduces October's 176 GB/s.
Not run: top against a real profile (needs llama-bench + iree-profile + the HRX add-on) and run mode.
Measured: the
|
|
Persistent review updated to latest commit 9b4c524 |
Your MoE term is confirmed by a measurement, and the doc sentence it contradicts is now settledI asked earlier for one flint-UD-Q4_K_XL (40B-A2.5B, 6 of 256 experts) on HRX, median 42.61 tok/s (2 runs, 35.24–49.97, spread 1.418×). So the tool's Two notes from the run for the table you are about to draw:
|
Refs CL-220GBS (owner goal 220 GB/s)
Step 1 of the 220 GB/s work is one re-baseline table over six models, with the tool that prints it committed so anyone can re-run it. This is that tool. Effective bandwidth is bytes one decode step must read / step time; this script computes the bytes from the GGUF and times the step with llama-bench.
The bytes formula (also in the script's docstring)
*_exps) count atexpert_used_count / expert_count;nextn_predict_layers) are left out unless--mtp.depth × n_head_kv × (key_length + value_length) × 2bytes (f16 cache).runcounts it atdepth + n/2, the mean position its generated tokens read.2 × time_step_rank × state_size × inner_size/time_step_rank × 4bytes.What it gives today, at depth 512, for the cl-hrxnan models:
Cross-check: Coder's 1.97 GB at the 11.20 ms/step measured in October gives 176 GB/s, against the 178.7 recorded then.
Checks
tools/hrx_decode_bw.py --self-testcovers a synthetic model with an MTP block, routed experts, a tied head, a full-attention and a linear-attention layer, and a split-file name. It is registered ashrx_decode_bw_self.tests_referenced_check,tool_help_checkandregistration_loss_check --base origin/main(481 kept, 1 added) pass.The measured rows (ms/step and GB/s, HRX and Vulkan) go in
~/briefs/CL-220GBS-STATE.mdas they are run. The step-1 table is posted there, not in this PR.🤖 Generated with Claude Code