Skip to content

tools: hrx_decode_bw.py, effective decode bandwidth of a model as one table row (CL-220GBS) - #2372

Merged
bong-water-water-bong merged 3 commits into
mainfrom
cl220/decode-bw-table
Nov 13, 2026
Merged

bong-water-water-bong merged 3 commits into
mainfrom
cl220/decode-bw-table

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Refs CL-220GBS (owner goal 220 GB/s)

Step 1 of the 220 GB/s work is one re-baseline table over six models, with the tool that prints it committed so anyone can re-run it. This is that tool. Effective bandwidth is bytes one decode step must read / step time; this script computes the bytes from the GGUF and times the step with llama-bench.

tools/hrx_decode_bw.py bytes MODEL.gguf [--depth N] [--mtp]
tools/hrx_decode_bw.py run MODEL.gguf --bench BUILD/bin/llama-bench --json row.json -- -ngl 99
tools/hrx_decode_bw.py table row-*.json

The bytes formula (also in the script's docstring)

  • weights: every tensor in the decode graph at its stored size, with three exceptions:
    • a token-embedding table is one row per token, unless it is also the tied output head;
    • routed experts (*_exps) count at expert_used_count / expert_count;
    • the MTP block(s) (nextn_predict_layers) are left out unless --mtp.
  • kv: for each full-attention layer, depth × n_head_kv × (key_length + value_length) × 2 bytes (f16 cache). run counts it at depth + n/2, the mean position its generated tokens read.
  • ssm: each linear-attention layer reads and writes its f32 recurrent state: 2 × time_step_rank × state_size × inner_size/time_step_rank × 4 bytes.

What it gives today, at depth 512, for the cl-hrxnan models:

model dense weights routed experts KV linear-attn state GB/step
Qwen3-Coder-30B-A3B Q4_K_M 0.823 1.097 (8/128) 0.050 — 1.970
Qwen3.8-27B UD-Q4_K_XL (no MTP) 16.482 — 0.034 0.302 16.818
Qwen3.8-Flash-Next UD-IQ3_XXS 4.001 0.950 (10/512) 0.013 0.226 5.190

Cross-check: Coder's 1.97 GB at the 11.20 ms/step measured in October gives 176 GB/s, against the 178.7 recorded then.

Checks

  • tools/hrx_decode_bw.py --self-test covers a synthetic model with an MTP block, routed experts, a tied head, a full-attention and a linear-attention layer, and a split-file name. It is registered as hrx_decode_bw_self.
  • tests_referenced_check, tool_help_check and registration_loss_check --base origin/main (481 kept, 1 added) pass.

The measured rows (ms/step and GB/s, HRX and Vulkan) go in ~/briefs/CL-220GBS-STATE.md as they are run. The step-1 table is posted there, not in this PR.

🤖 Generated with Claude Code

… one table row (CL-220GBS)

bytes one decode step reads, from the GGUF itself (every split file): every tensor at its
stored size, token-embedding tables at one row, routed experts at used/total, the MTP
block left out unless --mtp, KV for the full-attention layers at the mean depth the
generated tokens read, and the linear-attention state read and written. ms/step is the
median of llama-bench's per-repetition t/s; GB/s is the quotient. 'table' joins rows.
Self-test registered (hrx_decode_bw_self).

Refs CL-220GBS (owner goal 220 GB/s)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit 9b4c524)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

MTP Block Exclusion Logic

The logic for excluding MTP blocks when --mtp is not specified relies on nextn_predict_layers being present in the model fields. If the model does not have this field or it's set to 0, the exclusion logic may not work as intended, potentially including MTP blocks in the bandwidth calculation when they should be excluded. This is especially concerning for models where the MTP block count is not explicitly tracked.

mtp_from = blocks - nextn if nextn else blocks
has_output = any(n == "output.weight" for n, _, _ in tensors)
terms = {"weights_dense": 0, "weights_experts": 0, "embedding_rows": 0, "kv": 0, "ssm_state": 0}
attn_layers, ssm_layers = set(), set()
for name, nbytes, shape in tensors:
    m = re.match(r"blk\.(\d+)\.", name)
    blk = int(m.group(1)) if m else None
    if blk is not None and blk >= mtp_from and not mtp:
        continue
Potential Incorrect Tensor Counting for Routed Experts

The script computes the bytes for routed experts by using n_used / n_exp ratio, but this may not correctly reflect the actual memory access pattern in the hardware. If n_used is not representative of the actual number of experts used per token, the bandwidth calculation could be inaccurate. This is especially critical for models where expert usage varies significantly across tokens.

if "_exps" in name and n_exp:
    terms["weights_experts"] += nbytes * n_used // n_exp
else:
KV Cache Calculation Assumption

The script assumes that KV cache is read from depth to depth + n/2 for the mean position. This is a simplification and may not accurately represent the actual cache access pattern in hardware, especially for models with dynamic sequence lengths or complex attention mechanisms.

# the n generated tokens read the cache at depth .. depth + n: count KV at the mean, depth + n / 2
terms, notes = step_bytes(fields, tensors, a.depth + a.n // 2, a.mtp)
✅ Resolved findings

tools/hrx_decode_bw.py:102-103

MTP Block Exclusion Logic

The logic for excluding MTP blocks when --mtp is not specified relies on nextn_predict_layers being present in the model fields. If the model does not have this field or it's set to 0, the exclusion logic may not work as intended, potentially including MTP blocks in the bandwidth calculation when they should be excluded.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 6206d60

The formula is exact in its self-test and reproduces a number from October, which is the cross-check that makes a bytes model trustworthy.

want = {"weights_dense": 1000 + 40 + 40 + 64 + 64, "weights_experts": 200, "embedding_rows": 10,
        "kv": 1 * 10 * 2 * 8 * 2, "ssm_state": 1 * 2 * 2 * 4 * 4 * 4}
assert terms == want

A term-by-term expectation, not "it runs" -- and it pins all three exceptions the body names: routed experts at 2/8 (800 x 2/8 = 200), the MTP block left out (weights_dense + 999 and kv doubled only with --mtp), and the tied head (embedding_rows == 0 with no output.weight). Plus ssm_state and kv as their own keys, so a formula that folded them into weights would fail on the dict, not on a total.

And the body's cross-check reproduces:

1.970 GB/step ÷ 11.20 ms = 175.9 GB/s     (body says 176; 178.7 was the October record)

That is the difference between a model and a fit: the tool is asked to explain a measurement taken before it existed, and lands within 1.6 % of it.

One thing the body oversells, and it is the same shape as the census list: it says step 1 is "one re-baseline table over six models" and then shows three rows (Qwen3-Coder-30B-A3B, Qwen3.8-27B, Qwen3.8-Flash-Next). Three of six is a fine state to publish -- the tool is the deliverable -- but the sentence should say so, or a reader concludes the other three were dropped.

One convention note, not a defect: the self-test is assert-based, so the first failure aborts and a reviewer sees one arm per run. Every other tool in tools/ci prints ok/FAIL per arm and counts them (24 arms on moving_ref_check, 10 on notice_covers_deps), which is what makes "the arms bite" checkable in one run. For a ctest row the exit code is enough; for a reviewer it is one assertion at a time.

Worth keeping exactly as it is: the three exceptions are the ones that make a bytes model wrong rather than approximate -- a token-embedding read per step (not the whole table), experts at their used share, and MTP excluded unless asked. Each is a quantity that would silently inflate the bandwidth by a factor, and each has its own dict key.

Ran: --self-test (passes) and the arithmetic of the body's cross-check. Not run: run mode (needs llama-bench and a model) and the six-model table itself.

…rom the HRX dispatch profile (CL-220GBS)

rocprofv3 sees no HRX dispatch, so the step-1 table's top-10 column reads the backend's own
profile (HRX_PROFILE_MODE=dispatch, then iree-profile dispatch --dispatch_events): duration_ns
summed per kernel key over the decode steps, add-on ids named from the build's KERNEL-IDS.map.
Self-test extended.

Refs CL-220GBS (owner goal 220 GB/s)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 6206d60 — the right instrument for CL-220GBS, and the one thing it must settle is a definition, not a bug.

I ran it on a real MoE on this box

$ tools/hrx_decode_bw.py bytes ~/models/flint/flint-UD-Q4_K_XL.gguf
arch flint; 49 blocks; experts 6/256; 49 full-attention layers x depth 0
  weights_dense    1.673 GB
  weights_experts  0.540 GB
  total            2.213 GB

The term-by-term decomposition is exactly right: bytes from the GGUF, not from a parameter count; the MoE term scaled by expert_used_count / expert_count (nbytes * n_used // n_exp); embeddings as one row per token; KV at the cached depth; SSM state separate. And 2.213 GB/step is credible for 40B-A2.5B at ~5 bits/param — a tool that printed 25 GB here would have counted the whole file. The assert-based self-tests on the terms and the --mtp/tied-embedding cases are the right way to keep this honest.

The one thing that must be settled before this prints a row: what a step reads

docs/hrx.md:1071 says, about this exact model:

"flint UD-Q4_K_XL (routed experts) reads every weight stored and holds one copy per weight with or without the default."

Your tool models flint's step as 6 of 256 experts (2.213 GB). That sentence can be read as every stored weight is read per step (≈25 GB). The two readings differ by 10×, and they bracket the 220 GB/s goal differently:

tool reading   2.21 GB/step @ 100 tok/s =  221 GB/s   <- flint's ceiling is ~the goal
doc reading   25.05 GB/step @  10 tok/s =  250 GB/s   <- 10 tok/s already "exceeds 220"
                                                                50 tok/s would need 1,252 GB/s

The second reading would make a 40B model hit the owner's 220 GB/s target at ten tokens a second, which cannot be what the goal means; the first makes 220 GB/s ≈ 100 tok/s on flint, which is a real and demanding target. So: one line in the body saying which one the hardware does, and if the doc's sentence is about copies in memory (its paragraph is about duplicates) rather than bytes per step, say that too and fix the sentence — as written it is the contradiction. This is the definition of the metric, not a detail: every row this tool prints inherits it.

Two asks

  1. print the spread, not just its count. Line 157 does median of {len(ts)}) — good, --reps 3 clears the floor. But engine#2369 measured 7.94 / 11.78 / 10.39 / 12.57 tok/s on four identical runs of one command: a median whose spread is not printed is still a single number, and "220 GB/s" derived from it would be a sample. Print min…max (or the ratio) beside the median, and refuse when --reps < 3 rather than printing median of 1.
  2. state the row's depth=0 caveat where the row is — at depth 0 KV is 0.000 GB, which is correct and also means the printed GB/s is a floor-friendly best case: at 8192 cached positions the same model reads KV too, so the number moves with --depth. One clause, so nobody compares a depth-0 row against a depth-8192 one.

Why this is the right first step for the goal

Because "220 GB/s" has been a target without a denominator — GB/s of what, read by which step, at which depth. This tool answers all three from the artifact, and its self-tests make it re-runnable. Once (1) and (2) are in, the six-model re-baseline is a table someone can argue with instead of a claim.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 435d18b — the new top half is the strongest thing in the PR, and it can settle the open question by a prediction you can check in one run.

top is the right answer to a real obstacle

"top reads HRX's own dispatch profile (rocprofv3 does not see HRX dispatches)"

That sentence is worth a lot: it says the obvious instrument does not work on this backend and names the one that does, with the exact three commands. A tool that documents why the standard profiler is blind here saves the next person from a day of empty traces. And excluding the MTP block(s) unless --mtp, and counting KV at depth + n/2 (the mean position the n generated tokens read) instead of at depth, are both corrections that make the row mean what it says.

The one question still open — and top now answers it

docs/hrx.md:1071 says of this exact model: "flint UD-Q4_K_XL (routed experts) reads every weight stored". The tool models routed experts as expert_used_count / expert_count = 6/256. Ten times apart. Your own bytes on flint gives the two predictions, as a share of the decode step's device time:

weights_dense    1.673 GB
weights_experts  0.540 GB   =  24.4 % of 2.213 GB/step

tool's model     -> routed-expert kernels hold ~24 % of the step
docs line 1071   -> all 256 experts read   -> ~94 % of the step

So one top run on flint discriminates them: top EVENTS.jsonl --steps N, sum the expert-matmul keys, compare the share to 24 %. If it lands near 94 %, the bytes term is wrong for every MoE row this tool will ever print and docs/hrx.md:1071 is right; if it lands near 24 %, the doc sentence should be corrected (its paragraph is about duplicate copies, so it probably means footprint — but as written it says reads). Please put that comparison in the body as the tool's own first validation, because it is the cheapest possible test of the definition the whole 220 GB/s table rests on.

Still open from 6206d60

  1. print the spread beside median of {len(ts)} (engine#2369: 7.94 / 11.78 / 10.39 / 12.57 on four identical runs), and refuse --reps < 3 instead of printing median of 1;
  2. state the depth=0 KV caveat with the row (KV is 0.000 GB there; --depth moves it) — the new depth + n/2 wording in the docstring is good and the row itself should carry the same caveat.

Neither is a defect in the new code; both are the difference between a table someone can argue with and six numbers that look authoritative.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 435d18b

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 9b4c524

The new top subcommand is armed with the four quantities it computes, and both mutants I tried die on the numbers rather than on a missing key.

$ hrx_decode_bw.py --self-test        -> self-test passed

mutant A: divide by `steps + 1`       -> AssertionError: [('gemv_small', 2.0, 0.6, 0.667), ('loom_x', 1.333, 0.4, 0.333)]
mutant B: ignore the `--map` (no id resolution)
                                      -> AssertionError: [('k_aaaaaaaaaaaa', 3.0, 0.6, 1.0), ('loom_x', 2.0, 0.4, 0.5)]

The fixture is what makes those two separable: two dispatches of k_aaaaaaaaaaaa at 3000 ns plus one loom_x at 4000 ns over steps=2, and a map naming only the first. So the arm pins all four outputs at once --
gemv_small (the opaque id was resolved), 3.0 µs/step (2 x 3000 / 2), 0.6 share (6 of 10 µs), 1.0 dispatch/step, and dev == 5.0 µs per step for the total. A steps+1 off-by-one, which is exactly what the body warns about
("llama-bench's warm-up token counts as a step: give n + 1"), fails on the first two numbers; dropping the map fails on the name while leaving the arithmetic right. Two different defects, two different visible symptoms.

And the caveat about what the number means is stated in the tool, not only in the body: "Profiling slows a step down, so read the shares and the device-time total, not the step time it implies." That is the sentence that keeps top from being quoted as a
step-time measurement -- the same distinction as the device_gemm column including the weight fill (#2361).

One thing it explains that no other instrument here does: "top reads HRX's own dispatch profile (rocprofv3 does not see HRX dispatches)" -- with the two-command recipe (HRX_PROFILE_MODE=dispatch … then iree-profile dispatch --format=jsonl …). That is the missing link for anyone who has tried rocprofv3 on an HRX run and got
an empty timeline.

Unchanged from 6206d60: the bytes formula and its term-by-term arm; the three exceptions; and the cross-check that reproduces October's 176 GB/s.

Not run: top against a real profile (needs llama-bench + iree-profile + the HRX add-on) and run mode.

@bong-water-water-bong
bong-water-water-bong merged commit 16fdbca into main Nov 13, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the cl220/decode-bw-table branch November 13, 2026 14:15
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Measured: the depth=0 caveat is worth +85 % on a real model, not a footnote

I ran bytes on the flint pack on this box (header + config only, no device) across the depths the fleet actually serves at:

flint-UD-Q4_K_XL            total GB   KV GB    routed-expert GB   tok/s at 220 GB/s
--depth 0                     2.213     0.000        0.540              99.4
--depth 1024                  2.316     0.103        0.540              95.0
--depth 4096                  2.624     0.411        0.540              83.8
--depth 8192                  3.035     0.822        0.540              72.5
--depth 18800                 4.100     1.887        0.540              53.7

Two things this says, both about the row rather than the code:

  1. A depth=0 row is a best case that no long-context request sees. At 18.8k — the depth engine#679 is about — the KV term is 1.887 GB, 46 % of the step, and the model's speed-of-light at the 220 GB/s goal is 53.7 tok/s against 99.4 at depth 0. So a table that prints rows at depth=0 and labels them "effective bandwidth" will report a bandwidth roughly twice what a long-context request achieves, for the same model. Please print the depth in the row (not only in the invocation), so a depth=0 row cannot be compared against an --depth 18800 one.
  2. KV is a first-class term, not a correction. It grows 1.887/0.822 ≈ 2.3× from 8k to 18.8k, and on flint it is within a factor of 3.5 of all the weights in the model combined at 18.8k. That is worth a line in the docstring's kv paragraph: "at 18.8k on a 49-layer hybrid this term is comparable to the dense weights", because it tells a reader where to look when a long-context row comes out low.

Neither is a defect — the arithmetic is right and --depth exists. Both are why the six-model re-baseline needs its depth fixed before the table is drawn: with rows at different depths, the comparison would be measuring the depth, not the models.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 9b4c524

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Your MoE term is confirmed by a measurement, and the doc sentence it contradicts is now settled

I asked earlier for one top run to decide whether the routed-expert kernels hold ~24 % of a decode step (your expert_used_count / expert_count model) or ~94 % (docs/hrx.md:1071's "reads every weight stored"). I do not have the profile yet, but I served the model and got a number that settles it by arithmetic:

flint-UD-Q4_K_XL (40B-A2.5B, 6 of 256 experts) on HRX, median 42.61 tok/s (2 runs, 35.24–49.97, spread 1.418×).

your model:      2.213 GB/step × 42.61 tok/s =    94.3 GB/s effective   — plausible for this box
all 256 experts: 25.05 GB/step × 42.61 tok/s = 1,067.4 GB/s             — above the box's whole GTT bandwidth

So the tool's 6/256 is right, the 220 GB/s goal corresponds to ≈99.4 tok/s on this model, and docs/hrx.md:1071 must be describing resident copies, not bytes read per step. That row is worth correcting as part of this PR's first re-baseline, because as written it is the one sentence a reader would cite to argue your byte count is 10× low.

Two notes from the run for the table you are about to draw:

  • the depth caveat is now a real number for a real model: 2.213 GB/step at ~96 cached tokens, 4.100 GB/step at 18.8k — so the same model's speed-of-light moves from 99.4 to 53.7 tok/s at the 220 GB/s goal. The row must carry its depth.
  • 42.61 tok/s is a 2-run median with a 1.418× spread. By engine#2369's rule that is a sample, and the row should print 42.61 (35.24–49.97) rather than a single figure. medians.py refusing to print a median without its spread is exactly the tool this measurement wanted and did not have yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant