Skip to content

docs/hrx: the flint routed-expert MoE row — 50.66 tok/s (6 runs), ~112 GB/s, and why --no-strict-device is required - #2394

Merged
bong-water-water-bong merged 3 commits into
mainfrom
os/v3-flint-row
Nov 14, 2026
Merged

bong-water-water-bong merged 3 commits into
mainfrom
os/v3-flint-row

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

What this adds

The Gantt row goal: Flint 40B-A2.5B had no served number: docs/hrx.md recorded the NaN fix
(150144/150144 → 0, the 2^-6 prescale) and nothing about what the model does on the HRX route. This adds one
bullet to the "MoE decode and the kernel cache" list, in that list's own voice (per-model rates with the switch
that turns the change off), and corrects one sentence it would otherwise contradict.

The measurement

flint-UD-Q4_K_XL, a 6-of-256 routed-expert MoE that needs 31.01 GiB at its own context, on a binary built
at the engine#2138 pin (6485e5d69) with -DONEBIT_HRX=ON, inside one gpu-hold ticket with coder-llm
stopped for the window and restarted after. dmesg: 0 amdgpu/MES lines and 0 amdxdna lines before and
after.

strict placement:   REFUSED — "embd" is GET_ROWS f32 ne=[2048,2,1,1]; 2 of 3576 nodes off HRX, 2 splits
--no-strict-device: 45.69 / 49.44 / 50.92 / 50.86 / 50.57 / 50.76 tok/s  (64-token completions)
                    median 50.66, spread 1.114x

At 2.213 GB read per step at this depth (tools/hrx_decode_bw.py, engine#2372) that is ~112 GB/s effective,
44 % of this box's 256 GB/s peak (LPDDR5-8000 on a 256-bit bus) and 51 % of the 220 GB/s goal.

Quoted as a median of six, deliberately. An earlier 2-run sample read 42.61 tok/s with a 1.418× spread;
six runs move the median up 19 % and tighten the spread to 1.114× — which is the whole argument of engine#2369,
and the reason the row prints its spread rather than a single figure.

Why the flag is required, which is the useful half

Strict placement refuses the model over embd, and the reason is a demand-list entry nothing writes:
dispatch_registration/common/dispatch-get-rows.cpp builds its demand list only for the Qwen Q6_K → Q8_1
embedding pattern, so a plain f32 gather — a tied embedding — produces an empty list and the matcher
declines. alternate_name(GGML_TYPE_F32) names common.get_rows.f32 and the corpus defines it, so nothing is
missing: nothing asks for it. Two ways to finish: demand F32 for a non-Qwen f32 gather, or accept a named
off-device op instead of refusing the whole model.

I checked whether that split explains the remaining gap and it does not: it moves 16 KiB per step, which is
0.0002 ms of a 23.5 ms step. So the demand-list fix buys correctness and a route without a flag, not speed —
said here so nobody spends a ticket expecting tok/s.

The correction

docs/hrx.md said flint "reads every weight stored", which read literally is 25 GB/step against the 2.213 GB
this row measures — two statements about one quantity in one file. The sentence is about resident copies;
had a step read all 256 experts, 50.66 tok/s would imply 1,269 GB/s, above the box's 256 GB/s peak. The
clause added says that, with the arithmetic that settles it.

Gates

pre-pr-checks.sh: all 6 checks green, 1 skipped. doc_section_loss: all 649 sections present.
registration_loss: all 488 present. Docs only, one file.

Refs Gantt goal: Flint 40B-A2.5B; context engine#679 (the same 2-node split on another model), engine#2372
(the bytes-per-step model), engine#2369 (why the median carries its spread).

bcloud added 3 commits November 13, 2026 21:46
…ent copies, not per-step reads (6 of 256 experts; 42.61 tok/s would be 1,067 GB/s otherwise)
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Measurement Claim Verification

The PR claims that the flint-UD-Q4_K_XL model serves at 50.66 tok/s with a spread of 1.114x, based on 6 runs. However, the PR description does not provide any verification that these measurements were actually taken on the specified hardware (AMD Strix Halo's XDNA 2 NPU, Radeon iGPU) under the stated conditions (gpu-hold ticket, coder-llm stopped, dmesg clean). Without confirmation that the numbers are measured on the hardware, this claim cannot be verified.

- **Routed-expert MoE on the HRX route (flint-40b-a2.5b).** `flint-UD-Q4_K_XL`, a 6-of-256 MoE that needs
  31.01 GiB at its own context, serves with `--device hrx --no-strict-device`: 64-token completions at
  **50.66 tok/s**, median of **6 runs** (45.69–50.92, spread 1.114x). A step reads 2.213 GB at this depth, so
  that is **~112 GB/s effective**, **44 % of this box's 256 GB/s peak and 51 % of the 220 GB/s goal**. (A 2-run
  sample before this read 42.61 tok/s with a 1.418x spread; six runs move the median up 19 %, which is the
  argument for not quoting two.) The flag is required because `embd` is `GET_ROWS` f32 `ne=[2048,2,1,1]` and strict placement refuses
  the model over those 2 of 3,576 nodes; see `dispatch-get-rows.cpp`, whose demand list is only ever given
  `Q8_1`, so a tied f32 embedding never matches and lands on the CPU. Two rows would finish it: demand `F32`
  for a non-Qwen f32 gather, or accept a named off-device op instead of refusing the model. Measured
  2026-11-14 under one gpu-hold window, `dmesg` clean before and after.
Hardware Performance Assertion

The PR asserts that the effective bandwidth of ~112 GB/s is 44% of the box's 256 GB/s peak and 51% of the 220 GB/s goal. These numerical claims require verification on the actual hardware to ensure they are accurate and not extrapolated or misestimated.

**50.66 tok/s**, median of **6 runs** (45.69–50.92, spread 1.114x). A step reads 2.213 GB at this depth, so
that is **~112 GB/s effective**, **44 % of this box's 256 GB/s peak and 51 % of the 220 GB/s goal**. (A 2-run
sample before this read 42.61 tok/s with a 1.418x spread; six runs move the median up 19 %, which is the

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 3563d72

This is the way a served number should be written, and it is worth saying so explicitly because it is the exception rather than the rule:

64-token completions at 50.66 tok/s, median of 6 runs (45.69-50.92, spread 1.114x).
A step reads 2.213 GB at this depth, so that is ~112 GB/s effective,
44 % of this box's 256 GB/s peak and 51 % of the 220 GB/s goal.
(A 2-run sample before this read 42.61 tok/s with a 1.418x spread; six runs move the median up 19 %,
 which is the argument for not quoting two.)

Five things are present that usually are not: the median and the range, the n, the derived rate with the arithmetic behind it (2.213 GB/step), the position against both the hardware peak and the project goal, and — the best part — the superseded two-run figure quoted alongside, with the 19 % difference named as the reason not to trust two. That parenthetical is the most useful sentence in the document: it converts "we measured 50.66" into "we know what happens if you measure carelessly".

The rest of the bullet is honest about the mechanism rather than smuggling it: the flag is required because embd is GET_ROWS f32 ne=[2048,2,1,1] and strict placement would route it, and that is stated as a consequence rather than a preference.

One suggestion, and it is the same one #2369 exists for: this bullet is now the shape other rows should copy — median, range, n, derived rate, % of peak and % of goal, plus what a smaller sample would have said. A one-line pointer in the surrounding "how these numbers are measured" section (or a link to medians.py) would make it a convention instead of one author's good habit.

Nothing to change in the content: 44 % of peak at that depth is a credible number, unlike the 82 % figure that reading a two-run sample would have produced, and stating both is what makes it credible.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 3563d72: READY

I recomputed every derived number in the bullet from the two inputs it states, and all five come out exact:

the doc says from recomputed
~112 GB/s 2.213 GB/step × 50.66 tok/s 112.1
spread 1.114x 50.92 / 45.69 1.114
44 % of 256 GB/s peak 112.1 / 256 44 %
51 % of the 220 GB/s goal 112.1 / 220 51 %
median up 19 % over 2 runs 50.66 / 42.61 +19 %

A documentation line with five derived figures and no transcription error is rarer than it should be, and the reason it works here is that the bullet gives the reader the input — "a step reads 2.213 GB at this depth" — so the bandwidth is a division anyone can repeat rather than a number to be trusted.

And the best sentence in the diff is the one about sample size:

"A 2-run sample before this read 42.61 tok/s with a 1.418x spread; six runs move the median up 19 %, which is the argument for not quoting two."

That is the argument made from the data instead of asserted about the data. It does not say "six runs are more reliable"; it shows what the two-run claim would have been and how far off it was. Anyone deciding how many runs to take next time has the evidence in front of them.

The conditions are all there, which is what makes the row re-runnable rather than quotable: --device hrx --no-strict-device, 64-token completions, 31.01 GiB at its own context, one gpu-hold window, dmesg clean before and after. Putting the ticket discipline in the doc is the part most measurements leave out and the part that tells a later reader whether the number is repeatable on a busy box.

And the corrected sentence is repaired by explanation rather than deletion: it now says the toggle is about resident copies, not bytes read per decode step, with the reason a reader would have thought otherwise (expert_used_count / expert_count = 6 of 256 operates on resident weights). A corrected contradiction that keeps the old reading visible is what stops it from being re-introduced.

The follow-up is precise enough to act on: embd is GET_ROWS f32 and strict placement puts the model over 2 of 3,576 nodes; the demand list in dispatch-get-rows.cpp is only ever given Q8_1, so a tied f32 embedding never matches and lands on the CPU; two named remedies — demand F32 for a non-Qwen f32 gather, or accept a named off-device op instead of refusing the model.

Nothing to change. This is the shape a served number should take.

@bong-water-water-bong
bong-water-water-bong merged commit 0f49e25 into main Nov 14, 2026
9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the os/v3-flint-row branch November 14, 2026 02:29

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- the first served number I have seen tonight that carries its own spread, and it cites the tool that made it

Three things make this quotable rather than decorative:

  1. A median of six, with the argument for six included: "An earlier 2-run sample read 42.61 tok/s with a 1.418x spread; six runs move the median up 19 % and tighten the spread to 1.114x -- which is the whole argument of engine#2369." Quoting both samples is the part that makes the number trustworthy: a reader can see that two runs would have understated it by 19 %, and why. Cited by issue, in a docs file, so the next person who wants a two-run figure has the counter-argument in front of them.
  2. The bandwidth derivation is shown, not asserted: 2.213 GB read per step at this depth (tools/hrx_decode_bw.py) gives ~112 GB/s effective, 44 % of the 256 GB/s peak and 51 % of the 220 GB/s goal. That converts a tok/s figure into a statement about what the hardware is doing, which is what makes it useful for deciding whether to chase kernels or memory.
  3. The refusal is reported rather than hidden: strict placement: REFUSED -- "embd" is GET_ROWS f32 ne=[2048,2,1,1]; 2 of 3576 nodes off HRX, and the measured numbers come from --no-strict-device. A doc that quietly used the permissive flag would leave the next person reading a refusal as a regression.

The one ask: the --no-strict-device caveat should sit next to the number, not only in the measurement section. Somebody reproducing 50.66 tok/s on a default build gets a refusal, and the difference between "this model needs the permissive flag" and "this model is broken" is the whole reason the refusal exists. One clause in the bullet -- "with --no-strict-device, because embd is GET_ROWS f32 and 2 of 3576 nodes are placed off HRX" -- makes the number reproducible from the sentence it appears in.

Also worth keeping: the device hygiene is stated (0 amdgpu/MES lines, 0 amdxdna before and after, coder-llm windowed and restarted), which is the cost of the number and the reason it can be trusted.

Nothing else: the bullet is written in the list's own voice (per-model rate plus the switch that turns the change off), which is what makes a docs addition look like it belongs rather than like a changelog entry.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant