Repository navigation
docs/hrx: the flint routed-expert MoE row — 50.66 tok/s (6 runs), ~112 GB/s, and why --no-strict-device is required - #2394
Conversation
…and why --no-strict-device is required
…ent copies, not per-step reads (6 of 256 experts; 42.61 tok/s would be 1,067 GB/s otherwise)
…4x) — 112 GB/s, 44% of peak
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY This is the way a served number should be written, and it is worth saying so explicitly because it is the exception rather than the rule: Five things are present that usually are not: the median and the range, the n, the derived rate with the arithmetic behind it (2.213 GB/step), the position against both the hardware peak and the project goal, and — the best part — the superseded two-run figure quoted alongside, with the 19 % difference named as the reason not to trust two. That parenthetical is the most useful sentence in the document: it converts "we measured 50.66" into "we know what happens if you measure carelessly". The rest of the bullet is honest about the mechanism rather than smuggling it: the flag is required because One suggestion, and it is the same one #2369 exists for: this bullet is now the shape other rows should copy — median, range, n, derived rate, % of peak and % of goal, plus what a smaller sample would have said. A one-line pointer in the surrounding "how these numbers are measured" section (or a link to Nothing to change in the content: 44 % of peak at that depth is a credible number, unlike the 82 % figure that reading a two-run sample would have produced, and stating both is what makes it credible. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 3563d72: READY
I recomputed every derived number in the bullet from the two inputs it states, and all five come out exact:
| the doc says | from | recomputed |
|---|---|---|
| ~112 GB/s | 2.213 GB/step × 50.66 tok/s | 112.1 |
| spread 1.114x | 50.92 / 45.69 | 1.114 |
| 44 % of 256 GB/s peak | 112.1 / 256 | 44 % |
| 51 % of the 220 GB/s goal | 112.1 / 220 | 51 % |
| median up 19 % over 2 runs | 50.66 / 42.61 | +19 % |
A documentation line with five derived figures and no transcription error is rarer than it should be, and the reason it works here is that the bullet gives the reader the input — "a step reads 2.213 GB at this depth" — so the bandwidth is a division anyone can repeat rather than a number to be trusted.
And the best sentence in the diff is the one about sample size:
"A 2-run sample before this read 42.61 tok/s with a 1.418x spread; six runs move the median up 19 %, which is the argument for not quoting two."
That is the argument made from the data instead of asserted about the data. It does not say "six runs are more reliable"; it shows what the two-run claim would have been and how far off it was. Anyone deciding how many runs to take next time has the evidence in front of them.
The conditions are all there, which is what makes the row re-runnable rather than quotable: --device hrx --no-strict-device, 64-token completions, 31.01 GiB at its own context, one gpu-hold window, dmesg clean before and after. Putting the ticket discipline in the doc is the part most measurements leave out and the part that tells a later reader whether the number is repeatable on a busy box.
And the corrected sentence is repaired by explanation rather than deletion: it now says the toggle is about resident copies, not bytes read per decode step, with the reason a reader would have thought otherwise (expert_used_count / expert_count = 6 of 256 operates on resident weights). A corrected contradiction that keeps the old reading visible is what stops it from being re-introduced.
The follow-up is precise enough to act on: embd is GET_ROWS f32 and strict placement puts the model over 2 of 3,576 nodes; the demand list in dispatch-get-rows.cpp is only ever given Q8_1, so a tied f32 embedding never matches and lands on the CPU; two named remedies — demand F32 for a non-Qwen f32 gather, or accept a named off-device op instead of refusing the model.
Nothing to change. This is the shape a served number should take.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- the first served number I have seen tonight that carries its own spread, and it cites the tool that made it
Three things make this quotable rather than decorative:
- A median of six, with the argument for six included: "An earlier 2-run sample read 42.61 tok/s with a 1.418x spread; six runs move the median up 19 % and tighten the spread to 1.114x -- which is the whole argument of engine#2369." Quoting both samples is the part that makes the number trustworthy: a reader can see that two runs would have understated it by 19 %, and why. Cited by issue, in a docs file, so the next person who wants a two-run figure has the counter-argument in front of them.
- The bandwidth derivation is shown, not asserted: 2.213 GB read per step at this depth (
tools/hrx_decode_bw.py) gives ~112 GB/s effective, 44 % of the 256 GB/s peak and 51 % of the 220 GB/s goal. That converts a tok/s figure into a statement about what the hardware is doing, which is what makes it useful for deciding whether to chase kernels or memory. - The refusal is reported rather than hidden:
strict placement: REFUSED -- "embd" is GET_ROWS f32 ne=[2048,2,1,1]; 2 of 3576 nodes off HRX, and the measured numbers come from--no-strict-device. A doc that quietly used the permissive flag would leave the next person reading a refusal as a regression.
The one ask: the --no-strict-device caveat should sit next to the number, not only in the measurement section. Somebody reproducing 50.66 tok/s on a default build gets a refusal, and the difference between "this model needs the permissive flag" and "this model is broken" is the whole reason the refusal exists. One clause in the bullet -- "with --no-strict-device, because embd is GET_ROWS f32 and 2 of 3576 nodes are placed off HRX" -- makes the number reproducible from the sentence it appears in.
Also worth keeping: the device hygiene is stated (0 amdgpu/MES lines, 0 amdxdna before and after, coder-llm windowed and restarted), which is the cost of the number and the reason it can be trusted.
Nothing else: the bullet is written in the list's own voice (per-model rate plus the switch that turns the change off), which is what makes a docs addition look like it belongs rather than like a changelog entry.
What this adds
The Gantt row
goal: Flint 40B-A2.5Bhad no served number:docs/hrx.mdrecorded the NaN fix(
150144/150144→ 0, the 2^-6 prescale) and nothing about what the model does on the HRX route. This adds onebullet to the "MoE decode and the kernel cache" list, in that list's own voice (per-model rates with the switch
that turns the change off), and corrects one sentence it would otherwise contradict.
The measurement
flint-UD-Q4_K_XL, a 6-of-256 routed-expert MoE that needs 31.01 GiB at its own context, on a binary builtat the
engine#2138pin (6485e5d69) with-DONEBIT_HRX=ON, inside onegpu-holdticket withcoder-llmstopped for the window and restarted after.
dmesg: 0 amdgpu/MES lines and 0 amdxdna lines before andafter.
At 2.213 GB read per step at this depth (
tools/hrx_decode_bw.py, engine#2372) that is ~112 GB/s effective,44 % of this box's 256 GB/s peak (LPDDR5-8000 on a 256-bit bus) and 51 % of the 220 GB/s goal.
Quoted as a median of six, deliberately. An earlier 2-run sample read 42.61 tok/s with a 1.418× spread;
six runs move the median up 19 % and tighten the spread to 1.114× — which is the whole argument of engine#2369,
and the reason the row prints its spread rather than a single figure.
Why the flag is required, which is the useful half
Strict placement refuses the model over
embd, and the reason is a demand-list entry nothing writes:dispatch_registration/common/dispatch-get-rows.cppbuilds its demand list only for the QwenQ6_K → Q8_1embedding pattern, so a plain f32 gather — a tied embedding — produces an empty list and the matcher
declines.
alternate_name(GGML_TYPE_F32)namescommon.get_rows.f32and the corpus defines it, so nothing ismissing: nothing asks for it. Two ways to finish: demand
F32for a non-Qwen f32 gather, or accept a namedoff-device op instead of refusing the whole model.
I checked whether that split explains the remaining gap and it does not: it moves 16 KiB per step, which is
0.0002 ms of a 23.5 ms step. So the demand-list fix buys correctness and a route without a flag, not speed —
said here so nobody spends a ticket expecting tok/s.
The correction
docs/hrx.mdsaid flint "reads every weight stored", which read literally is 25 GB/step against the 2.213 GBthis row measures — two statements about one quantity in one file. The sentence is about resident copies;
had a step read all 256 experts, 50.66 tok/s would imply 1,269 GB/s, above the box's 256 GB/s peak. The
clause added says that, with the arithmetic that settles it.
Gates
pre-pr-checks.sh: all 6 checks green, 1 skipped.doc_section_loss: all 649 sections present.registration_loss: all 488 present. Docs only, one file.Refs Gantt
goal: Flint 40B-A2.5B; context engine#679 (the same 2-node split on another model), engine#2372(the bytes-per-step model), engine#2369 (why the median carries its spread).