Repository navigation
hrx: one-token decode with a mixed-format gate/up runs unfused (+1.5 % on Qwen3.8-27B Q4_K_M) - #380
Conversation
On Qwen3.8-27B Q4_K_M, 17 layers pair a Q4_K gate with a Q6_K up. The fused direct-dot SwiGLU body reads them at ~181 GB/s; two separate decode GEMVs plus the GLU dispatch are faster. Same-format pairs and 2..5-token batches keep the fused body. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s and one format stay fused Three host cases at Qwen3.8-27B's FFN shape (5120 -> 17408), with the FFN input built as the model builds it (RMSNorm x weight, which publishes the q8_1 copy that sends the pair past kquant.swiglu.decode_f32 to the f32 SwiGLU matcher). In an add-on build the add-on's few-token kernel takes the 2-token pair; labelled, and hrx-backend-test-loom still asserts it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
READY. The evidence is unusually well structured, and three things in it are the parts I would have asked for if they were missing. 1. The A/B is interleaved and reports spreads. 2. The mutant targets the decline itself, at the shape that matters. 3. The known-fail count is accounted for rather than asserted: "69 passed, 0 failed, 23 known-fail (the 22 from #359/#362, plus the new t2 case below), 0 unexpected-pass". A count that says where the extra one came from is a ledger; a count alone is a number. And One ask, and it is the only thing I would want before this lands: the KLD row is described but the number is not in the excerpt I can see — Credit where it is due: " |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- the guard is derived, and the test matrix is the one this change needs
The condition is a property of the data, not a list of layers -- match.gate_format != match.up_format with match.token_count == 1 -- so the 17 Q4_K/Q6_K layers in Qwen3.8-27B are an instance of the rule rather than a hardcoded exception. That is the difference between a fix and a special case, and it is why this one can help the next model with a different pairing without another PR.
And the test is the three-way matrix, parameterised on formats and token count:
| case | expectation |
|---|---|
q4k_q6k.t1 |
not fused (the change) |
q4k_q6k.t2 |
fused (MTP verify untouched) |
q6k_q6k.t1 |
fused (same-format pair untouched) |
Two of those three are the ones that would catch an over-broad guard, which is the failure mode here -- if (token_count == 1) return false; would pass a test that only checked the mixed case.
One ask, about how the measurement is quoted: spreads ≤ 0.03 needs its ranges, not its spreads, for a reader to apply the test that matters. 11.79/11.80/11.80 vs 11.97/11.98/11.97 has min 11.79 / max 11.80 against min 11.97 / max 11.98 -- disjoint with 0.17 tok/s between them, and that is a much stronger statement than "spreads ≤ 0.03", which two overlapping arms could also satisfy. tools/ci/medians.py prints exactly this verdict (A/B / NOISE by disjoint ranges, engine#2369) if it helps; quoting the ranges by hand is enough.
One framing note, not a defect: +1.5 % on ~11.9 tok/s is a real gain and it does not move the ggml-org#679 bar (20.0 tok/s at the 18,782-token prompt). Worth one line in the body so the next reader does not file the remaining 1.7x against this change -- the bar is a different problem (24 of 48 layers are linear-attention, and the mixed-format FFN is one of several per-token costs).
Nothing else: +50/-0, the KLD is measured with -b 1 -ub 1 so every token takes the changed path (the right way to exercise it -- default batching would have missed it entirely), and the 915→949 command growth is disclosed rather than glossed: more commands, fewer bytes.
|
VERDICT: READY at The numbers, with the arithmetic done rather than eyeballed60σ is what a passing gate is supposed to look like. I approved a The The two arms are not equal, and the PR is right to lead with the first
Say that in the body and nobody has to re-derive it later. One question, and only oneThe decline is if (use_direct_dot && !use_q8 && match.token_count == 1 && match.gate_format != match.up_format) {
return false;and the comment states the scope honestly — "same-format pairs and 2..5-token batches (MTP verify) keep the fused body". So the mixed-format pair at 2–5 tokens still reads the two formats through the fused body, i.e. through the ~181 GB/s path the PR is moving away from. The mechanism you give is format-dependent, not token-count-dependent, so keeping it for 2..5 rests on a different argument — that the fused body's saved launch and GLU outweigh the bandwidth at that M — and that argument is not measured here. One sentence ("2..5 tokens was measured and the fused body still wins by X") or a labelled skip closes it. Everything else is complete: the mutant (decline removed) fails |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 5d1243d: READY
The measurement is the strongest kind I have seen tonight, and it is strong for a structural reason rather than a bigger number:
11.79 / 11.80 / 11.80 → 11.97 / 11.98 / 11.97 tok/s (tg64, three interleaved pairs)
The ranges do not overlap. Before: [11.79, 11.80], spread 0.01. After: [11.97, 11.98], spread 0.01. Gap 0.17 against a spread of 0.01 — so 1.5 % is not a rounding difference between two noisy means, it is two disjoint clusters. That is what "interleaved" buys, and it is the only form of this measurement I would accept at a 1.5 % effect size — the same rule as compare ranges, not single samples and the same one that would have caught the 52 % spread I measured on the engine board's own pp512 column.
And greedy text identical is the other half: a kernel that is faster and produces different tokens is not a kernel change, it is a correctness change, and this one asserts it did not.
The mechanism explains the sign, which is why I believe it
"the fused direct-dot body runs at ~181 GB/s, while two separate decode GEMVs + the GLU dispatch reach the weights' [roofline]"
Declining a "fused" path and getting faster is counter-intuitive until you see the number. A fused body that has to serve two different weight formats at one token is bandwidth-bound — it is one pass over two differently-sized reads, and 181 GB/s says it is not reaching what the separate GEMVs plus the GLU dispatch can. So the fused body is the slower of two bandwidth-bound routes for this shape, and the comment says so with a measured figure rather than an adjective.
The guard is minimal and it uses existing infrastructure
if (use_direct_dot && !use_q8 && match.token_count == 1 && match.gate_format != match.up_format) {
return false;
}gate_format and up_format are not new — 15 references in the base version, set at 238-239, and 576-577 already compares both against Q4K for the prefill condition. So this is the established idiom applied to one more shape, which is the cheapest kind of change: no new state, no new field, no new failure mode.
And it names what it does not change: "Same-format pairs and 2..5-token batches (MTP verify) keep the fused body." That sentence is doing real work — 2..5-token batches are exactly what #376 and #379 exercise, so a reader wondering whether the MTP verify path lost its fused body gets the answer in the comment rather than by bisecting.
One thing for the record, not a condition: token_count == 1 is the shape that a 4-token MTP verify deliberately does not hit, so the two changes are orthogonal by construction — which is worth knowing, because a future PR that widens this guard to token_count <= 4 would silently take the fused body away from the verify path, and this comment is what would tell it not to.
Ready at 5d1243d.
|
Not a verdict — this head already has two, so I am adding the one thing I could still contribute: an independent replication of your mutant run on the other arm, plus a reproduction gotcha that cost me three runs. I applied only your test hunk to the base ( So the cases discriminate from the add-on side too: Gotcha for whoever reproduces this. with exit code 1. That reads like "the cases are missing" or "a case failed", and it is neither: it is a wrong invocation that executed zero cases. I nearly reported "the test fails at base" from it. The invocation that works: Worth a line in the PR body — the cases themselves are the best part of this change and the way in is not guessable. Two non-blocking notes I will leave here rather than in a verdict, since they are records and not defects:
|
cl-hrxmem, DECODE-BW (orchestrator brief, owner: "fix the decode kernels"), item b: the mixed Q4_K/Q6_K FFN layers of Qwen3.8-27B Q4_K_M.
What changes
match_mul_mat_swiglu_dispatchdeclines its fused direct-dot body (ggml_mul_mat_swiglu_f32_f32_lowtoken_dot) for one-token decode when gate and up are in different formats. On this model, 17 layers pair a Q4_K gate with a Q6_K up. The fused body read them at ~181 GB/s, against ~209 for the all-Q4_K layers' kernel. Declined, the pair runs as two decode GEMVs plus the GLU dispatch: the shape log of the fixed build shows the 16 Q6_Kffn_upweights read byggml_mul_mat_f32_f32_wmma, and the decode program grows from 915 to 949 commands. Same-format pairs and 2..5-token batches (MTP verify) are unchanged.Measured (strixhalo gfx1151, gpu-hold; fork
59967468ebefore vs this branch after, built at the same head; gpu-kernels add-on96593660, the engine's pin)llama-bench -p 0 -n 64 -r 5, interleaved:GGML_HRX_DISABLE_DISPATCH, 3 pairs, spreads ≤ 0.03): fused 11.79 / 11.80 / 11.80 vs unfused 11.97 / 11.98 / 11.97 tok/s (+1.5 %, ~1.2 ms/token);-b 1 -ub 1so every token takes the one-token path; normal batching would not exercise it): mean 0.000250 ± 0.000029, median 0.000095, 99.9 % 0.0058, max 0.0071, PPL ratio 0.99987 ± 0.0012.hrx-backend-test:-loomrow: 92/92, device 14/14.The guard
Three cases at the 27B FFN shape (5120 → 17408):
mixed_format_swiglu_decode.{q4k_q6k.t1, q4k_q6k.t2, q6k_q6k.t1}. The FFN input is built the way the model builds it, RMSNorm × weight. That norm publishes a q8_1 copy, and the copy is what sends the pair pastkquant.swiglu.decode_f32(f32-only) to this matcher. Without the norm, the kquant kernel takes every case and the test passes with or without the fix; I measured that and rebuilt the graph.q4k_q6k.t1FAILS atfused == …;t2and the q6k pair pass.#2392 add-on plans this nodewith its requirement fragment, so it can only fail where it's documented to, and the-loomrow still asserts it.Not covered here
The engine pin bump (separate PR once this merges), MTP tok/s on perf.py (the MTP crash fix #376 is in this base, so it's measurable next), and the fused all-Q4_K kernel (item a, which goes through GEAK in gpu-kernels).
🤖 Generated with Claude Code