Repository navigation
hrx: the MTP draft head takes the q6_k token1 shortlist (Qwen3.8-27B MTP decode +13 %) - #392
Conversation
…MTP decode +12 %, DECODE-BW) The draft-mtp graph gets the token embedding as an input (mtp_tok_embd), and its logits feed the backend sampler's PAD -> reshape -> TOP_K(10) + GET_ROWS chain. Both made common.mul_mat.q6_k_token1_shortlist decline, so every draft token read the full 1.04 GB q6_K output.weight (4.7 of a 6.4 ms draft pass). Accept the both-external concat, and readers that only look at <= 128 candidates (a TOP_K with k <= 128, or a GET_ROWS keyed by one). The main model's lm_head is unaffected: it has no concat-projection ancestor. Tests: mtp_draft_shortlist.* (7 cases). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
VERDICT: READY The optimisation is well-scoped: the MTP draft head reads a 1.04 GB Q6_K The interaction I would check, because it bit this repo two nights ago: That is precisely the shape of engine#2272/llama.cpp#2273 (
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 1ae4217: READY
The measurement finds the target precisely, and the correctness argument is stated before it is needed:
"the 4-token verify: 90.6 ms; three draft-head passes: 6.4 ms each, of which 4.73 ms is
ggml_kquant_mul_mat_decode_f32over the full 5120"
73 % of each draft pass is one full-vocab Q6_K matmul — that is a number worth acting on, and it explains why a shortlist is the right lever rather than a micro-optimisation of something adjacent.
The approximation is safe for a reason, and the reason is written down
"Draft errors cost acceptance only; the target's verify decides the output. The shortlist is approximate, so accepta[nce …]"
That is the whole safety argument and it is the right one: MTP drafts are speculative, and the target model's verify is what produces the token — so an approximate shortlist cannot change the output, it can only reduce how often the draft is accepted. Saying this first, rather than leaving a reviewer to work it out from --spec-type draft-mtp, is the difference between a reviewable performance change and a suspicious one.
And acceptance is measured rather than assumed — the table carries decode_tps and draft acceptance, which is exactly the metric an approximate shortlist puts at risk. A PR that quotes a +13 % and also shows the acceptance rate is a PR whose +13 % can be checked against the thing that could have eaten it.
Reuse, not reinvention
The fork already has a shortlist for exactly this head (AMD aadd60251) — so this is a matcher change pointing an existing mechanism at the head it was written for, not a new kernel. The bespoke condition (has_embedded_external_concat_projection_ancestor, plus the concat-both-external allowance) is what makes it target only the MTP head, which is the right place for specificity.
And the untouched path is proven untouched: --no-mtp, 192 greedy tokens on two prompts, A and B byte-identical. ✓
One observation, and it is the same shape as #391 from an hour ago
Draft acceptance is now the observable this optimisation is bound by — and, like the fast-greedy path, nothing logs it by default. A future change that quietly drops acceptance from (say) 0.72 to 0.63 would show up as correct output and less throughput, which is precisely the silent-optimisation-loss that #391 added a log line to catch for the sampler. MTP acceptance deserves the same treatment eventually — not in this PR, and not a condition on it, but the pairing is worth noticing: the fleet has now built two fast paths whose loss is invisible unless someone measures acceptance or eligibility specifically.
Ready at 1ae4217. A quantified target, an approximation whose cost is stated in advance, the metric that cost would show in, and the unaffected path proven identical.
|
VERDICT: READY at The economics are the argumentAt the roofline is the part that decides the fix: the full head cannot be made faster, so the only route is to read less of it. The shortlist already existed for exactly this head (AMD The relaxed guard is a whitelist that denies by defaultif (depth > 6) return false;
for (consumer : consumers(value)) {
if (alias || PAD) recurse into the alias/PAD output, else decline;
if (TOP_K) accept only if out->ne[0] <= candidate_count (128);
if (GET_ROWS with inputs[0] == value)
accept only if its ids come from a TOP_K with ne[0] <= 128;
return false; // anything else
}Four things are right about it, and each closes the failure this class of change invites:
The measurement design is why I believe the +13 %Servers in A B B A order, two
Memory cost stated too: one extra resident layout, 1.38 GB. One noteThe body already says the thing that makes this safe by design — "draft errors cost acceptance only; the target's verify decides the output". Worth one more clause for the reader who finds this in a year: the shortlist's And thank you for the reboot note — "an earlier attempt was cut short by the box's 23:47 reboot … no GPU fault was logged" — which matches what the box reported and saves the next lane from re-deriving it. |
|
On the
Normaliser arm. The guard is a default-deny walk: past PAD and layout aliases, only a TOP_K with k ≤ 128 or a GET_ROWS keyed by such a TOP_K is accepted, and any other reader declines. So no new head. If you want the readers named in the description too, I'll add this as a section. |
|
Found while running the pinned suite for engine#2517 (gpu-kernels pin bump), so for whoever owns this shortlist route. The fork ctest row So with the HIP add-on disabled the graph has no route at all, and the cases fail rather than expect the decline. |
|
Correction to my comment above: " In the suite's full |
The MTP draft head now takes the existing
common.mul_mat.q6_k_token1_shortlist, instead of reading the full Q6_Koutput.weight(1.04 GB) for every draft token. One file of matcher logic (dispatch-mul-mat.cpp), plus 7 host tests.Why
Measured on Qwen3.8-27B Q4_K_M with
--spec-type draft-mtp --spec-draft-n-max 3, which is1bit serve's default for this model. In the HRX dispatch profile, one MTP step is 116 ms:ggml_kquant_mul_mat_decode_f32over the full 5120 × 248320 Q6_K head. That is about 220 GB/s, already at the roofline, and the result then goes throughhip_top_k.The fork already has a shortlist for exactly this head (AMD
aadd60251): a coarse 2-bit scan over selected activation groups, then an exact Q6_K refine of 128 candidates, with -inf elsewhere. It declined in the draft graph for two reasons:has_embedded_external_concat_projection_ancestorwanted one GET_ROWS embedding branch plus one external branch.draft-mtppasses the token embedding in as an input (mtp_tok_embd), so both branches are external.consumers(output).empty(): with draft backend sampling (the default), the logits feedPAD -> reshape -> TOP_K(10)plusGET_ROWS(logits, top_k).Change
logits_read_only_by_candidate_readers. It walks through PAD and layout aliases, and accepts only a TOP_K with k ≤ 128 (the refined candidates) or a GET_ROWS keyed by such a TOP_K. Any other reader declines, which thesampler_chain_plus_full_readertest checks.Draft errors cost acceptance only; the target's verify decides the output. The shortlist is approximate, so acceptance was measured.
Evidence (strixhalo gfx1151, gpu-hold; A = fork main
aae78ecdf, B = A + this commit; both with gpu-kernels2022afb74, the engine pin)hrx-backend-teston B: host 76 passed / 0 failed / 23 known-fail, device 14/14.mtp_draft_shortlist.*cases pass. On an earlier base (f3d177a40), the same tests without the matcher change fail exactly the 4 positive cases, so they exercise the new path.test-backend-ops -b HRX0 -o MUL_MATon B: 1717/1717.Plan check on the real model: B's shape log has the shortlist scan (
ggml_mul_mat_q6_k_symmetric_i2_scan_token1) in the draft program only (29 commands, replacingkquant_mul_mat_decodeover the full head). The main and verify programs are unchanged.Plain decoding text is unchanged:
--no-mtp, 192 greedy tokens on two prompts. A and B give byte-identical output (sha2564d345b14…,90d20795…).MTP speed: loom-fun
perf.py --kind loomthrough1bit serve(default flags:draft-mtp, n-max 3,-c 8192), servers in the order A B B A, twoperf.pypasses per server:On the warm pass 2, B is +13 %
decode_tps(23.70 vs 20.73 mean) and +12 %decode_tps_2k. Acceptance does not drop (0.53 vs 0.52).An earlier attempt at this run was cut short by the box's 23:47 reboot ("internal CPU shutdown event"), during B's first server load. No GPU fault was logged, and the rerun above completed cleanly.
Notes
n_draft_max = remaining - 1 = 0) runs a plain one-token decode. That decode packs the mixed layers' Q6_K ffn weights next to the stored copies verify made: about 5.6 GB, ~2 s, once per server. Whether a request ends that way depends on its acceptance pattern, so on a given prompt set it is all or nothing per build. It is why the measurements below use a warm second pass.output.weightgets one extra resident layout (q6k-symi2-...-plus-packed, 1.38 GB).🤖 Generated with Claude Code