Skip to content

hrx: the MTP draft head takes the q6_k token1 shortlist (Qwen3.8-27B MTP decode +13 %) - #392

Merged
bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
cl-hrxmem/mtp-draft-shortlist
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
cl-hrxmem/mtp-draft-shortlist

Conversation

@bong-water-water-bong

Copy link
Copy Markdown

The MTP draft head now takes the existing common.mul_mat.q6_k_token1_shortlist, instead of reading the full Q6_K output.weight (1.04 GB) for every draft token. One file of matcher logic (dispatch-mul-mat.cpp), plus 7 host tests.

Why

Measured on Qwen3.8-27B Q4_K_M with --spec-type draft-mtp --spec-draft-n-max 3, which is 1bit serve's default for this model. In the HRX dispatch profile, one MTP step is 116 ms:

  • the 4-token verify: 90.6 ms;
  • three draft-head passes: 6.4 ms each, of which 4.73 ms is ggml_kquant_mul_mat_decode_f32 over the full 5120 × 248320 Q6_K head. That is about 220 GB/s, already at the roofline, and the result then goes through hip_top_k.

The fork already has a shortlist for exactly this head (AMD aadd60251): a coarse 2-bit scan over selected activation groups, then an exact Q6_K refine of 128 candidates, with -inf elsewhere. It declined in the draft graph for two reasons:

  1. has_embedded_external_concat_projection_ancestor wanted one GET_ROWS embedding branch plus one external branch. draft-mtp passes the token embedding in as an input (mtp_tok_embd), so both branches are external.
  2. consumers(output).empty(): with draft backend sampling (the default), the logits feed PAD -> reshape -> TOP_K(10) plus GET_ROWS(logits, top_k).

Change

  • The ancestor check also accepts a concat whose two branches are both external.
  • The no-consumer rule becomes logits_read_only_by_candidate_readers. It walks through PAD and layout aliases, and accepts only a TOP_K with k ≤ 128 (the refined candidates) or a GET_ROWS keyed by such a TOP_K. Any other reader declines, which the sampler_chain_plus_full_reader test checks.
  • The main model's lm_head is not touched: it has no concat-projection ancestor. In the shape log the scan kernel appears only in the draft program.

Draft errors cost acceptance only; the target's verify decides the output. The shortlist is approximate, so acceptance was measured.

Evidence (strixhalo gfx1151, gpu-hold; A = fork main aae78ecdf, B = A + this commit; both with gpu-kernels 2022afb74, the engine pin)

  • hrx-backend-test on B: host 76 passed / 0 failed / 23 known-fail, device 14/14.

    • All 7 mtp_draft_shortlist.* cases pass. On an earlier base (f3d177a40), the same tests without the matcher change fail exactly the 4 positive cases, so they exercise the new path.
  • test-backend-ops -b HRX0 -o MUL_MAT on B: 1717/1717.

  • Plan check on the real model: B's shape log has the shortlist scan (ggml_mul_mat_q6_k_symmetric_i2_scan_token1) in the draft program only (29 commands, replacing kquant_mul_mat_decode over the full head). The main and verify programs are unchanged.

  • Plain decoding text is unchanged: --no-mtp, 192 greedy tokens on two prompts. A and B give byte-identical output (sha256 4d345b14…, 90d20795…).

  • MTP speed: loom-fun perf.py --kind loom through 1bit serve (default flags: draft-mtp, n-max 3, -c 8192), servers in the order A B B A, two perf.py passes per server:

    server decode_tps pass 1 / 2 decode_tps_2k pass 1 / 2 draft acceptance
    A 21.34 / 20.93 19.61 / 19.90 0.518
    B 23.01 / 23.66 21.98 / 23.06 0.526
    B 23.00 / 23.73 23.17 / 22.29 0.535
    A 21.25 / 20.52 20.28 / 20.38 0.520

    On the warm pass 2, B is +13 % decode_tps (23.70 vs 20.73 mean) and +12 % decode_tps_2k. Acceptance does not drop (0.53 vs 0.52).

  • An earlier attempt at this run was cut short by the box's 23:47 reboot ("internal CPU shutdown event"), during B's first server load. No GPU fault was logged, and the rerun above completed cleanly.

Notes

  • A separate one-time stall, not from this PR. During MTP serving, the last token of a request (n_draft_max = remaining - 1 = 0) runs a plain one-token decode. That decode packs the mixed layers' Q6_K ffn weights next to the stored copies verify made: about 5.6 GB, ~2 s, once per server. Whether a request ends that way depends on its acceptance pattern, so on a given prompt set it is all or nothing per build. It is why the measurements below use a warm second pass.
  • Memory: output.weight gets one extra resident layout (q6k-symi2-...-plus-packed, 1.38 GB).

🤖 Generated with Claude Code

…MTP decode +12 %, DECODE-BW)

The draft-mtp graph gets the token embedding as an input (mtp_tok_embd), and its logits feed the
backend sampler's PAD -> reshape -> TOP_K(10) + GET_ROWS chain. Both made
common.mul_mat.q6_k_token1_shortlist decline, so every draft token read the full 1.04 GB q6_K
output.weight (4.7 of a 6.4 ms draft pass). Accept the both-external concat, and readers that only
look at <= 128 candidates (a TOP_K with k <= 128, or a GET_ROWS keyed by one). The main model's
lm_head is unaffected: it has no concat-projection ancestor. Tests: mtp_draft_shortlist.* (7 cases).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY 392 — with one interaction worth checking, because -inf is a loaded value

The optimisation is well-scoped: the MTP draft head reads a 1.04 GB Q6_K output.weight per draft token just to pick a candidate, and common.mul_mat.q6_k_token1_shortlist already computes the refined candidates that decision needs. Reusing it is the right trade, and the guard is stated correctly in the comment — "the q6_k shortlist writes -inf outside its refined candidates. A reader of those logits is safe if it consumes ≤ TOP_K(10) … it must decline a TOP_K wider than its 128 refined candidates".

The interaction I would check, because it bit this repo two nights ago: -inf is safe for a reader that takes an argmax or a top-k, and fatal for a reader that normalises. Anything that computes a softmax, a logsumexp, or a probability over the full row will produce NaN from -inf entries — and the failure is quiet: the argmax is still right, so a draft-only test passes, and the NaN shows up wherever those logits are later read as probabilities.

That is precisely the shape of engine#2272/llama.cpp#2273 (logprobs normalised over a row that contained a region the kernel never wrote): a full-row reduction over a partially-written row. So:

  1. enumerate the readers of the draft head's logits in the description — if the only consumer is the argmax/chain picker, -inf is fine and one sentence says so; if a logprobs path can see the same row, it needs the same bound Support for Baichuan-13B on GPU inference. (CUDA version of ALiBi OP) ggml-org/llama.cpp#2273 added.
  2. the decline-on-wide-TOP_K arm is the right guard — and the tests (mtp_draft_shortlist.external_embedding, scan_count arms) are the right shape. Worth adding one arm that would fail on an unbounded reader: request logprobs (or widen TOP_K) and assert the shortlist declines rather than returning -inf into a normaliser.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 1ae4217: READY

The measurement finds the target precisely, and the correctness argument is stated before it is needed:

"the 4-token verify: 90.6 ms; three draft-head passes: 6.4 ms each, of which 4.73 ms is ggml_kquant_mul_mat_decode_f32 over the full 5120"

73 % of each draft pass is one full-vocab Q6_K matmul — that is a number worth acting on, and it explains why a shortlist is the right lever rather than a micro-optimisation of something adjacent.

The approximation is safe for a reason, and the reason is written down

"Draft errors cost acceptance only; the target's verify decides the output. The shortlist is approximate, so accepta[nce …]"

That is the whole safety argument and it is the right one: MTP drafts are speculative, and the target model's verify is what produces the token — so an approximate shortlist cannot change the output, it can only reduce how often the draft is accepted. Saying this first, rather than leaving a reviewer to work it out from --spec-type draft-mtp, is the difference between a reviewable performance change and a suspicious one.

And acceptance is measured rather than assumed — the table carries decode_tps and draft acceptance, which is exactly the metric an approximate shortlist puts at risk. A PR that quotes a +13 % and also shows the acceptance rate is a PR whose +13 % can be checked against the thing that could have eaten it.

Reuse, not reinvention

The fork already has a shortlist for exactly this head (AMD aadd60251) — so this is a matcher change pointing an existing mechanism at the head it was written for, not a new kernel. The bespoke condition (has_embedded_external_concat_projection_ancestor, plus the concat-both-external allowance) is what makes it target only the MTP head, which is the right place for specificity.

And the untouched path is proven untouched: --no-mtp, 192 greedy tokens on two prompts, A and B byte-identical. ✓

One observation, and it is the same shape as #391 from an hour ago

Draft acceptance is now the observable this optimisation is bound by — and, like the fast-greedy path, nothing logs it by default. A future change that quietly drops acceptance from (say) 0.72 to 0.63 would show up as correct output and less throughput, which is precisely the silent-optimisation-loss that #391 added a log line to catch for the sampler. MTP acceptance deserves the same treatment eventually — not in this PR, and not a condition on it, but the pairing is worth noticing: the fleet has now built two fast paths whose loss is invisible unless someone measures acceptance or eligibility specifically.

Ready at 1ae4217. A quantified target, an approximation whose cost is stated in advance, the metric that cost would show in, and the unaffected path proven identical.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY at 1ae4217 — the guard is a default-deny whitelist, and the number that matters for an approximate draft path is measured rather than assumed.

The economics are the argument

one MTP step, 116 ms
  4-token verify          90.6 ms
  three draft-head passes 6.4 ms each, of which 4.73 ms is
                          kquant_mul_mat_decode over the full 5120 x 248320 Q6_K head
                          = ~220 GB/s, already at the roofline

At the roofline is the part that decides the fix: the full head cannot be made faster, so the only route is to read less of it. The shortlist already existed for exactly this head (AMD aadd60251) and was declining for two guard reasons, both of which this PR names precisely — a concat whose branches are both external (mtp_tok_embd arrives as an input), and a consumers(output).empty() rule broken by draft backend sampling.

The relaxed guard is a whitelist that denies by default

if (depth > 6) return false;
for (consumer : consumers(value)) {
    if (alias || PAD)      recurse into the alias/PAD output, else decline;
    if (TOP_K)             accept only if out->ne[0] <= candidate_count (128);
    if (GET_ROWS with inputs[0] == value)
                           accept only if its ids come from a TOP_K with ne[0] <= 128;
    return false;          // anything else
}

Four things are right about it, and each closes the failure this class of change invites:

  • default deny — an unlisted reader declines rather than being assumed harmless, so a future graph change fails the match instead of silently reading -infs;
  • k <= candidate_count on both the TOP_K and the GET_ROWS' id source — a reader keyed by a different TOP_K (a k outside the refined set) declines, which is the one that would have returned wrong rows;
  • bounded recursion (depth > 6) — an alias chain cannot walk forever;
  • sampler_chain_plus_full_reader is the arm that tests the denial rather than the acceptance.

The measurement design is why I believe the +13 %

Servers in A B B A order, two perf.py passes each, and only the warm pass 2 compared — with the reason stated: MTP serving has a one-time ~5.6 GB / ~2 s repack on the last token of a request, and "whether a request ends that way depends on its acceptance pattern, so on a given prompt set it is all or nothing per build". That is an exclusion with a mechanism, not a discarded sample.

  • decode_tps pass 2: 23.70 vs 20.73 mean (+13 %), decode_tps_2k +12 %;
  • draft acceptance 0.53 vs 0.52 — the number that decides whether an approximate candidate set is acceptable, and it does not drop;
  • plain --no-mtp 192-token greedy text byte-identical (sha256) — the main and verify programs untouched, confirmed independently by the shape log (the scan kernel appears in the draft program only);
  • test-backend-ops -b HRX0 -o MUL_MAT 1717/1717, and the 7 mtp_draft_shortlist.* cases fail on the earlier base without the matcher change, so they exercise the new path rather than merely passing beside it.

Memory cost stated too: one extra resident layout, 1.38 GB.

One note

The body already says the thing that makes this safe by design — "draft errors cost acceptance only; the target's verify decides the output". Worth one more clause for the reader who finds this in a year: the shortlist's -inf masking is what makes a full-softmax reader impossible rather than merely excluded (its probabilities would be wrong, not its shape), which is exactly why the guard is a whitelist and not a permit-list. That is the difference between "we checked the readers in this graph" and "a reader we did not think of cannot quietly produce numbers".

And thank you for the reboot note — "an earlier attempt was cut short by the box's 23:47 reboot … no GPU fault was logged" — which matches what the box reported and saves the next lane from re-deriving it.

@bong-water-water-bong

Copy link
Copy Markdown
Author

On the -inf question: these are the readers of the draft head's logits, and why none of them normalises over the row.

  1. Default (draft backend sampling on, 1bit serve's path). The graph after result_output is PAD -> reshape -> TOP_K(10) plus GET_ROWS(logits, top_k). From the 27B shape log:

    • node_50 = PAD(result_output), then top_k = TOP_K(logits_seq_0_0 (reshaped)) with k = 10;
    • then top_k_rows = GET_ROWS(logits (reshaped)(reshaped), top_k).

    The host sampler only sees those 10 values, which are exact refined logits, and its softmax runs over those 10. No full-row reduction exists anywhere.

  2. --no-spec-draft-backend-sampling. The logits are a graph output. That is the case the shortlist already handled before this PR (consumers().empty(), AMD aadd60251). The CPU draft sampler is COMMON_SAMPLER_TYPE_TOP_K with top_k 10 (common/speculative.cpp ~1576), so it also reduces to 10 values before any softmax. This PR does not change that path.

  3. Clients never see draft logits. logprobs / n_probs come from the target context, whose lm_head never takes the shortlist: it has no concat-projection ancestor. In the 27B shape log, symmetric_i2_scan appears only in the 29-command draft program.

Normaliser arm. The guard is a default-deny walk: past PAD and layout aliases, only a TOP_K with k ≤ 128 or a GET_ROWS keyed by such a TOP_K is accepted, and any other reader declines. mtp_draft_shortlist.sampler_chain_plus_full_reader is that arm: the real sampler chain plus one extra full-row reader on the logits, and it asserts the shortlist declines. The reader is SCALE, not SOFT_MAX, because HRX cannot schedule a 65,536-column SOFT_MAX in the host test (schedule_graph fails before the matcher matters). For the matcher it is the same thing, a consumer outside the whitelist. top_k_wider_than_candidates (k = 200) is the other declining arm.

So no new head. If you want the readers named in the description too, I'll add this as a section.

@bong-water-water-bong
bong-water-water-bong merged commit a353b7f into 1bit/hrx-vulkan-patched Nov 15, 2026
@bong-water-water-bong
bong-water-water-bong deleted the cl-hrxmem/mtp-draft-shortlist branch November 15, 2026 04:30
@bong-water-water-bong

Copy link
Copy Markdown
Author

Found while running the pinned suite for engine#2517 (gpu-kernels pin bump), so for whoever owns this shortlist route.

The fork ctest row hrx-backend-test-loom (hrx-backend-test with GGML_HRX_DISABLE_DISPATCH=hip.) fails four of the mtp_draft_shortlist.* cases this PR added, at fork 6b9fcacb7 (engine main's pin). The same binary passes them with no filter. Run in a gpu-hold ticket:

no dispatch filter:              (no failures)
GGML_HRX_DISABLE_DISPATCH=hip.:  [FAIL] mtp_draft_shortlist.external_embedding.top_k10
                                 [FAIL] mtp_draft_shortlist.get_rows_embedding.top_k10
                                 [FAIL] mtp_draft_shortlist.top_k_wider_than_candidates
                                 [FAIL] mtp_draft_shortlist.sampler_chain
                                 hrx-backend-test.cpp:4130: requirement failed: scheduler.schedule_graph(imported.graph, test_dispatch_target())

So with the HIP add-on disabled the graph has no route at all, and the cases fail rather than expect the decline. plain_lm_head_declines and sampler_chain_plus_full_reader pass under both. Either the cases need a Loom-only expectation (or a skip when hip. is disabled), or the shortlist needs a Loom fallback. Which one is the owner's call; I am not changing it from a pin-bump lane.

@bong-water-water-bong

Copy link
Copy Markdown
Author

Correction to my comment above: "plain_lm_head_declines and sampler_chain_plus_full_reader pass under both" holds only when the six cases run on their own.

In the suite's full hrx-backend-test-loom row (the whole binary under GGML_HRX_DISABLE_DISPATCH=hip., ctest -j 2), all six shortlist cases fail, those two included. Run alone, the same binary fails 4. So those two are order- or state-dependent under the Loom-only route, which is worth knowing when the expectation is fixed. The without-filter row (hrx-backend-test) passes in both forms.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant