speculative : give MTP targets their recurrent rollback slots back - #26
Conversation
f169e0b dropped DRAFT_MTP from need_n_rs_seq(), so an MTP target context is created with n_rs_seq = 0 and every partially accepted draft goes through the checkpoint restore path: restore the state, then decode the accepted tokens a second time as a replay batch. On Qwen3.8-27B that is a second target forward pass on most steps (90 draft decodes for 48 accepted steps in a verbose run, against 62 for 62 on mainline). Put MTP back on the list, as upstream has it. The target then rolls the recurrent state back in place and the replay disappears. Cost: the recurrent state holds n_max extra slots (150 MiB -> 750 MiB on this model). Qwen3.8-27B UD-Q4_K_XL, Vulkan (RADV, Mesa 26.1.7), MTP n_max 4, decode tok/s, 3 runs: 9.6/9.8/9.9 on master -> 12.0/12.2/12.0 with this change; with GGML_VK_MMV_NO_SPLIT=1 on top, 19.9/20.0/20.0, matching ggml-org master (19.7/19.7/20.2). Fixes #25 together with the mat-vec chunking change. Assisted-by: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01LTnaB4ucGc5vkzctYns4Nq
|
Tested baseline c7af5c6 against #26 + #27 on my second Strix Halo with RADV 25.3.6. With --spec-type draft-mtp --spec-draft-n-max 3 --ubatch-size 4, the baseline works, but the candidate aborts during startup. #26 sets n_rs_seq=3, making n_keep_tail=4, which fails GGML_ASSERT(n_ubatch > n_keep_tail). Could we validate the effective microbatch size and fall back to checkpoint/replay when it’s too small, or return a clear configuration error? A regression test for this boundary would help. |
|
Here the original fix (on Nathan's Vulkan branch) Nathanw1014@f25eefe |
|
Independent check of this PR on two backends, on the post-sync master (88c87dd), with #27 also applied. Model is the Huihui abliterated fine-tune of Qwen3.8-27B UD-Q4_K_XL (native MTP head in file). Workload is the issue #25 shape, thinking disabled, five rotated runs per arm, each server start verified for banner, model and n_rs_seq before use. Decode tok/s, mean +/- SD, n=5:
Mechanism on both backends, from the server logs: n_rs_seq 0 -> 4; recurrent buffer 149.62 -> 748.12 MiB, so the memory cost in the PR body matches what we see (+598.5 MiB on device); checkpoints per run 21 -> 0; checkpoint restores 14 -> 0. Prefill logits are byte-identical across arms on four fixed corpora, on both backends. One finding to weigh. With MTP active, the output of this PR's arm and master diverge at exactly token 56 of a fixed greedy-style run, with the same diverging token IDs on Vulkan and on HIP. #27 cannot cause this on HIP (it touches Vulkan dispatch only, and we verified the plain decode streams are identical there). So the divergence tracks the rollback path itself. This is the same property that Nathan's fork named when it chose full checkpoints for MTP instead: the RS-plane rollback "can diverge slightly after rejected drafts" (Nathanw1014/llama.cpp a17e843, later reapplied as f25eefe with a livelock fix). We did not classify the outputs against a ggml-org build, so we state it as an observation, not a defect verdict. Our read: this PR is fast and correct by upstream semantics, but it is not token-exact against master on rejected drafts. Nathan's full-checkpoint path is token-exact and slower. That trade is real and should be stated in the PR, and one of the two paths should be chosen on purpose. Assisted-by: Codex (runs and checks done locally on real hardware by the poster). -Astra |
3 participants
One of the two causes of #25. Written by an agent (Claude Fable 5.1); all numbers below were measured on a Strix Halo box in this session, against a baseline built from the merge-base (
c7af5c6).Problem
f169e0b ("server: keep speculative checkpoints on device", #18) removed
DRAFT_MTPfromcommon_params_speculative::need_n_rs_seq(). An MTP target context is therefore created withn_rs_seq = 0, and the server takes the checkpoint path on every partially accepted draft: restore the state, then decode the accepted tokens a second time as a replay batch. That is a second target forward pass on most steps.Verbose run, Qwen3.8-27B, 188 generated tokens:
Change
Put MTP back on the list, as upstream has it (
common/common.h, one line). The target then rolls the recurrent state back in place and the replay disappears. Cost: the recurrent state holdsn_maxextra slots, 150 MiB -> 750 MiB on this model withn_max 4.Measurements
Environment: Ryzen AI Max+ 395, Radeon 8060S (RADV, Mesa 26.1.7), Linux 7.0.0-30, Ubuntu 24.04. Model
unsloth/Qwen3.8-27B-GGUFUD-Q4_K_XL. Server flags as in the issue:-ngl 999 -c 8192 --flash-attn on --parallel 1 --spec-type draft-mtp --spec-draft-n-max 4. Workload: one chat completion, 95-token prompt, ~185 generated tokens, temperature 0.1, thinking off. Decode tok/s, 3 runs each, idle GPU otherwise.GGML_VK_MMV_NO_SPLIT=1The remaining gap between this PR and mainline is the mat-vec chunking, addressed separately. With both, decode matches mainline.
Not verified
--parallel > 1) not measured.🤖 Generated with Claude Code
https://claude.ai/code/session_01LTnaB4ucGc5vkzctYns4Nq