qwen4exp: support recurrent state rollback - #28123
Merged
ServeurpersoCom merged 1 commit intoSep 1, 2026
Merged
Conversation
MTP speculative decoding needs the target state to move back by the number of rejected draft tokens. Without rollback support the context is classified as SEQ_RM_TYPE_FULL and the server serializes the whole recurrent state to host memory on every round, which costs more than the drafting saves. The recurrent cache already holds n_rs_seq + 1 snapshot planes and the delta net writes its SSM state into them, but build_conv_state_at wrote a single plane, so a rollback restored a convolution history that was never captured. It now writes one snapshot per slot, each ending one token earlier, for the delta net QKV convolution and for the PLE convolution alike. Measured on Qwen3.8-Flash-Next UD-Q4_K_XL with the standalone MTP draft, n-max 3 and a single slot: decoding reaches 183 tok/s on code and 144 tok/s on prose. The same branch before this change, where the server falls back to checkpointing the state to host memory, reaches 123 and 83 tok/s, for 108 tok/s without a draft.
CISC
approved these changes
Aug 31, 2026
am17an
approved these changes
Sep 1, 2026
This was referenced Sep 1, 2026
MarkShark2
added a commit
to MarkShark2/llama.cpp
that referenced
this pull request
Sep 2, 2026
68 upstream commits, 9 conflicted files. Brings qwen4exp fixes needed for Qwen3.8-Flash-Next: recurrent state rollback (ggml-org#28123), seq_cp/block position keying/mtmd input (ggml-org#27941), indexer head slicing (ggml-org#28023), and the central n_layer_nextn load (ggml-org#28159). Resolutions: - ggml-rpc.cpp: resolved entirely to the fork side again. Upstream's rpc_dispatcher/proto-6.0 port stays deferred; the fork keeps 5.1.3. Note that upstream ggml-org#26500 now implements the same foreign-endpoint skip in serialize_graph that the fork carries, so that one is a candidate to drop when the dispatcher port finally happens. - llama-graph.cpp: adopted upstream's fused ggml_swiglu_clamp for both the dense and MoE clamp paths and kept the fork's LLM_ARCH_GLM5NEXT in the condition. The fused kernel is min(gate,limit) / clamp(up,+-limit) / swiglu_split, i.e. bit-identical to the three-op sequence it replaces, and Vulkan implements it, so the boards get it too. Costs the ffn_*_clamped cb() trace names, same tradeoff upstream took for DSV4. - speculative.cpp: upstream ggml-org#27310 folded the DFlash encoder into the injection decode and deleted features_buf. The fork's non-finite feature scrub moves onto batch_inject.embd in place; the last_tap_nextn tap automerged. - dflash.cpp: kept the fork's Laguna kv_inp norm and deferred gated o_proj, layered on upstream's newly-populated wk_s/wv_s/wo_s scale arguments (previously NULL). The gated path latches wo_s alongside wo. - nemotron-h.cpp: dropped the now-duplicate NEXTN_PREDICT_LAYERS get_key, kept the fork's stricter assert and n_layer_nextn_per_head derivation. - llama-model-loader.cpp: kept the fork's rpc_preloaded skip ahead of upstream's use_mmap || lazy.has(cur) predicate. - tests: kept both sets of flash-attn cases; ported the fork's GLM5NEXT test_dsa_kpool/test_mtp onto upstream's int-verbosity logging convention. Windows CUDA build green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U3H5motr51eTWujztSXykc
danielhanchen
added a commit
to unslothai/llama.cpp
that referenced
this pull request
Sep 2, 2026
TheTom
pushed a commit
to TheTom/llama-cpp-turboquant
that referenced
this pull request
Sep 3, 2026
Ports every upstream qwen4exp (Qwen3.8-Flash-Next) commit from the past 10 days that this fork's manual PR port had not received: - reduce graph splits by hoisting the PLE embedding gather out of the per-layer loop (ggml-org#27880) - sum indexer heads via strided adds instead of transpose+sum_rows (ggml-org#28023) - support recurrent state rollback for MTP speculative decoding (ggml-org#28123) - rewrite QSA sparse-attention block/bias selection: fixes NaN-producing bias rows for short sequences, fixes cross-sequence block pooling in a unified KV cache, adds mrope duplicate-position ranking, and fixes a CUDA rms_norm gridDim.y overflow (ggml-org#27941) - indexer cache seq_cp staleness fix, ext.x/ext.y state-restore fix, PLE-must-be-linear-attention validation, correct -sm tensor disablement (ggml-org#27941) - Hadamard k_rot context-shift crash fix, shared with other archs (ggml-org#27967) Also replaces raw GGML_ASSERT aborts in hparams loading with proper error messages, and adds test coverage: a PLE fixture in test-llama-archs (which required porting the per_layer_token_embd row-count-from-metadata fix to make it loadable) and a state round-trip test in test-save-load-state. Verified against the real Qwen3.8-Flash-Next model: correct generation at short and long (~66k token) context, and test-llama-archs passes qwen4exp on both CUDA and CPU.
Randozart
added a commit
to Randozart/llama.cpp
that referenced
this pull request
Sep 4, 2026
KV restore batching (ggml-org#27991), kv-cells seq-scan early stop (ggml-org#28011), MOE fusion to specdec + multi-token (ggml-org#27621), mm_ids_helper templated fast path (ggml-org#27978), qwen4exp recurrent state rollback (ggml-org#28123), n_layer_nextn load order (ggml-org#28159), FA K/V XOR-swizzle smem tiles (ggml-org#25635), --lazy-mode -lzm (ggml-org#27837/ggml-org#27969). TQ3/TurboQuant stack and vitriol-* integration auto-merged clean; no conflicts. Experiment E1 of mining-experiment-master-plan-2026-09-01.
TrevorS
added a commit
to TrevorS/llama.cpp
that referenced
this pull request
Sep 4, 2026
…ationale Three fork commits are now upstream's and were dropped at the rebase onto 3466812: the rollback allowlist entry and the per-token conv snapshots (ggml-org#28123) and the windowed n-gram scan (ggml-org#28040). The env toggles that went with them are gone too. The entry for the TENSOR_READ_LAZY revert claimed the GPU read the mmap'd PLE pages directly. It does not: integrated is hardcoded false on CUDA, GET_ROWS is never offloaded, and the gather runs on one CPU thread demand-faulting the mapping. The measurement stands; the explanation did not.
turbo-tan
pushed a commit
to turbo-tan/llama.cpp-tq3
that referenced
this pull request
Sep 4, 2026
…-org#28123) build_conv_state_at wrote a single conv-state snapshot plane, but seq-rollback restores state at multiple positions — a rollback restored a convolution history that was never captured. Now writes one snapshot per slot, each ending one token earlier, for both the delta-net QKV convolution and the PLE convolution. Adds QWEN4EXP to the rs_rollback arch list. Upstream 0eadefe, clean 3-way apply. Candidate fix for the sticky 'CUDA error: unspecified launch failure' seen after slot switches with partial KV reuse (rollback path) — crashed on both MTP and non-MTP serving with ggml-org#27941 already applied.
fewtarius
pushed a commit
to fewtarius/CachyLLama
that referenced
this pull request
Sep 5, 2026
MTP speculative decoding needs the target state to move back by the number of rejected draft tokens. Without rollback support the context is classified as SEQ_RM_TYPE_FULL and the server serializes the whole recurrent state to host memory on every round, which costs more than the drafting saves. The recurrent cache already holds n_rs_seq + 1 snapshot planes and the delta net writes its SSM state into them, but build_conv_state_at wrote a single plane, so a rollback restored a convolution history that was never captured. It now writes one snapshot per slot, each ending one token earlier, for the delta net QKV convolution and for the PLE convolution alike. Measured on Qwen3.8-Flash-Next UD-Q4_K_XL with the standalone MTP draft, n-max 3 and a single slot: decoding reaches 183 tok/s on code and 144 tok/s on prose. The same branch before this change, where the server falls back to checkpointing the state to host memory, reaches 123 and 83 tok/s, for 108 tok/s without a draft.
OllyJohnston
added a commit
to OllyJohnston/llama.cpp
that referenced
this pull request
Sep 5, 2026
MTP speculative decoding needs the target state to move back by the number of rejected draft tokens. Without rollback support the context is classified as SEQ_RM_TYPE_FULL and the server serializes the whole recurrent state to host memory on every round, which costs more than the drafting saves. The recurrent cache already holds n_rs_seq + 1 snapshot planes and the delta net writes its SSM state into them, but build_conv_state_at wrote a single plane, so a rollback restored a convolution history that was never captured. It now writes one snapshot per slot, each ending one token earlier, for the delta net QKV convolution and for the PLE convolution alike. Measured on Qwen3.8-Flash-Next UD-Q4_K_XL with the standalone MTP draft, n-max 3 and a single slot: decoding reaches 183 tok/s on code and 144 tok/s on prose, versus 123/83 without this change and 108 without a draft.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Qwen3.8-Flash-Next currently cannot roll back its recurrent state, so with an MTP draft the server falls back to serializing the whole state to host memory on every speculative round, which costs more than the drafting saves.
Adding the arch to llm_arch_supports_rs_rollback() is not enough on its own: qwen4exp overrides the shared convolution state write with its own build_conv_state_at(), which only ever wrote the current plane, so a rollback would restore a correct SSM state next to a convolution history that was never captured. It now writes one snapshot per rollback slot, like llm_build_delta_net_base::build_conv_state already does, covering both the delta net QKV convolution and the PLE one.
Tested on an RTX PRO 6000 with the MTP head and the draft patch from https://huggingface.co/dzannotti/Qwen3.8-Flash-Next-MTP-GGUF, n-max 3, one slot:
no draft: 108 tok/s
before: 123 tok/s code, 83 tok/s prose
after: 183 tok/s code, 144 tok/s prose
Note the 83: before this change MTP was slower than not drafting at all.
Additional information
Alternative to #28118
Requirements