WIP: DSv4 + Qwen3.8 Flash runtime fixes (MTP spec decode, converters, checkpoint-on-device, GDN graph order) - #14
Closed
Nathanw1014 wants to merge 18 commits into
Closed
Conversation
is_dflash2 was armed from the dflash.selector_top_k metadata key alone. A draft GGUF carrying the key without a decode graph that builds the selector lattice made the host read hidden states as a lattice: every draft rejected (0/1556 accepted), 2.4x slower than no speculation, no diagnostic. Gate on llama_model_dflash2_top_k(), which reports the selector top-k only when the decode graph builds the lattice, warn and fall back to the DFlash draft path on mismatch, and refuse at load to attach selector tensors to the DSV4 backbone (graph_dsv4 has no build_post_sampling). Assisted-by: Claude Fable 5
Drop COMMON_SPECULATIVE_TYPE_DRAFT_MTP from the need_n_rs_seq() predicate so no recurrent snapshot rollback planes are allocated for embedded MTP, forcing full checkpoints while keeping snapshot support for Eagle3, DFlash, and DSpark. Partial recurrent snapshot restoration diverged after rejected drafts; full checkpoints restore token-exact Qwen3.8 MTP output. Ported from gaetan-puleo/llama-cpp-strix-halo commit f82e59263. Assisted-by: Claude Fable 5
Fixes run-to-run non-determinism observed on Qwen3.6 35B-A3B UD on ROCm (Strix Halo): two consecutive runs of the same prompt produced different outputs at long context, and the stock build itself was not bit-stable (stock vs stock KL divergence 0.00384 without the fix, 0 with it). The layer loop called ggml_build_forward_expand(gf, cur) on the attention-norm output before building the recurrent attention layer. build_layer_attn_linear() and build_recurrent_attn() add the GDN state updates (ggml_cpy into the convolution and SSM state caches) later, and those writes are sink nodes with no consumers in the compute DAG, so their scheduling position is fixed only by the graph expansion order. Closing the graph branch at the pre-layer norm left the state cache writes ordered against the wrong node set, which could schedule them independently of the following layers and race with the state reads of the next token or layer. Move the expansion to after the recurrent layer is fully built. The full-attention branch keeps its original expansion position, so only the recurrent path changes. Matches upstream PR ggml-org#22661 (models: fix linear attention state corruption in recurrent layers, introduced by commit 2446419), which reported intermittent generation corruption on Vulkan. On this ROCm machine the same ordering surfaced as non-determinism; after the fix, repeated runs are bit-identical, and both the isolated LDS decode step and the MMQ-pair prefill show zero KL divergence against the stock reference. No throughput regression (tg128 47.08/45.21/41.71 t/s at 0/10k/30k, prefill unchanged). Assisted-by: OpenCode (cherry picked from commit be236e33d1e50be5b599cdc64c8b00a163963acf)
Request device-backed storage when saving and restoring the rolling target and draft checkpoints used by speculative decoding. Leave prompt-cache checkpoints host-resident because they retain multiple historical states, while each speculative sequence owns one rolling checkpoint. The validated path preserved all 128 generated tokens and the 106/77 draft trajectory while improving MTP3 from 12.61253 to 12.68163 tokens per second. Ported from gaetan-puleo/llama-cpp-strix-halo commit c8b681b6f. Assisted-by: OpenCode
…store With full-checkpoint rollback, a partial draft acceptance restores the pre-round state and re-decodes the accepted tokens to rebuild it. The replay went through the same verification as a fresh draft. On backends where logits change with batch shape or memory layout (Vulkan), that re-verification can reject a token the original verification accepted; the rejection restores the same checkpoint and replays again, and the slot loops on one position without emitting anything. qwen35moe with --spec-type draft-mtp stalled this way a few hundred tokens into long generations (the v0.6.8 MTP hang): the loop repeated "accepted 2/3, restore at pos 995" every 27 ms with the GPU at 90 percent. Accept the replayed tokens without re-verifying and sample only the continuation from the final position. The replayed prefix was accepted by the verification that triggered the restore; the replay exists to rebuild state. On backends with batch-shape invariant logits the re-verification always agreed, so behavior there is unchanged (verified bit-identical on CPU with and without this change, 800-token greedy pair, 118 restore rounds). Assisted-by: Claude Fable 5
The no_vocab branch returned before the special token KVs were read, so a declared mask token id was lost and llama_vocab_mask() returned -1. A diffusion drafter converted without a tokenizer never tokenizes text, but it does need the mask id to build its denoise batch. Read the ids in the no_vocab branch and validate them against the dummy vocab size. Assisted-by: Claude Fable 5
The qwen4exp converter drops the MTP head, but community sidecar drafters now
exist (agentionai's Qwen3.8-Flash-Next-MTP-Q8_0: arch qwen4exp, block_count 49,
nextn_predict_layers 1, upstream tensor naming, hc_norm gammas verified
(1+w)-folded against the raw HF checkpoint by range read - exact +1.0 offset).
Follows the deepseek4 pattern: optional NEXTN_PREDICT_LAYERS KV with a
tensor-presence downgrade, mtp_only detection when the trunk is absent, trunk
tensors TENSOR_NOT_REQUIRED in that case, NextN tensors TENSOR_SKIP unless the
context requests them.
qwen4exp-specific differences from deepseek4, both dictated by the file format:
- the NextN block is always a full-attention QSA layer (gated attention, own
nh=4 indexer, full 512-expert MoE, both HC pairs); is_recr() is derived from
full_attention_interval and would misclassify blk.48 as linear attention, so
the MTP layer forces the full-attention branch.
- nextn.hnorm is hc-space ({hc_dim}, 10240), not {n_embd}: the draft head
consumes the target's 4-stream hyper-connection state.
A standalone load of a sidecar now aborts with a clear message instead of
segfaulting in the first hc_mix (the trunk tensors are null by design; the
mainline graph cannot be built from a drafter file).
Draft graph and --spec-type draft-mtp wiring are the follow-up commit; this one
is loader-only and mainline GGUFs are unaffected (KV absent -> n_layer_nextn 0,
identical load path).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 23d8b2c)
…o end Second half of the MTP wiring (loader was the previous commit). Verified end to end on the 64 GB box: Q4_K_M target + agentionai's community MTP sidecar, --spec-type draft-mtp, coherent temp-0 generation through the Vulkan lane. - graph_mtp: [enorm(embd(tok)) repeated across streams ; grouped-RMS hnorm(h)] -> eh_proj per stream -> one HC-wrapped full-attention block -> the shared head mixer (which doubles as the output norm). h arrives hc-space: the mainline graph exports the 4-stream residual under cparams.embeddings_nextn. - the MTP context holds a PLAIN attention cache over the nextn layer(s), dense, no recurrent state - the deepseek32 pattern. Discovered the hard way: a hybrid memory with an empty recurrent layer set fails its buffer allocation. - exports hand the scheduler REAL nodes: a naked reshape view gets no backend assignment and the h_nextn extraction asserts. - the draft runs dense attention (QSA gate is null-safe now); the reference precedent is deepseek32, whose MTP head also runs dense. Correctness only at this point: no acceptance-rate or throughput claims - the smoke ran under heavy memory pressure and the interactive-trap log spin. Measured cleanly it may or may not pay; that is the next session's A/B. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit b5b7c8c)
Upstream's converter sets no_mtp unconditionally (HF and vLLM both drop the block), so a qwen4exp drafter could only come from a third party. Opt into the shared _QwenMtpMixin machinery so --mtp writes a sidecar and --no-mtp keeps the target clean, and add the three NEXTN tensors to the arch's tensor list. Two things the mixin does not cover: - this arch splits the eh projection into fc_embedding + fc_hidden where other Qwen MTP heads carry a fused fc. The draft graph builds ggml_concat(enorm(embedding), hnorm(hidden), 0), so eh_proj is cat on the input axis with embedding first. Verified byte-exact against the reference checkpoint rather than inferred: eh_proj[:2560] == fc_embedding, eh_proj[2560:] == fc_hidden. - qwen4exp has no output_norm - the hyper-connection head mixer is the final norm - and the MTP block carries its own under mtp.hyper_connection_mixer, outside the mixin's mtp.layers.N rewrite. Map it onto output_hc_*, folding hc_norm to (1 + w) as the inherited norm.weight rule would. The PLE key group is skipped for a sidecar: the draft block has no PLE layer and its hash constants are filtered out with the rest of the target. Dry run: 34 tensors, 7.8G, names and shapes matching a known-good sidecar. Assisted-by: Claude (Opus 5)
llama-quantize rejected our own MTP sidecar with key qwen4exp.attention.compress_ratios has wrong array length; expected 49, got 48 The array was built from num_hidden_layers, but the MTP block makes the model 49 blocks and the loader wants one entry per block including nextn layers. Worth recording how this was found: the sidecar was already 34/34 tensors byte-identical to a known-good reference. A tensor-level comparison cannot see a short metadata array; the quantizer caught it immediately. Load-bearing metadata deserves the same verification as weights. The MTP block gets 0 rather than the compression stride, matching the reference: it carries indexer weights, but the draft graph gates QSA off and runs dense (deepseek32's MTP precedent), so it has no stride of its own. Assisted-by: Claude (Opus 5)
Port of the fp8 scale gate onto the LazyChunkedTensor shard loader. The raw lazy re-read bypasses base.py's f32 upcast, so shards arrive as e4m3 and a plain cast writes the raw code values. Gate on the checkpoint scale, never on dtype. Assisted-by: Claude Fable 5
Port of the host-gather half of the old decode-graph commit onto the hoisted build_inp_ple: the in-graph get_rows on the CPU-resident table split the graph and cost a GPU->CPU->GPU round trip per layer group. The graph-reuse half is upstream already. LLAMA_PLE_HOST_GATHER=0 restores the in-graph gather. Assisted-by: Claude Fable 5
The half of the fork's lazy-read work upstream did not take: --lazy-mode applies MADV_RANDOM and drops the eager pull-in, but replaces the kernel readahead with nothing. Re-add prefetch_rows (posix_madvise WILLNEED / PrefetchVirtualMemory in page runs), gather_tables() nomination with start-offset matching, and the decline guard that strips the advice from any range no arch prefetches - the advice alone measures slower than off. qwen4exp nominates its PLE table and queues the faults in set_input. Assisted-by: Claude Fable 5
Port of danielhanchen's open PR, applied as-is (byte-clean on this base): - the update context never built ctx_idx, so a cross-stream seq_cp never copied the indexer key buffers and the destination kept stale keys (reachable through the OpenAI n parameter, silent wrong output); the indexer hparams also pin ROPE_TYPE_NONE so the now-live update path cannot K-shift keys that are stored unrotated - QSA blocks keyed on the cells' seq sets, not position alone (kv-unified) - M-RoPE rank ordering so image cells stop collapsing into one block slot - load_arch_hparams metadata asserts become throws - norm-only reshape dodges the CUDA gridDim.y 65535 cap at n_kv 262144 PPL verified bit-identical by the author on UD-IQ1_S; runtime-only, no GGUF rebuild. Assisted-by: Claude Fable 5
…ggml-org#27977) Port of ServeurpersoCom's open PR, applied as-is with one hardening: - for_each_token_in stops once every present sequence has been seen - get_prev_tokens scans only the n-gram window; the from-zero below[] fill is lazy and only runs on the position-gap fallback - QSA generation gathers the selected cells into a per-query window instead of masking the whole cache into flash attention; prefill keeps the scan path (Vulkan mask-opt already prunes it block-wise) - the indexer four-head sum is three strided-view adds, dropping the transpose that copied the score surface twice - the used-cell set is a bitmap walked 64 bits at a time Fork delta vs the PR: the gather's kq_mask stride requirement is part of the dispatch condition, falling back to the scan path instead of aborting on a padded mask. Assisted-by: Claude Fable 5
The generation-path checkpoint sites went on-device in v0.7.0; the prompt-phase create_checkpoint stayed host-resident, and with a draft model loaded it serialises the draft's whole KV cache through the one-blocking-copy-per-row host path. On UMA that collapses prefill 10x+ once the prompt spans more than one batch (toolbox issue #12). GTT is the same physical RAM on this platform, so the device-residency cost argument does not apply. Assisted-by: Claude Fable 5
Restores the stochastic verifier from pre-rebase 015f09c on top of upstream DFlash2 (ggml-org#27342, ggml-org#27816), which reverted draft sampling. The surviving delta: - common/speculative.h: common_speculative_token_dist plus dists / temperature / seed draft params - common/sampling: sample_and_accept_n overload doing maximal-coupling accept with residual sampling on rejection, using a dedicated verifier RNG (chain seed ^ 0x9e3779b9) threaded through init/reset/clone/copy - common/speculative.cpp: re-add the dist capture to the DFlash2 selector walk; temperature > 0 samples the predecessor from the softmaxed selector scores and records the distribution, temperature <= 0 keeps upstream's greedy path with its p_min break - server, speculative-simple: use the dist verifier when the context can roll back partial acceptance and sampling is stochastic Full Vulkan build passes (build-vk, glslc from ~/shaderc). Assisted-by: Claude Fable 5
The dispatcher clamps the draft to dp.n_max after the impl runs, which left dp.dists longer than the draft on the final block of a generation and the server's size guard fell back to the greedy verifier there. Keep the dists aligned so the truncated block is dist-verified too. Found in a DFlash2 smoke test at temp 0.7 (Qwen3.8-27B UD-Q4_K_XL + DFlash2 Q8_0 drafter): 41/42 verifies took the dist arm, the clamped final block read dists=7 draft=5 and took the plain arm; with this fix it reads dists=5 draft=5 and takes the dist arm. Assisted-by: Claude Fable 5
Author
|
Validation results for this branch (Strix Halo, gfx1151/RADV, build a6e38dd lineage):
DSv4 end-to-end still needs a 128GB tester; the DSv4 kernels themselves are covered in the backend suite above. |
Author
|
Closing along with #13. The Vulkan half is now in #17. The Qwen3.8/DSv4 runtime fixes from this branch (MTP spec decode, converters, checkpoint-on-device, GDN graph order) are held back deliberately: 7 of them are written against upstream changes this fork does not carry yet (ggml-org#27880 build_inp_ple hoist, ggml-org#27969 lazy-mode rename, the ggml-org#27941/ggml-org#27977 ports), so they will be re-proposed after the upstream sync (#9) lands. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Additive on top of #13 (base is set to its branch so the diff shows only these 18 commits). Focused on the two priority models: DeepSeek V4 and Qwen3.8 Flash.
Qwen3.8-Flash (qwen4exp)
MTP / speculative serving
Model correctness
Validation (Strix Halo, gfx1151/RADV)
AI usage disclosure: YES. Claude assisted with the rebase decomposition, cherry-picks, test execution, and this description. Nathan Wilson owns and maintains the changes.
🤖 Generated with Claude Code