spec: mtmd image chunks leave positional holes in the draft KV cache -> llama_decode(ctx_dft) rc=-1 -> HTTP 500 with draft-dflash (CUDA); zero-fill patch crash-eliminating but no speculative benefit on images

[indubwestep]

Summary

With --spec-type draft-dflash and an mmproj model, every image request stalls (~500 s) and then returns HTTP 500. Text-only requests with the same configuration work and speculate correctly. Root cause: a positional hole in the draft KV cache — mtmd image chunks advance the target context without being mirrored into the 1D draft KV cache, so llama_decode(ctx_dft) rejects the non-consecutive positions.

A community fix (z-lab#1, +74/−5 in common/speculative.cpp, zero-fills the hole) is crash-eliminating on CUDA in our testing (7/7 image regression + stability PASS, HTTP 500 gone) but provides no speculative benefit on image requests (draft/accept counters stay 0 for images; tolerated draft: llama_decode returned -1 is still logged), and a general position-gap case (non-consecutive positions across requests) can still terminate the server. Nothing here is claimed as fixed upstream.

Environment

• llama.cpp at PR spec : add DFlash2 support (local convolution + candidate selector) #27342 head: 5ecbe1ac17ec0484c5b44af0bd580cdc9c428ed4 (local build, MSVC 19.44, CUDA 12.4, sm_86)
• OS: Windows 11 x86_64
• GPU: NVIDIA RTX 3090 Ti 24 GB
• Target: empero-ai/Qwen3.8-27B-Ridge-GGUF Qwen3.8-27B-Ridge-3.7bpw.gguf (IQ2_M) + mmproj-Qwen3.8-27B-BF16.gguf
• Draft: incoai/Qwen3.8-27B-DFlash2-GGUF Qwen3.8-27B-DFlash2-Q4_K_M.gguf (1,143,006,752 bytes; SHA-256 18a380efc9b7ed8d88677fc895f5c11ae170653434ee378f7348f715c14d0594)

Failure log (image request, draft-dflash)

init: the tokens of sequence 0 in the input batch have inconsistent sequence positions:
 - the last position stored in ... KV cache ... is X
 - the tokens for sequence 0 ... have a starting position of Y
 it is required that the sequence positions remain consecutive: Y = X + 1
process: llama_decode(ctx_dft) failed rc=-1 (n_tokens=..., offset=...)
srv  update_slots: decode() failed: failed to process speculative batch   -> HTTP 500

Repro (llama-server)

llama-server --model <target> --mmproj <mmproj> --ctx-size 8192 --n-gpu-layers 999 \
  --flash-attn on --cache-type-k q4_0 --cache-type-v q4_0 --batch-size 256 \
  --ubatch-size 128 --threads 8 --parallel 1 --temp 0 --seed 42 \
  --spec-draft-model <draft> --spec-type draft-dflash \
  --spec-draft-ngl 999 --spec-draft-n-max 7 --spec-draft-p-min 0

Send any image in a chat-completions request → observe the ~500 s stall then HTTP 500. The same request with --spec-type none (or without mmproj) succeeds.

Root cause

• mtmd image chunks are decoded directly into the target context by the mtmd helper and never pass common_speculative_process() (the PR spec : add DFlash2 support (local convolution + candidate selector) #27342 DFlash path) — the draft KV cache therefore has a positional hole (M-RoPE targets use non-linear image positions; even a single-ubatch chunk injects draft rows at bogus positions, per the z-lab fix description).
• The 1D draft KV cache requires consecutive positions; the hole fails the consecutive check in llama_decode(ctx_dft) (rc=-1) → HTTP 500.

Upstream corroboration

• Eval bug: Failed to read image when using DFlash (Qwen3.6 27B) #26108 — Qwen3.6-27B + DFlash + image (CUDA): same llama_decode(ctx_dft) failed rc=-1, same position-mismatch log; works without --spec-type draft-dflash.
• PR spec : add DFlash2 support (local convolution + candidate selector) #27342 comment thread — Qwen3.8-27B-UD + DFlash2 Q4_K_M (sm_70): vision requests fail with draft-dflash; M-RoPE positions rejected by the draft context.
• dflash: zero-fill draft-cache holes left by mtmd chunks and reused prefixes z-lab/llama.cpp-fork#1 — candidate fix, Metal-validated (200 images, greedy byte-identical, acceptance recovers).

Evaluation of the z-lab fix on CUDA (this report)

• Patch: dflash: zero-fill draft-cache holes left by mtmd chunks and reused prefixes z-lab/llama.cpp-fork#1 (+74/−5, single file common/speculative.cpp); skips embedding batches in the DFlash process() and zero-fills the draft-cache positional hole (log: draft cache hole for seq 0: [46, 62) - seeding with zero features).
• Result: HTTP 500 gone. 7/7 image regression PASS (full-length generations, stable VRAM), stability addendum PASS (3x repeatability, text-after-image, restart resilience). Crash-eliminating on CUDA.
• Residual on CUDA: image requests still log 14+ tolerated draft: llama_decode returned -1 position-mismatch failures per request and run without speculative benefit — draft/accept counters stay 0 across all image tests, so images decode at full target speed (no vision speedup). Pure-text requests with the same binary speculate at ~100% acceptance.
• Remaining blocker: cross-request position gaps (non-consecutive positions spanning slots/requests) still terminate the server (exit -1). The z-lab patch covers the single mtmd hole only, not general position-gap handling.

Ask

1. Is there a path to a general fix (draft-KV position remapping / gap support) so DFlash2 can also speculate on image requests?
2. Maintainer guidance on where gap-tolerant draft-KV handling belongs (draft-KV refactor vs process() special-casing)?

Cross-links: #26108 (same failure, Qwen3.6-27B), PR #27342 (DFlash2; comment thread confirms M-RoPE rejection), z-lab#1 (zero-fill fix, Metal-validated), companion issue CUDA batched-verification parity.

[NikoCloud]

Third platform confirmation: this reproduces on Vulkan / AMD RDNA4 (gfx1201), so the bug is not CUDA- or Metal-specific.

Environment

• llama.cpp PR spec : add DFlash2 support (local convolution + candidate selector) #27342 head 5ecbe1ac17ec0484c5b44af0bd580cdc9c428ed4, built -DGGML_VULKAN=ON -DCMAKE_BUILD_TYPE=Release
• GPU: AMD Radeon AI PRO R9700 32 GB (gfx1201, RDNA4), Vulkan backend
• OS: CachyOS x86_64
• Target: mradermacher/Qwen3.8-27B-heretic-trohrbaugh-i1-GGUF Q4_K_S + unsloth/Qwen3.8-27B-Instruct-GGUF mmproj-BF16.gguf
• Draft: incoai/Qwen3.8-27B-DFlash2-GGUF Q8_0 (Q4_K_M behaves the same)
• Flags: --spec-type draft-dflash -md <draft> -devd Vulkan0 --spec-draft-n-max 4 -c 262144 -ctk q8_0 -ctv q8_0 -fa on --ctx-checkpoints 2 -np 1

Symptom

Text-only requests work and speculate normally (87.8 tok/s, mean accepted length 2.45). Any request containing an image fails:

W find_slot: non-consecutive token position 69 after 68 for sequence 0 with 512 new tokens
E decode: failed to initialize batch
E llama_decode: failed to decode, ret = -1
E process: llama_decode(ctx_dft) failed rc=-1 (n_tokens=512, offset=512)
E post-decode callback failed
E slot   operator(): id  0 | task 366 | failed to decode mtmd chunk, idx = 69, res = 1
E slot   operator(): id  0 | task 366 | failed to process mtmd chunk, res = -1
E srv    send_error: task id = 366, error: failed to process mtmd chunk

One difference from the original report: here it fails immediately (~3 s) and returns an error to the client, rather than stalling ~500 s before HTTP 500. Same positional-hole root cause, faster failure path.

The z-lab patch also fixes it on Vulkan

z-lab/llama.cpp-fork#1 applies cleanly to 5ecbe1a (+74/−5, common/speculative.cpp) and resolves it here. After rebuilding, image requests succeed and are answered correctly:

test	ground truth	output	tok/s
split image	left red, right blue	"Red Blue"	34.8
two shapes	green circle + yellow square top-left	"A yellow square in the top-left and a green circle in the center."	35.0

This also corroborates the "no speculative benefit on images" finding: draft acceptance and mean len are bit-identical before and after both image requests (0.36342 / 2.45), i.e. the drafter never engages on multimodal input. Image throughput lands at ~35 tok/s against a 37.4 tok/s no-speculation baseline for this target, so skipping speculation on images costs almost nothing in practice.

Text-only speculation is unaffected by the patch (87.8 tok/s, unchanged).

I have not exercised the general non-consecutive-position case mentioned in the original report, so I can't speak to whether that still terminates the server.

[fishlikeX]

We hit the exact same failure and traced it to the root cause — sharing the analysis and a complete fix, validated end-to-end on our Volta fork. Posting here since this answers both questions raised in the issue.

Root cause (measured, not inferred)

We instrumented the dflash process() in common/speculative.cpp and captured the actual position arrays of an mtmd image batch (752 embedding rows, Qwen3.8-27B + mmproj):

MTMD-DBG: n_tokens=752 pos[0]=53 pos[1]=53 ... pos[511]=53 pos[512]=53 pos[n-1]=53

Every image row arrives with the same position (the image's sequence position — the spatial structure lives in the target's M-RoPE machinery). And the text that follows does not continue at image_pos + n_rows either — in our capture the 752 rows span only 47 positions on the target's 1-D scale, so the text resumes at 53 + 47 = 100.

The draft's 1-D KV cache cannot store either shape:

• injecting image rows at their batch position lets ubatch chunk 1 pass (pos[0] == pos_max + 1 by coincidence) and chunk 2 fails the continuity check → llama_decode(ctx_dft) rc=-1 → HTTP 500;
• even a single-chunk image breaks the token-count arithmetic: those 752 rows advanced the 1-D position scale by only 47, so dp.n_past (token count) permanently diverges from the draft cache's position scale after any image — and the noise block, based on dp.n_past, lands in a hole.

Fix (three surgical changes in common/speculative.cpp, dflash impl)

1. process() skips embedding batches entirely. The hole they leave is zero-filled with zero-feature encoder rows when the next token batch arrives ([pos_max+1, first_pos) through the same encode+inject path).
2. draft() places the noise block at the draft cache's own pos_max + 1 instead of dp.n_past. The server's post-acceptance seq_rm(pos_next, -1) rolls the draft cache back to exactly the accepted context every round (pos_next is position-scale), so pos_max + 1 is always the correct base.
3. begin()'s pos_max < N-1 warning becomes a debug heuristic (token and position scales legitimately diverge after images).

Why speculative decoding keeps working on image requests

Draft tokens are always validated by the target, so zero-filled rows only dip the acceptance rate across the image span — the output distribution stays exact. Measured on V100 (CUDA), Qwen3.8-27B Q4_K_XL + DFlash2 draft (Q4_K_M) + mmproj:

• image requests: HTTP 200, correct visual content (a random-noise-grid test image is described as a "dense grid of small, uniformly sized squares"),
• draft acceptance 0.57 during image spans, recovering to 0.88 on text (the zero-fill patch described in the issue loses drafting on image requests — the difference is the noise-block base fix below: without it every draft round still fails and silently falls back to plain decoding),
• zero llama_decode failures, text-only throughput and acceptance unchanged.

The zero-fill also generalizes beyond mtmd: positional holes from cache-reused prefixes (previously a hard process() failure) are covered by the same mechanism.

On the two questions raised

1. General fix? — yes: keep the draft cache in the target's position scale and fill any hole with zero-feature rows at injection time. No KV remapping needed; positions never diverge from what the server's rollback machinery expects.
2. Where should it live? — the process() special-case above is minimal and sufficient (validated end-to-end). A draft-KV position remap would be a much larger change for no additional benefit on this failure mode.

Full repro + vision-check scripts and the regression harness are in our fork (where this fix is merged and validated end-to-end): https://github.com/fishlikeX/sm70-attn (verify/mtmd_repro.sh, verify/mtmd_vision_check.py).

Happy to open a PR — the fix applies cleanly both on master (DFlash v1 path has the same injection code) and rebased onto #27342; maintainer's choice which base.

[HermiG]

Full repro + vision-check scripts and the regression harness are in our fork (where this fix is merged and validated end-to-end): https://github.com/fishlikeX/sm70-attn (verify/mtmd_repro.sh, verify/mtmd_vision_check.py).

I can confirm your patch fixes the bug I encountered (0.0 dflash2 draft acceptance when the context contains an image).