bug: Qwen3.8 DFlash/MTP speculative emits OOB token id == n_vocab (248320) on Vulkan

[maciejdzialo]

Summary

On AMD Strix Point (gfx1150) Vulkan with llama.cpp b10724, Qwen3.8-27B + z-lab DFlash2 (or ggml MTP draft) fails decode with:

init: invalid token[1] = 248320
decode: failed to initialize batch

248320 equals tokenizer.ggml.tokens array length — valid token ids are 0..248319. This looks like a classic off-by-one in draft sampling / logits indexing.

Environment

• GPU: AMD Strix Point gfx1150, Vulkan/RADV
• Runtime: llama.cpp b10724 (also reproduced earlier on b10666)
• Target: Qwen3.8-27B-UD-Q4_K_XL.gguf (architecture qwen35)
• Draft A: z-lab/Qwen3.8-27B-DFlash2-GGUF → Qwen3.8-27B-DFlash2-Q4_K_M.gguf (dflash)
• Draft B: Unsloth mtp-Qwen3.8-27B-Q4_0.gguf (qwen35) — same 248320 with ggml MTP
• Flags: --spec-type draft-dflash --spec-draft-n-max 2 (or draft-mtp)

Note: Muse-Glimmer + DFlash on the same binary/hardware works (different vocab/arch ~200k). Vanilla Qwen3.8 without speculation is fine (~4.3 t/s).

GGUF metadata (streaming count of tokenizer.ggml.tokens)

Field	Value
n_vocab (tokenizer.ggml.tokens len)	248320
bos_token_id	248044
eos_token_id	248046

Same n_vocab / bos / eos on target, DFlash draft, and MTP draft.

Expected

Draft sampling never emits token id >= n_vocab.

Actual

First decode step emits token 248320 (exactly n_vocab). Server returns HTTP 500 / "Invalid input batch."

Suggested fix

1. Clamp draft token ids to [0, n_vocab - 1] after sampling in the DFlash/MTP verify path.
2. Audit qwen35 / dflash draft-head logits indexing for < n_vocab vs <= last index (or argmax over n_vocab+1 slots).

Repro

llama-server \
  --model Qwen3.8-27B-UD-Q4_K_XL.gguf \
  --model-draft Qwen3.8-27B-DFlash2-Q4_K_M.gguf \
  --spec-type draft-dflash --spec-draft-n-max 2 \
  --device Vulkan0 -ngl 999 --ctx-size 8192 \
  --cache-type-k f16 --cache-type-v f16 --flash-attn on \
  --host 127.0.0.1 --port 8080 --no-webui
# POST /v1/chat/completions → 500; log shows invalid token 248320

Related

• vulkan: fix missing view-alias dependencies in ggml_vk_graph_optimize #27812 fixed Vulkan graph-optimizer wrong tokens for speculative acceptance (merged); this is a separate hard OOB failure that remains on tip after that fix.
• Unsloth Qwen3.8 MTP on the same stack loads without 248320 in one layout but reports accept rate 0.0 (possible related draft-head issue; see also qwen4exp: support draft-head-only GGUFs (unsloth layout) + fix draft-load regression #28097).

Happy to provide full server logs / GGUF SHA if useful.

[0cc4m]

I can't reproduce this. Does it always happen to you or just sporadically?

[maciejdzialo]

Always, not sporadic. Same GGUFs, same llama-server build, same prompt — first decode step every time on my side.

Setup: AMD gfx1150, Vulkan/RADV, Linux, commit 2d8d612. Target is Unsloth Qwen3.8-27B-UD-Q4_K_XL.gguf, draft is z-lab Qwen3.8-27B-DFlash2-Q4_K_M.gguf (dflash.selector_top_k = 16, so DFlash2). Without --spec-type the same target runs fine on Vulkan.

With --spec-type draft-dflash I get:

init: invalid token[1] = 248320
decode: failed to initialize batch

248320 is exactly n_vocab (ids are 0..248319). Not the DFlash mask token — that's 248070 on my draft file. token[1] is the first draft token going into the target verify batch.

I also hit this with embedded draft-mtp on the same Qwen3.8 target (invalid token[2] = 248320). /v1/completions fails the same way, so I don't think it's chat-template related.

Quick backend check on the DFlash setup: Vulkan crashes, CPU (--device none -ngl 0, same for draft) completes and actually generates drafts. Same binary, same models. I haven't tried CUDA.

On Vulkan, when I added logging locally, the DFlash2 selector row had all 16 top-k slots at 248320 with NaN scores — might be a bad GPU lattice readback rather than one wrong sampled token, but I haven't root-caused it.

Repro:

llama-server \
  --model Qwen3.8-27B-UD-Q4_K_XL.gguf \
  --model-draft Qwen3.8-27B-DFlash2-Q4_K_M.gguf \
  --spec-type draft-dflash --spec-draft-n-max 2 \
  --device Vulkan0 -ngl 999 --ctx-size 8192 \
  --cache-type-k f16 --cache-type-v f16 --flash-attn on \
  --host 127.0.0.1 --port 8080 --no-webui

curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"local","messages":[{"role":"user","content":"Say hi."}],"max_tokens":8}'

CPU control: add --device none -ngl 0 --spec-draft-device none --spec-draft-ngl 0.

SHA256:

• target: 1373078c80a25e1b6ccd32e69fcf087b3eeedf503f3463b089d540afa89a0814
• draft: 1a25c56858e1ebe93f2718ac1d49d1151f9323325c1bbfd6209370f4db131ebd

[tblisss]

I'm seeing something similar, with what appears to be the same issue, with init emitting an invalid 248320 token through Ollama 0.33.1, using its bundled llama-server. My issue is sporadic. Also using --spec-draft-backend-sampling.

Environment:

llama.cpp / llama-server build: d222767 (Ollama 0.33.1 bundled runner, build 1)
Backend: Vulkan/RADV
GPU: AMD Radeon Graphics, GFX1201, 32 GiB VRAM
OS inside Ollama container: Ubuntu 24.04.4
Architecture: qwen35
Parameters: 27.3B
Quantization: Q4_K_M
Ollama num_ctx: 131072
Embedded MTP enabled with draft_num_predict: 3

This is a custom no-vision version of the model consisting of a single 16 GB blob, rather than the stock qwen3.8:27b model's two FROM blobs.
Model blob SHA-256:
a2e72548ba1e1a90fcf8456c1d3baa0dc1ee5900b0850a6ce7897a58158bc43d

Ollama launches the runner with:

--spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-backend-sampling
--cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on
-b 512 -ub 512 -ngl 999

init: invalid token[0] = 248320
decode: failed to initialize batch
llama_decode: failed to decode, ret = -1
spec draft: llama_decode[2] returned -1
init: invalid token[2] = 248320
decode: failed to initialize batch
llama_decode: failed to decode, ret = -1
srv decode: Invalid input batch. off = 0, n_batch = 512, ret = -1
srv send_error: task id = 24404, error: Invalid input batch.

I have 17 failure events over my retained logs. The failures are sometimes isolated and sometimes occur in bursts across repeated requests.

I'm a relative beginner for the specific details, but can provide additional details where possible. Apologies if not related.

[goodbull008]

Environment:

• Model: Qwen3.8-27B-Uncensored-Q4_K_M.gguf
• Backend: Vulkan (Intel Arc A770 ×2, layer split)
• MTP: --spec-type draft-mtp → "beho beho" repetition loop
• Without MTP: stable, 12-13 t/s decode
• Context: up to 131072, KV q8_0
• Trigger: long generation tasks (100K+ tokens)