Whole-host hard lock during MTP draft catch-up prefill on multi-GPU tensor-split (b9745+)

[taylorsatula]

Summary

Whole-host hard lock (requiring power cycle) during long-prompt prefill (100k+ tokens) when MTP speculative decoding is enabled on multi-GPU tensor-split configurations. MTP-off completes identical requests without issue (~295 s).

Root cause: MTP draft catch-up queues two GPU graphs against the same mutable KV cache without waiting for either to finish. Under prolonged tensor-split prefill, the overlapping work hits a dependency error inside the CUDA/meta backend and at least one GPU enters a non-completing kernel.

The exact defective CUDA event or tensor-meta dependency below the MTP layer remains unidentified. The fix serialized MTP ubatches at the application level; a backend fix may eventually permit safe asynchronous overlap again.

Hardware

• CPU: AMD Ryzen 7 9800X3D
• RAM: 30 GiB + 16 GiB zram swap
• GPUs: 2× NVIDIA RTX 3090 (24 GiB each), PCIe Gen4 x8 bifurcation
• Driver: NVIDIA 595.84
• No CUDA P2P peer access; cross-GPU transfers use non-P2P behavior

Model

Any Qwen3.5/3.6 architecture model with MTP (NextN) layers. Reproduced with:

• Qwopus3.6-27B-Fusion-Q5_K_M.gguf

This is not model-specific. The trigger is the MTP draft catch-up path, not a particular model's MTP head.

Reproduction

Build: llama.cpp b9745 (bug present since MTP support added in PR #22673)

Server command:

llama-server \
    --model Qwopus3.6-27B-Fusion-Q5_K_M.gguf \
    --n-gpu-layers 999 \
    --split-mode tensor \
    --tensor-split 12,12 \
    --ctx-size 262144 \
    --parallel 1 \
    --flash-attn on \
    --batch-size 2048 \
    --ubatch-size 1024 \
    --cache-type-k q8_0 \
    --cache-type-v q8_0 \
    --spec-type draft-mtp \
    --spec-draft-n-max 3 \
    --spec-draft-type-k f16 \
    --spec-draft-type-v f16

Trigger: Send a single completion request with a 100k+ token prompt. The host locks within a short period. Failure position varies from ~78k to ~143k tokens with no deterministic boundary.

curl http://127.0.0.1:8080/completion \
  -H "Content-Type: application/json" \
  -d '{"prompt": "<100k+ tokens>", "n_predict": 1, "cache_prompt": false, "temperature": 0.0}'

Expected: HTTP 200 with completion.
Actual: Whole-host hard lock. GPU enters non-completing kernel. System requires power cycle. No error logged — the host is frozen.

Causal isolation

1. MTP-off control: Same requests complete fine (~295 s). The failure only occurs with MTP active.
2. Synchronize after catch-up: Adding llama_synchronize(ctx_dft) after the catch-up call moves the wedge into the draft context — confirming the failing work is in the MTP draft, not the target.
3. Both ubatches report success before GPU completion: CTX-COMPUTE-EXIT status=0 and AR-EXIT ok=1 are logged for both ubatches, but the API accepts queued work without waiting for GPU completion. The overlap has already occurred by the time any post-hoc sync runs.
4. Per-ubatch serialization fixes it: Synchronizing after each internal MTP ubatch (no callback, normal graph UID reuse) completed 3/3 ~184k-token requests, zero failures.
5. Terminal-only callback also fixes it: A callback at terminal result_output (no interior graph cut) also succeeds, proving K/V or attention boundary segmentation is not required — serialization alone is sufficient.

What does NOT explain it

• Disabling CUDA graphs — did not prevent the wedge
• Switching to fallback/butterfly allreduce — did not prevent the wedge
• Disabling ASPM L1 — did not prevent the wedge
• Reverting kernel 7.0.0-29 → 7.0.0-28 — did not prevent the wedge
• Memory and VRAM remained available throughout (no OOM)

These results narrow the failure to the ubatch overlap condition itself.

When the bug was introduced

PR #22673 added llama_decode(ctx_dft, batch) for MTP catch-up on May 16, 2026 without synchronization between internally split ubatches. Short-context operations fit in one ubatch so they never triggered it. This workload guarantees multiple ubatches (2048 batch vs. 1024 ubatch).

Proposed fix

Synchronize after each internal ubatch for MTP contexts:

bool mtp_multi_ubatch = false;
do {
    const auto & ubatch = mctx->get_ubatch();
    // ... process ...
    bool has_next = mctx->next();
    mtp_multi_ubatch |= has_next;
    if (cparams.ctx_type == LLAMA_CONTEXT_TYPE_MTP && mtp_multi_ubatch) {
        synchronize();
    }
} while (has_next);

Only activates for MTP contexts with multiple ubatches. Target decode, non-MTP speculative decoding, and short MTP drafts pass through unchanged.

PR: #26827

Related issues

• Eval bug: llama-server crashes on CUDA with Qwen3.6-27B #23210 — llama-server crashes on CUDA with Qwen3.6-27B (same symptom cluster: Qwen3.6, CUDA, MTP, long-context prefill, multi-GPU, hard hangs)
• Eval bug: llama-server hard crash (cublasSgemm INVALID_VALUE) with --spec-type draft-mtp under KV-cache saturation #26558 — different MTP crash (CUDA graph-cache growth, invalid cuBLAS stream); GGML_CUDA_DISABLE_GRAPHS=1 did not fix this reproducer

Additional data point

Single-GPU MTP (one RTX 3090, Qwen3.8-27B, -ub 1024, FA on) has not reproduced this — consistent with the multi-GPU tensor-split overlap being the trigger. See comment from @47Hunter47.

An adjacent reproducer from @Anbeeld (#27750) shows a related multi-GPU tensor-split instability with MTP off, suggesting the MTP overlap is one trigger for a deeper tensor-split/backend ordering or lifetime problem rather than the only way to reach it.

[GrassfedMod]

It happens on mine as well. But with 2 v100s on nvlink and p2p also off. Mine crashed on the generation stage at 18k prefil though with my qwen3.8 27B Q4 UD K_Xl. Q8 kv at max native ctx.
It died on around 2-4th regression pass on generation with mtp on with draft length 4. Il send you the details later if you need it.