Eval bug: llama-server crashes on CUDA with Qwen3.6-27B

[t-jet]

Name and Version

llama-cli --version
version: 9186 (3a92bc9)
built with Clang 19.1.5 for Windows x86_64

Operating systems

Windows

GGML backends

CUDA

Hardware

AMD Ryzen 7900X, 64GB RAM, chipset X870E + 2 x NVIDIA RTX 5060 Ti 16 GB

Models

unsloth/Qwen3.6-27B-MTP-GGUF/Qwen3.6-27B-Q5_K_M.gguf

Problem description & steps to reproduce

After running a model for 3-4 prompts in the agentic workflow environment, the server silently crashes without any error message, sometime reporting "CUDA Error". There are no signs of GPU memory overload and enough amount of free RAM.

Command line parameters used to launch model:

llama-server.exe ^
  -m c:\models\unsloth\Qwen3.6-27B-MTP-GGUF\Qwen3.6-27B-Q5_K_M.gguf ^
  --port 35762 ^
  -c 65536 ^
  --parallel 1 ^
  --flash-attn on ^
  --no-context-shift ^
  -ngl -1 ^
  --threads -1 ^
  --jinja ^
  --reasoning on ^
  --no-mmap ^
  --mlock ^
  --spec-type draft-mtp ^
  --spec-draft-n-max 3 ^
  -lv 4 ^
  --cache-type-k q8_0 ^
  --cache-type-v q8_0 ^
  --metrics

First Bad Commit

No response

Relevant log output

Logs

�[34m2.28.454.754�[0m �[32mI �[0mslot create_check: id  0 | task 0 | created context checkpoint 3 of 32 (pos_min = 24575, pos_max = 24575, n_tokens = 24576, size = 201.095 MiB)
�[34m2.31.613.813�[0m �[32mI �[0mslot print_timing: id  0 | task 0 | prompt processing, n_tokens =  26624, progress = 0.91, t =  37.80 s / 704.26 tokens per second
�[34m2.31.613.819�[0m �[32mI �[0mslot update_slots: id  0 | task 0 | cached n_tokens = 26624, memory_seq_rm [26624, end)
�[34m2.34.775.559�[0m �[32mI �[0mslot print_timing: id  0 | task 0 | prompt processing, n_tokens =  28672, progress = 0.98, t =  40.97 s / 699.90 tokens per second
�[34m2.34.775.565�[0m �[32mI �[0mslot update_slots: id  0 | task 0 | cached n_tokens = 28672, memory_seq_rm [28672, end)
�[34m2.35.141.953�[0m �[32mI �[0mslot print_timing: id  0 | task 0 | prompt processing, n_tokens =  28878, progress = 0.98, t =  41.33 s / 698.68 tokens per second
�[34m2.35.141.959�[0m �[32mI �[0mslot update_slots: id  0 | task 0 | cached n_tokens = 28878, memory_seq_rm [28878, end)
�[34m2.35.191.665�[0m �[32mI �[0mslot create_check: id  0 | task 0 | created context checkpoint 4 of 32 (pos_min = 28877, pos_max = 28877, n_tokens = 28878, size = 210.104 MiB)
�[34m2.35.991.114�[0m �[32mI �[0mslot print_timing: id  0 | task 0 | prompt processing, n_tokens =  29390, progress = 1.00, t =  42.18 s / 696.75 tokens per second
�[34m2.35.991.120�[0m �[32mI �[0mslot update_slots: id  0 | task 0 | cached n_tokens = 29390, memory_seq_rm [29390, end)
�[34m2.35.993.615�[0m �[32mI �[0mslot init_sampler: id  0 | task 0 | init sampler, took 2.32 ms, tokens: text = 29394, total = 29394
�[34m2.36.041.609�[0m �[32mI �[0mslot create_check: id  0 | task 0 | created context checkpoint 5 of 32 (pos_min = 29389, pos_max = 29389, n_tokens = 29390, size = 211.177 MiB)
�[34m2.36.183.565�[0m �[32mI �[0msrv  log_server_r: done request: POST /v1/chat/completions 127.0.0.1 200
�[34m2.39.154.323�[0m �[32mI �[0mslot print_timing: id  0 | task 0 | n_decoded =    102, tg =  34.31 t/s
�[34m2.42.243.591�[0m �[32mI �[0mslot print_timing: id  0 | task 0 | n_decoded =    212, tg =  34.97 t/s
�[34m2.45.255.703�[0m �[32mI �[0mslot print_timing: id  0 | task 0 | n_decoded =    306, tg =  33.72 t/s
�[34m2.48.263.476�[0m �[32mI �[0mslot print_timing: id  0 | task 0 | n_decoded =    386, tg =  31.95 t/s
�[34m2.48.444.850�[0m �[32mI �[0mreasoning-budget: deactivated (natural end)
�[34m2.51.266.797�[0m �[32mI �[0mslot print_timing: id  0 | task 0 | n_decoded =    502, tg =  33.28 t/s
�[34m2.53.287.864�[0m �[32mI �[0mslot print_timing: id  0 | task 0 |
prompt eval time =   42371.37 ms / 29394 tokens (    1.44 ms per token,   693.72 tokens per second)
       eval time =   17106.68 ms /   585 tokens (   29.24 ms per token,    34.20 tokens per second)
      total time =   59478.05 ms / 29979 tokens
draft acceptance rate = 0.70945 (  398 accepted /   561 generated)
�[34m2.53.287.887�[0m �[32mI �[0mstatistics draft-mtp: #calls(b,g,a) = 1 187 187, #gen drafts = 187, #acc drafts = 162, #gen tokens = 561, #acc tokens = 398, dur(b,g,a) = 0.001, 2902.024, 0.192 ms
�[34m2.53.288.992�[0m �[32mI �[0mslot      release: id  0 | task 0 | stop processing: n_tokens = 29979, truncated = 0
�[34m2.53.289.001�[0m �[32mI �[0msrv  update_slots: all slots are idle
�[34m2.53.763.627�[0m �[32mI �[0msrv  params_from_: Chat format: peg-native
�[34m2.53.769.536�[0m �[32mI �[0mslot get_availabl: id  0 | task -1 | selected slot by LCP similarity, sim_best = 0.971 (> 0.100 thold), f_keep = 0.980
�[34m2.53.770.918�[0m �[32mI �[0mreasoning-budget: activated, budget=2147483647 tokens
�[34m2.53.771.286�[0m �[32mI �[0mslot launch_slot_: id  0 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> ?top-p -> min-p -> ?xtc -> ?temp-ext -> dist
�[34m2.53.771.296�[0m �[32mI �[0mslot launch_slot_: id  0 | task -1 | sampler params:
        repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
        dry_multiplier = 0.000, dry_base = 1.750, dry_allowed_length = 2, dry_penalty_last_n = 140032
        top_k = 20, top_p = 1.000, min_p = 0.050, xtc_probability = 0.000, xtc_threshold = 0.100, typical_p = 1.000, top_n_sigma = -1.000, temp = 1.000
        mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000, adaptive_target = -1.000, adaptive_decay = 0.900
�[34m2.53.771.298�[0m �[32mI �[0mslot launch_slot_: id  0 | task 205 | processing task, is_child = 0
�[34m2.53.771.314�[0m �[32mI �[0mslot update_slots: id  0 | task 205 | new prompt, n_ctx_slot = 140032, n_keep = 0, task.n_tokens = 30270
�[34m2.53.771.327�[0m �[35mW slot update_slots: id  0 | task 205 | n_past = 29393, slot.prompt.tokens.size() = 29979, seq_id = 0, pos_min = 29978, n_swa = 0
�[0m�[34m2.53.771.328�[0m �[32mI �[0mslot update_slots: id  0 | task 205 | Checking checkpoint with [29389, 29389] against 29393...
�[34m2.53.791.917�[0m �[35mW slot update_slots: id  0 | task 205 | restored context checkpoint (pos_min = 29389, pos_max = 29389, n_tokens = 29390, n_past = 29390, size = 211.177 MiB)
�[0m�[34m2.53.791.924�[0m �[32mI �[0mslot update_slots: id  0 | task 205 | cached n_tokens = 29390, memory_seq_rm [29390, end)
�[34m2.54.383.656�[0m �[32mI �[0mslot update_slots: id  0 | task 205 | cached n_tokens = 29754, memory_seq_rm [29754, end)
�[34m2.54.431.669�[0m �[32mI �[0mslot create_check: id  0 | task 205 | created context checkpoint 6 of 32 (pos_min = 29753, pos_max = 29753, n_tokens = 29754, size = 211.939 MiB)
�[34m2.55.238.829�[0m �[32mI �[0mslot update_slots: id  0 | task 205 | cached n_tokens = 30266, memory_seq_rm [30266, end)
�[34m2.55.241.379�[0m �[32mI �[0mslot init_sampler: id  0 | task 205 | init sampler, took 2.37 ms, tokens: text = 30270, total = 30270
�[34m2.55.290.705�[0m �[32mI �[0mslot create_check: id  0 | task 205 | created context checkpoint 7 of 32 (pos_min = 30265, pos_max = 30265, n_tokens = 30266, size = 213.011 MiB)
�[34m2.55.378.451�[0m �[32mI �[0msrv  log_server_r: done request: POST /v1/chat/completions 127.0.0.1 200
�[34m2.57.485.689�[0m �[32mI �[0mreasoning-budget: deactivated (natural end)
�[34m2.58.310.752�[0m �[32mI �[0mslot print_timing: id  0 | task 205 | n_decoded =    101, tg =  34.44 t/s
�[34m3.01.341.495�[0m �[32mI �[0mslot print_timing: id  0 | task 205 | n_decoded =    232, tg =  38.90 t/s
�[34m3.03.355.033�[0m �[32mI �[0mslot print_timing: id  0 | task 205 |
prompt eval time =    1606.20 ms /   880 tokens (    1.83 ms per token,   547.88 tokens per second)
       eval time =    7977.34 ms /   320 tokens (   24.93 ms per token,    40.11 tokens per second)
      total time =    9583.54 ms /  1200 tokens
draft acceptance rate = 0.88889 (  232 accepted /   261 generated)
�[34m3.03.355.050�[0m �[32mI �[0mstatistics draft-mtp: #calls(b,g,a) = 2 274 274, #gen drafts = 274, #acc drafts = 244, #gen tokens = 822, #acc tokens = 630, dur(b,g,a) = 0.001, 4202.304, 0.293 ms
�[34m3.03.356.068�[0m �[32mI �[0mslot      release: id  0 | task 205 | stop processing: n_tokens = 30589, truncated = 0
�[34m3.03.356.254�[0m �[32mI �[0msrv  update_slots: all slots are idle
�[34m3.06.066.990�[0m �[32mI �[0msrv  params_from_: Chat format: peg-native
�[34m3.06.072.514�[0m �[32mI �[0mslot get_availabl: id  0 | task -1 | selected slot by LCP similarity, sim_best = 0.843 (> 0.100 thold), f_keep = 0.990
�[34m3.06.073.976�[0m �[32mI �[0mreasoning-budget: activated, budget=2147483647 tokens
�[34m3.06.074.508�[0m �[32mI �[0mslot launch_slot_: id  0 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> ?top-p -> min-p -> ?xtc -> ?temp-ext -> dist
�[34m3.06.074.522�[0m �[32mI �[0mslot launch_slot_: id  0 | task -1 | sampler params:
        repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
        dry_multiplier = 0.000, dry_base = 1.750, dry_allowed_length = 2, dry_penalty_last_n = 140032
        top_k = 20, top_p = 1.000, min_p = 0.050, xtc_probability = 0.000, xtc_threshold = 0.100, typical_p = 1.000, top_n_sigma = -1.000, temp = 1.000
        mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000, adaptive_target = -1.000, adaptive_decay = 0.900
�[34m3.06.074.525�[0m �[32mI �[0mslot launch_slot_: id  0 | task 296 | processing task, is_child = 0
�[34m3.06.074.532�[0m �[32mI �[0mslot update_slots: id  0 | task 296 | new prompt, n_ctx_slot = 140032, n_keep = 0, task.n_tokens = 35888
�[34m3.06.074.547�[0m �[35mW slot update_slots: id  0 | task 296 | n_past = 30269, slot.prompt.tokens.size() = 30589, seq_id = 0, pos_min = 30588, n_swa = 0
�[0m�[34m3.06.074.548�[0m �[32mI �[0mslot update_slots: id  0 | task 296 | Checking checkpoint with [30265, 30265] against 30269...
�[34m3.06.097.455�[0m �[35mW slot update_slots: id  0 | task 296 | restored context checkpoint (pos_min = 30265, pos_max = 30265, n_tokens = 30266, n_past = 30266, size = 213.011 MiB)
�[0m�[34m3.06.097.463�[0m �[32mI �[0mslot update_slots: id  0 | task 296 | cached n_tokens = 30266, memory_seq_rm [30266, end)
�[34m3.09.436.106�[0m �[32mI �[0mslot print_timing: id  0 | task 296 | prompt processing, n_tokens =   2048, progress = 0.90, t =   3.36 s / 609.24 tokens per second
�[34m3.09.436.113�[0m �[32mI �[0mslot update_slots: id  0 | task 296 | cached n_tokens = 32314, memory_seq_rm [32314, end)
�[34m3.12.735.782�[0m �[32mI �[0mslot print_timing: id  0 | task 296 | prompt processing, n_tokens =   4096, progress = 0.96, t =   6.66 s / 614.90 tokens per second
�[34m3.12.735.787�[0m �[32mI �[0mslot update_slots: id  0 | task 296 | cached n_tokens = 34362, memory_seq_rm [34362, end)
D:\source\llama.cpp\ggml\src\ggml-cuda\ggml-cuda.cu:102: CUDA error

[t-jet]

Patch to fix it on my environment

Disclaimer 1 This patch doesn't pretend to become a PR, just provides a record of the investigation and fix for this specific issue.
Discaimer 2 I'm not an expert in llama-server internals, so information below produced with a help of GenAI as well as the patch itself.

Results

Instead of crashing after three-four request in the complex agentic workflow, configuration ran for 10+ hours without crashing on the same workflow and successfully processed all requests almost at the expected speed.

Branch and compilation.

Patch: 0001-feat-enhance-MTP-context-memory-management-and-check.patch

Patch was applied on top of main branch at commit 64b38b561b987679c4e1c6231f93860d3eec2638 (2026-05-16).

Build was done on Win11 with Visual Studio 2026 Developer prompt and CUDA 13.1 toolkit.

CMake commands was:

cmake -S . -B build-cuda -G "Visual Studio 17 2022" -A x64 -DGGML_CUDA=ON -DGGML_NATIVE=OFF -DBUILD_SHARED_LIBS=OFF
cmake --build build-cuda --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split

Workstation parameters: Windows 11 Pro 25H2, AMD Ryzen 7900X, 2 x NVIDIA RTX 5060 Ti 16 GB, 64GB RAM

llama-server launch parameters:

llama-server.exe ^
  -m c:\models\unsloth\Qwen3.6-27B-MTP-GGUF\Qwen3.6-27B-Q5_K_M.gguf ^
  --port 35762 ^
  -c 65536 ^
  --parallel 1 ^
  --flash-attn on ^
  --no-context-shift ^
  -ngl -1 ^
  --threads -1 ^
  --jinja ^
  --reasoning on ^
  --no-mmap ^
  --mlock ^
  --spec-type draft-mtp ^
  --spec-draft-n-max 3 ^
  -lv 4 ^
  --cache-type-k q8_0 ^
  --cache-type-v q8_0 ^
  --metrics

Summary

This change stabilizes llama-server on Qwen 3.6 hybrid/recurrent workloads with draft-mtp enabled in two separate places where memory management was previously too optimistic:

• the fit / startup path, where MTP draft-context overhead was not included in the memory budget
• the runtime prompt-cache / checkpoint path, where large prompt state and checkpoint exports could terminate the process or accumulate excessive RAM

The patch therefore addresses two related failure modes:

• CUDA OOM during real prompt processing even though -fit reported that the configuration should fit
• silent exits during prompt-cache save and later during checkpoint creation on long-running cached sessions

The patch keeps caching enabled, preserves the existing reuse logic, and makes both fitting and runtime state export account for MTP / checkpoint memory more realistically.

1. Initial Symptoms And Diagnostic Logic

Initial symptoms

The first symptom was not a silent exit. It was a CUDA failure during prompt processing on a later turn of a large draft-mtp session, even though the initial fit step had already reported that the configuration met all device-memory targets.

The relevant pattern was:

• common_fit_params reported that the projected memory usage fit available GPU memory
• the server then created the MTP draft context afterwards
• prompt processing failed later with a CUDA error during actual evaluation

That suggested a startup-time underestimation rather than a steady-state logic bug in the prompt processor itself.

The second symptom was a silent server exit immediately after a log line like:

srv prompt_save: - saving prompt with length ..., total state size = ... MiB

There was no subsequent prompt_load, no cache-state summary, and no explicit error from the server. That strongly suggested the process was dying inside or immediately after prompt-state export, not during normal slot selection or request execution.

After the MTP fit fix, and then after the first prompt-save fix, the crash point moved. The server now survived model initialization and later survived prompt_save, updated the cache successfully, continued serving requests, and only died later during a subsequent long request. The new tail showed the server reaching:

• slot init_sampler
• then stopping before the next normal checkpoint / decode progress line

That narrowed the second failure to the checkpoint creation path or the immediate decode handoff that follows it.

Diagnostic logic

The investigation separated the problem into three concrete paths:

1. MTP context fitting path

• common_fit_params() runs before the server creates the MTP draft context.
• For draft-mtp without a separate draft model, the fit step was only budgeting the primary context.
• The additional ctx_type = LLAMA_CONTEXT_TYPE_MTP context was created afterwards, so its context + compute overhead was outside the fit margins.

That explains why -fit could say the setup fit, but actual prompt processing still failed later with CUDA OOM once the real MTP context existed.

1. Prompt-cache save path

• get_available_slot() may save the old slot state before reusing the slot for a new task.
• server_slot::prompt_save() exported the live target and draft state into RAM and also duplicated prompt metadata, including the full checkpoint list.
• For long recurrent / MTP prompts, the live state was already large, and the checkpoint list added several more GiB of RAM pressure.

1. Checkpoint creation path

• During prompt processing, the server periodically creates rollback checkpoints for recurrent / bounded-removal contexts.
• These checkpoints were also exported through llama_state_seq_get_data_ext().
• The export path assumed success and would abort the process on short-copy / mismatch behavior.

The key log evidence was:

• fit reporting sufficient headroom before the MTP context existed
• the MTP draft context then allocating additional KV / compute memory after the fit step
• prompt-cache save no longer being the final line after the first patch
• checkpoint sizes staying very large on this model, often around 170-220 MiB each
• cached prompt entries growing to multi-GiB sizes because the cache stored both full prompt state and a long checkpoint list
• the server running with COMMON_CONTEXT_SEQ_RM_TYPE_RS, which means checkpoints remain part of the normal reuse path for this workload

From that, the working diagnosis was:

• the earliest CUDA failure came from missing MTP overhead in the fit budget
• after that was addressed, the next failure came from the prompt-cache save path under large-state pressure
• the remaining failure came from unguarded checkpoint export and uncontrolled checkpoint memory accumulation

2. Fix Logic

A. Account for MTP draft-context overhead during -fit

The common/ side of the patch adds a dedicated helper in common/fit.cpp and declares it in common/fit.h:

• common_get_mtp_ctx_memory_overhead(...)

That helper:

• loads the model with no_alloc = true
• creates a temporary llama_context with ctx_type = LLAMA_CONTEXT_TYPE_MTP
• forces n_rs_seq = 0 for that temporary MTP measurement context
• reads the memory breakdown from the constructed context
• sums only context + compute overhead per device and for host memory

The corresponding change in common/common.cpp uses that helper when all of the following are true:

• fitting is enabled
• draft-mtp is active
• there is no separate draft model

Instead of mutating the user’s original fit targets, the code copies params.fit_params_target into a temporary fit_targets vector, adds the measured MTP overhead to those margins, and passes the adjusted margins into common_fit_params().

This means the fit step now leaves enough headroom for the MTP context that will actually be created later by the server.

B. Harden prompt-cache save

The prompt-cache path was changed so that prompt_save() now:

• checks the exact byte count returned by llama_state_seq_get_data_ext() for both target and draft state
• discards incomplete cache entries if state export fails
• returns failure instead of leaving a partially initialized cache record behind

Related cache-management changes in server_prompt_cache support that behavior:

• alloc() now focuses on reserving the state buffers first, instead of eagerly cloning the full prompt metadata during allocation
• discard() removes incomplete cache entries when prompt export fails mid-save

This turns prompt-cache save from a crash-prone path into a recoverable path.

C. Avoid duplicating the full checkpoint list during cache save

When a slot is being repurposed, the old code effectively kept two large copies of prompt metadata alive at once:

• the live slot prompt and checkpoints
• the cached prompt and checkpoints being written into RAM

The fix changes this to move prompt metadata into the cache when appropriate, then clear the live slot before reloading. That removes a major temporary RAM spike during slot reuse.

D. Add fail-soft checkpoint export helpers

The server now has dedicated helpers for checkpoint export of target and draft state. These helpers:

• allocate checkpoint buffers explicitly
• catch std::bad_alloc
• validate the exact number of bytes returned by state export
• clear the failed checkpoint buffer and report failure instead of aborting the process

This logic is used for:

• normal prompt-processing checkpoints
• speculative / MTP checkpoint paths

E. Skip invalid checkpoints instead of aborting

Checkpoint creation now builds a temporary checkpoint first and only appends it to the prompt if export succeeds. If export fails, the server logs a warning and skips that checkpoint.

That means the request can continue with less rollback coverage instead of killing the process.

F. Trim checkpoint memory by bytes, not only by count

The previous logic limited checkpoints only by count. That is not enough when each checkpoint can be hundreds of MiB.

The fix adds byte-based trimming in two places:

1. Live slot checkpoints

• after creating a checkpoint, old checkpoints are dropped until the total checkpoint bytes are roughly bounded by one full prompt-state budget

1. Cached prompt checkpoints

• before keeping a prompt in the host prompt cache, its checkpoint list is trimmed using a similar budget

This keeps caching enabled while preventing checkpoint memory from growing far beyond the size of the actual reusable state.

G. Keep speculative decoding functional when checkpoint export fails

The speculative checkpoint path now degrades gracefully:

• if speculative checkpoint export fails, the speculative draft / checkpoint is cleared for that step
• the server continues without using that speculative checkpoint rather than aborting

This keeps draft-mtp usable even under high memory pressure.

3. Possible Effects On Other Functionality

Fit may choose more conservative settings for draft-mtp

Because MTP draft-context overhead is now included in the fit margins, automatic fitting may choose a smaller context size or otherwise leave more headroom on GPU devices than before.

Effect:

• -fit results for draft-mtp can become more conservative
• this is expected, because the previous fit result was undercounting real runtime memory
• non-MTP and separate-draft-model configurations should be unchanged by this logic

Slightly longer startup when -fit and draft-mtp are both used

The new helper creates a temporary MTP context to measure overhead before the final fit decision is applied.

Effect:

• startup can do one extra model/context measurement pass in this specific configuration
• this only affects the fitting path, not steady-state generation

Reduced rollback depth in long sessions

Because checkpoints are now trimmed by bytes as well as by count, long-running sessions may keep fewer old checkpoints than before, especially on recurrent / MTP models with large checkpoint payloads.

Effect:

• rollback may fall back to a newer checkpoint and replay a slightly longer suffix
• prompt reuse still works, but some restores may save less work than before in the extreme long-context case

Graceful degradation instead of hard failure

If checkpoint export or speculative checkpoint export fails, the server now skips the checkpoint.

Effect:

• some requests may lose a reuse optimization for that step
• the server stays alive instead of terminating

This is an intentional tradeoff in favor of stability.

Slightly different cache contents

Cached prompt entries may now retain fewer checkpoints because oversized checkpoint lists are trimmed before the prompt is stored in RAM.

Effect:

• host cache still works
• cross-request or cross-slot resume may sometimes require replaying a longer suffix than before
• total RAM use is significantly better bounded

No intended change to prompt selection or decoding semantics

The patch does not change:

• slot selection heuristics
• prompt similarity logic
• the main decode path when state export succeeds
• the correctness of existing cache reuse / checkpoint restore logic

The goal is not to change model behavior, but to make the server survive large-state export failures and avoid pathological checkpoint memory growth.

[arie-s]

I may have found a related fix/workaround for this crash class.

I reproduced a very similar failure on Linux/CUDA with Qwen3.6-27B MTP, llama-server, tensor split across 2x RTX 5060 Ti 16GB, -np 1, --spec-type draft-mtp, and server prompt cache enabled. The crash happened after repeated requests/checkpoint restores in the MTP draft path, with a stack like:

• ggml_new_tensor
• ggml_backend_meta_buffer_init_tensor
• ggml_gallocr_alloc_graph
• llama_context::process_ubatch
• common_speculative_state_draft_mtp::draft
• server_context_impl::update_slots

The assert/log was:

GGML_ASSERT(obj_new) failed
ggml_new_object: not enough space in the context's memory pool

In current main, ggml_backend_meta_buffer_reset() still only resets the simple child buffers:

static void ggml_backend_meta_buffer_reset(ggml_backend_buffer_t buffer) {
    const size_t n_buffers = ggml_backend_meta_buffer_n_bufs(buffer);
    for (size_t i = 0; i < n_buffers; i++) {
        ggml_backend_buffer_reset(ggml_backend_meta_buffer_simple_buffer(buffer, i));
    }
}

Locally, the crash went away after changing reset to clear the meta-buffer cached state and reset each cached buffer-config context:

static void ggml_backend_meta_buffer_reset(ggml_backend_buffer_t buffer) {
    GGML_ASSERT(ggml_backend_buffer_is_meta(buffer));
    ggml_backend_meta_buffer_context * buf_ctx = (ggml_backend_meta_buffer_context *) buffer->context;

    buf_ctx->split_state_cache.clear();
    buf_ctx->simple_tensors.clear();

    for (auto & [ctx, buf] : buf_ctx->buf_configs) {
        ggml_reset(ctx);
        ggml_backend_buffer_reset(buf);
    }
}

Validation so far:

• patched local build from current main (dd7cad719) plus the reset change above
• CUDA backend
• Qwen3.6-27B MTP IQ4_NL, tensor split 1,1
• one long-lived llama-server process
• 3 consecutive full agent task suites: 9/9 passed
• then a full uncapped prompt quality suite on the same server process
• no allocator assert, no ggml_new_object, no crash

I am not certain this is the complete/global fix for all reports here, but it looks like the meta backend reset path can leave stale cached state across prompt-cache/checkpoint reuse, and MTP + tensor split makes it much easier to hit.

[t-jet]

Fix is updated, now it becomes more general and incorporates @arie-s fixes, but still tested only on my own hardware and not a PR-grade.

Patch Notes

Disclaimer 1 This patch doesn't pretend to become a PR, just provides a record of the investigation and fix for this specific issue.
Disclaimer 2 I'm not an expert in llama-server internals, so information below produced with a help of GenAI as well as the patch itself.

Results

On the investigated Qwen 3.6 draft-mtp workload, the validated configuration completed a 10+ hour run without crashing and processed the complex agentic workflow at close to the expected speed.

That long-run result comes from the investigated configuration used during the fix development. A fresh equivalent long-run validation run on the exact current checked-out tree is not recorded in this note.

The current fix state is intended to cover the full observed and inferred failure set for this workload:

• CUDA OOM during real prompt processing even though -fit reported that the configuration should fit
• silent exits during prompt-cache save and during checkpoint creation on long-running cached sessions
• first-request failures on very large prompt-prefill batches
• startup-time CUDA failure in the MTP draft context after a failed reserve/fallback path
• silent process termination under severe host-memory pressure if a standard exception escaped task scheduling or slot updates
• allocator aborts in the GGML meta backend during repeated MTP draft graph reuse on similar tensor-split workloads

Branch and compilation

This note describes the diff between:

• current origin/master base: 40d5358d3c730b81729ba81cd5c44ed596d02510 (tests : move save-load-state from examples to tests (#23336))
• current branch head: a4303153fe84c1ce33fa071d604fc6e4f6e9d657 (fix: correct indentation in MTP pre-norm embedding setup)

The fixed branch is hosted in the forked repository here.

Relative to origin/master, the current state differs in ten files:

• common/common.cpp
• common/fit.cpp
• common/fit.h
• common/speculative.cpp
• ggml/src/ggml-backend-meta.cpp
• src/llama-context.cpp
• tools/server/server-context.cpp
• tools/server/server-queue.cpp
• tools/server/server-task.cpp
• tools/server/server-task.h

Build environment used during the investigation:

• Windows 11 Pro 25H2
• AMD Ryzen 7900X
• 2 x NVIDIA RTX 5060 Ti 16 GB
• 64 GB RAM
• Visual Studio 2026 Developer prompt
• CUDA 13.1 toolkit

CMake commands used during the investigation:

cmake -S . -B build-cuda -G "Visual Studio 17 2022" -A x64 -DGGML_CUDA=ON -DGGML_NATIVE=OFF -DBUILD_SHARED_LIBS=OFF
cmake --build build-cuda --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split

Incremental validation command:

cmake --build build-cuda --config Release -j --target llama-server

Long-run validation configuration used during the investigation:

llama-server.exe ^
  -m c:\models\unsloth\Qwen3.6-27B-MTP-GGUF\Qwen3.6-27B-Q5_K_M.gguf ^
  --port 35762 ^
  -c 65536 ^
  --parallel 1 ^
  --flash-attn on ^
  --no-context-shift ^
  -ngl -1 ^
  --threads -1 ^
  --jinja ^
  --reasoning on ^
  --no-mmap ^
  --mlock ^
  --spec-type draft-mtp ^
  --spec-draft-n-max 3 ^
  -lv 4 ^
  --cache-type-k q8_0 ^
  --cache-type-v q8_0 ^
  --metrics

Reproduction configuration used for the large-prompt and startup failures:

llama-server.exe ^
  -m c:\Users\think\.lmstudio\models\unsloth\Qwen3.6-27B-MTP-GGUF\Qwen3.6-27B-Q4_K_M.gguf ^
  --port 52411 ^
  -c 140000 ^
  --parallel 1 ^
  --flash-attn on ^
  --no-context-shift ^
  -ngl -1 ^
  --threads -1 ^
  --jinja ^
  --reasoning on ^
  --no-mmap ^
  --mlock ^
  --spec-type draft-mtp ^
  --spec-draft-n-max 3 ^
  -lv 4 ^
  --cache-type-k q8_0 ^
  --cache-type-v q8_0 ^
  --cache-ram 16384 ^
  --metrics

Summary

This change stabilizes llama-server on Qwen 3.6 hybrid/recurrent workloads with draft-mtp enabled in six places where memory management or speculative integration was too optimistic:

• the fit / startup path, where the secondary MTP draft context was outside the original fit budget
• the runtime prompt-cache / checkpoint path, where large prompt state and checkpoint exports could terminate the process or accumulate excessive RAM
• the speculative MTP prompt-prefill path, where the server was not recognizing speculative prompt batches correctly under the current upstream signaling contract
• the MTP draft-context startup path, where the reserve/fallback path could poison CUDA state on this workload
• the main runtime callback path, where uncaught standard exceptions from task scheduling or slot updates could unwind out of the queue loop under host-memory pressure
• the backend meta-buffer reset path, where cached split/tensor state could survive across repeated graph allocation cycles

The change therefore targets six related failure modes:

• CUDA OOM during real prompt processing even though -fit reported that the configuration should fit
• silent exits during prompt-cache save and during checkpoint creation on long-running cached sessions
• first-request failures on very large prompt-prefill batches because speculative prompt batches were not being capped and were not marking prompt output rows correctly
• startup-time CUDA failure in the MTP draft context after a failed pipeline-parallel reserve path
• silent process termination if a std::bad_alloc or another standard exception escaped the main runtime callbacks
• allocator aborts in ggml_new_object / meta-buffer graph allocation on similar repeated-reuse MTP workloads

The branch keeps caching enabled, preserves the existing reuse logic, includes active fit accounting for the MTP draft context, makes both fitting and runtime state export account for MTP / checkpoint memory more realistically, adapts the speculative prompt path to the current upstream implementation, adds a narrow runtime exception barrier around the main queue callbacks, and resets cached meta-buffer state before repeated graph allocation cycles.

1. Symptoms And Diagnostic Logic

Symptoms

One failure mode was a CUDA error during prompt processing on a later turn of a large draft-mtp session, even though the -fit step had already reported that the configuration met device-memory targets.

The relevant pattern was:

• common_fit_params reported that the projected memory usage fit available GPU memory
• the server then created the MTP draft context afterwards
• prompt processing then failed with a CUDA error during actual evaluation

That suggested a startup-time underestimation rather than a steady-state logic bug in the prompt processor itself.

Another failure mode was a silent server exit immediately after a log line like:

srv prompt_save: - saving prompt with length ..., total state size = ... MiB

There was no subsequent prompt_load, no cache-state summary, and no explicit error from the server. That strongly suggested the process was dying inside or immediately after prompt-state export, not during normal slot selection or request execution.

Another observed failure tail reached:

• slot init_sampler
• then stopping before the next normal checkpoint / decode progress line

That tail points to the checkpoint creation path or the immediate decode handoff that follows it.

On the large-prompt path, a separate failure mode was a first-request failure. The critical current upstream behavior is that MTP prompt hidden-state demand is reported through need_embd_pre_norm() rather than through need_embd() alone. If the server batching logic does not account for that, then:

• speculative MTP prompt batches are not recognized correctly
• the first prompt batch is no longer capped conservatively
• prompt tokens are no longer marked as output rows for the speculative pre-norm flow

That is enough to trigger first-request failure on very large prompt-prefill batches.

Another failure mode was a startup-time failure in the MTP draft context:

• the draft context attempted a pipeline-parallel reserve path during startup
• that path could fail with a large host-buffer allocation failure
• the reserve logic could retry without pipeline parallelism and appear to recover
• a subsequent CUDA call during server initialization could still abort with a generic CUDA error, consistent with a poisoned CUDA state after the failed reserve path

That made startup itself unstable even before the first request was processed.

Under severe host-memory pressure, another plausible failure mode was silent process termination from an escaped standard exception:

• server_queue::start_loop() invoked task scheduling and slot updates through callbacks without a queue-level exception barrier
• server_context::start_loop() executed that loop on the main thread
• an escaped std::bad_alloc or another standard exception from those runtime paths could therefore terminate the process instead of returning task errors

On a closely related CUDA / MTP / tensor-split workload, another failure class reached a hard allocator abort in the GGML meta backend:

• the stack ran through ggml_backend_meta_buffer_init_tensor(), ggml_gallocr_alloc_graph(), llama_context::process_ubatch(), the MTP draft path, and server_context_impl::update_slots()
• the terminal failure was ggml_new_object: not enough space in the context's memory pool
• current origin/master still reset the meta buffer by resetting only the simple child buffers, not the cached split/tensor state or the per-buffer ggml contexts

Diagnostic logic

The investigation separated the problem into seven concrete paths:

1. MTP context fitting path

• common_fit_params() runs before the server creates the MTP draft context.
• For draft-mtp without a separate draft model, the fit step budgets only the primary context.
• The additional ctx_type = LLAMA_CONTEXT_TYPE_MTP context is created afterwards, so its context + compute overhead is outside the fit margins.

That explains why -fit could say the setup fit, but prompt processing could still fail with CUDA OOM once the real MTP context existed.

2. Prompt-cache save path

• get_available_slot() may save the old slot state before reusing the slot for a new task.
• server_slot::prompt_save() exported the live target and draft state into RAM and also duplicated prompt metadata, including the full checkpoint list.
• For long recurrent / MTP prompts, the live state was already large, and the checkpoint list added several more GiB of RAM pressure.

3. Checkpoint creation path

• During prompt processing, the server periodically creates rollback checkpoints for recurrent / bounded-removal contexts.
• These checkpoints were also exported through llama_state_seq_get_data_ext().
• The export path assumed success and could abort the process on short-copy or mismatch behavior.

4. Speculative prompt-prefill path

• The current upstream signaling contract reports MTP prompt hidden-state demand through need_embd_pre_norm().
• The local server batching logic must still translate that requirement into output-row handling for prompt batches.
• If it does not, the required first-request protection is effectively disabled.

5. MTP draft-context startup path

• The draft context used a more aggressive startup scheduling path than this workload could tolerate.
• The fallback path after reserve failure was not cleanly recovering on the tested Windows setup.

6. Main runtime exception path

• server_queue::start_loop() invoked process_single_task() and update_slots() through callbacks without a queue-level catch boundary.
• Those runtime paths contain allocations and helper calls that are not all converted into task-local errors.
• Under severe host-memory pressure, an escaped std::bad_alloc or another standard exception could unwind out of the main loop and terminate the server.

7. Meta-backend reset / graph allocation path

• The GGML meta backend keeps cached split-state and simple-tensor mappings in the meta-buffer context.
• The reset path only reset the simple child buffers, leaving those caches and the per-buffer ggml contexts alive across allocation cycles.
• On repeated MTP draft graph allocation, especially with tensor split, that stale state could plausibly feed into a later allocator abort in the context memory pool.

The key log evidence was:

• fit reporting sufficient headroom before the MTP context existed
• the MTP draft context then allocating additional KV and compute memory after the fit step
• failure tails that reached slot init_sampler after prompt-save activity and then stopped before the next checkpoint or decode progress line
• checkpoint sizes staying very large on this model, often around 170 to 220 MiB each
• cached prompt entries growing to multi-GiB sizes because the cache stored both full prompt state and a long checkpoint list
• the server running with COMMON_CONTEXT_SEQ_RM_TYPE_RS, which means checkpoints remain part of the normal reuse path for this workload
• first-request failures clustering around very large prompt-prefill batches
• startup failures clustering around the MTP draft-context reserve/fallback path
• queue callbacks being wired directly to task scheduling and slot updates without a top-level runtime exception barrier
• a related allocator-abort stack on similar hardware reaching ggml_backend_meta_buffer_init_tensor() and ggml_new_object() during repeated MTP draft reuse

The resulting diagnosis was:

• the later-turn CUDA failure came from missing MTP overhead in the fit budget
• prompt-cache save failures came from large-state pressure
• checkpoint failures came from unguarded checkpoint export and uncontrolled checkpoint memory accumulation
• the large-prompt first-request failure came from the server not adapting its batching logic to the current upstream MTP signaling contract
• the startup failure came from the MTP draft-context reserve/fallback path
• silent host-memory-pressure crashes could also come from an escaped standard exception in the main runtime callbacks
• allocator aborts on similar repeated-reuse workloads could also come from stale GGML meta-buffer cached state surviving reset boundaries

2. Fix Logic

A. Account for MTP draft-context overhead during -fit

The common/ side of the change keeps the dedicated helper in common/fit.cpp and common/fit.h:

• common_get_mtp_ctx_memory_overhead(...)

That helper:

• loads the model with no_alloc = true
• creates a temporary llama_context with ctx_type = LLAMA_CONTEXT_TYPE_MTP
• forces n_rs_seq = 0 for that temporary MTP measurement context
• reads the memory breakdown from the constructed context
• sums only context + compute overhead per device and for host memory

In the current branch state, common/common.cpp uses that helper when all of the following are true:

• fitting is enabled
• draft-mtp is active
• there is no separate draft model

Instead of mutating the user's original fit targets, the code copies params.fit_params_target into a temporary fit_targets vector, adds the measured MTP overhead to the device margins, and passes the adjusted margins into common_fit_params().

Important current caveat:

• the current fit API still consumes per-device margins only
• so the helper's host-memory entry can be enforced only in the host-only case
• for GPU-backed runs the host overhead is logged, but not fully enforced by the current fit API

This leaves enough GPU headroom for the MTP context that the server creates at runtime.

B. Harden prompt-cache save

The prompt-cache path was changed so that prompt_save() now:

• checks the exact byte count returned by llama_state_seq_get_data_ext() for both target and draft state
• discards incomplete cache entries if state export fails
• returns failure instead of leaving a partially initialized cache record behind

Related cache-management changes in server_prompt_cache support that behavior:

• alloc() now focuses on reserving the state buffers first, instead of eagerly cloning the full prompt metadata during allocation
• discard() removes incomplete cache entries when prompt export fails mid-save

This turns prompt-cache save from a crash-prone path into a recoverable path.

C. Avoid duplicating the full checkpoint list during cache save

When a slot is being repurposed, the old code effectively kept two large copies of prompt metadata alive at once:

• the live slot prompt and checkpoints
• the cached prompt and checkpoints being written into RAM

The fix changes this to move prompt metadata into the cache when appropriate, then clear the live slot before reloading. That removes a major temporary RAM spike during slot reuse.

D. Add fail-soft checkpoint export helpers

The server now has dedicated helpers for checkpoint export of target and draft state. These helpers:

• allocate checkpoint buffers explicitly
• catch std::bad_alloc
• validate the exact number of bytes returned by state export
• clear the failed checkpoint buffer and report failure instead of aborting the process

This logic is used for:

• normal prompt-processing checkpoints
• speculative / MTP checkpoint paths

E. Skip invalid checkpoints instead of aborting

Checkpoint creation now builds a temporary checkpoint first and only appends it to the prompt if export succeeds. If export fails, the server logs a warning and skips that checkpoint.

That means the request can continue with less rollback coverage instead of killing the process.

F. Trim checkpoint memory by bytes, not only by count

The count-only checkpoint limit is not enough when each checkpoint can be hundreds of MiB.

The fix adds byte-based trimming in two places:

1. Live slot checkpoints

• before creating a new checkpoint, old checkpoints can be evicted so the next allocation fits the intended budget
• after creating a checkpoint, old checkpoints are dropped until total checkpoint bytes are bounded more realistically

2. Cached prompt checkpoints

• before keeping a prompt in the host prompt cache, its checkpoint list is trimmed using a similar budget

This keeps caching enabled while preventing checkpoint memory from growing far beyond the size of the actual reusable state.

G. Keep speculative decoding functional when checkpoint export fails

The speculative checkpoint path now degrades gracefully:

• if speculative checkpoint export fails, the speculative draft / checkpoint is cleared for that step
• the server continues without using that speculative checkpoint rather than aborting

This keeps draft-mtp usable even under high memory pressure.

H. Adapt speculative prompt handling to current origin/master

Relative to current origin/master, which already offloads MTP draft sampling to the backend when available, the current branch additionally treats speculative MTP prompt hidden-state demand as an output-row requirement in the server batching logic.

The current branch state:

• treats speculative MTP pre-norm needs as output-row needs through server_slot::need_embd() by folding in both common_speculative_need_embd(spec) and common_speculative_need_embd_pre_norm(spec)
• keeps the MTP target pre-norm path on masked rows in common/speculative.cpp
• preserves the masked/unmasked extraction split in src/llama-context.cpp
• uses a bounded n_batch_prompt for speculative prompt prefill
• keeps batch-wide llama_set_embeddings(ctx_tgt, ...) tied to actual embedding requests only

This preserves first-request protection while staying compatible with the current upstream speculative code.

I. Disable pipeline parallelism for LLAMA_CONTEXT_TYPE_MTP

The current branch disables pipeline parallelism up front for LLAMA_CONTEXT_TYPE_MTP in src/llama-context.cpp.

This is scoped narrowly because pipeline parallelism is only an optimization here and it increases memory usage during graph reservation. On this workload, the MTP draft context was the one that entered the failing reserve/retry path, so disabling it for LLAMA_CONTEXT_TYPE_MTP avoids that unstable startup path while leaving non-MTP contexts unchanged.

That is specifically to avoid the startup-time draft-context reserve/fallback failure seen on this workload.

J. Remove the response-queue logging race

The response-queue fix moves the waiting_task_ids.size() debug reads under mutex_results in:

• server_response::add_waiting_task_id()
• server_response::remove_waiting_task_id()

This removes the debug-level data race without changing queue semantics.

K. Add a runtime exception barrier around the main queue callbacks

The current branch now wraps the queue callbacks that drive:

• task scheduling
• slot updates
• sleeping-state transitions

Escaped std::bad_alloc and other standard exceptions in task scheduling or slot updates are now:

• logged explicitly
• converted into task errors when possible
• followed by slot release and prompt clearing for any affected active slots

For parent tasks, the same failure handling is applied to the whole task family so child tasks do not remain waiting indefinitely.

Sleeping-state callback exceptions are logged and terminate the queue loop cleanly instead of unwinding out of the main server loop.

Important caveat:

• this hardening covers escaped standard exceptions
• it does not intercept explicit GGML_ABORT paths or backend-level fatal failures

L. Reset cached state in the GGML meta-buffer reset path

The current branch now hardens ggml_backend_meta_buffer_reset() in ggml/src/ggml-backend-meta.cpp.

The reset path now:

• clears the cached split-state map
• clears the cached simple-tensor map
• calls ggml_reset() on each per-buffer ggml context
• resets each child backend buffer afterwards

This keeps per-reset tensor/context state from leaking into the next graph allocation cycle.

Important caveat:

• this is a coherence fix for repeated reuse of the meta backend
• it does not convert allocator aborts into recoverable runtime errors if some other backend-level failure still occurs

3. Possible Effects On Other Functionality

Fit may choose more conservative settings for draft-mtp

Because MTP draft-context overhead is included in the active fit margins, automatic fitting may choose a smaller context size or otherwise leave more headroom on GPU devices than origin/master.

Effect:

• -fit results for draft-mtp can become more conservative
• this is expected, because the baseline fit result undercounts real runtime memory on this workload
• host overhead is still not fully enforced for GPU-backed runs because the current fit API has only per-device margins
• non-MTP and separate-draft-model configurations should otherwise be unchanged by this logic

Slightly longer startup when -fit and draft-mtp are both used

The helper creates a temporary MTP context to measure overhead before the final fit decision is applied.

Effect:

• startup can do one extra model/context measurement pass in this specific configuration
• this only affects the fitting path, not steady-state generation

More conservative speculative prompt prefill

Because speculative MTP prompt slices are bounded more aggressively than origin/master, prompt-prefill throughput may be somewhat lower on some systems.

Effect:

• very large MTP prompts may be processed in more, smaller prompt batches
• peak host-output pressure is lower
• first-request stability is favored over maximum prompt-prefill throughput

Reduced rollback depth in long sessions

Because checkpoints are now trimmed by bytes as well as by count, long-running sessions may keep fewer old checkpoints than a count-only policy, especially on recurrent / MTP models with large checkpoint payloads.

Effect:

• rollback may fall back to a newer checkpoint and replay a slightly longer suffix
• prompt reuse still works, but some restores may save less work in the extreme long-context case

Graceful degradation instead of hard failure

If checkpoint export or speculative checkpoint export fails, the server now skips the checkpoint.

Effect:

• some requests may lose a reuse optimization for that step
• the server stays alive instead of terminating

This is an intentional tradeoff in favor of stability.

Slightly different cache contents

Cached prompt entries may now retain fewer checkpoints because oversized checkpoint lists are trimmed before the prompt is stored in RAM.

Effect:

• host cache still works
• cross-request or cross-slot resume may sometimes require replaying a longer suffix
• total RAM use is significantly better bounded

MTP startup is less aggressive than origin/master

Because pipeline parallelism is disabled for LLAMA_CONTEXT_TYPE_MTP, the draft MTP context no longer follows the same startup scheduling path as origin/master.

Effect:

• startup avoids the observed reserve/fallback/CUDA-error path on this workload
• draft-context startup and steady-state scheduling may be slightly more conservative than origin/master on multi-GPU systems

Severe host-memory pressure may fail requests instead of killing the server

Because the main queue callbacks now catch escaped standard exceptions, severe host-memory pressure can now degrade into request failure instead of immediate process termination.

Effect:

• escaped std::bad_alloc and other standard exceptions in task scheduling or slot updates are logged and translated into task errors when possible
• affected slots are released and their prompt state is cleared before reuse
• explicit GGML_ABORT and backend-level fatal failures are still outside this guard

Slightly more work on meta-buffer reset paths

Because the GGML meta-buffer reset now clears cached split/tensor state and resets the child ggml contexts, the next graph allocation cycle may have to rebuild that cached metadata.

Effect:

• repeated graph reuse after reset may do a bit more setup work than before
• this is intended to preserve allocator correctness across reset boundaries

No intended change to prompt selection or decoding semantics

This change does not alter:

• slot selection heuristics
• prompt similarity logic
• the main decode path when state export succeeds
• the correctness of existing cache reuse / checkpoint restore logic

The goal is not to change model behavior, but to make the server survive large-state export failures, adapt correctly to the current upstream MTP behavior, and avoid pathological checkpoint or draft-context memory failures.

[TacoTakumi]

Adding a Linux-side data point that may be useful here.

Setup

• llama.cpp 5a4126adc (build 9328, master as of 2026-05-25)
• NVIDIA open kernel module 595.58.03, Linux 6.8.0
• 4-GPU rig: RTX 3090 + RTX 5060 Ti + 2× RTX 4060 Ti
• Model: unsloth/Qwen3.6-27B-MTP-GGUF (UD-Q6_K_XL, 25.6 GB)
• llama-server with --spec-type draft-mtp, --spec-draft-n-max 6,
  -fa on, -ctk q4_0 -ctv q4_0, -c 160000, --mlock, --no-mmap
• Multi-GPU split: --device CUDA0,CUDA3 (3090 + 5060 Ti). With the
  dense 27B sequential layer split, the MTP head and tail layers sit
  on the 5060 Ti (back card).

Symptom

Mid-generation hang on long restored-KV sessions (~44K+ KV). Same
process two crashes in a row. Both times the abort comes out of:

common_speculative_impl_draft_mtp::process
  → llama_get_embeddings_pre_norm
    → llama_context::synchronize
      → ggml_backend_cuda_synchronize  (device 3 = 5060 Ti)

with cudaStreamSynchronize ... the launch timed out and was terminated at ggml-cuda.cu:3235.

dmesg side, both crashes:

NVRM: krcWatchdog_IMPL: RC watchdog: GPU is probably locked!
  Notify Timeout Seconds: 7
NVRM: Xid (PCI:0000:01:00): 8, pid=<pid>, name=llama-server, channel <chan>

PCI:0000:01:00 is the RTX 5060 Ti. Xid 8 (channel locked >7 s,
killed by NVRM RC watchdog) rather than the Xid 43 illegal-memory
that #21564 saw on 5090. Different terminal symptom, same Blackwell
arch, adjacent code path.

Back-card A/B isolation

The single-variable swaps localize this fairly cleanly:

Test	Front	Back	MTP	KV depth reached	Result
Original	3090 (CUDA0)	5060 Ti (CUDA3)	on	~44K	Xid 8
1 — swap back card	3090 (CUDA0)	4060 Ti (CUDA1)	on	93K	clean
2 — same card, no MTP	3090 (CUDA0)	5060 Ti (CUDA3)	off	99.9K	clean

Test 2 used the same Qwen3.6-27B-UD-Q6_K_XL.gguf weights (the
non-MTP variant of the same GGUF family), same device split, same
-c 160000, same -fa on, same q4_0/q4_0 KV — only spec-type
removed. The 5060 Ti was driven to 99.9K KV at ~15.5 t/s steady,
44% util, 95 W, 53 °C through the long tail. No NVRM/Xid.

So the trigger here is specifically: RTX 5060 Ti as back card +
MTP draft + long restored KV. Hardware is not the issue (Test 2),
and there's nothing intrinsic about CUDA3's slot in this layer split
either (Test 1).

Relationship to #21564

#21564 was closed as fixed by driver 595.58.03 (Xid 43 in
flash_attn_stream_k_fixup on RTX 5090). This system already runs
595.58.03 and reproduces an adjacent Xid 8 in the MTP draft path on
the same architecture. Whatever the 595 driver fixed in the FA
kernel selection on Blackwell does not cover the kernel(s) reached
through llama_get_embeddings_pre_norm in
common_speculative_impl_draft_mtp::process.

On the t-jet patch

The patch on t-jet/llama.cpp-jet:fix_mtp_server_instability looks
relevant — particularly the MTP fit-budget accounting, the pipeline-
parallelism-off-for-LLAMA_CONTEXT_TYPE_MTP change, and the
ggml_backend_meta_buffer_reset() cache coherence fix arie-s
proposed in a comment above. I haven't applied it to this checkout
yet; happy to test against any subset if it would help narrow which
piece addresses the back-card-Blackwell signature specifically.

[matthewlai]

I am also hitting this.

[87100.986160] NVRM: krcWatchdog_IMPL: RC watchdog: GPU is probably locked!  Notify Timeout Seconds: 7
[87100.987407] NVRM: GPU at PCI:0000:06:00: GPU-5da51bf6-4f66-f4eb-481f-d41dba3a8342
[87100.987408] NVRM: GPU Board Serial Number: 0
[87100.987408] NVRM: Xid (PCI:0000:06:00): 8, pid=128606, name=llama-server, channel 0x00000006

Also running Qwen3.6-27B, on 2x 5070 Ti. About 50K context, but only happens maybe once in 50 prompts. No caching (I'm benchmarking with random prefill). Also using MTP.

I've tried switching to a very small quant (model and KV) so there's lots of free VRAM, and this still happens. Also happens without MTP.

llama-server -hf unsloth/Qwen3.6-27B-MTP-GGUF:UD-IQ3_XXS \
	-ctk q4_0 -ctv q4_0 \
	-fa 1 -np 1 -ub 128 -ngl 99 \
	-c 100000 \
	 --spec_type draft-mtp --spec-draft-n-max 2

[craftgear]

I have hit the same problem.

I revert back to b9180 and the problem has gone.

[matthewlai]

I went back to the 580 driver and that seems to have fixed it for me, but my workload also changed, so I'm not 100% sure.

[madmax550]

Adding a data point that may help narrow this, because our setup has none of the features
the fixes discussed above target: no --spec-type draft-mtp, no server prompt cache /
checkpoints, no tensor split (single GPU), and not Blackwell. We still hit a hard GPU hang in
the same workload shape — large-prompt prefill on a Qwen3.6 hybrid (Gated DeltaNet +
periodic full-attention) model. So the MTP draft-context and meta-buffer-reset fixes, whatever
they fix, don't cover this whole crash class.

Setup

• Model: Qwen3.6-35B-A3B (qwen35moe, full_attention_interval=4), Q4_K_M GGUF (an
  uncensored finetune, but the base MoE hybrid arch)
• llama.cpp commit 9e3b928f (vendored via llama-cpp-rs 0.1.150), CUDA backend, Linux
• RTX 3080 Ti 12 GB (sm_86), driver 595.45.04 (Xorg shares the GPU)
• Custom Rust server on libllama, stateless: fresh llama_context per request, so every
  request is a full prefill from position 0 — we hit the "full prompt re-processing" trigger
  on request Merging tensors of larger models #1, which is probably why our repro rate is much higher than multi-turn
  llama-server setups
• -fa on, n_batch=8192, n_ubatch=512, MoE experts CPU-offloaded via
  tensor_buft_overrides (equivalent of --n-cpu-moe 30), n_ctx varied 32k–262k across
  incidents, KV cache q4_0 in some incidents and f16 in others

Symptom

7 hard freezes over a week, all during prefill, never during generation. Unlike the Xid 8
reports above, ours is completely silent: no Xid, no kernel panic, no MCE, empty pstore —
journald just stops mid-write and the whole host needs a physical power cycle (journal
"corrupted or uncleanly shut down" + fsck dirty bit on every reboot). Docker's json log shows
the app log dying seconds into prefill each time.

What we've isolated

1. The fused chunked GDN kernel is implicated but is not the root cause. We patched in an
   env toggle that forces fused_gdn_ch = false (falling back to the non-fused
   build_delta_net_chunking path; happy to share the 20-line patch). On the fused path a
   4.4k-token prefill froze reproducibly. Disabling it raised the ceiling — ~9.0–9.5k prefills
   then ran repeatedly clean — but prefills of ~16k, ~17k and ~26k tokens still froze the host.
   So the failure lives in code shared by (or downstream of) both the fused and non-fused
   chunked-GDN prefill paths.
2. KV cache quantization is not required. The 16k/17k freezes were q4_0 K/V; the 26k
   freeze and the one below were f16 K/V.
3. It is not size-deterministic. Most recently (with f16 KV + non-fused path): a
   8,046-token prefill completed and generated fine, and an 8,133-token prefill froze the host
   two minutes later, same process, same config. Failure probability seems to rise with
   prefill size rather than crossing a hard threshold — consistent with the "once in 50
   prompts" report above.
4. Generation is unaffected. After a ~9.4k prefill the same config generated 8,779 tokens
   cleanly. Only prefill (multi-token ubatches through the chunked recurrent path) triggers it.
5. Not general instability. 5 min of gpu-burn tensor-core load is clean, and VRAM had
   headroom at every freeze (e.g. ~10.3/12.3 GB with per-layer accounting, another at 54%).

Given the MoE-hybrid model, no-MTP, no-cache repro, this looks like the same underlying bug as
#22450 (closed by stale-bot, never fixed — the theory there was a position desync between the
recurrent state and the attention KV causing an OOB access), rather than anything specific to
MTP or the server's checkpoint path. Happy to provide full logs, the env-toggle patch, or run
diagnostics on this setup if useful — though each repro costs us a physical power cycle, so
we'd prefer targeted asks over blind bisection.

[madmax550]

A silent (no-Xid) variant of this crash class, and a workaround that stopped it: locking the GPU clocks

We spent a week chasing what looks like a sibling of the crashes in this thread and ended somewhere unexpected but cheap to test, so posting the full arc including the dead ends.

Setup: custom Rust server on libllama (llama.cpp 9e3b928f, CUDA 12.4, Linux), single RTX 3080 Ti 12 GB on a desktop host (Xorg on the same GPU), MoE experts CPU-offloaded via tensor_buft_overrides (the --n-cpu-moe 30 mechanism), -fa on, n_ubatch=512. Stateless server — every request is a full prefill from position 0. No MTP, no speculative decoding, no prompt cache, no multi-GPU.

Symptom: 12 hard freezes of the whole host, always during large-prompt prefill, never during generation. Unlike the Xid 8 reports above, completely silent: no Xid, no panic, no MCE, empty pstore — journald stops mid-write, physical power-cycle required.

What we falsified along the way (each of these looked like the answer for a while):

• The fused chunked GDN kernel — an env-toggle patch forcing the non-fused build_delta_net_chunking path raised the freeze threshold (~4.4k → ~16k tokens at first) but freezes returned at smaller sizes later. Contributing, not root cause.
• KV-cache quantization — froze on q4_0 and on f16.
• Prompt-size caps — there is no safe threshold. Same day, same config: 8,046-token prefill fine, 8,133 froze, 7,114 froze, ~5.8k froze; days earlier 9.0–9.5k ran repeatedly clean. Probability rises with prefill length, no hard line.
• Driver 595 → 580.173.02 — two more silent freezes on 580. (To the earlier comment: we suspect "580 fixed it" observations are the bug's non-determinism — a quiet stretch after any change looks curative. Ours looked cured for four hours after a model swap.)
• The hybrid-GDN architecture itself — swapping to Qwen3-Coder-30B-A3B (plain MoE, no Gated DeltaNet) validated clean at 8.1k×3 and 16.1k prefills, then froze the host a few hours later on a real workload. Both architectures freeze; the common ingredient is the CPU-expert-offload prefill, not GDN.

The observation that cracked it: comparing power draw. A small fully-VRAM-resident model (via Ollama, same box) pulls ~350 W during prompt processing and never crashes. Our offloaded prefill reads "100 % utilization" but draws only ~146 W of a 350 W limit — i.e. the GPU is mostly stalled, running thousands of short kernels separated by PCIe round-trips to the CPU-resident experts. That stall/burst pattern makes the card bounce between boost/P-states continuously, and rapid clock transitions are a known way to wedge a GPU silently (no Xid — same signature as ours). It also predicts the probabilistic size correlation: longer prefill = more transitions.

The workaround:

nvidia-smi -pm 1
nvidia-smi -lgc 1365,1365    # pin SM clock at base; pick per card

I created a service to start with OS to make sure the clock is always pinned between reboots (this is not permanent after reboot)

With clocks pinned (verified flat P2/1365 MHz under load), the workload that had frozen this host roughly every other attempt has run clean since — including the exact prompt sizes that froze it the same day. Honest caveats: one host, a probabilistic bug, and days rather than weeks of soak so far; we'll follow up if it recurs.

Why this might matter for this thread: the common thread across the reports here — MTP draft passes, multi-GPU tensor splits, checkpoint restores, and our CPU-expert offload — is prefill-adjacent phases with lots of cross-device synchronization and short-kernel stalls, exactly the shape that thrashes clock states. #22450's deterministic repro also ran --n-cpu-moe 40. If you're hitting the silent-hang variant (no Xid), nvidia-smi -lgc <base>,<base> is a zero-cost experiment; even for the Xid 8 watchdog cases it may be worth a try. Cost of the workaround: the card idles at its P2 floor (~100 W here) and loses boost headroom.

I hope this will help :)

[taylorsatula]

I tagged this Issue in a PR. I solved the issue. It was a multi-GPU sync issue.