Name and Version
version: 0.1.0-dev (build 1, commit 9f0d017)
built with GNU 13.3.0 for Linux x86_64
Full commit: 9f0d017efb4a388bd5c60a27a575c90f20868e51 (2026-08-17)
Operating systems
Linux
GGML backends
CUDA
Hardware
|
|
| GPU |
NVIDIA GeForce RTX 5090 Laptop GPU (GB203M), 24463 MiB, compute capability 12.0 |
| CPU |
Intel Core Ultra 9 275HX |
| Laptop |
HP OMEN MAX 16 (board 8D41) |
| Driver |
595.84 |
| CUDA runtime |
13.2 |
| CUDA toolkit used to build |
13.3, V13.3.73 |
| OS |
Ubuntu 24.04.4 LTS, kernel 7.0.0-29-generic |
Build configuration:
CMAKE_BUILD_TYPE = Release
CMAKE_CUDA_ARCHITECTURES = 120
GGML_CUDA = ON
GGML_CUDA_GRAPHS = ON
GGML_CUDA_FA = ON
GGML_CUDA_FORCE_CUBLAS = OFF
GGML_CUDA_FORCE_MMQ = OFF
GGML_SCHED_MAX_COPIES = 1
Models
Qwen3.8-27B Q4_K_M GGUF (arch qwen35, 27.32B params, 15.65 GiB, hybrid attention + SSM, 64 layers, n_embd_head_k/v = 256), loaded together with the F16 mmproj for vision.
The hang reproduces with two different Q4_K_M builds of this architecture: the stock unsloth Qwen3.8-27B-Q4_K_M.gguf and a third party abliterated build of the same model. So it is not specific to one quantization run or one finetune.
Problem description and steps to reproduce
Under sustained inference load, the GPU stops servicing a command channel. The NVIDIA kernel driver's Robust Channel watchdog declares the GPU locked after a 7 second notify timeout, raises Xid 8, and llama-server aborts inside ggml_abort with a CUDA error.
Setting GGML_CUDA_DISABLE_GRAPHS=1 eliminates the failure completely. Nothing else needs to change.
Command line that reproduces the hang:
llama-server \
--model Qwen3.8-27B-Q4_K_M.gguf \
--mmproj mmproj-F16.gguf \
--host 0.0.0.0 --port 8000 \
-b 2048 -ub 512 -fitt 0 \
--cache-type-k q8_0 --cache-type-v q8_0 \
--cache-ram 32768 --ctx-checkpoints 32 \
--threads 12 --threads-batch 12 \
-ngl 999 --no-mmap --mlock \
--reasoning-preserve \
--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \
--presence-penalty 0.0 --repeat-penalty 1.0 \
-np 1 -lv 4
The exact same command line prefixed with GGML_CUDA_DISABLE_GRAPHS=1 does not hang.
Steps:
- Start
llama-server with the command above.
- Drive it with a real coding agent workload (I used OpenCode) so the slot stays busy with long prompts and long generations.
- Wait. The hang is stochastic, not deterministic. It never happens on the first request.
Failure rate
Across 11 observed hangs, time from server start to hang was:
min 4 minutes
mean 8.6 minutes
max 15 minutes
With GGML_CUDA_DISABLE_GRAPHS=1 the same workload has run 50 minutes and then 32+ minutes without a single hang, on the same boot, same GPU, same model, same client. Assuming the failure rate is unchanged, the probability of surviving 32 minutes by chance is under 3 percent.
A/B evidence, same boot, same hour, same model
| PID |
GGML_CUDA_DISABLE_GRAPHS |
fusion |
Result |
| 4555 |
not set |
enabled |
hang at 16:27:41 |
| 10643 |
not set |
enabled |
hang at 16:37:15 |
| 3568 |
=1 |
disabled |
50 minutes clean, stopped manually |
| 40527 |
=1 |
enabled |
32+ minutes clean and still running |
PID 40527 is the important row. It has kernel fusion enabled and only CUDA graphs disabled, and it is stable. So this is not the fusion path, it is the graph path.
Collateral damage
While llama-server holds the GPU, the X server hits Xid 8 on its own channel too:
NVRM: Xid (PCI:0000:02:00): 8, pid=2434, name=Xorg, channel 0x00000002
This is consistent with a single long running or non terminating kernel blocking the GPU for more than the 7 second watchdog window, starving other channels.
What I ruled out
- Power / TGP. Hangs occur identically at the stock 95 W limit and at the full 175 W (150 W base + 25 W Dynamic Boost). Not a power delivery problem.
- Thermal.
HW Thermal Slowdown counter is 0. Temperatures ranged 67 to 82 C against a 87 C target, 89 C slowdown, 92 C shutdown. No thermal event.
- Hardware fault. No Xid 79, no Xid 63/64, no ECC errors. A clean reboot does not prevent the hang, it recurs 16 minutes into a fresh boot.
- MTP / speculative decoding. Hangs with and without
--spec-type draft-mtp.
- Kernel fusion.
GGML_CUDA_DISABLE_FUSION=1 is not required. With fusion enabled and only graphs disabled the server is stable.
- KV cache quantization.
q8_0 K and V are used in both the hanging and the stable configuration.
First Bad Commit
Not bisected. Present in 9f0d017.
This looks distinct from the two existing Blackwell reports:
Both of those are illegal accesses. What I am seeing is a hang, not an illegal access: the RC watchdog fires on a 7 second timeout and the hardware exception counters stay at zero. Different failure mode, and the workaround is different as well.
Relevant log output
Kernel log
Every hang is preceded by the watchdog line:
kernel: NVRM: krcWatchdog_IMPL: RC watchdog: GPU is probably locked! Notify Timeout Seconds: 7
kernel: NVRM: Xid (PCI:0000:02:00): 8, pid=4555, name=llama-server, channel 0x00000007
Sample sequence over one afternoon:
15:05:41 Xid 8 llama-server channel 0x00000007
15:20:45 Xid 8 llama-server channel 0x00000007
15:24:35 Xid 8 llama-server channel 0x00000007
15:28:33 Xid 8 llama-server channel 0x00000007
15:35:44 Xid 8 llama-server channel 0x00000007
16:24:12 Xid 8 Xorg channel 0x00000002
16:27:41 Xid 8 llama-server channel 0x00000007
16:37:15 Xid 8 llama-server channel 0x00000005
Backtrace A, synchronous, names the graph path
This is the informative one. The error is caught in the frame that launched the kernel, inside the CUDA graph compute path:
#3 ggml_cuda_error(char const*, char const*, char const*, int, char const*)
from libggml-cuda.so.0
#4 rms_norm_mul_f32_cuda(float const*, float const*, float const*, float*, int, int, int, int,
long, long, long, long, long, long, unsigned int, unsigned int, unsigned int, unsigned int,
long, long, long, unsigned int, unsigned int, unsigned int, unsigned int, float, CUstream_st*)
from libggml-cuda.so.0
#5 ggml_cuda_op_rms_norm_fused(ggml_backend_cuda_context&, ggml_tensor*, ggml_tensor*)
from libggml-cuda.so.0
#6 ggml_cuda_try_fuse(ggml_backend_cuda_context*, ggml_cgraph*, int)
from libggml-cuda.so.0
#7 ggml_backend_cuda_graph_compute(ggml_backend*, ggml_cgraph*)
from libggml-cuda.so.0
#8 ggml_backend_sched_graph_compute_async
from libggml-base.so.0
#9 llama_context::graph_compute(ggml_cgraph*, bool)
#10 llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&)
#11 llama_context::decode(llama_batch const&)
#12 llama_decode
#15 server_context_impl::decode(int&, int, llama_batch&)
#16 server_context_impl::update_slots()
#17 server_queue::start_loop(long)
Because CUDA errors are sticky, rms_norm_mul_f32_cuda may be the first call to observe an already dead context rather than the culprit itself. I am reporting the frame as observed, not as a claim about which kernel hangs.
Backtrace B, asynchronous, most common form
Most hangs surface later, at the synchronization point, so no kernel name is available:
#3 ggml_cuda_error(char const*, char const*, char const*, int, char const*)
#4 ggml_backend_cuda_synchronize(ggml_backend*)
#5 ggml_backend_sched_synchronize
#6 llama_context::synchronize()
#7 server_queue::yield_to_queue(std::function<void ()>&&)
#8 server_context_impl::decode(int&, int, llama_batch&)
#9 server_context_impl::update_slots()
#10 server_queue::start_loop(long)
GPU state at the time of the hangs
GPU Current Temp : 82 C
GPU Target Temperature : 87 C
HW Thermal Slowdown : Not Active (counter 0 us)
SW Power Cap : Active (expected, at the 175 W ceiling)
Power draw : 167 W of a 175 W limit
Additional notes
I am happy to run any diagnostic build, apply a patch, bisect, or capture a CUDA_LAUNCH_BLOCKING=1 trace. This machine reproduces the hang within 15 minutes of load, so turnaround is fast.
Name and Version
Full commit:
9f0d017efb4a388bd5c60a27a575c90f20868e51(2026-08-17)Operating systems
Linux
GGML backends
CUDA
Hardware
Build configuration:
Models
Qwen3.8-27B Q4_K_M GGUF (arch
qwen35, 27.32B params, 15.65 GiB, hybrid attention + SSM, 64 layers,n_embd_head_k/v = 256), loaded together with the F16 mmproj for vision.The hang reproduces with two different Q4_K_M builds of this architecture: the stock unsloth
Qwen3.8-27B-Q4_K_M.ggufand a third party abliterated build of the same model. So it is not specific to one quantization run or one finetune.Problem description and steps to reproduce
Under sustained inference load, the GPU stops servicing a command channel. The NVIDIA kernel driver's Robust Channel watchdog declares the GPU locked after a 7 second notify timeout, raises Xid 8, and
llama-serveraborts insideggml_abortwith a CUDA error.Setting
GGML_CUDA_DISABLE_GRAPHS=1eliminates the failure completely. Nothing else needs to change.Command line that reproduces the hang:
The exact same command line prefixed with
GGML_CUDA_DISABLE_GRAPHS=1does not hang.Steps:
llama-serverwith the command above.Failure rate
Across 11 observed hangs, time from server start to hang was:
With
GGML_CUDA_DISABLE_GRAPHS=1the same workload has run 50 minutes and then 32+ minutes without a single hang, on the same boot, same GPU, same model, same client. Assuming the failure rate is unchanged, the probability of surviving 32 minutes by chance is under 3 percent.A/B evidence, same boot, same hour, same model
GGML_CUDA_DISABLE_GRAPHS=1=1PID 40527 is the important row. It has kernel fusion enabled and only CUDA graphs disabled, and it is stable. So this is not the fusion path, it is the graph path.
Collateral damage
While
llama-serverholds the GPU, the X server hits Xid 8 on its own channel too:This is consistent with a single long running or non terminating kernel blocking the GPU for more than the 7 second watchdog window, starving other channels.
What I ruled out
HW Thermal Slowdowncounter is 0. Temperatures ranged 67 to 82 C against a 87 C target, 89 C slowdown, 92 C shutdown. No thermal event.--spec-type draft-mtp.GGML_CUDA_DISABLE_FUSION=1is not required. With fusion enabled and only graphs disabled the server is stable.q8_0K and V are used in both the hanging and the stable configuration.First Bad Commit
Not bisected. Present in
9f0d017.This looks distinct from the two existing Blackwell reports:
flash_attn_stream_k_fixup.mul_mat_q<Q8_0,128>MMA write-back epilogue — intermittent crash; eliminated by removing Q8_0 weights #24399 reports a device exception from an out of range shared memory store inmul_mat_q<Q8_0,128>.Both of those are illegal accesses. What I am seeing is a hang, not an illegal access: the RC watchdog fires on a 7 second timeout and the hardware exception counters stay at zero. Different failure mode, and the workaround is different as well.
Relevant log output
Kernel log
Every hang is preceded by the watchdog line:
Sample sequence over one afternoon:
Backtrace A, synchronous, names the graph path
This is the informative one. The error is caught in the frame that launched the kernel, inside the CUDA graph compute path:
Because CUDA errors are sticky,
rms_norm_mul_f32_cudamay be the first call to observe an already dead context rather than the culprit itself. I am reporting the frame as observed, not as a claim about which kernel hangs.Backtrace B, asynchronous, most common form
Most hangs surface later, at the synchronization point, so no kernel name is available:
GPU state at the time of the hangs
Additional notes
I am happy to run any diagnostic build, apply a patch, bisect, or capture a
CUDA_LAUNCH_BLOCKING=1trace. This machine reproduces the hang within 15 minutes of load, so turnaround is fast.