Misc. bug: CUDA illegal memory access on GLM-5.3-Flash (glm5next) long prefill at -ub 2048 (Blackwell/sm_120)

[eexxoo]

Name and Version

version: 0.3.0-dev (build 10749, commit 949f7efb0)

Built from PR #27754 HEAD (glm5next / GLM-5.3-Flash), which is not merged to
master, so this could only be tested on the PR branch. -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120, gcc-13, CUDA 13.0.

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

llama-server

Command line

llama-server -m GLM-5.3-Flash-UD-IQ4_XS-00001-of-00005.gguf \
  -ngl auto -fitt 1500,1500,3500 -fa on -ctk f16 -ctv f16 \
  -c 131072 -b 2048 -ub 2048 --jinja

Problem description & steps to reproduce

A single chat completion with a long prompt (7k+ tokens) crashes with a CUDA illegal
memory access during prefill, around the 2nd–3rd context checkpoint.

-ub 512 (and -ub 128 ) is a full workaround; -ub 2048 crashes
reliably. Independent of the DFlash2 drafter and of --cache-reuse (crashes
with both disabled). Reproduces on both unsloth's UD-IQ4_XS and a self-quant.

Hardware: RTX 5090 + 2× RTX PRO 5000 Blackwell (all sm_120 / compute cap 12.0).

Repro script

import json, urllib.request
URL   = "http://127.0.0.1:8090/v1/chat/completions"
MODEL = "GLM-5.3-Flash"   # adjust to your served model id
# ~7k-token prompt: long enough that h*ub*n_kv crosses 2^31 during prefill
prompt = ("Summarize the following document.\n\n" +
          "Raft uses leader election, log replication, and safety guarantees to "
          "maintain consistency across a cluster of nodes even under partitions "
          "and failures. " * 120)
body = json.dumps({"model": MODEL,
                   "messages": [{"role": "user", "content": prompt}],
                   "max_tokens": 200}).encode()
req = urllib.request.Request(URL, body, {"Content-Type": "application/json"})
try:
    r = urllib.request.urlopen(req, timeout=900)
    print("no crash:", json.load(r)["choices"][0]["message"]["content"][:80])
except Exception as e:
    print("CRASH:", type(e).__name__, e)   # 502/BadGateway => server died mid-prefill

First Bad Commit

PR#27754 HEAD 949f7ef

Relevant log output

Logs

Sep 02 13:09:05 K12.local llama-swap[2575151]: 2026-09-02T13:09:05+01:00 [DEBUG] <glm-5.3-flash-iq4xs> request /v1/chat/completions - start: 35.386890689s, total: 39.982425493s
Sep 02 13:09:05 K12.local llama-swap[2575151]: 2026-09-02T13:09:05+01:00 [DEBUG] Capture 0 compressed and saved: 1283 bytes -> 785 bytes (38.8% compression)
Sep 02 13:09:05 K12.local llama-swap[2575151]: 2026-09-02T13:09:05+01:00 [INFO] Request 127.0.0.1 "POST /v1/chat/completions HTTP/1.1" 200 909 "curl/8.5.0" 39.990856211s
Sep 02 13:09:05 K12.local llama-swap[2575151]: 2026-09-02T13:09:05+01:00 [DEBUG] ProxyManager using local Process for model: glm-5.3-flash-iq4xs
Sep 02 13:09:05 K12.local llama-swap[2575151]: 2026-09-02T13:09:05+01:00 [DEBUG] Matrix: model=glm-5.3-flash-iq4xs already running in set=glm53_flash_abl dsl="glm53abl & t"
Sep 02 13:09:05 K12.local llama-swap[2575151]: 0.40.224.664 I srv  server_strea: conv_id= (empty=1)
Sep 02 13:09:05 K12.local llama-swap[2575151]: 0.40.227.374 I srv    operator(): chat format: peg-native
Sep 02 13:09:05 K12.local llama-swap[2575151]: 0.40.227.654 I slot get_availabl: id  0 | task -1 |  - checking sim = 0.003 (10/3149) > 0.300
Sep 02 13:09:05 K12.local llama-swap[2575151]: 0.40.227.661 I slot get_availabl: id  0 | task -1 | selected slot by LRU, t_last = 1636267350720
Sep 02 13:09:05 K12.local llama-swap[2575151]: 0.40.227.662 I srv  get_availabl: updating prompt cache
Sep 02 13:09:05 K12.local llama-swap[2575151]: 0.40.228.040 I srv   prompt_save:  - saving prompt with length 42, total state size = 147.176 MiB (draft: 0.821 MiB)
Sep 02 13:09:05 K12.local llama-swap[2575151]: 0.40.452.721 I srv          load:  - looking for better prompt, base f_keep = 0.238, f_sim = 0.003
Sep 02 13:09:05 K12.local llama-swap[2575151]: 0.40.452.728 I srv          load:    - prompt with length      42, lcp =      10, f_keep = 0.238, f_sim = 0.003
Sep 02 13:09:05 K12.local llama-swap[2575151]: 0.40.452.732 I srv        update:  - cache state: 1 prompts, 292.915 MiB (limits: 512.000 MiB, 131072 tokens, 131072 est)
Sep 02 13:09:05 K12.local llama-swap[2575151]: 0.40.452.733 I srv        update:    - prompt 0x5812ca44e660:      42 tokens, checkpoints:  1,   292.915 MiB
Sep 02 13:09:05 K12.local llama-swap[2575151]: 0.40.452.735 I srv  get_availabl: prompt cache update took 225.07 ms
Sep 02 13:09:05 K12.local llama-swap[2575151]: 0.40.452.803 I slot launch_slot_: id  0 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> temp-ext -> dist
Sep 02 13:09:05 K12.local llama-swap[2575151]: 0.40.452.812 I slot launch_slot_: id  0 | task -1 | sampler params:
Sep 02 13:09:05 K12.local llama-swap[2575151]:         repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
Sep 02 13:09:05 K12.local llama-swap[2575151]:         dry_multiplier = 0.000, dry_base = 1.750, dry_allowed_length = 2, dry_penalty_last_n = 64
Sep 02 13:09:05 K12.local llama-swap[2575151]:         top_k = 40, top_p = 0.950, min_p = 0.050, xtc_probability = 0.000, xtc_threshold = 0.100, typical_p = 1.000, top_n_sigma = -1.000, temp = 0.700
Sep 02 13:09:05 K12.local llama-swap[2575151]:         mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000, adaptive_target = -1.000, adaptive_decay = 0.900
Sep 02 13:09:05 K12.local llama-swap[2575151]: 0.40.452.816 I slot launch_slot_: id  0 | task 13 | processing task, is_child = 0
Sep 02 13:09:05 K12.local llama-swap[2575151]: 0.40.452.829 I slot   operator(): id  0 | task 13 | new prompt, n_ctx_slot = 131072, n_keep = 0, task.n_tokens = 3149
Sep 02 13:09:05 K12.local llama-swap[2575151]: 0.40.452.834 I slot   operator(): id  0 | task 13 | checking checkpoint with [8, 8] against 10...
Sep 02 13:09:05 K12.local llama-swap[2575151]: 0.40.467.027 I slot   operator(): id  0 | task 13 | restored context checkpoint (pos_min = 8, pos_max = 8, n_tokens = 9, n_past = 9, size = 145.739 MiB)
Sep 02 13:09:05 K12.local llama-swap[2575151]: 0.40.467.032 I slot   operator(): id  0 | task 13 | cached n_tokens = 9, memory_seq_rm [9, end)
Sep 02 13:09:06 K12.local llama-swap[2575151]: 0.40.578.631 I slot create_check: id  0 | task 13 | created context checkpoint 2 of 32 (pos_min = 8, pos_max = 8, n_tokens = 9, size = 145.739 MiB)
Sep 02 13:09:06 K12.local llama-swap[2575151]: 2026-09-02T13:09:06+01:00 [INFO] Request 127.0.0.1 "GET /v1/models HTTP/1.1" 200 1378 "" 58.9µs
Sep 02 13:09:10 K12.local llama-swap[2575151]: 0.45.372.166 I slot print_timing: id  0 | task 13 | prompt processing, n_tokens =   1092, progress = 0.35, t =   4.18 s / 260.94 tokens per second
Sep 02 13:09:10 K12.local llama-swap[2575151]: 0.45.372.169 I slot   operator(): id  0 | task 13 | cached n_tokens = 1101, memory_seq_rm [1101, end)
Sep 02 13:09:10 K12.local llama-swap[2575151]: 0.45.473.026 I slot create_check: id  0 | task 13 | created context checkpoint 3 of 32 (pos_min = 1100, pos_max = 1100, n_tokens = 1101, size = 167.080 MiB)
Sep 02 13:09:11 K12.local llama-swap[2575151]: /llm/llamacpp-src/ggml/src/ggml-cuda/ggml-cuda.cu:107: CUDA error
Sep 02 13:09:11 K12.local llama-swap[2575151]: 0.45.658.726 E CUDA error: an illegal memory access was encountered
Sep 02 13:09:11 K12.local llama-swap[2575151]: 0.45.658.731 E   current device: 1, in function ggml_backend_cuda_synchronize at /llm/llamacpp-src/ggml/src/ggml-cuda/ggml-cuda.cu:2541
Sep 02 13:09:11 K12.local llama-swap[2575151]: 0.45.658.731 E   cudaStreamSynchronize(cuda_ctx->stream())
Sep 02 13:09:11 K12.local llama-swap[2575151]: [New LWP 2576595]
Sep 02 13:09:11 K12.local llama-swap[2575151]: [New LWP 2576594]
Sep 02 13:09:11 K12.local llama-swap[2575151]: [New LWP 2576593]
...
Sep 02 13:09:11 K12.local llama-swap[2575151]: [New LWP 2575690]
Sep 02 13:09:11 K12.local llama-swap[2575151]: [New LWP 2575689]
Sep 02 13:09:11 K12.local llama-swap[2575151]: [Thread debugging using libthread_db enabled]
Sep 02 13:09:11 K12.local llama-swap[2575151]: Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Sep 02 13:09:11 K12.local llama-swap[2575151]: 0x000078b05bf10813 in __GI___wait4 (pid=2576618, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
Sep 02 13:09:11 K12.local llama-swap[2575151]: warning: 30        ../sysdeps/unix/sysv/linux/wait4.c: No such file or directory
Sep 02 13:09:11 K12.local llama-swap[2575151]: #0  0x000078b05bf10813 in __GI___wait4 (pid=2576618, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
Sep 02 13:09:11 K12.local llama-swap[2575151]: 30        in ../sysdeps/unix/sysv/linux/wait4.c
Sep 02 13:09:11 K12.local llama-swap[2575151]: #1  0x000078b05c547853 in ggml_print_backtrace () from /llm/llamacpp-src/build/bin/libggml-base.so.0
Sep 02 13:09:11 K12.local llama-swap[2575151]: #2  0x000078b05c5479fb in ggml_abort () from /llm/llamacpp-src/build/bin/libggml-base.so.0
Sep 02 13:09:11 K12.local llama-swap[2575151]: #3  0x000078b057478e77 in ggml_cuda_error(char const*, char const*, char const*, int, char const*) () from /llm/llamacpp-src/build/bin/libggml-cuda.so.0
Sep 02 13:09:11 K12.local llama-swap[2575151]: #4  0x000078b0574806e8 in ggml_backend_cuda_synchronize(ggml_backend*) () from /llm/llamacpp-src/build/bin/libggml-cuda.so.0
Sep 02 13:09:11 K12.local llama-swap[2575151]: #5  0x000078b05c566c25 in ggml_backend_sched_graph_compute_async () from /llm/llamacpp-src/build/bin/libggml-base.so.0
Sep 02 13:09:11 K12.local llama-swap[2575151]: #6  0x000078b05b5062f1 in llama_context::graph_compute(ggml_cgraph*, bool) () from /llm/llamacpp-src/build/bin/libllama.so.0
Sep 02 13:09:11 K12.local llama-swap[2575151]: #7  0x000078b05b50a45a in llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&) () from /llm/llamacpp-src/build/bin/libllama.so.0
Sep 02 13:09:11 K12.local llama-swap[2575151]: #8  0x000078b05b51162d in llama_context::decode(llama_batch const&) () from /llm/llamacpp-src/build/bin/libllama.so.0
Sep 02 13:09:11 K12.local llama-swap[2575151]: #9  0x000078b05b512b5f in llama_decode () from /llm/llamacpp-src/build/bin/libllama.so.0
Sep 02 13:09:11 K12.local llama-swap[2575151]: #10 0x000078b05c753259 in std::_Function_handler<void (), server_context_impl::decode(int&, int, llama_batch&)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /llm/llamacpp->
Sep 02 13:09:11 K12.local llama-swap[2575151]: #11 0x000078b05c717676 in server_queue::yield_to_queue(std::function<void ()>&&) () from /llm/llamacpp-src/build/bin/libllama-server-impl.so
Sep 02 13:09:11 K12.local llama-swap[2575151]: #12 0x000078b05c767b09 in server_context_impl::decode(int&, int, llama_batch&) () from /llm/llamacpp-src/build/bin/libllama-server-impl.so
Sep 02 13:09:11 K12.local llama-swap[2575151]: #13 0x000078b05c7695a1 in server_context_impl::update_slots() () from /llm/llamacpp-src/build/bin/libllama-server-impl.so
Sep 02 13:09:11 K12.local llama-swap[2575151]: #14 0x000078b05c7182b5 in server_queue::start_loop(long) () from /llm/llamacpp-src/build/bin/libllama-server-impl.so
Sep 02 13:09:11 K12.local llama-swap[2575151]: #15 0x000078b05c6d0000 in llama_server(common_params&, int, char**) () from /llm/llamacpp-src/build/bin/libllama-server-impl.so
Sep 02 13:09:11 K12.local llama-swap[2575151]: #16 0x000078b05c6d1046 in llama_server(int, char**) () from /llm/llamacpp-src/build/bin/libllama-server-impl.so
Sep 02 13:09:11 K12.local llama-swap[2575151]: #17 0x000078b05be2a1ca in __libc_start_call_main (main=main@entry=0x5812a6b29270 <main>, argc=argc@entry=54, argv=argv@entry=0x7ffdd2cc7918) at ../sysdeps/nptl/libc_start_call_main.h:58
Sep 02 13:09:11 K12.local llama-swap[2575151]: warning: 58        ../sysdeps/nptl/libc_start_call_main.h: No such file or directory
Sep 02 13:09:11 K12.local llama-swap[2575151]: #18 0x000078b05be2a28b in __libc_start_main_impl (main=0x5812a6b29270 <main>, argc=54, argv=0x7ffdd2cc7918, init=<optimised out>, fini=<optimised out>, rtld_fini=<optimised out>, stack_e>
Sep 02 13:09:11 K12.local llama-swap[2575151]: warning: 360        ../csu/libc-start.c: No such file or directory
Sep 02 13:09:11 K12.local llama-swap[2575151]: #19 0x00005812a6b292a5 in _start ()
Sep 02 13:09:11 K12.local llama-swap[2575151]: [Inferior 1 (process 2575687) detached]