Skip to content

cuda: fix self-recursive static init in ggml_cuda_tq_mmq_enabled() (hangs every AMD prefill) - #348

Merged
TheTom merged 1 commit into
TheTom:feature/turboquant-kv-cachefrom
jasstrong:pr/fix-tq-mmq-enabled-recursion
Sep 5, 2026
Merged

cuda: fix self-recursive static init in ggml_cuda_tq_mmq_enabled() (hangs every AMD prefill)#348
TheTom merged 1 commit into
TheTom:feature/turboquant-kv-cachefrom
jasstrong:pr/fix-tq-mmq-enabled-recursion

Conversation

@jasstrong

Copy link
Copy Markdown

What

One-line fix: ggml_cuda_tq_mmq_enabled() initialized its cached static const bool from a call to itself.

static bool ggml_cuda_tq_mmq_enabled() {
-    static const bool enabled = ggml_cuda_tq_mmq_enabled();
+    static const bool enabled = getenv("GGML_TQ_MMQ") != nullptr;
    return enabled;
}

Why it matters

The first call re-enters the static's initialization guard from the same thread and waits on it forever: the process sits in futex_wait with the GPU idle. The function is only reached on the AMD MMQ path (a TQ mul_mat with n >= 8 and GGML_TQ_MMQ=1), so on RDNA and CDNA any prefill hangs. NVIDIA builds short-circuit the environment lookup away and never call the function, which is why the GB10 gates, CI and the MUL_MAT 1697/1697 sweep stayed green.

This came in with #337 (commit 439fe67, mine): the review asked for the getenv() to be read once instead of per node, and the cached version called the wrong thing. Sorry about that.

How it showed up

test-backend-ops -o MUL_MAT -p type_a=tq4_1s on an MI210 stops at the eighth case, m=16,n=8,k=256 (the first with n >= 8), every time; the earlier cases are below the MMQ threshold. Backtrace of the stuck thread:

ggml_cuda_tq_mmq_enabled() -> ggml_cuda_mul_mat -> ggml_cuda_graph_evaluate_and_capture -> ggml_backend_cuda_graph_compute

Test

MI210 (gfx90a), GGML_TQ_MMQ=1 GGML_TQ_NATIVE=1:

suite before (1208c59) after (this branch)
-o MUL_MAT -p type_a=tq4_1s hangs at case 8 (m=16,n=8,k=256) 149/149
-o MUL_MAT_ID -p type_a=tq4_1s hangs before the first case 44/44
-o MUL_MAT -p type_a=tq3_1s (not run, same gate) 158/158
-o MUL_MAT_ID -p type_a=tq3_1s (not run, same gate) 44/44

🤖 Generated with Claude Code

https://claude.ai/code/session_01NxP6x5bmUDYFvmouceN2mR

…ed()

The cached GGML_TQ_MMQ lookup initialized its static from a call to itself, so
the first call on the AMD MMQ path (a TQ mul_mat with n >= 8, i.e. any prefill)
re-entered the static's guard from the same thread and waited on it forever.
NVIDIA builds short-circuit the getenv away and never call the function, which
is why the CUDA gates stayed green. Read the environment variable directly.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NxP6x5bmUDYFvmouceN2mR
@TheTom
TheTom merged commit 27d17bd into TheTom:feature/turboquant-kv-cache Sep 5, 2026
7 of 21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants