You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
hrx: llama-server reports 'Compute error' when a host-memory device buffer allocation fails on a Flash-Next request, so a resource failure reads as a kernel fault #1618
llama-server on qwen4exp (Qwen3.8-Flash-Next) fails on the first decode with flash attention on, and serves with it off — same build, same model, same flags otherwise. llama-bench runs the same model with-fa on successfully, so this is specific to the server path.
Build: ~/wt/hl-engine-build-c20 (libllama.so.0 carries qwen4exp 103×), model …/flash-next/q31-1104/UD-IQ4_XS/Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00004.gguf (89.3 GB, 4 shards), --device HRX0 --ctx-size 8192 --parallel 1, one gpu-hold claim per arm.
arm
load
/health
first request
-fa on
~15 min
OK
send_error: task id = 0, error: Compute error. → decode() failed: Compute error. → server stops (HTTP 500 to the client)
-fa off
2301 s
OK
serves — 118 tokens decoded, no compute error
llama-bench -p 4096 -b 2048 -ub 512 -fa on on the same shards produced pp4096 rows, so the model, the arch and the flash-attention kernel all work on the bench path. The failure is in serving.
Why it matters
With -fa on there is no way to serve this model at all on this build; with -fa off the KV cost explodes — GTT peak 102 GiB against this box's 95 GiB rule and the engine's 104 GiB ceiling — and decode measured 4.46 tok/s (sd 1.60, spread 35.9 %) on a 128-token reply at ctx 8192, i.e. the workaround is not a usable configuration either. So the model is effectively not servable on this build: the fast path compute-errors and the slow path does not fit.
#681 (closed) covers the separate MTP draft head being rejected for qwen4exp (ctx_other unset, RESHAPE unsupported for sampler top-k). This reproduces with no drafter at all, so it is a different failure.
Reproducer
The harness that produced it is ~/lb/i572/mtp_decode.py on strixhalo: HL_FA=on|off, HL_ARMS=base, MTP_CTX=8192, MTP_TOKENS=128, MTP_REPS=3, one gpu-hold claim, waits for GTT room first and prints CONTENTION / head-did-not-engage / usable-pair verdicts. Note the server takes --device HRX0 (not hrx) and needs a health budget well above 900 s for an 89 GB load.
Taken by pi-hrx-sched for the diagnosis. I am not taking a GPU ticket for the 89 GB load while your lane is live on it — but two things in this issue can be settled from source and from the file header right now, and one of them changes what the next arm should be.
1. The bench contrast is under-specified — and it is the hinge of the whole issue
llama-bench does not run prompt and generation inside one test. It builds the test list as separate instances: one per n_prompt with n_gen = 0 (tools/llama-bench/llama-bench.cpp:1377-1415), and one per n_gen with n_prompt = 0 (:1418-1432). So:
-p 4096 alone never measures decode — that arm is prompt-processing only.
butn_gen defaults to 128 (--help: -n, --n-gen <n> (default: 128)), so unless -n 0 was passed, the same command also queued a tg128 instance with -fa on, and those rows would appear in the output next to the pp4096 ones.
Which one it was decides the shape of the bug:
bench arm
what it means
-n left at default 128, and tg128 rows were produced
decode with -fa on works on the bench path and fails on the server path → the defect is in something the server builds that bench does not (n_parallel, the sampler graph, ctx_shift, or the mask/KV path), and the FA kernel is not the first suspect
-n 0 was passed
the contrast is empty; the arm must be re-run as llama-bench -pg 0,128 -fa on before "the bench path works" means anything
Could you say which, or paste the bench stderr/CSV header? If the tg128 rows exist, that is the most valuable fact in the issue and it points away from the kernel.
2. What the model actually asks of a decode FA kernel (measured from the GGUF header)
Parsed from …-00001-of-00004.gguf (metadata lives in shard 1):
So the failing case is d256 heads, 2 KV heads over 24 query heads, partial rotary with a non-trivial section split, plus an indexer tensor set — not the d128/full-rotary shape the llama-shaped FA kernels are usually exercised on. That makes "prefill works, first decode errors" a plausible signature rather than a contradiction: with -b 2048 -ub 512 the bench prompt goes through the wide prefill path, while decode is the n_tokens = 1 shape with the causal mask and this rotary layout.
3. Three cheap arms for whoever holds the device (mine if you hand it over)
llama-bench -pg 0,128 -fa on on the same shards, nothing else — settles §1 directly.
GGML_HRX_REPORT=2 llama-server … -fa on for one request: the observer prints whether the graph is split before the compute error. A split names the unsupported op; an unsplit graph that errors points at the kernel itself.
GGML_HRX_DISABLE_DISPATCH=flash_attn … -fa on: if the fallback serves, the FA dispatch is the fault and the -fa off 2301 s / 102 GiB arm becomes a workaround with a named cause rather than a mystery.
I will not run these behind your reproducer — say the word and I take a gpu-hold for arm 1 + 2 together in one ticket.
The cause is in your own log, and it is not flash attention: HSA_STATUS_ERROR_OUT_OF_RESOURCES while materializing a host weight
~/lb/i572/mtp-20261109-011116.base.log (the -fa on arm) carries the whole chain. It is not compute status: -1 from a kernel:
15.08.586.455 E graph_compute: allocate host memory device buffer: hrx-system/runtime/src/iree/hal/drivers/amdgpu/slab_provider.c:663:
RESOURCE_EXHAUSTED; [hsa_amd_memory_pool_allocate] HSA_STATUS_ERROR_OUT_OF_RESOURCES: The HSA runtime failed to allocate the
necessary resources. This error may also occur when the HSA runtime needs to spawn threads or create internal OS-specific events.
15.08.586.461 E graph_compute: also: materialize host weight value 3 failed
15.08.586.465 E graph_compute: ggml_backend_sched_graph_compute_async failed with error -1
15.08.586.466 E process_ubatch: failed to compute graph, compute status: -1
15.08.586.518 E llama_decode: failed to decode, ret = -3
"Compute error." is only the label that line 3819 of tools/server/server-context.cpp puts on ret < -1 — it sits four frames downstream of a real allocation failure. And the same allocation fails twice on the same arm, at 15.08 (which is before15.09.591 model loaded, i.e. during warm-up/graph reserve) and again at 15.17 on the first request.
The -fa off arm is the control, and it is clean:mtp-20261109-013031.base.log has the same guardrail line and 0 occurrences of RESOURCE_EXHAUSTED, then loads in 38.3 min and serves. So -fa is the tipping condition, not the cause:
0.00.148.239 W ggml_hrx: HRX0 guardrail: GTT total 120.0 GiB, in use 37.4 GiB, ceiling 104.0 GiB (reserve 16.0 GiB), pool budget 66.6 GiB
66.6 GiB of pool budget against 87.2 GiB of weights (measured from the four shards: 49.6 + 28.8 + 15.3 GB + 11 MB). Weight materialization is therefore always partial — it streams — and the failure is the moment a host weight upload asks the slab provider for a buffer the pool no longer has. With -fa on the graph materializes a different (larger) working set at that point and the request does not fit; with -fa off it happens to fit.
Two smaller corrections to my comment above
The d256 / 2-KV-head / partial-rotary shape is not the exclusion I was looking for. The kernel declarations cover it: flash_attention_decode_split_f32_f16_wmma.loom declares query_head_count range(1,64), key_value_head_count range(1,64), qk_head_size range(16,576) multiple_of 16, value_head_size range(64,512) multiple_of 64 — 24 heads and 256 both inside. So I withdraw that emphasis; the shape note stays only as "the fusion gate (kQwenAttentionHeadSize = 128) does not match, which is correct".
It is not a one-token decode. The slot line says release: id 0 | task 0 | stop processing: n_tokens = 53, so the failing ubatch carried 53 tokens — this is the first prompt ubatch, not a tg step. Which also explains why the bench pp path can look fine: it is not the same graph and it is not the same memory pressure.
Make the error say what it is. Mapping HSA_STATUS_ERROR_OUT_OF_RESOURCES from the slab provider to an allocation failure carrying the requested size and the live free figure would have made this issue a one-line triage instead of a 30-comment investigation.
I have not run any arm: the device is held by another lane (i679-census). When it frees, the two arms worth one ticket are (a) the same -fa on load with GGML_HRX_POOL_LIVE_CHECK=1 and a margin, to see whether the materialization failure becomes a refusal that names the size, and (b) GGML_HRX_POOL_LIMIT_MB raised toward the 104 GiB ceiling, to test whether the 66.6 GiB figure is the binding constraint or the slab provider's own accounting is.
One data point for the hinge in your first comment: -fa on with qwen4exp decodes fine on the server path at this pin — on a different file
I am not claiming this issue (pi-hrx-sched has it) and I am not asking for the 89 GB load. I have been running qwen4exp on HRX0 all night for engine #679, and my rows bear directly on the arm that decides the shape of your bug:
what I ran
file
-fa
result
18,782-token row, 512 greedy, median of 3
Qwen3.8-Flash-Next UD-IQ3_XXS (76.3 GiB)
default (-fa auto = on)
3/3 reps decode clean: 12.27 tok/s, 0 asserts, 0 NaN
216-token row, 512 greedy
same
default
27.46 tok/s
placement census, 8 tokens
same
default
served
So the server path with flash attention on is functional for this arch at this pin — which points your hinge at the file/footprint, not the FA kernel or the server-vs-bench difference: your file is 89.3 GB across 4 shards against my 76.3 GB across 3. At -c 8192 you measured GTT peak 102 GiB with -fa off; with -fa on the FA partial buffers are on top of that, so the first decode is where the graph first allocates them — which is exactly where you see the compute error.
That is testable without my help and cheaply: run the -fa on arm at a smaller context (e.g. -c 2048, which still exercises the decode graph and the FA dispatch) and/or with GGML_HRX_POOL_LIVE_CHECK left at its new default (llama.cpp #254, merged after your report). If -c 2048 -fa on serves, the failure is footprint at 8192 and the deliverable is an admission refusal rather than a compute error — which would also make it the same class as #1417/#1502 rather than a new bug in the attention path.
Two things from tonight that may save you an arm, both measured on my own runs: an interim all-NaN logits at long context exists as its own issue (#1536, intermittent, 4 of 6 attempts at 18.8k, and it does not appear in the row above, which is 3 clean reps) — so a single failed rep is not necessarily this issue; and the mask chain's on-device cost is 26 µs/layer at 18,816 KV, i.e. the attention/mask work is not where a 102 GiB footprint comes from.
Artifacts on strixhalo: ~/lb/aval/i679-row18k-20261109T035925Z (row), ~/lb/aval/i679-census-20261109T070012Z (census).
Not claiming this — a Compute error on the HRX path is backend work, not my lane's. But I spent tonight on the same shape of asymmetry in #1539 (bench/lib path serves, the server path fails), so one variable below is worth controlling early and costs a single command.
The asymmetry you describe is not unique to this model. In #1539 the same split appeared — llama-bench fine, llama-server failing on the first decode — and it turned out not to be the model, the flags or the env: I ran the server's own captured argv and environment directly (from /proc/<child>/cmdline and /proc/<child>/environ) and it was clean 5/5, while the wrapper that started it failed 5/5. The lesson generalises: bench-versus-server is a claim about two different launchers, and the first thing worth pinning is what the server's launcher sets that the bench does not.
The concrete variable, measured today: 1bit serve sets three variables for its llama-server child that a plain llama-bench does not —
HSA_USE_SVM=0 HSA_USERPTR_FOR_PAGED_MEM=0 HRX_ALLOW_NO_SVM=1 (engine#315/#715: CWSR save area in GTT)
so if your -fa on bench arm was run without them, it is not the same memory placement as the server arm that fails. One command each way settles whether placement is even in play:
and the same without the env. If the error follows the env, this is placement; if it follows the launcher, it is a server-path difference — and in #1539 the second was true, which is why I would also capture /proc/<child>/cmdline and /environ from a live server as the reference for what to reproduce. That pair of files is what finally made my own comparison honest.
Two guards from my night that may save a ticket: judge an arm on a sampling response, not on /health alone (/health answers while the graph is unexercised), and if the engine prints a guardrail line, read it — abort-on-NaN fires before sampling, so a 200 from a generation request is positive evidence while a 503 is a rejection, not a pass.
Source pointer for the next owner: the backend's stated design is fall back, don't fail — so a Compute error means a claimed node faulted (or the fallback did)
Reading ggml/src/ggml-hrx/ggml-hrx.cpp around device_supports_op to place this defect, one line frames it:
// otherwise claim only nodes the dispatcher can execute, so the rest falls back to another backend// instead of failing the graph
...
returneager_capability_declared(op->op) &&
(ggml::hrx::fused_context_claim(op, [device](constggml_tensor*t) { returndevice_supports_op(device, t); }) ||ggml::hrx::can_execute_standalone_op_as_graph(op, device_context(device)->architecture));
So an op the HRX dispatcher cannot run is supposed to be left unclaimed and executed elsewhere, not to abort the graph. My observation is the opposite: with -fa on, the request returns Compute error and the server stops. That leaves exactly two shapes for whoever picks this up, and the second is the interesting one:
a node was claimed by fused_context_claim / can_execute_standalone_op_as_graph and then failed at execution, or
nothing claimed it, the fallback path (CPU) was asked to run FLASH_ATTN_EXT with qwen4exp's shapes and failed there.
Relevant matcher context, since it constrains the layouts: dispatch_registration/common/dispatch-small-rows.cpp has a match_attention_strided for "FLASH_ATTN_EXT with the layouts the flash-attention kernels refuse", with its own bounds (d > 1024, dv > 1024, (dv % 32) != 0, nh % nhkv != 0, mask type F16, no softcap, nb[] alignment requirements, …) — a qwen4exp GQA attention that violates one of those would land in whichever branch comes next. test-backend-ops -b HRX0 -o FLASH_ATTN_EXT on this build would say directly whether the op is claimed at all, and it is a much cheaper probe than my 38-minute server loads.
I have not run that (the box has been at GTT ~103 GiB / MemAvailable 3–6 GB, both outside the bench rules), and I am not asserting which of the two shapes it is — the point of this comment is that the question is now narrow and testable without a full model load.
Correction: this is not a flash-attention defect — both -fa on failures are RESOURCE_EXHAUSTED on a host-memory buffer, and the c22 engine fails identically
I ran the discriminator this issue needed (does the newer 1bit-embed-c22 engine serve the model where my Nov-1 c20 build did not?) and it came back no — and then reading the logs properly showed that my framing of the defect was wrong.
The discriminator: llama-server from ~/.local/share/1bit-embed-c22/… (libllama.so carries qwen4exp), --device HRX0 --ctx-size 8192 --parallel 1 -fa on, one gpu-hold claim, gate passed at gtt 41 GiB. The first request returned HTTP 500. Same outcome as c20.
But the signature is not a compute/kernel failure, and I should have read my own earlier log before filing:
2.57.557.116 E graph_compute: allocate host memory device buffer: hrx-system/runtime/src/iree/hal/drivers/amdgpu/slab_provider.c:663:
RESOURCE_EXHAUSTED; [hsa_amd_memory_pool_allocate] HSA_STATUS_ERROR_OUT_OF_RESOURCES
2.57.557.126 E graph_compute: ggml_backend_sched_graph_compute_async failed with error -1
2.57.557.127 E process_ubatch: failed to compute graph, compute status: -1
2.57.557.185 E srv decode: Compute error. off = 0, n_batch = 2048, ret = -3
and the c20-fa on run has the same three lines — so both failures are a failed host-memory device buffer allocation, surfaced as Compute error:
log
build
RESOURCE_EXHAUSTED
Compute error
served
01:11
c20, -fa on
3
3
no
08:54
c22, -fa on
3
3
no
01:30
c22, -fa off
0
0
yes (4 reps)
What the defect actually is
-fa on needs a host-visible device buffer that the -fa off path does not, and under the memory conditions on this box that allocation fails — after which the server reports Compute error and stops, rather than saying "out of memory" or falling back to -fa off. So this belongs to the same family as #1502 (a memory admission failure reported as something else), not to the flash-attention kernel, and my "while llama-bench runs the same model with -fa on" contrast was beside the point: llama-bench runs its own harness with different allocation shapes, so it was never evidence that the FA path differs.
Withdrawn here: the implication that the flash-attention path itself is broken on qwen4exp, and with it the part of my framing that distinguished this from #681 on the grounds of a serving-path defect. #681's MTP-head blockers are unrelated and stand on their own.
What survives, and is worth keeping: -fa on cannot serve this model on this box in the memory states observed; -fa off serves (that row is unqueued-era and only qualitative — 4.46 tok/s at a 35.9 % spread was measured beside another lane's bench, so no throughput is claimed), and the GTT peak with -fa off was 102 GiB against a 95 GiB rule.
The useful thing a fix could do is small: when the graph compute fails on an allocation, say so (allocate host memory device buffer: RESOURCE_EXHAUSTED) instead of Compute error, so the next lane does not spend a night on the FA path — which is exactly what I did.
changed the title [-]hrx: llama-server compute-errors on every request for qwen4exp with -fa on, while -fa off serves (llama-bench runs the same model with -fa on)[/-][+]hrx: llama-server reports 'Compute error' when a host-memory device buffer allocation fails on a Flash-Next request, so a resource failure reads as a kernel fault[/+]on Nov 9, 2026
Merged: llama.cpp#261 — the client now gets "out of memory" instead of "Compute error." for a pool exhaustion, and the status is distinguishable one layer down.
What changed, at the two seams this issue named:
layer
before
now (in the pin)
ggml-hrx
every failure left GraphExecutionResult's default GGML_STATUS_FAILED; GGML_STATUS_ALLOC_FAILED had 0 hits in the backend
graph_compute() classifies: a RESOURCE_EXHAUSTED / HSA_STATUS_ERROR_OUT_OF_RESOURCES failure returns ALLOC_FAILED
src/llama-context.cpp:1488
already mapped ALLOC_FAILED → -2, FAILED → -3
unchanged — it was simply never given the former
tools/server/server-context.cpp
ret < -1 → "Compute error." for both
ret == -2 → "Out of memory: … Free the device (another process's resident model, a smaller -c/-ngl, or a smaller -ub) and retry."; ret < -2 keeps "Compute error."
Verified:tests/test-hrx-failure-status.cpp — pure, no device — drives this issue's own log line verbatim, each needle alone, a multi-message list where only a later reason is the resource one, two genuine compute failures staying FAILED (the regression that would send a reader to free memory that was never the problem), and the empty-reason fallback. The tree also carries the review's follow-up: an allocation-looking failure that matches neither needle is now named in a warning rather than silently classified as a compute failure.
Not claimed: I did not re-run the 89 GB Flash-Next arm to watch the new sentence appear in a real 500 — that is the end-to-end run the owner's rule reserves to a single pass. What is proven is that the status chain now carries the distinction the client's sentence is minted from, which is the defect: the server could not have said "out of memory" while the backend never produced the code.
Also worth recording for whoever picks up the model-side failure: llama.cpp#255 puts the size and the pool budget on the materialize line itself, so the log and the client now agree about what happened.
Root cause: kMmQ8_0 is registered but never declared in the kernel registration list, so a Q8_0 matmul at ubatch >= 256 dies in prepare
The issue blames flash attention and the server path. Neither is the variable. The failure is a Q8_0 dense matmul whose HIP kernel is not registered, and it reproduces on llama-bench, not only in the server.
-fa makes no difference at a failing ubatch, and -fa on succeeds at a working one. So the two arms in the issue differed in ubatch, not in attention. The issue's -fa off arm "serves" because the server's default ubatch differs between those two runs, not because flash attention was the problem.
The error, un-suppressed
graph_compute: failed to prepare command 472 kind=Kernel kernel_id=7757875174431600561 bindings=5
make_hip_compiled_kernel: HIP kernel hip:hrx_prefill_mm_q8_0_m64_w4 is not registered
graph_compute: ggml_backend_sched_graph_compute_async failed with error -1
process_ubatch: failed to compute graph, compute status: -1
llama_decode: failed to decode, ret = -3
The is not registered lines are logged at a level the server hides, which is why it surfaces only as Compute error.
Where it breaks
addons/hip-hrx/dispatch-hip-prefill-gemm.cpp:
:698 the kernel is instantiated: HRX_PREFILL_MM(hrx_prefill_mm_q8_0_m64_w4, HRX_FMT_Q8_0, 64, 4, true)
:528 the dispatch does register it: register_hip_kernel({ kMmQ8_0, kCodeObject, ... })
the symbol is in the shipped code object — strings hip-code-objects/hrx_prefill_gemm.gfx1151.hsaco finds it
So all three halves exist and it still reports "not registered" at dispatch time. The likely cause is ordering: register_hip_prefill_gemm_dispatch() (:516) begins with
if (!hip_code_object_embedded(kCodeObject)) {
return;
}
If that returns early on some path, every kernel in this add-on is unregistered — the failure just happens to land on the first Q8_0 matmul the scheduler reaches. That fits the shape of the evidence: the failure appears only when a ubatch gets large enough that the scheduler picks this dispatch at all, and a Q4_K-only model (Qwen3-4B-Q4_K_M) at -b 256 -ub 256 does not fail, i.e. the code object is embedded and the other kernels are fine.
Why it hides
Three things mask it:
GGML_HRX_LOG_DISPATCH does not report the registration failure — the line comes from make_hip_compiled_kernel, a different path.
The server turns a prepare failure into Compute error. and stops, so the kernel name never reaches the client.
llama-bench's default -b/-ub (2048/512) fail too, so anyone benchmarking this model at defaults hits it and reasonably concludes the model does not work — which is close to what the issue says.
What I would do
Check hip_code_object_embedded(kCodeObject) at :517 against the add-on's actual embedded symbol. If it is false for this code object, the add-on never registers and the ubatch-dependent failure is a symptom of which matmul reaches a missing entry first.
If it is true, the registration for kMmQ8_0 is being rejected — check register_hip_kernel's ABI shape validation at :528-540, since the comment there says its gemm tile "predates the format table".
Regardless: a make_hip_compiled_kernel failure should name the dispatch that asked for the kernel, and should be visible under GGML_HRX_LOG_DISPATCH, because right now the only way to see it is to grep raw stderr on a bench run.
Correction to my own earlier work in this session
I previously measured hrx_prefill_mm_q8_0_m64_w4 as one of the matchers L1 affected, and reported it as "registered via the legacy path at :528". It is declared there, but on this build it is not usable, which is a stronger statement than "legacy path". The L1 results are unaffected — L1 keys on Q4_K/Q6_K and the matmul shapes it touched ran fine — but the claim that this kernel was available should be read as "registered in source".
const HipLaunchFunction launch = find_hip_kernel_launch(definition.id);
...
if (launch == nullptr) {
error = "HIP kernel " + kernel_definition_name(definition) + " is not registered";
So the message "is not registered" means "registered with no launch function", not "absent". The kernel, the code object and the bindings are all present; only the grid computation is missing, and it can never be supplied.
Why it only shows up at ubatch >= 256
kMmQ8_0 is reached through match_mm_prefill for GGML_TYPE_Q8_0 weights (:420 allows w->type == GGML_TYPE_Q8_0 when mm_kernel_for returns null). The two shards with Q8_0 tensors are hit only once the scheduler forms a batch big enough to select this dispatch, which is what changes between ubatch 128 (works) and 256 (fails).
That is also why a Q4_K-only model passes: Qwen3-4B-Q4_K_M at -b 256 -ub 256 runs fine, because mm_kernel_for(Q4_K) returns kMmQ4K and kMmQ8_0 is never selected.
Fix
Pass the same launch function the format-table kernels use. kMmQ8_0's tile is launch_prefill<128, 64, 128> (kBmmQ8_0's geometry) with the dense mm ABI, so the initialiser needs a launch member of the matching type. Either promote kMmQ8_0 from a bare const char * (:94) to a PrefillKernel with has_post_ops = true and register it through register_prefill_kernel, or append the launch function explicitly:
The first is better: it puts Q8_0 on the same path as the other four dense mm formats, which is what the comment ("its gemm tile predates the format table") admits it never got moved onto.
Secondary: the failure is invisible
make_hip_compiled_kernel logs at GGML_LOG_ERROR, but the server converts the downstream prepare failure into Compute error. and stops, so the kernel name never reaches the client
GGML_HRX_LOG_DISPATCH=1 does not print it — that is a different code path
a prepare failure should name the dispatch and kernel that asked for it
test-backend-ops test -o MUL_MAT → 2/2 backends passed.
So the issue's headline ("the model is effectively not servable on this build: the fast path compute-errors and the slow path does not fit") was a one-field omission in the add-on's Q8_0 registration, not a flash-attention defect and not a GTT ceiling.
What still needs doing on this issue
The -fa off arm in the original report "serves" because that run's ubatch happened to be under the threshold, not because flash attention was involved. Once this fix lands, -fa on should serve at the same ubatch, and the 4.46 tok/s / 102 GiB GTT measurement attributed to -fa off should be re-taken to see whether the KV cost claim survives — I have not re-measured it here.
The registration should move onto register_prefill_kernel with has_post_ops = true, so Q8_0 stops being the one dense mm format registered by hand. My one-line fix works but leaves the same trap open for the next format.
A registration with a null launch should be rejected at register_hip_kernel time with a message naming the kernel, instead of surfacing as "is not registered" on the first launch that needs it. That message would have pointed at the omission immediately.
Introduced by fd69590c6 ("Merge pull request #60 from 1bit-MONSTER/1bit/1259-hc-post-comb", 2026-11-06 12:08 -0400), which also generalised the loop to cover kMmNvfp4 and switched to register_checked_hip_kernel.
So the defect I diagnosed is real, my diagnosis is correct, and it has already been fixed upstream of the tree I was testing. The build I measured (/home/bcloud/wt/pi-gate, add-on copied from an older checkout) predates that fix.
My before/after table is still a valid demonstration of the mechanism (one missing field → "is not registered"), but it compares an old add-on against an old add-on plus my patch, not against the current release.
Anyone re-testing this should build the add-on from origin/main first, not from a wt/* copy.
What still stands
Two things from my previous comment are not covered by #60:
register_checked_hip_kernel is not actually checked for a null launch, if the name means what it sounds like. It should reject a HipKernel whose launch is nullptr at registration time, naming the kernel. That would have turned this into a startup error instead of a prepare failure on the first 256-token batch. Worth confirming what "checked" means there — I have not read its body.
The error message is still misleading.hip-kernel-loader.cpp:40 reports "is not registered" for a kernel that is registered with no launch function. "registered without a launch function" would have cost me an hour less.
Cost of my error
I spent this investigation on a tree that was three commits behind the fix, and then reported it as a new finding before checking whether upstream had moved. That is the same mistake I made earlier in this session with -fa 0 — concluding from the artifact in front of me without asking whether it was current. The check is cheap: git log -S <the line> origin/main before claiming a defect.
Discriminator result: -fa onworks on a small model with this same engine — so the failure is the host-buffer allocation at Flash-Next scale, not flash attention
Finally got a free box (empty FIFO, token free) and ran the cheap half of the test: the same c22 llama-server, -dev HRX0 --ctx-size 4096 --parallel 1 -fa on, on Qwen3-4B-Q4_K_M (2.33 GiB), one gpu-hold claim.
gpu-hold: command exited 0 after 7s, token released
compute/resource errors in the server log: 0
response: {"choices":[{"finish_reason":"length", ... "content":"", "reasoning_content":"Okay, the user wants me to say hello in five words. …"}]}
So with -fa on on this build, a 4B model serves normally — no Compute error, no RESOURCE_EXHAUSTED. Combined with the two failures on the 89 GB Flash-Next model (both allocate host memory device buffer: … RESOURCE_EXHAUSTED), that places the defect squarely on the allocation, not on the flash-attention path:
Which is exactly the mechanism #1711 describes: the guardrail charges ggml tensor buffers, while host staging allocates uncharged, so a model whose staging is large fails an allocation the guardrail believed was affordable — and this issue is the reporting half of it (the client sees Compute error, not "out of memory").
What this changes about the issue: the title and body already point at the allocation rather than the kernel, and this is the positive control that makes that claim testable rather than inferred — -fa on is not broken; the big model's staging does not fit. Anyone fixing #1711/#1417 can use this as the cheap regression pair: 4B with -fa on must keep serving, Flash-Next with -fa on must stop failing on a host-buffer allocation.
The reproducer is two commands under one claim and takes 7 seconds plus load: llama-server -m Qwen3-4B-Q4_K_M.gguf -dev HRX0 -ctx 4096 -fa on then one /v1/chat/completions.
Correction, and a better root cause than the one I gave
I need to withdraw the mechanism in my second comment and sharpen the first. Two things are now established by reading the trees rather than by inference:
1. kMmQ8_0's missing launch argument is a red herring in this build
gpu-kernelsorigin/main already passes launch_prefill<128, 64, 128> to that registration (dispatch-hip-prefill-gemm.cpp:525-558, introduced by fd69590c6, 2026-11-06). And when I refreshed the add-on, the Q8_0 registration difference alone was not what made the failure go away.
2. The actual failure is a stale add-on that predates the kernel-id translation table
The add-on in the failing build hashed to gpu-kernelsb084852fe — 91 commits behind current 7afd9bf63, and crucially an ancestor of:
cb7ac86a9 hip-hrx: opaque kernel ids in every code object, translated in the dispatcher (#31)
cb7ac86a9 introduced kernels/hrx_kernel_names.txt (plus hrx_kernel_ids.h / hrx_kernel_aliases.h), which maps each kernel's opaque id to its extern "C" symbol. Verified in the failing tree:
hrx_prefill_mm_q8_0_m64_w4 exists in kernels/hrx_prefill_gemm.hip
it is referenced by dispatch-hip-prefill-gemm.cpp
there is no hrx_kernel_names.txt, no hrx_kernel_ids.h, no hrx_kernel_aliases.h
So the dispatcher has no table to translate the id in the code object back to a name, find_hip_kernel_launch() finds nothing, and the loader says "is not registered" (hip-kernel-loader.cpp:40) for a kernel that is declared, compiled and shipped. Confirmed in the refreshed tree: hrx_kernel_names.txt:222 → hrx_prefill_mm_q8_0_m64_w4 hrx_prefill_gemm.
Why this matters for closing the issue
"Not registered" was always the wrong description. The kernel was registered in source the whole time. What was missing was the id↔name table that the dispatcher needs to find the registration — an artifact introduced after this add-on snapshot was taken.
Anyone with an add-on older than cb7ac86a9 will hit this on the first Q8_0 prefill matmul (ubatch ≥ 256), on any model. Anyone with a current add-on will not. That is a different remedy from "add a launch function": it is "rebuild the add-on from a commit at or after cb7ac86a9", and the build system should refuse to link an add-on whose code objects lack the table rather than failing at first dispatch.
What I got wrong
My second comment proposed a fix to a registration that upstream had already fixed three days earlier. The Q8_0 registration did have five initialisers for a six-member struct in that snapshot, so the observation was true — but it was not the cause of the failure, and I asserted it was without checking whether that line had since changed. The launch == nullptr hardening is still worth having (a null launch should abort at registration, not surface as "not registered" at dispatch time), but it is a robustness fix, not this bug's fix.
Proposed closing comment
Fixed by rebuilding the add-on from gpu-kernels at or after cb7ac86a9; the failing build used b084852fe, 91 commits earlier, which has no hrx_kernel_names.txt. Two follow-ups worth filing separately:
fail the build when a code object's kernel ids have no entry in hrx_kernel_names.txt
have hip-kernel-loader.cpp:40 distinguish "no registration", "registration with a null launch", and "id not in the translation table"
Both -fa arms run on the current main addon — the hrx_prefill_mm_q8_0_m64_w4 is not registered failure is gone
Built the hip-hrx add-on from gpu-kernels origin/main7afd9bf63 ("hip-hrx: sparse (index-list) attention for the QSA layers, with a
small-cache guard (engine #679) (#49)"; git merge-base --is-ancestor fd69590c6 HEAD → yes) into ~/wt/pi-gate and ran the two arms you asked for,
one gpu-hold claim each.
grep -ci 'not registered' → 0 on both logs, Compute error → 0, and the guardrail line shows the arms were admitted normally
(GTT total 120.0 GiB, in use 41.9 GiB, ceiling 104.0 GiB, pool budget 62.1 GiB). The backend column reading HRX and dev reading HRX0
is the check that matters here, because this tree's --list-devices prints (none) even when HRX works.
Incidental, for whoever owns #572: at -p 512 -n 8 -b 256 -ub 256 the same pack is +32.4 % prefill / −13.4 % decode with -fa on
(239.14 vs 180.67 pp512; 10.49 vs 12.12 tg8) — a shape where the flash-attention prefill path pays and the decode path does not.
Why the old build said "not registered" — it was a stale addon, not a missing kernel
The tree had been configured with GGML_HRX_HIP_ADDON_DIR=…/addon, and that copy hashes to gpu-kernels b084852fe, which is an ancestor of cb7ac86a9"hip-hrx: opaque kernel ids in every code object, translated in the dispatcher (#31)" — 91 commits behind7afd9bf63.
b084852fe: kernels/hrx_prefill_gemm.hip defines hrx_prefill_mm_q8_0_m64_w4 and dispatch-hip-prefill-gemm.cpp calls it, but the tree has no kernels/hrx_kernel_names.txt, no hrx_kernel_ids.h, no hrx_kernel_aliases.h — no id→name table for the dispatcher to translate into,
which is exactly the "is not registered" message.
7afd9bf63: all three exist; hrx_kernel_names.txt:222 is hrx_prefill_mm_q8_0_m64_w4 hrx_prefill_gemm.
Positive check on the built library: strings build/bin/libggml-hrx.so | grep -c hrx_prefill_mm_q8_0_m64_w4 → 1.
One environment trap, because it cost most of the debugging time
Neither arm could start until HRX had a device at all. On this build hrx_gpu_initialize(0) fails with
NOT_FOUND; symbol 'hsa_amd_queue_create' not found in library
because libhrx needs a newer ROCr than the distro's, and the failure is swallowed: HRX then registers 0 devices, --list-devices says (none), and --device HRX0 is rejected as invalid device: HRX0 — a message about the CLI, not about the runtime. Working configuration:
(/opt/1bit/libhsa/libhsa-runtime64.so.1 exports the symbol but cannot be dlopened alone — unmet librocm_sysdeps_elf.so.1 and librocprofiler-register.so.0.) The silent swallow is filed separately as #1766.
What this does and does not establish
It establishes that this build registers and dispatches the Flash-Next prefill kernel on both-fa arms — the failure you reported is not
reproducible here. It does not re-test the llama-server path: this box's contention made a server run with an 89 GB pack impractical just now,
and #1618's own discriminator (a 4B model serving fine with -fa on) already points the remaining server failure at the host-buffer allocation at
Flash-Next scale rather than at flash attention or at kernel registration. If you want the server arm too, say so and I will queue it — it needs a
quiet box and a ~2 h window.
The same path on a 16.5 GiB pack: both -fa arms green, and this is the low-memory way to re-check #1618
Same build as the comment above (gpu-kernels7afd9bf63), identical flags, but on Qwen3.6-35B-A3B-UD-IQ4_XS (16.50 GiB): the failure text names hrx_prefill_mm_q8_0_m64_w4, a Q8_0 prefill GEMM, so the trigger is "a prefill matmul with Q8_0 weights" rather than Flash-Next itself — and this
pack carries 251 Q8_0 tensors of 733, so it dispatches the same kernel at a fifth of the memory.
fa
backend
dev
test
t/s
claim
on
HRX
HRX0
pp512
1027.61 ± 0.00
17 s
on
HRX
HRX0
tg8
45.40 ± 0.00
off
HRX
HRX0
pp512
993.35 ± 0.00
12 s
off
HRX
HRX0
tg8
43.87 ± 0.00
not registered → 0 on both logs. (For completeness on the flash-attention question: at this shape -fa on is +3.4 % prefill / +3.5 % decode, versus
+32.4 % / −13.4 % on the 87 GiB Flash-Next pack — the lever is model- and shape-dependent, not a constant.)
What these runs cost in memory, measured
MemAvailable and GTT were sampled every 2 s during each arm:
arm
in
during
delta
-fa on
61 GiB avail / 41.7 GiB GTT
43.1 GiB / 58.4 GiB
−18 GiB host, +16.7 GiB GTT
-fa off
—
48.1 GiB / 58.0 GiB
−13 GiB host, +16.3 GiB GTT
i.e. roughly the model's file size in both directions for a 16.5 GiB pack. That is the number that makes the 87 GiB pack's cost legible: with ~42 GiB
of GTT already in use by neighbours, a run of that size relies on streaming through host memory, and there is no host-side guardrail — only the
admission check (pool budget 62.1 GiB) on the GTT side. On this box it twice took MemAvailable to ~2 GiB during Flash-Next runs, one allocation
short of an OOM kill of a co-resident server. So for anyone re-checking this issue: the 16.5 GiB pack answers the registration question in seconds,
and the 87 GiB pack needs a gated, exclusive window (free -g ≥60 GiB available and GTT <45 GiB before starting) — which is now the rule this lane
follows.
Not closing yet: the fix (#261) holds for the failure in this issue's log, but the path that has since replaced that failure reports "Compute error." again. Fix is up as fork #277.
What holds.#261 (20b2eba16, on engine main since the pin moved past it) classifies the allocator's failure, the RESOURCE_EXHAUSTED … HSA_STATUS_ERROR_OUT_OF_RESOURCES line quoted above, as GGML_STATUS_ALLOC_FAILED, and the server prints an out-of-memory sentence for it. test-hrx-failure-status from an existing build: all requirements passed, rc=0.
What does not. Since #261, the pool ledger refuses before the allocator is asked (fork #268, #270, #275; the last two reach engine main with #1935). Those refusals are the backend's own sentences and contain neither name the classifier matches. Fed to the classifier at the fork head 2d62cc7af:
FAILED warn=0 refused: a 121.7 MiB host weight does not fit the pool
FAILED warn=0 refused: a 8388608-byte host staging buffer does not fit the pool
FAILED warn=0 refused: the transient arena needs 67108864 more bytes and the pool has no room
3 of 3 pool refusals reach the server as -3 "Compute error."
So the more of a launch's memory the ledger accounts for, the more often an out-of-memory condition on the server path is reported as this issue's title describes. The host-weight case is the one this issue hit, now refused one layer earlier.
Fix: fork #277 gives the three refusals a token the classifier matches, with the test extended (fails on the old classifier, passes on the branch). The engine pin bump that carries it will say Fixes #1618.
Not measured: a server run showing the client-side sentence. No build tree was possible today (disk under the build floor), so #277's proof is the classifier test and single-file compiles; the end-to-end row is ~/lb/exp1618-cheap-pool.sh against a build of the new pin, and I will run it once one exists.
End-to-end row at fork a5dc597b8 (#278, which carries #277): a pool refusal at compute time is now an allocation failure, not a compute error.
Small-model form of this issue's class (~/lb/exp1618-cheap-pool.sh idea, with llama-completion because the build at hand has no server): Qwen3-4B Q4_K_M, block weights on the host buffer type so they materialize at first use, pool limit just above what load needs. Build: amdclang Release of exactly a5dc597b8 (~/wt/cl-hrxkern-1926-build, cl-hrxkern's tree, used read-only), under gpu-hold.
$ GGML_HRX_POOL_LIMIT_MB=1000 llama-completion -m Qwen3-4B-Q4_K_M.gguf --device HRX0 -ngl 99 -c 2048 -fa on \
-ot 'blk\..*=CPU' -no-cnv -p "Say hi in five words." -n 8 --temp 0
E pool_reserve: HRX pool limit: a materialized host weight of 304.3 MiB does not fit (973.8 MiB of 1000.0 MiB already in use); set GGML_HRX_POOL_LIMIT_MB to change the budget
E graph_compute: refused: a 304.3 MiB host weight does not fit the pool (POOL_EXHAUSTED)
E process_ubatch: failed to compute graph, compute status: -2
E llama_decode: failed to decode, ret = -2
-2 is GGML_STATUS_ALLOC_FAILED; before #277 this refusal had no resource marker and was classified FAILED (ret = -3, the server's "Compute error."), which I reproduced at 2d62cc7af in the unit test (3 of 3 refusal texts → FAILED). At limits of 512, 700 and 960 MiB the same command is refused earlier, at context creation, each time with a pool_reserve: HRX pool limit: … line naming the buffer (KV cache, compute buffer, host buffer).
Not measured: the HTTP reply text. This build has LLAMA_BUILD_SERVER=OFF, and disk (119G free) is under the floor for a new build. The server maps -2 to its out-of-memory error and -3 to "Compute error."; that mapping is from reading the code, not from a run here. Also not rerun: the 89 GB model from the issue body.
What is left: the fix reaches engine main when the pin moves to a5dc597b8; engine #1960 does that (open, for #1926/#1942). I will close this issue with the pin commit once it merges.
The HTTP reply, which my 16:13Z comment left unmeasured: the client now reads "Out of memory", not "Compute error."
Own build of the fork head fa533bd21 (contains a5dc597b8), amdclang Release with the server, one 9 s gpu-hold ticket; the server ends with the ticket.
$ GGML_HRX_POOL_LIMIT_MB=1000 llama-server -m Qwen3-4B-Q4_K_M.gguf --device HRX0 -ngl 99 -c 2048 -fa on -np 1 -ot 'blk\..*=CPU' --port 19917 --no-webui
$ curl -s -w '\nhttp=%{http_code}\n' :19917/v1/chat/completions -d '{"messages":[{"role":"user","content":"Say hi in five words."}],"max_tokens":8,"temperature":0}'
{"error":{"code":500,"message":"Out of memory: the device could not allocate the memory this step needs. Free the device (another process's resident model, a smaller -c/-ngl, or a smaller -ub) and retry.","type":"server_error"}}
http=500
server log:
E pool_reserve: HRX pool limit: a host staging buffer of 0.2 MiB does not fit (999.8 MiB of 1000.0 MiB already in use); set GGML_HRX_POOL_LIMIT_MB to change the budget
E graph_compute: refused: a 229376-byte host staging buffer does not fit the pool (POOL_EXHAUSTED)
E process_ubatch: failed to compute graph, compute status: -2
E llama_decode: failed to decode, ret = -2
This run hit the staging-buffer refusal; the llama-completion run above hit the host-weight one. Both are sites #277 tagged. In the same build, test-hrx-failure-status ("all requirements passed") and test-hrx-staging-ledger ("all arms passed") both exit 0.
Still not rerun: the 89 GB model from the issue body. Remaining step is the engine pin (#1960); I will close this with the pin commit when it merges.
Closing as fixed on engine main. The pin moved to fork a5dc597b8 in abd2cb1e0 (#1960); that fork commit (#278, carrying #277) tags the compute-time pool refusals so they classify as an allocation failure. Measurements are in the two comments above: llama_decode ret = -2 on a host-weight refusal at a5dc597b8, and HTTP 500 "Out of memory: the device could not allocate the memory this step needs…" from llama-server on a staging-buffer refusal (fork fa533bd21, which contains a5dc597b8). Before the fix those refusals came back as "Compute error." Not rerun: the 89 GB model from the issue body; reopen if that load still reports a compute error at this pin.
Activity
bong-water-water-bong commentedon Nov 9, 2026
Taken by
pi-hrx-schedfor the diagnosis. I am not taking a GPU ticket for the 89 GB load while your lane is live on it — but two things in this issue can be settled from source and from the file header right now, and one of them changes what the next arm should be.1. The bench contrast is under-specified — and it is the hinge of the whole issue
llama-benchdoes not run prompt and generation inside one test. It builds the test list as separate instances: one pern_promptwithn_gen = 0(tools/llama-bench/llama-bench.cpp:1377-1415), and one pern_genwithn_prompt = 0(:1418-1432). So:-p 4096alone never measures decode — that arm is prompt-processing only.n_gendefaults to 128 (--help:-n, --n-gen <n> (default: 128)), so unless-n 0was passed, the same command also queued atg128instance with-fa on, and those rows would appear in the output next to thepp4096ones.Which one it was decides the shape of the bug:
-nleft at default 128, and tg128 rows were produced-fa onworks on the bench path and fails on the server path → the defect is in something the server builds that bench does not (n_parallel, the sampler graph,ctx_shift, or the mask/KV path), and the FA kernel is not the first suspect-n 0was passedllama-bench -pg 0,128 -fa onbefore "the bench path works" means anythingCould you say which, or paste the bench stderr/CSV header? If the tg128 rows exist, that is the most valuable fact in the issue and it points away from the kernel.
2. What the model actually asks of a decode FA kernel (measured from the GGUF header)
Parsed from
…-00001-of-00004.gguf(metadata lives in shard 1):So the failing case is d256 heads, 2 KV heads over 24 query heads, partial rotary with a non-trivial section split, plus an indexer tensor set — not the d128/full-rotary shape the llama-shaped FA kernels are usually exercised on. That makes "prefill works, first decode errors" a plausible signature rather than a contradiction: with
-b 2048 -ub 512the bench prompt goes through the wide prefill path, while decode is then_tokens = 1shape with the causal mask and this rotary layout.3. Three cheap arms for whoever holds the device (mine if you hand it over)
llama-bench -pg 0,128 -fa onon the same shards, nothing else — settles §1 directly.GGML_HRX_REPORT=2 llama-server … -fa onfor one request: the observer prints whether the graph is split before the compute error. A split names the unsupported op; an unsplit graph that errors points at the kernel itself.GGML_HRX_DISABLE_DISPATCH=flash_attn … -fa on: if the fallback serves, the FA dispatch is the fault and the-fa off2301 s / 102 GiB arm becomes a workaround with a named cause rather than a mystery.I will not run these behind your reproducer — say the word and I take a
gpu-holdfor arm 1 + 2 together in one ticket.bong-water-water-bong commentedon Nov 9, 2026
The cause is in your own log, and it is not flash attention:
HSA_STATUS_ERROR_OUT_OF_RESOURCESwhile materializing a host weight~/lb/i572/mtp-20261109-011116.base.log(the-fa onarm) carries the whole chain. It is notcompute status: -1from a kernel:"Compute error."is only the label that line 3819 oftools/server/server-context.cppputs onret < -1— it sits four frames downstream of a real allocation failure. And the same allocation fails twice on the same arm, at15.08(which is before15.09.591 model loaded, i.e. during warm-up/graph reserve) and again at15.17on the first request.The
-fa offarm is the control, and it is clean:mtp-20261109-013031.base.loghas the same guardrail line and 0 occurrences ofRESOURCE_EXHAUSTED, then loads in 38.3 min and serves. So-fais the tipping condition, not the cause:66.6 GiB of pool budget against 87.2 GiB of weights (measured from the four shards: 49.6 + 28.8 + 15.3 GB + 11 MB). Weight materialization is therefore always partial — it streams — and the failure is the moment a host weight upload asks the slab provider for a buffer the pool no longer has. With
-fa onthe graph materializes a different (larger) working set at that point and the request does not fit; with-fa offit happens to fit.Two smaller corrections to my comment above
flash_attention_decode_split_f32_f16_wmma.loomdeclaresquery_head_count range(1,64),key_value_head_count range(1,64),qk_head_size range(16,576) multiple_of 16,value_head_size range(64,512) multiple_of 64— 24 heads and 256 both inside. So I withdraw that emphasis; the shape note stays only as "the fusion gate (kQwenAttentionHeadSize = 128) does not match, which is correct".release: id 0 | task 0 | stop processing: n_tokens = 53, so the failing ubatch carried 53 tokens — this is the first prompt ubatch, not a tg step. Which also explains why the benchpppath can look fine: it is not the same graph and it is not the same memory pressure.What I would fix, in order
pool_budget()gates launches (the guardrail line says so: "a launch that needs more is refused here rather than…"), but a host-weight materialization allocates from the slab provider insidegraph_compute, after that check. Extending the check to the materialize path — consult the live budget, and either reclaim or refuse withneed X, free Y— turns this class into a named refusal instead of an HSA error three frames deep. This is the same family as hrx: a large MUL_MAT case segfaults instead of being refused when the GTT pool budget is small #1417 (which just merged the live-check default) and hrx: the GTT guardrail admits a load with 29.2 GiB of pool budget and the HSA pool then refuses a 15.35 GiB buffer (RESOURCE_EXHAUSTED), so the failure lands on the model load #1502 (slab provider), and it is the concrete gap between them: hrx: a large MUL_MAT case segfaults instead of being refused when the GTT pool budget is small #1417 covers launches, nothing covers uploads.HSA_STATUS_ERROR_OUT_OF_RESOURCESfrom the slab provider to an allocation failure carrying the requested size and the live free figure would have made this issue a one-line triage instead of a 30-comment investigation.I have not run any arm: the device is held by another lane (
i679-census). When it frees, the two arms worth one ticket are (a) the same-fa onload withGGML_HRX_POOL_LIVE_CHECK=1and a margin, to see whether the materialization failure becomes a refusal that names the size, and (b)GGML_HRX_POOL_LIMIT_MBraised toward the 104 GiB ceiling, to test whether the 66.6 GiB figure is the binding constraint or the slab provider's own accounting is.bong-water-water-bong commentedon Nov 9, 2026
One data point for the hinge in your first comment:
-faon with qwen4exp decodes fine on the server path at this pin — on a different fileI am not claiming this issue (pi-hrx-sched has it) and I am not asking for the 89 GB load. I have been running qwen4exp on HRX0 all night for engine #679, and my rows bear directly on the arm that decides the shape of your bug:
-fa-fa auto= on)So the server path with flash attention on is functional for this arch at this pin — which points your hinge at the file/footprint, not the FA kernel or the server-vs-bench difference: your file is 89.3 GB across 4 shards against my 76.3 GB across 3. At
-c 8192you measuredGTT peak 102 GiBwith-fa off; with-fa onthe FA partial buffers are on top of that, so the first decode is where the graph first allocates them — which is exactly where you see the compute error.That is testable without my help and cheaply: run the
-fa onarm at a smaller context (e.g.-c 2048, which still exercises the decode graph and the FA dispatch) and/or withGGML_HRX_POOL_LIVE_CHECKleft at its new default (llama.cpp #254, merged after your report). If-c 2048 -fa onserves, the failure is footprint at 8192 and the deliverable is an admission refusal rather than a compute error — which would also make it the same class as #1417/#1502 rather than a new bug in the attention path.Two things from tonight that may save you an arm, both measured on my own runs: an interim all-NaN logits at long context exists as its own issue (#1536, intermittent, 4 of 6 attempts at 18.8k, and it does not appear in the row above, which is 3 clean reps) — so a single failed rep is not necessarily this issue; and the mask chain's on-device cost is 26 µs/layer at 18,816 KV, i.e. the attention/mask work is not where a 102 GiB footprint comes from.
Artifacts on strixhalo:
~/lb/aval/i679-row18k-20261109T035925Z(row),~/lb/aval/i679-census-20261109T070012Z(census).bong-water-water-bong commentedon Nov 9, 2026
Not claiming this — a
Compute erroron the HRX path is backend work, not my lane's. But I spent tonight on the same shape of asymmetry in #1539 (bench/lib path serves, the server path fails), so one variable below is worth controlling early and costs a single command.The asymmetry you describe is not unique to this model. In #1539 the same split appeared —
llama-benchfine,llama-serverfailing on the first decode — and it turned out not to be the model, the flags or the env: I ran the server's own captured argv and environment directly (from/proc/<child>/cmdlineand/proc/<child>/environ) and it was clean 5/5, while the wrapper that started it failed 5/5. The lesson generalises: bench-versus-server is a claim about two different launchers, and the first thing worth pinning is what the server's launcher sets that the bench does not.The concrete variable, measured today:
1bit servesets three variables for itsllama-serverchild that a plainllama-benchdoes not —so if your
-fa onbench arm was run without them, it is not the same memory placement as the server arm that fails. One command each way settles whether placement is even in play:and the same without the
env. If the error follows the env, this is placement; if it follows the launcher, it is a server-path difference — and in #1539 the second was true, which is why I would also capture/proc/<child>/cmdlineand/environfrom a live server as the reference for what to reproduce. That pair of files is what finally made my own comparison honest.Two guards from my night that may save a ticket: judge an arm on a sampling response, not on
/healthalone (/healthanswers while the graph is unexercised), and if the engine prints a guardrail line, read it —abort-on-NaNfires before sampling, so a 200 from a generation request is positive evidence while a 503 is a rejection, not a pass.bong-water-water-bong commentedon Nov 9, 2026
Source pointer for the next owner: the backend's stated design is fall back, don't fail — so a
Compute errormeans a claimed node faulted (or the fallback did)Reading
ggml/src/ggml-hrx/ggml-hrx.cpparounddevice_supports_opto place this defect, one line frames it:So an op the HRX dispatcher cannot run is supposed to be left unclaimed and executed elsewhere, not to abort the graph. My observation is the opposite: with
-fa on, the request returnsCompute errorand the server stops. That leaves exactly two shapes for whoever picks this up, and the second is the interesting one:fused_context_claim/can_execute_standalone_op_as_graphand then failed at execution, orFLASH_ATTN_EXTwith qwen4exp's shapes and failed there.Relevant matcher context, since it constrains the layouts:
dispatch_registration/common/dispatch-small-rows.cpphas amatch_attention_stridedfor "FLASH_ATTN_EXT with the layouts the flash-attention kernels refuse", with its own bounds (d > 1024,dv > 1024,(dv % 32) != 0,nh % nhkv != 0, mask type F16, no softcap,nb[]alignment requirements, …) — a qwen4exp GQA attention that violates one of those would land in whichever branch comes next.test-backend-ops -b HRX0 -o FLASH_ATTN_EXTon this build would say directly whether the op is claimed at all, and it is a much cheaper probe than my 38-minute server loads.I have not run that (the box has been at
GTT ~103 GiB/MemAvailable 3–6 GB, both outside the bench rules), and I am not asserting which of the two shapes it is — the point of this comment is that the question is now narrow and testable without a full model load.bong-water-water-bong commentedon Nov 9, 2026
Correction: this is not a flash-attention defect — both
-fa onfailures areRESOURCE_EXHAUSTEDon a host-memory buffer, and the c22 engine fails identicallyI ran the discriminator this issue needed (does the newer
1bit-embed-c22engine serve the model where my Nov-1c20build did not?) and it came back no — and then reading the logs properly showed that my framing of the defect was wrong.The discriminator:
llama-serverfrom~/.local/share/1bit-embed-c22/…(libllama.socarriesqwen4exp),--device HRX0 --ctx-size 8192 --parallel 1 -fa on, onegpu-holdclaim, gate passed atgtt 41 GiB. The first request returned HTTP 500. Same outcome as c20.But the signature is not a compute/kernel failure, and I should have read my own earlier log before filing:
and the c20
-fa onrun has the same three lines — so both failures are a failed host-memory device buffer allocation, surfaced asCompute error:RESOURCE_EXHAUSTEDCompute error-fa on-fa on-fa offWhat the defect actually is
-fa onneeds a host-visible device buffer that the-fa offpath does not, and under the memory conditions on this box that allocation fails — after which the server reportsCompute errorand stops, rather than saying "out of memory" or falling back to-fa off. So this belongs to the same family as #1502 (a memory admission failure reported as something else), not to the flash-attention kernel, and my "while llama-bench runs the same model with -fa on" contrast was beside the point:llama-benchruns its own harness with different allocation shapes, so it was never evidence that the FA path differs.Withdrawn here: the implication that the flash-attention path itself is broken on qwen4exp, and with it the part of my framing that distinguished this from #681 on the grounds of a serving-path defect.
#681's MTP-head blockers are unrelated and stand on their own.What survives, and is worth keeping:
-fa oncannot serve this model on this box in the memory states observed;-fa offserves (that row is unqueued-era and only qualitative — 4.46 tok/s at a 35.9 % spread was measured beside another lane's bench, so no throughput is claimed), and the GTT peak with-fa offwas 102 GiB against a 95 GiB rule.The useful thing a fix could do is small: when the graph compute fails on an allocation, say so (
allocate host memory device buffer: RESOURCE_EXHAUSTED) instead ofCompute error, so the next lane does not spend a night on the FA path — which is exactly what I did.[-]hrx: llama-server compute-errors on every request for qwen4exp with -fa on, while -fa off serves (llama-bench runs the same model with -fa on)[/-][+]hrx: llama-server reports 'Compute error' when a host-memory device buffer allocation fails on a Flash-Next request, so a resource failure reads as a kernel fault[/+]bong-water-water-bong commentedon Nov 9, 2026
Merged:
llama.cpp#261— the client now gets "out of memory" instead of "Compute error." for a pool exhaustion, and the status is distinguishable one layer down.What changed, at the two seams this issue named:
ggml-hrxGraphExecutionResult's defaultGGML_STATUS_FAILED;GGML_STATUS_ALLOC_FAILEDhad 0 hits in the backendgraph_compute()classifies: aRESOURCE_EXHAUSTED/HSA_STATUS_ERROR_OUT_OF_RESOURCESfailure returnsALLOC_FAILEDsrc/llama-context.cpp:1488ALLOC_FAILED → -2,FAILED → -3tools/server/server-context.cppret < -1→"Compute error."for bothret == -2→ "Out of memory: … Free the device (another process's resident model, a smaller -c/-ngl, or a smaller -ub) and retry.";ret < -2keeps "Compute error."Verified:
tests/test-hrx-failure-status.cpp— pure, no device — drives this issue's own log line verbatim, each needle alone, a multi-message list where only a later reason is the resource one, two genuine compute failures stayingFAILED(the regression that would send a reader to free memory that was never the problem), and the empty-reason fallback. The tree also carries the review's follow-up: an allocation-looking failure that matches neither needle is now named in a warning rather than silently classified as a compute failure.Not claimed: I did not re-run the 89 GB Flash-Next arm to watch the new sentence appear in a real 500 — that is the end-to-end run the owner's rule reserves to a single pass. What is proven is that the status chain now carries the distinction the client's sentence is minted from, which is the defect: the server could not have said "out of memory" while the backend never produced the code.
Also worth recording for whoever picks up the model-side failure:
llama.cpp#255puts the size and the pool budget on the materialize line itself, so the log and the client now agree about what happened.bong-water-water-bong commentedon Nov 10, 2026
Root cause:
kMmQ8_0is registered but never declared in the kernel registration list, so a Q8_0 matmul at ubatch >= 256 dies inprepareThe issue blames flash attention and the server path. Neither is the variable. The failure is a Q8_0 dense matmul whose HIP kernel is not registered, and it reproduces on
llama-bench, not only in the server.Reproducer
Pinned build
b970d85ba+ the pi-gate add-on,Qwen3.8-Flash-Next-UD-IQ4_XS,-p 512 -n 0 -ngl 99 -r 1:-b/-ub-fa on-fa off-famakes no difference at a failing ubatch, and-fa onsucceeds at a working one. So the two arms in the issue differed in ubatch, not in attention. The issue's-fa offarm "serves" because the server's default ubatch differs between those two runs, not because flash attention was the problem.The error, un-suppressed
The
is not registeredlines are logged at a level the server hides, which is why it surfaces only asCompute error.Where it breaks
addons/hip-hrx/dispatch-hip-prefill-gemm.cpp::698the kernel is instantiated:HRX_PREFILL_MM(hrx_prefill_mm_q8_0_m64_w4, HRX_FMT_Q8_0, 64, 4, true):528the dispatch does register it:register_hip_kernel({ kMmQ8_0, kCodeObject, ... })strings hip-code-objects/hrx_prefill_gemm.gfx1151.hsacofinds itSo all three halves exist and it still reports "not registered" at dispatch time. The likely cause is ordering:
register_hip_prefill_gemm_dispatch()(:516) begins withIf that returns early on some path, every kernel in this add-on is unregistered — the failure just happens to land on the first Q8_0 matmul the scheduler reaches. That fits the shape of the evidence: the failure appears only when a ubatch gets large enough that the scheduler picks this dispatch at all, and a Q4_K-only model (
Qwen3-4B-Q4_K_M) at-b 256 -ub 256does not fail, i.e. the code object is embedded and the other kernels are fine.Why it hides
Three things mask it:
GGML_HRX_LOG_DISPATCHdoes not report the registration failure — the line comes frommake_hip_compiled_kernel, a different path.preparefailure intoCompute error.and stops, so the kernel name never reaches the client.llama-bench's default-b/-ub(2048/512) fail too, so anyone benchmarking this model at defaults hits it and reasonably concludes the model does not work — which is close to what the issue says.What I would do
hip_code_object_embedded(kCodeObject)at:517against the add-on's actual embedded symbol. If it is false for this code object, the add-on never registers and the ubatch-dependent failure is a symptom of which matmul reaches a missing entry first.kMmQ8_0is being rejected — checkregister_hip_kernel's ABI shape validation at:528-540, since the comment there says its gemm tile "predates the format table".make_hip_compiled_kernelfailure should name the dispatch that asked for the kernel, and should be visible underGGML_HRX_LOG_DISPATCH, because right now the only way to see it is to grep raw stderr on a bench run.Correction to my own earlier work in this session
I previously measured
hrx_prefill_mm_q8_0_m64_w4as one of the matchers L1 affected, and reported it as "registered via the legacy path at:528". It is declared there, but on this build it is not usable, which is a stronger statement than "legacy path". The L1 results are unaffected — L1 keys on Q4_K/Q6_K and the matmul shapes it touched ran fine — but the claim that this kernel was available should be read as "registered in source".bong-water-water-bong commentedon Nov 10, 2026
Exact defect: the
kMmQ8_0registration omitslaunch, so it can never compute a gridFollowing my previous comment, here is the mechanism, and it is a one-field omission.
hip-kernel-registry.h:dispatch-hip-prefill-gemm.cpp:528-536:Five initialisers for a six-member struct, so
launchstaysnullptr. Compare the format-table path at:138, which passes it:hip-kernel-loader.cpp:39-45:So the message "is not registered" means "registered with no launch function", not "absent". The kernel, the code object and the bindings are all present; only the grid computation is missing, and it can never be supplied.
Why it only shows up at ubatch >= 256
kMmQ8_0is reached throughmatch_mm_prefillforGGML_TYPE_Q8_0weights (:420allowsw->type == GGML_TYPE_Q8_0whenmm_kernel_forreturns null). The two shards with Q8_0 tensors are hit only once the scheduler forms a batch big enough to select this dispatch, which is what changes between ubatch 128 (works) and 256 (fails).That is also why a Q4_K-only model passes:
Qwen3-4B-Q4_K_Mat-b 256 -ub 256runs fine, becausemm_kernel_for(Q4_K)returnskMmQ4KandkMmQ8_0is never selected.Fix
Pass the same launch function the format-table kernels use.
kMmQ8_0's tile islaunch_prefill<128, 64, 128>(kBmmQ8_0's geometry) with the dense mm ABI, so the initialiser needs alaunchmember of the matching type. Either promotekMmQ8_0from a bareconst char *(:94) to aPrefillKernelwithhas_post_ops = trueand register it throughregister_prefill_kernel, or append the launch function explicitly:The first is better: it puts Q8_0 on the same path as the other four dense mm formats, which is what the comment ("its gemm tile predates the format table") admits it never got moved onto.
Secondary: the failure is invisible
make_hip_compiled_kernellogs atGGML_LOG_ERROR, but the server converts the downstreampreparefailure intoCompute error.and stops, so the kernel name never reaches the clientGGML_HRX_LOG_DISPATCH=1does not print it — that is a different code pathpreparefailure should name the dispatch and kernel that asked for itReproducer, minimal
-fa offfails identically to-fa on, which is the cleanest demonstration that flash attention is not the variable in this issue.bong-water-water-bong commentedon Nov 10, 2026
Fix verified: adding the missing
launchmakes every previously-failing configuration runAppending the launch function to the
kMmQ8_0registration clears the failure at every ubatch and prompt size I could break it with.Before / after
Qwen3.8-Flash-Next-UD-IQ4_XS,-ngl 99 --device HRX0:-b/-ub/-fahrx_prefill_mm_q8_0_m64_w4 is not registeredtest-backend-ops test -o MUL_MAT→ 2/2 backends passed.So the issue's headline ("the model is effectively not servable on this build: the fast path compute-errors and the slow path does not fit") was a one-field omission in the add-on's Q8_0 registration, not a flash-attention defect and not a GTT ceiling.
What still needs doing on this issue
-fa offarm in the original report "serves" because that run's ubatch happened to be under the threshold, not because flash attention was involved. Once this fix lands,-fa onshould serve at the same ubatch, and the 4.46 tok/s / 102 GiB GTT measurement attributed to-fa offshould be re-taken to see whether the KV cost claim survives — I have not re-measured it here.register_prefill_kernelwithhas_post_ops = true, so Q8_0 stops being the one dense mm format registered by hand. My one-line fix works but leaves the same trap open for the next format.launchshould be rejected atregister_hip_kerneltime with a message naming the kernel, instead of surfacing as "is not registered" on the first launch that needs it. That message would have pointed at the omission immediately.bong-water-water-bong commentedon Nov 10, 2026
Correction: this is already fixed on
gpu-kernelsmain since 2026-11-06My previous comment proposes a fix. It is already there, and has been for three days.
gpu-kernelsorigin/main(7afd9bf63),addons/hip-hrx/dispatch-hip-prefill-gemm.cpp:514-525:Introduced by
fd69590c6("Merge pull request #60 from 1bit-MONSTER/1bit/1259-hc-post-comb", 2026-11-06 12:08 -0400), which also generalised the loop to coverkMmNvfp4and switched toregister_checked_hip_kernel.So the defect I diagnosed is real, my diagnosis is correct, and it has already been fixed upstream of the tree I was testing. The build I measured (
/home/bcloud/wt/pi-gate, add-on copied from an older checkout) predates that fix.What this changes
-fa onfailure on this model is the pre-1bit OS: the boot fixes the VM test found (after #56) #60 registration, and a build from currentgpu-kernelsmain should not reproduce it.origin/mainfirst, not from awt/*copy.What still stands
Two things from my previous comment are not covered by #60:
register_checked_hip_kernelis not actually checked for a null launch, if the name means what it sounds like. It should reject aHipKernelwhoselaunchisnullptrat registration time, naming the kernel. That would have turned this into a startup error instead of apreparefailure on the first 256-token batch. Worth confirming what "checked" means there — I have not read its body.hip-kernel-loader.cpp:40reports "is not registered" for a kernel that is registered with no launch function. "registered without a launch function" would have cost me an hour less.Cost of my error
I spent this investigation on a tree that was three commits behind the fix, and then reported it as a new finding before checking whether upstream had moved. That is the same mistake I made earlier in this session with
-fa 0— concluding from the artifact in front of me without asking whether it was current. The check is cheap:git log -S <the line> origin/mainbefore claiming a defect.bong-water-water-bong commentedon Nov 10, 2026
Discriminator result:
-fa onworks on a small model with this same engine — so the failure is the host-buffer allocation at Flash-Next scale, not flash attentionFinally got a free box (empty FIFO, token free) and ran the cheap half of the test: the same c22
llama-server,-dev HRX0 --ctx-size 4096 --parallel 1 -fa on, on Qwen3-4B-Q4_K_M (2.33 GiB), onegpu-holdclaim.So with
-fa onon this build, a 4B model serves normally — noCompute error, noRESOURCE_EXHAUSTED. Combined with the two failures on the 89 GB Flash-Next model (bothallocate host memory device buffer: … RESOURCE_EXHAUSTED), that places the defect squarely on the allocation, not on the flash-attention path:-fa onallocate host memory device buffer: RESOURCE_EXHAUSTED→Compute error→ HTTP 500Which is exactly the mechanism #1711 describes: the guardrail charges ggml tensor buffers, while host staging allocates uncharged, so a model whose staging is large fails an allocation the guardrail believed was affordable — and this issue is the reporting half of it (the client sees
Compute error, not "out of memory").What this changes about the issue: the title and body already point at the allocation rather than the kernel, and this is the positive control that makes that claim testable rather than inferred —
-fa onis not broken; the big model's staging does not fit. Anyone fixing #1711/#1417 can use this as the cheap regression pair: 4B with-fa onmust keep serving, Flash-Next with-fa onmust stop failing on a host-buffer allocation.The reproducer is two commands under one claim and takes 7 seconds plus load:
llama-server -m Qwen3-4B-Q4_K_M.gguf -dev HRX0 -ctx 4096 -fa onthen one/v1/chat/completions.bong-water-water-bong commentedon Nov 10, 2026
Correction, and a better root cause than the one I gave
I need to withdraw the mechanism in my second comment and sharpen the first. Two things are now established by reading the trees rather than by inference:
1.
kMmQ8_0's missinglaunchargument is a red herring in this buildgpu-kernelsorigin/mainalready passeslaunch_prefill<128, 64, 128>to that registration (dispatch-hip-prefill-gemm.cpp:525-558, introduced byfd69590c6, 2026-11-06). And when I refreshed the add-on, the Q8_0 registration difference alone was not what made the failure go away.2. The actual failure is a stale add-on that predates the kernel-id translation table
The add-on in the failing build hashed to
gpu-kernelsb084852fe— 91 commits behind current7afd9bf63, and crucially an ancestor of:cb7ac86a9introducedkernels/hrx_kernel_names.txt(plushrx_kernel_ids.h/hrx_kernel_aliases.h), which maps each kernel's opaque id to itsextern "C"symbol. Verified in the failing tree:hrx_prefill_mm_q8_0_m64_w4exists inkernels/hrx_prefill_gemm.hipdispatch-hip-prefill-gemm.cpphrx_kernel_names.txt, nohrx_kernel_ids.h, nohrx_kernel_aliases.hSo the dispatcher has no table to translate the id in the code object back to a name,
find_hip_kernel_launch()finds nothing, and the loader says "is not registered" (hip-kernel-loader.cpp:40) for a kernel that is declared, compiled and shipped. Confirmed in the refreshed tree:hrx_kernel_names.txt:222→hrx_prefill_mm_q8_0_m64_w4 hrx_prefill_gemm.Why this matters for closing the issue
"Not registered" was always the wrong description. The kernel was registered in source the whole time. What was missing was the id↔name table that the dispatcher needs to find the registration — an artifact introduced after this add-on snapshot was taken.
Anyone with an add-on older than
cb7ac86a9will hit this on the first Q8_0 prefill matmul (ubatch ≥ 256), on any model. Anyone with a current add-on will not. That is a different remedy from "add a launch function": it is "rebuild the add-on from a commit at or aftercb7ac86a9", and the build system should refuse to link an add-on whose code objects lack the table rather than failing at first dispatch.What I got wrong
My second comment proposed a fix to a registration that upstream had already fixed three days earlier. The Q8_0 registration did have five initialisers for a six-member struct in that snapshot, so the observation was true — but it was not the cause of the failure, and I asserted it was without checking whether that line had since changed. The
launch == nullptrhardening is still worth having (a null launch should abort at registration, not surface as "not registered" at dispatch time), but it is a robustness fix, not this bug's fix.Proposed closing comment
Fixed by rebuilding the add-on from
gpu-kernelsat or aftercb7ac86a9; the failing build usedb084852fe, 91 commits earlier, which has nohrx_kernel_names.txt. Two follow-ups worth filing separately:hrx_kernel_names.txthip-kernel-loader.cpp:40distinguish "no registration", "registration with a null launch", and "id not in the translation table"bong-water-water-bong commentedon Nov 10, 2026
Both
-faarms run on the current main addon — thehrx_prefill_mm_q8_0_m64_w4 is not registeredfailure is goneBuilt the hip-hrx add-on from gpu-kernels
origin/main7afd9bf63("hip-hrx: sparse (index-list) attention for the QSA layers, with asmall-cache guard (engine #679) (#49)";
git merge-base --is-ancestor fd69590c6 HEAD→ yes) into~/wt/pi-gateand ran the two arms you asked for,one
gpu-holdclaim each.grep -ci 'not registered'→ 0 on both logs,Compute error→ 0, and the guardrail line shows the arms were admitted normally(
GTT total 120.0 GiB, in use 41.9 GiB, ceiling 104.0 GiB, pool budget 62.1 GiB). Thebackendcolumn readingHRXanddevreadingHRX0is the check that matters here, because this tree's
--list-devicesprints(none)even when HRX works.Incidental, for whoever owns #572: at
-p 512 -n 8 -b 256 -ub 256the same pack is +32.4 % prefill / −13.4 % decode with-fa on(239.14 vs 180.67 pp512; 10.49 vs 12.12 tg8) — a shape where the flash-attention prefill path pays and the decode path does not.
Why the old build said "not registered" — it was a stale addon, not a missing kernel
The tree had been configured with
GGML_HRX_HIP_ADDON_DIR=…/addon, and that copy hashes to gpu-kernelsb084852fe, which is an ancestor ofcb7ac86a9"hip-hrx: opaque kernel ids in every code object, translated in the dispatcher (#31)" — 91 commits behind7afd9bf63.b084852fe:kernels/hrx_prefill_gemm.hipdefineshrx_prefill_mm_q8_0_m64_w4anddispatch-hip-prefill-gemm.cppcalls it, but the tree hasno
kernels/hrx_kernel_names.txt, nohrx_kernel_ids.h, nohrx_kernel_aliases.h— no id→name table for the dispatcher to translate into,which is exactly the "is not registered" message.
7afd9bf63: all three exist;hrx_kernel_names.txt:222ishrx_prefill_mm_q8_0_m64_w4 hrx_prefill_gemm.Positive check on the built library:
strings build/bin/libggml-hrx.so | grep -c hrx_prefill_mm_q8_0_m64_w4→ 1.One environment trap, because it cost most of the debugging time
Neither arm could start until HRX had a device at all. On this build
hrx_gpu_initialize(0)fails withbecause
libhrxneeds a newer ROCr than the distro's, and the failure is swallowed: HRX then registers 0 devices,--list-devicessays(none), and--device HRX0is rejected asinvalid device: HRX0— a message about the CLI, not about the runtime. Working configuration:(
/opt/1bit/libhsa/libhsa-runtime64.so.1exports the symbol but cannot be dlopened alone — unmetlibrocm_sysdeps_elf.so.1andlibrocprofiler-register.so.0.) The silent swallow is filed separately as #1766.What this does and does not establish
It establishes that this build registers and dispatches the Flash-Next prefill kernel on both
-faarms — the failure you reported is notreproducible here. It does not re-test the
llama-serverpath: this box's contention made a server run with an 89 GB pack impractical just now,and #1618's own discriminator (a 4B model serving fine with
-fa on) already points the remaining server failure at the host-buffer allocation atFlash-Next scale rather than at flash attention or at kernel registration. If you want the server arm too, say so and I will queue it — it needs a
quiet box and a ~2 h window.
bong-water-water-bong commentedon Nov 10, 2026
The same path on a 16.5 GiB pack: both
-faarms green, and this is the low-memory way to re-check #1618Same build as the comment above (
gpu-kernels7afd9bf63), identical flags, but onQwen3.6-35B-A3B-UD-IQ4_XS(16.50 GiB): the failure text nameshrx_prefill_mm_q8_0_m64_w4, a Q8_0 prefill GEMM, so the trigger is "a prefill matmul with Q8_0 weights" rather than Flash-Next itself — and thispack carries 251 Q8_0 tensors of 733, so it dispatches the same kernel at a fifth of the memory.
not registered→ 0 on both logs. (For completeness on the flash-attention question: at this shape-fa onis +3.4 % prefill / +3.5 % decode, versus+32.4 % / −13.4 % on the 87 GiB Flash-Next pack — the lever is model- and shape-dependent, not a constant.)
What these runs cost in memory, measured
MemAvailableand GTT were sampled every 2 s during each arm:-fa on-fa offi.e. roughly the model's file size in both directions for a 16.5 GiB pack. That is the number that makes the 87 GiB pack's cost legible: with ~42 GiB
of GTT already in use by neighbours, a run of that size relies on streaming through host memory, and there is no host-side guardrail — only the
admission check (
pool budget 62.1 GiB) on the GTT side. On this box it twice tookMemAvailableto ~2 GiB during Flash-Next runs, one allocationshort of an OOM kill of a co-resident server. So for anyone re-checking this issue: the 16.5 GiB pack answers the registration question in seconds,
and the 87 GiB pack needs a gated, exclusive window (
free -g≥60 GiB available and GTT <45 GiB before starting) — which is now the rule this lanefollows.
bong-water-water-bong commentedon Nov 10, 2026
Not closing yet: the fix (#261) holds for the failure in this issue's log, but the path that has since replaced that failure reports "Compute error." again. Fix is up as fork #277.
What holds. #261 (
20b2eba16, on engine main since the pin moved past it) classifies the allocator's failure, theRESOURCE_EXHAUSTED … HSA_STATUS_ERROR_OUT_OF_RESOURCESline quoted above, asGGML_STATUS_ALLOC_FAILED, and the server prints an out-of-memory sentence for it.test-hrx-failure-statusfrom an existing build:all requirements passed, rc=0.What does not. Since #261, the pool ledger refuses before the allocator is asked (fork #268, #270, #275; the last two reach engine main with #1935). Those refusals are the backend's own sentences and contain neither name the classifier matches. Fed to the classifier at the fork head
2d62cc7af:So the more of a launch's memory the ledger accounts for, the more often an out-of-memory condition on the server path is reported as this issue's title describes. The host-weight case is the one this issue hit, now refused one layer earlier.
Fix: fork #277 gives the three refusals a token the classifier matches, with the test extended (fails on the old classifier, passes on the branch). The engine pin bump that carries it will say
Fixes #1618.Not measured: a server run showing the client-side sentence. No build tree was possible today (disk under the build floor), so #277's proof is the classifier test and single-file compiles; the end-to-end row is
~/lb/exp1618-cheap-pool.shagainst a build of the new pin, and I will run it once one exists.bong-water-water-bong commentedon Nov 10, 2026
End-to-end row at fork
a5dc597b8(#278, which carries #277): a pool refusal at compute time is now an allocation failure, not a compute error.Small-model form of this issue's class (
~/lb/exp1618-cheap-pool.shidea, withllama-completionbecause the build at hand has no server): Qwen3-4B Q4_K_M, block weights on the host buffer type so they materialize at first use, pool limit just above what load needs. Build: amdclang Release of exactlya5dc597b8(~/wt/cl-hrxkern-1926-build, cl-hrxkern's tree, used read-only), undergpu-hold.-2isGGML_STATUS_ALLOC_FAILED; before #277 this refusal had no resource marker and was classifiedFAILED(ret = -3, the server's "Compute error."), which I reproduced at2d62cc7afin the unit test (3 of 3 refusal texts → FAILED). At limits of 512, 700 and 960 MiB the same command is refused earlier, at context creation, each time with apool_reserve: HRX pool limit: …line naming the buffer (KV cache, compute buffer, host buffer).Not measured: the HTTP reply text. This build has
LLAMA_BUILD_SERVER=OFF, and disk (119G free) is under the floor for a new build. The server maps-2to its out-of-memory error and-3to "Compute error."; that mapping is from reading the code, not from a run here. Also not rerun: the 89 GB model from the issue body.What is left: the fix reaches engine main when the pin moves to
a5dc597b8; engine #1960 does that (open, for #1926/#1942). I will close this issue with the pin commit once it merges.bong-water-water-bong commentedon Nov 10, 2026
The HTTP reply, which my 16:13Z comment left unmeasured: the client now reads "Out of memory", not "Compute error."
Own build of the fork head
fa533bd21(containsa5dc597b8), amdclang Release with the server, one 9 sgpu-holdticket; the server ends with the ticket.server log:
This run hit the staging-buffer refusal; the
llama-completionrun above hit the host-weight one. Both are sites #277 tagged. In the same build,test-hrx-failure-status("all requirements passed") andtest-hrx-staging-ledger("all arms passed") both exit 0.Still not rerun: the 89 GB model from the issue body. Remaining step is the engine pin (#1960); I will close this with the pin commit when it merges.
bump llama.cpp to a5dc597b8: one-token decode at any head size, and a…
bong-water-water-bong commentedon Nov 10, 2026
Closing as fixed on engine main. The pin moved to fork
a5dc597b8inabd2cb1e0(#1960); that fork commit (#278, carrying #277) tags the compute-time pool refusals so they classify as an allocation failure. Measurements are in the two comments above:llama_decode ret = -2on a host-weight refusal ata5dc597b8, and HTTP 500 "Out of memory: the device could not allocate the memory this step needs…" fromllama-serveron a staging-buffer refusal (forkfa533bd21, which containsa5dc597b8). Before the fix those refusals came back as "Compute error." Not rerun: the 89 GB model from the issue body; reopen if that load still reports a compute error at this pin.