Repository navigation
Description
Summary
On Qwen3.8-Flash-Next UD-IQ3_XXS (76.3 GiB), our HRX route loses three quarters of its decode rate on a
long context: 9.70 tok/s at an 18,782-token prompt against 31.15 tok/s for a llama.cpp fork built for
gfx1151, on the same GGUF, same requests, same hour. Prefill falls from 74–93 tok/s to 459. Short-prompt
decode is parity (28.6 vs 28.9), which is what makes this a long-context finding rather than a backend
finding.
Numbers
Protocol for every row: 512 greedy tokens (ignore_eos), median of 3, short 216-token / long 18,782-token
prompt, -c 32768 --parallel 1, one gpu-hold claim per arm, strixhalo gfx1151, 22:31–23:20 ADT.
Bold = the two rows being compared.
| arm | config | short tg | short pp | long tg (18.8k) | long pp | GTT peak | peak RSS |
|---|---|---|---|---|---|---|---|
| c1 — our HRX route | house route's own server, --device HRX0 -ngl 99 -np 1 --cache-reuse 256, house defaults, no -fa, no MTP |
28.61 (28.53/28.61/28.67) | 465.7 (warm) | 9.70 (9.23/9.85/9.70) | 74 – 93 | 106.5 GiB | 45.3 GiB |
| b1 — fork, no MTP | -fa on -ctk q8_0 -ctv q4_0, shipped default (its expert cache resolves off at this much device memory) |
28.95 | 336.6 | 31.15 | 459.0 | 50.4 GiB | 28.1 GiB |
b3 — fork, --moe-cache off |
same config as b1 | 31.72 | 350.9 | 31.14 | 465.5 | 50.4 GiB | 28.1 GiB |
| c3 — fork + its cut-vocab MTP head, n=2 | as b1 + drafting | 44.86 | 329.8 | 47.87 | 458.4 | 52.2 GiB | 28.4 GiB |
- Short prompt: 28.6 vs 28.9 — parity. With neither side drafting, the fork is not faster on this box.
- Long prompt: 9.70 vs 31.15 — 3.2×, and prefill 74–93 vs 459 — 5–6×.
- b1 vs b3 differ only in a flag whose value resolved off in both, so their 29–32 t/s spread is run-to-run
noise on this box, not an effect. The c1 numbers are medians of three, spread 9.23–9.85.
The configuration difference, stated so it is auditable
Our arm ran the house route's own server with the argv/env that 1bit serve --device hrx --print-backend
prints (engine hrxls): --device HRX0 -ngl 99 --jinja -c 32768 -np 1 --cache-reuse 256,
GGML_HRX_REPORT=1, IREE_HAL_AMDGPU_LIBHSA_PATH=…, plus --no-strict-device because 1bit refuses
ffn_moe_gate-47 (MUL_MAT_ID, f32, ne=[640,10,0,1]) under strict placement — the house route leaves 2 of
6060 nodes on the CPU for this model. The fork's arms ran -fa on -ctk q8_0 -ctv q4_0.
The -fa on control on our route was never run. Arm d1 (-fa on -ctk q8_0 -ctv q4_0 on c1's exact
server) was killed during its load when the sequence was stopped, so there is no number for "our route with
flash attention at 18.8k". That is the one missing row, and it decides how the 3.2× is attributed:
- if
-fa oncloses most of the gap → the finding is "the house defaults for this model must enable
flash attention at long context", and it is a one-line default, not a backend defect; - if it does not → the long-context attention path (not the backend as a whole, given short-prompt parity)
is where the work is.
What is already excluded by the data above: "the fork is faster" (it is not, at short context, without
drafting), and "the file or the quant is the difference" (same GGUF, byte-identical).
Why this matters
This is the first file where a published long-context number exists for the house route, and it is the
kind of row a user will hit first: a 76 GiB model at a 32k context. It also sits next to #572 (Flash-Next
prefill 285 tok/s on UD-IQ4_XS vs 1,679 for a tuned Vulkan engine), which is the same family at a short
context — different file, different mechanism, and the two should not be conflated: #572's blocker is a CPU
GET_ROWS node and f32 dispatch overhead; this one is decode attention over 18.8k of history.
Reproduce
# house route (one gpu-hold claim)
llama-server -m Qwen3.8-Flash-Next-UD-IQ3_XXS-00001-of-00003.gguf \
--device HRX0 -ngl 99 --jinja -c 32768 -np 1 --cache-reuse 256 --no-strict-device
# env: GGML_HRX_REPORT=1 IREE_HAL_AMDGPU_LIBHSA_PATH=<TheRock libhsa-runtime64.so.1>
# fork, same box, same file, same hour
llama-server ... -fa on -ctk q8_0 -ctv q4_0 # + the fork's own cached build for gfx1151
# client: 512 greedy tokens, ignore_eos, short 216 / long 18,782 tokens, medians of 3
Sampler: peak GTT from /sys/class/drm/card0/device/mem_info_gtt_used at 1 Hz, peak RSS from /proc.
Per-arm server logs, bench.json and the client sample.txt are under ~/lb/aval/*/ on strixhalo.
Next
One row: arm d1 (-fa on -ctk q8_0 -ctv q4_0 on the same server, same protocol). It needs one GPU claim and
about four minutes, and it turns this issue from "3.2× slower" into either a default to change or a kernel
gap to size. Until it exists I am not calling it a backend defect.
Related
- HRX: Qwen3.8-Flash-Next prefill is 287 tok/s on UD-IQ4_XS vs 1,679 for gufo on the same box (and 500-2,400 across published Strix Halo engines) #572 — Flash-Next prefill on UD-IQ4_XS (short context, CPU
GET_ROWSnode); different mechanism. - The same arm produced two
[gfxhub] page fault+ MODE2 GPU reset bursts at GTT 106.5 of 120 GiB; filed as hrx: gfxhub page fault + MODE2 GPU reset when GTT nears the 120 GiB ceiling (Flash-Next UD-IQ3_XXS, 18.8k ctx, 106.5 GiB GTT) #680,
separately to keep this issue about throughput.
Activity
bong-water-water-bong commentedon Oct 31, 2026
Attribution for #679 — long-context decode on HRX (Qwen3.8-Flash-Next UD-IQ3_XXS)
Everything below is measured on strixhalo (gfx1151), medians of 3 unless the row says otherwise, one
gpu-holdclaim per arm, GTT sampled at 1 Hz with a hard 95 GiB abort guard.
1. The "one missing row" cannot be measured as written, and it is not the mechanism
Arm d1 (
-fa on -ctk q8_0 -ctv q4_0on c1's exact server argv) does not run: the server loads (49.7 GiB peakGTT) and the first decode dies with
HRX's only quantized-KV writer (
match_set_rows_addon,ggml/src/ggml-hrx/dispatch_registration/common/dispatch-rope-set-rows.cpp)accepts
GGML_TYPE_Q8_0only, so aq4_0V cache has no writer at all; and the HRX flash-attention matchers requireKV element size
sizeof(ggml_fp16_t), while llama.cpp force-enables FA for a quantized V cache. So the fork's KVconfiguration is doubly unreachable on our route, and
-ctk q8_0 -ctv q4_0would not use the HRX FA kernel even if thewrite existed.
The
-fahalf of the difference is also a non-issue, measured on the same arm at-c 8192, 5.4k prompt, 64 tokens:-fa on-fa(llama.cpp default isauto)-fa offi.e. c1 already ran the HRX FA kernel, and turning it off is both slower and unstable.
2. What actually scales with context: the QSA selection path
Qwen3.8-Flash-Next's full-attention layers are indexer-sparse (QSA,
src/models/qwen4exp.cpp). Whilen_kv <= indexer_top_k + r - 1 = 2051the top-k graph is skipped entirely (dense); above it, every such layer, everystep, runs: indexer gather over the whole indexer KV -> block pooling ->
ggml_top_k(2051 of n_kv)-> a full-n_kvmask rebuild ->
flash_attn_ext.One server, one process, decode 64 greedy tokens, median of 3 (
-c 8192, f16 KV, no-faoverride):the 103 ms (9.70 tok/s) that c1 measured there -- the whole gap to the fork's flat 31.15;
LLAMA_QSA_DENSE=0) at fixed n_kv ~1,480 costs 13x prefill(676 -> 52.1 tok/s).
GGML_OP_TOP_Kis the only op in that path with no HRX dispatch: it is absent fromeager_capability_declared()(
ggml-hrx.cpp), no matcher registers for it, sodevice_supports_opis false, the scheduler puts it on the CPUbackend and splits every full-attention layer's graph around it. The CPU implementation is a single-row
std::partial_sortwith an indirect comparator (ggml-cpu/ops.cpp), and QSA has one row per layer per step.3. The fix and what it buys
PR 1bit-MONSTER/llama.cpp#161:
GGML_OP_TOP_Kon HRX as a radix select on a monotone uint32 key map (three levelsof 11/11/10 bits to the exact rank-k key, then one emit pass), one workgroup per row, 8 KiB LDS histogram, no global
scratch, no host round trip; plus a bounded claim in
device_supports_opso shapes the kernel cannot take keep theirCPU fallback.
test-backend-ops -o TOP_K -b HRX0: 484/484 withggml_hrx dispatch: hip.top_k_f32 (single), and theGGML_HRX_DISABLE_DISPATCH=hip.top_k_f32control degrades to the CPU backend instead of failing the graph.Long-context rows with the kernel, same binary, same pin, paired against
GGML_HRX_DISABLE_DISPATCH=hip.top_k_f32(identical to the pre-PR behaviour):
hip.top_k_f32)GGML_HRX_DISABLE_DISPATCH=hip.top_k_f32, = pre-PR behaviour)Those rows are queued behind a release-gate window on the box (a 76.3 GiB Flash-Next load needs ~88+ GiB of
MemAvailable and the box is currently hosting other work), together with the 18,782-token protocol row (512 greedy
tokens, median of 3,
-c 19456, 95 GiB GTT cap). I will post them here as a follow-up rather than guess.What is measured for the kernel so far, at the op level (
test-backend-ops):-o TOP_K -b HRX0: 484/484,ggml_hrx dispatch: hip.top_k_f32 (single)in the dispatch log, and theGGML_HRX_DISABLE_DISPATCHcontrol falls back to the CPU backend instead of failing the graph;perf -o TOP_K -b HRX0: 106 us (n=1000), 230 us (n=65,000), 623 us (n=200,000) per single row -- so the QSA decodeshape (12 layers, one row each, n=n_kv) should cost ~1.5-2 ms per step against the ~49 ms the CPU path costs at
18.8k. The fixed part is the serial 2048-bucket scan in the radix select, which the GEAK loop will attack next.
4. Not the cause (so no one re-does these)
in any server log of these runs.
was missing.
MUL_MAT_IDexpert nodes (--no-strict-device): they are per step, not per layer, and have nocontext term.
5. Residual, after the kernel
The O(n_kv) mask rebuild that follows the selection (a
[1,n_kv]f32zeros+set_rows+add, then a cast for FA)is still per-layer-per-step work that grows with context, on HRX but serial. If the numbers above leave a slope, that
is where it is, and the next step is either a fused
qsa_mask_builddispatch or an FA variant that reads the selectedindices directly instead of rebuilding a dense mask.
bong-water-water-bong commentedon Oct 31, 2026
Follow-up on #679 — the long-context rows with the fix in (and what it does not fix)
Same box (strixhalo, gfx1151), same model (Qwen3.8-Flash-Next UD-IQ3_XXS), one
gpu-holdclaim per arm, GTT sampled at1 Hz under a 95 GiB abort guard, 18,782-token prompt, 512 greedy tokens,
ignore_eos, median of 3.The fix: fork PR 1bit-MONSTER/llama.cpp#161 —
GGML_OP_TOP_Kon HRX (radix select on a monotone uint32 key map ofthe f32 bit pattern, one workgroup per row, 8 KiB LDS histogram, no global scratch, no host round trip) plus a bounded
claim in
device_supports_op.test-backend-ops -o TOP_K -b HRX0: 484/484 withggml_hrx dispatch: hip.top_k_f32 (single), andGGML_HRX_DISABLE_DISPATCH=hip.top_k_f32degrades to the CPU backendinstead of failing the graph — which is what makes a clean pre/post comparison possible on one binary.
The rows
Both rows: same binary, same pin (4a58da9b0 = hrx-base + #159 + #160 + #161), same flags
(
-c 19456 --no-repack, c1's argv otherwise). The only difference is the environment variable.GGML_HRX_DISABLE_DISPATCH=hip.top_k_f32(pre-PR behaviour)hip.top_k_f32)-c 32768, no--no-repack)(the disabled row has two long reps rather than three: I cut it short to hand the GPU to two lanes already queued behind
my claim; the two agree to 2%, and the enabled row's three agree to 3%.)
What the dispatch buys, measured on one binary
Prefill is the win, and it is the expected one: the QSA layers call TOP_K once per layer per ubatch with one row per
token, so prefill was 512 serial
std::partial_sorts on 4 CPU threads (with a graph split around every layer); on deviceit is 512 workgroups. Long-context prefill now sits at 389 t/s, i.e. within reach of the gfx1151 fork's 459 rather than
5–6x under it.
Decode at 18.8k only moves 1.16x, so the selection was not where decode time went. What remains is the rest of the
QSA path, all O(n_kv) per full-attention layer per step: the indexer
get_rowsover the whole indexer cache, theblock→cell expand, and the
[1, n_kv]f32 mask rebuild (zeros+set_rows+add, then a cast for FA) ahead of adense FA over all n_kv. Consistently, the 2051-token cliff survives with the kernel in (26.86 → 21.55 tok/s across it in
the same run): crossing 2051 switches the whole path on, not just the sort.
Correction on the memory side (#680)
I expected the ~24 GiB long-prompt GTT transient (c1: 106.5 GiB) to be the per-layer graph splits, and therefore to
disappear with the dispatch. It does not: the disabled row on the new pin/flags also peaks at 84.17 GiB. So the kernel
has no GTT effect (83.82 vs 84.17, noise), and the 106.5 → 83.8 GiB difference comes from the pin move and/or
-c 19456+--no-repack, not from this PR. The operationally useful fact for #680 stands either way: at-c 19456 --no-repackon the current pin, this model's 18.8k row never exceeds 84.2 GiB, ~11 GiB clear of the 95 GiBrule, with or without the kernel.
Next, in order
qsa_mask_builddispatch or an FA variant that consumes the selected indices directly instead of rebuilding a densemask.
scan / warp-aggregated histogram / 8-bit levels.
hyperloom custom: size the long-context prompt by what the tokeniser …
hyperloom custom: a long-context decode leg, so the tuner can score e…
bong-water-water-bong commentedon Nov 1, 2026
TOP_Kon HRX is merged on the fork (#161→199165047), but this issue stays open for two reasons, both stated so it is not closed on the merge:mainpinsthird_party/llama.cpp=4a58da9b0, 8 commits behind the fork tip199165047, and there is no open bump PR (#759→37ca4d442was closed as superseded at 15:54:53Z).HSA_USE_SVM=0 HSA_USERPTR_FOR_PAGED_MEM=0 HRX_ALLOW_NO_SVM=1faults on its first graph replay (hrx: gfxhub permission page faults (UTCL2 client CPF) on the rebuilt box with the TheRock nightly libhsa — the no-SVM env trio is the only fault-free arm #715,gfxhub … UTCL2 client CPF), andtest-backend-opshangs on the fault. No fault-free measurement of the 18.8k row exists yet.Closes on: the pin bump, then arm d1 (
-fa on -ctk q8_0 -ctv q4_0) at 18.8k, memory-capped and alone, fault delta 0, beating the 28.43 tok/s short-prompt baseline.bong-water-water-bong commentedon Nov 2, 2026
Follow-up: the TOP_K kernel from #161 now has a 2.38x geomean improvement (op-level), branch
1bit/hrx-top-k-coop-scanready.The dispatch fixed the cliff and the prefill gap, but it left one measured defect: inside the radix select the descending
bucket scan ran serial in lane 0 — 3 levels x 2,048 buckets = 6,144 dependent LDS reads plus integer work,
independent of
n. That is the ~100 us fixed costtest-backend-ops perf -o TOP_K -b HRX0shows atn=1000, and it ispaid by all 12 full-attention layers on every decode step.
Each lane now owns 8 consecutive buckets; the 256 per-lane chunk sums get a reverse (suffix) inclusive scan as an LDS
Hillis-Steele tree (8 steps, 2 syncs each); the crossing lane is the one with
suffix_incl >= remaining > suffix_excl,and only that lane walks its own 8 buckets. Same bucket, same residue, same tie behaviour as the serial scan.
Same-hold A/B against the merged kernel, arms interleaved A/B/A/B, best of 3 rounds per shape, only that one
.hipdiffers:
geomean 2.38x, worst 1.29x. The 111 -> 13 us at small
nis the fixed cost disappearing; the n-proportional slope isuntouched, which is why the large-
nrows only gain 1.16-1.25x. Correctness:-o TOP_K -b HRX0484/484 withhip.top_k_f32matched, theGGML_HRX_DISABLE_DISPATCHcontrol still declining every case as unsupported, CPUreference 529/529. Registers 17 VGPRs (was 8), 0 spills, occupancy 16 waves/SIMD, LDS 9,228 B (was 8,204). Only the
kernel changed — the matcher, its shape bounds and the
device_supports_opclaim are byte-identical, and the add-on ABIcheck still reports 336 kernels matching their code objects.
Two things worth recording for whoever touches this next:
__shfl_down_synchere. HIP rejects a 32-bit mask on this toolchain, and with a 64-bit mask anunguarded add inflates every suffix sum at the top of the wave, because
shfl_downreturns the caller's own valuefor an out-of-range source lane rather than zero:
ERR = 4.0on n=2/rows=2,ERR = 0.667after adding the rangeguard. The LDS tree is wave-agnostic and cheap; the shuffle is not worth the tuning.
(
k > n, non-f32,nabove the template range) and they correctly fall to the CPU backend — not failures.Not measured, and it is the one thing left: the model row (18.8k decode, kernel ON vs
GGML_HRX_DISABLE_DISPATCH=hip.top_k_f32). It needs the 76 GiB Flash-Next file at-c 19456with ~92 GiB ofMemAvailable and the owner's serve holds ~35 GiB of this 122 GiB box, so the 18.8k rows for #679 are still
memory-blocked. Every number above is op-level.
Branch
1bit/hrx-top-k-coop-scan@8b7966f90(base199165047, one commit, one file) is pushed and ready for a PR;the round log with the measurement protocol (including why a single 81-shape sweep is not usable: a byte-identical
candidate "measured" 0.59x-2.05x on it) is at
~/geak-tasks/tattn-top-k/ROUNDS.mdon strixhalo.bong-water-water-bong commentedon Nov 2, 2026
Addendum to the comment above — one more in-contract change is in the same branch: geomean 2.38x -> 2.77x.
The emit phase ran two full passes over the row (keys strictly above the rank-k key, then keys equal to it). The
split is already known when the level loop ends — exactly
top_k - remainingkeys are above,remainingare equal — soboth groups can be written in one pass into disjoint slot ranges,
[0, above_slots)and[above_slots, top_k),with the per-group slot guard keeping the result duplicate-free (the ties variant's requirement). Five passes over the
row become four.
geomean 2.77x, worst 1.40x, baseline's own per-round spread 1.01x (quiet box). Correctness re-verified on this exact
candidate: 484/484, control declining all 529, CPU 529/529, 17 VGPRs, 0 spills. Branch head is now
1b292a11b(still onecommit, one file).
One thing the measurement says that changes the next step, worth flagging rather than leaving for a later PR: the
remaining cost is parallelism, not atomics or bandwidth. The kernel reads the row 5 times (now 4), and at
rows=1it moves ~8 GB/s on a ~100+ GB/s box whilerows=16reaches ~83 GB/s — the 16-row case does 16x the work in1.54x the time, so throughput scales with workgroup count and
rows=1(one workgroup, ~1/40th of the GPU) is starved.Splitting a long row across P workgroups is therefore the biggest remaining lever for the QSA shape — but it is not
doable inside the kernel: it needs a global histogram and a device-scope counter, i.e. either an extra binding for
scratch (the ABI is fixed) or a cooperative launch (the matcher launches one workgroup per row). That is a
matcher-level task, so it should be scheduled as one instead of being smuggled into the kernel work.
bong-water-water-bong commentedon Nov 2, 2026
TOP_K on HRX, round 3 (vec4 row reads):
1.909xover round 1,5.436xcumulative against the #161 pin. Branch1bit/hrx-top-k-coop-scan@b25b0485d(2 commits, 1 file, +101/-28, base199165047). Op-level only — the 18.8kmodel row is still unmeasured and is not being claimed.
Why vec4, and the number that pointed there
Round 1 (
1b292a11b: cooperative bucket scan + single emit pass) removed the fixed ~100 µs and one pass, and measured2.77x against the pin. I then tried candidate compaction (after level 0, write the higher-bucket indices straight to
dst and collect the selected bucket into an LDS candidate list, so levels 1/2 and the emit run over ≤2048 entries — 4
row passes to 2). It was rejected on measurement: 0.973x geomean, worst 0.95x against round 1, on a quiet box. The
passes it removes are cheap because the row is cache-warm by then; its restructured base was ~5-10% slower to start with.
That failure is the useful part: at
rows=1one pass is 200 000/256 = 781 iterations per lane and ~143 ns each(~300 cycles) = one un-hidden DRAM latency per iteration, because a lane has one outstanding load and the workgroup is
only 4 waves. So the limit is memory-level parallelism, not work. Every pass (3 histograms + emit) now reads
float4when the row base is 16-byte aligned (
src + row*column_count, so the guard covers pointer and pitch), with a scalartail for
column_count % 4and a fully scalar fallback. Index order is unchanged, so the ties rule and slot numberingare untouched; the ABI, the matcher bounds and the
device_supports_opclaim are byte-identical to the pin.geomean 1.909x, worst 1.03x; cumulative against the pin on the same gate 5.436x, worst 4.23x. Protocol:
same-hold, arms interleaved A/B/A/B, best of 3 rounds per shape, raw per-round values printed (this box's per-sweep
noise is one-sided and reaches 1.6x, so a phase-ordered or single-sample protocol measures the box, not the kernel).
Gates (all re-run on the pushed head)
test-backend-ops -o TOP_K -b HRX0484/484 withhip.top_k_f32 (single)taken;GGML_HRX_DISABLE_DISPATCHcontroldeclines all 529 cases as
not supported [HRX0]with nothing failing; CPU reference 529/529; whole-backend guardidentical 4289 per-case outcomes (the pin's own 320
[VIEW]/TOPMOEfailures are present on both arms and theirerror text is normalised). Compiler: 18 VGPRs, 34 SGPRs, 0 spills, occupancy 16, LDS 8 228 B. The algorithm and the
rejected design were both modelled in
scripts/ref_topk.pyagainst a sort reference first (616 cases, 0 failures).What this does and does not mean for the #679 gap
There is no expected decode jump:
top_kat 18.8k is ~12 x 50 µs ≈ 0.6 ms of a ~79 ms step (~0.8%), so the decoderesidue stays where the attribution put it — the O(n_kv) mask rebuild and the dense FA over n_kv (P2 in the kernel
plan). Where it should show is prefill: the same dispatch was worth 2.97x there, and it is now ~5x faster at the
n=8192..262144shapes prefill produces. The row that would confirm both (18 782 tokens, 512 greedy, median of 3,-c 19456 --no-repack, kernel on vsGGML_HRX_DISABLE_DISPATCH) needs the 76.33 GiB qwen4exp file and ~92 GiB ofavailable memory; it is queued behind a headroom gate and no model-level number is claimed here.
bong-water-water-bong commentedon Nov 3, 2026
Measured tonight: the attention path is ~14 % of this gap, not the gap
The 9.70 tok/s row is from
c1(pre-#161). Since then the QSA path has been fixed in three measured steps(
TOP_Kon device #161, the mask rebuild'sSET_ROWS, and now sparse index-list attention), so it is worth sayingplainly where the long-context decode time actually goes, because the answer is not the attention.
The measurement that settles it (new tonight, no model needed): the model's entire full-attention set for one
decode step — 12 QSA layers, each with indexer gather + pool + top_k(2051) + the mask rebuild + attention — built as
one graph at the GGUF's real shapes, and timed as one step:
So at 18.8k the whole attention path — the thing this issue was opened about — costs ~10 ms of a decode step. The
model decodes at 12.67 tok/s on today's main, i.e. ~79 ms/step, of which the short-prompt floor is ~35 ms; the
context-dependent slope is therefore ~44 ms, and the 12 full-attention layers account for 10 ms of it (~23 % of the
slope, ~13 % of the step). The other ~34 ms/step of context-dependent cost is not in these layers.
That matches the per-stage probes rather than contradicting them: indexer 0.60 ms, top_k 0.07, mask rebuild 0.07,
attention 0.81 per layer at 18.8k — and 12 layers in one graph cost exactly 12x that, so there is no hidden per-node or
per-split overhead inside the attention graph (the earlier "1.5 ms measured against 5.9 ms/layer implied" gap was me
dividing the model's whole slope by 12 QSA layers, which assumed the slope lives there; it does not).
What is left, with what each candidate needs to be settled
The remaining context-dependent cost has to be one of these, and every one of them needs the model to attribute:
is the largest unmeasured block in the step, and "expected" is not measured.
(
GGML_HRX_GRAPH_PROGRAM_CACHE, earlier sessions counted 933 trims in 40 min), so at least one program is evicted andre-JITed per step.
GGML_HRX_GRAPH_PROGRAM_CACHE=128was measured to give 0 trims on another model and was nevermeasured here — the row never ran.
and the weights are 76.33 GiB. A context-dependent fault rate is exactly the shape of a slope that no kernel-level
profile shows.
Why the model row still has not run
It needs the 76.33 GiB file with ~92 GiB MemAvailable (the arm's own GTT peak is 83.8 GiB). Tonight the box has
49-51 GiB available with the owner's
forward-serveholding 31.7 GiB, and the waiter is armed for the moment thatclears. I am not lowering the gate to manufacture the row — the last time this arm ran near the ceiling it took the box
down (engine #680).
Ours, fixed tonight
gpu-kernels#49 @22dc9f6: sparse index-list attention for these layers (3.0 ms/step off the attention at 18.8k,verified against the dense path at NMSE 1.7e-07), including the guard the 12-layer row exposed — at n_kv ~2048 the
selection is the whole cache and the same dispatch was 1.7 ms/step slower, so it now declines when the selection covers
half the cache or more.
What I would do next, in this order, once the box has the memory: (a) the same 18.8k row with
GGML_HRX_GRAPH_PROGRAM_CACHE=128— one env var, and it either moves the step or kills candidate 2 for good; (b) thesame row with the 36 GDN layers' graph timed separately (they can be built model-free the way the QSA set was); (c) only
then a kernel, if what is left is arithmetic.
bong-water-water-bong commentedon Nov 4, 2026
Long-context follow-up: multi-token prediction is not a long-context lever (paired A/B, one claim, both orders)
Different file from this issue's headline row, said up front so the numbers are not moved onto Flash-Next: this is
Qwen3.8-27B UD-Q4_K_XL (17.56 GB), 20,480 ctx, 18,762-token prompt, via the Hyperloom custom workload on strixhalo
(
tools/hyperloom/custom/benchmark.py, engine7d7c46f>=d3d2667a6, llama.cpp1d32c226, hrx-system82804197bcbe,kernel
7.3.0-rc4-1bit-perfopt+). Metric: decode tok/s at depth (the serve long leg), plus the short serve legs.Why paired, and why two orders
The arm-to-arm spread of this metric on this box is 19.3 % (12.938 / 15.435 / 13.342 on three identical no-lever
arms) -- larger than any lever effect measured here. So the lever was measured as A and B inside one
gpu-holdclaim,then repeated with the order reversed, because "B second, onto a hotter die" is a real confound (the 2026-11-01 11:13
drafter fault reset the box).
HL_MTP_MAX=6Both halves of both pairs gate PASS at KLD 0 (greedy texts identical),
gfxhubfault lines 1 -> 1 (MES banner only).The sign did not flip, and the prose/short penalty replicates to within 0.7 pp across the two orders (-23.5/-26.8 vs
-23.6/-27.3). Running B first, onto a cooler box, makes the decode penalty larger, so the effect is the lever, not a
thermal ramp; the pair-1 ordering was masking part of it. The decode magnitude itself (-9.7 vs -17.9 %) is not stable
across pairs because the two halves' prefill differed by 27 % in pair 2 (250.85 -> 318.77, second half faster) -- and that
drift runs against the conclusion (the faster half is the one that won), so -17.9 % is conservative. Prefill is not
comparable across days at all (139.57 / 279.93 / 318.77 on the same binary), so decode is the metric of record here.
Mechanism, captured in the same runs: draft acceptance falls 0.6746 -> 0.4381 with depth (drafts 85/126 -> 92/210,
mean draft length 3.02 -> 3.63). Deeper drafts are converted less often, so at depth the lever spends draft work it cannot
turn into tokens.
What this means for this issue
HL_MTP_MAX=6must not be carried onto long context, even though it is a +13.8 % win on the short-prompt geomean(50.87 vs 44.69 clean). The two metrics disagree; only the short-prompt number stands.
drafting is measured harmful (above);
HL_DFLASHat long context is refused in committed code (hyperloom custom: refuse the drafter at long context (#764's GTT ceiling does not cover it) #773) because it is thearm that faulted the graph on 2026-11-01;
HL_UB/HL_Bmove thellama-benchlegs but not the serve decode leg(
1bit serverejects-b/-ub/-ctk/-ctvand sizes those per route);ONEBIT_HRX_DECODE_SPLIT=0is documented as a lossat ctx >= 512. That points at the kernels -- the attention work in this lane's own thread -- not at the tuner.
prefill 318.77). Use that, not the 14.72 from the older binary, and not the 12.94 or 15.44 single arms.
bong-water-water-bong commentedon Nov 4, 2026
#679 follow-up comment (draft, 2026-11-03 night)
Posted by pi-attn (strixhalo). Numbers are from this box, this pin, one
gpu-holdclaim per row; nothing below is aprojection unless it says so.
Two things changed since the last comment, and one candidate is now dead. All rows are on the fork build
b2b545d06(= based8bd0ff60+ thetop_kGEAK branch).1. The "our route with
-ctk q8_0 -ctv q4_0" row is runnable now — and it is 3x to 17x SLOWERThe 10-31 comment said that row could not run: a quantized V had no
SET_ROWSwriter at all (match_set_rows_addonaccepts
GGML_Q8_0only), so the graph was refused outright.hip.set_rows(PR-READY branch1bit/hrx-set-rows)supplies that writer, so quantized KV now runs end to end on HRX with 0 rejected nodes. Measured on Qwen3-4B-Q4_K_M,
-c 32768, 29,150-token prompt, 64 greedy tokens:hip.fa_prompt_f16-ctk q8_0hip.fa_prompt_q8-ctk q8_0 -ctv q8_0hip.fa_prompt_q8Same three at
-c 8192/ 3,257 tokens: f16 1153.9 / 59.90, q8k 378.1 / 53.62, q8k8v 350.8 / 46.70.So the answer to "is quantized KV the missing row" is no, by a wide margin: it costs 3.0x prefill at 3.3k and 17.2x
at 29k on this route, in the same binary, with the quantization itself doing the expected thing (KV bytes halved - the
f16 row also shows prefill falling from 1154 to 979 t/s over the same range, i.e. KV bytes are not the limit). This is a
kernel gap, not a policy one, and not explained by bytes: the q8 cache is half the traffic of f16, so the 17x is in
hip.fa_prompt_q8's own schedule (one workgroup per (query head, 32-query tile), each walking the whole cache anddequantizing its chunk into LDS - so the cache is dequantized once per query tile per head), not in the quantization.
The KLD/needle A/B for quantized KV is moot at these numbers.
2. Long context is not generically slow on HRX — control row on a non-QSA model
Qwen3.6-35B-A3B-UD-IQ4_XS (17.7 GB, 3B active), same binary, 64 greedy tokens:
A 35B-A3B MoE decodes at 26-34 tok/s at 82k tokens (30-38 ms/step, 1.4 GiB of KV) where Flash-Next at 18,782 tokens
measures 79 ms/step. "HRX decode degrades with context" is not a general property of the route; the residual is
Flash-Next's own layer mix, which is why the model-free layer probes below are the right instrument.
3. Candidate killed: the HRX program-cache trims do not cost decode time
Flash-Next at 18.8k needed 66 programs against a 64-entry cache and showed 933 trims in a 40-minute run, so
GGML_HRX_GRAPH_PROGRAM_CACHE=128was one of the three ranked candidates. Measured on Qwen3.6 at 82k with decode-onlyreps (the first request pays the prefill, the next two reuse the KV), arms alternating so the box's 8-15%/hour drift
lands on both:
GGML_HRX_GRAPH_PROGRAM_CACHE=128GGML_HRX_GRAPH_PROGRAM_CACHE=128, second serverIndistinguishable. Caveat stated so this is not over-read: that model does not overflow the cache the way Flash-Next was
counted doing, so this weakens the candidate rather than closing it on the real model. It also means the 128 setting is
not worth a model row before something else does.
4. The 36 GDN layers: 15.0 ms of the step, and an instrument that is not yet numerics-clean
~/wt/hrx-probe/gdn_layer.cpprebuilds one GDN layer's decode step fromsrc/models/qwen4exp.cpp(
build_layer_attn_linear+build_conv_state_at) at the GGUF's real shapes (state 128, K heads 16, V heads 48, inner6144, conv 4, channels 10240, the file's own Q6_K projections) and sums N of them into one graph:
unattributed, which is now the whole question.
The caveat, measured rather than assumed. That graph's output is wrong on HRX: 1,400 of 2,560 values are nonfinite
and the finite ones are 29 distinct garbage values repeated 40 times each (
~2e8), byte-identically across processes,and unchanged when I shrink the probe's leaf ranges to mild values. I stage-bisected it by copying each layer-0
intermediate into a leaf (the only reliable read-back on HRX):
llm.ssm_convoutcommon.rmsnorm_gateout (normed)ssm_outmatvecggml_mul_mat(ssm_out, final), Q6_K x 1-token f32)So the recurrent math, the conv state, the state gather and the gated norm are all correct, and the corruption is in the
last node: a quantized-weight matvec with a one-token f32 activation. The shapes are not the issue either - I parsed both
GGUFs, and Bonsai-2-27B (
qwen35) has the identical GDN config (group_count 16, time_step_rank 48, inner 6144, state128) and runs correctly on HRX with the same fused kernel. Switching that closing projection from Q6_K to f32 (same
graph, same values, one env var) makes the output finite and equal to the CPU reference (rms 4.20265 vs 4.20273,
NMSE 1.8e-4) and removes
kquant.mul_mat.decode_f32from the dispatch list, while the same kernel on a leaf activationis correct in the same graph. So the recurrent math is right and the single broken node is the quantized-weight matvec
whose activation comes from another fused dispatch rather than from a leaf - the shape of a missing activation staging,
not a coverage or shape problem.
What that costs the numbers above: the 15.0 ms is a dispatch/shape timing, and with the file's own Q6_K weights it is
timed on a graph whose last node mis-stages its activation. I am quoting it as provisional, not as a validated row.
The ~40 ms unattributed arithmetic is unchanged either way.
And one thing this lane should say out loud about its own evidence: every number in this issue and in my earlier
comments is a speed row - the 76 GiB model has been memory-blocked for four days, so no KLD, needle or even
coherence check has ever been run on Flash-Next on HRX. If the model's GDN layers take the path found broken above (in
the model the projection may instead be fused into the GDN dispatch -
..._projection_epilogueexists in the kernelcatalog, and
GGML_HRX_LOG_DISPATCH=1on a real load would answer it in one line), the model would still generate tokensat 12.67 tok/s and nobody would notice. Before the next parity number is quoted, the speed row and a quality check should
be taken in the same window.
Not run, and why: the 18.8k model row. Its own waiter logged 18:27 -> 19:50 on
pool >= 88 GiBandMemAvailable >= 92 GiB; the pool never exceeded 72 GiB and MemAvailable peaked at 66 GiB, because the owner's1bit forward-serveholds ~30 GiB of GTT. The gate is not being lowered: the one attempt near the ceiling was the#680 reset.
bong-water-water-bong commentedon Nov 4, 2026
The long-context config surface is empty, and here is the log evidence — so nobody spends a GPU window on these two names.
This lane measured the 27B UD-Q4_K_XL at 20,480 ctx (
HL_LONGCTX=18800, 18,762 tokens as the server counted them). Its numbers of record: 14.5893 tok/s decode (pair-2 half A, one claim) — not Flash-Next, not the 12.67 of the issue. Two sweepable names were left untested at long context (GGML_HRX_GRAPH_PROGRAM_CACHE,GGML_HRX_Q8_PREFILL_RELAX); both are now ruled out from the artifacts rather than by another arm:grep -c 'graph program cache trimmed'over all 11 leg logs (serve.log,llama-bench.log,kld.log) of the four pair-2 arm dirs → 0 in every one, including the arm that did the 18,762-token prefill and the 128-token decode. A lever that sets the count cap to 0 (graph_cache_unbounded) cannot move a cache that never evicts, and it does not touch the separate 8 GiB byte budget (plan_evictions(budget, max_programs, …)applies both;GGML_HRX_GRAPH_PROGRAM_CACHE_MBdefaults to 8192 and is not in the grid).HL_UB=1024/2048explore runs —trimmed (1 of 39 programs, 8583 -> 8323 MiB of arenas, budget 8192 MiB, 64 max). A quality-leg cost at widened chunk shapes; no score depends on it.q8_prefill_relax_offis a prefill-routing knob (docs/hrx.md: prompt matmuls on the q8_1 x4 kernel, 94 % of HRX prompt time, prompt chunks 256–2048), and=0restores the older, slower routing. The long-context score here is the decode rate, so it can only cost the 320 tok/s prefill and cannot raise decode.Combined with what this issue's thread already has —
HL_MTP_MAX=6measured harmful at depth in both orders (−10…−18 % decode, −24…−27 % prose/short, draft acceptance 0.6746 → 0.4381),HL_DFLASHrefused by committed code (#773) after faulting the HRX graph at 18.8k on 2026-11-01, andHL_UB/HL_B/-ctk/-ctvnot reaching1bit serveat all — there is no configuration lever left for the 18.8k decode rate. The remaining gain is attention-kernel work, which is not this lane's.Separately, so the artifacts above are attributable from now on: #947 → PR #948 adds
bench_summary.device_stack = {kernel, hsa_runtime, hsa_runtime_sha256}to every measurement, because the report named the binary (#933) but not the kernel, and this box rebooted five times in 18 h.hyperloom custom: refuse a long-context leg that does not fit the pin…
bong-water-water-bong commentedon Nov 4, 2026
#679 follow-up (pi-attn, strixhalo): the re-JIT price is measured, and it is the first thing in this issue with a number in the right order of magnitude
Model-free work only; the 18.8k row is still blocked (see the end). One number matters here:
A graph-program re-JIT costs ~41 ms, and it is flat in graph size
Measured by timing the first
graph_computeof a fresh graph against the immediately following one - the second isserved from the HRX program cache - same graph, same leaves, one
gpu-holdclaim, 0 faults:It is compile+load overhead, not per-command emission, so any single trim costs the same whether it is a small decode
program or a large prefill one; and it is 3-4x the whole steady decode of a 36-layer GDN graph. (First compute in a
cold process is 110-136 ms; 41 ms is the warm figure, cross-checked by the steady p50 in those same runs scaling with
the clock ramp and the JIT delta scaling with it.)
Why this matters for this issue, and what it does not say
This model's own 18.8k server log carries 933
trim_graph_programs ... (1 of 66 programs, ... 64 max)lines in one40-minute run, and 779 in the pre-rebuild run of the same row; each trim evicts one program and frees ~6 MiB, so the
8192 MiB byte budget is never the constraint. Candidate (a) was "raise
GGML_HRX_GRAPH_PROGRAM_CACHE; one env var, andit either moves the step or kills the candidate" - ranked on intuition because nobody had priced a re-JIT.
It is priced now, and it sits inside the unexplained ~44 ms/step: at the observed rate of ~1 trim per 10 decode steps it
is ~4 ms/step; at 1 trim per step it would be ~41 ms/step, i.e. the whole residual. Which of those it is, is
still unmeasured, because it needs the 18.8k row itself.
So I shipped the cheap half as data rather than guess: engine #1006 adds one built-in recipe,
hrx-graph-program-cache-128(hrx+qwen4exp->GGML_HRX_GRAPH_PROGRAM_CACHE=128), with the trim counts and thisprice table in its
measuredfield and an explicit "the tok/s row is not measured" sentence. It is a recipe and not adefault change for the same reason. Raising the count does not preallocate: the byte budget still caps the cache, and
128 slots of ~6 MiB are ~768 MiB against a 120 GiB GTT budget.
One retraction that removes a red herring from this issue's history
The "GDN layers produce 1400/2560 nonfinite, diverging from CPU" rows were my probe, not the engine:
gdn_layer.cppbound every tensor in one usage-
ANYbuffer, so the packed Q6_K weight layout was requested and never repacked - thesame class as the retracted #969. With the weight tensors in their own
WEIGHTS-flagged buffer the fused GDN graph isnon-finite-free (0/2560) and its timing rows stand unchanged. A bounded difference from CPU remains (rel_rms
~9 %, no NaN) which I am not claiming as a defect: the delta-net recurrence runs 128 steps in f16 on device against f32
on CPU. The next step if anyone picks it up is
~/lb/tattn-gdn-f32.sh(f32 vs Q6_K projections through the samecomparison), which separates the quantised path from the fused op.
Still the same block
The 18.8k row needs the 76.33 GiB file with the box to itself: the load alone measured >= 109 GiB of GTT on
2026-11-04 (against a 120 GiB ceiling and #680's 106.5 GiB reset regime), and this box has carried other tenants' serves
(32-40 GiB) continuously since 10-31. Nothing in this comment is a substitute for that row, and none of it is a tok/s
claim.
122 remaining items
bong-water-water-bong commentedon Nov 14, 2026
The 6.4 tok/s is the quant's kernel path, not long-context work -- measured, same run, same binary (23:52Z)
The open question after the first IQ3_XXS run was whether 6.39 tok/s was (a) the 3-bit subject taking a different, slower kernel path, or (b) attention/context work that the 4-bit subject also pays. It matters because (a) means the 3-bit number cannot be compared to the bar at all, while (b) would mean the bar is simply lower here than the 20.0 target.
Same model, same binary (
9dd1a1b72), same flags; only the prompt length changes:A 6.7x increase in context costs 10.8 % of decode. The 4-bit subject is 1.70x faster than the 3-bit one. So of the gap between the 3-bit subject and the 4-bit numbers, roughly 83 % is the quant and 17 % is context -- and the context part matches the historical 4-bit behaviour, which was 11.09 at 19k and ~10.4 at 32k (nearly flat).
What this settles: the bar cannot be re-baselined on
UD-IQ3_XXS. A 3-bit subject measures a 1.7x-slower decode path, so any number taken on it is a statement about that path rather than about the decode work the row is chasing.UD-IQ4_XSneeds 96.1 GB (93.7 weights + 0.46 KV + 2.0 buffers) against the 94 GB this box gives with coder-llm stopped, measured twice;UD-Q4_K_XLneeds 113.8 GB against a 104 GiB HRX pool ceiling (the engine#680 reset regime), so it was never feasible.So the row's open question is unchanged and is the one already ORCH'd: free ~4 GB in the window for a 4-bit subject (fewer co-tenant embed servers during it), or restate the bar per quant. What is now closed is the alternative I was asked to consider -- "just measure it on the smaller quant and compare" -- with numbers rather than a hunch.
One caveat kept honest: the historical 11.09 was taken on a Nov 11 build and the bench binary was rebuilt today, so the 1.70x is a quant effect with a possible binary component. The 3k-vs-18.8k comparison, by contrast, holds both of those fixed, which is why it is the part of this that can be stated without qualification.
bong-water-water-bong commentedon Nov 15, 2026
Measured in a 12-min
gpu-holdwindow (06:33:15–06:35:30Z,coder-llmstopped inside the ticket and confirmed back on:8090). The dispatch-log instrument now runs end to end, and it answers the matcher question this thread has been carrying.Both requests complete
Three blockers fell in order, each fixed and none of them the GPU: the wrapper serves
/v1/completions(not/completion, which is the child's warm-up path); a body sent withcurl -dis form-labelled, so httplib's 8 KiB form cap answered an empty 413; and the long prompt was 411 tokens overn_ctx(18800passed as words × 1.111 = 20891 tokens vs 20480), which the server now reports itself asexceed_context_size_error.The matcher answer: exactly one registration is long-context-only
Slicing the dispatch log by the request boundaries (line ranges, not marker text — see below):
From the log itself, immediately after the long request starts:
So the "a different matcher is chosen at long context" hypothesis has exactly one named suspect, and everything else in the long path was already registered during startup or the short request. That does not make it the cause — a matcher registered once is used on every subsequent iteration, so a slow one shows up in the per-token time, not in a count. Next step is a per-kernel profile of long-context decode to see whether
rmsnorm_binary_q8_1_x4dominates, rather than more matcher census.Prefill at 16k, measured per band (one run, so bands are not separable from noise)
Aggregate 150 tok/s, and the band-to-band spread (1.7k–29k) is larger than any trend in a single run — so no claim about where prefill falls over until there are three runs.
Two instrument defects found by disagreement, both fixed on my side
fa-matcher-diff.shattributed phases by marker TEXT, but the markers are only ever written to the sidecar — the serve owns the log — sosed -n '/SHORT/,/LONG/p'matched nothing and the run printed "25 on the SHORT request, 0 on the LONG": the startup set counted as short-context and an empty range counted as long-context. It now slices by the sidecar's line numbers, and on the same log it agrees with the report script and with a hand count (0 short, 1 long, the same candidate).WINDOWS.logstart line at queue time rather than grant time, so the record looked like two windows overlapping (mine "started" 06:31:46 while another lane still held the token until 06:33:11).gpu-holdhad serialised them correctly; only the log lied. The line now comes from inside the ticket.What this does not claim
One run, at 15,765 tokens, not at the bar's 18,782 — the bar is a median of 3 at 18,782 and this is a single sample below it, so it neither meets nor refutes it. It does place decode at 20.02 tok/s at 15.8k, against the 9.70 tok/s at 18.8k that opened this issue.
bong-water-water-bong commentedon Nov 15, 2026
Window 6 result: median decode 20.30 tok/s at 18,545 tokens, three reps, and the long-context-only matcher is the same one in all three.
gpu-holdwindow 07:27:06–07:35:54Z,coder-llmstopped inside the ticket and confirmed back on:8090afterwards; no kernel-log fault lines.Two honest qualifications
LONG=19027puts the next window at 18,782 exactly (19027 + 32 + 512 ≤ 20,480✓).The matcher answer repeats exactly
All three reps produced the same one candidate, by line-range attribution:
That is now four independent runs (one at 15,765 tokens earlier, three here at 18,545) naming one op and nothing else. So the "a different matcher is chosen at long context" hypothesis has a single, reproducible suspect — and a matcher registered once is used on every iteration afterwards, so the per-token cost question still needs a per-kernel profile, not another census.
What I would run next, in this order
LONG=19027, three reps — the bar's own context, to convert "20.30 at 18,545" into the exact number the bar is written against. ~9 min in a 15-min window.rocprofv3(the engine'sGGML_HRX_REPORTprints placement, not time — I checked) and see whetherrmsnorm_binary_q8_1_x4dominates the per-token time.bong-water-water-bong commentedon Nov 15, 2026
At the bar's own context now: two samples above 20.0 tok/s at 18,785 tokens — but n = 2, so the median-of-3 is not yet satisfied.
So the two valid reps are 21.21 and 21.84, both above the bar's 20.0, at the bar's context (within 3 tokens) rather than 1.3 % short as last time. What that does not do is meet the bar's own definition — median of 3 — and I am not going to average two samples and call it a median. The 502 on rep 1 is a real event, not a harness slip: the short request in the same rep succeeded, so the server was up; the long request came back
502with notimings.One more instrument bug, mine, found by rep 1
Rep 1's candidate list contained
common.soalongside the real name:common.sois a library, matched by my own extraction patterncommon\.[a-z0-9_]+— so any shared object mentioned in the log could appear as a "matcher first seen at long context". Fixed: the pattern now requires an underscore (common\.[a-z0-9_]*_[a-z0-9_]+), because matcher names in this tree carry one (common.rmsnorm_binary_q8_1_x4) and object suffixes do not. Verified: the real name still matches,common.sono longer does. Reps 2 and 3 never showed it, which is exactly why a single run is not enough to trust a candidate list.The state of the question
common.rmsnorm_binary_q8_1_x4, present in every rep that completed and in the two earlier windows (15,765 and 18,545 tokens) — five runs now.bong-water-water-bong commentedon Nov 15, 2026
The bar is met at its own context: median 21.60 tok/s at the 18,782-token prompt, four samples, and every one of them above 20.0.
The minimum is above the bar. This is not "one lucky sample carries the median": the four span 4.1 %, and the worst of them (20.97) still clears 20.0. For comparison, the number that opened this issue was 9.70 tok/s at 18.8k — so this is roughly 2.2×, at the same prompt length the bar is written against rather than 1.3 % short of it.
How to read it
1bit serve --no-strict-device(the flag is required while engine#2450's two tiedembdGET_ROWS are off HRX),GGML_HRX_LOG_DISPATCH=1,-c 20480 --parallel 1, in agpu-holdticket withcoder-llmstopped inside it — so the number is not sharing the GPU with the PR-agent model.common.rmsnorm_binary_q8_1_x4, now across six runs (15,765 tokens once, 18,545 ×3, 18,785 ×2). That hypothesis — "a different matcher is chosen at long context" — has a single reproducible suspect and nothing else, in six attempts.--no-strict-deviceflag means the run does not assert placement, only speed.Where the credit belongs
The decode number moved from 9.70 to ~21.5 across this thread's fixes, which came from the attention/mask work — not from anything in this measurement lane. What this lane contributed is the instrument (dispatch-log diff by line ranges, a completion-sized prompt guard against
n_ctx, and the completion-based readiness probe that now lives inwindow-run.sh), and the six-run reproducibility of the one candidate above.If a per-kernel share is still wanted, the next step is a
rocprofv3profile of decode —GGML_HRX_REPORTprints placement, not time, so the census cannot answer it — but the bar question this issue was opened for is now answered.bong-water-water-bong commentedon Nov 15, 2026
The HRX dispatch profile works, and the answer is not a matcher threshold: one generic attention kernel takes 86.8 % of the GPU dispatch time in this 18.8k request.
Instrument chain (each step measured, each one cost me something)
iree-profileis not on PATH — it exists only as source in~/amd-oss/hrx-system, and the built binaries are in~/amd-oss/toolchains/hrx-stock-b142-build/…/iree-profile. Worth a line inTOOLCHAINS.md: the doc namesHRX_PROFILE_MODE=dispatch → hrx2kineto.pyas if the chain were one step, and it is three.tools/hyperloom/hrx2kineto.pycannot read that output. It filtersr.get("type") in ("dispatch_event", "dispatch"), but the v18 interchange emitsrecord_type— so it exits with "no dispatch events in …" on a file holding 53,342 of them. One-word fix (record_type), and it is the kind of mismatch that reads as "the profile is empty".The finding, from the events themselves
262,818 valid
dispatch_eventrecords, aggregated bykeyoverduration_ticks:ggml_attention_sink_f32is 87 % of the dispatch time, and it is not one of thehip./hrx.specialised matchers — which reframes this issue. The long-context cost is not a matcher being chosen late (that wascommon.rmsnorm_binary_q8_1_x4, 1 of 24 registrations); it is one generic attention kernel doing nearly all the work. That is consistent with the O(n²) shape of a 19,031-token prefill, and it is where the next 2× lives.What I am not claiming
ggml_kquant_mul_mat_decode_f32occurs 7,496 times, so it is not decode-only, and my "decode" half was 49 % of the events with the same 87 % attention share. The profile carries no phase markers, so prefill vs decode needs either two runs (n_predict=1vsn_predict=32, differenced) or thederived_time_*fields decoded to wall clock. I report the whole-request share only.device_tickand I did not establish ticks→seconds, so the percentages above are rate-independent and the absolute numbers are not offered.http=200, prompt 19,031, decode 18.86 tok/s while profiling (vs 21.60 unprofiled — the profiler's own overhead, and 44 MB written for one request).The three things worth doing next, in order
hrx2kineto.py'srecord_typeso the documented chain works end to end; until then the doc's one-line description produces an empty result.ggml_attention_sink_f32for long prefill — 87 % of the time in a kernel that is not a specialised matcher is the largest single lever this profile shows.bong-water-water-bong commentedon Nov 15, 2026
Correction to my own wording, and the mechanism behind the 87 %.
I called
ggml_attention_sink_f32"one generic attention kernel … not one of the specialised matchers". That is wrong, and the file says so plainly:ggml_attention_sink_f32is the export name of the gfx11 wave32 target — the profile records exports, not targets, so the 87 % is the specialised, subgroup-32 kernel for this box. There is no faster sibling being passed over; I checked before repeating the claim.What the launch geometry shows
One workgroup per (query head × token), 256 lanes, with the key range walked inside the workgroup — so a 19,031-token prefill with this model's head count launches on the order of 10⁵–10⁶ workgroups, each scanning up to 19,031 keys. That is the naive O(n²) shape, and it matches the profile exactly:
ggml_attention_sink_f32(gfx11 wave32, per-head×token workgroups)ggml_flash_attention_f32_f16_wmma(tiled flash attention)So the 22× gap in this profile is tiled vs untiled attention, not matcher selection. The wmma flash kernel is the tiled implementation and it costs 3.9 % of the run; the sink kernel is the same problem without key tiling and it costs 87 %.
What that means for this issue
The lever is tiling the sink path (or, where the sinks allow it, running flash and applying the sink correction
S / (S + exp(sink − M))afterwards) — not another matcher threshold. That is a kernel change inloom-libs/ops/attention_sink_f32.loom, and it is the largest single item this profile contains. My earlier three next-steps stand, with (2) sharpened from "route or replace" to "tile it".And the thing I still cannot say
The phase split: the profile carries no phase markers, my marker kernel occurs 7,496 times (so it is not decode-only), and the whole-request share above therefore mixes prefill and decode. Given the geometry and that a 19k prefill is where the O(n²) term lives, the 87 % is almost certainly prefill — but "almost certainly" is not a measurement, and two differenced runs (
n_predict=1vsn_predict=32) would settle it.bong-water-water-bong commentedon Nov 15, 2026
Retracting two things I posted an hour ago, with the correct numbers — and the correct numbers make the finding sharper.
1.
hrx2kineto.pyis not broken. My conversion was.I wrote "
hrx2kineto.pycannot read that output … one-word fix." That is wrong. The script's own argument help says the input isand that command emits exactly the
"type":"dispatch_event"rows the reader expects:I had used
export --format=ireeperf-jsonl, a different interchange format (record_type, schema_version 18), which the reader does not read — and I concluded the tool was defective instead of reading its--help. No fix is needed; I have reverted the change I made in my working tree. The wrappertools/hyperloom/hrx-tracelens.shalready does this correctly (HRX_PROFILE_MODE=dispatch→ locateiree-profile→ convert →hrx2kineto.py), sitting next to the reader I had misdiagnosed.2. My dispatch count was doubled.
export --format=ireeperf-jsonlemits each event in two record types, so my hand aggregation counted 262,818 dispatches where there are 131,409. The shares were unaffected (same kernel mix on both halves), but the count was wrong and the numbers below supersede them.3. The tool's own summary, which is better than my aggregation
Per call: 30.0 ms for the sink kernel against 1.2 ms for flash attention — a 24.5× gap, at 4,067 and 4,487 calls respectively. Nearly the same number of invocations, a 25× difference in cost each, and the grids say why:
8192×1(one workgroup per query head per token, keys looped inside) versus256×32(tiled).So the conclusion of my earlier comment stands and is now measured by the tool rather than by me: the long-context prefill is untiled attention, 87 % of the dispatch time, and the tiled implementation for the neighbouring kernel is 25× cheaper per call. The lever is tiling the sink path (or running flash and applying the sink correction afterwards) — not a matcher threshold.
4. What is still open
The phase split.
--phase decodeneeds the trace to contain a decode section, and this run's dispatch time is essentially all prefill (140.4 s of dispatch vs a ~140 s prefill and ~1.7 s of decode), so a decode view needs a request with more decode and fewer prompt tokens. That is a second window with-p 512 -n 512, not something this profile can yield.bong-water-water-bong commentedon Nov 15, 2026
The decode profile — and it shares almost nothing with prefill, which is why the phase split mattered.
Window 17:07:40–17:08:59Z:
LONG=512 TOK=512(prompt 521 tokens, decode 35.22 tok/s), 846,410 dispatch rows, thenhrx2kineto --phase decode:ggml_mul_mat_id_skinny_pair_input_f32_swiglu_publish_f32ggml_kquant_mul_mat_decode_f32ggml_flash_attention_f32_f16_wmmaggml_mul_mat_id_decode_f32_wave64ggml_kquant_swiglu_decode_f32ggml_mul_mat_f32_f32_decode_wave64ggml_attention_sink_f32The two phases have different levers
ggml_attention_sink_f3286.8 % — untiled, 30.0 ms/call against 1.2 ms for the tiled flash kernel.ggml_attention_sink_f324.0 %; the expert/matmul mix is 65.8 % (28.5 + 24.3 + 7.1 + 5.9). Attention in total is 12.7 %.So an optimisation aimed at the prefill number does nothing for decode, and vice versa. The decode lever is the MoE matmul path —
mul_mat_id_*andkquant_*_decode— not attention.Does that hold at 18.8k, where this issue's bar lives?
This profile is at short context (n_kv 512→~1024), so the honest question is whether attention takes over at long context, since attention is O(n_kv) per token and matmul is O(1). Two measurements bound it without another window:
So the matmul mix dominates decode across the whole range and there is no crossover. That also explains why the bar moved from 9.70 → 21.60 without any attention-kernel change: the work that got faster was the matmul side.
Limits
device_tick, so these are shares, not seconds (profile-top.pydeliberately prints no absolute time).--phase decodeattributes by the trace's own step boundary; the prefill tail (521 tokens ≈ 0.1 s of the 1.8 s run) is small but not zero, so treat the 4.0 % as an upper bound on prefill bleed, not a floor.-n 512shape at the bar's context.bong-water-water-bong commentedon Nov 16, 2026
Where the prefill 86.8 % comes from — and it is a dispatch ordering, not a slow kernel.
From the 17:19 profile: prefill at 18.8k is
ggml_attention_sink_f32at 30.0 ms/call against 1.2 ms for the tiled flash kernel — 86.8 % of prefill for one kernel that is 25× slower per call. That reads as "the sink kernel needs optimising". It is not that.The selection is structural, not sized.
dispatch_registration/common/dispatch-attention-sink.cppselects on shape: the node isGGML_OP_FLASH_ATTN_EXTwith 5 inputs,sinksis F32/contiguous, andsinks->ne[0] == query_head_countwith the rest 1. Nothing in it looks atn_kv, the query length, or the batch. So prefill at 18.8k and decode at 521 take the same path, and only the cost per call differs because the sink kernel is untiled.And the faster path already accepts the same nodes: the flash-attention registration admits a 5-input node when
attention_sinks_supported(...)is true —— i.e. the tiled kernel is already sink-capable. It simply does not get the node, because the sink dispatch is attempted first and succeeds:
So the 25× is the price of which registration wins, not of attention sinks being incompatible with tiling.
The shape of a fix (not made yet)
Make the sink registration decline long-query nodes so they fall through to the tiled path: a sink node whose query length is above the tile's useful size is exactly the prefill case, and the flash registration already handles sinks. The arm exists —
tests/test-hrx-attention-sink.cppalready assertsattention_sinks_supported(graph, node)— so the change can be armed by extending it with a length-based decline and asserting the chosen registration, rather than by adding a new test file.Before anyone does that, one measurement is owed: the two registrations must be shown to agree numerically on a sink node at prefill length. If the tiled path's sink handling is numerically different, 25× is the price of correctness and this is a documented constraint, not a bug. That is the decode-window run's sibling and does not need a new issue.
Caveat on my evidence: read from the
cl-flm-2138-bumppin ofthird_party/llama.cpp;~/engine's pin does not containdispatch_registration/, so line numbers and file names here are that pin's, not main's.bong-water-water-bong commentedon Nov 16, 2026
The controlled version of the bound — one serve, A B B A,
n_predict=512on both armsThe published 1.63× came from two different runs (21.60 tok/s at 18,785 tokens vs 35.22 at a 521-token prompt). This is the same question asked inside one process, with the same decode length on both sides, so context length is the only variable:
Read the
prompt_ncolumn with care — it is the prefix cache, not a mistake. The 2nd and 4th requests repeat a prefix the server has already processed, so theirprompt_nis only the uncached remainder (1 and 5 tokens). Both long-context samples therefore decode against the same 18.5k KV, which is better for this comparison than re-prefilling: the arm comparison is decode-against-18.5k-KV vs decode-against-521-token-KV, with no prefill cost inside either sample.What it establishes
Method, for re-running:
bash ~/wt/os3i/679-controlled.shinside awindow-run.shticket — one serve, four requests in A B B A, so one load, one residency and no thermal step between arms. It refuses to start without its own ticket (holder must be an ancestor) and, because the installedgpu-holdon this box predates the take-time guard, it runs the 10-minute fault check itself and refuses with rc 78.bong-water-water-bong commentedon Nov 16, 2026
Correcting my own comment 11 minutes ago: the tiled flash kernel is NOT sink-capable, so the 25× is the price of correctness, not a dispatch bug.
I wrote that "the tiled kernel is already sink-capable" and proposed declining long-query nodes from the sink registration. That was wrong, and the method to check it was one grep:
kernel-corpus/kernels/loom-libs/ops/flash_attention_f32_f16_wmma.loom—grep -c sink→0. Its exported entry@ggml_flash_attention_gfx11_wave64takes(%tokens, %key_count, %key_capacity): no sink argument at all.attention_sink_f32.loom— 62 sink references, same three parameters plus the sink term, and it ships its ownggml_attention_sink_reference_f32with an explicit%use_sinkflag.So a 5-input sink node cannot be served by the tiled path. Routing prefills there would silently drop the sink term — which is exactly the failure the sink registration exists to prevent.
What line 226 actually means.
(node->inputs.size() != 4 && !attention_sinks_supported(graph, *node))reads as "5-input nodes are fine if sinks are supported". Butappend_attention_sink_dispatchis attempted first (line 631) and claims every node for whichattention_sinks_supportedis true — so that admission never fires for a sink node. It is a guard for a case the ordering makes unreachable, not a capability claim. I read the guard as the capability and mistook dead code for a plan.What that leaves. The 25× is the honest cost of attention sinks on this backend: an untiled kernel that carries the extra term, versus a tiled one that cannot. So the real options are:
What I got wrong in method: I inferred capability from a guard's negated condition instead of from the kernel's parameters. Two greps —
grep -c sinkon both.loomfiles — would have settled it before I posted. The correct predicate for "can the tiled path take sinks" is the kernel's own argument list, not the dispatcher's admission test.Caveat unchanged: read from the
cl-flm-2138-bumppin;~/engine's pin does not containdispatch_registration/or the loom corpus, so this is that pin's state.bong-water-water-bong commentedon Nov 16, 2026
Where this row stands, on the issue's own table
Three measurements from this lane, mapped onto the arms above. The short version: decode is bounded and understood; prefill is the open gap, and it has one named cause.
--moe-cache off1. The decode gap is not a long-context collapse any more
Controlled A/B (one serve, four requests A B B A,
n_predict=512both arms — the same question the original finding asked, with context as the only variable):So the 3.2× collapse the issue opened with (9.70 → 31.15) is gone: what remains is a 1.71× attention cost at 35× n_kv, and that is consistent with the decode profile (attention is 12.7 % of dispatches; MoE matmul is 65.8 %). Long context is no longer where our decode loses to the fork — the remaining 1.44× to b3 is in the kernel mix, not in context length.
2. Prefill is the open gap, and it is one kernel
The issue's prefill line (74–93 vs 459) is a 5× gap, and the profile names the cause:
attention_sinkflashSame work class, 24.5× the per-call cost, and the grid is the tell:
attention_sinklaunches 8192 blocks in one dimension (×1 in the other) where flash uses 256 × 32. A 1-D grid that wide with 32× fewer total blocks per call is a kernel that is not distributing the head/sequence dimension — which is exactly the shape GEAK is for.3. What I would do next on this row, in order
attention_sinkprefill — it is 86.8 % of prefill and 24.5× flash's per-call cost with a visibly wrong grid shape. A 2–4× there closes most of the 74–93 → 459 gap without touching decode.What this lane has delivered on #679
Bar met and posted (21.60 median at 18,785 tok/s, four samples above the 20.0 bar); prefill profile (86.8 %
attention_sink, 30.0 ms/call, grid 8192×1); decode profile (65.8 % MoE matmul, 12.7 % attention); and the controlled A/B above, which is what turns the original two-run comparison into a ratio with a spread. The row stays open on prefill, and item 1 is the next measurement this lane would take if it gets the kernel.bong-water-water-bong commentedon Nov 16, 2026
Correction to my own prefill diagnosis: the grids are the same size, and my inference from them was wrong
Two hours ago I wrote, on this issue:
The block counts are identical: 8192 either way. 8192 × 1 and 256 × 32 are the same number of workgroups, differently factored, and I read a factorisation as a deficit. Checked at the source rather than from the profile:
So the shape is
(heads, ⌈tokens/64⌉)with 128 threads per workgroup, and a8192×1line in a profile is that product flattened — not a 1-D launch.What survives, and what the next step actually is
attention_sinkis 86.8 % of prefill (121,949 ms / 4,067 calls = 30.0 ms/call) against flash at 1.2 ms/call — 24.5× per call on the same shape. That gap is real and is still the largest single lever on this row.tokensfor these 4,067 calls, because a call that covers 64 tokens needs 294 calls for an 18.8k prompt and the per-call cost would then be dominated by launch/parity rather than arithmetic.I would rather retract this now than have someone aim a GEAK run at a grid that was never wrong. The rule I keep relearning: a profiler field is a report of the artefact, not the artefact — the dispatch source is the latter.
bong-water-water-bong commentedon Nov 16, 2026
The open question from my retraction, answered from a capture already on disk:
attention_sinkis compute-bound, and 99 % of its cost is one grid shapeI retracted the "32× fewer blocks" diagnosis two hours ago and left one question: is the 30.0 ms/call launch overhead or work? That needed the per-dispatch grid dimensions rather than the flattened line, and it turns out I still had them —
t2.csv(13 MB, kept when I deleted the raw dumps) carriesGrid_Size_X/Y/Zper row.By shape, not by average
The 1,862 calls with the small grid — the ones a "launch overhead" theory would blame — are 0.1 % of the total. 99.0 % is in 1,813 calls at 8192 × 512, i.e. 4.19 M workgroups each.
What those numbers are
The dispatcher computes
workgroup_count = {heads, (tokens + 63) / 64}(addons/hip-hrx/dispatch-hip-attention-sink.cpp:29), so:⌈tokens/64⌉-c 32768, and 32768/64 = 512 exactlyAnd the constant 8192 is the flattened q dimension:
64 heads × 128 head_dim = 8192✓. (The parameter is namedheadsin the dispatcher; for this op it is heads × head_dim.)So the lever is the sink kernel at full context, not its launch count
attention_sink, and within it 99.0 % is the full-context shape — 1,813 dispatches each covering 32,768 tokens across the flattened q dimension.hrx_attention_sink.hipat 32k tokens — the sink scan itself — not the grid, and not the dispatch count. For reference,flash_attentionin the same capture runs256×32/8192×32/4096×32(2-D, y = 32) at 3.9 % of prefill, so the two kernels are not doing comparable work per call and their timings should not be read as "same shape, 24.5× slower" — which is what my retracted note said.