Skip to content

hrx: long-context decode 9.70 tok/s at 18.8k ctx (prefill 74-93) vs 31.15 / 459 on a gfx1151 fork — same GGUF, same hour, short-prompt parity 28.6 vs 28.9 #679

Open

Description

@bong-water-water-bong

Summary

On Qwen3.8-Flash-Next UD-IQ3_XXS (76.3 GiB), our HRX route loses three quarters of its decode rate on a
long context
: 9.70 tok/s at an 18,782-token prompt against 31.15 tok/s for a llama.cpp fork built for
gfx1151, on the same GGUF, same requests, same hour. Prefill falls from 74–93 tok/s to 459. Short-prompt
decode is parity (28.6 vs 28.9), which is what makes this a long-context finding rather than a backend
finding.

Numbers

Protocol for every row: 512 greedy tokens (ignore_eos), median of 3, short 216-token / long 18,782-token
prompt, -c 32768 --parallel 1, one gpu-hold claim per arm, strixhalo gfx1151, 22:31–23:20 ADT.
Bold = the two rows being compared.

arm config short tg short pp long tg (18.8k) long pp GTT peak peak RSS
c1 — our HRX route house route's own server, --device HRX0 -ngl 99 -np 1 --cache-reuse 256, house defaults, no -fa, no MTP 28.61 (28.53/28.61/28.67) 465.7 (warm) 9.70 (9.23/9.85/9.70) 74 – 93 106.5 GiB 45.3 GiB
b1 — fork, no MTP -fa on -ctk q8_0 -ctv q4_0, shipped default (its expert cache resolves off at this much device memory) 28.95 336.6 31.15 459.0 50.4 GiB 28.1 GiB
b3 — fork, --moe-cache off same config as b1 31.72 350.9 31.14 465.5 50.4 GiB 28.1 GiB
c3 — fork + its cut-vocab MTP head, n=2 as b1 + drafting 44.86 329.8 47.87 458.4 52.2 GiB 28.4 GiB
  • Short prompt: 28.6 vs 28.9 — parity. With neither side drafting, the fork is not faster on this box.
  • Long prompt: 9.70 vs 31.15 — 3.2×, and prefill 74–93 vs 459 — 5–6×.
  • b1 vs b3 differ only in a flag whose value resolved off in both, so their 29–32 t/s spread is run-to-run
    noise on this box, not an effect. The c1 numbers are medians of three, spread 9.23–9.85.

The configuration difference, stated so it is auditable

Our arm ran the house route's own server with the argv/env that 1bit serve --device hrx --print-backend
prints (engine hrxls): --device HRX0 -ngl 99 --jinja -c 32768 -np 1 --cache-reuse 256,
GGML_HRX_REPORT=1, IREE_HAL_AMDGPU_LIBHSA_PATH=…, plus --no-strict-device because 1bit refuses
ffn_moe_gate-47 (MUL_MAT_ID, f32, ne=[640,10,0,1]) under strict placement — the house route leaves 2 of
6060 nodes on the CPU for this model. The fork's arms ran -fa on -ctk q8_0 -ctv q4_0.

The -fa on control on our route was never run. Arm d1 (-fa on -ctk q8_0 -ctv q4_0 on c1's exact
server) was killed during its load when the sequence was stopped, so there is no number for "our route with
flash attention at 18.8k". That is the one missing row, and it decides how the 3.2× is attributed:

  • if -fa on closes most of the gap → the finding is "the house defaults for this model must enable
    flash attention at long context"
    , and it is a one-line default, not a backend defect;
  • if it does not → the long-context attention path (not the backend as a whole, given short-prompt parity)
    is where the work is.

What is already excluded by the data above: "the fork is faster" (it is not, at short context, without
drafting), and "the file or the quant is the difference" (same GGUF, byte-identical).

Why this matters

This is the first file where a published long-context number exists for the house route, and it is the
kind of row a user will hit first: a 76 GiB model at a 32k context. It also sits next to #572 (Flash-Next
prefill 285 tok/s on UD-IQ4_XS vs 1,679 for a tuned Vulkan engine), which is the same family at a short
context — different file, different mechanism, and the two should not be conflated: #572's blocker is a CPU
GET_ROWS node and f32 dispatch overhead; this one is decode attention over 18.8k of history.

Reproduce

# house route (one gpu-hold claim)
llama-server -m Qwen3.8-Flash-Next-UD-IQ3_XXS-00001-of-00003.gguf \
    --device HRX0 -ngl 99 --jinja -c 32768 -np 1 --cache-reuse 256 --no-strict-device
    # env: GGML_HRX_REPORT=1 IREE_HAL_AMDGPU_LIBHSA_PATH=<TheRock libhsa-runtime64.so.1>
# fork, same box, same file, same hour
llama-server ... -fa on -ctk q8_0 -ctv q4_0      # + the fork's own cached build for gfx1151
# client: 512 greedy tokens, ignore_eos, short 216 / long 18,782 tokens, medians of 3

Sampler: peak GTT from /sys/class/drm/card0/device/mem_info_gtt_used at 1 Hz, peak RSS from /proc.
Per-arm server logs, bench.json and the client sample.txt are under ~/lb/aval/*/ on strixhalo.

Next

One row: arm d1 (-fa on -ctk q8_0 -ctv q4_0 on the same server, same protocol). It needs one GPU claim and
about four minutes, and it turns this issue from "3.2× slower" into either a default to change or a kernel
gap to size. Until it exists I am not calling it a backend defect.

Related

Activity

  1. bong-water-water-bong commented on Oct 31, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Attribution for #679 — long-context decode on HRX (Qwen3.8-Flash-Next UD-IQ3_XXS)

    Everything below is measured on strixhalo (gfx1151), medians of 3 unless the row says otherwise, one gpu-hold
    claim per arm, GTT sampled at 1 Hz with a hard 95 GiB abort guard.

    1. The "one missing row" cannot be measured as written, and it is not the mechanism

    Arm d1 (-fa on -ctk q8_0 -ctv q4_0 on c1's exact server argv) does not run: the server loads (49.7 GiB peak
    GTT) and the first decode dies with

    E graph_compute: unsupported HRX node: SET_ROWS output=621:q4_0[512,8192,1,1]<-SET_ROWS
    E llama_decode: failed to decode, ret = -3
    

    HRX's only quantized-KV writer (match_set_rows_addon, ggml/src/ggml-hrx/dispatch_registration/common/dispatch-rope-set-rows.cpp)
    accepts GGML_TYPE_Q8_0 only, so a q4_0 V cache has no writer at all; and the HRX flash-attention matchers require
    KV element size sizeof(ggml_fp16_t), while llama.cpp force-enables FA for a quantized V cache. So the fork's KV
    configuration is doubly unreachable on our route, and -ctk q8_0 -ctv q4_0 would not use the HRX FA kernel even if the
    write existed.

    The -fa half of the difference is also a non-issue, measured on the same arm at -c 8192, 5.4k prompt, 64 tokens:

    arm flags tg pp
    d2 -fa on 14.65 205
    c1 argv no -fa (llama.cpp default is auto) 15.91 443
    c1 argv -fa off 20.65 (short ctx only) segfault during the 5.4k prefill

    i.e. c1 already ran the HRX FA kernel, and turning it off is both slower and unstable.

    2. What actually scales with context: the QSA selection path

    Qwen3.8-Flash-Next's full-attention layers are indexer-sparse (QSA, src/models/qwen4exp.cpp). While
    n_kv <= indexer_top_k + r - 1 = 2051 the top-k graph is skipped entirely (dense); above it, every such layer, every
    step, runs: indexer gather over the whole indexer KV -> block pooling -> ggml_top_k(2051 of n_kv) -> a full-n_kv
    mask rebuild -> flash_attn_ext.

    One server, one process, decode 64 greedy tokens, median of 3 (-c 8192, f16 KV, no -fa override):

    n_kv pp t/s tg t/s ms/step
    1,001 694.9 28.06 35.6
    1,498 676.1 25.71 38.9
    1,986 683.8 25.49 39.2
    2,096 595.7 20.42 49.0
    2,398 620.6 19.40 51.5
    3,005 293.1 17.86 56.0
    3,995 491.5 17.33 57.7
    5,391 414.6 16.43 60.9
    6,978 351.7 14.84 67.4
    • stepping from 1,986 to 2,096 tokens (the only change is crossing 2051) costs +9.7 ms per decode step;
    • above it the slope is ~3.8 us per KV token per step, which extrapolates to ~112 ms/step at 18,782 tokens against
      the 103 ms (9.70 tok/s) that c1 measured there -- the whole gap to the fork's flat 31.15;
    • forcing the same path below the threshold (LLAMA_QSA_DENSE=0) at fixed n_kv ~1,480 costs 13x prefill
      (676 -> 52.1 tok/s).

    GGML_OP_TOP_K is the only op in that path with no HRX dispatch: it is absent from eager_capability_declared()
    (ggml-hrx.cpp), no matcher registers for it, so device_supports_op is false, the scheduler puts it on the CPU
    backend
    and splits every full-attention layer's graph around it. The CPU implementation is a single-row
    std::partial_sort with an indirect comparator (ggml-cpu/ops.cpp), and QSA has one row per layer per step.

    3. The fix and what it buys

    PR 1bit-MONSTER/llama.cpp#161: GGML_OP_TOP_K on HRX as a radix select on a monotone uint32 key map (three levels
    of 11/11/10 bits to the exact rank-k key, then one emit pass), one workgroup per row, 8 KiB LDS histogram, no global
    scratch, no host round trip; plus a bounded claim in device_supports_op so shapes the kernel cannot take keep their
    CPU fallback.

    test-backend-ops -o TOP_K -b HRX0: 484/484 with ggml_hrx dispatch: hip.top_k_f32 (single), and the
    GGML_HRX_DISABLE_DISPATCH=hip.top_k_f32 control degrades to the CPU backend instead of failing the graph.

    Long-context rows with the kernel, same binary, same pin, paired against GGML_HRX_DISABLE_DISPATCH=hip.top_k_f32
    (identical to the pre-PR behaviour):

    arm n_kv tg t/s pp t/s
    kernel ON (hip.top_k_f32) pending
    kernel OFF (GGML_HRX_DISABLE_DISPATCH=hip.top_k_f32, = pre-PR behaviour) pending

    Those rows are queued behind a release-gate window on the box (a 76.3 GiB Flash-Next load needs ~88+ GiB of
    MemAvailable and the box is currently hosting other work), together with the 18,782-token protocol row (512 greedy
    tokens, median of 3, -c 19456, 95 GiB GTT cap). I will post them here as a follow-up rather than guess.

    What is measured for the kernel so far, at the op level (test-backend-ops):

    • -o TOP_K -b HRX0: 484/484, ggml_hrx dispatch: hip.top_k_f32 (single) in the dispatch log, and the
      GGML_HRX_DISABLE_DISPATCH control falls back to the CPU backend instead of failing the graph;
    • perf -o TOP_K -b HRX0: 106 us (n=1000), 230 us (n=65,000), 623 us (n=200,000) per single row -- so the QSA decode
      shape (12 layers, one row each, n=n_kv) should cost ~1.5-2 ms per step against the ~49 ms the CPU path costs at
      18.8k. The fixed part is the serial 2048-bucket scan in the radix select, which the GEAK loop will attack next.

    4. Not the cause (so no one re-does these)

    • flash-attention kernel coverage on the full-attention layers: matched and HRX-placed at f16 KV; no rejected nodes
      in any server log of these runs.
    • KV layout / KV bytes: 462 MB of KV traffic per decode step at 18.8k is ~5 ms at 100 GB/s -- not the ~49 ms that
      was missing.
    • The GDN / linear-attention state path: fixed-size state, 36 of 48 layers, no context term in the curve.
    • The 2 CPU MUL_MAT_ID expert nodes (--no-strict-device): they are per step, not per layer, and have no
      context term.

    5. Residual, after the kernel

    The O(n_kv) mask rebuild that follows the selection (a [1,n_kv] f32 zeros + set_rows + add, then a cast for FA)
    is still per-layer-per-step work that grows with context, on HRX but serial. If the numbers above leave a slope, that
    is where it is, and the next step is either a fused qsa_mask_build dispatch or an FA variant that reads the selected
    indices directly instead of rebuilding a dense mask.

  2. bong-water-water-bong commented on Oct 31, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Follow-up on #679 — the long-context rows with the fix in (and what it does not fix)

    Same box (strixhalo, gfx1151), same model (Qwen3.8-Flash-Next UD-IQ3_XXS), one gpu-hold claim per arm, GTT sampled at
    1 Hz under a 95 GiB abort guard, 18,782-token prompt, 512 greedy tokens, ignore_eos, median of 3.

    The fix: fork PR 1bit-MONSTER/llama.cpp#161 — GGML_OP_TOP_K on HRX (radix select on a monotone uint32 key map of
    the f32 bit pattern, one workgroup per row, 8 KiB LDS histogram, no global scratch, no host round trip) plus a bounded
    claim in device_supports_op. test-backend-ops -o TOP_K -b HRX0: 484/484 with
    ggml_hrx dispatch: hip.top_k_f32 (single), and GGML_HRX_DISABLE_DISPATCH=hip.top_k_f32 degrades to the CPU backend
    instead of failing the graph — which is what makes a clean pre/post comparison possible on one binary.

    The rows

    Both rows: same binary, same pin (4a58da9b0 = hrx-base + #159 + #160 + #161), same flags
    (-c 19456 --no-repack, c1's argv otherwise). The only difference is the environment variable.

    arm short tg (216) long tg (18,782) long pp peak GTT
    GGML_HRX_DISABLE_DISPATCH=hip.top_k_f32 (pre-PR behaviour) 28.61 10.90 (11.02, 10.78) 131.0 (134.1, 127.9) 84.17 GiB
    kernel in (hip.top_k_f32) 28.43 12.67 (12.56/12.94/12.67) 388.7 (388.7/411.3/251.4) 83.82 GiB
    c1, published above (old pin, -c 32768, no --no-repack) 28.61 9.70 74–93 (84.5) 106.5 GiB

    (the disabled row has two long reps rather than three: I cut it short to hand the GPU to two lanes already queued behind
    my claim; the two agree to 2%, and the enabled row's three agree to 3%.)

    What the dispatch buys, measured on one binary

    disabled kernel factor
    long-context prefill 131.0 t/s 388.7 t/s 2.97x
    long-context decode 10.90 t/s 12.67 t/s 1.16x
    short-context decode 28.61 28.43 flat

    Prefill is the win, and it is the expected one: the QSA layers call TOP_K once per layer per ubatch with one row per
    token, so prefill was 512 serial std::partial_sorts on 4 CPU threads (with a graph split around every layer); on device
    it is 512 workgroups. Long-context prefill now sits at 389 t/s, i.e. within reach of the gfx1151 fork's 459 rather than
    5–6x under it.

    Decode at 18.8k only moves 1.16x, so the selection was not where decode time went. What remains is the rest of the
    QSA path, all O(n_kv) per full-attention layer per step: the indexer get_rows over the whole indexer cache, the
    block→cell expand, and the [1, n_kv] f32 mask rebuild (zeros + set_rows + add, then a cast for FA) ahead of a
    dense FA over all n_kv. Consistently, the 2051-token cliff survives with the kernel in (26.86 → 21.55 tok/s across it in
    the same run): crossing 2051 switches the whole path on, not just the sort.

    Correction on the memory side (#680)

    I expected the ~24 GiB long-prompt GTT transient (c1: 106.5 GiB) to be the per-layer graph splits, and therefore to
    disappear with the dispatch. It does not: the disabled row on the new pin/flags also peaks at 84.17 GiB. So the kernel
    has no GTT effect (83.82 vs 84.17, noise), and the 106.5 → 83.8 GiB difference comes from the pin move and/or
    -c 19456 + --no-repack, not from this PR. The operationally useful fact for #680 stands either way: at
    -c 19456 --no-repack on the current pin, this model's 18.8k row never exceeds 84.2 GiB, ~11 GiB clear of the 95 GiB
    rule, with or without the kernel.

    Next, in order

    1. The residual decode term — the O(n_kv) mask rebuild and indexer gathers, not the selection: either a fused
      qsa_mask_build dispatch or an FA variant that consumes the selected indices directly instead of rebuilding a dense
      mask.
    2. GEAK on the new kernel — its fixed cost is the serial 2,048-bucket scan (~100 us/row); the plan is cooperative
      scan / warp-aggregated histogram / 8-bit levels.
  3. bong-water-water-bong commented on Nov 1, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    TOP_K on HRX is merged on the fork (#161 → 199165047), but this issue stays open for two reasons, both stated so it is not closed on the merge:

    1. it is in no build: engine main pins third_party/llama.cpp = 4a58da9b0, 8 commits behind the fork tip 199165047, and there is no open bump PR (#759 → 37ca4d442 was closed as superseded at 15:54:53Z).
    2. the row still needs a fault-free arm: with the TheRock nightly libhsa, any run without HSA_USE_SVM=0 HSA_USERPTR_FOR_PAGED_MEM=0 HRX_ALLOW_NO_SVM=1 faults on its first graph replay (hrx: gfxhub permission page faults (UTCL2 client CPF) on the rebuilt box with the TheRock nightly libhsa — the no-SVM env trio is the only fault-free arm #715, gfxhub … UTCL2 client CPF), and test-backend-ops hangs on the fault. No fault-free measurement of the 18.8k row exists yet.

    Closes on: the pin bump, then arm d1 (-fa on -ctk q8_0 -ctv q4_0) at 18.8k, memory-capped and alone, fault delta 0, beating the 28.43 tok/s short-prompt baseline.

  4. bong-water-water-bong commented on Nov 2, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Follow-up: the TOP_K kernel from #161 now has a 2.38x geomean improvement (op-level), branch 1bit/hrx-top-k-coop-scan ready.

    The dispatch fixed the cliff and the prefill gap, but it left one measured defect: inside the radix select the descending
    bucket scan ran serial in lane 0 — 3 levels x 2,048 buckets = 6,144 dependent LDS reads plus integer work,
    independent of n. That is the ~100 us fixed cost test-backend-ops perf -o TOP_K -b HRX0 shows at n=1000, and it is
    paid by all 12 full-attention layers on every decode step.

    Each lane now owns 8 consecutive buckets; the 256 per-lane chunk sums get a reverse (suffix) inclusive scan as an LDS
    Hillis-Steele tree (8 steps, 2 syncs each); the crossing lane is the one with suffix_incl >= remaining > suffix_excl,
    and only that lane walks its own 8 buckets. Same bucket, same residue, same tie behaviour as the serial scan.

    Same-hold A/B against the merged kernel, arms interleaved A/B/A/B, best of 3 rounds per shape, only that one .hip
    differs:

    shape before after ratio
    n=1,000, rows=1, k=400 111 us 13 us 8.74x
    n=1,000, rows=16, k=400 108 us 14 us 7.48x
    n=65,000, rows=1, k=400 246 us 155 us 1.50x
    n=65,000, rows=16, k=400 341 us 258 us 1.29x
    n=200,000, rows=1, k=400 643 us 511 us 1.25x
    n=200,000, rows=16, k=400 908 us 766 us 1.16x

    geomean 2.38x, worst 1.29x. The 111 -> 13 us at small n is the fixed cost disappearing; the n-proportional slope is
    untouched, which is why the large-n rows only gain 1.16-1.25x. Correctness: -o TOP_K -b HRX0 484/484 with
    hip.top_k_f32 matched, the GGML_HRX_DISABLE_DISPATCH control still declining every case as unsupported, CPU
    reference 529/529. Registers 17 VGPRs (was 8), 0 spills, occupancy 16 waves/SIMD, LDS 9,228 B (was 8,204). Only the
    kernel changed — the matcher, its shape bounds and the device_supports_op claim are byte-identical, and the add-on ABI
    check still reports 336 kernels matching their code objects.

    Two things worth recording for whoever touches this next:

    • Do not use __shfl_down_sync here. HIP rejects a 32-bit mask on this toolchain, and with a 64-bit mask an
      unguarded add inflates every suffix sum at the top of the wave, because shfl_down returns the caller's own value
      for an out-of-range source lane rather than zero: ERR = 4.0 on n=2/rows=2, ERR = 0.667 after adding the range
      guard. The LDS tree is wave-agnostic and cheap; the shuffle is not worth the tuning.
    • The harness's "484/484" is 484 of 529 cases. The other 45 are shapes the bounded claim declines by design
      (k > n, non-f32, n above the template range) and they correctly fall to the CPU backend — not failures.

    Not measured, and it is the one thing left: the model row (18.8k decode, kernel ON vs
    GGML_HRX_DISABLE_DISPATCH=hip.top_k_f32). It needs the 76 GiB Flash-Next file at -c 19456 with ~92 GiB of
    MemAvailable and the owner's serve holds ~35 GiB of this 122 GiB box, so the 18.8k rows for #679 are still
    memory-blocked. Every number above is op-level.

    Branch 1bit/hrx-top-k-coop-scan @ 8b7966f90 (base 199165047, one commit, one file) is pushed and ready for a PR;
    the round log with the measurement protocol (including why a single 81-shape sweep is not usable: a byte-identical
    candidate "measured" 0.59x-2.05x on it) is at ~/geak-tasks/tattn-top-k/ROUNDS.md on strixhalo.

  5. bong-water-water-bong commented on Nov 2, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Addendum to the comment above — one more in-contract change is in the same branch: geomean 2.38x -> 2.77x.

    The emit phase ran two full passes over the row (keys strictly above the rank-k key, then keys equal to it). The
    split is already known when the level loop ends — exactly top_k - remaining keys are above, remaining are equal — so
    both groups can be written in one pass into disjoint slot ranges, [0, above_slots) and [above_slots, top_k),
    with the per-group slot guard keeping the result duplicate-free (the ties variant's requirement). Five passes over the
    row become four.

    shape before (merged #161) + cooperative scan + single emit pass vs before
    n=1,000, rows=1 106 us 13 us 11 us 9.36x
    n=1,000, rows=16 106 us 14 us 12 us 9.05x
    n=65,000, rows=1 225 us 155 us 133 us 1.70x
    n=65,000, rows=16 328 us 258 us 211 us 1.55x
    n=200,000, rows=1 617 us 511 us 423 us 1.46x
    n=200,000, rows=16 865 us 766 us 620 us 1.40x

    geomean 2.77x, worst 1.40x, baseline's own per-round spread 1.01x (quiet box). Correctness re-verified on this exact
    candidate: 484/484, control declining all 529, CPU 529/529, 17 VGPRs, 0 spills. Branch head is now 1b292a11b (still one
    commit, one file).

    One thing the measurement says that changes the next step, worth flagging rather than leaving for a later PR: the
    remaining cost is parallelism, not atomics or bandwidth. The kernel reads the row 5 times (now 4), and at
    rows=1 it moves ~8 GB/s on a ~100+ GB/s box while rows=16 reaches ~83 GB/s — the 16-row case does 16x the work in
    1.54x the time
    , so throughput scales with workgroup count and rows=1 (one workgroup, ~1/40th of the GPU) is starved.
    Splitting a long row across P workgroups is therefore the biggest remaining lever for the QSA shape — but it is not
    doable inside the kernel
    : it needs a global histogram and a device-scope counter, i.e. either an extra binding for
    scratch (the ABI is fixed) or a cooperative launch (the matcher launches one workgroup per row). That is a
    matcher-level task, so it should be scheduled as one instead of being smuggled into the kernel work.

  6. bong-water-water-bong commented on Nov 2, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    TOP_K on HRX, round 3 (vec4 row reads): 1.909x over round 1, 5.436x cumulative against the #161 pin. Branch
    1bit/hrx-top-k-coop-scan @ b25b0485d (2 commits, 1 file, +101/-28, base 199165047). Op-level only — the 18.8k
    model row is still unmeasured and is not being claimed.

    Why vec4, and the number that pointed there

    Round 1 (1b292a11b: cooperative bucket scan + single emit pass) removed the fixed ~100 µs and one pass, and measured
    2.77x against the pin. I then tried candidate compaction (after level 0, write the higher-bucket indices straight to
    dst and collect the selected bucket into an LDS candidate list, so levels 1/2 and the emit run over ≤2048 entries — 4
    row passes to 2). It was rejected on measurement: 0.973x geomean, worst 0.95x against round 1, on a quiet box. The
    passes it removes are cheap because the row is cache-warm by then; its restructured base was ~5-10% slower to start with.

    That failure is the useful part: at rows=1 one pass is 200 000/256 = 781 iterations per lane and ~143 ns each
    (~300 cycles) = one un-hidden DRAM latency per iteration
    , because a lane has one outstanding load and the workgroup is
    only 4 waves. So the limit is memory-level parallelism, not work. Every pass (3 histograms + emit) now reads float4
    when the row base is 16-byte aligned (src + row*column_count, so the guard covers pointer and pitch), with a scalar
    tail for column_count % 4 and a fully scalar fallback. Index order is unchanged, so the ties rule and slot numbering
    are untouched; the ABI, the matcher bounds and the device_supports_op claim are byte-identical to the pin.

    shape round 1 round 3 ratio
    n=1 000, rows=1, k=400 12 µs 11 µs 1.03x
    n=1 000, rows=16, k=400 12 µs 11 µs 1.07x
    n=65 000, rows=1, k=400 127 µs 55 µs 2.30x
    n=65 000, rows=16, k=400 211 µs 81 µs 2.61x
    n=200 000, rows=1, k=400 414 µs 165 µs 2.51x
    n=200 000, rows=16, k=400 622 µs 212 µs 2.93x

    geomean 1.909x, worst 1.03x; cumulative against the pin on the same gate 5.436x, worst 4.23x. Protocol:
    same-hold, arms interleaved A/B/A/B, best of 3 rounds per shape, raw per-round values printed (this box's per-sweep
    noise is one-sided and reaches 1.6x, so a phase-ordered or single-sample protocol measures the box, not the kernel).

    Gates (all re-run on the pushed head)

    test-backend-ops -o TOP_K -b HRX0 484/484 with hip.top_k_f32 (single) taken; GGML_HRX_DISABLE_DISPATCH control
    declines all 529 cases as not supported [HRX0] with nothing failing; CPU reference 529/529; whole-backend guard
    identical 4289 per-case outcomes (the pin's own 320 [VIEW]/TOPMOE failures are present on both arms and their
    error text is normalised). Compiler: 18 VGPRs, 34 SGPRs, 0 spills, occupancy 16, LDS 8 228 B. The algorithm and the
    rejected design were both modelled in scripts/ref_topk.py against a sort reference first (616 cases, 0 failures).

    What this does and does not mean for the #679 gap

    There is no expected decode jump: top_k at 18.8k is ~12 x 50 µs ≈ 0.6 ms of a ~79 ms step (~0.8%), so the decode
    residue stays where the attribution put it — the O(n_kv) mask rebuild and the dense FA over n_kv (P2 in the kernel
    plan). Where it should show is prefill: the same dispatch was worth 2.97x there, and it is now ~5x faster at the
    n=8192..262144 shapes prefill produces. The row that would confirm both (18 782 tokens, 512 greedy, median of 3,
    -c 19456 --no-repack, kernel on vs GGML_HRX_DISABLE_DISPATCH) needs the 76.33 GiB qwen4exp file and ~92 GiB of
    available memory; it is queued behind a headroom gate and no model-level number is claimed here.

  7. bong-water-water-bong commented on Nov 3, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Measured tonight: the attention path is ~14 % of this gap, not the gap

    The 9.70 tok/s row is from c1 (pre-#161). Since then the QSA path has been fixed in three measured steps
    (TOP_K on device #161, the mask rebuild's SET_ROWS, and now sparse index-list attention), so it is worth saying
    plainly where the long-context decode time actually goes, because the answer is not the attention.

    The measurement that settles it (new tonight, no model needed): the model's entire full-attention set for one
    decode step — 12 QSA layers, each with indexer gather + pool + top_k(2051) + the mask rebuild + attention — built as
    one graph at the GGUF's real shapes, and timed as one step:

    n_kv 12 QSA layers, today's path (Loom decode split) with sparse index-list attention (#49)
    18,816 13.00 ms 9.95 ms (829 us/layer)
    2,048 8.02 ms 9.74 ms → guarded back to 7.75 ms

    So at 18.8k the whole attention path — the thing this issue was opened about — costs ~10 ms of a decode step. The
    model decodes at 12.67 tok/s on today's main, i.e. ~79 ms/step, of which the short-prompt floor is ~35 ms; the
    context-dependent slope is therefore ~44 ms, and the 12 full-attention layers account for 10 ms of it (~23 % of the
    slope, ~13 % of the step)
    . The other ~34 ms/step of context-dependent cost is not in these layers.

    That matches the per-stage probes rather than contradicting them: indexer 0.60 ms, top_k 0.07, mask rebuild 0.07,
    attention 0.81 per layer at 18.8k — and 12 layers in one graph cost exactly 12x that, so there is no hidden per-node or
    per-split overhead inside the attention graph (the earlier "1.5 ms measured against 5.9 ms/layer implied" gap was me
    dividing the model's whole slope by 12 QSA layers, which assumed the slope lives there; it does not).

    What is left, with what each candidate needs to be settled

    The remaining context-dependent cost has to be one of these, and every one of them needs the model to attribute:

    1. The 36 GDN (linear-attention) layers. Their state is fixed-size, so this is expected to be context-flat — but it
      is the largest unmeasured block in the step, and "expected" is not measured.
    2. Per-step host/JIT overhead. This model needs 66 graph programs and the cache holds 64
      (GGML_HRX_GRAPH_PROGRAM_CACHE, earlier sessions counted 933 trims in 40 min), so at least one program is evicted and
      re-JITed per step. GGML_HRX_GRAPH_PROGRAM_CACHE=128 was measured to give 0 trims on another model and was never
      measured here — the row never ran.
    3. Memory pressure. The 18.8k arm peaked at 83.8 GiB GTT against ~50 GiB at 8k, on a box where GTT is DRAM-backed
      and the weights are 76.33 GiB. A context-dependent fault rate is exactly the shape of a slope that no kernel-level
      profile shows.

    Why the model row still has not run

    It needs the 76.33 GiB file with ~92 GiB MemAvailable (the arm's own GTT peak is 83.8 GiB). Tonight the box has
    49-51 GiB available with the owner's forward-serve holding 31.7 GiB, and the waiter is armed for the moment that
    clears. I am not lowering the gate to manufacture the row — the last time this arm ran near the ceiling it took the box
    down (engine #680).

    Ours, fixed tonight

    gpu-kernels #49 @ 22dc9f6: sparse index-list attention for these layers (3.0 ms/step off the attention at 18.8k,
    verified against the dense path at NMSE 1.7e-07), including the guard the 12-layer row exposed — at n_kv ~2048 the
    selection is the whole cache and the same dispatch was 1.7 ms/step slower, so it now declines when the selection covers
    half the cache or more.

    What I would do next, in this order, once the box has the memory: (a) the same 18.8k row with
    GGML_HRX_GRAPH_PROGRAM_CACHE=128 — one env var, and it either moves the step or kills candidate 2 for good; (b) the
    same row with the 36 GDN layers' graph timed separately (they can be built model-free the way the QSA set was); (c) only
    then a kernel, if what is left is arithmetic.

  8. bong-water-water-bong commented on Nov 4, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Long-context follow-up: multi-token prediction is not a long-context lever (paired A/B, one claim, both orders)

    Different file from this issue's headline row, said up front so the numbers are not moved onto Flash-Next: this is
    Qwen3.8-27B UD-Q4_K_XL (17.56 GB), 20,480 ctx, 18,762-token prompt, via the Hyperloom custom workload on strixhalo
    (tools/hyperloom/custom/benchmark.py, engine 7d7c46f >= d3d2667a6, llama.cpp 1d32c226, hrx-system 82804197bcbe,
    kernel 7.3.0-rc4-1bit-perfopt+). Metric: decode tok/s at depth (the serve long leg), plus the short serve legs.

    Why paired, and why two orders

    The arm-to-arm spread of this metric on this box is 19.3 % (12.938 / 15.435 / 13.342 on three identical no-lever
    arms) -- larger than any lever effect measured here. So the lever was measured as A and B inside one gpu-hold claim,
    then repeated with the order reversed, because "B second, onto a hotter die" is a real confound (the 2026-11-01 11:13
    drafter fault reset the box).

    pair (date) order A: no levers B: HL_MTP_MAX=6 B vs A
    1 (11-02 19:21-19:27) AB (B second, die 43-64 C) 14.72 decode 13.29 decode -9.7 % decode; prose -23.5 %, short -26.8 %
    2 (11-03 15:19-15:27) BA (B first, die 53-66 C) 14.5893 decode, geomean 48.8571 11.9825 decode, geomean 47.9897 -17.9 % decode; prose -23.6 %, short -27.3 %, code +3.1 %, geomean -1.8 %

    Both halves of both pairs gate PASS at KLD 0 (greedy texts identical), gfxhub fault lines 1 -> 1 (MES banner only).

    The sign did not flip, and the prose/short penalty replicates to within 0.7 pp across the two orders (-23.5/-26.8 vs
    -23.6/-27.3). Running B first, onto a cooler box, makes the decode penalty larger, so the effect is the lever, not a
    thermal ramp; the pair-1 ordering was masking part of it. The decode magnitude itself (-9.7 vs -17.9 %) is not stable
    across pairs because the two halves' prefill differed by 27 % in pair 2 (250.85 -> 318.77, second half faster) -- and that
    drift runs against the conclusion (the faster half is the one that won), so -17.9 % is conservative. Prefill is not
    comparable across days at all (139.57 / 279.93 / 318.77 on the same binary), so decode is the metric of record here.

    Mechanism, captured in the same runs: draft acceptance falls 0.6746 -> 0.4381 with depth (drafts 85/126 -> 92/210,
    mean draft length 3.02 -> 3.63). Deeper drafts are converted less often, so at depth the lever spends draft work it cannot
    turn into tokens.

    What this means for this issue

    • HL_MTP_MAX=6 must not be carried onto long context, even though it is a +13.8 % win on the short-prompt geomean
      (50.87 vs 44.69 clean). The two metrics disagree; only the short-prompt number stands.
    • The long-context gain this issue is after is not going to come from the config levers the tuning loop can sweep:
      drafting is measured harmful (above); HL_DFLASH at long context is refused in committed code (hyperloom custom: refuse the drafter at long context (#764's GTT ceiling does not cover it) #773) because it is the
      arm that faulted the graph on 2026-11-01; HL_UB/HL_B move the llama-bench legs but not the serve decode leg
      (1bit serve rejects -b/-ub/-ctk/-ctv and sizes those per route); ONEBIT_HRX_DECODE_SPLIT=0 is documented as a loss
      at ctx >= 512. That points at the kernels -- the attention work in this lane's own thread -- not at the tuner.
    • Number of record for the shape on the current binary: 14.59 tok/s decode at 18,762 tokens (no levers, pair-2 half A,
      prefill 318.77). Use that, not the 14.72 from the older binary, and not the 12.94 or 15.44 single arms.
  9. bong-water-water-bong commented on Nov 4, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    #679 follow-up comment (draft, 2026-11-03 night)

    Posted by pi-attn (strixhalo). Numbers are from this box, this pin, one gpu-hold claim per row; nothing below is a
    projection unless it says so.


    Two things changed since the last comment, and one candidate is now dead. All rows are on the fork build
    b2b545d06 (= base d8bd0ff60 + the top_k GEAK branch).

    1. The "our route with -ctk q8_0 -ctv q4_0" row is runnable now — and it is 3x to 17x SLOWER

    The 10-31 comment said that row could not run: a quantized V had no SET_ROWS writer at all (match_set_rows_addon
    accepts GGML_Q8_0 only), so the graph was refused outright. hip.set_rows (PR-READY branch 1bit/hrx-set-rows)
    supplies that writer, so quantized KV now runs end to end on HRX with 0 rejected nodes. Measured on Qwen3-4B-Q4_K_M,
    -c 32768, 29,150-token prompt, 64 greedy tokens:

    KV fa dispatch prefill t/s decode t/s
    f16 hip.fa_prompt_f16 979.5 25.26
    -ctk q8_0 hip.fa_prompt_q8 57.1 14.77
    -ctk q8_0 -ctv q8_0 hip.fa_prompt_q8 55.7 18.92

    Same three at -c 8192 / 3,257 tokens: f16 1153.9 / 59.90, q8k 378.1 / 53.62, q8k8v 350.8 / 46.70.

    So the answer to "is quantized KV the missing row" is no, by a wide margin: it costs 3.0x prefill at 3.3k and 17.2x
    at 29k on this route, in the same binary, with the quantization itself doing the expected thing (KV bytes halved - the
    f16 row also shows prefill falling from 1154 to 979 t/s over the same range, i.e. KV bytes are not the limit). This is a
    kernel gap, not a policy one, and not explained by bytes: the q8 cache is half the traffic of f16, so the 17x is in
    hip.fa_prompt_q8's own schedule (one workgroup per (query head, 32-query tile), each walking the whole cache and
    dequantizing its chunk into LDS - so the cache is dequantized once per query tile per head), not in the quantization.
    The KLD/needle A/B for quantized KV is moot at these numbers.

    2. Long context is not generically slow on HRX — control row on a non-QSA model

    Qwen3.6-35B-A3B-UD-IQ4_XS (17.7 GB, 3B active), same binary, 64 greedy tokens:

    ctx prompt_n prefill t/s decode t/s ms/step
    8,192 4,909 607.4 31.53 31.7
    32,768 21,227 755.2 33.99 29.4
    131,072 82,017 387.0 28.15 35.5

    A 35B-A3B MoE decodes at 26-34 tok/s at 82k tokens (30-38 ms/step, 1.4 GiB of KV) where Flash-Next at 18,782 tokens
    measures 79 ms/step. "HRX decode degrades with context" is not a general property of the route; the residual is
    Flash-Next's own layer mix, which is why the model-free layer probes below are the right instrument.

    3. Candidate killed: the HRX program-cache trims do not cost decode time

    Flash-Next at 18.8k needed 66 programs against a 64-entry cache and showed 933 trims in a 40-minute run, so
    GGML_HRX_GRAPH_PROGRAM_CACHE=128 was one of the three ranked candidates. Measured on Qwen3.6 at 82k with decode-only
    reps (the first request pays the prefill, the next two reuse the KV), arms alternating so the box's 8-15%/hour drift
    lands on both:

    arm trims decode t/s (2 reps)
    cache default (64) 99 31.47, 33.20
    cache default (64), second server 99 30.97, 32.22
    GGML_HRX_GRAPH_PROGRAM_CACHE=128 89 33.47, 33.35
    GGML_HRX_GRAPH_PROGRAM_CACHE=128, second server 89 30.30, 31.58

    Indistinguishable. Caveat stated so this is not over-read: that model does not overflow the cache the way Flash-Next was
    counted doing, so this weakens the candidate rather than closing it on the real model. It also means the 128 setting is
    not worth a model row before something else does.

    4. The 36 GDN layers: 15.0 ms of the step, and an instrument that is not yet numerics-clean

    ~/wt/hrx-probe/gdn_layer.cpp rebuilds one GDN layer's decode step from src/models/qwen4exp.cpp
    (build_layer_attn_linear + build_conv_state_at) at the GGUF's real shapes (state 128, K heads 16, V heads 48, inner
    6144, conv 4, channels 10240, the file's own Q6_K projections) and sums N of them into one graph:

    layers HRX0 p50 per layer CPU p50
    1 582 us 582 913 us
    12 5,597 us 466 8,043 us
    36 14,973 us 416 21,866 us
    • 36 layers = 15.0 ms of the decode step, context-flat by construction (fixed recurrent state).
    • Drift over a fixed 36-layer graph, 600 reps: first quarter 14,993 us vs last quarter 14,976 us, ratio 0.999.
    • With the 12 QSA layers at 10-13 ms and the ~13 ms short-context floor, ~40 ms of the 79 ms 18.8k step remains
      unattributed
      , which is now the whole question.

    The caveat, measured rather than assumed. That graph's output is wrong on HRX: 1,400 of 2,560 values are nonfinite
    and the finite ones are 29 distinct garbage values repeated 40 times each (~2e8), byte-identically across processes,
    and unchanged when I shrink the probe's leaf ranges to mild values. I stage-bisected it by copying each layer-0
    intermediate into a leaf (the only reliable read-back on HRX):

    stage HRX nonfinite max|err|/max|ref| vs CPU
    llm.ssm_conv out 0 3.3e-3
    v (convolved) 0 3.3e-3
    recurrent-state gather (786k) 0 6.1e-3
    common.rmsnorm_gate out (normed) 0 4.7e-3
    gated (ngated) 0 5.8e-3
    the activation into the closing ssm_out matvec 0 5.8e-3
    layer output (ggml_mul_mat(ssm_out, final), Q6_K x 1-token f32) 1400 -

    So the recurrent math, the conv state, the state gather and the gated norm are all correct, and the corruption is in the
    last node: a quantized-weight matvec with a one-token f32 activation. The shapes are not the issue either - I parsed both
    GGUFs, and Bonsai-2-27B (qwen35) has the identical GDN config (group_count 16, time_step_rank 48, inner 6144, state
    128) and runs correctly on HRX with the same fused kernel. Switching that closing projection from Q6_K to f32 (same
    graph, same values, one env var) makes the output finite and equal to the CPU reference (rms 4.20265 vs 4.20273,
    NMSE 1.8e-4) and removes kquant.mul_mat.decode_f32 from the dispatch list, while the same kernel on a leaf activation
    is correct in the same graph. So the recurrent math is right and the single broken node is the quantized-weight matvec
    whose activation comes from another fused dispatch rather than from a leaf - the shape of a missing activation staging,
    not a coverage or shape problem.

    What that costs the numbers above: the 15.0 ms is a dispatch/shape timing, and with the file's own Q6_K weights it is
    timed on a graph whose last node mis-stages its activation. I am quoting it as provisional, not as a validated row.
    The ~40 ms unattributed arithmetic is unchanged either way.

    And one thing this lane should say out loud about its own evidence: every number in this issue and in my earlier
    comments is a speed row - the 76 GiB model has been memory-blocked for four days, so no KLD, needle or even
    coherence check has ever been run on Flash-Next on HRX. If the model's GDN layers take the path found broken above (in
    the model the projection may instead be fused into the GDN dispatch - ..._projection_epilogue exists in the kernel
    catalog, and GGML_HRX_LOG_DISPATCH=1 on a real load would answer it in one line), the model would still generate tokens
    at 12.67 tok/s and nobody would notice. Before the next parity number is quoted, the speed row and a quality check should
    be taken in the same window.


    Not run, and why: the 18.8k model row. Its own waiter logged 18:27 -> 19:50 on pool >= 88 GiB and
    MemAvailable >= 92 GiB; the pool never exceeded 72 GiB and MemAvailable peaked at 66 GiB, because the owner's
    1bit forward-serve holds ~30 GiB of GTT. The gate is not being lowered: the one attempt near the ceiling was the
    #680 reset.

  10. bong-water-water-bong commented on Nov 4, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The long-context config surface is empty, and here is the log evidence — so nobody spends a GPU window on these two names.

    This lane measured the 27B UD-Q4_K_XL at 20,480 ctx (HL_LONGCTX=18800, 18,762 tokens as the server counted them). Its numbers of record: 14.5893 tok/s decode (pair-2 half A, one claim) — not Flash-Next, not the 12.67 of the issue. Two sweepable names were left untested at long context (GGML_HRX_GRAPH_PROGRAM_CACHE, GGML_HRX_Q8_PREFILL_RELAX); both are now ruled out from the artifacts rather than by another arm:

    1. The graph-program cache never trims in the long arm. grep -c 'graph program cache trimmed' over all 11 leg logs (serve.log, llama-bench.log, kld.log) of the four pair-2 arm dirs → 0 in every one, including the arm that did the 18,762-token prefill and the 128-token decode. A lever that sets the count cap to 0 (graph_cache_unbounded) cannot move a cache that never evicts, and it does not touch the separate 8 GiB byte budget (plan_evictions(budget, max_programs, …) applies both; GGML_HRX_GRAPH_PROGRAM_CACHE_MB defaults to 8192 and is not in the grid).
    2. Trims happen in the KLD/perplexity leg, not the scored legs: 163 / 164 / 92 in the HL_UB=1024/2048 explore runs — trimmed (1 of 39 programs, 8583 -> 8323 MiB of arenas, budget 8192 MiB, 64 max). A quality-leg cost at widened chunk shapes; no score depends on it.
    3. q8_prefill_relax_off is a prefill-routing knob (docs/hrx.md: prompt matmuls on the q8_1 x4 kernel, 94 % of HRX prompt time, prompt chunks 256–2048), and =0 restores the older, slower routing. The long-context score here is the decode rate, so it can only cost the 320 tok/s prefill and cannot raise decode.

    Combined with what this issue's thread already has — HL_MTP_MAX=6 measured harmful at depth in both orders (−10…−18 % decode, −24…−27 % prose/short, draft acceptance 0.6746 → 0.4381), HL_DFLASH refused by committed code (#773) after faulting the HRX graph at 18.8k on 2026-11-01, and HL_UB/HL_B/-ctk/-ctv not reaching 1bit serve at all — there is no configuration lever left for the 18.8k decode rate. The remaining gain is attention-kernel work, which is not this lane's.

    Separately, so the artifacts above are attributable from now on: #947 → PR #948 adds bench_summary.device_stack = {kernel, hsa_runtime, hsa_runtime_sha256} to every measurement, because the report named the binary (#933) but not the kernel, and this box rebooted five times in 18 h.

  11. bong-water-water-bong commented on Nov 4, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    #679 follow-up (pi-attn, strixhalo): the re-JIT price is measured, and it is the first thing in this issue with a number in the right order of magnitude

    Model-free work only; the 18.8k row is still blocked (see the end). One number matters here:

    A graph-program re-JIT costs ~41 ms, and it is flat in graph size

    Measured by timing the first graph_compute of a fresh graph against the immediately following one - the second is
    served from the HRX program cache - same graph, same leaves, one gpu-hold claim, 0 faults:

    graph nodes first second price of a re-JIT steady p50
    1 GDN layer 44 41.9 ms 0.94 ms 40.9 ms 0.65 ms
    12 GDN layers 528 52.0 6.04 46.0 ms 5.39 ms
    36 GDN layers 1584 58.4 15.68 42.7 ms 15.29 ms
    36 layers, cache limit 1, twice 1584 56.8 / 59.6 15.8 / 18.5 41.0 / 41.1 ms 15.5 / 15.3 ms

    It is compile+load overhead, not per-command emission, so any single trim costs the same whether it is a small decode
    program or a large prefill one; and it is 3-4x the whole steady decode of a 36-layer GDN graph. (First compute in a
    cold process is 110-136 ms; 41 ms is the warm figure, cross-checked by the steady p50 in those same runs scaling with
    the clock ramp and the JIT delta scaling with it.)

    Why this matters for this issue, and what it does not say

    This model's own 18.8k server log carries 933 trim_graph_programs ... (1 of 66 programs, ... 64 max) lines in one
    40-minute run, and 779 in the pre-rebuild run of the same row; each trim evicts one program and frees ~6 MiB, so the
    8192 MiB byte budget is never the constraint. Candidate (a) was "raise GGML_HRX_GRAPH_PROGRAM_CACHE; one env var, and
    it either moves the step or kills the candidate" - ranked on intuition because nobody had priced a re-JIT.

    It is priced now, and it sits inside the unexplained ~44 ms/step: at the observed rate of ~1 trim per 10 decode steps it
    is ~4 ms/step; at 1 trim per step it would be ~41 ms/step, i.e. the whole residual. Which of those it is, is
    still unmeasured
    , because it needs the 18.8k row itself.

    So I shipped the cheap half as data rather than guess: engine #1006 adds one built-in recipe,
    hrx-graph-program-cache-128 (hrx + qwen4exp -> GGML_HRX_GRAPH_PROGRAM_CACHE=128), with the trim counts and this
    price table in its measured field and an explicit "the tok/s row is not measured" sentence. It is a recipe and not a
    default change for the same reason. Raising the count does not preallocate: the byte budget still caps the cache, and
    128 slots of ~6 MiB are ~768 MiB against a 120 GiB GTT budget.

    One retraction that removes a red herring from this issue's history

    The "GDN layers produce 1400/2560 nonfinite, diverging from CPU" rows were my probe, not the engine: gdn_layer.cpp
    bound every tensor in one usage-ANY buffer, so the packed Q6_K weight layout was requested and never repacked - the
    same class as the retracted #969. With the weight tensors in their own WEIGHTS-flagged buffer the fused GDN graph is
    non-finite-free (0/2560) and its timing rows stand unchanged. A bounded difference from CPU remains (rel_rms
    ~9 %, no NaN) which I am not claiming as a defect: the delta-net recurrence runs 128 steps in f16 on device against f32
    on CPU. The next step if anyone picks it up is ~/lb/tattn-gdn-f32.sh (f32 vs Q6_K projections through the same
    comparison), which separates the quantised path from the fused op.

    Still the same block

    The 18.8k row needs the 76.33 GiB file with the box to itself: the load alone measured >= 109 GiB of GTT on
    2026-11-04 (against a 120 GiB ceiling and #680's 106.5 GiB reset regime), and this box has carried other tenants' serves
    (32-40 GiB) continuously since 10-31. Nothing in this comment is a substitute for that row, and none of it is a tok/s
    claim.

  12. 122 remaining items

  13. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The 6.4 tok/s is the quant's kernel path, not long-context work -- measured, same run, same binary (23:52Z)

    The open question after the first IQ3_XXS run was whether 6.39 tok/s was (a) the 3-bit subject taking a different, slower kernel path, or (b) attention/context work that the 4-bit subject also pays. It matters because (a) means the 3-bit number cannot be compared to the bar at all, while (b) would mean the bar is simply lower here than the 20.0 target.

    Same model, same binary (9dd1a1b72), same flags; only the prompt length changes:

    ctx= 3000   rep1  prompt_n= 2758  pp=56.0  tg=6.80
    ctx= 3000   rep2  prompt_n= 2758  pp=91.7  tg=7.82
    ctx=18800   rep1  prompt_n=18609  pp=72.6  tg=6.41
    ctx=18800   rep2  prompt_n=18609  pp=84.1  tg=6.63
    
    decode tok/s note
    2,758 tokens 7.31 (6.80-7.82, spread 1.15x)
    18,609 tokens 6.52 (6.41-6.63, spread 1.03x)
    4-bit subject, historical 11.09 different binary, so see below

    A 6.7x increase in context costs 10.8 % of decode. The 4-bit subject is 1.70x faster than the 3-bit one. So of the gap between the 3-bit subject and the 4-bit numbers, roughly 83 % is the quant and 17 % is context -- and the context part matches the historical 4-bit behaviour, which was 11.09 at 19k and ~10.4 at 32k (nearly flat).

    What this settles: the bar cannot be re-baselined on UD-IQ3_XXS. A 3-bit subject measures a 1.7x-slower decode path, so any number taken on it is a statement about that path rather than about the decode work the row is chasing. UD-IQ4_XS needs 96.1 GB (93.7 weights + 0.46 KV + 2.0 buffers) against the 94 GB this box gives with coder-llm stopped, measured twice; UD-Q4_K_XL needs 113.8 GB against a 104 GiB HRX pool ceiling (the engine#680 reset regime), so it was never feasible.

    So the row's open question is unchanged and is the one already ORCH'd: free ~4 GB in the window for a 4-bit subject (fewer co-tenant embed servers during it), or restate the bar per quant. What is now closed is the alternative I was asked to consider -- "just measure it on the smaller quant and compare" -- with numbers rather than a hunch.

    One caveat kept honest: the historical 11.09 was taken on a Nov 11 build and the bench binary was rebuilt today, so the 1.70x is a quant effect with a possible binary component. The 3k-vs-18.8k comparison, by contrast, holds both of those fixed, which is why it is the part of this that can be stated without qualification.

  14. bong-water-water-bong commented on Nov 15, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Measured in a 12-min gpu-hold window (06:33:15–06:35:30Z, coder-llm stopped inside the ticket and confirmed back on :8090). The dispatch-log instrument now runs end to end, and it answers the matcher question this thread has been carrying.

    Both requests complete

    short  (~216 tok)   http=200   decode 35.39 tok/s   prefill  543.2 tok/s   prompt 251
    long   (~15765 tok) http=200   decode 20.02 tok/s   prefill  150.0 tok/s   prompt 15765
    

    Three blockers fell in order, each fixed and none of them the GPU: the wrapper serves /v1/completions (not /completion, which is the child's warm-up path); a body sent with curl -d is form-labelled, so httplib's 8 KiB form cap answered an empty 413; and the long prompt was 411 tokens over n_ctx (18800 passed as words × 1.111 = 20891 tokens vs 20480), which the server now reports itself as exceed_context_size_error.

    The matcher answer: exactly one registration is long-context-only

    Slicing the dispatch log by the request boundaries (line ranges, not marker text — see below):

    first matched during STARTUP:        24   (not attributable to either request)
    first matched on the SHORT request:   0
    first matched on the LONG request:    1
    ONLY AT LONG CONTEXT:  common.rmsnorm_binary_q8_1_x4   (fused)
    

    From the log itself, immediately after the long request starts:

    0.25.404.768 I slot launch_slot_: id 0 | task 48 | processing task
    ggml_hrx dispatch: common.rmsnorm_binary_q8_1_x4 (fused)
    0.30.213.876 I slot print_timing: … prompt processing, n_tokens = 2048, progress = …
       … to 15761 tokens, prompt eval 105121.57 ms / 15765 tokens
    

    So the "a different matcher is chosen at long context" hypothesis has exactly one named suspect, and everything else in the long path was already registered during startup or the short request. That does not make it the cause — a matcher registered once is used on every subsequent iteration, so a slow one shows up in the per-token time, not in a count. Next step is a per-kernel profile of long-context decode to see whether rmsnorm_binary_q8_1_x4 dominates, rather than more matcher census.

    Prefill at 16k, measured per band (one run, so bands are not separable from noise)

     2048 ->  4096 tok : 0.07 s = 29257 tok/s
     4096 ->  6144 tok : 0.09 s = 22756 tok/s
     6144 ->  8192 tok : 0.12 s = 17067 tok/s
     8192 -> 10240 tok : 0.55 s =  3724 tok/s
    10240 -> 12288 tok : 0.18 s = 11378 tok/s
    12288 -> 14336 tok : 0.21 s =  9752 tok/s
    14336 -> 15249 tok : 0.52 s =  1756 tok/s
    15249 -> 15761 tok : 0.06 s =  8533 tok/s
    

    Aggregate 150 tok/s, and the band-to-band spread (1.7k–29k) is larger than any trend in a single run — so no claim about where prefill falls over until there are three runs.

    Two instrument defects found by disagreement, both fixed on my side

    • fa-matcher-diff.sh attributed phases by marker TEXT, but the markers are only ever written to the sidecar — the serve owns the log — so sed -n '/SHORT/,/LONG/p' matched nothing and the run printed "25 on the SHORT request, 0 on the LONG": the startup set counted as short-context and an empty range counted as long-context. It now slices by the sidecar's line numbers, and on the same log it agrees with the report script and with a hand count (0 short, 1 long, the same candidate).
    • My window wrapper wrote its WINDOWS.log start line at queue time rather than grant time, so the record looked like two windows overlapping (mine "started" 06:31:46 while another lane still held the token until 06:33:11). gpu-hold had serialised them correctly; only the log lied. The line now comes from inside the ticket.

    What this does not claim

    One run, at 15,765 tokens, not at the bar's 18,782 — the bar is a median of 3 at 18,782 and this is a single sample below it, so it neither meets nor refutes it. It does place decode at 20.02 tok/s at 15.8k, against the 9.70 tok/s at 18.8k that opened this issue.

  15. bong-water-water-bong commented on Nov 15, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Window 6 result: median decode 20.30 tok/s at 18,545 tokens, three reps, and the long-context-only matcher is the same one in all three.

    rep 1   prompt 18545   decode 20.30 tok/s   prefill 126.5 tok/s
    rep 2   prompt 18545   decode 20.54 tok/s   prefill 132.7 tok/s
    rep 3   prompt 18545   decode 19.99 tok/s   prefill 129.5 tok/s
    median 20.30   spread 1.028x        (bar: 20.0 tok/s at the 18,782-token prompt, median of 3)
    

    gpu-hold window 07:27:06–07:35:54Z, coder-llm stopped inside the ticket and confirmed back on :8090 afterwards; no kernel-log fault lines.

    Two honest qualifications

    1. The context is 1.3 % short of the bar's. 18,545 vs 18,782 tokens — my word→token conversion (×0.9 for this prompt) landed 237 tokens low. The bar is 20.0 at 18,782, so this is evidence toward it, not formal satisfaction of it. The conversion is measured now: 16,903 words → 18,545 tokens = 1.097 tokens/word, so LONG=19027 puts the next window at 18,782 exactly (19027 + 32 + 512 ≤ 20,480 ✓).
    2. Single window, three reps, one box state. Spread is 1.028× (19.99–20.54), which is tight — tighter than the 1.114× I measured for the same model earlier tonight — so the median is a fair summary of this state, not of every state.

    The matcher answer repeats exactly

    All three reps produced the same one candidate, by line-range attribution:

    FIRST SEEN ONLY AT LONG CONTEXT (the candidates):
      common.rmsnorm_binary_q8_1_x4
    

    That is now four independent runs (one at 15,765 tokens earlier, three here at 18,545) naming one op and nothing else. So the "a different matcher is chosen at long context" hypothesis has a single, reproducible suspect — and a matcher registered once is used on every iteration afterwards, so the per-token cost question still needs a per-kernel profile, not another census.

    What I would run next, in this order

    1. LONG=19027, three reps — the bar's own context, to convert "20.30 at 18,545" into the exact number the bar is written against. ~9 min in a 15-min window.
    2. Then, if the per-kernel share matters: profile decode with rocprofv3 (the engine's GGML_HRX_REPORT prints placement, not time — I checked) and see whether rmsnorm_binary_q8_1_x4 dominates the per-token time.
  16. bong-water-water-bong commented on Nov 15, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    At the bar's own context now: two samples above 20.0 tok/s at 18,785 tokens — but n = 2, so the median-of-3 is not yet satisfied.

    window 10:08:17–10:20:04Z, LONG=19027 -> prompt 18785 tokens   (bar: 18,782 -- within 3 tokens)
    
    rep 1   short 200 (34.12 tok/s)    long 502  "no timings"        <- request failed, no figure
    rep 2   short 200 (38.47 tok/s)    long 200  decode 21.21 tok/s   prefill 131.5
    rep 3   short 200 (34.45 tok/s)    long 200  decode 21.84 tok/s   prefill 136.0
    

    So the two valid reps are 21.21 and 21.84, both above the bar's 20.0, at the bar's context (within 3 tokens) rather than 1.3 % short as last time. What that does not do is meet the bar's own definition — median of 3 — and I am not going to average two samples and call it a median. The 502 on rep 1 is a real event, not a harness slip: the short request in the same rep succeeded, so the server was up; the long request came back 502 with no timings.

    One more instrument bug, mine, found by rep 1

    Rep 1's candidate list contained common.so alongside the real name:

    FIRST SEEN ONLY AT LONG CONTEXT (the candidates):
      common.rmsnorm_binary_q8_1_x4
      common.so
    

    common.so is a library, matched by my own extraction pattern common\.[a-z0-9_]+ — so any shared object mentioned in the log could appear as a "matcher first seen at long context". Fixed: the pattern now requires an underscore (common\.[a-z0-9_]*_[a-z0-9_]+), because matcher names in this tree carry one (common.rmsnorm_binary_q8_1_x4) and object suffixes do not. Verified: the real name still matches, common.so no longer does. Reps 2 and 3 never showed it, which is exactly why a single run is not enough to trust a candidate list.

    The state of the question

    • The one long-context-only matcher is stable: common.rmsnorm_binary_q8_1_x4, present in every rep that completed and in the two earlier windows (15,765 and 18,545 tokens) — five runs now.
    • The bar: two of three samples exceed 20.0 at 18,785 tokens. A third rep in a later window settles it; until then the honest statement is "2/2 valid samples above the bar, one request failed".
  17. bong-water-water-bong commented on Nov 15, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The bar is met at its own context: median 21.60 tok/s at the 18,782-token prompt, four samples, and every one of them above 20.0.

    window 11:30:31–11:36:10Z, LONG=19027 -> prompt 18,785 tokens (bar: 18,782 — within 3)
    
    rep 1  short 200 (36.44 tok/s)   long 200  decode 20.97 tok/s  prefill 127.6
    rep 2  short 200 (35.60 tok/s)   long 200  decode 21.60 tok/s  prefill 130.5
    
    with the two valid samples from the previous window (21.21, 21.84 at the same 18,785 tokens):
    
      all four:  20.97  21.21  21.60  21.84
      median of three:  21.60        median of four:  21.41
      min 20.97   max 21.84   spread 1.041x
    
      bar: 20.0 tok/s decode at 18,782, median of 3        ->  MET
    

    The minimum is above the bar. This is not "one lucky sample carries the median": the four span 4.1 %, and the worst of them (20.97) still clears 20.0. For comparison, the number that opened this issue was 9.70 tok/s at 18.8k — so this is roughly 2.2×, at the same prompt length the bar is written against rather than 1.3 % short of it.

    How to read it

    • Configuration: the flint 40B-A2.5B GGUF through 1bit serve --no-strict-device (the flag is required while engine#2450's two tied embd GET_ROWS are off HRX), GGML_HRX_LOG_DISPATCH=1, -c 20480 --parallel 1, in a gpu-hold ticket with coder-llm stopped inside it — so the number is not sharing the GPU with the PR-agent model.
    • What the instrument answers besides the number: every rep names one long-context-only dispatch, common.rmsnorm_binary_q8_1_x4, now across six runs (15,765 tokens once, 18,545 ×3, 18,785 ×2). That hypothesis — "a different matcher is chosen at long context" — has a single reproducible suspect and nothing else, in six attempts.
    • Honest limits: one box state, four samples, one build; the spread across windows was wider earlier in the night (1.114×) than within this one, so treat 21.5 as the median of this state rather than a promise. The --no-strict-device flag means the run does not assert placement, only speed.

    Where the credit belongs

    The decode number moved from 9.70 to ~21.5 across this thread's fixes, which came from the attention/mask work — not from anything in this measurement lane. What this lane contributed is the instrument (dispatch-log diff by line ranges, a completion-sized prompt guard against n_ctx, and the completion-based readiness probe that now lives in window-run.sh), and the six-run reproducibility of the one candidate above.

    If a per-kernel share is still wanted, the next step is a rocprofv3 profile of decode — GGML_HRX_REPORT prints placement, not time, so the census cannot answer it — but the bar question this issue was opened for is now answered.

  18. bong-water-water-bong commented on Nov 15, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The HRX dispatch profile works, and the answer is not a matcher threshold: one generic attention kernel takes 86.8 % of the GPU dispatch time in this 18.8k request.

    Instrument chain (each step measured, each one cost me something)

    HRX_PROFILE_MODE=dispatch        -> 085021.iree            44,303,764 bytes   (binary, IRPF magic)
    iree-profile export --format=ireeperf-jsonl
                                     -> 085021.ireeperf.jsonl  257,502,044 bytes   413,484 records
    
    • iree-profile is not on PATH — it exists only as source in ~/amd-oss/hrx-system, and the built binaries are in ~/amd-oss/toolchains/hrx-stock-b142-build/…/iree-profile. Worth a line in TOOLCHAINS.md: the doc names HRX_PROFILE_MODE=dispatch → hrx2kineto.py as if the chain were one step, and it is three.
    • tools/hyperloom/hrx2kineto.py cannot read that output. It filters r.get("type") in ("dispatch_event", "dispatch"), but the v18 interchange emits record_type — so it exits with "no dispatch events in …" on a file holding 53,342 of them. One-word fix (record_type), and it is the kind of mismatch that reads as "the profile is empty".

    The finding, from the events themselves

    262,818 valid dispatch_event records, aggregated by key over duration_ticks:

    86.8%   12171.8 kticks   ggml_attention_sink_f32
     3.9%     548.1          ggml_flash_attention_f32_f16_wmma
     2.8%     386.1          ggml_mul_mat_id_tiled_pair_input_f32_swiglu_publish_f32
     1.6%     219.6          ggml_mul_mat_f32_f32_wmma
     1.3%     176.8          ggml_mul_mat_id_f32_f32_wmma
     1.0%     137.5          ggml_mul_mat_add_f32_f32_wmma
    

    ggml_attention_sink_f32 is 87 % of the dispatch time, and it is not one of the hip./hrx. specialised matchers — which reframes this issue. The long-context cost is not a matcher being chosen late (that was common.rmsnorm_binary_q8_1_x4, 1 of 24 registrations); it is one generic attention kernel doing nearly all the work. That is consistent with the O(n²) shape of a 19,031-token prefill, and it is where the next 2× lives.

    What I am not claiming

    • No phase split. I tried to cut prefill from decode with a marker kernel and the cut was invalid — ggml_kquant_mul_mat_decode_f32 occurs 7,496 times, so it is not decode-only, and my "decode" half was 49 % of the events with the same 87 % attention share. The profile carries no phase markers, so prefill vs decode needs either two runs (n_predict=1 vs n_predict=32, differenced) or the derived_time_* fields decoded to wall clock. I report the whole-request share only.
    • Shares, not times. The tick domain is device_tick and I did not establish ticks→seconds, so the percentages above are rate-independent and the absolute numbers are not offered.
    • The run itself: http=200, prompt 19,031, decode 18.86 tok/s while profiling (vs 21.60 unprofiled — the profiler's own overhead, and 44 MB written for one request).

    The three things worth doing next, in order

    1. Fix hrx2kineto.py's record_type so the documented chain works end to end; until then the doc's one-line description produces an empty result.
    2. Route or replace ggml_attention_sink_f32 for long prefill — 87 % of the time in a kernel that is not a specialised matcher is the largest single lever this profile shows.
    3. Then the phase split, if it is still needed after (2).
  19. bong-water-water-bong commented on Nov 15, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correction to my own wording, and the mechanism behind the 87 %.

    I called ggml_attention_sink_f32 "one generic attention kernel … not one of the specialised matchers". That is wrong, and the file says so plainly:

    amdgpu.target<gfx11-generic> @ggml_attention_sink_gfx11_wave32 {subgroup_size = 32}
    kernel.def target(@ggml_attention_sink_gfx11_wave32) export("ggml_attention_sink_f32") @ggml_attention_sink_f32(…)
    

    ggml_attention_sink_f32 is the export name of the gfx11 wave32 target — the profile records exports, not targets, so the 87 % is the specialised, subgroup-32 kernel for this box. There is no faster sibling being passed over; I checked before repeating the claim.

    What the launch geometry shows

    kernel.launch.config workgroups(%query_head_count, %tokens, %c1) workgroup_size(%c256, %c1, %c1)
    

    One workgroup per (query head × token), 256 lanes, with the key range walked inside the workgroup — so a 19,031-token prefill with this model's head count launches on the order of 10⁵–10⁶ workgroups, each scanning up to 19,031 keys. That is the naive O(n²) shape, and it matches the profile exactly:

    kernel share of dispatch time
    ggml_attention_sink_f32 (gfx11 wave32, per-head×token workgroups) 86.8 %
    ggml_flash_attention_f32_f16_wmma (tiled flash attention) 3.9 %

    So the 22× gap in this profile is tiled vs untiled attention, not matcher selection. The wmma flash kernel is the tiled implementation and it costs 3.9 % of the run; the sink kernel is the same problem without key tiling and it costs 87 %.

    What that means for this issue

    The lever is tiling the sink path (or, where the sinks allow it, running flash and applying the sink correction S / (S + exp(sink − M)) afterwards) — not another matcher threshold. That is a kernel change in loom-libs/ops/attention_sink_f32.loom, and it is the largest single item this profile contains. My earlier three next-steps stand, with (2) sharpened from "route or replace" to "tile it".

    And the thing I still cannot say

    The phase split: the profile carries no phase markers, my marker kernel occurs 7,496 times (so it is not decode-only), and the whole-request share above therefore mixes prefill and decode. Given the geometry and that a 19k prefill is where the O(n²) term lives, the 87 % is almost certainly prefill — but "almost certainly" is not a measurement, and two differenced runs (n_predict=1 vs n_predict=32) would settle it.

  20. bong-water-water-bong commented on Nov 15, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Retracting two things I posted an hour ago, with the correct numbers — and the correct numbers make the finding sharper.

    1. hrx2kineto.py is not broken. My conversion was.

    I wrote "hrx2kineto.py cannot read that output … one-word fix." That is wrong. The script's own argument help says the input is

    iree-profile dispatch --format=jsonl --dispatch_events <file.ireeprof>
    

    and that command emits exactly the "type":"dispatch_event" rows the reader expects:

    iree-profile dispatch --format=jsonl --dispatch_events 085021.iree  ->  80,841,630 bytes
      131409 "type":"dispatch_event"
           1 "type":"dispatch_summary"
    

    I had used export --format=ireeperf-jsonl, a different interchange format (record_type, schema_version 18), which the reader does not read — and I concluded the tool was defective instead of reading its --help. No fix is needed; I have reverted the change I made in my working tree. The wrapper tools/hyperloom/hrx-tracelens.sh already does this correctly (HRX_PROFILE_MODE=dispatch → locate iree-profile → convert → hrx2kineto.py), sitting next to the reader I had misdiagnosed.

    2. My dispatch count was doubled.

    export --format=ireeperf-jsonl emits each event in two record types, so my hand aggregation counted 262,818 dispatches where there are 131,409. The shares were unaffected (same kernel mix on both halves), but the count was wrong and the numbers below supersede them.

    3. The tool's own summary, which is better than my aggregation

    131,409 dispatches, 140,440 ms of dispatch time   (prefill alone was ~140 s wall — this is the prefill)
    
    86.8%  121,949 ms   n=4,067   grid 8192x1    ggml_attention_sink_f32
     3.9%    5,492 ms   n=4,487   grid  256x32   ggml_flash_attention_f32_f16_wmma
     2.8%    3,869 ms   n=1,840   grid 1024x256  ggml_mul_mat_id_tiled_pair_input_f32_swiglu_publish_f32
     1.6%    2,201 ms   n=7,259   grid 1024x16   ggml_mul_mat_f32_f32_wmma
    

    Per call: 30.0 ms for the sink kernel against 1.2 ms for flash attention — a 24.5× gap, at 4,067 and 4,487 calls respectively. Nearly the same number of invocations, a 25× difference in cost each, and the grids say why: 8192×1 (one workgroup per query head per token, keys looped inside) versus 256×32 (tiled).

    So the conclusion of my earlier comment stands and is now measured by the tool rather than by me: the long-context prefill is untiled attention, 87 % of the dispatch time, and the tiled implementation for the neighbouring kernel is 25× cheaper per call. The lever is tiling the sink path (or running flash and applying the sink correction afterwards) — not a matcher threshold.

    4. What is still open

    The phase split. --phase decode needs the trace to contain a decode section, and this run's dispatch time is essentially all prefill (140.4 s of dispatch vs a ~140 s prefill and ~1.7 s of decode), so a decode view needs a request with more decode and fewer prompt tokens. That is a second window with -p 512 -n 512, not something this profile can yield.

  21. bong-water-water-bong commented on Nov 15, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The decode profile — and it shares almost nothing with prefill, which is why the phase split mattered.

    Window 17:07:40–17:08:59Z: LONG=512 TOK=512 (prompt 521 tokens, decode 35.22 tok/s), 846,410 dispatch rows, then hrx2kineto --phase decode:

    kernel share calls grid
    ggml_mul_mat_id_skinny_pair_input_f32_swiglu_publish_f32 28.5 % 24,627 1024×256
    ggml_kquant_mul_mat_decode_f32 24.3 % 102,056 131072×1
    ggml_flash_attention_f32_f16_wmma 8.7 % 25,823 256×32
    ggml_mul_mat_id_decode_f32_wave64 7.1 % 24,581 65536×6
    ggml_kquant_swiglu_decode_f32 5.9 % 25,387 65536×1
    ggml_mul_mat_f32_f32_decode_wave64 4.8 % 24,351 8192×1
    ggml_attention_sink_f32 4.0 % 25,823 8192×1

    The two phases have different levers

    • Prefill (18.8k): ggml_attention_sink_f32 86.8 % — untiled, 30.0 ms/call against 1.2 ms for the tiled flash kernel.
    • Decode (521 prompt): ggml_attention_sink_f32 4.0 %; the expert/matmul mix is 65.8 % (28.5 + 24.3 + 7.1 + 5.9). Attention in total is 12.7 %.

    So an optimisation aimed at the prefill number does nothing for decode, and vice versa. The decode lever is the MoE matmul path — mul_mat_id_* and kquant_*_decode — not attention.

    Does that hold at 18.8k, where this issue's bar lives?

    This profile is at short context (n_kv 512→~1024), so the honest question is whether attention takes over at long context, since attention is O(n_kv) per token and matmul is O(1). Two measurements bound it without another window:

    • decode 35.22 tok/s at ~768 average n_kv, 21.60 tok/s at 18,785 — a 1.63× slowdown across a ~24× n_kv increase.
    • If attention were dominant at 18.8k, the slowdown would have to track n_kv; it tracks it to the 1.6 power of nothing like 24.

    So the matmul mix dominates decode across the whole range and there is no crossover. That also explains why the bar moved from 9.70 → 21.60 without any attention-kernel change: the work that got faster was the matmul side.

    Limits

    • One run, one prompt shape; the tick domain is device_tick, so these are shares, not seconds (profile-top.py deliberately prints no absolute time).
    • --phase decode attributes by the trace's own step boundary; the prefill tail (521 tokens ≈ 0.1 s of the 1.8 s run) is small but not zero, so treat the 4.0 % as an upper bound on prefill bleed, not a floor.
    • The 1.63× bound uses two different runs (512-prompt vs 18.8k-prompt); it is an argument, not a controlled experiment. A single run at 18.8k with 512 decode tokens would settle it, and is the same -n 512 shape at the bar's context.
  22. bong-water-water-bong commented on Nov 16, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Where the prefill 86.8 % comes from — and it is a dispatch ordering, not a slow kernel.

    From the 17:19 profile: prefill at 18.8k is ggml_attention_sink_f32 at 30.0 ms/call against 1.2 ms for the tiled flash kernel — 86.8 % of prefill for one kernel that is 25× slower per call. That reads as "the sink kernel needs optimising". It is not that.

    The selection is structural, not sized. dispatch_registration/common/dispatch-attention-sink.cpp selects on shape: the node is GGML_OP_FLASH_ATTN_EXT with 5 inputs, sinks is F32/contiguous, and sinks->ne[0] == query_head_count with the rest 1. Nothing in it looks at n_kv, the query length, or the batch. So prefill at 18.8k and decode at 521 take the same path, and only the cost per call differs because the sink kernel is untiled.

    And the faster path already accepts the same nodes: the flash-attention registration admits a 5-input node when attention_sinks_supported(...) is true —

    (node->inputs.size() != 4 && !attention_sinks_supported(graph, *node))   // -> not a flash candidate
    

    — i.e. the tiled kernel is already sink-capable. It simply does not get the node, because the sink dispatch is attempted first and succeeds:

    if (!append_attention_sink_dispatch(context.graph, *context.root_node, dispatch_match)) { … flash … }
    

    So the 25× is the price of which registration wins, not of attention sinks being incompatible with tiling.

    The shape of a fix (not made yet)

    Make the sink registration decline long-query nodes so they fall through to the tiled path: a sink node whose query length is above the tile's useful size is exactly the prefill case, and the flash registration already handles sinks. The arm exists — tests/test-hrx-attention-sink.cpp already asserts attention_sinks_supported(graph, node) — so the change can be armed by extending it with a length-based decline and asserting the chosen registration, rather than by adding a new test file.

    Before anyone does that, one measurement is owed: the two registrations must be shown to agree numerically on a sink node at prefill length. If the tiled path's sink handling is numerically different, 25× is the price of correctness and this is a documented constraint, not a bug. That is the decode-window run's sibling and does not need a new issue.

    Caveat on my evidence: read from the cl-flm-2138-bump pin of third_party/llama.cpp; ~/engine's pin does not contain dispatch_registration/, so line numbers and file names here are that pin's, not main's.

  23. bong-water-water-bong commented on Nov 16, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The controlled version of the bound — one serve, A B B A, n_predict=512 on both arms

    The published 1.63× came from two different runs (21.60 tok/s at 18,785 tokens vs 35.22 at a 521-token prompt). This is the same question asked inside one process, with the same decode length on both sides, so context length is the only variable:

    ctx 20480, n_predict 512, one serve on :31491     (gpu-hold 1822 s = 30.4 min wait, die 46 °C, job 223 s)
    
      A  prompt_n=521     decode_n=512  tok/s=38.20    (words=512)
      B  prompt_n=18511   decode_n=512  tok/s=22.38    (words=19027)
      B  prompt_n=1       decode_n=512  tok/s=22.51    (words=19027)
      A  prompt_n=5       decode_n=512  tok/s=38.45    (words=512)
    
      short: median 38.33 tok/s (2 samples)   long: median 22.45 tok/s (2 samples)
      n_kv ratio 35.5x                        decode 1.71x slower at long context
      spread: short 0.25, long 0.13 tok/s
    

    Read the prompt_n column with care — it is the prefix cache, not a mistake. The 2nd and 4th requests repeat a prefix the server has already processed, so their prompt_n is only the uncached remainder (1 and 5 tokens). Both long-context samples therefore decode against the same 18.5k KV, which is better for this comparison than re-prefilling: the arm comparison is decode-against-18.5k-KV vs decode-against-521-token-KV, with no prefill cost inside either sample.

    What it establishes

    • The bound reproduces, slightly wider: 1.71× here against 1.63× published, both at ~24–35× n_kv. The ratio is stable across the two independent methods, which is the point of running it controlled.
    • The bar still holds, from a cleaner measurement: long-context median 22.45 tok/s against the bar's 20.0 at 18,782 tokens — and the bar's own samples were 20.97 / 21.21 / 21.60 / 21.84, so this run sits at the top of that range rather than outside it.
    • The spread is the evidence the ratio is real: 0.25 and 0.13 tok/s max−min over the two samples per arm. A 1.71× difference with 0.3 tok/s of noise is not two runs compared.
    • No crossover: 35.5× more KV for 1.71× less decode speed, consistent with the attention term being a minority of decode time at this size (decode profile: MoE matmul 65.8 %, attention 12.7 %).

    Method, for re-running: bash ~/wt/os3i/679-controlled.sh inside a window-run.sh ticket — one serve, four requests in A B B A, so one load, one residency and no thermal step between arms. It refuses to start without its own ticket (holder must be an ancestor) and, because the installed gpu-hold on this box predates the take-time guard, it runs the 10-minute fault check itself and refuses with rc 78.

  24. bong-water-water-bong commented on Nov 16, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correcting my own comment 11 minutes ago: the tiled flash kernel is NOT sink-capable, so the 25× is the price of correctness, not a dispatch bug.

    I wrote that "the tiled kernel is already sink-capable" and proposed declining long-query nodes from the sink registration. That was wrong, and the method to check it was one grep:

    • kernel-corpus/kernels/loom-libs/ops/flash_attention_f32_f16_wmma.loom — grep -c sink → 0. Its exported entry @ggml_flash_attention_gfx11_wave64 takes (%tokens, %key_count, %key_capacity): no sink argument at all.
    • attention_sink_f32.loom — 62 sink references, same three parameters plus the sink term, and it ships its own ggml_attention_sink_reference_f32 with an explicit %use_sink flag.

    So a 5-input sink node cannot be served by the tiled path. Routing prefills there would silently drop the sink term — which is exactly the failure the sink registration exists to prevent.

    What line 226 actually means. (node->inputs.size() != 4 && !attention_sinks_supported(graph, *node)) reads as "5-input nodes are fine if sinks are supported". But append_attention_sink_dispatch is attempted first (line 631) and claims every node for which attention_sinks_supported is true — so that admission never fires for a sink node. It is a guard for a case the ordering makes unreachable, not a capability claim. I read the guard as the capability and mistook dead code for a plan.

    What that leaves. The 25× is the honest cost of attention sinks on this backend: an untiled kernel that carries the extra term, versus a tiled one that cannot. So the real options are:

    1. Add the sink term to the tiled kernel — the only route to prefill speed here, and a kernel change, not a dispatch change.
    2. Document the constraint — "sink models pay 30 ms/call at prefill; non-sink models take the tiled path" — so nobody spends a window chasing a dispatch fix that cannot exist.

    What I got wrong in method: I inferred capability from a guard's negated condition instead of from the kernel's parameters. Two greps — grep -c sink on both .loom files — would have settled it before I posted. The correct predicate for "can the tiled path take sinks" is the kernel's own argument list, not the dispatcher's admission test.

    Caveat unchanged: read from the cl-flm-2138-bump pin; ~/engine's pin does not contain dispatch_registration/ or the loom corpus, so this is that pin's state.

  25. bong-water-water-bong commented on Nov 16, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Where this row stands, on the issue's own table

    Three measurements from this lane, mapped onto the arms above. The short version: decode is bounded and understood; prefill is the open gap, and it has one named cause.

    issue's arm long tg (18.8k) what our profile says
    c1 — our HRX route 9.70 → 21.60 (bar 20.0 met) decode = 65.8 % MoE matmul, 12.7 % attention
    b3 — fork, --moe-cache off 31.14 1.44× above us with neither side drafting
    c3 — fork + cut-vocab MTP, n=2 47.87 2.22× above us — drafting, a different axis from the kernel gap

    1. The decode gap is not a long-context collapse any more

    Controlled A/B (one serve, four requests A B B A, n_predict=512 both arms — the same question the original finding asked, with context as the only variable):

    short 521 tok   38.33 tok/s median   (spread 0.25)
    long  18,511    22.45 tok/s median   (spread 0.13)
    n_kv 35.5x  ->  1.71x slower         (published two-run bound: 1.63x)
    

    So the 3.2× collapse the issue opened with (9.70 → 31.15) is gone: what remains is a 1.71× attention cost at 35× n_kv, and that is consistent with the decode profile (attention is 12.7 % of dispatches; MoE matmul is 65.8 %). Long context is no longer where our decode loses to the fork — the remaining 1.44× to b3 is in the kernel mix, not in context length.

    2. Prefill is the open gap, and it is one kernel

    The issue's prefill line (74–93 vs 459) is a 5× gap, and the profile names the cause:

    kernel share of prefill ms/call grid
    attention_sink 86.8 % (121,949 ms / 4,067 calls) 30.0 8192 × 1
    flash — 1.2 256 × 32

    Same work class, 24.5× the per-call cost, and the grid is the tell: attention_sink launches 8192 blocks in one dimension (×1 in the other) where flash uses 256 × 32. A 1-D grid that wide with 32× fewer total blocks per call is a kernel that is not distributing the head/sequence dimension — which is exactly the shape GEAK is for.

    3. What I would do next on this row, in order

    1. GEAK on attention_sink prefill — it is 86.8 % of prefill and 24.5× flash's per-call cost with a visibly wrong grid shape. A 2–4× there closes most of the 74–93 → 459 gap without touching decode.
    2. The MTP axis is separate and worth a row of its own — c3 at 47.87 vs b3 at 31.14 is a 1.54× from drafting alone, and it is not a kernel question. If our route wants to be compared with c3 rather than b3, the comparison has to say which.

    What this lane has delivered on #679

    Bar met and posted (21.60 median at 18,785 tok/s, four samples above the 20.0 bar); prefill profile (86.8 % attention_sink, 30.0 ms/call, grid 8192×1); decode profile (65.8 % MoE matmul, 12.7 % attention); and the controlled A/B above, which is what turns the original two-run comparison into a ratio with a spread. The row stays open on prefill, and item 1 is the next measurement this lane would take if it gets the kernel.

  26. bong-water-water-bong commented on Nov 16, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correction to my own prefill diagnosis: the grids are the same size, and my inference from them was wrong

    Two hours ago I wrote, on this issue:

    "attention_sink launches 8192 blocks in one dimension (×1 in the other) where flash uses 256 × 32. A 1-D grid that wide with 32× fewer total blocks per call is a kernel that is not distributing the head/sequence dimension."

    The block counts are identical: 8192 either way. 8192 × 1 and 256 × 32 are the same number of workgroups, differently factored, and I read a factorisation as a deficit. Checked at the source rather than from the profile:

    // addons/hip-hrx/dispatch-hip-attention-sink.cpp:29
    geometry.workgroup_count = { static_cast<uint32_t>(heads->second),
                                 static_cast<uint32_t>((tokens->second + 63) / 64), … };
    
    // addons/hip-hrx/kernels/hrx_attention_sink.hip
    const uint32_t head = blockIdx.x;                    // one workgroup per head
    const uint32_t t0   = blockIdx.y * 64 + wave * 16;   // 64 tokens per workgroup, in 16-token waves
    __global__ void __launch_bounds__(128) …

    So the shape is (heads, ⌈tokens/64⌉) with 128 threads per workgroup, and a 8192×1 line in a profile is that product flattened — not a 1-D launch.

    What survives, and what the next step actually is

    • The measurement stands: attention_sink is 86.8 % of prefill (121,949 ms / 4,067 calls = 30.0 ms/call) against flash at 1.2 ms/call — 24.5× per call on the same shape. That gap is real and is still the largest single lever on this row.
    • The diagnosis does not: 24.5× the time on the same workgroup count is work per workgroup, not parallelism. Either the kernel does far more arithmetic per call (a sink kernel may not be windowing the way its name suggests), or its call shape differs (tokens per dispatch) in a way the profile summary does not show.
    • So the GEAK target changes: not "fix the grid", but the kernel body and its per-workgroup work — with the first question being what is tokens for these 4,067 calls, because a call that covers 64 tokens needs 294 calls for an 18.8k prompt and the per-call cost would then be dominated by launch/parity rather than arithmetic.

    I would rather retract this now than have someone aim a GEAK run at a grid that was never wrong. The rule I keep relearning: a profiler field is a report of the artefact, not the artefact — the dispatch source is the latter.

  27. bong-water-water-bong commented on Nov 16, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The open question from my retraction, answered from a capture already on disk: attention_sink is compute-bound, and 99 % of its cost is one grid shape

    I retracted the "32× fewer blocks" diagnosis two hours ago and left one question: is the 30.0 ms/call launch overhead or work? That needed the per-dispatch grid dimensions rather than the flattened line, and it turns out I still had them — t2.csv (13 MB, kept when I deleted the raw dumps) carries Grid_Size_X/Y/Z per row.

    By shape, not by average

    attention_sink total ticks 121,949,281,196
      grid=8192 x 512   n=1813   share= 99.0%   ticks/call=66,618,466
      grid=8192 x  83   n=  49   share=  0.8%
      grid=8192 x   1   n=1862   share=  0.1%   ticks/call=    78,914
      grid=8192 x   4   n= 147   share=  0.0%
      (three more shapes, each <0.1%)
    

    The 1,862 calls with the small grid — the ones a "launch overhead" theory would blame — are 0.1 % of the total. 99.0 % is in 1,813 calls at 8192 × 512, i.e. 4.19 M workgroups each.

    What those numbers are

    The dispatcher computes workgroup_count = {heads, (tokens + 63) / 64} (addons/hip-hrx/dispatch-hip-attention-sink.cpp:29), so:

    grid ⌈tokens/64⌉ tokens what it is
    8192 × 512 512 32,768 the full context — the run used -c 32768, and 32768/64 = 512 exactly
    8192 × 1 1 ≤ 64 a single-token call (the cheap tail)

    And the constant 8192 is the flattened q dimension: 64 heads × 128 head_dim = 8192 ✓. (The parameter is named heads in the dispatcher; for this op it is heads × head_dim.)

    So the lever is the sink kernel at full context, not its launch count

    • 86.8 % of prefill time is attention_sink, and within it 99.0 % is the full-context shape — 1,813 dispatches each covering 32,768 tokens across the flattened q dimension.
    • The per-call overhead theory is dead: the many tiny calls are numerically irrelevant.
    • GEAK target, corrected again and for the right reason: the work inside hrx_attention_sink.hip at 32k tokens — the sink scan itself — not the grid, and not the dispatch count. For reference, flash_attention in the same capture runs 256×32/8192×32/4096×32 (2-D, y = 32) at 3.9 % of prefill, so the two kernels are not doing comparable work per call and their timings should not be read as "same shape, 24.5× slower" — which is what my retracted note said.
    • Retained data beats a new window: this analysis needed no ticket and no GPU. The lesson I am taking is to keep the derived CSVs (they cost 13 MB and answer questions the raw dumps were deleted for).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

backlogReal, but does not hold the release buildperfMeasured performance gap against a named target

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions