Skip to content

HRX: Qwen3.8-Flash-Next prefill is 287 tok/s on UD-IQ4_XS vs 1,679 for gufo on the same box (and 500-2,400 across published Strix Halo engines) #572

Closed

Description

@bong-water-water-bong

Split out of #390 (whose reporting half is done). With the HIP add-on the Flash-Next experts run on HRX (fork #142/#144, gpu-kernels #38/#39), and the file now measures PPL 1.8090, prefill 287 tok/s, decode 24.0 (UD-IQ4_XS, HRX0, no drafter; UD-IQ3_XXS 297 / 26.9). The field on the same chip reports prefill 510 (tuned Vulkan llama.cpp at 8k), 1,175-1,309 (halogen), 1,679-2,013 (gufo), 2,372 (hipfire), and decode 33-60 with drafters.

What gufo does that we do not (from the 2026-10-11 comparison, memory gufo-benchmark): prefill batches up to 4096 (dense) / 8192-token chunks (MoE), WMMA quant GEMMs or dequant-to-BF16 hipBLASLt for big shapes, FP16 activations, WMMA attention, MMQ experts with fitted tile widths, int8 WMMA for Q8_0 dense. Our expert prefill path is the HIP mmid GEMM at m64 tiles; the IQ2_S kernel GEAK authored is latency-bound at ~135 GB/s (52%).

Ask: a Flash-Next prefill plan on HRX: (1) dispatch profile of pp512 / pp4096 on UD-IQ4_XS with owners and the CPU splits (engine #389: -ub > 512 fails res=-3 on HRX, so chunking is capped), (2) expert-GEMM tile width fitted to the 512x640x2560 routed shapes, (3) the ub cap, (4) an MTP head on HRX for Flash-Next for the decode side. Numbers to beat: 1,679 prefill (gufo, same file), 33 decode (Vulkan llama.cpp with MTP).

🤖 Generated with Claude Code

Activity

  1. bong-water-water-bong commented on Oct 28, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The claim still holds at the current pin, and the numbers are now in-tree rather than in a memory file — docs/npu-leaderboard.md:27-29:

    Flash-Next (IQ4_XS 4.25 bpw) | hrx | prefill pp512  | 286.63 tok/s | ngl=99, fa=1, ub=512,  engine=0d0c846
    Flash-Next (IQ4_XS 4.25 bpw) | hrx | prefill p1024 ub576 | 269.34 tok/s | ngl=99, fa=1, ub=576, engine=0d0c846
    Flash-Next (IQ4_XS 4.25 bpw) | hrx | prefill pp4096 | 260.59 tok/s | ngl=99, fa=1, ub=512,  engine=0d0c846
    

    So 287 tok/s vs gufo's 1,679 is real and reproducible on paper, and nothing in the tree since has moved it: the #622 tiled transpose and the #623 first-token pair are 27B NPU rows (435.9 s -> 145.7 s, docs/npu-leaderboard.md:34-35), not this prefill path. The gap analysis in the issue body (batch size, WMMA/dequant GEMM, FP16 activations, MMQ experts, int8 WMMA for Q8_0, latency-bound IQ2_S expert kernel at ~135 GB/s) is the right shape and is unimplemented.

    Classification: needs GPU measurement — specifically llama-bench -fa 1 -ngl 99 -ub 512 -p 512 (3 repeats, and -ub 576 -p 1024 plus -p 4096 to match the three rows) on UD-IQ4_XS with the HIP add-on, on the current pin and again on 40c4ccb76e48 (the #610 bump target), reported as a table against 286.63 / 269.34 / 260.59. Without that row the next change to the expert GEMM has no before-number to beat. Left to the measurement lane — this box is CPU-only today.

  2. bong-water-water-bong commented on Oct 30, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Two things, and the second one is a retraction.

    (a) Where this should look next. ggml_mul_mat_f32_f32_wmma is 8.6% of device time at pp512 — 56.8 ms of the pass — over 522 dispatches in the profiled bundle (docs/hrx.md:414-422), inside a 4.25 bpw model. That is a defect regardless of the per-layer accounting. One correction to the mechanism, though: it is not "some projection is falling back to an unquantized matmul" (docs/hrx.md:429-430). The operands are the file's own F32/BF16 tensors — 48 ffn_gate_inp (F32 [2560,512] router), 96 hc_{attn,ffn}_inject (F32 [10240,4]; N=4, so every quantized matcher declines), 72 ssm_alpha/ssm_beta (F32 [2560,48]), 24 indexer.{q,k}_proj (BF16) — 240 of the 261 per-forward f32 matmuls (perf-572-findings.md). ~95 GFLOP inside 56.8 ms ≈ 1.7 TFLOPS against ~9.5 for the expert GEMM, so the cost is dispatch overhead on tiny shapes, not a bad matmul. The lever is fusing or dispatch-batching those, not quantization.

    (b) Retraction. An earlier reading today held that the bundle contains two forwards (92 swiglu dispatches = 2 × 46 layers) and therefore that device time was 37% rather than 76% of the pass. That does not survive checking. The GGUF says qwen4exp.block_count = 48 (header read of Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf, also qwen4exp.expert_count = 512), and the dispatch counts divide by neither 48 nor 46 — 92, 86 and 522 are 4×23, 2×43 and 2×261. So docs/hrx.md:409-411 stands as written and the 76% (1,355 ms of the 1,786 ms pp512 pass) is not being changed. The per-layer attribution inside the bundle is unresolved, not settled in the other direction either.

    Next: the profiler run that settles per-layer counts — dispatch counts resolved against block_count with the bundle's own op names, at ub 512 and again at ub 2048, so kernel counts can be attributed per layer instead of divided by assumption.

  3. bong-water-water-bong commented on Oct 30, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Item 1: the host gap, measured. It is not there on any model that fits on this box, and the ~1.0 s/forward premise does not survive either instrument I could bring to it.

    First the premise. The brief takes device time as 755.7 ms of the 1,786 ms pp512 pass (41.9 %) and the rest as host. This file's own profile records 1,355 ms of 1,786 ms as device (76 %) (the section merged as #616), and comment (b) above re-affirmed that 76 % after retracting the two-forwards reading. Those two splits cannot both be quoted: at 76 % the host leg is 431 ms, not 1,030 ms. The 755.7 ms figure is not in this issue or in docs/hrx.md.

    Two instruments, both on engine 0d0c846 + llama.cpp 8ee20a0 + gpu-kernels 2b626798a.

    1. A host-path profiler (GGML_HRX_HOST_GAP=1, llama.cpp ggml-hrx: an env-gated host-gap profiler (engine #572) llama.cpp#154, our own new file, off by default) that times every phase of a graph compute. Fully offloaded, llama-bench -ngl 99 -fa 1 -ub 512 -p 512 -dev HRX0, steady-state forward:
    model forward graph build (outside the backend) lookup bind upload flush wait (device)
    Qwen3-Coder-30B-A3B Q4_K_M, 1,161 dispatches 263.5 ms 1.5 ms 0.004 ms 0.002 ms 0.005 ms 0.001 ms 263.46 ms
    Qwen3.8-27B UD-Q4_K_XL, 1,519 dispatches 1369.0 ms 0.9 ms 0.003 ms 0.003 ms 0.004 ms 0.001 ms 1368.89 ms

    Every item the brief names is in that row: graph build 1.5 ms, program-cache lookups 4 µs, host copies 5 µs, GET_ROWS fallback absent (one graph, no split, no CPU node), wait spin inside replay.wait where the device leg is. Total host: 1.5 ms of 263.5 ms, 0.6 %.

    1. libhrx's own dispatch profile (HRX_PROFILE_FILE=<bundle> HRX_PROFILE_MODE=dispatch, then iree-profile statistics) — the only profiler that sees HRX dispatches:
    model dispatches in bundle device kernel time wall for the same 2 forwards device/wall
    Qwen3-Coder-30B-A3B Q4_K_M 2,322 = 2 × 1,161 538.14 ms 523.8 ms 103 %
    Qwen3.8-27B UD-Q4_K_XL 3,038 = 2 × 1,519 1,871.15 ms 1,657.2 ms 113 %

    Both instruments agree on the dispatch counts to the unit (1,161 and 1,519), and device kernel time is the whole forward; the >100 % is the profiler writing records.

    rocprofv3 cannot do this job, and that may be where the 41.9 % came from. rocprofv3 --kernel-trace --stats on an HRX run exits 0 and writes no trace file: HRX dlopens TheRock's libhsa-runtime64.so.1 itself and libhrx writes its own AQL packets, so the profiler's HSA interposition never sees a dispatch. Anything summed from a "device trace" that returns nothing, or from a partial one, gives an inflated host leg.

    What a host leg looks like when there is one. The same 27B at -ngl 20 (a third of the layers on the CPU) runs 3,746 graph computes per forward instead of 2, and its host-staging upload alone costs 145.7 ms per forward (39 µs per graph), with the CPU backend's own compute on top. That is the shape of a real host component: a graph that is not fully on HRX.

    Counts the last comment asked for (GGML_HRX_HOST_GAP=1, dispatches per forward):

    model p/ub 512 p/ub 1024 p/ub 2048
    Qwen3.8-27B UD-Q4_K_XL 1,519 1,616 1,813
    Qwen3-Coder-30B-A3B Q4_K_M 1,161 — 1,137

    Resolved against block_count: 27B 62 layers → 24.5 dispatches/layer at ub 512, 29.2 at ub 2048; 30B-A3B 48 layers → 24.2 and 23.7. The counts are stable, so per-layer accounting can be done from them, but the timings are not: the same 27B pp512 read 828.6 / 870.2 / 1120.8 / 1369.9 / 1518.2 ms across five runs on this box today (±57 %). Any before/after on this lane needs repeats inside one gpu-hold claim.

    What I could not measure. Flash-Next UD-IQ4_XS is 93.7 GB (87.3 GiB) of weights; this box has 75–82 GiB available with the owner's 27B NPU job resident, so loading it would swap and every number from it would be a swap number. UD-IQ3_XXS (76.4 GiB) is the one Flash-Next file that could be tried here, and only with that job stopped — your call, not mine. One run of the profiler on either file answers this issue's item 1 for Flash-Next itself.

    On your item (a) (the f32 matmuls), from the same profile: on the 27B they are fused — common.mul_mat.f32_f32_wmma appears in the matcher census as fused and no ggml_mul_mat_f32_f32_wmma kernel row appears at all — so the 522-dispatch / 8.6 % shape is Flash-Next's own (hc_*_inject N=4 each getting a dispatch). The batching fix already has a precedent in this tree on a file that fits: qwen3_moe_router_projection_f32_four_row_wave32 batches the 30B-A3B's f32 router projection — 16 dispatches per forward, 3.3 % of its device time, instead of one per token.

    PRs, neither merged: #646 (docs/hrx.md, the whole profile above) and 1bit-MONSTER/llama.cpp#154 (the profiler).

  4. bong-water-water-bong commented on Oct 30, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Follow-up with the long-prompt case closed, since it was the one place the host path did grow.

    -p 4096 -ub 2048 on Qwen3-Coder-30B-A3B Q4_K_M, steady-state graph, every step named (llama.cpp #154 gained the remaining scopes: cache.key, replay.total, replay.rebind, replay.transient, replay.launch, replay.download):

    wall 1158.581 ms = graph.lookup 0.179 + graph.bind 0.016 + graph.execute 1158.385
      execute          = cache.key 0.007 + replay.total 1145.534 + ~12.8 ms (cache mutex/prologue)
      replay.total     = replay.wait 1143.718 + rebind 0.002 + transient 0.001 + upload 0.004
                         + launch 0.042 + flush 0.001 + download 0.002
    

    so host time outside the device wait is ~15 ms of a 1158.6 ms forward (1.3 %), and at ubatch 512 it is 0.03 ms of 263.5 ms (0.01 %). The 145 ms/forward I first saw at this ubatch was the one-time cost of the ubatch's first graph — prepare plus kernel JIT for the new shapes, 4.5 s above the following graphs — not a per-forward path.

    Summary of what item 1 asked for, for every model that fits on this box:

    ub 512 ub 2048
    host per prefill forward outside the device wait 0.03 ms (0.01 %) ~15 ms (1.3 %)
    graph build (llama.cpp side) 1.5 ms ~2 ms
    program-cache lookup 4 µs 0.18 ms
    host copies (bind + staging) 7 µs 25 µs
    GET_ROWS fallback absent, one graph, no split same
    mutex/wait spin inside replay.wait, which is the device leg same

    That is as far as item 1 goes on this box. Items 2 and 3 both need the 87 GiB Flash-Next file: 2 has no host item left to fix, and 3 is a device-side defect that only exists in that model's graph. One decision unblocks both — stop the owner's 27B NPU job, or accept UD-IQ3_XXS (76.4 GiB vs 75-82 GiB available) — and the run is one command each now that the profiler and the libhrx dispatch bundle are both in place.

    PRs, neither merged: #646 (docs/hrx.md) and 1bit-MONSTER/llama.cpp#154 (the profiler, twelve phases, off by default).

  5. bong-water-water-bong commented on Oct 30, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Item 1 is closed, on the file itself, and item 2 has a one-line target. The owner's NPU job exited at 04:41 (114 GiB free), so I loaded the 87.3 GiB UD-IQ4_XS and measured it.

    Qwen3.8-Flash-Next UD-IQ4_XS, -ngl 99 -fa 1 -ub 512 -p 512 -dev HRX0
      llama-bench                      1,795.4 ms/forward = 285.2 tok/s   (in-tree row: 286.63)
      HRX graph computes per forward   2 graphs: 2,566 dispatches in 618.6 ms, and a 1-dispatch one in 0.2 ms
      host phases inside HRX           <= 0.13 ms total (lookup, bind, cache key, transient, rebind,
                                       upload, launch, flush, download)
      libhrx dispatch bundle           2,567 dispatches/forward, 1,353.5 ms of device kernels per 2 forwards
                                       = 676.8 ms/forward = 37.7 %
      llama-server -lv 4 placement     6060 nodes, 1 off HRX (GET_ROWS x 1), 3 graph splits
    

    So: 676.8 ms device (37.7 %), 0.13 ms of HRX host path, and ~1,177 ms (65.6 %) in one CPU node. The ~1.0 s is real; it is GET_ROWS fallback, the last item on this issue's own suspect list, and it is a single declined dispatch:

    • the node reads per_layer_token_embd.weight, which this file carries as IQ3_S [160, 320001536] — a ~20 GB table (token_embd.weight, the ordinary one, is Q8_0 [2560, 248320]);
    • ggml/src/ggml-hrx/dispatch_registration/common/dispatch-get-rows.cpp declines it at is_supported_row_count(): kMaxGetRowsRowCount = 262208 (line 23) against 320,001,536 rows;
    • the CPU then gathers 512 rows out of 20 GB per forward, and the cost is the page faults of touching it: ~35 ms at -p 128 (565.5 ms/forward) vs ~1.18 s at -p 512. Super-linear in tokens, because more tokens means more distinct rows and more cold pages. The graph splits 3 ways around it (HRX → CPU → HRX), so the device and host legs cannot overlap.

    Item 2 is therefore: let that gather stay on HRX — raise/relax kMaxGetRowsRowCount for a table this shape, or add a dispatch for a gather from a table this large. Ceiling if it lands: the forward goes from 1,795.4 ms to ~677 ms of device plus whatever the gather costs on HRX, i.e. up to ~2.3x pp512 on this file — which is the ~2.4x roof in the brief, arrived at from the other end.

    A correction to the tree while I was there: docs/hrx.md's "1,355 ms of the 1,786 ms pass (76 %)" is a two-forwards device total measured against a one-forward pass. 1,353.5 ms is exactly 2 × 676.8 ms from the bundle I measured. Per forward the device leg is 37.7 %, so the brief's 41.9 % was the closer figure and the published 76 % should come out. Fixed in #646.

    Controls (a graph with no off-HRX node has no host leg at all): Qwen3-Coder-30B-A3B Q4_K_M 263.5 ms/forward with 0.02 ms of host phases and 1,161 dispatches; Qwen3.8-27B UD-Q4_K_XL 870.2 ms with 0.01 ms and 1,519. Both stay ~100 % device. rocprofv3 collects nothing on HRX (it exits 0 and writes no traceback files — HRX dlopens TheRock's libhsa-runtime64.so.1 itself and libhrx writes its own AQL packets), so the device leg has to come from HRX_PROFILE_FILE + iree-profile statistics.

    PRs, neither merged: #646 (docs: the whole profile, the correction, the GET_ROWS target) and 1bit-MONSTER/llama.cpp#154 (the host-path profiler, twelve phases, off by default).

  6. bong-water-water-bong commented on Oct 30, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The Flash-Next prefill forward on a gfx1151 roofline (TraceLens at ff705ed; no GPU run needed for the projection).

    Method: #1175's TraceLens_model_roofline_projection parses a HF checkpoint, and this model exists here only as GGUF — qwen4exp is a hybrid llama.cpp architecture with no HF counterpart. So I used the same machinery a level down: TraceLens.PerfModel's GEMM / flash_attention formulas over shapes and exact per-tensor bytes read from the GGUF directory, against the peaks TraceLens keeps for gfx1151 after #1172 removed the bundled Radeon_8060S.json (measured spec already on this box: 256 GB/s, 59.39 TFLOP/s fp16, 118.78 TFLOP/s int4). Measured device time per class is from the libhrx dispatch bundle of the same model, so each row is roof against what the device actually did.

    class weights GB touched GB FLOPs T mem roof ms compute roof ms measured ms
    MoE experts 59.52 59.52 2.416 232.5 20.3 306.1
    gated-delta-net layers (36) 2.25 2.25 2.126 8.8 35.8 33.7
    token embedding + LM head 1.20 1.20 0.651 4.7 11.0 (dense)
    full-attention layers (12) 1.02 1.02 0.671 4.0 11.3 136.3
    MoE shared expert + router 0.50 0.50 0.306 2.0 5.2 (dense)
    hyper-connections 0.34 0.34 0.644 1.3 10.8 35.1
    PLE table 28.80 0.002 0 0.0 0.0 0.5
    total 93.67 64.87 6.81 253 115 676.8

    touched < weights because at 512 tokens with top-10 of 512 experts the expected number of untouched experts per layer is 512·e⁻¹⁰ = 0.02 — the expert weights are compulsory traffic — while the 28.8 GB PLE table is only gathered, one 90-byte IQ4_NL row per token per layer (2.2 MB). So the 87.3 GiB file does not put a 366 ms floor under a 512-token pass: 253 ms is the floor.

    Three consequences:

    1. The forward is memory-bound by 2.2× (253 ms against a 115 ms compute roof). No compute-side kernel rewrite can beat 253 ms on this file, which is worth knowing before anyone chases the expert GEMM's tile width again.
    2. The device leg is at 37 % of its own compulsory-traffic roof — 2.7× the floor. The MoE expert GEMMs are the class closest to roof (306.1 ms against 232.5 ms, 76 %), so the slack is not there: 1,189 elementwise/copy/norm dispatches cost 106.6 ms with no weight traffic at all (~90 µs each, dispatch-bound), 273 f32 small matmuls cost 58.4 ms, and 395 dense projection GEMMs 136.3 ms.
    3. The host leg is the largest single line in the forward: the one CPU GET_ROWS costs ~1,177 ms = 4.6× the memory roof of the entire pass, to move 2.2 MB. It is page faults on a cold 28.8 GB mmap'd table, not arithmetic — which is why it does not warm up between forwards (1,788.6 then 1,802.1 ms), while the 77 device-side get_rows dispatches cost 0.5 ms.

    So item 2's ceiling, in roofline terms: give that gather a dispatch and the forward becomes ~620 ms of device, i.e. ~2.9× pp512 (285 → ~830 tok/s), and the file then sits at 37 % of its memory roof. The remaining gap to the floor is the dispatch-bound small ops — item 3's f32 batching is 58.4 ms of it and the elementwise/copy stream 106.6 ms.

    Reproduce: ~/lb/i572/roofline_fn.py <gguf-prefix> <iree-profile statistics --format=jsonl> 512. The same table is in #646.

  7. bong-water-water-bong commented on Oct 30, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Status: item 1 shipped, item 2 merged-but-unproven, item 3 is the same file's graph. Both remaining measurements are now blocked on #669, and I have stopped running the 87.3 GiB file on HRX.

    What shipped. The host-path profiler (llama.cpp #154) and the GET_ROWS row bound (#159) are both merged; the docs section in docs/hrx.md (#646) is merged. Fork 1bit/hrx-vulkan-patched is now at 5e6388a5a852281c6929303c1219665b2a6d4c0a, and the next HRX pin bump moves the engine's third_party/llama.cpp from d8d2f5b to it, carrying six commits: fe04824 (profiler), b8861a9 (#155 KV padding), 9e8453a/f61f290/01d7df4 (#156/#157/#158 dsv41) and 5e6388a (#159, this issue's item 2). Note for whoever raises it: tools/check_pins.py gates only the hrx-system pin against config/hrx-long-prompt-validated-pin.config, and that pin does not move in this bump — so this llama.cpp-only bump does not need the long-prompt probe re-run.

    Item 2 numbers, partial and honest. #159 alone is not enough, which is what the placement analysis predicted:

    Item 3 is parked for the same reason, and I checked it rather than assuming. From the dispatch bundles already on disk: on Qwen3.8-27B UD-Q4_K_XL there is no ggml_mul_mat_f32_f32_wmma row at all — the f32 matmuls are fused (common.mul_mat.f32_f32_wmma (fused) in its matcher census), with only hrx_dp_f32_to_f16 (246 dispatches, 10.9 ms over 2 forwards) as the f32 traffic. On Qwen3-Coder-30B-A3B the router projection is already batched (qwen3_moe_router_projection_f32_four_row_wave32, 16 dispatches/forward). So Flash-Next's 273 standalone f32 dispatches / 58.4 ms are its own hc_*_inject (N=4) and ssm_alpha/beta shapes, and there is no runnable file on this box that exercises that pattern — a fix could be written but not validated, which the lane rules do not allow.

    So: item 2 needs #669 (or the owner freeing the box for one long run with the table on the CPU and the gather dispatched — not viable at 88.7 GB), and item 3 needs item 2's file. I would rather leave both recorded than publish numbers from another thrash event.

  8. 47 remaining items

  9. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Orchestrator: parked on the owner word of 2026-11-11 (performance work waits; correctness issues come first). No lane is assigned until it is un-parked.

  10. bong-water-water-bong commented on Nov 12, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    cl-1077 (#572 lead from 2026-11-12, cl-hrxkern supporting): baseline at the current pin, the top five dispatches, and the first change.

    Run: one gpu-hold ticket, 15:46–15:56Z, coder-llm stopped for the window and back at 15:55:54Z (:8090/v1/models 200). Engine main's llama.cpp pin 962507f8e + release add-on (cl-hrxkern's build, library sha256 5093dd59be29, ldd checked). Model Qwen3.8-Flash-Next-UD-IQ3_XXS (3 shards, 81.95 GB), -dev HRX0 -ngl 99 -fa 1, default -b 2048 -ub 512, MemAvailable 98–108 GiB throughout.

    cell tok/s samples
    pp512, llama-bench -p 512 -n 0 -r 2 716.10 ± 8.98 709.75, 722.45
    pp2048, same arm no number the arm hit its 420 s timeout: the pp512 row was only written 6 min 43 s after the arm started (rows take 0.7 s), so the first graph's setup took the window
    pp2048 under tools/hrx_dispatch_profile.py (serial mode) 611.17 1 sample; serial dispatch adds a sync per launch, so read it as a lower bound for an unprofiled pp2048

    Top five dispatches by device time, whole profiled pp2048 run (warmup + 1 rep, 19,958 dispatches, 5,345.7 ms device):

    # role kernel ms share dispatches
    1 routed expert gate/up (MoE, SwiGLU fused) hrx_prefill_mmid_swiglu_iq2_s_m64_w4 1084.6 20.3 % 368
    2 routed expert down (MoE) hrx_prefill_mmid_iq4_nl_m64_w4 734.0 13.7 % 376
    3 q6_K dense projections ggml_mul_mat_q6_k_f16_wmma_prefill_wave32 722.6 13.5 % 1232
    4 f32 x f32 matmul (an unquantized projection; docs/hrx.md already flags it) ggml_mul_mat_f32_f32_wmma 477.4 8.9 % 2088
    5 q8_0 dense projections hrx_dp_mm_q8_0_m64_w8_f2_sb 347.6 6.5 % 760

    The two routed-expert GEMMs are 34.0 % of device time. At -ub 512 each launch routes 512 tokens x 10 experts over 512 experts: about 10 rows per expert against a 64-row tile. That is the mechanism behind the ubatch gains already measured on this thread and in docs/hrx.md (-ub 2048: 1.16x at pp2048 and 1.60x at pp4096 in llama-bench on this IQ3_XXS file, 11-09; 2.03x at pp4096 on the product path, 11-05).

    The first change. 1bit serve starts the chat llama-server without -b/-ub (app/serve.cpp: only the mmproj, embedding and rerank launches set them). So the product prefills Flash-Next at llama-server's default ubatch of 512, which is the slow arm of every pair above. I will make 1bit serve pass -b 2048 -ub 2048 on HRX for a GGUF whose header declares experts (<arch>.expert_count > 0). Dense models keep the default, because docs/hrx.md's dense rows put 512 at the optimum. An explicit -ub from the user still wins. The PR comes with an argv unit test and the docs. The acceptance pair is product-path pp512/pp4096 through 1bit serve, before and after, in one window: that needs a second memory window, requested through the orchestrator.

    Artifacts (box-local): ~/.cache/scratch-cl-1077/572/run-20261112T154603Z/ (run.log, bench/prof jsonl, the 20 MB serial trace and the per-program JSON).

  11. bong-water-water-bong commented on Nov 12, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    cl-npurt (lead, claimed 11-12 14:4xZ after the unpark). Ask (2) is in review: 1bit-MONSTER/gpu-kernels#67 (the routed-expert prefill GEMM fitted to this model's routed shapes), plus 1bit-MONSTER/llama.cpp#329 (test cases at the file's real 512 experts).

    What it does:

    • On UD-IQ4_XS, about two thirds of the routed gate/up and down blocks were not on the HIP prefill path at all. They now are.
    • The existing HIP formats get a better-fitted width for the routed down shape and for IQ2_S at larger batches.

    Proof:

    • Op-level measurements and the correctness record are in the private PR.
    • On HRX0 against the CPU, the new 512-expert cases pass with the defaults and with both escape hatches (GGML_HRX_HIP_MMID_TILE_FIT=0, GGML_HRX_HIP_MMID_IQ4XS_Q5_0=0). The dispatch log confirms each arm takes the path it claims.

    Not measured yet: the model-level number. The paired llama-bench pp512/pp4096 on UD-IQ4_XS and UD-IQ3_XXS (default vs both knobs off) needs a ~100 GiB memory window; it is on an ORCH line behind the correctness rows. Expect UD-IQ3_XXS to move little: its experts were already on the HIP path. The gap to 1,679 tok/s is not closed by this PR. The next lever named in the PR is the IQ3_XXS gate/up blocks that still run on the generic path.

  12. bong-water-water-bong commented on Nov 12, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    CLOSE CONDITION

    cl-1077 (#572 lead).

    The bar in the body cannot be reached this week. The body asks for 1,679 tok/s prefill (gufo, UD-IQ4_XS) and 33 tok/s decode.

    Proposed bar (for the orchestrator to accept or change):

    On engine main with every PR below merged, tools/bench.py --prompt-chars 18000 (one ~4,100-token prompt, 3 reps, interleaved arms, one window) through 1bit serve --device hrx -c 8192 -fa on --no-strict-device gives prefill of:

    • ≥ 500 tok/s on Qwen3.8-Flash-Next-UD-IQ3_XXS;
    • ≥ 220 tok/s on Qwen3.8-Flash-Next-UD-IQ4_XS (the issue's file).

    The same run must show no pp512 regression against the pre-#2289 config (-ub 512, gpu-kernels fd69590c6). Asks (1)–(3) of the body are then answered; the rest of the gap to 1,679 is not this issue's close condition.

    Where the numbers come from:

    • 500 (UD-IQ3_XXS): -ub 2048 measured 548.94 at pp4096 on this file (11-09, older pin), and 611 at pp2048 today under the serial profiler.
    • 220 (UD-IQ4_XS): 205 measured at pp4096 with -ub 2048. gpu-kernels#67's own arithmetic is about 0.25 s saved per 512-token graph on that file.

    --no-strict-device is needed because today's serve run refused the model under strict placement: two input-embedding GET_ROWS nodes were off HRX at an 87 GiB pool. It moves the same two nodes in both arms.

    What is left, as PRs (no slices beyond these):

    1. engine#2292. A recipe's -ub no longer replaces a user's LLAMA_ARG_UBATCH; this is a follow-up to recipes: Flash-Next (qwen4exp) on HRX prefills at -ub 2048 (#572) #2289. READY twice, waits for merge.
    2. engine#2316 (pi-pkg). It moves the gpu-kernels pin fd69590c6 → eb0f3839, which brings in gpu-kernels#67, the routed-expert tile fit (merged, with its tests in llama.cpp#329), and docs/lemonade.md: the embedding checklist (measured) #68 (hrx: 4 of 10 identical 8.8k-token prefills on flint give different logits (a block of 16 tokens goes non-finite mid-network); usually silent, sometimes NaN logits and a server abort #2159's fix). Without that pin move the product does not run the fitted tiles. Not a new PR of mine.
    3. The close evidence: docs/hrx.md's HRX: Qwen3.8-Flash-Next prefill is 287 tok/s on UD-IQ4_XS vs 1,679 for gufo on the same box (and 500-2,400 across published Strix Halo engines) #572 section.

    If PR 3's numbers miss the bar, I will post the measured ceiling and the reason rather than add slices.

  13. bong-water-water-bong commented on Nov 13, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    #572 acceptance, row 1 of 2: UD-IQ3_XXS passes the bar (652.0 tok/s, bar 500). cl-1077, lead.

    Setup:

    UD-IQ3_XXS, 5,742-token prompt prefill reps, tok/s reported
    main (-ub 2048 recipe, tile fit) 483.6, 622.1, 648.3, 711.2, 653.7, 652.0, 646.6, 675.8 652.0 (median of the last 3)
    pre-#2289 config: llama.cpp 962507f8e, gpu-kernels fd69590c6, -ub 512 (00:10Z, a different window) 231.0, 434.6, 478.7 434.6 (median; only 3 reps, still rising)

    Decode on main was 25.1, 25.2 and 25.6 tok/s on code, prose and short prompts.

    Two corrections to my CLOSE CONDITION, at the same command and bar:

    1. Prompt size: --prompt-chars 18000 is 5,742 tokens on this model's tokenizer (from the server's prompt eval lines), not the "~4,100" I wrote. The bar is set by the command, and the real prompt is longer, so it is the harder test.
    2. Reps: the run takes 8 reps, reported as the median of the last 3, not 3 reps. The first reps compile each new graph shape. With 3 reps the same build gave 49 / 179 / 372 at 05:42Z, which measures the compiles and not the rate.

    The pre-#2289 row shows the shape of the change and is not a paired A/B: it is a different window with fewer reps. The no-regression check at about 512 tokens is still owed. Main gave 504.7, 598.1 and 651.0 at 647 tokens with 3 reps (rerun with 6). The old-config arm runs in its own window, because its server needs more than 8 minutes to become ready.

    Still to run:

  14. bong-water-water-bong commented on Nov 13, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    #572 acceptance, row 2 of 2: UD-IQ4_XS passes the bar (387.3 tok/s, bar 220). cl-1077, lead.

    Same setup and command as row 1:

    • Engine: main b1a4a97.
    • llama.cpp: pin e6c80014e.
    • gpu-kernels: b0ed2060e.
    • Command: tools/bench.py --prompt-chars 18000 (5,742 tokens), 8 reps, reported as the median of the last 3.
    • Window: one self-service window, 14:42:47–14:55:16Z. coder-llm was down 12.5 min and :8090 answered 200 afterwards.
    main, Qwen3.8-Flash-Next-UD-IQ4_XS prefill reps, tok/s reported
    bar, 5,742-token prompt 377.7, 480.8, 510.4, 377.4, 380.1, 329.9, 387.3, 398.1 387.3 (every rep is above 220)
    regnew, 647-token prompt 325.7, 448.7, 531.9, 530.3, 533.6, 518.2 530.3

    Decode was 18.0–19.7 tok/s.

    Both bar rows have passed: UD-IQ3_XXS at 652.0 ≥ 500 and UD-IQ4_XS at 387.3 ≥ 220.

    The remaining item on the accepted bar is the pp512 no-regression check. The new side is now measured on both files at 647 tokens: UD-IQ3_XXS gave 651.0 as the last of 3 reps, and UD-IQ4_XS 530.3. The pre-#2289 old side (llama.cpp 962507f8e, gpu-kernels fd69590c6, -ub 512) runs next, one window per file. I will close #572 when that check holds.

  15. bong-water-water-bong commented on Nov 13, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    #572 acceptance, no-regression check, and close. cl-1077, lead.

    The no-regression check on UD-IQ3_XXS passes. Prompt: tools/bench.py --prompt-chars 2200, which is 647 tokens. Prefill is the median of the last 3 reps.

    UD-IQ3_XXS, 647 tokens before #2289 (-ub 512, gpu-kernels fd69590c6, pin 962507f8e) main b1a4a97 (pin e6c80014e)
    prefill tok/s 584.3 (577.7, 584.3, 584.5) 598.1 (504.7, 598.1, 651.0)
    decode tok/s (code prompt) 24.9 26.3
    window 11-13 17:32Z 11-13 07:25Z
    • Command: both arms ran through 1bit serve --device hrx -c 8192 -fa on --no-strict-device. The "before" arm used a recipes file without qwen4exp-hrx-ub2048.
    • Caveat: the two arms ran in separate windows. The difference is small, so read it as "no regression", not as a speed-up.

    UD-IQ4_XS no-regression check: not run. On main this file measured 530.3 tok/s at 647 tokens (14:42Z window). Before its "before" arm could run, the owner had both local Flash-Next quants deleted (ISSUE-OWNERS.md, ORCH 11-13). The box's Flash-Next file is now UD-Q4_K_XL, so this row cannot be run on the files the bar names.

    Result against the accepted bar:

    • Prefill, UD-IQ3_XXS: 652.0 tok/s, bar ≥ 500. Pass. See the row 1 comment.
    • Prefill, UD-IQ4_XS: 387.3 tok/s, bar ≥ 220. Pass. See the row 2 comment.
    • No regression: UD-IQ3_XXS passes, as above. UD-IQ4_XS is not measurable now.

    PRs #2289 (the recipe), #2292 (the user's -ub wins) and #2316 (the kernel pin) are merged.

    Closing #572. A UD-Q4_K_XL baseline would be new work for Flash-Next, not a slice of this issue. Reopen if the orchestrator wants the missing UD-IQ4_XS row re-run on a re-downloaded file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    backlogReal, but does not hold the release buildperfMeasured performance gap against a named target

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions