You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
HRX: Qwen3.8-Flash-Next prefill is 287 tok/s on UD-IQ4_XS vs 1,679 for gufo on the same box (and 500-2,400 across published Strix Halo engines) #572
Split out of #390 (whose reporting half is done). With the HIP add-on the Flash-Next experts run on HRX (fork #142/#144, gpu-kernels #38/#39), and the file now measures PPL 1.8090, prefill 287 tok/s, decode 24.0 (UD-IQ4_XS, HRX0, no drafter; UD-IQ3_XXS 297 / 26.9). The field on the same chip reports prefill 510 (tuned Vulkan llama.cpp at 8k), 1,175-1,309 (halogen), 1,679-2,013 (gufo), 2,372 (hipfire), and decode 33-60 with drafters.
What gufo does that we do not (from the 2026-10-11 comparison, memory gufo-benchmark): prefill batches up to 4096 (dense) / 8192-token chunks (MoE), WMMA quant GEMMs or dequant-to-BF16 hipBLASLt for big shapes, FP16 activations, WMMA attention, MMQ experts with fitted tile widths, int8 WMMA for Q8_0 dense. Our expert prefill path is the HIP mmid GEMM at m64 tiles; the IQ2_S kernel GEAK authored is latency-bound at ~135 GB/s (52%).
Ask: a Flash-Next prefill plan on HRX: (1) dispatch profile of pp512 / pp4096 on UD-IQ4_XS with owners and the CPU splits (engine #389: -ub > 512 fails res=-3 on HRX, so chunking is capped), (2) expert-GEMM tile width fitted to the 512x640x2560 routed shapes, (3) the ub cap, (4) an MTP head on HRX for Flash-Next for the decode side. Numbers to beat: 1,679 prefill (gufo, same file), 33 decode (Vulkan llama.cpp with MTP).
So 287 tok/s vs gufo's 1,679 is real and reproducible on paper, and nothing in the tree since has moved it: the #622 tiled transpose and the #623 first-token pair are 27B NPU rows (435.9 s -> 145.7 s, docs/npu-leaderboard.md:34-35), not this prefill path. The gap analysis in the issue body (batch size, WMMA/dequant GEMM, FP16 activations, MMQ experts, int8 WMMA for Q8_0, latency-bound IQ2_S expert kernel at ~135 GB/s) is the right shape and is unimplemented.
Classification: needs GPU measurement — specifically llama-bench -fa 1 -ngl 99 -ub 512 -p 512 (3 repeats, and -ub 576 -p 1024 plus -p 4096 to match the three rows) on UD-IQ4_XS with the HIP add-on, on the current pin and again on 40c4ccb76e48 (the #610 bump target), reported as a table against 286.63 / 269.34 / 260.59. Without that row the next change to the expert GEMM has no before-number to beat. Left to the measurement lane — this box is CPU-only today.
(a) Where this should look next.ggml_mul_mat_f32_f32_wmma is 8.6% of device time at pp512 — 56.8 ms of the pass — over 522 dispatches in the profiled bundle (docs/hrx.md:414-422), inside a 4.25 bpw model. That is a defect regardless of the per-layer accounting. One correction to the mechanism, though: it is not "some projection is falling back to an unquantized matmul" (docs/hrx.md:429-430). The operands are the file's own F32/BF16 tensors — 48 ffn_gate_inp (F32 [2560,512] router), 96 hc_{attn,ffn}_inject (F32 [10240,4]; N=4, so every quantized matcher declines), 72 ssm_alpha/ssm_beta (F32 [2560,48]), 24 indexer.{q,k}_proj (BF16) — 240 of the 261 per-forward f32 matmuls (perf-572-findings.md). ~95 GFLOP inside 56.8 ms ≈ 1.7 TFLOPS against ~9.5 for the expert GEMM, so the cost is dispatch overhead on tiny shapes, not a bad matmul. The lever is fusing or dispatch-batching those, not quantization.
(b) Retraction. An earlier reading today held that the bundle contains two forwards (92 swiglu dispatches = 2 × 46 layers) and therefore that device time was 37% rather than 76% of the pass. That does not survive checking. The GGUF says qwen4exp.block_count = 48 (header read of Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf, also qwen4exp.expert_count = 512), and the dispatch counts divide by neither 48 nor 46 — 92, 86 and 522 are 4×23, 2×43 and 2×261. So docs/hrx.md:409-411 stands as written and the 76% (1,355 ms of the 1,786 ms pp512 pass) is not being changed. The per-layer attribution inside the bundle is unresolved, not settled in the other direction either.
Next: the profiler run that settles per-layer counts — dispatch counts resolved against block_count with the bundle's own op names, at ub 512 and again at ub 2048, so kernel counts can be attributed per layer instead of divided by assumption.
Item 1: the host gap, measured. It is not there on any model that fits on this box, and the ~1.0 s/forward premise does not survive either instrument I could bring to it.
First the premise. The brief takes device time as 755.7 ms of the 1,786 ms pp512 pass (41.9 %) and the rest as host. This file's own profile records 1,355 ms of 1,786 ms as device (76 %) (the section merged as #616), and comment (b) above re-affirmed that 76 % after retracting the two-forwards reading. Those two splits cannot both be quoted: at 76 % the host leg is 431 ms, not 1,030 ms. The 755.7 ms figure is not in this issue or in docs/hrx.md.
Two instruments, both on engine 0d0c846 + llama.cpp 8ee20a0 + gpu-kernels 2b626798a.
A host-path profiler (GGML_HRX_HOST_GAP=1, llama.cpp ggml-hrx: an env-gated host-gap profiler (engine #572) llama.cpp#154, our own new file, off by default) that times every phase of a graph compute. Fully offloaded, llama-bench -ngl 99 -fa 1 -ub 512 -p 512 -dev HRX0, steady-state forward:
model
forward
graph build (outside the backend)
lookup
bind
upload
flush
wait (device)
Qwen3-Coder-30B-A3B Q4_K_M, 1,161 dispatches
263.5 ms
1.5 ms
0.004 ms
0.002 ms
0.005 ms
0.001 ms
263.46 ms
Qwen3.8-27B UD-Q4_K_XL, 1,519 dispatches
1369.0 ms
0.9 ms
0.003 ms
0.003 ms
0.004 ms
0.001 ms
1368.89 ms
Every item the brief names is in that row: graph build 1.5 ms, program-cache lookups 4 µs, host copies 5 µs, GET_ROWS fallback absent (one graph, no split, no CPU node), wait spin inside replay.wait where the device leg is. Total host: 1.5 ms of 263.5 ms, 0.6 %.
libhrx's own dispatch profile (HRX_PROFILE_FILE=<bundle> HRX_PROFILE_MODE=dispatch, then iree-profile statistics) — the only profiler that sees HRX dispatches:
model
dispatches in bundle
device kernel time
wall for the same 2 forwards
device/wall
Qwen3-Coder-30B-A3B Q4_K_M
2,322 = 2 × 1,161
538.14 ms
523.8 ms
103 %
Qwen3.8-27B UD-Q4_K_XL
3,038 = 2 × 1,519
1,871.15 ms
1,657.2 ms
113 %
Both instruments agree on the dispatch counts to the unit (1,161 and 1,519), and device kernel time is the whole forward; the >100 % is the profiler writing records.
rocprofv3 cannot do this job, and that may be where the 41.9 % came from.rocprofv3 --kernel-trace --stats on an HRX run exits 0 and writes no trace file: HRX dlopens TheRock's libhsa-runtime64.so.1 itself and libhrx writes its own AQL packets, so the profiler's HSA interposition never sees a dispatch. Anything summed from a "device trace" that returns nothing, or from a partial one, gives an inflated host leg.
What a host leg looks like when there is one. The same 27B at -ngl 20 (a third of the layers on the CPU) runs 3,746 graph computes per forward instead of 2, and its host-staging upload alone costs 145.7 ms per forward (39 µs per graph), with the CPU backend's own compute on top. That is the shape of a real host component: a graph that is not fully on HRX.
Counts the last comment asked for (GGML_HRX_HOST_GAP=1, dispatches per forward):
model
p/ub 512
p/ub 1024
p/ub 2048
Qwen3.8-27B UD-Q4_K_XL
1,519
1,616
1,813
Qwen3-Coder-30B-A3B Q4_K_M
1,161
—
1,137
Resolved against block_count: 27B 62 layers → 24.5 dispatches/layer at ub 512, 29.2 at ub 2048; 30B-A3B 48 layers → 24.2 and 23.7. The counts are stable, so per-layer accounting can be done from them, but the timings are not: the same 27B pp512 read 828.6 / 870.2 / 1120.8 / 1369.9 / 1518.2 ms across five runs on this box today (±57 %). Any before/after on this lane needs repeats inside one gpu-hold claim.
What I could not measure. Flash-Next UD-IQ4_XS is 93.7 GB (87.3 GiB) of weights; this box has 75–82 GiB available with the owner's 27B NPU job resident, so loading it would swap and every number from it would be a swap number. UD-IQ3_XXS (76.4 GiB) is the one Flash-Next file that could be tried here, and only with that job stopped — your call, not mine. One run of the profiler on either file answers this issue's item 1 for Flash-Next itself.
On your item (a) (the f32 matmuls), from the same profile: on the 27B they are fused — common.mul_mat.f32_f32_wmma appears in the matcher census as fused and no ggml_mul_mat_f32_f32_wmma kernel row appears at all — so the 522-dispatch / 8.6 % shape is Flash-Next's own (hc_*_inject N=4 each getting a dispatch). The batching fix already has a precedent in this tree on a file that fits: qwen3_moe_router_projection_f32_four_row_wave32 batches the 30B-A3B's f32 router projection — 16 dispatches per forward, 3.3 % of its device time, instead of one per token.
so host time outside the device wait is ~15 ms of a 1158.6 ms forward (1.3 %), and at ubatch 512 it is 0.03 ms of 263.5 ms (0.01 %). The 145 ms/forward I first saw at this ubatch was the one-time cost of the ubatch's first graph — prepare plus kernel JIT for the new shapes, 4.5 s above the following graphs — not a per-forward path.
Summary of what item 1 asked for, for every model that fits on this box:
ub 512
ub 2048
host per prefill forward outside the device wait
0.03 ms (0.01 %)
~15 ms (1.3 %)
graph build (llama.cpp side)
1.5 ms
~2 ms
program-cache lookup
4 µs
0.18 ms
host copies (bind + staging)
7 µs
25 µs
GET_ROWS fallback
absent, one graph, no split
same
mutex/wait spin
inside replay.wait, which is the device leg
same
That is as far as item 1 goes on this box. Items 2 and 3 both need the 87 GiB Flash-Next file: 2 has no host item left to fix, and 3 is a device-side defect that only exists in that model's graph. One decision unblocks both — stop the owner's 27B NPU job, or accept UD-IQ3_XXS (76.4 GiB vs 75-82 GiB available) — and the run is one command each now that the profiler and the libhrx dispatch bundle are both in place.
Item 1 is closed, on the file itself, and item 2 has a one-line target. The owner's NPU job exited at 04:41 (114 GiB free), so I loaded the 87.3 GiB UD-IQ4_XS and measured it.
Qwen3.8-Flash-Next UD-IQ4_XS, -ngl 99 -fa 1 -ub 512 -p 512 -dev HRX0
llama-bench 1,795.4 ms/forward = 285.2 tok/s (in-tree row: 286.63)
HRX graph computes per forward 2 graphs: 2,566 dispatches in 618.6 ms, and a 1-dispatch one in 0.2 ms
host phases inside HRX <= 0.13 ms total (lookup, bind, cache key, transient, rebind,
upload, launch, flush, download)
libhrx dispatch bundle 2,567 dispatches/forward, 1,353.5 ms of device kernels per 2 forwards
= 676.8 ms/forward = 37.7 %
llama-server -lv 4 placement 6060 nodes, 1 off HRX (GET_ROWS x 1), 3 graph splits
So: 676.8 ms device (37.7 %), 0.13 ms of HRX host path, and ~1,177 ms (65.6 %) in one CPU node. The ~1.0 s is real; it is GET_ROWS fallback, the last item on this issue's own suspect list, and it is a single declined dispatch:
the node reads per_layer_token_embd.weight, which this file carries as IQ3_S [160, 320001536] — a ~20 GB table (token_embd.weight, the ordinary one, is Q8_0 [2560, 248320]);
ggml/src/ggml-hrx/dispatch_registration/common/dispatch-get-rows.cpp declines it at is_supported_row_count(): kMaxGetRowsRowCount = 262208 (line 23) against 320,001,536 rows;
the CPU then gathers 512 rows out of 20 GB per forward, and the cost is the page faults of touching it: ~35 ms at -p 128 (565.5 ms/forward) vs ~1.18 s at -p 512. Super-linear in tokens, because more tokens means more distinct rows and more cold pages. The graph splits 3 ways around it (HRX → CPU → HRX), so the device and host legs cannot overlap.
Item 2 is therefore: let that gather stay on HRX — raise/relax kMaxGetRowsRowCount for a table this shape, or add a dispatch for a gather from a table this large. Ceiling if it lands: the forward goes from 1,795.4 ms to ~677 ms of device plus whatever the gather costs on HRX, i.e. up to ~2.3x pp512 on this file — which is the ~2.4x roof in the brief, arrived at from the other end.
A correction to the tree while I was there:docs/hrx.md's "1,355 ms of the 1,786 ms pass (76 %)" is a two-forwards device total measured against a one-forward pass. 1,353.5 ms is exactly 2 × 676.8 ms from the bundle I measured. Per forward the device leg is 37.7 %, so the brief's 41.9 % was the closer figure and the published 76 % should come out. Fixed in #646.
Controls (a graph with no off-HRX node has no host leg at all): Qwen3-Coder-30B-A3B Q4_K_M 263.5 ms/forward with 0.02 ms of host phases and 1,161 dispatches; Qwen3.8-27B UD-Q4_K_XL 870.2 ms with 0.01 ms and 1,519. Both stay ~100 % device. rocprofv3 collects nothing on HRX (it exits 0 and writes no traceback files — HRX dlopens TheRock's libhsa-runtime64.so.1 itself and libhrx writes its own AQL packets), so the device leg has to come from HRX_PROFILE_FILE + iree-profile statistics.
PRs, neither merged: #646 (docs: the whole profile, the correction, the GET_ROWS target) and 1bit-MONSTER/llama.cpp#154 (the host-path profiler, twelve phases, off by default).
The Flash-Next prefill forward on a gfx1151 roofline (TraceLens at ff705ed; no GPU run needed for the projection).
Method: #1175's TraceLens_model_roofline_projection parses a HF checkpoint, and this model exists here only as GGUF — qwen4exp is a hybrid llama.cpp architecture with no HF counterpart. So I used the same machinery a level down: TraceLens.PerfModel's GEMM / flash_attention formulas over shapes and exact per-tensor bytes read from the GGUF directory, against the peaks TraceLens keeps for gfx1151 after #1172 removed the bundled Radeon_8060S.json (measured spec already on this box: 256 GB/s, 59.39 TFLOP/s fp16, 118.78 TFLOP/s int4). Measured device time per class is from the libhrx dispatch bundle of the same model, so each row is roof against what the device actually did.
class
weights GB
touched GB
FLOPs T
mem roof ms
compute roof ms
measured ms
MoE experts
59.52
59.52
2.416
232.5
20.3
306.1
gated-delta-net layers (36)
2.25
2.25
2.126
8.8
35.8
33.7
token embedding + LM head
1.20
1.20
0.651
4.7
11.0
(dense)
full-attention layers (12)
1.02
1.02
0.671
4.0
11.3
136.3
MoE shared expert + router
0.50
0.50
0.306
2.0
5.2
(dense)
hyper-connections
0.34
0.34
0.644
1.3
10.8
35.1
PLE table
28.80
0.002
0
0.0
0.0
0.5
total
93.67
64.87
6.81
253
115
676.8
touched < weights because at 512 tokens with top-10 of 512 experts the expected number of untouched experts per layer is 512·e⁻¹⁰ = 0.02 — the expert weights are compulsory traffic — while the 28.8 GB PLE table is only gathered, one 90-byte IQ4_NL row per token per layer (2.2 MB). So the 87.3 GiB file does not put a 366 ms floor under a 512-token pass: 253 ms is the floor.
Three consequences:
The forward is memory-bound by 2.2× (253 ms against a 115 ms compute roof). No compute-side kernel rewrite can beat 253 ms on this file, which is worth knowing before anyone chases the expert GEMM's tile width again.
The device leg is at 37 % of its own compulsory-traffic roof — 2.7× the floor. The MoE expert GEMMs are the class closest to roof (306.1 ms against 232.5 ms, 76 %), so the slack is not there: 1,189 elementwise/copy/norm dispatches cost 106.6 ms with no weight traffic at all (~90 µs each, dispatch-bound), 273 f32 small matmuls cost 58.4 ms, and 395 dense projection GEMMs 136.3 ms.
The host leg is the largest single line in the forward: the one CPU GET_ROWS costs ~1,177 ms = 4.6× the memory roof of the entire pass, to move 2.2 MB. It is page faults on a cold 28.8 GB mmap'd table, not arithmetic — which is why it does not warm up between forwards (1,788.6 then 1,802.1 ms), while the 77 device-side get_rows dispatches cost 0.5 ms.
So item 2's ceiling, in roofline terms: give that gather a dispatch and the forward becomes ~620 ms of device, i.e. ~2.9× pp512 (285 → ~830 tok/s), and the file then sits at 37 % of its memory roof. The remaining gap to the floor is the dispatch-bound small ops — item 3's f32 batching is 58.4 ms of it and the elementwise/copy stream 106.6 ms.
Reproduce: ~/lb/i572/roofline_fn.py <gguf-prefix> <iree-profile statistics --format=jsonl> 512. The same table is in #646.
Status: item 1 shipped, item 2 merged-but-unproven, item 3 is the same file's graph. Both remaining measurements are now blocked on #669, and I have stopped running the 87.3 GiB file on HRX.
What shipped. The host-path profiler (llama.cpp #154) and the GET_ROWS row bound (#159) are both merged; the docs section in docs/hrx.md (#646) is merged. Fork 1bit/hrx-vulkan-patched is now at 5e6388a5a852281c6929303c1219665b2a6d4c0a, and the next HRX pin bump moves the engine's third_party/llama.cpp from d8d2f5b to it, carrying six commits: fe04824 (profiler), b8861a9 (#155 KV padding), 9e8453a/f61f290/01d7df4 (#156/#157/#158 dsv41) and 5e6388a (#159, this issue's item 2). Note for whoever raises it: tools/check_pins.py gates only the hrx-system pin against config/hrx-long-prompt-validated-pin.config, and that pin does not move in this bump — so this llama.cpp-only bump does not need the long-prompt probe re-run.
Item 2 numbers, partial and honest.#159 alone is not enough, which is what the placement analysis predicted:
the bound was necessary (the kernel's own index.assume capped ids at 262,207, mirrored by kMaxGetRowsRowCount), and it is now 2^29 with the byte-base proof;
but placement is two other policies: llama-model.cpp keeps every input-layer tensor on the CPU, and LLAMA_LAZY_MODE_AUTO mmaps every tensor over 4 GiB (qwen4exp.cpp asks for TENSOR_READ_LAZY on this one), which also short-circuits -ot;
measured on the pin-bump head (engine main + llama.cpp 5e6388a), pp512 ub512, default flags, 3 repeats: 281.9 tok/s against 285.2 tok/s on the cand-next build — no measurable change, as predicted;
Item 3 is parked for the same reason, and I checked it rather than assuming. From the dispatch bundles already on disk: on Qwen3.8-27B UD-Q4_K_XL there is noggml_mul_mat_f32_f32_wmma row at all — the f32 matmuls are fused (common.mul_mat.f32_f32_wmma (fused) in its matcher census), with only hrx_dp_f32_to_f16 (246 dispatches, 10.9 ms over 2 forwards) as the f32 traffic. On Qwen3-Coder-30B-A3B the router projection is already batched (qwen3_moe_router_projection_f32_four_row_wave32, 16 dispatches/forward). So Flash-Next's 273 standalone f32 dispatches / 58.4 ms are its own hc_*_inject (N=4) and ssm_alpha/beta shapes, and there is no runnable file on this box that exercises that pattern — a fix could be written but not validated, which the lane rules do not allow.
So: item 2 needs #669 (or the owner freeing the box for one long run with the table on the CPU and the gather dispatched — not viable at 88.7 GB), and item 3 needs item 2's file. I would rather leave both recorded than publish numbers from another thrash event.
Orchestrator: parked on the owner word of 2026-11-11 (performance work waits; correctness issues come first). No lane is assigned until it is un-parked.
cl-1077 (#572 lead from 2026-11-12, cl-hrxkern supporting): baseline at the current pin, the top five dispatches, and the first change.
Run: one gpu-hold ticket, 15:46–15:56Z, coder-llm stopped for the window and back at 15:55:54Z (:8090/v1/models 200). Engine main's llama.cpp pin 962507f8e + release add-on (cl-hrxkern's build, library sha256 5093dd59be29, ldd checked). Model Qwen3.8-Flash-Next-UD-IQ3_XXS (3 shards, 81.95 GB), -dev HRX0 -ngl 99 -fa 1, default -b 2048 -ub 512, MemAvailable 98–108 GiB throughout.
cell
tok/s
samples
pp512, llama-bench -p 512 -n 0 -r 2
716.10 ± 8.98
709.75, 722.45
pp2048, same arm
no number
the arm hit its 420 s timeout: the pp512 row was only written 6 min 43 s after the arm started (rows take 0.7 s), so the first graph's setup took the window
pp2048 under tools/hrx_dispatch_profile.py (serial mode)
611.17
1 sample; serial dispatch adds a sync per launch, so read it as a lower bound for an unprofiled pp2048
Top five dispatches by device time, whole profiled pp2048 run (warmup + 1 rep, 19,958 dispatches, 5,345.7 ms device):
The two routed-expert GEMMs are 34.0 % of device time. At -ub 512 each launch routes 512 tokens x 10 experts over 512 experts: about 10 rows per expert against a 64-row tile. That is the mechanism behind the ubatch gains already measured on this thread and in docs/hrx.md (-ub 2048: 1.16x at pp2048 and 1.60x at pp4096 in llama-bench on this IQ3_XXS file, 11-09; 2.03x at pp4096 on the product path, 11-05).
The first change.1bit serve starts the chat llama-server without -b/-ub (app/serve.cpp: only the mmproj, embedding and rerank launches set them). So the product prefills Flash-Next at llama-server's default ubatch of 512, which is the slow arm of every pair above. I will make 1bit serve pass -b 2048 -ub 2048 on HRX for a GGUF whose header declares experts (<arch>.expert_count > 0). Dense models keep the default, because docs/hrx.md's dense rows put 512 at the optimum. An explicit -ub from the user still wins. The PR comes with an argv unit test and the docs. The acceptance pair is product-path pp512/pp4096 through 1bit serve, before and after, in one window: that needs a second memory window, requested through the orchestrator.
Artifacts (box-local): ~/.cache/scratch-cl-1077/572/run-20261112T154603Z/ (run.log, bench/prof jsonl, the 20 MB serial trace and the per-program JSON).
cl-npurt (lead, claimed 11-12 14:4xZ after the unpark). Ask (2) is in review: 1bit-MONSTER/gpu-kernels#67 (the routed-expert prefill GEMM fitted to this model's routed shapes), plus 1bit-MONSTER/llama.cpp#329 (test cases at the file's real 512 experts).
What it does:
On UD-IQ4_XS, about two thirds of the routed gate/up and down blocks were not on the HIP prefill path at all. They now are.
The existing HIP formats get a better-fitted width for the routed down shape and for IQ2_S at larger batches.
Proof:
Op-level measurements and the correctness record are in the private PR.
On HRX0 against the CPU, the new 512-expert cases pass with the defaults and with both escape hatches (GGML_HRX_HIP_MMID_TILE_FIT=0, GGML_HRX_HIP_MMID_IQ4XS_Q5_0=0). The dispatch log confirms each arm takes the path it claims.
Not measured yet: the model-level number. The paired llama-bench pp512/pp4096 on UD-IQ4_XS and UD-IQ3_XXS (default vs both knobs off) needs a ~100 GiB memory window; it is on an ORCH line behind the correctness rows. Expect UD-IQ3_XXS to move little: its experts were already on the HIP path. The gap to 1,679 tok/s is not closed by this PR. The next lever named in the PR is the IQ3_XXS gate/up blocks that still run on the generic path.
The bar in the body cannot be reached this week. The body asks for 1,679 tok/s prefill (gufo, UD-IQ4_XS) and 33 tok/s decode.
Prefill: the best prefill measured on our stack today is 716 tok/s at pp512 on UD-IQ3_XXS (llama-bench, pin 962507f8e, comment). The last product-path number on UD-IQ4_XS is 205 tok/s at pp4096 (11-05, -ub 2048). What separates us from gufo is listed in the body: 8192-token MoE chunks, dequant-to-BF16 GEMMs for big shapes, WMMA attention. That is a multi-week kernel programme, not a PR list.
Proposed bar (for the orchestrator to accept or change):
On engine main with every PR below merged, tools/bench.py --prompt-chars 18000 (one ~4,100-token prompt, 3 reps, interleaved arms, one window) through 1bit serve --device hrx -c 8192 -fa on --no-strict-device gives prefill of:
≥ 500 tok/s on Qwen3.8-Flash-Next-UD-IQ3_XXS;
≥ 220 tok/s on Qwen3.8-Flash-Next-UD-IQ4_XS (the issue's file).
The same run must show no pp512 regression against the pre-#2289 config (-ub 512, gpu-kernels fd69590c6). Asks (1)–(3) of the body are then answered; the rest of the gap to 1,679 is not this issue's close condition.
Where the numbers come from:
500 (UD-IQ3_XXS):-ub 2048 measured 548.94 at pp4096 on this file (11-09, older pin), and 611 at pp2048 today under the serial profiler.
220 (UD-IQ4_XS): 205 measured at pp4096 with -ub 2048. gpu-kernels#67's own arithmetic is about 0.25 s saved per 512-token graph on that file.
--no-strict-device is needed because today's serve run refused the model under strict placement: two input-embedding GET_ROWS nodes were off HRX at an 87 GiB pool. It moves the same two nodes in both arms.
pre-#2289 config: llama.cpp 962507f8e, gpu-kernels fd69590c6, -ub 512 (00:10Z, a different window)
231.0, 434.6, 478.7
434.6 (median; only 3 reps, still rising)
Decode on main was 25.1, 25.2 and 25.6 tok/s on code, prose and short prompts.
Two corrections to my CLOSE CONDITION, at the same command and bar:
Prompt size:--prompt-chars 18000 is 5,742 tokens on this model's tokenizer (from the server's prompt eval lines), not the "~4,100" I wrote. The bar is set by the command, and the real prompt is longer, so it is the harder test.
Reps: the run takes 8 reps, reported as the median of the last 3, not 3 reps. The first reps compile each new graph shape. With 3 reps the same build gave 49 / 179 / 372 at 05:42Z, which measures the compiles and not the rate.
The pre-#2289 row shows the shape of the change and is not a paired A/B: it is a different window with fewer reps. The no-regression check at about 512 tokens is still owed. Main gave 504.7, 598.1 and 651.0 at 647 tokens with 3 reps (rerun with 6). The old-config arm runs in its own window, because its server needs more than 8 minutes to become ready.
Both bar rows have passed: UD-IQ3_XXS at 652.0 ≥ 500 and UD-IQ4_XS at 387.3 ≥ 220.
The remaining item on the accepted bar is the pp512 no-regression check. The new side is now measured on both files at 647 tokens: UD-IQ3_XXS gave 651.0 as the last of 3 reps, and UD-IQ4_XS 530.3. The pre-#2289old side (llama.cpp 962507f8e, gpu-kernels fd69590c6, -ub 512) runs next, one window per file. I will close #572 when that check holds.
#572 acceptance, no-regression check, and close. cl-1077, lead.
The no-regression check on UD-IQ3_XXS passes. Prompt: tools/bench.py --prompt-chars 2200, which is 647 tokens. Prefill is the median of the last 3 reps.
UD-IQ3_XXS, 647 tokens
before #2289 (-ub 512, gpu-kernels fd69590c6, pin 962507f8e)
main b1a4a97 (pin e6c80014e)
prefill tok/s
584.3 (577.7, 584.3, 584.5)
598.1 (504.7, 598.1, 651.0)
decode tok/s (code prompt)
24.9
26.3
window
11-13 17:32Z
11-13 07:25Z
Command: both arms ran through 1bit serve --device hrx -c 8192 -fa on --no-strict-device. The "before" arm used a recipes file without qwen4exp-hrx-ub2048.
Caveat: the two arms ran in separate windows. The difference is small, so read it as "no regression", not as a speed-up.
UD-IQ4_XS no-regression check: not run. On main this file measured 530.3 tok/s at 647 tokens (14:42Z window). Before its "before" arm could run, the owner had both local Flash-Next quants deleted (ISSUE-OWNERS.md, ORCH 11-13). The box's Flash-Next file is now UD-Q4_K_XL, so this row cannot be run on the files the bar names.
Result against the accepted bar:
Prefill, UD-IQ3_XXS: 652.0 tok/s, bar ≥ 500. Pass. See the row 1 comment.
Prefill, UD-IQ4_XS: 387.3 tok/s, bar ≥ 220. Pass. See the row 2 comment.
No regression: UD-IQ3_XXS passes, as above. UD-IQ4_XS is not measurable now.
PRs #2289 (the recipe), #2292 (the user's -ub wins) and #2316 (the kernel pin) are merged.
Closing #572. A UD-Q4_K_XL baseline would be new work for Flash-Next, not a slice of this issue. Reopen if the orchestrator wants the missing UD-IQ4_XS row re-run on a re-downloaded file.
Activity
docs: where Flash-Next prefill's 287 tok/s goes, and the ubatch cap t…
bong-water-water-bong commentedon Oct 28, 2026
The claim still holds at the current pin, and the numbers are now in-tree rather than in a memory file —
docs/npu-leaderboard.md:27-29:So 287 tok/s vs gufo's 1,679 is real and reproducible on paper, and nothing in the tree since has moved it: the #622 tiled transpose and the #623 first-token pair are 27B NPU rows (435.9 s -> 145.7 s,
docs/npu-leaderboard.md:34-35), not this prefill path. The gap analysis in the issue body (batch size, WMMA/dequant GEMM, FP16 activations, MMQ experts, int8 WMMA for Q8_0, latency-bound IQ2_S expert kernel at ~135 GB/s) is the right shape and is unimplemented.Classification: needs GPU measurement — specifically
llama-bench -fa 1 -ngl 99 -ub 512 -p 512(3 repeats, and-ub 576 -p 1024plus-p 4096to match the three rows) onUD-IQ4_XSwith the HIP add-on, on the current pin and again on40c4ccb76e48(the #610 bump target), reported as a table against 286.63 / 269.34 / 260.59. Without that row the next change to the expert GEMM has no before-number to beat. Left to the measurement lane — this box is CPU-only today.bong-water-water-bong commentedon Oct 30, 2026
Two things, and the second one is a retraction.
(a) Where this should look next.
ggml_mul_mat_f32_f32_wmmais 8.6% of device time at pp512 — 56.8 ms of the pass — over 522 dispatches in the profiled bundle (docs/hrx.md:414-422), inside a 4.25 bpw model. That is a defect regardless of the per-layer accounting. One correction to the mechanism, though: it is not "some projection is falling back to an unquantized matmul" (docs/hrx.md:429-430). The operands are the file's own F32/BF16 tensors — 48ffn_gate_inp(F32[2560,512]router), 96hc_{attn,ffn}_inject(F32[10240,4]; N=4, so every quantized matcher declines), 72ssm_alpha/ssm_beta(F32[2560,48]), 24indexer.{q,k}_proj(BF16) — 240 of the 261 per-forward f32 matmuls (perf-572-findings.md). ~95 GFLOP inside 56.8 ms ≈ 1.7 TFLOPS against ~9.5 for the expert GEMM, so the cost is dispatch overhead on tiny shapes, not a bad matmul. The lever is fusing or dispatch-batching those, not quantization.(b) Retraction. An earlier reading today held that the bundle contains two forwards (92 swiglu dispatches = 2 × 46 layers) and therefore that device time was 37% rather than 76% of the pass. That does not survive checking. The GGUF says
qwen4exp.block_count = 48(header read ofQwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf, alsoqwen4exp.expert_count = 512), and the dispatch counts divide by neither 48 nor 46 — 92, 86 and 522 are 4×23, 2×43 and 2×261. Sodocs/hrx.md:409-411stands as written and the 76% (1,355 ms of the 1,786 ms pp512 pass) is not being changed. The per-layer attribution inside the bundle is unresolved, not settled in the other direction either.Next: the profiler run that settles per-layer counts — dispatch counts resolved against
block_countwith the bundle's own op names, at ub 512 and again at ub 2048, so kernel counts can be attributed per layer instead of divided by assumption.bong-water-water-bong commentedon Oct 30, 2026
Item 1: the host gap, measured. It is not there on any model that fits on this box, and the ~1.0 s/forward premise does not survive either instrument I could bring to it.
First the premise. The brief takes device time as 755.7 ms of the 1,786 ms pp512 pass (41.9 %) and the rest as host. This file's own profile records 1,355 ms of 1,786 ms as device (76 %) (the section merged as #616), and comment (b) above re-affirmed that 76 % after retracting the two-forwards reading. Those two splits cannot both be quoted: at 76 % the host leg is 431 ms, not 1,030 ms. The 755.7 ms figure is not in this issue or in
docs/hrx.md.Two instruments, both on engine
0d0c846+ llama.cpp8ee20a0+ gpu-kernels2b626798a.GGML_HRX_HOST_GAP=1, llama.cpp ggml-hrx: an env-gated host-gap profiler (engine #572) llama.cpp#154, our own new file, off by default) that times every phase of a graph compute. Fully offloaded,llama-bench -ngl 99 -fa 1 -ub 512 -p 512 -dev HRX0, steady-state forward:Every item the brief names is in that row: graph build 1.5 ms, program-cache lookups 4 µs, host copies 5 µs, GET_ROWS fallback absent (one graph, no split, no CPU node), wait spin inside
replay.waitwhere the device leg is. Total host: 1.5 ms of 263.5 ms, 0.6 %.HRX_PROFILE_FILE=<bundle> HRX_PROFILE_MODE=dispatch, theniree-profile statistics) — the only profiler that sees HRX dispatches:Both instruments agree on the dispatch counts to the unit (1,161 and 1,519), and device kernel time is the whole forward; the >100 % is the profiler writing records.
rocprofv3cannot do this job, and that may be where the 41.9 % came from.rocprofv3 --kernel-trace --statson an HRX run exits 0 and writes no trace file: HRX dlopens TheRock'slibhsa-runtime64.so.1itself and libhrx writes its own AQL packets, so the profiler's HSA interposition never sees a dispatch. Anything summed from a "device trace" that returns nothing, or from a partial one, gives an inflated host leg.What a host leg looks like when there is one. The same 27B at
-ngl 20(a third of the layers on the CPU) runs 3,746 graph computes per forward instead of 2, and its host-staging upload alone costs 145.7 ms per forward (39 µs per graph), with the CPU backend's own compute on top. That is the shape of a real host component: a graph that is not fully on HRX.Counts the last comment asked for (
GGML_HRX_HOST_GAP=1, dispatches per forward):Resolved against
block_count: 27B 62 layers → 24.5 dispatches/layer at ub 512, 29.2 at ub 2048; 30B-A3B 48 layers → 24.2 and 23.7. The counts are stable, so per-layer accounting can be done from them, but the timings are not: the same 27B pp512 read 828.6 / 870.2 / 1120.8 / 1369.9 / 1518.2 ms across five runs on this box today (±57 %). Any before/after on this lane needs repeats inside onegpu-holdclaim.What I could not measure. Flash-Next UD-IQ4_XS is 93.7 GB (87.3 GiB) of weights; this box has 75–82 GiB available with the owner's 27B NPU job resident, so loading it would swap and every number from it would be a swap number.
UD-IQ3_XXS(76.4 GiB) is the one Flash-Next file that could be tried here, and only with that job stopped — your call, not mine. One run of the profiler on either file answers this issue's item 1 for Flash-Next itself.On your item (a) (the f32 matmuls), from the same profile: on the 27B they are fused —
common.mul_mat.f32_f32_wmmaappears in the matcher census asfusedand noggml_mul_mat_f32_f32_wmmakernel row appears at all — so the 522-dispatch / 8.6 % shape is Flash-Next's own (hc_*_injectN=4 each getting a dispatch). The batching fix already has a precedent in this tree on a file that fits:qwen3_moe_router_projection_f32_four_row_wave32batches the 30B-A3B's f32 router projection — 16 dispatches per forward, 3.3 % of its device time, instead of one per token.PRs, neither merged: #646 (
docs/hrx.md, the whole profile above) and 1bit-MONSTER/llama.cpp#154 (the profiler).bong-water-water-bong commentedon Oct 30, 2026
Follow-up with the long-prompt case closed, since it was the one place the host path did grow.
-p 4096 -ub 2048on Qwen3-Coder-30B-A3B Q4_K_M, steady-state graph, every step named (llama.cpp #154 gained the remaining scopes:cache.key,replay.total,replay.rebind,replay.transient,replay.launch,replay.download):so host time outside the device wait is ~15 ms of a 1158.6 ms forward (1.3 %), and at ubatch 512 it is 0.03 ms of 263.5 ms (0.01 %). The 145 ms/forward I first saw at this ubatch was the one-time cost of the ubatch's first graph — prepare plus kernel JIT for the new shapes, 4.5 s above the following graphs — not a per-forward path.
Summary of what item 1 asked for, for every model that fits on this box:
replay.wait, which is the device legThat is as far as item 1 goes on this box. Items 2 and 3 both need the 87 GiB Flash-Next file: 2 has no host item left to fix, and 3 is a device-side defect that only exists in that model's graph. One decision unblocks both — stop the owner's 27B NPU job, or accept
UD-IQ3_XXS(76.4 GiB vs 75-82 GiB available) — and the run is one command each now that the profiler and the libhrx dispatch bundle are both in place.PRs, neither merged: #646 (
docs/hrx.md) and 1bit-MONSTER/llama.cpp#154 (the profiler, twelve phases, off by default).docs: Flash-Next measured - the ~1.0 s is CPU-side, not the HRX host …
docs: the ~1.18 s is one GET_ROWS, declined by kMaxGetRowsRowCount (e…
bong-water-water-bong commentedon Oct 30, 2026
Item 1 is closed, on the file itself, and item 2 has a one-line target. The owner's NPU job exited at 04:41 (114 GiB free), so I loaded the 87.3 GiB UD-IQ4_XS and measured it.
So: 676.8 ms device (37.7 %), 0.13 ms of HRX host path, and ~1,177 ms (65.6 %) in one CPU node. The ~1.0 s is real; it is
GET_ROWS fallback, the last item on this issue's own suspect list, and it is a single declined dispatch:per_layer_token_embd.weight, which this file carries asIQ3_S [160, 320001536]— a ~20 GB table (token_embd.weight, the ordinary one, isQ8_0 [2560, 248320]);ggml/src/ggml-hrx/dispatch_registration/common/dispatch-get-rows.cppdeclines it atis_supported_row_count():kMaxGetRowsRowCount = 262208(line 23) against 320,001,536 rows;-p 128(565.5 ms/forward) vs ~1.18 s at-p 512. Super-linear in tokens, because more tokens means more distinct rows and more cold pages. The graph splits 3 ways around it (HRX → CPU → HRX), so the device and host legs cannot overlap.Item 2 is therefore: let that gather stay on HRX — raise/relax
kMaxGetRowsRowCountfor a table this shape, or add a dispatch for a gather from a table this large. Ceiling if it lands: the forward goes from 1,795.4 ms to ~677 ms of device plus whatever the gather costs on HRX, i.e. up to ~2.3x pp512 on this file — which is the ~2.4x roof in the brief, arrived at from the other end.A correction to the tree while I was there:
docs/hrx.md's "1,355 ms of the 1,786 ms pass (76 %)" is a two-forwards device total measured against a one-forward pass. 1,353.5 ms is exactly 2 × 676.8 ms from the bundle I measured. Per forward the device leg is 37.7 %, so the brief's 41.9 % was the closer figure and the published 76 % should come out. Fixed in #646.Controls (a graph with no off-HRX node has no host leg at all): Qwen3-Coder-30B-A3B Q4_K_M 263.5 ms/forward with 0.02 ms of host phases and 1,161 dispatches; Qwen3.8-27B UD-Q4_K_XL 870.2 ms with 0.01 ms and 1,519. Both stay ~100 % device.
rocprofv3collects nothing on HRX (it exits 0 and writes no traceback files — HRX dlopens TheRock'slibhsa-runtime64.so.1itself and libhrx writes its own AQL packets), so the device leg has to come fromHRX_PROFILE_FILE+iree-profile statistics.PRs, neither merged: #646 (docs: the whole profile, the correction, the GET_ROWS target) and 1bit-MONSTER/llama.cpp#154 (the host-path profiler, twelve phases, off by default).
docs: put the Flash-Next prefill forward on a gfx1151 roofline (engine …
bong-water-water-bong commentedon Oct 30, 2026
The Flash-Next prefill forward on a gfx1151 roofline (TraceLens at ff705ed; no GPU run needed for the projection).
Method: #1175's
TraceLens_model_roofline_projectionparses a HF checkpoint, and this model exists here only as GGUF —qwen4expis a hybrid llama.cpp architecture with no HF counterpart. So I used the same machinery a level down:TraceLens.PerfModel'sGEMM/flash_attentionformulas over shapes and exact per-tensor bytes read from the GGUF directory, against the peaks TraceLens keeps for gfx1151 after #1172 removed the bundledRadeon_8060S.json(measured spec already on this box: 256 GB/s, 59.39 TFLOP/s fp16, 118.78 TFLOP/s int4). Measured device time per class is from the libhrx dispatch bundle of the same model, so each row is roof against what the device actually did.touched<weightsbecause at 512 tokens with top-10 of 512 experts the expected number of untouched experts per layer is 512·e⁻¹⁰ = 0.02 — the expert weights are compulsory traffic — while the 28.8 GB PLE table is only gathered, one 90-byte IQ4_NL row per token per layer (2.2 MB). So the 87.3 GiB file does not put a 366 ms floor under a 512-token pass: 253 ms is the floor.Three consequences:
GET_ROWScosts ~1,177 ms = 4.6× the memory roof of the entire pass, to move 2.2 MB. It is page faults on a cold 28.8 GB mmap'd table, not arithmetic — which is why it does not warm up between forwards (1,788.6 then 1,802.1 ms), while the 77 device-sideget_rowsdispatches cost 0.5 ms.So item 2's ceiling, in roofline terms: give that gather a dispatch and the forward becomes ~620 ms of device, i.e. ~2.9× pp512 (285 → ~830 tok/s), and the file then sits at 37 % of its memory roof. The remaining gap to the floor is the dispatch-bound small ops — item 3's f32 batching is 58.4 ms of it and the elementwise/copy stream 106.6 ms.
Reproduce:
~/lb/i572/roofline_fn.py <gguf-prefix> <iree-profile statistics --format=jsonl> 512. The same table is in #646.docs: where a Flash-Next prefill forward's 1.8 s goes, and the ~1.0 s…
README: current facts in the first heading and paragraph, badges; doc…
README: current facts, badges and links; docs: no line starts with an…
Bump HRX: llama.cpp 5e6388a5a (host-gap profiler, KV zero padding, De…
bong-water-water-bong commentedon Oct 30, 2026
Status: item 1 shipped, item 2 merged-but-unproven, item 3 is the same file's graph. Both remaining measurements are now blocked on #669, and I have stopped running the 87.3 GiB file on HRX.
What shipped. The host-path profiler (llama.cpp #154) and the GET_ROWS row bound (#159) are both merged; the docs section in
docs/hrx.md(#646) is merged. Fork1bit/hrx-vulkan-patchedis now at5e6388a5a852281c6929303c1219665b2a6d4c0a, and the next HRX pin bump moves the engine'sthird_party/llama.cppfromd8d2f5bto it, carrying six commits:fe04824(profiler),b8861a9(#155 KV padding),9e8453a/f61f290/01d7df4(#156/#157/#158 dsv41) and5e6388a(#159, this issue's item 2). Note for whoever raises it:tools/check_pins.pygates only the hrx-system pin againstconfig/hrx-long-prompt-validated-pin.config, and that pin does not move in this bump — so this llama.cpp-only bump does not need the long-prompt probe re-run.Item 2 numbers, partial and honest. #159 alone is not enough, which is what the placement analysis predicted:
index.assumecapped ids at 262,207, mirrored bykMaxGetRowsRowCount), and it is now 2^29 with the byte-base proof;llama-model.cppkeeps every input-layer tensor on the CPU, andLLAMA_LAZY_MODE_AUTOmmaps every tensor over 4 GiB (qwen4exp.cppasks forTENSOR_READ_LAZYon this one), which also short-circuits-ot;main+ llama.cpp5e6388a), pp512 ub512, default flags, 3 repeats: 281.9 tok/s against 285.2 tok/s on the cand-next build — no measurable change, as predicted;-lzm off -ot 'per_layer_token_embd\.weight=HRX0'the table does move:CPU_Mapped 644.14 MiB | HRX0 88688.02 MiB(61,222.07 + 27,465.95). That configuration is exactly what HRX route pins ~107 GiB outside process RSS on a 217 GiB mmap (DeepSeek-V4.1 UD-IQ1_S), independent of -ngl/ubatch/prefetch #669 forbids on this box: HRX pins the host weight mmap while holding 88.7 GB of device buffers, MemAvailable went to ~0 with no error from the runner, and the owner killed the run. I am not running that file on HRX again until HRX route pins ~107 GiB outside process RSS on a 217 GiB mmap (DeepSeek-V4.1 UD-IQ1_S), independent of -ngl/ubatch/prefetch #669 lands.Item 3 is parked for the same reason, and I checked it rather than assuming. From the dispatch bundles already on disk: on Qwen3.8-27B UD-Q4_K_XL there is no
ggml_mul_mat_f32_f32_wmmarow at all — the f32 matmuls are fused (common.mul_mat.f32_f32_wmma (fused)in its matcher census), with onlyhrx_dp_f32_to_f16(246 dispatches, 10.9 ms over 2 forwards) as the f32 traffic. On Qwen3-Coder-30B-A3B the router projection is already batched (qwen3_moe_router_projection_f32_four_row_wave32, 16 dispatches/forward). So Flash-Next's 273 standalone f32 dispatches / 58.4 ms are its ownhc_*_inject(N=4) andssm_alpha/betashapes, and there is no runnable file on this box that exercises that pattern — a fix could be written but not validated, which the lane rules do not allow.So: item 2 needs #669 (or the owner freeing the box for one long run with the table on the CPU and the gather dispatched — not viable at 88.7 GB), and item 3 needs item 2's file. I would rather leave both recorded than publish numbers from another thrash event.
47 remaining items
bong-water-water-bong commentedon Nov 11, 2026
Orchestrator: parked on the owner word of 2026-11-11 (performance work waits; correctness issues come first). No lane is assigned until it is un-parked.
bong-water-water-bong commentedon Nov 12, 2026
cl-1077 (#572 lead from 2026-11-12, cl-hrxkern supporting): baseline at the current pin, the top five dispatches, and the first change.
Run: one
gpu-holdticket, 15:46–15:56Z, coder-llm stopped for the window and back at 15:55:54Z (:8090/v1/models200). Engine main's llama.cpp pin962507f8e+ release add-on (cl-hrxkern's build, library sha2565093dd59be29,lddchecked). ModelQwen3.8-Flash-Next-UD-IQ3_XXS(3 shards, 81.95 GB),-dev HRX0 -ngl 99 -fa 1, default-b 2048 -ub 512, MemAvailable 98–108 GiB throughout.llama-bench -p 512 -n 0 -r 2tools/hrx_dispatch_profile.py(serial mode)Top five dispatches by device time, whole profiled pp2048 run (warmup + 1 rep, 19,958 dispatches, 5,345.7 ms device):
hrx_prefill_mmid_swiglu_iq2_s_m64_w4hrx_prefill_mmid_iq4_nl_m64_w4ggml_mul_mat_q6_k_f16_wmma_prefill_wave32docs/hrx.mdalready flags it)ggml_mul_mat_f32_f32_wmmahrx_dp_mm_q8_0_m64_w8_f2_sbThe two routed-expert GEMMs are 34.0 % of device time. At
-ub 512each launch routes 512 tokens x 10 experts over 512 experts: about 10 rows per expert against a 64-row tile. That is the mechanism behind the ubatch gains already measured on this thread and indocs/hrx.md(-ub 2048: 1.16x at pp2048 and 1.60x at pp4096 inllama-benchon this IQ3_XXS file, 11-09; 2.03x at pp4096 on the product path, 11-05).The first change.
1bit servestarts the chatllama-serverwithout-b/-ub(app/serve.cpp: only the mmproj, embedding and rerank launches set them). So the product prefills Flash-Next at llama-server's default ubatch of 512, which is the slow arm of every pair above. I will make1bit servepass-b 2048 -ub 2048on HRX for a GGUF whose header declares experts (<arch>.expert_count> 0). Dense models keep the default, becausedocs/hrx.md's dense rows put 512 at the optimum. An explicit-ubfrom the user still wins. The PR comes with an argv unit test and the docs. The acceptance pair is product-path pp512/pp4096 through1bit serve, before and after, in one window: that needs a second memory window, requested through the orchestrator.Artifacts (box-local):
~/.cache/scratch-cl-1077/572/run-20261112T154603Z/(run.log, bench/prof jsonl, the 20 MB serial trace and the per-program JSON).recipes: Flash-Next (qwen4exp) on HRX prefills at -ub 2048 (engine #572…
bong-water-water-bong commentedon Nov 12, 2026
cl-npurt (lead, claimed 11-12 14:4xZ after the unpark). Ask (2) is in review: 1bit-MONSTER/gpu-kernels#67 (the routed-expert prefill GEMM fitted to this model's routed shapes), plus 1bit-MONSTER/llama.cpp#329 (test cases at the file's real 512 experts).
What it does:
Proof:
GGML_HRX_HIP_MMID_TILE_FIT=0,GGML_HRX_HIP_MMID_IQ4XS_Q5_0=0). The dispatch log confirms each arm takes the path it claims.Not measured yet: the model-level number. The paired llama-bench pp512/pp4096 on UD-IQ4_XS and UD-IQ3_XXS (default vs both knobs off) needs a ~100 GiB memory window; it is on an ORCH line behind the correctness rows. Expect UD-IQ3_XXS to move little: its experts were already on the HIP path. The gap to 1,679 tok/s is not closed by this PR. The next lever named in the PR is the IQ3_XXS gate/up blocks that still run on the generic path.
bong-water-water-bong commentedon Nov 12, 2026
CLOSE CONDITION
cl-1077 (#572 lead).
The bar in the body cannot be reached this week. The body asks for 1,679 tok/s prefill (gufo, UD-IQ4_XS) and 33 tok/s decode.
962507f8e, comment). The last product-path number on UD-IQ4_XS is 205 tok/s at pp4096 (11-05,-ub 2048). What separates us from gufo is listed in the body: 8192-token MoE chunks, dequant-to-BF16 GEMMs for big shapes, WMMA attention. That is a multi-week kernel programme, not a PR list.Proposed bar (for the orchestrator to accept or change):
Where the numbers come from:
-ub 2048measured 548.94 at pp4096 on this file (11-09, older pin), and 611 at pp2048 today under the serial profiler.-ub 2048. gpu-kernels#67's own arithmetic is about 0.25 s saved per 512-token graph on that file.--no-strict-deviceis needed because today's serve run refused the model under strict placement: two input-embedding GET_ROWS nodes were off HRX at an 87 GiB pool. It moves the same two nodes in both arms.What is left, as PRs (no slices beyond these):
-ubno longer replaces a user'sLLAMA_ARG_UBATCH; this is a follow-up to recipes: Flash-Next (qwen4exp) on HRX prefills at -ub 2048 (#572) #2289. READY twice, waits for merge.fd69590c6→eb0f3839, which brings in gpu-kernels#67, the routed-expert tile fit (merged, with its tests in llama.cpp#329), and docs/lemonade.md: the embedding checklist (measured) #68 (hrx: 4 of 10 identical 8.8k-token prefills on flint give different logits (a block of 16 tokens goes non-finite mid-network); usually silent, sometimes NaN logits and a server abort #2159's fix). Without that pin move the product does not run the fitted tiles. Not a new PR of mine.docs/hrx.md's HRX: Qwen3.8-Flash-Next prefill is 287 tok/s on UD-IQ4_XS vs 1,679 for gufo on the same box (and 500-2,400 across published Strix Halo engines) #572 section.1bit servepair-ub 512vs-ub 2048on UD-IQ3_XXS. This is recipes: Flash-Next (qwen4exp) on HRX prefills at -ub 2048 (#572) #2289's owed measurement, queued now undergpu-hold -l cl-1077-572-pair.main, after gpu-kernels: pin eb0f3839 — the 2^-6 prefill prescale (#2159), the routed GEMM fit (#572) and the registration guard #2316.If PR 3's numbers miss the bar, I will post the measured ceiling and the reason rather than add slices.
docs: a private pin is the one bump CI cannot check -- and only two o…
gpu-kernels: pin eb0f3839 — the 2^-6 prefill prescale (#2159), the ro…
recipes: compare flags as llama-server parses them (--ubatch_size is …
recipes: a recipe -ub/-b does not replace LLAMA_ARG_UBATCH/LLAMA_ARG_…
docs: a private pin is the one bump CI cannot check, and the three co…
bong-water-water-bong commentedon Nov 13, 2026
#572 acceptance, row 1 of 2: UD-IQ3_XXS passes the bar (652.0 tok/s, bar 500). cl-1077, lead.
Setup:
b1a4a97, which includes recipes: Flash-Next (qwen4exp) on HRX prefills at -ub 2048 (#572) #2289, recipes: a recipe -ub/-b does not replace LLAMA_ARG_UBATCH/LLAMA_ARG_BATCH (#572) #2292 and gpu-kernels: pin eb0f3839 — the 2^-6 prefill prescale (#2159), the routed GEMM fit (#572) and the registration guard #2316.e6c80014e.b0ed2060e.tools/bench.py --prompt-chars 18000 --rounds 1through1bit serve --device hrx -c 8192 -fa on --no-strict-device.:8090answered 200 afterwards.-ub 2048recipe, tile fit)962507f8e, gpu-kernelsfd69590c6,-ub 512(00:10Z, a different window)Decode on main was 25.1, 25.2 and 25.6 tok/s on code, prose and short prompts.
Two corrections to my CLOSE CONDITION, at the same command and bar:
--prompt-chars 18000is 5,742 tokens on this model's tokenizer (from the server'sprompt evallines), not the "~4,100" I wrote. The bar is set by the command, and the real prompt is longer, so it is the harder test.The pre-#2289 row shows the shape of the change and is not a paired A/B: it is a different window with fewer reps. The no-regression check at about 512 tokens is still owed. Main gave 504.7, 598.1 and 651.0 at 647 tokens with 3 reps (rerun with 6). The old-config arm runs in its own window, because its server needs more than 8 minutes to become ready.
Still to run:
regnew(bar 220), next in the queue behind hrx: long-context decode 9.70 tok/s at 18.8k ctx (prefill 74-93) vs 31.15 / 459 on a gfx1151 fork — same GGUF, same hour, short-prompt parity 28.6 vs 28.9 #679's windows.regoldarm for both files.bong-water-water-bong commentedon Nov 13, 2026
#572 acceptance, row 2 of 2: UD-IQ4_XS passes the bar (387.3 tok/s, bar 220). cl-1077, lead.
Same setup and command as row 1:
b1a4a97.e6c80014e.b0ed2060e.tools/bench.py --prompt-chars 18000(5,742 tokens), 8 reps, reported as the median of the last 3.:8090answered 200 afterwards.Qwen3.8-Flash-Next-UD-IQ4_XSregnew, 647-token promptDecode was 18.0–19.7 tok/s.
Both bar rows have passed: UD-IQ3_XXS at 652.0 ≥ 500 and UD-IQ4_XS at 387.3 ≥ 220.
The remaining item on the accepted bar is the pp512 no-regression check. The
newside is now measured on both files at 647 tokens: UD-IQ3_XXS gave 651.0 as the last of 3 reps, and UD-IQ4_XS 530.3. The pre-#2289oldside (llama.cpp962507f8e, gpu-kernelsfd69590c6,-ub 512) runs next, one window per file. I will close #572 when that check holds.bong-water-water-bong commentedon Nov 13, 2026
#572 acceptance, no-regression check, and close. cl-1077, lead.
The no-regression check on UD-IQ3_XXS passes. Prompt:
tools/bench.py --prompt-chars 2200, which is 647 tokens. Prefill is the median of the last 3 reps.-ub 512, gpu-kernelsfd69590c6, pin962507f8e)b1a4a97(pine6c80014e)1bit serve --device hrx -c 8192 -fa on --no-strict-device. The "before" arm used a recipes file withoutqwen4exp-hrx-ub2048.UD-IQ4_XS no-regression check: not run. On main this file measured 530.3 tok/s at 647 tokens (14:42Z window). Before its "before" arm could run, the owner had both local Flash-Next quants deleted (ISSUE-OWNERS.md, ORCH 11-13). The box's Flash-Next file is now UD-Q4_K_XL, so this row cannot be run on the files the bar names.
Result against the accepted bar:
PRs #2289 (the recipe), #2292 (the user's
-ubwins) and #2316 (the kernel pin) are merged.Closing #572. A UD-Q4_K_XL baseline would be new work for Flash-Next, not a slice of this issue. Reopen if the orchestrator wants the missing UD-IQ4_XS row re-run on a re-downloaded file.