You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
npu: prefill is host-pack bound (24 ms/launch pack vs 1.3 ms device, 93 % of the pass) — 18.64 tok/s is the pack rate, not the array's #1416
The dx route's prefill is host-pack bound: the array is idle through ~93 % of a prefill pass. Measured on strixhalo, Bonsai-2-27B t2s2 container, NPU_PHASE_PROF=1, M=128 launches (~/briefs/TBONSAI-SPEED-STATE.md, 2026-11-02 20:05 entry):
role (M=128)
host pack ms/launch
device wait ms/launch
G / U / D
24.2 / 23.8 / 23.8
1.20 / 1.24 / 1.35
O
8.2
0.60
QKV / Z
14.2 / 8.3
0.85 / 0.59
Consistency check that makes this a fact rather than a profile artefact: 320 launches × ~24 ms ≈ 7.7 s per 128 rows = 16.6 tok/s, against the measured prefill of 18.64 tok/s. Prefill throughput is host-pack throughput (~0.19 ms/row), and it does not depend on the weight-stream width at all — which is also why the 2-bit stream (plan P3, #802, 1.76x per kernel, 1.31x per decode step) buys nothing at prefill.
Cost of leaving it: prefill 18.64 tok/s against HRX's 149-160 tok/s on the same model and box, and 82-105 for FLM. Device-bound for the same pass would be 320 × ~1.3 ms ≈ 0.43 s ≈ ~308 tok/s, i.e. the same order as the GPU/FLM numbers, with no new kernel arithmetic.
Where the time goes (code, not guess): the launch path calls quantize_async_rows(A, am, ak, ascales) (npu/forward/engine-src/npu_engine_i8ctx_inc.h:682) per launch, called from the pb16i/t2s launch path (:535), and it re-packs all MD rows on every launch even when only am rows are real. Weight packing (q4nx_pack_B_int8, npu/forward/include/q4nx_pack.h:268) is not in this number — it happens at load.
Candidate fixes, cheapest first:
Pack only the rows that exist (am rows, not MD): the measured 5-token-prompt pass packs 128 rows when 5 exist. Wins short prompts (~0.5 s of a 3.2 s request) and nothing at long ones.
Fuse the three passes over each row into one (amax and quantise in one sweep, keeping the row in L1) and OMP over rows — the array is idle, so there are 32 cores free for this.
Only then consider moving the expansion on-core for prefill (the T-MAN shape, ~/amd-oss/npu-re/ASSESS-tman-2bit-dx.md §4), which needs a new M=128 ELF family and is 10x the work for maybe 2x more.
Gate as always: byte-exact prefill logits against the current route on the same prompt set (the harness for that already exists), plus the [prof] pre pack column before/after and the whole-prompt tok/s row.
Not a device-blocked issue: 1-3 are host code, measurable with a host bench first and confirmed in one npu-hold window.
Correction to my own fix list, measured before anyone acts on step 2 — the row quantiser's arithmetic is already fast and already vectorised; that is not what the 24 ms is.
Device-free bench of the engine's own routine (npu/forward/include/quant_i8.h, q4nx_quant_i8::row, AVX2 path active), a whole M=128 block per iteration, median of 20, run under ~/lb/thermal-run.sh:
row width K
128 rows
per row
per element
int8 write rate
896
0.038 ms
0.0003 ms
0.33 ns
3.0 GB/s
4864
0.203 ms
0.0016 ms
0.33 ns
3.1 GB/s
6144
0.279 ms
0.0022 ms
0.35 ns
2.8 GB/s
17408
0.584 ms
0.0046 ms
0.26 ns
3.8 GB/s
So quantising 128 rows of 4864 takes 0.203 ms against the ~24 ms the in-loop [prof] pre column reports for the same launch. ~99 % of "pack" is not arithmetic. Step 2 of the list above (vectorise the row quantiser) is therefore cancelled — it would optimise 0.8 % of the term — and step 1 (pack only real rows) keeps its value only for short prompts, where it is a real 5-fold saving on this same 24 ms.
What is left, and it is now the whole issue: quantize_async_rows() writes directly into the activation BO mapping (Am + (size_t)m * KD), then the caller does sync_a_to_device(). Two candidates, and they are separable in one host-side experiment:
byte-granular stores into the mapped BO (partial cache-line writes → read-modify-write or per-store flush) — the fix is the shape we already use for weights after npu: fill a weight image straight into the BO, one pass instead of four #871: quantise into a RAM staging buffer, then one bulk write/DMA of the block;
sync_a_to_device() itself — 622 KB per launch × 320 launches is a lot of driver sync for a pass whose device wait is 1.3 ms.
Next diagnostic (no device needed): split the pack column into quantise→RAM, RAM→BO copy, and sync, and report each. If (1) dominates, the staging fix is a few lines in the launch path; if (2) dominates, the lever is fewer/bigger syncs (one per layer rather than one per launch), which is a different PR and also cheap.
The prefill ceiling math in the issue body is unchanged: device-bound for the same pass is ~0.43 s per 128 rows ≈ ~308 tok/s, and it is reachable once "pack" stops being 24 ms of a 25.3 ms launch.
Cross-lane evidence for your fix list, from the TileFuse reproduction attempt: the kernel is 1.54 TFLOP/s and the route is pack-bound — and their stack has no host pack at all
I came at the same wall from the other side (owner task: reproduce TileFuse's Llama3-8B W4A16 prefill on Max+ 395). Two measurements worth
having here, both from one 11 s npu-hold, amdxdna 0 → 0, mosaic_bench --mode npu against our own gemm_q4_prefill T=256 sets:
shape (Llama3-8B)
M
our ms/tile
our TFLOP/s
up_gate K=4096 N=14336
1024
19.500
1.542
up_gate K=4096 N=14336
2048
19.589
1.535
down K=14336 N=4096
1024
18.953
1.586
So the kernel alone is 1.54–1.59 TFLOP/s, and per-tile time is flat in M — which is the microbenchmark's device-only number, i.e. a
different quantity from your 24 ms pack column, and it is not the thing that makes the route 18.64 tok/s.
What TileFuse does that removes the pack column structurally rather than incrementally. Their artifact
(glassescrab/mlir-aie, branch feature/update-mix-mm-int4-verification, Apache-2.0 with LLVM Exceptions — licence read, no code copied)
packs weights on disk as [packed int4 | bf16 scales | int8 zero-points] blocks at group 128, 128 B-padded, in an interleaved
column-major order so K/N reach 32K, and the kernel does unpack+dequant+GEMM in one pass (mix_int4_ATB.cc, their §3.3). There is no
host pack at all, which is why their numbers are end-to-end: 501.9 tok/s at 1K context on Krackan, which at 8.03 B parameters is
16.06 GFLOP/token ⇒ ~8.06 TFLOP/s end to end. Our 1.54 TFLOP/s kernel rate is ~5x below that, and our route is ~16x below its own
kernel because of the pack — so the two deficits are independent and fixing one leaves the other.
What that means for the ordering in your fix list: items 1-3 (pack only am rows, the A-path stores / sync_a_to_device, and the
fused-pass + OMP shape) recover the 16x and are host code, exactly as you have it. But your item 4 ("only then consider moving the
expansion on-core for prefill") now has a published target rather than a guess: on-core dequant with the weights pre-tiled on disk is
their whole design, and it buys from 1.54 TFLOP/s toward ~8 — i.e. it is the ~5x, and it is worth ~4x more than any further arithmetic work
on the host side once the pack is gone. The two levers multiply into the ~80x that separates 18.64 tok/s from their class of number.
One caveat so this is not over-read: their 501.9 tok/s is Krackan (Ryzen AI 7 350) at 1K context and they report the NPU slower than the
iGPU below ~1k tokens on HX 370 — Max+ 395, the part I measured on, is the iGPU-heavy one in HeteroMosaic's own classification, so the
crossover likely sits higher here. Their reproduction on 395 is blocked on this box anyway (no aiecc/xchesscc, empty ironenv* venvs),
which is why I measured our kernel on their shapes instead.
Second correction, from reading current main rather than trusting my own table: the pack figure needs a re-take before it is optimised, and I have located exactly what to meter.
The table I quoted has no print site in current main. The [prof] pre per-role pack/wait rows came from the 2026-11-02 build; grep -rn '"\[prof\]' npu/forward/src npu/forward/include on main now finds one site (q4nx_forward.cpp:4666, the token-level line). So the 24 ms/launch is a historical measurement, not something a reader can reproduce with the current tree, and the first step of this issue is re-instrumenting the prefill phase split, not writing the fix.
The paths are now identified precisely. The M=16/pb16i launch that carries the per-call meters is xdna_target.cpp:473 (tp) → :552 (tl) — the pack window is everything from function entry to the launch, which includes the Hadamard rotation, the int8 quantise, and the BO work; that branch is hardcoded M=1 (decode). The batched/prefill path is I8Ctx::launch_async_with_bo_rows (npu/forward/engine-src/npu_engine_i8ctx_inc.h:532), which is quantize_async_rows(...) → sync_a_to_device() → (*kext)(...): three separable terms, none of them currently timed separately.
The arithmetic is not one of them. Device-free bench of the engine's own q4nx_quant_i8::row (AVX2, median of 20, 128-row blocks): 0.203 ms for K=4864, 0.584 ms for K=17408 — see the previous comment. So within any re-taken pack figure, quantise can account for a small fraction, and the candidates that remain are the writes into the mapped activation BO (Am + m*KD, byte stores) and sync_a_to_device().
Proposed instrumentation (env-gated, no behaviour change, ~30 lines): in the rows path, four steady_clock stamps around (a) rotation, (b) quantise into a RAM staging buffer, (c) RAM→BO copy, (d) sync_a_to_device(), printed as one [ps] role=… M=… rot=… quant=… copy=… sync=… ms line per 100 launches. That single line decides between the two fixes: if (c) dominates → the #871 staging shape (quantise to RAM, one bulk write); if (d) dominates → one sync per layer instead of per launch. It also, as a side effect, tests the staging hypothesis without committing to it.
I am not proposing the fix before that line exists: this issue has already been wrong once (the quantiser), and the honest state is "the symptom is real and large, the profiler that measured it is gone, the split takes one short window".
The other two inner terms are now bounded device-free, and together they leave ~90 % of the historical 24 ms unaccounted for — which points at the weight side. Same method as before: the engine's own routines, 128-row blocks, median of 20, under ~/lb/thermal-run.sh.
term
K=896
K=4864
K=6144
K=17408
Hadamard rotation, 128 rows (q4nx_hadamard::rotate, incl. the row copy)
0.021 ms
0.695 ms
0.977 ms
2.927 ms
int8 quantise, 128 rows (q4nx_quant_i8::row)
0.038 ms
0.203 ms
0.279 ms
0.584 ms
device wait per launch (2026-11-02, M=128)
—
1.20
1.24
1.35
Cross-check on the rotation: 5.4 µs/rotation at K=4864, which sits beside the 7.6 µs/rotation that #857 measured in-loop — the two instruments agree, so these are not benchmark artefacts.
For the G/U/D rows that reported 24.2 / 23.8 / 23.8 ms "pack", the three terms above sum to roughly 2.3 ms (rotation ~0.75 + quantise ~0.21 + launch 1.2-1.35). So ~90 % of the figure is neither rotation, nor activation quantise, nor the launch — and by elimination it is the weight-side term that sits in the same tp→tl window before the call: kctx.packB_into(...) on a weight-cache miss or kctx.packB(0, …) on the uncached path (xdna_target.cpp, the else branch of the weight_cache_ block).
That makes "is the weight cache missing per pass?" the question the run has to answer, and the run is instrumented for it ([ps] inside the call, the engine's own per-design table for the tp→tl total, and the [fill] line count for pb16i image reads — a cache miss that re-reads a 23.67 MB image shows up as ~1 GB/s of fill, which is exactly the 24 ms).
If that is what it is, the fix is not in the pack code at all: it is why a cache keyed on (layer, wkey) misses on every prefill pass, and prefill's whole 18.64 tok/s would be repacking weights that were packed once at serve start. That would also explain why the 2-bit stream bought 1.31x in decode (where the cache holds) and nothing in prefill.
Status of the code side: the [ps] instrumentation is written and pushed (npu/1416-pack-split @ 8cfa62bc, env-gated, unset path unchanged). The window is queued (first attempt expired after 900 s behind a 20-minute ticket — the FIFO is doing its job, not failing), and the split will be posted here when it lands.
First window ran (654 s wait, die 56 C, amdxdna 0) — and the useful result is that my instrumentation hits the wrong launch path. Reporting it before the numbers so nobody reads the split as the prefill split.
What the run shows
== request wall 98.65 s for 308 prompt tokens -> 3.12 tok/s prompt rate
[ps] n=1300 M=1 K=6144 per-launch: quant=0.024 sync=0.004 canary=0.008 launch=0.096 ms
[ps] n=1300 M=1 K=5120 per-launch: quant=0.019 sync=0.003 canary=0.024 launch=0.077 ms
[ps] n=1000 M=1 K=17408 per-launch: quant=0.066 sync=0.007 canary=0.008 launch=0.096 ms
[prof] dispatch decode=15 prefill=485 … prefill=1170 … fill lines: 0
Prefill did run batched — the dispatch counter climbs to 1170 prefill dispatches, and the log shows M=128 contexts being created for QKV/O/G/U. So the 2026-11-02 premise (a batched prefill path exists) holds on current main.
My [ps] meters never saw it. Every sampled line is M=1, and the launch_async_with_bo call sites in xdna_target.cpp are all hardcoded M=1 (:351, :365, :629, :712), so the M=128 prefill launches go through one of the other entry points — most likely launch_async_rows(0, A, M, K, scales) (the no-BO variant) or a path that does not share my instrumented function. n=1300 for a 316-token request is also far too small for 316 × 64 × ~5 launches, which confirms the meters only caught a minority of calls.
fill lines: 0 — no pb16i image re-reads at all in the window, so the "weight cache misses per pass" hypothesis is not supported by this run either: nothing re-read the weights.
The 3.12 tok/s prompt rate is not a prefill measurement: the box was at load 52 with a ticket ahead of mine, and prefill is the host-bound path, so contention hits it hardest. I am not comparing it to the 18.64 tok/s quiet figure — that would be the exact error this lane's own §6 rule exists to prevent. A quiet-window pair is needed before any regression claim.
So the next step is code, not a ticket: put the four stamps in a single helper and call it from every launch entry point (launch_async_rows, launch_async_with_bo, launch_async_with_bo_rows), so whichever path prefill takes is measured. Then one window gives the split, and the attribution question from the previous comment (rotation + quantise + launch ≈ 2.3 ms vs the historical 24 ms) finally gets its other half — or dissolves, if the 24 ms was itself a contended-window figure.
That possibility deserves saying out loud: the 24 ms/launch may be a contended measurement of a host-bound path, in which case this issue's real content is "prefill is host-bound" (which the release check's own 18.64 tok/s vs ~308 tok/s device-bound arithmetic still supports) and not "24 ms of pack" at all.
Two retractions of things I wrote in the last two comments — both wrong, both mine, both worth fixing before anyone plans around them.
1. "The 24 ms may itself be a contended measurement" — retracted. I checked the source before letting it stand: the [prof] pre table with pack 24.2 / 23.8 / 23.8 ms at M=128 comes from the same three npu-hold tickets (2026-11-02 19:47/19:48/19:49) whose entry describes the box as "load 4-9", i.e. a deliberately quiet window (~/briefs/TBONSAI-SPEED-STATE.md, 20:05 entry, the paragraph introducing the pair). So the figure is a low-load number and contention is not an explanation for it. Today's 3.12 tok/s prompt rate at load 52 stays what I said it was — uncomparable — but the historical 24 ms is innocent of that charge.
2. "fill lines: 0 retires the weight-cache hypothesis" — too strong, retracted.[fill] is emitted by the pb16i image read path. A packB/packB_into repack on a weight-cache miss is host compute over K×N floats (dequantise + int8 quantise + blocked-layout assembly of a 23.67 MB matrix) and would print no fill line at all. So fills=0 rules out "the images are re-read per pass" and says nothing about "the weights are re-packed per pass" — which is exactly the candidate that would explain ~22 ms of host time at M=128, and it remains the leading one.
Net state of the attribution, honestly: at a quiet load, the three terms I can measure inside the launch entry points sum to ~2.2 ms (rotation 0.70 + quantise 0.20 + launch 1.20-1.35, all benched with the engine's own routines). The historical pack column for the same launches was 24.2 ms. So ~22 ms sits in that same tp→tl window and in neither of the three — and the only other thing in that window is the weight side (packB_into on a cache miss / packB on the uncached branch).
What the currently-queued run will and will not settle.[ps] now stamps every launch entry point, so it pins the three inner terms on the real prefill path (the last window only saw M=1 decode-shaped calls — that is fixed). It does not cover packB, which happens before those calls in npu_gemm_rows. The decisive instrument for the remaining 22 ms is one stamp pair around the packB_into/packB calls plus a counter of cache hits and misses — if a prefill pass misses the cache once per layer, this issue's fix is a cache-key bug, not a pack optimisation, and prefill's ~308 tok/s device-bound ceiling becomes reachable without touching a kernel.
I would rather retract two of my own statements in one comment than have a lane spend a day on either.
Third instrument bug of mine, caught before it misled anyone: the [ps] line printed the last call's M next to a cumulative average, so "M=1" was never evidence that prefill is unbatched.
The second window (with the splitter on all four entry points) came back the same way as the first — [ps] n=1300 M=1 K=… — and I nearly wrote "this container does not batch prefill at all" on the strength of it. It does not follow: am/ak in that print are taken from the most recent invocation, while n, quant, sync etc. are accumulated over every launch since the context started. A mixed counter whose last call is a single-row decode launch prints M=1 no matter how many M=128 prefill launches it also contains. That also explains n=1300 for a 316-token request without contradicting the dispatch counter.
Fixed: the reporter now buckets by row count and prints every bucket —
[ps] total=1300 launches, by rows: | M=128 K=5120 n=… quant=… sync=… canary=… launch=… ms | M=1 K=6144 n=… …
— so one line answers the question the last two windows could not: whether prefill's launches are in there, and what their four terms cost. Rebuilt.
What this leaves standing from the two runs that did happen (both at load 27-52, so quoted as structure, not as absolutes):
amdxdna errors 0 in both windows; fill lines: 0 in both, i.e. no pb16i image re-reads — which still only rules out image re-reads per pass, not a packB repack (§ my previous comment).
The decode-shaped calls that are visible cost, per launch, quant 0.019-0.073 ms, sync 0.003-0.008, canary 0.008-0.029, launch-issue 0.077-0.481 ms — the quantise figure matching the device-free bench (0.203 ms for 128×4864 ≈ 0.0016 ms/row, so 1 row ≈ 0.002 ms… which is lower than the 0.019 ms seen here, i.e. per-launch overhead dominates at M=1, as expected).
[wm] (cache-miss packB_into / uncached packB, with counts) is built and pushed but was not in the binary that these two windows used; it needs its own window.
Net: the attribution question in this issue is still open, and it is open because of my instrumentation rather than because of the engine. I would rather say that plainly than publish a fourth attribution from a print whose fields mean something different from what they look like.
Independent confirmation, on a different model and route — and one detail that narrows the cause
This lane hit the same wall from the other side (Qwen3-VL-2B text half on the array, q4_1/rm container, engine#1311's
TQVL work). Rows from NPU_PHASE_PROF=1 and the [lh] ranked call sites, one warm 16-token decode:
[ph] wall=4260.2 ms layers=3445.0 (pack=0.0 ql=0.0 wait=0.0 rb=0.0 sum=0.0 host=3445.0) head=813.8 (host=813.8)
[lh] npu_gemm (total) 3442.16 ms n=112 30.734 ms/call 99.9%
[lh] gemm host (amax+deq) 3442.16 ms 99.9% of gemm-free host
Same shape as yours: 112 launches/pass (28 layers × 4 designs), ~30.7 ms per launch of host work, device phases
~0. Three A/Bs to sharpen where it goes:
A/B
result
packed-weight cache off (NPU_NO_WCACHE=1) vs on
0.021 vs 0.158 tok/s → the cache is worth 7.6× and is doing its job
NPU_FFN_CACHE=1 forced on (the ffn_cache_default() gate wants lin_kd_ > 0)
0.766 vs 0.812 tok/s → no gain, so that gate is not costing anything
The last row is the useful one for your issue: the per-launch cost tracks the design's padded row count (MD), not
the live row count (am = 1 during decode, 128 during prefill) — halving MD from 128 to 16 cut it 5.5×, and the phase
table's pack column stays 0.0 while gemm host (amax+deq) carries it. On this tree quantize_async_rows already
loops m < am and prepare_activation already zeroes/dirties only [am, a_written_rows_) and sync_a_to_device is narrow unless NPU_DBG/NPU_NO_A_NARROW is set, so your candidate #1 looks partly in —
which is why I would re-run your 24 ms/launch figure on current main before taking it as the target. If it is still
24 ms with am small, the remaining suspect is whatever still touches the full MD × KD image per launch.
Claiming it, if nobody is on it
Your issue is unassigned and this lane is the NPU forward's owner; my TQVL work is blocked on an owner decision, so I
have room. I would start exactly at your gate: re-measure the pack column on current main (Bonsai-2-27B t2s2, M=128, prompt_ms/[prof] pre before and after), then take candidate #1/#2 (single-sweep amax+quantise, OMP over rows,
AVX2) with byte-exact prefill logits as the control. Say if you are already in it and I will stay off.
The control answers it, and the answer retires the regression hypothesis: the 10x gap between my runs and the 18.64 tok/s row is the box, not the build.
Control, deployed binary, same 308-token prompt, same port, same contended box:389.92 s → 0.79 tok/s prompt rate, amdxdna 0, fill lines: 0. My build on the same prompt: 1.94 tok/s (and 3.12 on an earlier, less loaded window). So on identical conditions main is ~2.4x faster than the deployed t2s2-20261104 binary at this prompt, not slower — there is no prefill-routing regression.
Why the numbers are so far below the 18.64 tok/s row, and why I should have checked this before drawing conclusions: that row is pp512 | 18.64 tok/s (425-tok prompt) from ~/lb/supreme/rows-bonsai-tokJ.md, produced by ~/wt/npu-power-row.sh, whose own rule is MAXLOAD=${MAXLOAD:-12} — it refuses to run above load 12. My windows today ran at load 27, 49.8 and 55 (load15 64.6). Prefill is the host-bound path in this engine, so a 4-16x CPU oversubscription inflates exactly the term under investigation; a 10x drop in prompt rate at load ~50 is what the hypothesis predicts, not evidence against the engine.
What that means for the two structural claims I made from those windows — one stands, one is withdrawn:
Stands: the sampled M=1 prints come from a call site that only ever handles single-row launches, so they say nothing about whether prefill batches. That was already my correction two comments ago and it remains the position.
Withdrawn: any implication that prefill is not batching, or that the 24 ms pack figure is suspect, because both rested on windows that could not measure this path. The quiet-box figure stands as the only measurement of prefill this issue has.
Consequently: my bucketed [ps] prints never fired in these windows either — which is consistent with per-context launch counts staying under 100 once ctx_for(rows) splits the 192 prefill launches across contexts, and is therefore also not evidence.
Protocol from here, so this does not repeat a sixth time: prefill magnitudes on this box are only measurable under ~/wt/npu-power-row.sh's own rule (load ≤ 12). The instrumentation in #1431 (READY) is the tool that extracts the split in one such window; until the fleet is quiet, this issue should carry no new magnitude numbers, and I will not take another ticket for them. The attribution question stays open with the same evidence it had before today: three terms benched at ~2.2 ms, 24.2 ms reported, packB on a cache miss as the live candidate, and a meter ready to measure it.
I would rather withdraw a structural claim than let it stand on a load-50 measurement — this lane has published that error class twice before (TBONSAI-HYBRID.md §6) and it is still the easiest mistake to make on this box.
Cross-lane datum for this issue, since the launch count is what your table multiplies: as of 5bdc441b (#1382, merged
about an hour ago) a prefill longer than the activation block runs as several launches, not one.
The device BO holds exactly max_batch_rows() rows (128 for the full_i8 family). Before #1382 a wider batch was not
quantized at all — it corrupted memory (4/4 SIGSEGV on the d3 pack's real 225-token questions, #1372) — so for a long prompt
there was no pack cost to speak of, just a crash. It is now walked in max_batch_rows()-sized blocks inside npu_gemm_rows(), so:
a 225-token prefill (the published d3 states) issues ⌈225/128⌉ = 2 blocks per projection per layer, and the 687-token
tail of that benchmark needs 6;
at the ~24 ms pack cost in your table, that is ~48 ms per G/U/D launch pair per block on a 2-block prompt — i.e. the pack
cost scales with ⌈M/128⌉ for any prompt that is not a multiple of the block, where a single-launch design paid it once;
measured end to end on the route that motivates it (d3-lite 0.8B, 24 layers, 225-token questions, amdxdna errors 0): 3.8–4.0 s per decision on the array versus 13 h of the same work on host GEMM at 24 s/decision — so the blocking is
a large net win, but its cost is exactly the per-launch pack term you are attacking.
Nothing here disputes your table; it says the multiplier is now ⌈M/128⌉ rather than 1. If a future change removes the
128-row BO limit (or the pack moves to the device), the block loop collapses back to one launch per projection and that
term disappears — which is another reason the pack cost is worth fixing rather than working around.
Sources: npu/forward/src/q4nx_forward.cpp (npu_gemm_rows, the blk > 0 && M > blk path, commit 5bdc441b); rows in ~/lb/d3eval/dev0-results.json and #796's rung-3 comments; crash/fix evidence on #1372.
Narrowing it further: on the q4_1/rm path the per-launch cost is the call itself, linear in launched rows — not host pack
Follow-up to my comment above, with three more A/Bs and then the decisive one (device-only per-call timings from onebit_npu_gemm_q8_probe, which is gemm_q8_raw: BO pack + launch + wait + readback, no forward in the loop).
Two more host-side hypotheses closed (Qwen3-VL-2B text container, m16 decode designs, warm):
A/B
result
BO cache-maintenance syncs off (NPU_NO_A_FLUSH=1 NPU_NO_C_INVAL=1)
0.762 vs 0.835 tok/s — no gain, and 0 of 8 logprob probes identical to the synced arm, i.e. the syncs are load-bearing; they must not be shipped as a speed lever
NPU_FFN_CACHE=1 forced (the gate wants lin_kd_ > 0)
no gain (0.766 vs 0.812)
The decisive measurement — device-only per-call cost, my geometry, M=16 vs M=128:
shape (int8, q4_1/rm)
M=16
M=128
per row
K2048 × N4096 (QKV)
4.28 ms/call
39.55 ms/call
0.27 / 0.31 ms
K2048 × N12288 (GU)
15.34 ms/call
—
~0.31 ms (at 16 rows)
Two things follow. First, the forward's 5.55 ms/call at M=16 is the kernel call itself (4.3–15.3 ms depending on
the shape, averaging with the head to ~5.5) — there is no separate 24 ms host-pack layer on this container to attack. Second, the cost is linear in the launched rows (~0.28 ms/row), not fixed: 8× the rows costs 9.2× the time. So what
this route leaves on the table for decode is launched rows that hold no data — 16 rows carrying 1 token — and the
m16 twins already took 8× of that (30.7 → 5.55 ms/call, 0.166 → 0.90 tok/s). The next lever for decode is not packing
but a genuinely narrow design (or batching several requests' tokens into one launch).
And the underlying efficiency, stated plainly because it bounds everything above: 16.8 M MACs per row in 0.27 ms is ~56 GFLOP/s per row, ~2 % of this array's int8 peak. On the q4_1/rm layout the kernel itself is the ceiling; your
Bonsai-27B t2s2 numbers (24 ms/launch pack vs 1.3 ms device) are a different layout, where the pack column really is
the pass. So this issue may be two issues: "host-pack bound" for pb16i/t2s2, and "kernel-efficiently-low + wasted rows"
for q4_1/rm. I would keep the fix scoped to the layout each measurement names rather than one pack-only fix.
Found my own comparison error, and it explains the whole discrepancy: the 18.64 tok/s row is prompt_tokens / TTFT of a streaming request, while my "prompt rate" was prompt tokens over the entire request wall. They are different quantities and I compared them anyway.
The row (~/lb/supreme/rows-bonsai-tokJ.md, pp512 18.64) is produced by ~/wt/prefill-bench.py, whose own doc says: "With stream: true the first chunk is the prefill plus one token (= TTFT) and the last chunk is the last token, so prefill tok/s = prompt_tokens / ttft".
My pack-split.sh measured == request wall 143.75 s for 308 prompt tokens -> 2.14 tok/s prompt rate — that divides prompt tokens by a wall clock that also contains 8 decode steps and the whole per-token walk. Calling it a "prompt rate" and setting it against TTFT-based tok/s is an apples-to-oranges comparison, and it is the fourth measurement error of mine in this issue (after the wrong function, the wrong coverage, and the misleading M=1 print).
What this does and does not change:
The "10x gap" I was chasing, and the load-12 experiment I built a gate for, were both aimed at the wrong quantity. Load is still a real confound for prefill, but it was never the explanation for a 10x difference between two different units.
The control result stands on its own terms and is still useful: on identical conditions, same script, same prompt, main's request wall was 143.75 s against the deployed binary's 389.92 s — same quantity, so main is ~2.7x faster at this prompt, no regression.
The attribution question is untouched: three terms at ~2.2 ms, 24.2 ms reported, packB on a cache miss live.
Running now, with the row's own instrument:ab-prefill-ttft.sh — prefill-bench.py --rows prefill --greedy --prompts 512 --decode 1 against deployed then main in adjacent windows inside one quiet-gated ticket, printing prompt_tokens, ttft_s, prefill_tps and text_sha256 for each. That produces numbers in the row's units from both binaries, and the sha tells us whether the A/B changed the model (it must not). If deployed reproduces ≈18.64 and main does not, the issue is a main-side prefill regression in the row's own units — worth far more than the pack question. If both land near each other, the row is reproducible and the attribution continues from a sound baseline.
The gate is the same quiet-gate.sh (load1 ≤ 12, bounded, prints load at start and end), so this cannot produce another withdrawal.
The fused chunks→int8 prototype is written and not proven byte-exact yet — here is exactly what it shows, and why I am not pushing it.
The piece: dequantise one 32×256 q4_1 tile into a 32 KB scratch, fold it into the per-column amax, then a second pass emits the codes. No K×N float buffer, so it removes the 356 MB per projection instead of making the passes over it faster.
Test: a synthetic 2×2 tile grid (K=64, N=512), running both paths (dequant_tensor_chunked → pack_B_int8, and the fused function) and comparing every code, every scale and the mean.
ok fused: every column scale is byte-identical as well
ok fused: the mean column scale matches the two-step return value
FAIL fused: every code is byte-identical (316 of 32768 differ)
mismatch at row 1 col 160: fused 127 ref 0
mismatch at row 1 col 161: fused 35 ref 0
So: all 512 scales agree exactly, and 3,2652 of 32,768 codes agree — which rules out the packing arithmetic and the amax/floor rule. What differs is that the reference has zeros where the tile decode has values, inside tile (0,0), i.e. it is not the nibble decode or the scale.
Two candidate explanations, and I did not have time to separate them honestly:
q4nx_dequant_tensor_chunked does not write plain row-major for this geometry — the mismatches concentrated at col 160 onwards in row 1 look like a layout/stride difference, not a value difference. If the tensor path lays tiles out in a blocked order that only coincides with row-major for some dimensions, then pack_B_int8 reading it as B[i*N+j] is the thing to question, not the fused path.
My synthetic tile is not a legal q4_1 tile in every byte — I filled 5120 bytes with a pattern without encoding a valid per-group header, so the two decoders may legitimately disagree on the 16-byte header region (scales would then differ too, and they do not).
The first explanation fits the evidence better (scales identical, layout-shaped mismatch), but "fits better" is not "proven", and a byte-exactness claim is exactly the thing not to push on a plausible story. So the prototype is stashed, the test is written, and the remaining work is one debug session on the tile→B layout.
What this does not change:#1707 (row-major packing, 2.8×, byte-identical against #1698's fixture) stands on its own and is pushed. The fused path is the change that removes the traffic rather than the per-pass cost, and it stays unshipped until the codes match.
The production packer packB_blocked is the worse traversal, not the better one — measured, because the review of #1707 asserted the opposite and I did not want to leave it either way.
I8Ctx::packB_blocked (npu_engine_i8ctx_inc.h:495) walks CB=64 stripes: for jb += 64, then for i < K, then 64 columns. The stripe is 64 floats = 256 B per row while rows are 69 KB apart, so every row still opens fresh cache lines — the stripe is too narrow to amortise a row-crossing. Replicating its exact loops (OpenMP pragma removed, K=5120, N=17408, 356.5 MB):
So in production the packer is fast because of #pragma omp parallel for, not because of its traversal: it divides 928 ms by the core count, and a row-major traversal would divide 299 ms by the same count. Per thread the difference is 3.1×, and that is on the pass the non-folded route runs once per weight.
It needs its own PR, with two things npu: pack_B_int8 walks the matrix in row order (2.8x, byte-identical) — #1416's dominant host pass #1707 does not need: the blocked index (bpos) is a permutation of the row-major position, so the traversal has to be rewritten without breaking that map; and the #pragma omp parallel for reduction(+:ssum) has to keep summing the same values in a deterministic way (the mean scale must not become order-dependent).
Not claimed: that 3.1× survives with threads (memory bandwidth may cap it — it caps the serial run at 1.2 GB/s already), or that the blocked map is trivially reorderable. Both are the first things to check in that PR.
Ledger of what landed for this issue tonight, and the one step that remains — so the next reader does not have to reconstruct it from merged PRs.
The claim in the title (prefill is host-pack bound: 24 ms/launch pack vs 1.3 ms device) produced five merged PRs, and the split between "landed" and "landed but not yet in
the path" is the only thing that matters now:
npu: the fused q4_1 tile->int8 packer (not called yet; wiring is the next slice) #1713 — the fused tiles → int8 packer (q4nx_pack_q4_1_tiles_to_int8), which allocates O(N + tile) instead of O(K·N): 356 KB instead of 356 MB at the 27B
shape. On main it has 0 call sites (grep -c in npu/forward/src/q4nx_forward.cpp → 0), and the merge commit says so: "not called yet; wiring is the next
slice". So the float matrix is still materialised on every weight; the saving is available but unused.
What I measured, rather than repeated: the row walk is 1.75× on the row-major layout and 2.18× on the blocked layout at K=5120 N=17408 — five interleaved
repetitions, medians, both variants extracted from their own branches into one binary — with memcmp over 89 M codes confirming byte-identity in both layouts. The
"2.7×" in #1720's title is directionally right and larger than what this box reproduces under load; the byte-identity is exact.
Remaining, and it is one edit:packB_blocked should take the fused path when the container can hand it tiles, and fall back to the two-step path otherwise (the -2 return from q4nx_pack_q4_1_tiles_to_int8 already exists for exactly that). Until then this issue's memory half is unclaimed and its throughput half is
partially delivered.
I am leaving the issue open on that basis rather than closing it on the throughput win: the title names a host-pack bound problem, and the bound is still there for the
allocation that dominates it.
So the wiring is: at the pack site, obtain the container's tile range for that weight and call q4nx_pack_q4_1_tiles_to_int8 directly, falling back to
assemble→pack when the tiles are not available (the fused function already returns -2 for a byte count that is not q4_1, which is exactly the fallback signal).
That is a plumb through the weight-loading path with 34 assemble sites as context and the tile-range lookup to add — a real change, with two properties that make it
worth doing carefully rather than quickly:
A wrong tile range means wrong weights, silently. The fused function decodes whatever bytes it is handed; if it is given a range offset by one tile, every code is
plausible and the model just answers differently. That is the engine#1605 class of failure, and the reason the golden pin (tests: pin the assemble -> pack composition a fused weight packer must reproduce (#1416) #1698) should be asserted per shape at
each pack site rather than once.
Its benefit is only visible on a device window. The O(N + tile) footprint is structural, but confirming this doesn't regress throughput needs a load of a real
pack, which is a ticket.
So I am not writing it in this pass. What remains for this issue is now stated accurately: the extraction, the golden and the row walk are in the path (#1720/#1707/#1698), the
fused packer is landed and uncalled (#1713), and the wiring is a defined plumb — 34 assemble sites, 3 pack sites, the -2 fallback already present — that nobody has
claimed and that needs a device window at the end. Anyone picking it up should start from the golden pin, not from the allocation.
Claiming this (pi-moe-npu, 0 assignees) — and the first two candidate fixes in the body are already in main, verified by reading the code at today's origin/main, so nobody needs to redo them.
At npu/forward/engine-src/npu_engine_i8ctx_inc.h:734 and npu/forward/include/quant_i8.h:
"Pack only the rows that exist (am, not MD)" — done.quantize_async_rows loops for (int m = 0; m < am; m++) (line ~755) and refuses loudly when am > MD rather than overrunning (RESULTS-coverage-multifamily 82 is quoted in the code comment). The companion half — not re-zeroing the whole BO on every launch — is also done: prepare_activation (line ~751) memsets only the row range that changed (a_written_rows_ bookkeeping + the ak != KD special case) instead of MD * KD.
"Vectorise the row quantiser" — done.q4nx_quant_i8::row() dispatches to an AVX2 implementation (row_avx2, target("avx2"), have_avx2() runtime check) whose comments are written around byte-identity with row_scalar, including the isfinite-vs-overflow distinction, the ±2^31 cast saturation, and a scalar tail for n % 8.
So the issue's opener ("the launch path … re-packs all MD rows on every launch") no longer describes main. What remains from the list is candidate 3 (fuse the amax sweep and the quantise sweep per row, and OMP across rows) and candidate 4 (on-core expansion — a new M=128 family, 10x the work).
What I am not doing tonight, and why: candidate 3 is a change to the per-launch path, and this plan's own rule is that such a change has to be proven byte-exact on the device plus three identical repeats before its speed is claimed. The box is at its 150 GB disk floor and the NPU queue is saturated by 13 lanes, so I can build and reason but I cannot measure — and shipping an unmeasured perf change to the launch path is exactly what §2 forbids. I would rather post this verified status than a half-gated diff.
What I can add to the measurement side immediately, if it is useful to whoever takes 3: a 1.4 s/token device run of an 8B pack on our per-op route is consistent with the host-pack claim from the other direction — the array answers (the first token matches the container's f32 reference on every prompt tried) while the pass is dominated by host work, i.e. the device is not the limit. That data point is in #1740 (issuecomment-6757626883).
Measured: the activation pack is no longer 24 ms. It is 16-149 microseconds — so this issue's premise is stale, and the remaining per-launch cost is somewhere else.
Built a host benchmark against the engine's own header (npu/forward/include/quant_i8.h, so it measures the shipped path, not a copy) and ran it on this box, origin/main:
shape
cost per launch
byte-identical to row_scalar?
am=16, KD=4096 (decode tile, 8B)
15.7 µs
yes
am=128, KD=4096 (prefill tile, 8B)
128 µs
yes
am=128, KD=4864 (prefill tile, 27B)
149 µs
yes
am=16, KD=14336 (decode tile, 27B)
55 µs
yes
Against the 1.20-1.35 ms device wait per launch that the body quotes, the pack is now ~1-12 % of a launch, not 93 %. That is what the two already-merged fixes did: quantize_async_rows packs only am rows (with a loud refusal instead of the old MD-row overrun), and q4nx_quant_i8::row() has the AVX2 path. The 24 ms/launch figure describes an older build.
Consequence for whoever takes this next: optimising the host pack further (candidate 3, fusion + OMP) has at most ~150 µs per launch to win on the 8B shapes — it cannot move prefill throughput. The 1.4 s/token this lane measures on an 8B pack through the per-op route is therefore not host packing: the array answers (the first token matches the container's f32 reference on every prompt tried — #1740, issuecomment-6757626883), so the cost is in the launch path around it (per-launch context switching, BO syncs, or the number of launches per token), and that is where the next measurement should go.
Reproducer committed (private repo, npu-dxb91593929): scripts/bench-quant-i8.cpp, with the build line in its header. It also asserts byte-identity of the AVX2 path against row_scalar on the same data, so a future "vectorise more" change has a one-command check that it did not change a single byte.
Status of this issue: the two cheapest candidates are done in main; candidate 3's ceiling is ~150 µs/launch; candidate 4 (on-core expansion) is unaffected by any of this. I am not opening a PR for candidate 3 because the measurement above says it cannot buy what the issue was opened to buy — the honest next step is a launch-path profile, not another host optimisation.
Re-measured at main (ee5b436), and the PR that closes this is #1949 (cl-npuperf).
The title no longer holds. Same pack as the issue (bonsai2-27b-t2s2), one npu-hold ticket, amdxdna error count unchanged:
issue body
main today
host pack per launch
24 ms
0.13 ms (53.6 ms over 416 launches)
share of a prefill block
93 %
7 %
first request of a serve vs third (5-token prompt, 8 tokens)
cold start paid the fill
2.99 s vs 2.90 s
prefill by TTFT (510-536-token prompts)
18.64 tok/s (since retracted)
20.8-21.4 tok/s
A 16-row block at main is 736 ms: device wait 215 (29 %), pack + quantise/launch + readback 110 (15 %), host work outside the launches 384 (52 %). The 24 ms was the first-touch weight fill, which #1621 moved in front of "ready".
What #1949 does: the largest single host term left was the GDN alpha/beta projection, serial inside the row loop (76 ms per block). It now runs inside the parallel region the layer already opens. Measured A B B A in one ticket: alpha/beta 77.5 -> 13.1-13.5 ms, block wall 734-749 -> 688-704 ms (21.4-21.8 -> 22.7-23.3 rows/s, about 6 %), greedy text identical, and the bit-pattern check is 336 of 336 layer passes identical on the device.
The body's candidate list: 1 and 2 were already in main (verified above in this thread), 3 is bounded at 0.13 ms/launch so it cannot move throughput, and 4 is #1421.
Activity
bong-water-water-bong commentedon Nov 8, 2026
Correction to my own fix list, measured before anyone acts on step 2 — the row quantiser's arithmetic is already fast and already vectorised; that is not what the 24 ms is.
Device-free bench of the engine's own routine (
npu/forward/include/quant_i8.h,q4nx_quant_i8::row, AVX2 path active), a whole M=128 block per iteration, median of 20, run under~/lb/thermal-run.sh:So quantising 128 rows of 4864 takes 0.203 ms against the ~24 ms the in-loop
[prof] precolumn reports for the same launch. ~99 % of "pack" is not arithmetic. Step 2 of the list above (vectorise the row quantiser) is therefore cancelled — it would optimise 0.8 % of the term — and step 1 (pack only real rows) keeps its value only for short prompts, where it is a real 5-fold saving on this same 24 ms.What is left, and it is now the whole issue:
quantize_async_rows()writes directly into the activation BO mapping (Am + (size_t)m * KD), then the caller doessync_a_to_device(). Two candidates, and they are separable in one host-side experiment:sync_a_to_device()itself — 622 KB per launch × 320 launches is a lot of driver sync for a pass whose device wait is 1.3 ms.Next diagnostic (no device needed): split the
packcolumn into quantise→RAM, RAM→BO copy, and sync, and report each. If (1) dominates, the staging fix is a few lines in the launch path; if (2) dominates, the lever is fewer/bigger syncs (one per layer rather than one per launch), which is a different PR and also cheap.The prefill ceiling math in the issue body is unchanged: device-bound for the same pass is ~0.43 s per 128 rows ≈ ~308 tok/s, and it is reachable once "pack" stops being 24 ms of a 25.3 ms launch.
bong-water-water-bong commentedon Nov 8, 2026
Cross-lane evidence for your fix list, from the TileFuse reproduction attempt: the kernel is 1.54 TFLOP/s and the route is pack-bound — and their stack has no host pack at all
I came at the same wall from the other side (owner task: reproduce TileFuse's Llama3-8B W4A16 prefill on Max+ 395). Two measurements worth
having here, both from one 11 s
npu-hold,amdxdna0 → 0,mosaic_bench --mode npuagainst our owngemm_q4_prefillT=256 sets:up_gateK=4096 N=14336up_gateK=4096 N=14336downK=14336 N=4096So the kernel alone is 1.54–1.59 TFLOP/s, and per-tile time is flat in M — which is the microbenchmark's device-only number, i.e. a
different quantity from your 24 ms
packcolumn, and it is not the thing that makes the route 18.64 tok/s.What TileFuse does that removes the pack column structurally rather than incrementally. Their artifact
(
glassescrab/mlir-aie, branchfeature/update-mix-mm-int4-verification, Apache-2.0 with LLVM Exceptions — licence read, no code copied)packs weights on disk as
[packed int4 | bf16 scales | int8 zero-points]blocks at group 128, 128 B-padded, in an interleavedcolumn-major order so K/N reach 32K, and the kernel does unpack+dequant+GEMM in one pass (
mix_int4_ATB.cc, their §3.3). There is nohost pack at all, which is why their numbers are end-to-end: 501.9 tok/s at 1K context on Krackan, which at 8.03 B parameters is
16.06 GFLOP/token ⇒ ~8.06 TFLOP/s end to end. Our 1.54 TFLOP/s kernel rate is ~5x below that, and our route is ~16x below its own
kernel because of the pack — so the two deficits are independent and fixing one leaves the other.
What that means for the ordering in your fix list: items 1-3 (pack only
amrows, the A-path stores /sync_a_to_device, and thefused-pass + OMP shape) recover the 16x and are host code, exactly as you have it. But your item 4 ("only then consider moving the
expansion on-core for prefill") now has a published target rather than a guess: on-core dequant with the weights pre-tiled on disk is
their whole design, and it buys from 1.54 TFLOP/s toward ~8 — i.e. it is the ~5x, and it is worth ~4x more than any further arithmetic work
on the host side once the pack is gone. The two levers multiply into the ~80x that separates 18.64 tok/s from their class of number.
One caveat so this is not over-read: their 501.9 tok/s is Krackan (Ryzen AI 7 350) at 1K context and they report the NPU slower than the
iGPU below ~1k tokens on HX 370 — Max+ 395, the part I measured on, is the iGPU-heavy one in HeteroMosaic's own classification, so the
crossover likely sits higher here. Their reproduction on 395 is blocked on this box anyway (no
aiecc/xchesscc, emptyironenv*venvs),which is why I measured our kernel on their shapes instead.
bong-water-water-bong commentedon Nov 8, 2026
Second correction, from reading current main rather than trusting my own table: the
packfigure needs a re-take before it is optimised, and I have located exactly what to meter.[prof] preper-rolepack/waitrows came from the 2026-11-02 build;grep -rn '"\[prof\]' npu/forward/src npu/forward/includeon main now finds one site (q4nx_forward.cpp:4666, the token-level line). So the 24 ms/launch is a historical measurement, not something a reader can reproduce with the current tree, and the first step of this issue is re-instrumenting the prefill phase split, not writing the fix.xdna_target.cpp:473(tp) →:552(tl) — thepackwindow is everything from function entry to the launch, which includes the Hadamard rotation, the int8 quantise, and the BO work; that branch is hardcodedM=1(decode). The batched/prefill path isI8Ctx::launch_async_with_bo_rows(npu/forward/engine-src/npu_engine_i8ctx_inc.h:532), which isquantize_async_rows(...)→sync_a_to_device()→(*kext)(...): three separable terms, none of them currently timed separately.q4nx_quant_i8::row(AVX2, median of 20, 128-row blocks): 0.203 ms for K=4864, 0.584 ms for K=17408 — see the previous comment. So within any re-takenpackfigure, quantise can account for a small fraction, and the candidates that remain are the writes into the mapped activation BO (Am + m*KD, byte stores) andsync_a_to_device().Proposed instrumentation (env-gated, no behaviour change, ~30 lines): in the rows path, four steady_clock stamps around (a) rotation, (b) quantise into a RAM staging buffer, (c) RAM→BO copy, (d)
sync_a_to_device(), printed as one[ps] role=… M=… rot=… quant=… copy=… sync=… msline per 100 launches. That single line decides between the two fixes: if (c) dominates → the #871 staging shape (quantise to RAM, one bulk write); if (d) dominates → one sync per layer instead of per launch. It also, as a side effect, tests the staging hypothesis without committing to it.I am not proposing the fix before that line exists: this issue has already been wrong once (the quantiser), and the honest state is "the symptom is real and large, the profiler that measured it is gone, the split takes one short window".
bong-water-water-bong commentedon Nov 8, 2026
The other two inner terms are now bounded device-free, and together they leave ~90 % of the historical 24 ms unaccounted for — which points at the weight side. Same method as before: the engine's own routines, 128-row blocks, median of 20, under
~/lb/thermal-run.sh.q4nx_hadamard::rotate, incl. the row copy)q4nx_quant_i8::row)Cross-check on the rotation: 5.4 µs/rotation at K=4864, which sits beside the 7.6 µs/rotation that #857 measured in-loop — the two instruments agree, so these are not benchmark artefacts.
For the G/U/D rows that reported 24.2 / 23.8 / 23.8 ms "pack", the three terms above sum to roughly 2.3 ms (rotation ~0.75 + quantise ~0.21 + launch 1.2-1.35). So ~90 % of the figure is neither rotation, nor activation quantise, nor the launch — and by elimination it is the weight-side term that sits in the same
tp→tlwindow before the call:kctx.packB_into(...)on a weight-cache miss orkctx.packB(0, …)on the uncached path (xdna_target.cpp, theelsebranch of theweight_cache_block).That makes "is the weight cache missing per pass?" the question the run has to answer, and the run is instrumented for it (
[ps]inside the call, the engine's own per-design table for thetp→tltotal, and the[fill]line count for pb16i image reads — a cache miss that re-reads a 23.67 MB image shows up as ~1 GB/s of fill, which is exactly the 24 ms).If that is what it is, the fix is not in the pack code at all: it is why a cache keyed on
(layer, wkey)misses on every prefill pass, and prefill's whole 18.64 tok/s would be repacking weights that were packed once at serve start. That would also explain why the 2-bit stream bought 1.31x in decode (where the cache holds) and nothing in prefill.Status of the code side: the
[ps]instrumentation is written and pushed (npu/1416-pack-split@8cfa62bc, env-gated, unset path unchanged). The window is queued (first attempt expired after 900 s behind a 20-minute ticket — the FIFO is doing its job, not failing), and the split will be posted here when it lands.bong-water-water-bong commentedon Nov 8, 2026
First window ran (654 s wait, die 56 C,
amdxdna0) — and the useful result is that my instrumentation hits the wrong launch path. Reporting it before the numbers so nobody reads the split as the prefill split.What the run shows
[ps]meters never saw it. Every sampled line isM=1, and thelaunch_async_with_bocall sites inxdna_target.cppare all hardcodedM=1(:351, :365, :629, :712), so the M=128 prefill launches go through one of the other entry points — most likelylaunch_async_rows(0, A, M, K, scales)(the no-BO variant) or a path that does not share my instrumented function. n=1300 for a 316-token request is also far too small for 316 × 64 × ~5 launches, which confirms the meters only caught a minority of calls.fill lines: 0— no pb16i image re-reads at all in the window, so the "weight cache misses per pass" hypothesis is not supported by this run either: nothing re-read the weights.So the next step is code, not a ticket: put the four stamps in a single helper and call it from every launch entry point (
launch_async_rows,launch_async_with_bo,launch_async_with_bo_rows), so whichever path prefill takes is measured. Then one window gives the split, and the attribution question from the previous comment (rotation + quantise + launch ≈ 2.3 ms vs the historical 24 ms) finally gets its other half — or dissolves, if the 24 ms was itself a contended-window figure.That possibility deserves saying out loud: the 24 ms/launch may be a contended measurement of a host-bound path, in which case this issue's real content is "prefill is host-bound" (which the release check's own 18.64 tok/s vs ~308 tok/s device-bound arithmetic still supports) and not "24 ms of pack" at all.
npu: [ps] covers every launch entry point, not just the with_bo_rows …
bong-water-water-bong commentedon Nov 8, 2026
Two retractions of things I wrote in the last two comments — both wrong, both mine, both worth fixing before anyone plans around them.
1. "The 24 ms may itself be a contended measurement" — retracted. I checked the source before letting it stand: the
[prof] pretable withpack 24.2 / 23.8 / 23.8 msat M=128 comes from the same threenpu-holdtickets (2026-11-02 19:47/19:48/19:49) whose entry describes the box as "load 4-9", i.e. a deliberately quiet window (~/briefs/TBONSAI-SPEED-STATE.md, 20:05 entry, the paragraph introducing the pair). So the figure is a low-load number and contention is not an explanation for it. Today's 3.12 tok/s prompt rate at load 52 stays what I said it was — uncomparable — but the historical 24 ms is innocent of that charge.2. "
fill lines: 0retires the weight-cache hypothesis" — too strong, retracted.[fill]is emitted by the pb16i image read path. ApackB/packB_intorepack on a weight-cache miss is host compute over K×N floats (dequantise + int8 quantise + blocked-layout assembly of a 23.67 MB matrix) and would print no fill line at all. So fills=0 rules out "the images are re-read per pass" and says nothing about "the weights are re-packed per pass" — which is exactly the candidate that would explain ~22 ms of host time at M=128, and it remains the leading one.Net state of the attribution, honestly: at a quiet load, the three terms I can measure inside the launch entry points sum to ~2.2 ms (rotation 0.70 + quantise 0.20 + launch 1.20-1.35, all benched with the engine's own routines). The historical
packcolumn for the same launches was 24.2 ms. So ~22 ms sits in that sametp→tlwindow and in neither of the three — and the only other thing in that window is the weight side (packB_intoon a cache miss /packBon the uncached branch).What the currently-queued run will and will not settle.
[ps]now stamps every launch entry point, so it pins the three inner terms on the real prefill path (the last window only saw M=1 decode-shaped calls — that is fixed). It does not coverpackB, which happens before those calls innpu_gemm_rows. The decisive instrument for the remaining 22 ms is one stamp pair around thepackB_into/packBcalls plus a counter of cache hits and misses — if a prefill pass misses the cache once per layer, this issue's fix is a cache-key bug, not a pack optimisation, and prefill's ~308 tok/s device-bound ceiling becomes reachable without touching a kernel.I would rather retract two of my own statements in one comment than have a lane spend a day on either.
npu: [wm] times the weight-side pack in the same window as the activa…
bong-water-water-bong commentedon Nov 8, 2026
Third instrument bug of mine, caught before it misled anyone: the
[ps]line printed the last call'sMnext to a cumulative average, so "M=1" was never evidence that prefill is unbatched.The second window (with the splitter on all four entry points) came back the same way as the first —
[ps] n=1300 M=1 K=…— and I nearly wrote "this container does not batch prefill at all" on the strength of it. It does not follow:am/akin that print are taken from the most recent invocation, whilen,quant,syncetc. are accumulated over every launch since the context started. A mixed counter whose last call is a single-row decode launch printsM=1no matter how many M=128 prefill launches it also contains. That also explainsn=1300for a 316-token request without contradicting the dispatch counter.Fixed: the reporter now buckets by row count and prints every bucket —
— so one line answers the question the last two windows could not: whether prefill's launches are in there, and what their four terms cost. Rebuilt.
What this leaves standing from the two runs that did happen (both at load 27-52, so quoted as structure, not as absolutes):
amdxdnaerrors 0 in both windows;fill lines: 0in both, i.e. no pb16i image re-reads — which still only rules out image re-reads per pass, not apackBrepack (§ my previous comment).[wm](cache-misspackB_into/ uncachedpackB, with counts) is built and pushed but was not in the binary that these two windows used; it needs its own window.Net: the attribution question in this issue is still open, and it is open because of my instrumentation rather than because of the engine. I would rather say that plainly than publish a fourth attribution from a print whose fields mean something different from what they look like.
npu: [ps] buckets by row count instead of printing the last call's M …
bong-water-water-bong commentedon Nov 8, 2026
Independent confirmation, on a different model and route — and one detail that narrows the cause
This lane hit the same wall from the other side (Qwen3-VL-2B text half on the array, q4_1/rm container, engine#1311's
TQVL work). Rows from
NPU_PHASE_PROF=1and the[lh]ranked call sites, one warm 16-token decode:Same shape as yours: 112 launches/pass (28 layers × 4 designs), ~30.7 ms per launch of host work, device phases
~0. Three A/Bs to sharpen where it goes:
NPU_NO_WCACHE=1) vs on[WCHK i8]counter (NPU_DBG=1) during a request_m16+plan.jsondecode block)NPU_FFN_CACHE=1forced on (theffn_cache_default()gate wantslin_kd_ > 0)The last row is the useful one for your issue: the per-launch cost tracks the design's padded row count (MD), not
the live row count (
am= 1 during decode, 128 during prefill) — halving MD from 128 to 16 cut it 5.5×, and the phasetable's
packcolumn stays 0.0 whilegemm host (amax+deq)carries it. On this treequantize_async_rowsalreadyloops
m < amandprepare_activationalready zeroes/dirties only[am, a_written_rows_)andsync_a_to_deviceis narrow unlessNPU_DBG/NPU_NO_A_NARROWis set, so your candidate #1 looks partly in —which is why I would re-run your 24 ms/launch figure on current
mainbefore taking it as the target. If it is still24 ms with
amsmall, the remaining suspect is whatever still touches the fullMD × KDimage per launch.Claiming it, if nobody is on it
Your issue is unassigned and this lane is the NPU forward's owner; my TQVL work is blocked on an owner decision, so I
have room. I would start exactly at your gate: re-measure the pack column on current
main(Bonsai-2-27B t2s2, M=128,prompt_ms/[prof] prebefore and after), then take candidate #1/#2 (single-sweep amax+quantise, OMP over rows,AVX2) with byte-exact prefill logits as the control. Say if you are already in it and I will stay off.
bong-water-water-bong commentedon Nov 8, 2026
The control answers it, and the answer retires the regression hypothesis: the 10x gap between my runs and the 18.64 tok/s row is the box, not the build.
Control, deployed binary, same 308-token prompt, same port, same contended box:
389.92 s → 0.79 tok/sprompt rate,amdxdna0,fill lines: 0. My build on the same prompt:1.94 tok/s(and3.12on an earlier, less loaded window). So on identical conditions main is ~2.4x faster than the deployedt2s2-20261104binary at this prompt, not slower — there is no prefill-routing regression.Why the numbers are so far below the 18.64 tok/s row, and why I should have checked this before drawing conclusions: that row is
pp512 | 18.64 tok/s (425-tok prompt)from~/lb/supreme/rows-bonsai-tokJ.md, produced by~/wt/npu-power-row.sh, whose own rule isMAXLOAD=${MAXLOAD:-12}— it refuses to run above load 12. My windows today ran at load 27, 49.8 and 55 (load15 64.6). Prefill is the host-bound path in this engine, so a 4-16x CPU oversubscription inflates exactly the term under investigation; a 10x drop in prompt rate at load ~50 is what the hypothesis predicts, not evidence against the engine.What that means for the two structural claims I made from those windows — one stands, one is withdrawn:
M=1prints come from a call site that only ever handles single-row launches, so they say nothing about whether prefill batches. That was already my correction two comments ago and it remains the position.packfigure is suspect, because both rested on windows that could not measure this path. The quiet-box figure stands as the only measurement of prefill this issue has.[ps]prints never fired in these windows either — which is consistent with per-context launch counts staying under 100 oncectx_for(rows)splits the 192 prefill launches across contexts, and is therefore also not evidence.Protocol from here, so this does not repeat a sixth time: prefill magnitudes on this box are only measurable under
~/wt/npu-power-row.sh's own rule (load ≤ 12). The instrumentation in #1431 (READY) is the tool that extracts the split in one such window; until the fleet is quiet, this issue should carry no new magnitude numbers, and I will not take another ticket for them. The attribution question stays open with the same evidence it had before today: three terms benched at ~2.2 ms, 24.2 ms reported,packBon a cache miss as the live candidate, and a meter ready to measure it.I would rather withdraw a structural claim than let it stand on a load-50 measurement — this lane has published that error class twice before (
TBONSAI-HYBRID.md§6) and it is still the easiest mistake to make on this box.bong-water-water-bong commentedon Nov 8, 2026
Cross-lane datum for this issue, since the launch count is what your table multiplies: as of
5bdc441b(#1382, mergedabout an hour ago) a prefill longer than the activation block runs as several launches, not one.
The device BO holds exactly
max_batch_rows()rows (128 for thefull_i8family). Before #1382 a wider batch was notquantized at all — it corrupted memory (4/4 SIGSEGV on the d3 pack's real 225-token questions, #1372) — so for a long prompt
there was no pack cost to speak of, just a crash. It is now walked in
max_batch_rows()-sized blocks insidenpu_gemm_rows(), so:tail of that benchmark needs 6;
cost scales with ⌈M/128⌉ for any prompt that is not a multiple of the block, where a single-launch design paid it once;
amdxdnaerrors 0):3.8–4.0 s per decision on the array versus
13 hof the same work on host GEMM at 24 s/decision — so the blocking isa large net win, but its cost is exactly the per-launch pack term you are attacking.
Nothing here disputes your table; it says the multiplier is now
⌈M/128⌉rather than 1. If a future change removes the128-row BO limit (or the pack moves to the device), the block loop collapses back to one launch per projection and that
term disappears — which is another reason the pack cost is worth fixing rather than working around.
Sources:
npu/forward/src/q4nx_forward.cpp(npu_gemm_rows, theblk > 0 && M > blkpath, commit5bdc441b); rows in~/lb/d3eval/dev0-results.jsonand#796's rung-3 comments; crash/fix evidence on #1372.bong-water-water-bong commentedon Nov 8, 2026
Narrowing it further: on the q4_1/rm path the per-launch cost is the call itself, linear in launched rows — not host pack
Follow-up to my comment above, with three more A/Bs and then the decisive one (device-only per-call timings from
onebit_npu_gemm_q8_probe, which isgemm_q8_raw: BO pack + launch + wait + readback, no forward in the loop).Two more host-side hypotheses closed (Qwen3-VL-2B text container, m16 decode designs, warm):
NPU_NO_A_FLUSH=1 NPU_NO_C_INVAL=1)NPU_FFN_CACHE=1forced (the gate wantslin_kd_ > 0)The decisive measurement — device-only per-call cost, my geometry, M=16 vs M=128:
Two things follow. First, the forward's 5.55 ms/call at M=16 is the kernel call itself (4.3–15.3 ms depending on
the shape, averaging with the head to ~5.5) — there is no separate 24 ms host-pack layer on this container to attack.
Second, the cost is linear in the launched rows (~0.28 ms/row), not fixed: 8× the rows costs 9.2× the time. So what
this route leaves on the table for decode is launched rows that hold no data — 16 rows carrying 1 token — and the
m16 twins already took 8× of that (30.7 → 5.55 ms/call, 0.166 → 0.90 tok/s). The next lever for decode is not packing
but a genuinely narrow design (or batching several requests' tokens into one launch).
And the underlying efficiency, stated plainly because it bounds everything above: 16.8 M MACs per row in 0.27 ms is
~56 GFLOP/s per row, ~2 % of this array's int8 peak. On the q4_1/rm layout the kernel itself is the ceiling; your
Bonsai-27B t2s2 numbers (24 ms/launch pack vs 1.3 ms device) are a different layout, where the pack column really is
the pass. So this issue may be two issues: "host-pack bound" for pb16i/t2s2, and "kernel-efficiently-low + wasted rows"
for q4_1/rm. I would keep the fix scoped to the layout each measurement names rather than one pack-only fix.
bong-water-water-bong commentedon Nov 8, 2026
Found my own comparison error, and it explains the whole discrepancy: the 18.64 tok/s row is
prompt_tokens / TTFTof a streaming request, while my "prompt rate" was prompt tokens over the entire request wall. They are different quantities and I compared them anyway.~/lb/supreme/rows-bonsai-tokJ.md,pp512 18.64) is produced by~/wt/prefill-bench.py, whose own doc says: "Withstream: truethe first chunk is the prefill plus one token (= TTFT) and the last chunk is the last token, so prefill tok/s = prompt_tokens / ttft".pack-split.shmeasured== request wall 143.75 s for 308 prompt tokens -> 2.14 tok/s prompt rate— that divides prompt tokens by a wall clock that also contains 8 decode steps and the whole per-token walk. Calling it a "prompt rate" and setting it against TTFT-based tok/s is an apples-to-oranges comparison, and it is the fourth measurement error of mine in this issue (after the wrong function, the wrong coverage, and the misleadingM=1print).What this does and does not change:
packBon a cache miss live.Running now, with the row's own instrument:
ab-prefill-ttft.sh—prefill-bench.py --rows prefill --greedy --prompts 512 --decode 1against deployed then main in adjacent windows inside one quiet-gated ticket, printingprompt_tokens,ttft_s,prefill_tpsandtext_sha256for each. That produces numbers in the row's units from both binaries, and the sha tells us whether the A/B changed the model (it must not). If deployed reproduces ≈18.64 and main does not, the issue is a main-side prefill regression in the row's own units — worth far more than the pack question. If both land near each other, the row is reproducible and the attribution continues from a sound baseline.The gate is the same
quiet-gate.sh(load1 ≤ 12, bounded, prints load at start and end), so this cannot produce another withdrawal.18 remaining items
bong-water-water-bong commentedon Nov 9, 2026
The fused chunks→int8 prototype is written and not proven byte-exact yet — here is exactly what it shows, and why I am not pushing it.
The piece: dequantise one 32×256 q4_1 tile into a 32 KB scratch, fold it into the per-column amax, then a second pass emits the codes. No
K×Nfloat buffer, so it removes the 356 MB per projection instead of making the passes over it faster.Test: a synthetic 2×2 tile grid (K=64, N=512), running both paths (
dequant_tensor_chunked→pack_B_int8, and the fused function) and comparing every code, every scale and the mean.So: all 512 scales agree exactly, and 3,2652 of 32,768 codes agree — which rules out the packing arithmetic and the amax/floor rule. What differs is that the reference has zeros where the tile decode has values, inside tile (0,0), i.e. it is not the nibble decode or the scale.
Two candidate explanations, and I did not have time to separate them honestly:
q4nx_dequant_tensor_chunkeddoes not write plain row-major for this geometry — the mismatches concentrated atcol 160onwards inrow 1look like a layout/stride difference, not a value difference. If the tensor path lays tiles out in a blocked order that only coincides with row-major for some dimensions, thenpack_B_int8reading it asB[i*N+j]is the thing to question, not the fused path.The first explanation fits the evidence better (scales identical, layout-shaped mismatch), but "fits better" is not "proven", and a byte-exactness claim is exactly the thing not to push on a plausible story. So the prototype is stashed, the test is written, and the remaining work is one debug session on the tile→B layout.
What this does not change: #1707 (row-major packing, 2.8×, byte-identical against #1698's fixture) stands on its own and is pushed. The fused path is the change that removes the traffic rather than the per-pass cost, and it stays unshipped until the codes match.
serve: forward-serve warms the weight cache before it says ready (44.…
bong-water-water-bong commentedon Nov 9, 2026
The production packer
packB_blockedis the worse traversal, not the better one — measured, because the review of #1707 asserted the opposite and I did not want to leave it either way.I8Ctx::packB_blocked(npu_engine_i8ctx_inc.h:495) walks CB=64 stripes:for jb += 64, thenfor i < K, then 64 columns. The stripe is 64 floats = 256 B per row while rows are 69 KB apart, so every row still opens fresh cache lines — the stripe is too narrow to amortise a row-crossing. Replicating its exact loops (OpenMP pragma removed, K=5120, N=17408, 356.5 MB):packB_blocked(CB=64), productionq4nx_pack_B_int8j-outer (what I fixed in #1707)q4nx_pack_B_int8row-major (#1707)So in production the packer is fast because of
#pragma omp parallel for, not because of its traversal: it divides 928 ms by the core count, and a row-major traversal would divide 299 ms by the same count. Per thread the difference is 3.1×, and that is on the pass the non-folded route runs once per weight.What that means for this issue's fix list
packB_blocked, which is a different function from the one npu: pack_B_int8 walks the matrix in row order (2.8x, byte-identical) — #1416's dominant host pass #1707 touches — the review of npu: pack_B_int8 walks the matrix in row order (2.8x, byte-identical) — #1416's dominant host pass #1707 was right that my PR had no production caller, and the measurement shows the production path has the same problem worse.bpos) is a permutation of the row-major position, so the traversal has to be rewritten without breaking that map; and the#pragma omp parallel for reduction(+:ssum)has to keep summing the same values in a deterministic way (the mean scale must not become order-dependent).Not claimed: that 3.1× survives with threads (memory bandwidth may cap it — it caps the serial run at 1.2 GB/s already), or that the blocked map is trivially reorderable. Both are the first things to check in that PR.
tests: pin the assemble -> pack composition a fused weight packer mus…
npu: the profiler dumps for a prefill-only run (it required a decode …
npu: pack_B_int8 walks the matrix in row order (2.8x, byte-identical) —
npu: the fused q4_1 tile->int8 packer (not called yet; wiring is the …
npu: the production packer walks rows, not 64-wide stripes (2.7x seri…
bong-water-water-bong commentedon Nov 10, 2026
Ledger of what landed for this issue tonight, and the one step that remains — so the next reader does not have to reconstruct it from merged PRs.
The claim in the title (prefill is host-pack bound: 24 ms/launch pack vs 1.3 ms device) produced five merged PRs, and the split between "landed" and "landed but not yet in
the path" is the only thing that matters now:
In the production path:
q4nx_pack_B_int8_blockedandI8Ctx::packB_blockeddelegates to it (2 references innpu_forward_i8ctx_inc.h), so the packer is testable without the device runtime. That PR also walks the matrix in row order instead of 64-wide stripes.amaxpass is row-major (it was stride-N, 424 MB/s against the scatter pass's 5.2 GB/s), same bytes out.against itself.
Landed, tested, and not yet called:
tiles → int8packer (q4nx_pack_q4_1_tiles_to_int8), which allocatesO(N + tile)instead ofO(K·N): 356 KB instead of 356 MB at the 27Bshape. On
mainit has 0 call sites (grep -cinnpu/forward/src/q4nx_forward.cpp→ 0), and the merge commit says so: "not called yet; wiring is the nextslice". So the float matrix is still materialised on every weight; the saving is available but unused.
What I measured, rather than repeated: the row walk is 1.75× on the row-major layout and 2.18× on the blocked layout at
K=5120 N=17408— five interleavedrepetitions, medians, both variants extracted from their own branches into one binary — with
memcmpover 89 M codes confirming byte-identity in both layouts. The"2.7×" in #1720's title is directionally right and larger than what this box reproduces under load; the byte-identity is exact.
Remaining, and it is one edit:
packB_blockedshould take the fused path when the container can hand it tiles, and fall back to the two-step path otherwise (the-2return fromq4nx_pack_q4_1_tiles_to_int8already exists for exactly that). Until then this issue's memory half is unclaimed and its throughput half ispartially delivered.
I am leaving the issue open on that basis rather than closing it on the throughput win: the title names a host-pack bound problem, and the bound is still there for the
allocation that dominates it.
bong-water-water-bong commentedon Nov 10, 2026
Correction to my own "remaining: one edit" line above — it is not one edit, and the structure says why.
I said
packB_blockedshould "take the fused path when the container can hand it tiles". Checking the signature and the call graph, that is wrong:By the time
packB_blockedruns, theK × Nmatrix already exists, so the tiles are not reachable there. The matrix is produced one level up:So the wiring is: at the pack site, obtain the container's tile range for that weight and call
q4nx_pack_q4_1_tiles_to_int8directly, falling back toassemble→pack when the tiles are not available (the fused function already returns -2 for a byte count that is not q4_1, which is exactly the fallback signal).
That is a plumb through the weight-loading path with 34 assemble sites as context and the tile-range lookup to add — a real change, with two properties that make it
worth doing carefully rather than quickly:
plausible and the model just answers differently. That is the engine#1605 class of failure, and the reason the golden pin (tests: pin the assemble -> pack composition a fused weight packer must reproduce (#1416) #1698) should be asserted per shape at
each pack site rather than once.
O(N + tile)footprint is structural, but confirming this doesn't regress throughput needs a load of a realpack, which is a ticket.
So I am not writing it in this pass. What remains for this issue is now stated accurately: the extraction, the golden and the row walk are in the path (#1720/#1707/#1698), the
fused packer is landed and uncalled (#1713), and the wiring is a defined plumb — 34 assemble sites, 3 pack sites, the
-2fallback already present — that nobody hasclaimed and that needs a device window at the end. Anyone picking it up should start from the golden pin, not from the allocation.
bong-water-water-bong commentedon Nov 10, 2026
Claiming this (pi-moe-npu, 0 assignees) — and the first two candidate fixes in the body are already in main, verified by reading the code at today's
origin/main, so nobody needs to redo them.At
npu/forward/engine-src/npu_engine_i8ctx_inc.h:734andnpu/forward/include/quant_i8.h:am, notMD)" — done.quantize_async_rowsloopsfor (int m = 0; m < am; m++)(line ~755) and refuses loudly whenam > MDrather than overrunning (RESULTS-coverage-multifamily 82is quoted in the code comment). The companion half — not re-zeroing the whole BO on every launch — is also done:prepare_activation(line ~751) memsets only the row range that changed (a_written_rows_bookkeeping + theak != KDspecial case) instead ofMD * KD.q4nx_quant_i8::row()dispatches to an AVX2 implementation (row_avx2,target("avx2"),have_avx2()runtime check) whose comments are written around byte-identity withrow_scalar, including the isfinite-vs-overflow distinction, the ±2^31 cast saturation, and a scalar tail forn % 8.So the issue's opener ("the launch path … re-packs all
MDrows on every launch") no longer describes main. What remains from the list is candidate 3 (fuse the amax sweep and the quantise sweep per row, and OMP across rows) and candidate 4 (on-core expansion — a new M=128 family, 10x the work).What I am not doing tonight, and why: candidate 3 is a change to the per-launch path, and this plan's own rule is that such a change has to be proven byte-exact on the device plus three identical repeats before its speed is claimed. The box is at its 150 GB disk floor and the NPU queue is saturated by 13 lanes, so I can build and reason but I cannot measure — and shipping an unmeasured perf change to the launch path is exactly what §2 forbids. I would rather post this verified status than a half-gated diff.
What I can add to the measurement side immediately, if it is useful to whoever takes 3: a 1.4 s/token device run of an 8B pack on our per-op route is consistent with the host-pack claim from the other direction — the array answers (the first token matches the container's f32 reference on every prompt tried) while the pass is dominated by host work, i.e. the device is not the limit. That data point is in
#1740(issuecomment-6757626883).bong-water-water-bong commentedon Nov 10, 2026
Measured: the activation pack is no longer 24 ms. It is 16-149 microseconds — so this issue's premise is stale, and the remaining per-launch cost is somewhere else.
Built a host benchmark against the engine's own header (
npu/forward/include/quant_i8.h, so it measures the shipped path, not a copy) and ran it on this box,origin/main:row_scalar?am=16, KD=4096(decode tile, 8B)am=128, KD=4096(prefill tile, 8B)am=128, KD=4864(prefill tile, 27B)am=16, KD=14336(decode tile, 27B)Against the 1.20-1.35 ms device wait per launch that the body quotes, the pack is now ~1-12 % of a launch, not 93 %. That is what the two already-merged fixes did:
quantize_async_rowspacks onlyamrows (with a loud refusal instead of the oldMD-row overrun), andq4nx_quant_i8::row()has the AVX2 path. The 24 ms/launch figure describes an older build.Consequence for whoever takes this next: optimising the host pack further (candidate 3, fusion + OMP) has at most ~150 µs per launch to win on the 8B shapes — it cannot move prefill throughput. The 1.4 s/token this lane measures on an 8B pack through the per-op route is therefore not host packing: the array answers (the first token matches the container's f32 reference on every prompt tried —
#1740,issuecomment-6757626883), so the cost is in the launch path around it (per-launch context switching, BO syncs, or the number of launches per token), and that is where the next measurement should go.Reproducer committed (private repo,
npu-dxb91593929):scripts/bench-quant-i8.cpp, with the build line in its header. It also asserts byte-identity of the AVX2 path againstrow_scalaron the same data, so a future "vectorise more" change has a one-command check that it did not change a single byte.Status of this issue: the two cheapest candidates are done in main; candidate 3's ceiling is ~150 µs/launch; candidate 4 (on-core expansion) is unaffected by any of this. I am not opening a PR for candidate 3 because the measurement above says it cannot buy what the issue was opened to buy — the honest next step is a launch-path profile, not another host optimisation.
bong-water-water-bong commentedon Nov 10, 2026
Re-measured at
main(ee5b436), and the PR that closes this is #1949 (cl-npuperf).The title no longer holds. Same pack as the issue (
bonsai2-27b-t2s2), onenpu-holdticket,amdxdnaerror count unchanged:maintodaypackper launchA 16-row block at
mainis 736 ms: device wait 215 (29 %), pack + quantise/launch + readback 110 (15 %), host work outside the launches 384 (52 %). The 24 ms was the first-touch weight fill, which #1621 moved in front of "ready".What #1949 does: the largest single host term left was the GDN alpha/beta projection, serial inside the row loop (76 ms per block). It now runs inside the parallel region the layer already opens. Measured A B B A in one ticket: alpha/beta 77.5 -> 13.1-13.5 ms, block wall 734-749 -> 688-704 ms (21.4-21.8 -> 22.7-23.3 rows/s, about 6 %), greedy text identical, and the bit-pattern check is 336 of 336 layer passes identical on the device.
What remains after it, none of it packing:
gdn_layerhost work (about 60 ms), Hadamard rotation (about 44 ms), attention (35-38 ms), MLP activation (28-30 ms).The body's candidate list: 1 and 2 were already in
main(verified above in this thread), 3 is bounded at 0.13 ms/launch so it cannot move throughput, and 4 is #1421.npu: a GDN layer's alpha/beta runs inside the region it already opens…