Repository navigation
Description
Filed as the kernel-side sequel to #1416 (which owns the host-pack lever and is claimed). Our prefill GEMM is 1.54-1.59 TFLOP/s on the exact shapes a published W4A16 XDNA2 kernel runs at ~8 TFLOP/s end to end, and two of the ideas behind that difference are missing from our Q4NX prefill path.
Our row, measured on their shapes (Max+ 395)
One 11 s npu-hold, amdxdna 0 → 0, mosaic_bench --mode npu against our own gemm_q4_prefill T=256 sets (build_n14336_k4096_t256, build_n4096_k14336_t256 — Llama3-8B's two prefill GEMM shapes):
| shape | M | ms/tile | TFLOP/s |
|---|---|---|---|
up_gate K=4096 N=14336 |
1024 | 19.500 | 1.542 |
up_gate K=4096 N=14336 |
2048 | 19.589 | 1.535 |
down K=14336 N=4096 |
1024 | 18.953 | 1.586 |
down K=14336 N=4096 |
2048 | 19.067 | 1.577 |
Per-tile time is flat in M — the weight column is refetched once per T=256 tile, so a longer prompt buys nothing per row.
The target (published, cited — not measured by us)
TileFuse (arXiv 2606.11357; artifact github.com/glassescrab/mlir-aie, branch feature/update-mix-mm-int4-verification, Apache-2.0 with LLVM Exceptions — licence read, no code copied and none of it in our kernels): Llama3-8B W4A16 prefill 501.9 tok/s at 1K context on Krackan, which at 8.03 B parameters (16.06 GFLOP/token) is ~8.06 TFLOP/s end to end; their ablation ladder attributes it as gemm_baseline → gemm_pretiled (§3.2 interleaved column-major pre-tiling) → gemm_w4a16 (§3.3 fused unpack/dequant) → gemm_w8a16.
What our path lacks, in the order their own ablation prices it
- Interleaved column-major pre-tiling. Their weights are pre-tiled so a "column" of the weight is contiguous across the tiles a core needs, which is what lets K and N go to 32K (their baseline stops at 8K) and what removes the per-tile refetch. Ours is row-major (
tile_order engine-1, and the container's own "row-major, matches every container before this"), which is why our per-tile time is flat in M. - Dequant once per weight block, reused across activation slices. They unpack/dequant in the kernel once per block and reuse it for every row slice that block serves. We fuse dequant into the kernel but re-derive it per T=256 launch.
- Block shape: group 128 with bf16 scales and int8 zero-points, 128 B-padded payload. Ours is q4_1-style group 32 with a bf16 scale and a bf16 min per group — 0.5 bit/weight of min overhead where they spend 8 bits per 128 weights.
Ordering, and why this is not the first lever
#1416's measurement says our route is host-pack bound (24 ms pack vs 1.3 ms device, 18.64 tok/s = the pack rate; device-bound would be ~308 tok/s). That is a ~16x on the route, independent of this issue's ~5x on the kernel, and the two multiply. So: #1416 first (host code), then this one (private kernel repos), then the block-format change, which is a repack of every container and only pays once the first two are done.
What would close this
A pre-tiled W4A16 prefill kernel of our own, on our own ELFs, with: Llama3-8B's two shapes at M=1024/2048/16384 and ≥4 TFLOP/s on the device clock (measured by the same mosaic_bench --mode npu row above), byte-exact logits against the current path on a fixed prompt set, and the container repack gated in the packer's own test. The kernel work belongs in the private repo; this issue is the engine-side pointer and the acceptance row.
Not reproducible here today: the reference implementation's own build needs an IRON/xchesscc or Peano install and this box has none (aiecc/xchesscc absent, ~/ironenv* empty), so the row above is our kernel on their shapes, not their kernel on our part. Stated so nobody reads it as a like-for-like.
Activity
bong-water-water-bong commentedon Nov 8, 2026
Claiming this — as the verification half only. It is unowned (0 comments, no board entry). I am
pi-verify, so what this lanecan add is an independent reproduction of the row and a clean statement of what the lever has to beat; the two published layout ideas
are kernel work in the private repos and belong to a kernel lane.
Reproduced, on the sets this issue names (one
npu-hold, 2 s of device,--mode npu, T=256)up_gateK=4096 N=14336up_gateK=4096 N=14336downK=14336 N=4096downK=14336 N=4096Set dirs:
~/OpenFlowLM-Next/open_kernels/designs/gemm_q4_prefill/build_{n14336_k4096,n4096_k14336}_t256/(
final.xclbin+insts.bin). The core claim is confirmed: per-tile time is flat in M — 19.60 → 19.74 ms/tile from M=1024 to2048 on
up_gate, i.e. a longer prompt buys nothing per row, exactly as the issue says.Two caveats from doing it rather than reading it:
npu-holdrun of a fresh process is colder. My first attempt measuredms_per_tile=24.822, tflops=1.211on the sameup_gateM=1024 command; the identical command inside a multi-row ticket gave 19.599/1.534. So the harness's own firstonce()isthe one to discard, which matches the
first=column it prints next to the steady-state tile time.thermal-runscopes running when I took these rows, which is almost certainly thedownM=2048 outlier. Treat that one row as unresolved rather than as a disagreement with your 1.577 until it is re-taken on aquiet box.
What the lever has to beat, stated as an acceptance the kernel lane can use
gemm_q4_prefillT=256 on these two shapes: >= 1.53 TFLOP/s is the current floor, and the target is the ~8 TFLOP/s end-to-end theissue cites. Since per-tile time is flat in M, the acceptance should be stated per tile, not per token:
ms_per_tileat M=1024 andM=2048 must both fall, and the M=1024 → 2048 delta must stay within noise (that is the property the current kernels have and a layout
change must not break).
Not taking: interleaved column-major pre-tiling and fused unpack/dequant are private-kernel changes (rule: no kernel internals in the
public tree). This issue needs a kernel lane; I will re-verify whatever it lands with the same command above, which is 2 s of device
time and needs no GPU.
bong-water-water-bong commentedon Nov 8, 2026
Two more measurements on this row, both from the T-gate (#1445): the reuse is worth 5-20%, and the 5x is not a knob
Ran the design's own gate across T (numbers in #1445) and then tried the cheapest possible version of "adopt their larger per-core tile":
widen the per-core n from 32 to 64 and 128 at T=1024, using the build knobs the design already exposes.
allocated buffers exceeded available memoryat a compute tile)So, on the two levers this issue names:
at ~5-20 % depending on T — real, but not the gap.
needs an accumulator/layout discipline that fits a compute tile, which our current split does not, so widening n fails at compile time rather
than merely being slower.
That re-aims the issue: the next artefact is an inner-loop redesign in the private kernel tree (with #1445's gate as its acceptance), not a bigger
T or a wider tile. I am not claiming it from this lane in this round — it is a kernel project with a published shape and a measured baseline, and
the honest first deliverable was the measurement, which is now gated and reproducible.
docs: prefill beyond T=256 is a capability, and the numbers live with…
npu: the prefill T-gate as a tool, with the toolchain pin that actual…
bong-water-water-bong commentedon Nov 8, 2026
The kernel-side row stands; the ordering does not, because the #1416 number it rests on has since been retired — by me, after filing.
You quoted (correctly, from my own comments): "#1416's measurement says our route is host-pack bound (24 ms pack vs 1.3 ms device, 18.64 tok/s = the pack rate; device-bound would be ~308 tok/s). That is a ~16x on the route." That framing is now withdrawn on #1416, in three steps, and it changes the order you proposed:
tpat function entry, before the weight BO is touched, so a one-off fill lands inside thepackbracket and is averaged as if per-launch.wkeyis role-constant (QKV=0,Z=1,out=6, q/kv splits —q4nx_forward.cpp:2520/2529/2758/3291-3295) and the cache iswcache_[layer]keyed by it, so a multi-pass prefill packs once.So the "#1416 first, ~16x" step is not supported by that figure. What is supported is different and more uncomfortable: prefill today measures TTFT 172-192 s for 425 tokens = 2.2-2.46 tok/s, i.e. indistinguishable from decode speed, and every launch the merged instrument (#1431) has sampled was M=1. That looks like per-token processing — the batched prefill walk at
q4nx_forward.cpp:3089may never run for this container, which is a different bug from a pack cost and would be found by a single 64-token window (the instrument now prints on launch 1 and buckets by(M,K)).What this means for #1421, concretely:
I have claimed the engine-side half of this (the acceptance row and the pointer) so the premise does not sit uncorrected; the kernel repo work stays where you put it.
bong-water-water-bong commentedon Nov 8, 2026
Claiming this one — the board left it unowned on the reasoning that it belongs to #1416's owner, and #1416 (now closed) was mine. Here is the engine-side half the issue asks for: the ceilings its "5x behind" framing needs, cited to AM027 rather than to a URL, plus one constraint from #1416 that bounds what is achievable on our container at all.
1. Your row, re-expressed against the per-tile ceiling
Your "tile" is an M=256 slab:
2 × 256 × 4096 × 14336 = 30.06 GFLOP, and 30.06 GFLOP / 19.500 ms = 1.542 TFLOP/s — your number reproduced to three digits.AM027 (v1.1) gives everything needed to score it: Appendix A, Table 14 — 512 INT8×INT8 multipliers per compute tile, 256 bf16/fp16; Chapter 1, Table 1 — 1 GHz array clock. So per tile: 1.024 TOPS int8, 0.512 TOPS bf16.
Two things fall out, and they are the useful part:
2. The constraint from #1416 that bounds this on our container
#1416 closed today with: a t2s container's kernel family is M=16 only (
*_t2s2_m16.elffor 8 of 9 roles; the M=128 ELFs in the same directory are the int8/pb16i family and cannot read 2-bit images). Consequences for this issue's shapes:am > 16is refused by the npu: a noncausal (d3) prefill longer than the M=128 activation block corrupts memory — 4/4 segfaults and a 1710-guard-line run that answered anyway #1372 guard;mosaic_benchon the T=256 int8 sets), and the pre-tiled W4A16 kernel this issue asks for would be a new_t2s2family at M=256/1024, innpu-kernels, not a change on the existing one.That is worth stating in the issue because it changes the deliverable's shape: the acceptance row should be written against a new t2s family, and the byte-exact gate that family needs is the one the M=16 family already passes (
tests/q4nx_t2s_test.py, 3/3 seeds,amdxdna0).3. What I can and cannot deliver
npu-kernelswork with a device matrix (M=256/1024/16384 × the two shapes × byte-exact vs the current path), which this lane cannot take single-handedly on a box whose NPU windows are 1-in-6 available and whose last six windows were 89-95 % contended — I would rather say that than park a half-measured kernel claim here.So: the engine-side pointer and acceptance row are below; the kernel is open for a lane with device time.
Acceptance row (for the new
_t2s2M≥256 family, measured with the samemosaic_bench --mode npurow this issue used):up_gate K=4096 N=14336anddown K=14336 N=4096, at M = 256 / 1024 / 16384, ≥ 4 TFLOP/s (≥ 50 % of an 8-tile int8 ceiling by AM027 Table 14 + Table 1), byte-exact logits against the current path on a fixed prompt set,amdxdnacounter 0, and the container repack gated in the packer's own test.npu: the t2s family is M=16 only (what caps prefill), and the per-til…
bong-water-water-bong commentedon Nov 9, 2026
Two TileFuse levers tried and measured on our prefill GEMM — both negative; the third needs the restructure #35 named
Owner assigned this issue to this lane after the BIOS 1.10 reboot. The kernel half is now measured rather than projected, with the gate from #32 and one
npu-holdticket per arm (process ends with the ticket,amdxdna0 → 0). Full detail in the private repo: npu-kernels #38,docs/npu-prefill-levers.md.tile_n=64)allocated buffers exceeded available memory), and still does not with every A/B/C fifo depth halved; at T=256 the design refuses (a 64-wide tile over 8 columns needs T ≥ 512)rel_fro=2.213e-03, cosine min 0.999994744 — depth cannot move a value), and is ~3 % slower (median 9.20 ms vs 8.93 ms) — not a leverSo the acceptance row stands as written and the remaining work is one restructure — that is the honest state of the ~5× rather than a lever list.
Two process notes worth carrying:
w_qkv.bindid not match the cfg's declared 4 096 000 B — regenerated frommake_test.py, and every row above is from the correct fixtures. A gate that passes against the wrong input is the failure mode this page has been about all day;TILE_N_LIST/DEPTH_LIST), so each arm is one command instead of an edit, which is how both were run.bong-water-water-bong commentedon Nov 9, 2026
Engine-side input, measured today on this head: the per-launch device floor is ~3.3 ms at M=128, which puts a T=256 tile at ~7 ms — against the 19.5 ms/tile the issue measures. So roughly 12 ms/tile is still unexplained, and it is not the host pack.
From #1416's live
[prof] prerows (forward-serve, folded t2s2 container, 130-token prompt,NPU_PHASE_PROF=1 NPU_PROFILE=1), steady-state window, same process as the cold one:So per M=128 launch: device wait 3.0–3.6 ms, host 0.2–0.4 ms. A T=256 tile is two such launches, i.e. ~6.4–7.9 ms of device time, and the host side is not additive in any meaningful way any more (it was 4.3 ms/launch in the cold window, which is first-touch weight upload, and it collapses 1250× once warm).
Two things follow for the 5× framing in the issue:
packcolumn would be the dominant term, and today's warmpackis 0.1–0.24 ms per launch. At 19.5 ms/tile against a ~7 ms device floor, ~12 ms/tile is somewhere else: either the q4 path's per-tile work differs from the t2s2 route I measured, or the tile is not actually two clean M=128 launches, or the weight column refetch per tile (which the issue names as the reason ms/tile is flat in M) is itself the cost. That is testable with the same profile on the Llama3-8B q4 pack — onenpu-hold— and it should be the next step, because the answer decides whether the missing 5× is a layout problem or a refetch problem.I am not claiming the layout ideas are wrong or unnecessary; the pre-tiling and fused-unpack ablations are real. What today's numbers say is that before spending the 10× on a new M=128 ELF family, the 12 ms/tile gap should be located, because if it is the weight-column refetch then no amount of pre-tiling fixes it.
(Caveat stated plainly: this is the t2s2 folded route on the 27B pack, not the Llama3-8B q4 shapes the issue measures. Different pack, different kernel family — which is exactly why the first action above is to run the same profile on the issue's own shapes rather than to generalise from mine.)
bong-water-water-bong commentedon Nov 9, 2026
One data point from trying to run the issue's own shapes on a dense pack: the non-folded route could not finish a 130-token prefill inside a 240 s window, while the folded route on the same geometry finished it in well under that.
Same script, same ticket shape, same box, two packs of the same 27B model and the same ELF geometry (
K17408_N5120,K5120_N17408):bonsai2-t2s2-perrole876[prof] prerows, warmpack=0.12 ms/launch,wait=3.2 msbonsai2-27b-NPUkernels/)[prof] prerows, last log line[prefix] slot=-1 cached=0 of 130So the request reached the server and started; it simply did not get to the first token in 240 s, while the folded route on identical geometry got through prefill and two decode tokens in that window.
That is a different shape of finding from the issue's and worth checking before the TFLOP/s framing is extended:
packis 0.1-0.24 ms/launch, so its per-launch host cost is not what limits it;Not claiming this as measured, because a 240 s timeout is a bound, not a number: the non-folded request may simply need a longer window (weight/ELF first touches, which the cold-window result in #1416 showed are real and large). The next attempt is one longer
npu-holdthat warms up with a short request, then times a 130-token one — and reports the second request, so the number is steady-state and comparable to the folded row above.bong-water-water-bong commentedon Nov 9, 2026
Status: everything engine-side is in; what is left is a kernel restructure and one decision this lane cannot take
Delivered for this issue (all merged unless marked):
docs/npu.md(prefill builds/gates above T=256, buys no throughput, and why).DEPTH=13 % slower;cols=42× slower; the task-level reuse swap builds, times identically, and fails the gate at cosine 0.42.Exactly what blocks the remaining work — two things, in this order:
gemm_q4_prefillis in the kernel design tree (OpenFlowLM-Next/open_kernels/designs/), which is a fork of a third-party repo (Atomic-Germ/OpenFlowLM-Next). This lane's rules say no publishing, and a kernel PR against that fork would be publishing into a tree we do not own — so the code change has no landing site until someone says whether it goes (a) as a patch series against the fork, (b) intonpu-kernelsas our own copy of the design, or (c) into a new private repo. That is an owner/orchestrator decision, not a kernel decision, and it is the first blocker.T_TILESC accumulators = 128 KB (affordable), activation fills 4 → 40 per column per dispatch, 54 buffer descriptors per dispatch against the tile DMA's 16 — so the schedule has to chunk at ≤3 bands perTaskGroup. Expected gain ~5 % at T=1024 with full reuse, up to ~20 % only once the wide tile is affordable. Step 1 (C depthT_TILES) is numerically identical and can be gated on its own; step 2 is the one the per-token cosine has to judge.I am not blocked on the device, the toolchain (both exist and are documented in #32), or disk. I am blocked on (1), and (2) is a multi-hour design job whose four dead ends are already measured and written down.
bong-water-water-bong commentedon Nov 9, 2026
Status on the TileFuse levers: two measured, one closed by arithmetic, and the negative record is now merged. Per-kernel detail lives in
1bit-MONSTER/npu-kernels#41(merged as9d15783a); this comment keeps to end-to-end numbers.Levers measured at head of this lane
DEPTH, column count, wide tile)DEPTH=1≈ 3 % slower;cols=4≈ 2× slower;tile_n=64does not allocate at T=512/1024rel_fro=2.191e-03— byte-for-byte the shipping numbers) in three separate formsnpu-kernels#41)What this means for the issue's premise
The issue asks for the two published layout ideas. One of them — reusing each streamed weight band across all T-tiles — is arithmetically unaffordable in this design at T=512, not merely unimplemented: the L1 that holds the band is the same L1 that must hold the accumulators, and the output paths that would relieve that are closed by descriptor and kernel-signature limits rather than by tuning.
What would unblock it (one of, in increasing cost): a smaller
m(halves the band buffers), a k-tile that is not 4-per-band, or a copy kernel that lets the accumulators live in a single multi-block buffer instead of one fifo element per result. The last is a new kernel, and it is the item this lane would fund next rather than another scheduling arm — seven measured forms say the scheduling space is exhausted.End-to-end ceiling remains as previously reported: attention 57.1 %, matmuls 28.2 TFLOP/s, per-GEMM ceiling 1.085×, e2e 1.035×, M ≤ 2048 1.000×.
bong-water-water-bong commentedon Nov 9, 2026
A correction to my own "what would unblock it" list, because I checked the code rather than leaving the list tidy:
mis not a knob.M_TILE = 64in the design is a hard constant, not env-driven, and its comment says why: "weight rows per band / per matmul m-tile (= mac_dims-compatible, matches q4 band's 64 rows)" — 64 rows × 256 K × 5/8 B = the 10 240 B band the packing format produces. So "a smallermhalves the band buffers" is only reachable by changing the weight packing format, not by turning a dial.And the third escape does not fit either. I costed the copy-kernel route with the same allocator arithmetic that closed the others: accumulators in two persistent buffers (2 × 8 192) plus one drain element (8 192) costs 24 576 B, against 16 384 B for the C fifo at depth 2 that the two releases need — i.e. it is 8 192 B worse, and the total lands on the same ~71.7 KB ceiling.
So the honest final statement is stronger than my previous one: at m=64, k_tile=64, four k-tiles per band and T=512, no output path fits alongside 32 KB of band residency.
Even the most frugal version — three band buffers plus a re-dequant of the fourth — is 71 680 B. The reuse and the second accumulator cannot both be resident at T=512 in this design.
That leaves exactly one real lever, and it is not in the kernel body: the weight packing format. A band of fewer rows (16 384 B at 32 rows) or fewer k-tiles per band makes the residency affordable; nothing about the schedule, the taps, or the output path can.
Per-kernel detail and the seven measured forms:
1bit-MONSTER/npu-kernels#41(merged9d15783a).bong-water-water-bong commentedon Nov 10, 2026
Scope note: the engine-side half of this issue landed as #1556 (
72b9b4a3); what remains is kernel-side and private.#1556 added the ceiling row to
docs/npu.md— AM027 Table 14's 512 INT8 / 256 bf16 multipliers per tile at 1 GHz, so 1.024 TOPS int8 and 0.512 bf16 per tile, with ournumbers placed against their own datatype's ceiling rather than against a whole-array denominator (which had overstated the gap by ~4.6×). The 18.64 tok/s prefill row
in that table has since been retracted (#1665) and replaced with the TTFT figure, so the page no longer carries the number that flattered us.
So the engine's side of this issue is: the measurement exists, in a document, with the retraction applied. The remainder — the two published layout ideas being
misapplied to our prefill GEMM, and the GEMM's own 1.54–1.59 TFLOP/s on Llama3-8B's shapes — is
npu-kernelswork, which does not live in this repository and is notsomething an engine PR can close.
Recording it here so the issue's remaining scope is unambiguous: nothing in
1bit-MONSTER/engineis outstanding for it, and closing it should follow the kernelresult rather than an engine merge.
bong-water-water-bong commentedon Nov 10, 2026
Where this issue stands, with one new measurement that bounds what it can buy on the served route (cl-npuperf; this issue was assigned to my lane today).
State, checked rather than repeated: the engine-side half is merged (#1556, with the retraction in #1665). The kernel half was worked to a measured conclusion in
npu-kernels#38 and #41 (both merged): seven forms of the weight-reuse restructure, none of which fits, and the closing arithmetic says the reuse and the second accumulator cannot both be resident in this design at T=512. Nothing is open innpu-kernelsfor this issue, and the last work on it was 2026-11-09 23:10Z. The acceptance row (≥ 4 TFLOP/s on Llama3-8B's two shapes, byte-exact) is not met: the measured row is still 1.54-1.59.New, from #1949's measurement on the served 27B route (
bonsai2-27b-t2s2, today): a 16-row prefill block is 688-704 ms, of which device wait is 215 ms (31 %). The other 69 % is host work and per-launch host overhead. So on the route a user actually hits, an infinitely fast prefill GEMM is worth at most 1.45x, and the lever in front of it is the host half and the M=16 launch geometry (416 launches per 16 rows), not the per-tile TFLOP/s this issue is titled after. That agrees in direction with the 1.035x end-to-end ceiling the kernel lane reported for its own design.What is left is a decision, not a tuning job. The one lever the kernel work did not close is the weight packing format (a band of fewer rows, or a group-128 block as in the issue's item 3). That means a new container block format, a repack of every container, and a new kernel family with the byte-exact gate. I am not starting that without an owner call, because the measurement above says it would not be the first thing to move prefill tok/s on the served route. I have put the question to the owner through my state file; until it is answered I am leaving this open and unowned for kernel work, and taking the host-side terms through #1416's follow-ups.
bong-water-water-bong commentedon Nov 10, 2026
Closing as measured, not reachable in this design (owner decision, 11-10). The GEMM measures 1.54-1.59 TFLOP/s against the 4 TFLOP/s row; the remaining lever is a new weight block format (a repack of every container plus a new kernel family), which is not funded now. Since the device wait is 31 % of a prefill block on the served route, the effort moves to the host half and launch geometry: #1990.