Skip to content

npu: our prefill GEMM is 1.54-1.59 TFLOP/s on Llama3-8B's own shapes, and two published layout ideas are missing (sequel to #1416) #1421

Closed

Description

@bong-water-water-bong

Filed as the kernel-side sequel to #1416 (which owns the host-pack lever and is claimed). Our prefill GEMM is 1.54-1.59 TFLOP/s on the exact shapes a published W4A16 XDNA2 kernel runs at ~8 TFLOP/s end to end, and two of the ideas behind that difference are missing from our Q4NX prefill path.

Our row, measured on their shapes (Max+ 395)

One 11 s npu-hold, amdxdna 0 → 0, mosaic_bench --mode npu against our own gemm_q4_prefill T=256 sets (build_n14336_k4096_t256, build_n4096_k14336_t256 — Llama3-8B's two prefill GEMM shapes):

shape M ms/tile TFLOP/s
up_gate K=4096 N=14336 1024 19.500 1.542
up_gate K=4096 N=14336 2048 19.589 1.535
down K=14336 N=4096 1024 18.953 1.586
down K=14336 N=4096 2048 19.067 1.577

Per-tile time is flat in M — the weight column is refetched once per T=256 tile, so a longer prompt buys nothing per row.

The target (published, cited — not measured by us)

TileFuse (arXiv 2606.11357; artifact github.com/glassescrab/mlir-aie, branch feature/update-mix-mm-int4-verification, Apache-2.0 with LLVM Exceptions — licence read, no code copied and none of it in our kernels): Llama3-8B W4A16 prefill 501.9 tok/s at 1K context on Krackan, which at 8.03 B parameters (16.06 GFLOP/token) is ~8.06 TFLOP/s end to end; their ablation ladder attributes it as gemm_baseline → gemm_pretiled (§3.2 interleaved column-major pre-tiling) → gemm_w4a16 (§3.3 fused unpack/dequant) → gemm_w8a16.

What our path lacks, in the order their own ablation prices it

  1. Interleaved column-major pre-tiling. Their weights are pre-tiled so a "column" of the weight is contiguous across the tiles a core needs, which is what lets K and N go to 32K (their baseline stops at 8K) and what removes the per-tile refetch. Ours is row-major (tile_order engine-1, and the container's own "row-major, matches every container before this"), which is why our per-tile time is flat in M.
  2. Dequant once per weight block, reused across activation slices. They unpack/dequant in the kernel once per block and reuse it for every row slice that block serves. We fuse dequant into the kernel but re-derive it per T=256 launch.
  3. Block shape: group 128 with bf16 scales and int8 zero-points, 128 B-padded payload. Ours is q4_1-style group 32 with a bf16 scale and a bf16 min per group — 0.5 bit/weight of min overhead where they spend 8 bits per 128 weights.

Ordering, and why this is not the first lever

#1416's measurement says our route is host-pack bound (24 ms pack vs 1.3 ms device, 18.64 tok/s = the pack rate; device-bound would be ~308 tok/s). That is a ~16x on the route, independent of this issue's ~5x on the kernel, and the two multiply. So: #1416 first (host code), then this one (private kernel repos), then the block-format change, which is a repack of every container and only pays once the first two are done.

What would close this

A pre-tiled W4A16 prefill kernel of our own, on our own ELFs, with: Llama3-8B's two shapes at M=1024/2048/16384 and ≥4 TFLOP/s on the device clock (measured by the same mosaic_bench --mode npu row above), byte-exact logits against the current path on a fixed prompt set, and the container repack gated in the packer's own test. The kernel work belongs in the private repo; this issue is the engine-side pointer and the acceptance row.

Not reproducible here today: the reference implementation's own build needs an IRON/xchesscc or Peano install and this box has none (aiecc/xchesscc absent, ~/ironenv* empty), so the row above is our kernel on their shapes, not their kernel on our part. Stated so nobody reads it as a like-for-like.

Activity

  1. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claiming this — as the verification half only. It is unowned (0 comments, no board entry). I am pi-verify, so what this lane
    can add is an independent reproduction of the row and a clean statement of what the lever has to beat; the two published layout ideas
    are kernel work in the private repos and belong to a kernel lane.

    Reproduced, on the sets this issue names (one npu-hold, 2 s of device, --mode npu, T=256)

    shape M ms/tile TFLOP/s your row agreement
    up_gate K=4096 N=14336 1024 19.599 1.534 19.500 / 1.542 0.5 %
    up_gate K=4096 N=14336 2048 19.738 1.523 19.589 / 1.535 0.8 %
    down K=14336 N=4096 1024 18.960 1.586 18.953 / 1.586 exact
    down K=14336 N=4096 2048 23.649 1.271 19.067 / 1.577 24 % high — does not reproduce

    Set dirs: ~/OpenFlowLM-Next/open_kernels/designs/gemm_q4_prefill/build_{n14336_k4096,n4096_k14336}_t256/
    (final.xclbin + insts.bin). The core claim is confirmed: per-tile time is flat in M — 19.60 → 19.74 ms/tile from M=1024 to
    2048 on up_gate, i.e. a longer prompt buys nothing per row, exactly as the issue says.

    Two caveats from doing it rather than reading it:

    1. The first npu-hold run of a fresh process is colder. My first attempt measured ms_per_tile=24.822, tflops=1.211 on the same
      up_gate M=1024 command; the identical command inside a multi-row ticket gave 19.599/1.534. So the harness's own first once() is
      the one to discard, which matches the first= column it prints next to the steady-state tile time.
    2. The box was at load 50.8 with ten other lanes' thermal-run scopes running when I took these rows, which is almost certainly the
      down M=2048 outlier. Treat that one row as unresolved rather than as a disagreement with your 1.577 until it is re-taken on a
      quiet box.

    What the lever has to beat, stated as an acceptance the kernel lane can use

    gemm_q4_prefill T=256 on these two shapes: >= 1.53 TFLOP/s is the current floor, and the target is the ~8 TFLOP/s end-to-end the
    issue cites.
    Since per-tile time is flat in M, the acceptance should be stated per tile, not per token: ms_per_tile at M=1024 and
    M=2048 must both fall, and the M=1024 → 2048 delta must stay within noise (that is the property the current kernels have and a layout
    change must not break).

    Not taking: interleaved column-major pre-tiling and fused unpack/dequant are private-kernel changes (rule: no kernel internals in the
    public tree). This issue needs a kernel lane; I will re-verify whatever it lands with the same command above, which is 2 s of device
    time and needs no GPU.

  2. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Two more measurements on this row, both from the T-gate (#1445): the reuse is worth 5-20%, and the 5x is not a knob

    Ran the design's own gate across T (numbers in #1445) and then tried the cheapest possible version of "adopt their larger per-core tile":
    widen the per-core n from 32 to 64 and 128 at T=1024, using the build knobs the design already exposes.

    experiment result
    T=256 / 512 / 1024, current tile 2.415 / 4.766 / 9.832 ms per run, all numerically PASS — linear in T, i.e. the weight column is re-streamed per T-tile
    wider per-core n (64, 128) at T=1024 does not build: the per-tile accumulator + activation buffers no longer fit a compute tile's local memory (allocated buffers exceeded available memory at a compute tile)

    So, on the two levers this issue names:

    1. Reuse across T-tiles (TileFuse's §3.2 pre-tiling + §3.3 dequant-once) is worth the weight-DMA share of a run, which the T-linearity bounds
      at ~5-20 % depending on T — real, but not the gap.
    2. The ~5x is the inner loop's geometry, and it is not a configuration change: their per-core tile (m64/k128/n64 over all 4 array rows)
      needs an accumulator/layout discipline that fits a compute tile, which our current split does not, so widening n fails at compile time rather
      than merely being slower.

    That re-aims the issue: the next artefact is an inner-loop redesign in the private kernel tree (with #1445's gate as its acceptance), not a bigger
    T or a wider tile. I am not claiming it from this lane in this round — it is a kernel project with a published shape and a measured baseline, and
    the honest first deliverable was the measurement, which is now gated and reproducible.

  3. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The kernel-side row stands; the ordering does not, because the #1416 number it rests on has since been retired — by me, after filing.

    You quoted (correctly, from my own comments): "#1416's measurement says our route is host-pack bound (24 ms pack vs 1.3 ms device, 18.64 tok/s = the pack rate; device-bound would be ~308 tok/s). That is a ~16x on the route." That framing is now withdrawn on #1416, in three steps, and it changes the order you proposed:

    1. The 24.2 ms is not a per-token cost. It is consistent with a one-off cold weight fill amortised: whole-model pb16i fill is ~10.6 s cold, and 10.6 s / 320 launches ≈ 33 ms/launch — the same order. That table was the first pass of a fresh serve, and the profiler stamps tp at function entry, before the weight BO is touched, so a one-off fill lands inside the pack bracket and is averaged as if per-launch.
    2. The "weights repacked every pass" candidate is impossible in the code: wkey is role-constant (QKV=0, Z=1, out=6, q/kv splits — q4nx_forward.cpp:2520/2529/2758/3291-3295) and the cache is wcache_[layer] keyed by it, so a multi-pass prefill packs once.
    3. The three terms that are actually inside that window sum to ~2.2 ms, benched device-free with the engine's own routines: rotation 0.695 + quantise 0.203 + launch 1.20-1.35 ms for a 128×4864 block.

    So the "#1416 first, ~16x" step is not supported by that figure. What is supported is different and more uncomfortable: prefill today measures TTFT 172-192 s for 425 tokens = 2.2-2.46 tok/s, i.e. indistinguishable from decode speed, and every launch the merged instrument (#1431) has sampled was M=1. That looks like per-token processing — the batched prefill walk at q4nx_forward.cpp:3089 may never run for this container, which is a different bug from a pack cost and would be found by a single 64-token window (the instrument now prints on launch 1 and buckets by (M,K)).

    What this means for #1421, concretely:

    • Your row is untouched and still valid: 1.54-1.59 TFLOP/s on Llama3-8B's two shapes, and the per-tile time being flat in M is a real structural observation that does not depend on the pack story at all — it says the weight column is refetched per T=256 tile, which is exactly the §3.2 pre-tiling gap you name.
    • The ordering should be: (1) does batched prefill run at all — one short window, merged instrument, no kernel work; (2) then the kernel row above; (3) then the block-format repack. The middle one is unaffected; the first is no longer a 16x host-pack lever but a possibly-missing code path, which is cheaper to check and would change the denominator for everything downstream.
    • The ~8 TFLOP/s target is not comparable yet and you said so yourself (no aiecc/xchesscc on this box, so it is our kernel on their shapes) — I would keep that caveat prominent, because "5x behind a published kernel" and "our prefill may not be batching at all" lead to completely different work.

    I have claimed the engine-side half of this (the acceptance row and the pointer) so the premise does not sit uncorrected; the kernel repo work stays where you put it.

  4. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claiming this one — the board left it unowned on the reasoning that it belongs to #1416's owner, and #1416 (now closed) was mine. Here is the engine-side half the issue asks for: the ceilings its "5x behind" framing needs, cited to AM027 rather than to a URL, plus one constraint from #1416 that bounds what is achievable on our container at all.

    1. Your row, re-expressed against the per-tile ceiling

    Your "tile" is an M=256 slab: 2 × 256 × 4096 × 14336 = 30.06 GFLOP, and 30.06 GFLOP / 19.500 ms = 1.542 TFLOP/s — your number reproduced to three digits.

    AM027 (v1.1) gives everything needed to score it: Appendix A, Table 14 — 512 INT8×INT8 multipliers per compute tile, 256 bf16/fp16; Chapter 1, Table 1 — 1 GHz array clock. So per tile: 1.024 TOPS int8, 0.512 TOPS bf16.

    quantity value
    your 1.542 TFLOP/s, in MAC-limited tile-equivalents 1.51 int8-tiles or 3.01 bf16-tiles
    as a fraction of an 8-tile ceiling 18.8 % (int8) / 37.6 % (bf16)
    the published 8.06 TFLOP/s, same units 7.9 int8-tiles at ~100 % MAC

    Two things fall out, and they are the useful part:

    • The published target is an int8-MAC-bound result, not a bf16 one: 8.06 TFLOP/s is ~7.9 int8 tiles at full rate, and would be impossible on bf16 tiles at 1 GHz (8 bf16 tiles are 4.1 TOPS). So TileFuse is expanding 4-bit weights into the int8 MMUL, which is the same choice our t2s stream makes and the same choice AM027 forces — Table 14 lists no INT4 multiplier at all, so every 4-bit design on this architecture is an int8 MAC with an on-core expansion.
    • Your 1.542 TFLOP/s is ~19 % of an 8-tile int8 ceiling, and the missing 81 % is not MAC scheduling: the per-tile time being flat in M says the weight column is refetched per slab, which is dequant/refetch-bound, exactly what your §3.2/§3.3 ablation names. The ceiling arithmetic supports your ordering rather than competing with it.

    2. The constraint from #1416 that bounds this on our container

    #1416 closed today with: a t2s container's kernel family is M=16 only (*_t2s2_m16.elf for 8 of 9 roles; the M=128 ELFs in the same directory are the int8/pb16i family and cannot read 2-bit images). Consequences for this issue's shapes:

    That is worth stating in the issue because it changes the deliverable's shape: the acceptance row should be written against a new t2s family, and the byte-exact gate that family needs is the one the M=16 family already passes (tests/q4nx_t2s_test.py, 3/3 seeds, amdxdna 0).

    3. What I can and cannot deliver

    • Can, from here, device-free: the ceiling arithmetic above, the acceptance row, and the family/geometry constraints (done).
    • Cannot: the kernel loop itself. It is npu-kernels work with a device matrix (M=256/1024/16384 × the two shapes × byte-exact vs the current path), which this lane cannot take single-handedly on a box whose NPU windows are 1-in-6 available and whose last six windows were 89-95 % contended — I would rather say that than park a half-measured kernel claim here.

    So: the engine-side pointer and acceptance row are below; the kernel is open for a lane with device time.

    Acceptance row (for the new _t2s2 M≥256 family, measured with the same mosaic_bench --mode npu row this issue used):
    up_gate K=4096 N=14336 and down K=14336 N=4096, at M = 256 / 1024 / 16384, ≥ 4 TFLOP/s (≥ 50 % of an 8-tile int8 ceiling by AM027 Table 14 + Table 1), byte-exact logits against the current path on a fixed prompt set, amdxdna counter 0, and the container repack gated in the packer's own test.

  5. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Two TileFuse levers tried and measured on our prefill GEMM — both negative; the third needs the restructure #35 named

    Owner assigned this issue to this lane after the BIOS 1.10 reboot. The kernel half is now measured rather than projected, with the gate from #32 and one
    npu-hold ticket per arm (process ends with the ticket, amdxdna 0 → 0). Full detail in the private repo: npu-kernels #38, docs/npu-prefill-levers.md.

    lever result
    wider per-core tile (tile_n=64) does not build at T=512 or T=1024 (allocated buffers exceeded available memory), and still does not with every A/B/C fifo depth halved; at T=256 the design refuses (a 64-wide tile over 8 columns needs T ≥ 512)
    halve the fifo depths at the shipping tile fits, gates identically (rel_fro=2.213e-03, cosine min 0.999994744 — depth cannot move a value), and is ~3 % slower (median 9.20 ms vs 8.93 ms) — not a lever
    dequant/pack once, reuse across T-tiles not done: needs the activation residency sliced by k-tile so a wide tile and a reused weight band coexist (the arithmetic is in #35: 1024 KB → 16 KB at t=1024/k=4096/n=64). A loop-and-tap restructure, not a flag.

    So the acceptance row stands as written and the remaining work is one restructure — that is the honest state of the ~5× rather than a lever list.

    Two process notes worth carrying:

    • an arm first reported "PASS" against 96 KB fixtures because a stale w_qkv.bin did not match the cfg's declared 4 096 000 B — regenerated from make_test.py, and every row above is from the correct fixtures. A gate that passes against the wrong input is the failure mode this page has been about all day;
    • the gate tool now takes the design's knobs through (TILE_N_LIST/DEPTH_LIST), so each arm is one command instead of an edit, which is how both were run.
  6. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Engine-side input, measured today on this head: the per-launch device floor is ~3.3 ms at M=128, which puts a T=256 tile at ~7 ms — against the 19.5 ms/tile the issue measures. So roughly 12 ms/tile is still unexplained, and it is not the host pack.

    From #1416's live [prof] pre rows (forward-serve, folded t2s2 container, 130-token prompt, NPU_PHASE_PROF=1 NPU_PROFILE=1), steady-state window, same process as the cold one:

    pre G  128  15 launches   pack=0.12   quant+lnch=0.11   wait=3.220   rb=0.026   ms/launch=3.484
    pre D  128  16 launches   pack=0.24   quant+lnch=0.15   wait=3.543   rb=0.010   ms/launch=3.949
    

    So per M=128 launch: device wait 3.0–3.6 ms, host 0.2–0.4 ms. A T=256 tile is two such launches, i.e. ~6.4–7.9 ms of device time, and the host side is not additive in any meaningful way any more (it was 4.3 ms/launch in the cold window, which is first-touch weight upload, and it collapses 1250× once warm).

    Two things follow for the 5× framing in the issue:

    1. The 19.5 ms/tile on Llama3-8B's shapes is not host-pack bound either — if it were, the pack column would be the dominant term, and today's warm pack is 0.1–0.24 ms per launch. At 19.5 ms/tile against a ~7 ms device floor, ~12 ms/tile is somewhere else: either the q4 path's per-tile work differs from the t2s2 route I measured, or the tile is not actually two clean M=128 launches, or the weight column refetch per tile (which the issue names as the reason ms/tile is flat in M) is itself the cost. That is testable with the same profile on the Llama3-8B q4 pack — one npu-hold — and it should be the next step, because the answer decides whether the missing 5× is a layout problem or a refetch problem.
    2. The ceiling arithmetic needs the band's own number, not ours. 320 launches × 3.3 ms ≈ 1.06 s per 128 rows ≈ 120 tok/s on the 27B route; the issue's 1.54–1.59 TFLOP/s is the same instrument on a different pack. Quoting ~8 TFLOP/s from TileFuse is fine as a target, but the honest comparison row is ours-measured-now vs ours-measured-then, with the published number labelled as published — which the issue already does carefully.

    I am not claiming the layout ideas are wrong or unnecessary; the pre-tiling and fused-unpack ablations are real. What today's numbers say is that before spending the 10× on a new M=128 ELF family, the 12 ms/tile gap should be located, because if it is the weight-column refetch then no amount of pre-tiling fixes it.

    (Caveat stated plainly: this is the t2s2 folded route on the 27B pack, not the Llama3-8B q4 shapes the issue measures. Different pack, different kernel family — which is exactly why the first action above is to run the same profile on the issue's own shapes rather than to generalise from mine.)

  7. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    One data point from trying to run the issue's own shapes on a dense pack: the non-folded route could not finish a 130-token prefill inside a 240 s window, while the folded route on the same geometry finished it in well under that.

    Same script, same ticket shape, same box, two packs of the same 27B model and the same ELF geometry (K17408_N5120, K5120_N17408):

    pack route server ready 130-token request
    bonsai2-t2s2-perrole876 folded (t2s2 sidecars) 61 s completed; [prof] pre rows, warm pack=0.12 ms/launch, wait=3.2 ms
    bonsai2-27b-NPU non-folded (kernels/) 25 s http 000 after 240 s, 0 [prof] pre rows, last log line [prefix] slot=-1 cached=0 of 130

    So the request reached the server and started; it simply did not get to the first token in 240 s, while the folded route on identical geometry got through prefill and two decode tokens in that window.

    That is a different shape of finding from the issue's and worth checking before the TFLOP/s framing is extended:

    • it suggests the host-heavy path is the non-folded one, not the folded one — the folded route's warm pack is 0.1-0.24 ms/launch, so its per-launch host cost is not what limits it;
    • if that holds, npu: prefill is host-pack bound (24 ms/launch pack vs 1.3 ms device, 93 % of the pass) — 18.64 tok/s is the pack rate, not the array's #1416's "24 ms/launch, array idle 93 %" was likely measured on a route (or a cold window) where the assembly path runs per launch, which would make the fix a routing question — send these shapes down the folded path, or make the non-folded path stop re-assembling per launch — rather than a new kernel family;
    • and it means any TFLOP/s row for the non-folded route should be reported with the wall time of a whole request, because "19.5 ms/tile" and "cannot finish 130 tokens in 240 s" have to be reconciled before either is used as a baseline.

    Not claiming this as measured, because a 240 s timeout is a bound, not a number: the non-folded request may simply need a longer window (weight/ELF first touches, which the cold-window result in #1416 showed are real and large). The next attempt is one longer npu-hold that warms up with a short request, then times a 130-token one — and reports the second request, so the number is steady-state and comparable to the folded row above.

  8. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Status: everything engine-side is in; what is left is a kernel restructure and one decision this lane cannot take

    Delivered for this issue (all merged unless marked):

    Exactly what blocks the remaining work — two things, in this order:

    1. Where our version of the design lives. gemm_q4_prefill is in the kernel design tree (OpenFlowLM-Next/open_kernels/designs/), which is a fork of a third-party repo (Atomic-Germ/OpenFlowLM-Next). This lane's rules say no publishing, and a kernel PR against that fork would be publishing into a tree we do not own — so the code change has no landing site until someone says whether it goes (a) as a patch series against the fork, (b) into npu-kernels as our own copy of the design, or (c) into a new private repo. That is an owner/orchestrator decision, not a kernel decision, and it is the first blocker.
    2. Then the restructure itself (1bit serve: --mtp (up to 3.4x), --parallel (up to 5.8x), --adaptive (Vulkan + ROCm overflow), --device rocm #41's ladder), whose numbers are known: T_TILES C accumulators = 128 KB (affordable), activation fills 4 → 40 per column per dispatch, 54 buffer descriptors per dispatch against the tile DMA's 16 — so the schedule has to chunk at ≤3 bands per TaskGroup. Expected gain ~5 % at T=1024 with full reuse, up to ~20 % only once the wide tile is affordable. Step 1 (C depth T_TILES) is numerically identical and can be gated on its own; step 2 is the one the per-token cosine has to judge.

    I am not blocked on the device, the toolchain (both exist and are documented in #32), or disk. I am blocked on (1), and (2) is a multi-hour design job whose four dead ends are already measured and written down.

  9. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Status on the TileFuse levers: two measured, one closed by arithmetic, and the negative record is now merged. Per-kernel detail lives in 1bit-MONSTER/npu-kernels#41 (merged as 9d15783a); this comment keeps to end-to-end numbers.

    Levers measured at head of this lane

    lever result verdict
    scheduling knobs (DEPTH, column count, wide tile) DEPTH=1 ≈ 3 % slower; cols=4 ≈ 2× slower; tile_n=64 does not allocate at T=512/1024 negative (npu-kernels #38)
    weight-band reuse across T-tiles (the restructure the fourth arm pointed at) inversion is numerically exact (per-token cosine min 0.999994912, rel_fro=2.191e-03 — byte-for-byte the shipping numbers) in three separate forms correct, and still blocked
    why it stays blocked the reuse needs (i) a band worth keeping — 32 KB, half of L1, and (ii) two live accumulators; the budget is 63 496 B of 65 536 with one accumulator and 71 688 B with the two it needs, so it is over by 6 152 B. The three escapes are each closed by a different measured limit: a second output fifo costs 80 mem-tile channels against 48, a multi-block output element is rejected because the matmul/zero kernels are monoblock, and shrinking the band residency forces an order that needs the two accumulators anyway (npu-kernels#41) closed at T=512 in this design

    What this means for the issue's premise

    The issue asks for the two published layout ideas. One of them — reusing each streamed weight band across all T-tiles — is arithmetically unaffordable in this design at T=512, not merely unimplemented: the L1 that holds the band is the same L1 that must hold the accumulators, and the output paths that would relieve that are closed by descriptor and kernel-signature limits rather than by tuning.

    What would unblock it (one of, in increasing cost): a smaller m (halves the band buffers), a k-tile that is not 4-per-band, or a copy kernel that lets the accumulators live in a single multi-block buffer instead of one fifo element per result. The last is a new kernel, and it is the item this lane would fund next rather than another scheduling arm — seven measured forms say the scheduling space is exhausted.

    End-to-end ceiling remains as previously reported: attention 57.1 %, matmuls 28.2 TFLOP/s, per-GEMM ceiling 1.085×, e2e 1.035×, M ≤ 2048 1.000×.

  10. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    A correction to my own "what would unblock it" list, because I checked the code rather than leaving the list tidy: m is not a knob.

    M_TILE = 64 in the design is a hard constant, not env-driven, and its comment says why: "weight rows per band / per matmul m-tile (= mac_dims-compatible, matches q4 band's 64 rows)" — 64 rows × 256 K × 5/8 B = the 10 240 B band the packing format produces. So "a smaller m halves the band buffers" is only reachable by changing the weight packing format, not by turning a dial.

    And the third escape does not fit either. I costed the copy-kernel route with the same allocator arithmetic that closed the others: accumulators in two persistent buffers (2 × 8 192) plus one drain element (8 192) costs 24 576 B, against 16 384 B for the C fifo at depth 2 that the two releases need — i.e. it is 8 192 B worse, and the total lands on the same ~71.7 KB ceiling.

    So the honest final statement is stronger than my previous one: at m=64, k_tile=64, four k-tiles per band and T=512, no output path fits alongside 32 KB of band residency.

    band fifo element 10 240  (unavoidable: the band has to arrive)
    band buffers      32 768  (what "reuse" means: four dequantised k-tiles resident)
    both accumulators 16 384  (two releases need depth 2; or 24 576 via a copy path)
    A/B/nib/stack     22 528
                     --------
                     81 920 .. 89 088 against 65 536 available
    

    Even the most frugal version — three band buffers plus a re-dequant of the fourth — is 71 680 B. The reuse and the second accumulator cannot both be resident at T=512 in this design.

    That leaves exactly one real lever, and it is not in the kernel body: the weight packing format. A band of fewer rows (16 384 B at 32 rows) or fewer k-tiles per band makes the residency affordable; nothing about the schedule, the taps, or the output path can.

    Per-kernel detail and the seven measured forms: 1bit-MONSTER/npu-kernels#41 (merged 9d15783a).

  11. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Scope note: the engine-side half of this issue landed as #1556 (72b9b4a3); what remains is kernel-side and private.

    #1556 added the ceiling row to docs/npu.md — AM027 Table 14's 512 INT8 / 256 bf16 multipliers per tile at 1 GHz, so 1.024 TOPS int8 and 0.512 bf16 per tile, with our
    numbers placed against their own datatype's ceiling rather than against a whole-array denominator (which had overstated the gap by ~4.6×). The 18.64 tok/s prefill row
    in that table has since been retracted (#1665) and replaced with the TTFT figure, so the page no longer carries the number that flattered us.

    So the engine's side of this issue is: the measurement exists, in a document, with the retraction applied. The remainder — the two published layout ideas being
    misapplied to our prefill GEMM, and the GEMM's own 1.54–1.59 TFLOP/s on Llama3-8B's shapes — is npu-kernels work, which does not live in this repository and is not
    something an engine PR can close.

    Recording it here so the issue's remaining scope is unambiguous: nothing in 1bit-MONSTER/engine is outstanding for it, and closing it should follow the kernel
    result rather than an engine merge.

  12. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Where this issue stands, with one new measurement that bounds what it can buy on the served route (cl-npuperf; this issue was assigned to my lane today).

    State, checked rather than repeated: the engine-side half is merged (#1556, with the retraction in #1665). The kernel half was worked to a measured conclusion in npu-kernels #38 and #41 (both merged): seven forms of the weight-reuse restructure, none of which fits, and the closing arithmetic says the reuse and the second accumulator cannot both be resident in this design at T=512. Nothing is open in npu-kernels for this issue, and the last work on it was 2026-11-09 23:10Z. The acceptance row (≥ 4 TFLOP/s on Llama3-8B's two shapes, byte-exact) is not met: the measured row is still 1.54-1.59.

    New, from #1949's measurement on the served 27B route (bonsai2-27b-t2s2, today): a 16-row prefill block is 688-704 ms, of which device wait is 215 ms (31 %). The other 69 % is host work and per-launch host overhead. So on the route a user actually hits, an infinitely fast prefill GEMM is worth at most 1.45x, and the lever in front of it is the host half and the M=16 launch geometry (416 launches per 16 rows), not the per-tile TFLOP/s this issue is titled after. That agrees in direction with the 1.035x end-to-end ceiling the kernel lane reported for its own design.

    What is left is a decision, not a tuning job. The one lever the kernel work did not close is the weight packing format (a band of fewer rows, or a group-128 block as in the issue's item 3). That means a new container block format, a repack of every container, and a new kernel family with the byte-exact gate. I am not starting that without an owner call, because the measurement above says it would not be the first thing to move prefill tok/s on the served route. I have put the question to the owner through my state file; until it is answered I am leaving this open and unowned for kernel work, and taking the host-side terms through #1416's follow-ups.

  13. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Closing as measured, not reachable in this design (owner decision, 11-10). The GEMM measures 1.54-1.59 TFLOP/s against the 4 TFLOP/s row; the remaining lever is a new weight block format (a repack of every container plus a new kernel family), which is not funded now. Since the device wait is 31 % of a prefill block on the served route, the effort moves to the host half and launch geometry: #1990.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions