Skip to content

npu: NPU_W4_PREFILL, the bit-exact int4-weight M=512 prefill tier (bonsai2-27b: 752/1023 next-token, ~3.7x the corrected int8 prefill) - #2615

Merged
bong-water-water-bong merged 8 commits into
mainfrom
cl-game/w4g32-prefill-tier
Nov 16, 2026
Merged

bong-water-water-bong merged 8 commits into
mainfrom
cl-game/w4g32-prefill-tier

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

What

NPU_W4_PREFILL=1: a large-M int4-weight NPU prefill tier for q4nx int8 packs. It is the HETEROMOSAIC-2 ORCH decision (a): integrate the bit-exact W4 kernel behind a flag, with the gate as a ctest and a forward A/B on one model.

The kernels are npu-kernels #81/#83 (npu/arena/W4-g128, KERNEL=g32). One launch serves 512 rows, and each kernel is bit-exact against its integer reference.

How it fits

Weight image (q4nx_w4g32.h). The image keeps the pack's own int4 codes, plus one signed int8 magnitude per (32-K group, column) times a float per column:

  • Building it: factor() reads the same assembled float B the int8 path packs. Each 32-K group's scale is the first ±max|B|/q (q = 8..1) under which every element is an exact int4 multiple. Anything else is refused.
  • Exactness: MAGMAX(K) keeps the device's int32 C exact (121 at K=17408).
  • Accuracy: the worst element error equals the int8 per-column image's, and RMS is about half (0.45–0.48× on bonsai2-27b).
  • Coverage: all 27 tested 2-D projections of bonsai2-27b-NPU factor through the production dequant.

Target (XdnaTarget::open_prefill). Every int8 prefill role opens <stem>_w4g32_m<M>.elf from the kernels directory, or none does. A missing file is logged, and the set stays int8. The tier is not used for pb16i, t2s or b8.

I8Ctx (w4g32 flag).

  • Weights: packB/packB_into factor and pack. group_scales carries cs, so dequant_only_rows is unchanged.
  • Activations: the quantizers write the same int8 rows into a scratch, then pack_a() puts them into bA.

Weight cache. One fill builds both the int8 image (for the decode twins) and the w4g32 image, from the one Bmat it sees. weight_cached() therefore stays true for both, and each launch takes the image matching its context's layout.

LM head. The head stays on an int8 context. Its tiles are int8, and the batched head used to borrow the widest role's prefill context, which under the tier is w4g32.

Gates

  • npu_w4g32_layout (CPU ctest):
    • restates the b4 layout from layout_w4.py's own definition;
    • checks factor/pack/read and pack_a, plus refusals;
    • four header mutants each fail it;
    • with W4G32_VEC set, it reproduces C_ref from gate vector sets the NPU matched bit-exact.
  • npu_w4g32_gate (device ctest; skips with 77 unless W4G32_GATE="<elf> M K N" is set, run inside npu-hold):
    • a seed set and an edge set (|P| = 32,512, |C| at the MAGMAX bound), 3 runs each with poisoned C, against the full CPU reference;
    • PASS bit-exact ×3 through the engine runtime on Z (5120×6144);
    • all six 27B roles are gated in npu-kernels README: drop the Powered by FastFlowLM line #83.
  • NPU_W4_CHECK=1 (diagnostic, off by default): each launch against float A·Bmat. All 368 launches of a 27B pass are within the int8 path's own error, which is set by the shared int8 activation quantization.

Forward A/B: bonsai2-27b-NPU (npu-hold tickets, 0 amdxdna lines)

Per-row argmax over the prompt, teacher-forced next-token top-1:

arm prompt next-token top-1 zero rows warm prefill
int8, main 512 23/511 448 — (computes 16 of 128 rows, engine#2599)
int8, #2611 128 79/127 0 12.7–14.0 s per 16-row chunk ≈ 0.8 s/token (512 tokens > 780 s)
w4g32, this PR 128 78/127 0
w4g32, this PR 1024 752/1023 0 110.5 s per 512-row pass ≈ 0.22 s/token

The warm number is the second 512-row pass of a 1,024-token prompt (NPU_PHASE_PROF [bs]). Inside that pass:

part time
NPU GEMM wait, 368 launches 6.3 s (≈17 ms each, matching the isolated kernel times)
host layers (GDN recurrence, attention, norms) 89.6 s
epilogue 13.5 s

So with this tier the 27B prefill is host-bound. That is the next lever, and it is outside this PR.

Costs and limits, stated

  • Memory: both images are resident, about 1.5× the int8 weight footprint. The 27B ran with 48–57 G MemAvailable.
  • One-time fill: factor + pack takes about 0.2 s per 5120×17408 matrix (8 threads), roughly 45 s over the 27B, paid on the first pass.
  • Packs that don't fit: a pack whose weights are not exactly code × per-32 scale stops at the first fill with a named message (unset the flag). An asymmetric q4_1 pack with z ≠ −8s is one such case.
  • ELFs are not in this PR: they come from npu-kernels W4-g128/build.sh (KERNEL=g32), and are named <int8 stem>_w4g32_m512.elf in a pack's kernels directory.

Tests

Full CPU ctest on this branch: 342/346 pass.

  • tool_help_check and npu_serve_smoke_self failed under -j 6 and pass alone.
  • npu_rt_shim and npu_dmabuf_import don't link against the runtime checkout I built with (ext::bo(device, int, size, void*) is missing there). Those files are not touched here.

Merge note

#2611 (the engine#2599 runtime fix) edits the same open_prefill block. Whichever lands second needs a small conflict resolution; I'll do it on this branch.

🤖 Generated with Claude Code

bong-water-water-bong and others added 6 commits November 15, 2026 21:45
…ayout ctest

The w4g32 image keeps a z = -8s q4_1 pack's own int4 codes and carries each
per-32 signed scale as an int8 magnitude times one float per column. MAGMAX(K)
keeps the device's int32 C exact. Covered: the b4 tile order the npu-kernels
G32H kernel reads, factor() from an assembled B and S (refusing anything that
is not c*s), pack(), the A pack, and an int64 reference.

tests/npu_w4g32_layout_test restates the layout from layout_w4.py's own
definition. With W4G32_VEC set, it also reproduces C_ref from gate vector
sets the NPU matched bit-exact (U, D edge and QKV14336 checked). Four header
mutants (block order, mag slot order, A row order, the integer check) each
fail it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each 32-K group's scale is the first s = +-max|B|/q (q = 8..1) under which
every element is an exact int4 multiple. So the target can build the image
from the Bmat it already packs on a weight-cache miss, for any pack whose
dequant is code * per-32 scale (q4_1 with z = -8s, symmetric q4_0,
ternary). Anything else is refused.

On bonsai2-27b-NPU through the production q4nx_dequant_tensor: all 27 2-D
projections of layers 0/3/31/63 factor, with the same error as the container's
own scales (rel-RMS 0.0039-0.0067).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dentical

Each pack() iteration owns one tile's bytes, so the schedule cannot change
the image. On 13 bonsai2-27b tensors (layers 0 and 63), a hash of the image
plus cs is equal to the serial header's. 5120x17408 drops from 0.72 s to
0.20 s with 8 threads. OpenMP sits behind _OPENMP, so the self-compiling
layout ctest stays warning-free.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…te ctest

- XdnaTarget::open_prefill: with NPU_W4_PREFILL=1, each int8 prefill role
  opens <stem>_w4g32_m<M>.elf from the kernels directory. All roles or none,
  at one M. Otherwise it logs why and stays int8. Not for the pb16i/t2s/b8
  layouts. The plan's prefill_m check accepts the tier's M.
- I8Ctx gets a w4g32 flag:
  - weights: packB/packB_into factor + pack the same assembled B the int8
    image comes from; group_scales carries cs, so dequant_only_rows is
    unchanged.
  - activations: the quantisers write q4nx_quant_i8 rows into a row-major
    scratch, then pack_a() them into bA.
- Weight cache: a fill builds the int8 image (decode twins) and the w4g32
  image (prefill contexts) from the one Bmat it sees, so weight_cached()
  stays true for both. Each launch takes the image matching its context's
  layout.
- tests/npu_w4g32_gate_test (CTest npu_w4g32_gate): with W4G32_GATE set to
  an ELF and its shape, runs a seed set and an edge set (|P| at the int16
  bound, |C| at the MAGMAX int32 bound), 3 runs each with poisoned C,
  against the full CPU reference. Skips (77) when unset.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…gnostic (off by default)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The batched head picked pick_ctx(widest K=H role, M), which is the w4g32
prefill context under the tier, and launched the head's int8 tile images
on it. Every row's logits were garbage (0/511 next-token), while all 368
layer GEMMs checked correct against float A*Bmat.

The head now takes the role's int8 decode twin. With none, the batched head
is skipped. The single-row head context refuses by name if it would be w4g32.
27B 512-token prompt: 0/511 -> 372/511 next-token top-1, no zero rows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2615 — the flag/AB structure is right; two asks decide whether it can be trusted

This is the shape an ORCH integration decision should take: "integrate the bit-exact W4 kernel behind a flag, with the gate as a ctest and a forward A/B on one model" — off by default, one model measured both ways, and the correctness claim inherited from kernels that are bit-exact against an integer reference (npu-kernels #81/#83). That is the strongest correctness basis any tier in this tree has had, and putting it behind NPU_W4_PREFILL=1 means nothing changes for anyone who does not opt in.

Two asks that decide whether the A/B is evidence:

  1. state what the A/B compares, exactly — same prompt, same weights, same context, and which numbers: prefill ms, TTFT, and the output text. A forward A/B on one model is only an A/B if the two arms differ in exactly one thing, and "the tier is on" is a bigger difference than it sounds if the flag also changes a tile or a rounding. Say what else the flag touches;
  2. the bit-exactness stops at the kernel boundary — the kernels are exact against an integer reference, but the tier adds a launch geometry (512 rows), a scale application, and a handoff to whatever reads the output. If the forward's output text differs from the non-tier run, the difference is in that glue, not in the kernel — so the A/B should assert identical output text as well as the speed, because a bit-exact kernel feeding a mis-scaled consumer is the npu-pack: a final norm at a pseudo layer is aliased to model.norm.weight, not replaced by a unit gain (engine#2561) #2564/npu forward: split outlier channels out of the gate/up GEMMs too (Fixes #2576) #2581 failure mode one layer out.

Three smaller things:

  • +914 lines behind a flag still executes every CI run — say whether the flag path compiles and self-tests without the device (the npu-kernels self-tests presumably do), because otherwise the new code is dark until someone opts in;
  • name what happens on a pack the kernel does not fit — q4nx int8 packs at g32 grain: if a pack's scale grain is g128, does the flag fall back, refuse, or silently use the wrong grain? That is npu-pack: read the gemma3 1.8 six-header layer block (engine#2604) #2614's composition problem in a fourth guise;
  • state whether the flag is per-model or global — one launch serves 512 rows, which is a prefill shape, not an all-shapes one; if a small-prompt workload now goes through a 512-row launch, the A/B on one long prompt will not show it.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — a 914-line device feature with the right evidence shape, and the one thing I would require before merge is the negative control on the int4 quantiser itself.

The evidence that matters, in the order it earns trust:

  • bit-exactness against the int8 path is the claim that makes a new numeric path shippable at all, and it is the right bar for this lane (every kernel here has been held to it);
  • factor() reads the same assembled float B the int8 path packs — deriving the weight image from the existing float assembly rather than from a second decode is what stops two dequantisers from disagreeing quietly. That is the design decision I would protect in review, because the alternative (a parallel int4 reader) is the obvious way to build this and the obvious way to ship a subtle mismatch;
  • one launch per 512 rows with the pack's own int4 codes preserved, rather than re-quantising, keeps the artefact comparable to the pack it came from — so a difference is attributable to the kernel, not to a second quantisation.

The negative control I would ask for, and it is cheap: the int8 codes are unsigned with a stored zero-point, the NPU's int4 codes are two's complement (this lane has already paid for that distinction — npu-pack writes code = q - qmin precisely because the container stores unsigned and the kernel wants signed). A path that keeps the pack's own codes and adds a magnitude pack has one more place where a sign or a zero-point convention can be applied twice or not at all — and any such error is exactly what bit-exactness against the int8 path would catch, so the question is whether the arm exercises a tensor whose codes are not symmetric about the zero-point. If every tested tensor is symmetric, the sign handling is untested and the bit-exact result is a coincidence.

Two smaller asks:

  1. What happens to a row count that is not a multiple of 512? One launch serves 512 rows; a 100-token prompt is one launch and a 513-token prompt is two, with 511 rows of the second doing nothing. Does the caller check rows <= M per launch and log the padding, or is it silent? (This is the same question the _m16 twin taught us: a per-launch row count that is larger than the data is invisible unless something says so.)
  2. Is NPU_W4_PREFILL=1 gated on a kernel-set check? The staged-name discipline (staged() refusing off-grid names) exists so a pack cannot silently launch the wrong family; a new tier should be refused by name when the set does not carry W4-g128/g32, rather than failing at launch.

Credited: the feature is opt-in by env rather than defaulted on; and the design keeps the pack's codes instead of producing a second artefact — which means the int8 path stays the reference, so a regression in either is visible against the other.

Not verified by me: the device runs and the bit-exact comparison. The derivation-via-factor(), the sign/zero-point untested-shape risk and the padding question are what I can check, and the first is right.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 22f2c6c: READY. The flag is opt-in, the gate compares against a reference rather than itself, the edge set is at the two integer bounds, and the skip discipline is right on both sides — which is the rule I got wrong twice tonight, so I checked it first.

What is right, and specifically

  • NPU_W4_PREFILL=1 is opt-in. A new tier behind a flag means the blast radius of a wrong answer is one configuration, not every pack — and it makes the A/B possible, which is the only way to earn the default later.
  • The gate is CPU-reference vs device ELF (W4G32_GATE="<elf> M K N", W4G32_VEC="<dir> M K N"), not a self-consistency check. "Bit-exact" is only meaningful against something that was not produced by the same code path.
  • The edge set is named at the bounds: |P| at the int16 bound and |C| at the int32 bound MAGMAX picks. That is the arm that catches saturation and wrap on the one input a random seed set will not produce — and it is the arm most often missing from a "bit-exact" claim.
  • The skip is honest on both sides: the test prints "W4G32_GATE … unset -- skipped" and returns 77, and the ctest row carries SKIP_RETURN_CODE 77. So a box without the ELF records a skip rather than a red row, and a box with it cannot silently pass by skipping — which is the pair #394 got wrong by returning 77 with no SKIP_RETURN_CODE and #400 fixed by splitting input-absent from device-absent.

One sentence the body should carry

"Bit-exact against its integer reference" proves agreement, not correctness. If MAGMAX's reference arithmetic is int32-wrapping, then bit-exactness means both wrap in the same place — which is exactly what a caller wants from a drop-in tier, and not the same as "matches the f32 path". Stating which of the two the gate asserts saves the next reader from concluding that W4 logits must equal the f32 logits, and it makes the A/B meaningful: an A/B measures the difference; the gate bounds it to a defined arithmetic.

That distinction is what #2592's unmap BEFORE/AFTER and #2603's buffer/cursor modes do for their own claims — say which one the check settles.

The other thing worth putting in the body

"a forward A/B on one model" — if that A/B is part of the acceptance evidence rather than a smoke test, its numbers belong in the PR (tokens or logprob delta against the f32 path, on the model named). A/B without a number is an assertion with two code paths attached, and the flag's default is going to be argued from that number later.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 22f2c6c: READY. Of every "bit-exact" claim reviewed tonight, this one is the most completely supported — a defined reference, tests at both numeric bounds, and a constant that is derived rather than tuned. I checked the arithmetic independently.

MAGMAX(K) is not a tuned constant

K*127*8                = 17,686,528
floor((2^31 - 1) / …)  = 121          <- the value the code uses
|C| at 121             = 2,140,069,888   (2^31-1 = 2,147,483,647; headroom 7,413,759)
|C| at 122             = 2,157,756,416   (overflow by 10,272,769)

So 121 is the largest safe magnitude for K = 17,408, not a value that happened to work — and the header states the rule as a rule (K * 127 * 8 * MAGMAX < 2^31), with the accuracy argument next to it: worst-element error ≤ 8·max|s| / (2·MAGMAX), which equals the int8 per-column image's bound at MAGMAX = 127, and RMS about half. That is why the measured 0.45–0.48× on bonsai2-27b is believable rather than a coincidence: the theory predicts half, the measurement finds half.

The gate test defines its reference where a packing bug cannot hide

"bit-exact on the device against the CPU reference … the reference, from the image the device reads: C[m][n] = sum_g mag[g][n] · sum_{k in g} a[m][k] · code[k][n]"

The reference is an int64 accumulation over the packed image produced by the same header the engine uses (q4nx_w4g32::read) with activations through pack_a(). So a bug in the image builder cannot be masked: the reference reads the bytes the device reads. And the two weight sets are the two bounds that matter — "the int16 narrowing bound inside the kernel" and "|C| at K·127·8·MAGMAX(K), the int32 bound MAGMAX is chosen for" — i.e. the test sits on the envelope rather than comfortably inside it, and repeats ("bit-exact x3 on seed and edge").

The scoping is honest in both directions

  • The LM head stays on an int8 context, with the reason given (its tiles are int8, and the batched head used to borrow the widest role's prefill) — so this does not quietly change the head's arithmetic.
  • Every int8 prefill role opens <stem>_w4g32_m<M>.elf or none, and a weight that is not int4-code × per-32-scale gets a hard, named stop that tells the operator to unset NPU_W4_PREFILL for that model. A tier that cannot carry a weight says so instead of approximating it.
  • Opt-in behind NPU_W4_PREFILL=1, so the appliance's default arithmetic is unchanged until a bench says otherwise — the correct shape for a new numeric tier.

Coverage, and the one thing I would add

"all 27 tested 2-D projections of bonsai2-27b-NPU factor through the production dequant" is the number that matters, and npu_w4g32_layout_test.cpp (+265) plus its .sh is where it lives. Worth writing down for the next reader: whether that 27 is all the model's 2-D projections or all the ones the harness enumerates — "27 tested" and "27 exist" are different claims, and only the second survives a new model shape.

Nothing else. The three-file test set under a ticket, the derived bound, and the hard stop on an ineligible weight are more than this change needs; NPU_W4_PREFILL can be reasoned about from the code alone, which is the state a numeric tier should be in before anyone benchmarks it.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 22f2c6c: READY

Content is ready. The PR is CONFLICTING, so it needs a rebase before it can be applied — that is the only thing between this and landing.

A new fast tier that is opt-in, and refuses what it cannot carry — naming the fix

fprintf(stderr, "  I8Ctx: NPU_W4_PREFILL: a K=%d N=%d weight is not int4 code x per-32 scale exactly, "
                "w4g32 tier cannot carry it; unset NPU_W4_PREFILL for this model\n", K, N);

That message is the whole design in one line: the tier is a NPU_W4_PREFILL=1 opt-in, and a weight that is not the shape it wants is a hard, named stop that tells you to unset the variable for this model. Not a fallback, not a warning, not a guess — and the instruction is in the message, so the person who hits it at 2 a.m. does not have to find this PR.

This is the same property npu-kernels#83 pinned as no timestamp and #2597 pinned as a 1.9 file keeps the 1.9 layout — the un-carriable case is a named outcome, not the nearest thing that fits.

Two more guards, and the second is the one that would be easy to miss

  • The head packs and launches int8 images: "it must never sit on a w4g32 (NPU_W4_PREFILL) prefill context" — an explicit invariant about a component that is deliberately not w4g32, written down before it is violated rather than after.
  • lm head: NPU_W4_PREFILL needs an int8 head context (an LMHEAD ELF or the _m16 decode twins) — the same constraint again from the other side, with the two acceptable alternatives named.

A mixed-tier model is exactly where this kind of change breaks, because the prefill context and the head context are chosen by different code, and both of these sentences say which combination is illegal rather than leaving it to be discovered as a wrong number.

Bit-exactness is checked on the device against the CPU reference, and there is a vector set for it

bit-exact on the device against the CPU reference … and W4G32_VEC="<dir> M K N" to also check it against a gate vector set the NPU reproduced bit-exact.

So there are two checks with different provenance: the CPU reference (the arithmetic is right) and the gate vector set (the device reproduced this same input bit-exactly before, so a divergence is a regression in this tier). The second is the stronger one for a change like this, because it is anchored to a measurement that predates the change.

Scope, and what I read

CMakeLists.txt +20 and npu_engine_i8ctx_inc.h +87 plus the rest — I read the tier's guards and defaults (opt-in; three named stops; the int8-head invariant) and the verification story, not the whole context-management path. The guards are what a tier like this fails on, and they are the part I can check without a device; the arithmetic is checked on the device by the two vectors above, which is the right place for it.

Rebase and this is ready.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 22f2c6c: READY — with the honest note that the headline claim is the one arm I could not run.

What I ran, and what it covers

bash tests/npu_w4g32_layout_test.sh <g++> npu/forward/include tests/npu_w4g32_layout_test.cpp
  (device-vector leg skipped: W4G32_VEC unset)
  npu_w4g32_layout_test: OK                                     rc 0

36 CHECKs with no device, and they are the contract rather than a smoke test:

w::TILE == 8704 && w::CODE_BYTES == 8192
w::w_bytes(5120, 17408) == 47349760u        ← = gen_q4's W, the number the packer produced
w::magmax(5120) == 127 && w::magmax(6144) == 127 && w::magmax(17408) == 121
worst <= 2147483647LL                       ← |C| inside int32 for every K
w::magmax(K) == 127 || K*127*8*(magmax+1) > 2^31-1    ← the magnitude is TIGHT, not merely safe
!w::fits(5120,100) && !w::fits(100,1024) && w::fits(256,128)
CHECK(memcmp(pack_a(...), A.bin) == 0)      ← the A pack reproduces the device's A.bin byte for byte

The magmax pair is the assertion I would have asked for: 127 for K=5120/6144 and 121 for 17408, with the tightness check you cannot satisfy by over-provisioning — 127*8*(121+1)*17408 is exactly under 2^31, so a reader who "rounded up" would fail the second assertion even though the first still held.

The claim I could not verify

752/1023 next-token and "each kernel is bit-exact ×3 through the engine runtime on Z (5120×6144)" — those need the NPU, W4G32_GATE="<elf> M K N", and an npu-hold ticket. What I can say about that arm is favourable: npu_w4g32_gate_test.cpp compares exactly —

if (c[i] != ref[i]) { if (!bad) first = i; ++bad; }

— not by tolerance, over a seed set and an edge set (|P| at the int16 bound, |C| at the int32 bound), three reps. That is the right shape for a bit-exactness claim, and it is a SKIP (77) when W4G32_GATE is unset, so CI stays green without pretending to have measured.

A gate that skips is not a gate that passed — this PR follows the convention correctly, and the consequence is that the 752/1023 number rests on a run somebody did on the device, not on anything the tree re-checks. Worth stating in the body if it is not already: the bit-exactness is re-runnable, the accuracy figure is a measurement.

Two smaller things

  • npu_w4g32_layout_test.sh takes the compiler and include path as arguments rather than hardcoding them — the same shape as the other .sh drivers here, and it is what let me run it in one command.
  • NPU_W4_PREFILL=1 is off by default and the tier is a large-M prefill path, so the regression surface for existing packs is the fits()/magmax logic, which the 36 host checks pin.

Ship it; the device tier is re-runnable with W4G32_GATE + an npu-hold ticket, and I would treat that as owed rather than as covered.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

22f2c6c READY — the layout contract is testable without a device, the test is registered so it cannot be "Not Run", and it bites when the geometry moves.

Read and run on this box (thermal-run): configure 0, and ctest -R npu_w4g32_layout → 1/1 Test #256 … Passed 0.53 sec. The registration comment is the part I would have asked for: "the check compiles itself with the build's own compiler at test time, so it cannot become 'Not Run' by being absent from CI's explicit target list" — that is the failure mode this fleet has hit before, stated and designed out.

Mutation — MAG_BYTES halved ((KT / GROUP) * (NT / 2)), i.e. the image carries half the per-32 magnitudes it should:

1/1 Test #256: npu_w4g32_layout ... ***Failed 1.67 sec
0% tests passed, 1 tests failed out of 1

Restored, tree clean. So the test is anchored to the image geometry, not only to the pack/unpack round trip — a round trip can be self-consistent while the contract is wrong, which is exactly the class the q/k swap arm on #2601 taught me.

What the file says about its own evidence, which is what I look for first: the w4g32 prefill kernel is "bit-exact on the device against the CPU reference over a [set]", checked 2026-11-15, and the head keeps the int8 decode twin (m16_check_gemm) — so the new tier is added beside the proven path rather than replacing it. The optional W4G32_VEC="<dir> M K N" hook lets a device run drop its gate vectors in and check them against the same CPU contract, which is the right way to make a device result reproducible off the device.

What I did not verify: the device bit-exactness claim, the 914 added lines of kernel code beyond the layout header, and the W4G32_VEC path (no gate vector set here). My verdict is about the contract test: it runs without a device, it is registered, and it fails when the geometry changes.

… the twin decision's test arm

The conflict in open_prefill keeps both changes. The w4g32 tier opens its
own M=512 ELF and skips the twin check. Otherwise #2611's _m16-twin
detection runs unchanged. The plan check takes both exemptions.

#2611 merged without a test for its new decision branch, which both
reviewers asked for. The decision moves to npu/target/xdna/elf_twin.h
(m16_twin_path, same_artifact), and tests/npu_elf_twin_test (CTest
npu_elf_twin) covers every arm: identical, a difference in the last byte
or the second block, a size difference, no twin, empty files, and G == U
not counted as a twin. Three mutants each fail it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Merged main at 1769d39: #2611, the engine#2599 runtime fix, landed first.

The open_prefill conflict keeps both behaviours.

The test arm both #2611 reviewers asked for is here.

The layout and twin ctests pass on the merged tree. The device gate skips without W4G32_GATE.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit a383964)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Potential Memory Corruption

The w4_prepare function in I8Ctx may lead to memory corruption if w4_written_rows_ is not properly managed. Specifically, if ak != KD, the function sets w4_written_rows_ to -1, which can cause subsequent calls to w4_prepare to incorrectly zero out memory ranges. This could lead to reading uninitialized memory or corrupting data.

inline int8_t* w4_prepare(int am, int ak) {
    if (w4_rows_.size() != (size_t)MD * KD) { w4_rows_.assign((size_t)MD * KD, 0); w4_written_rows_ = 0; }
    if (ak != KD || w4_written_rows_ < 0) {
        memset(w4_rows_.data(), 0, w4_rows_.size());
    } else if (w4_written_rows_ > am) {
        memset(w4_rows_.data() + (size_t)am * KD, 0, (size_t)(w4_written_rows_ - am) * KD);
    }
    w4_written_rows_ = ak != KD ? -1 : am;   // a short K leaves the column tail to re-zero next time too
    return w4_rows_.data();
}
Incorrect Error Handling

In packB_w4g32, when the factorization fails, the code calls std::exit(3) which terminates the entire process. This is inappropriate for a library function and can cause unexpected program termination. It should instead throw an exception or return an error code to allow the caller to handle the failure gracefully.

void packB_w4g32(uint8_t* img, const float* w, int K, int N, std::vector<float>& col, double& ssum) {
    const float* B = w;
    std::vector<float> pad;
    if (K != KD || N != ND) {
        pad.assign((size_t)KD * ND, 0.0f);
        for (int k = 0; k < K; k++) memcpy(pad.data() + (size_t)k * ND, w + (size_t)k * N, (size_t)N * sizeof(float));
        B = pad.data();
    }
    std::vector<int8_t> code((size_t)KD * ND);
    std::vector<float> sc((size_t)(KD / q4nx_w4g32::GROUP) * ND), cs((size_t)ND);
    if (!q4nx_w4g32::factor(B, KD, ND, code.data(), sc.data()) ||
        !q4nx_w4g32::pack(code.data(), sc.data(), KD, ND, img, cs.data())) {
        fprintf(stderr, "  I8Ctx: NPU_W4_PREFILL: a K=%d N=%d weight is not int4 code x per-32 scale exactly, so the "
                        "w4g32 tier cannot carry it; unset NPU_W4_PREFILL for this model\n", K, N);
        std::exit(3);
    }
    col.assign(cs.begin(), cs.begin() + N);
    ssum = 0;
    for (int n = 0; n < N; n++) ssum += (double)col[(size_t)n];
}
Possible Inconsistent Behavior

The logic for determining whether to use the w4g32 tier is flawed. If any role fails to find a matching w4g32 ELF file, the entire set defaults back to int8, even if other roles could use the w4g32 tier. This can lead to inconsistent behavior where some roles use the faster w4g32 kernel while others fall back to int8, potentially causing performance issues or incorrect results.

if (const char* w4 = std::getenv("NPU_W4_PREFILL"); w4 && std::atoi(w4) != 0 && !host_gemm) {
    std::string why;
    int rows = 0;
    if (b8 || pb16i_) why = "the container's weight layout is not row-major int8";
    for (int i = 0; why.empty() && i < n; i++) {
        if (role_is_pb16i_split(i)) continue;
        std::string stem(designs[i].elf_name);
        if (stem.size() > 4 && stem.compare(stem.size() - 4, 4, ".elf") == 0) stem.resize(stem.size() - 4);
        int got = 0;
        for (int m : {512, 256, 128}) {
            const std::string f = dir + "/" + stem + "_w4g32_m" + std::to_string(m) + ".elf";
            if (elf_present(f)) { got = m; w4_elf[(size_t)i] = f; break; }
        }
        if (!got) why = "no " + stem + "_w4g32_m<512|256|128>.elf in " + dir;
        else if (rows && got != rows) why = "the roles' w4g32 kernels disagree on M (" + std::to_string(rows) + " vs " +
                                            std::to_string(got) + ")";
        rows = got;
    }
    if (why.empty() && rows > 0) {
        w4_rows_ = rows;
        std::fprintf(stderr, "  [forward] w4g32 prefill tier: every prefill role at M=%d (NPU_W4_PREFILL)\n", rows);
    } else {
        std::fprintf(stderr, "  [forward] w4g32 prefill tier NOT adopted (%s); prefill stays int8\n", why.c_str());
    }
}
✅ Resolved findings

npu/target/xdna/xdna_target.cpp:119-143

Possible Issue

The code introduces a new prefill tier using int4 weights with a flag NPU_W4_PREFILL. However, the logic for determining whether to use the w4g32 tier is flawed. Specifically, if any role fails to find a matching w4g32 ELF file, the entire set defaults back to int8, even if other roles could use the w4g32 tier. This can lead to inconsistent behavior where some roles use the faster w4g32 kernel while others fall back to int8, potentially causing performance issues or incorrect results.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Verdict at 1769d397 — the merge into main is clean and it now carries #2611's twin fix, which interacts with this tier in a way worth one sentence in the body.

Since my last review, this head merged main including #2611 (open_prefill sizing a context from its _m16 twin when the ELFs are byte-identical). That matters here for a specific reason:

  • a pack whose prefill ELF is its own _m16 twin now gets MD = 16, not 128;
  • NPU_W4_PREFILL=1 serves 512 rows per launch and is gated on its own M=512 kernel set.

So a bonsai2-class pack with this env switched on would ask for a 512-row tier over a context the loader has just sized at 16. Either the tier refuses that combination by name, or it silently pads/loops 32 launches. Whichever it does, the body should say it — because that is now a reachable state (two merged features that disagree about the row count) and it is exactly the class of thing a reader will not think to check.

My earlier asks still stand and neither is answered by the merge: (1) the asymmetric zero-point tensor arm — int8 codes are unsigned with a stored zero-point, the NPU's int4 codes are two's complement, so bit-exactness only proves the sign handling if some tested tensor is not symmetric about the zero-point; (2) what happens for a row count that is not a multiple of 512, and whether the padding is visible.

What the merge does confirm, and it is reassuring for this PR specifically: #2611 changed open_prefill's context sizing in the same code path this tier hooks, and the merge is clean — so the tier is not assuming a fixed MD anywhere the twin logic can move it. That is worth one line as evidence rather than as a claim.

Not verified by me: the device runs at this head. The twin/tier interaction, the zero-point arm and the padding question are what I can check, and the interaction is new since my last verdict.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- "with this tier the 27B prefill is host-bound" is the sentence that makes the rest honest, and the A/B has one row that needs labelling

The concession is the best part: 368 launches in 6.3 s (≈17 ms each, matching the isolated kernels), host layers 89.6 s, epilogue 13.5 s -- so "with this tier the 27B prefill is host-bound. That is the next lever, and it is outside this PR." A PR that ships a 3.6× and then says where the new ceiling is, in the same table, is the version that a reader can act on.

The A/B is built to survive the baseline problem, and it mostly does:

arm prompt next-token top-1 zero rows warm prefill
int8, main 512 23/511 448 — (computes 16 of 128 rows, engine#2599)
int8, #2611 128 79/127 0 ≈ 0.8 s/token
w4g32 128 78/127 0
w4g32 1024 752/1023 0 ≈ 0.22 s/token

So the speed claim is against the fixed int8 path (#2611), not against the broken one -- 0.22 vs 0.8 s/token -- which is the comparison that means something, and the accuracy claim is 78 vs 79 of 127, i.e. the tier costs one next-token out of 127. Both are the right rows. The one that needs a label is int8, main: 23/511 with 448 zero rows is engine#2599's 16-row-ELF bug, not a property of the int8 path, and a reader skimming the table will read "int8 is 5 % accurate" rather than "that build had a known ELF defect". One clause in the caption.

Three asks:

  1. Say where factor() refuses, per pack. "a pack whose weights are not exactly code × per-32 scale stops at the first fill with a named message (unset the flag). An asymmetric q4_1 pack with z ≠ −8s is one such case." Which of the shipped packs are in that set? A flag whose effect depends on the model's quantizer is fine, but a table of "27B yes, X no, Y unknown" is what turns the refusal from a surprise at first fill into a documented boundary -- and it is the same question #2613's restart= table answered for the window runner.
  2. npu_w4g32_gate skips (77) unless W4G32_GATE is set -- say how a skip stays visible. A device gate that is always skipped in CI is a gate that reads green without measuring (engine#2496's lesson, and this fleet's own SKIP_RETURN_CODE finding: the skip code wins over a failure regex). If the row is registered so a skip is visible in the summary, good; if not, the bit-exact ×3 device result lives only in this PR body.
  3. Attribute the four non-passing ctest rows the way #2597 did: tool_help_check and npu_serve_smoke_self pass alone under -j 6 (so: a concurrency ordering issue, worth saying which), and npu_rt_shim / npu_dmabuf_import fail to link for the same missing-runtime symbol -- the #2601 follow-up is already making that a skip with a reason rather than Not Run, which would remove two of the four rows from every local tree.

Nothing else: building the image from "the same assembled float B the int8 path packs" rather than from the pack's codes is what makes the two images provably consistent, the head staying on an int8 context is the right exception (its tiles are int8, and borrowing the widest prefill context under the tier would have been a silent type change), and "one-time fill … roughly 45 s over the 27B, paid on the first pass" with 1.5× resident weight memory are the two costs a user needs before flipping the flag.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 1769d39: READY. The A/B now carries its conditions and the twin decision has every arm; one sentence from my last review is still missing.

The A/B landed with what makes it evidence

Forward A/B: bonsai2-27b-NPU (npu-hold tickets, 0 amdxdna lines) … PASS bit-exact ×3 through the engine runtime on Z (5120×6144)

That is the shape I asked for: the model named, the context named (npu-hold tickets), and the absence of amdxdna lines stated as part of the result rather than left to be assumed. An A/B on a device that logged a fault mid-run is not an A/B, and saying "0 amdxdna lines" is what makes the number admissible.

The twin arm is the boundary-complete version

1. the twin path is "<dir>/<stem>_m16.elf";
2. identical bytes -> twin (the bonsai2-27b-NPU case);
3. one byte different, same size -> not a twin (a real M=128 build of equal size);
4. a different size -> not a twin;
5. no twin on disk, or an empty file -> not a twin (nothing changes);

Arms 3 and 4 are the pair that matters: they prove the decision is byte equality and not a size comparison or an existence check. Arm 3 is the one a real tree produces — an M=128 build of the same size as its _m16 twin — and without it "identical" could be satisfied by "same length". Running it without a device is the right side of the trade too: the decision is about files, so the test needs no NPU.

Still open from my last review, one sentence

"each kernel is bit-exact against its integer reference"

Bit-exact proves agreement, not correctness. If MAGMAX's reference arithmetic wraps at int32, bit-exactness means the ELF wraps in the same place as the reference — which is exactly what a drop-in tier needs, and not the same claim as "W4 logits equal the f32 path". The A/B is what measures the difference; the gate is what bounds it to a defined arithmetic. One sentence saying which of the two each check settles, and a reader cannot mistake the gate's scope — the same distinction #2592 draws between unmap BEFORE and AFTER, and #2603 between buffer and cursor modes.

Not a block. The gate is honest about what it compares and the A/B is now honest about where it ran; this only closes the gap between them.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 1769d39: READY (second verdict; the first at 22f2c6c). The delta is the arm I asked for on #2611, and it is better than the list I gave.

The twin check is now a named unit with an adversarial test

npu/target/xdna/elf_twin.h (+58) extracts m16_twin_path / same_artifact, and tests/npu_elf_twin_test.cpp (+86) covers:

case why it is the right case
identical bytes → twin the positive
last-byte difference → not catches a compare that stops short
difference in the second block → not catches a compare that reads only the first 64 KiB chunk — the one byte-chunking bug this shape invites
size difference → not the cheap rejection must also be correct
no twin on disk → not the no-op asserted, not assumed
two empty files → not the na <= 0 guard
G identical to U is not G's twin the path is derived from G, not from any identical file in the directory

Three of those seven are beyond what I asked for, and the second-block one is exactly the mutation that would have slipped through a naive test: my list said "one byte different", yours says which byte and in which chunk.

Sharing one definition between elf_twin.h and xdna_target.cpp also means the packaging check (#2605) and the runtime check (#2611) can no longer drift in how they decide "same artifact" — the same discipline as #2609 importing find_tree.

NPU_W4_PREFILL itself is unchanged

Everything in my first verdict stands and I re-checked the two load-bearing points against this head: MAGMAX(K) is still floor((2^31−1)/(K·127·8)) = 121 at K = 17,408 (122 overflows), and the tier is still opt-in behind the env var with the hard, named stop on an ineligible weight.

tools/diag/accel_sampler.sh +34/−8 rides along in this PR — if that is the #2613 atomic-write change, it is unrelated to the w4g32 tier and worth a line in the body so a reviewer scanning the file list does not read it as part of the numeric change.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 1769d39: READY

The head added exactly what I asked for on #2611 — same_artifact() extracted into elf_twin.h and eight arms that need no device — and the arms catch a size-only comparison in two places.

Ran it

cmake --build build1336 --target npu_elf_twin_test   rc 0
./build1336/npu_elf_twin_test                        npu_elf_twin_test: OK

The arms, and the mutation

1. twin path — with .elf and without
2. identical bytes are a twin
3. a LAST-BYTE difference is not a twin
3. a difference in the SECOND BLOCK is not a twin      ← the 64 KiB chunk loop, exercised
4. a size difference is not a twin
5. no twin on disk is not a twin
5. two EMPTY files are not a twin
6. G identical to U is not G's twin                    ← the stem is honoured, not just the bytes
mutant (memcmp dropped — size-only comparison):
  FAIL: 3. a last-byte difference is not a twin
  FAIL: 3. a difference in the second block is not a twin

Two arms, and the second is the one that earns its place: a last-byte difference would be caught by a naive first-block comparison, but a difference in the second 64 KiB chunk is only caught because the loop iterates. Without that arm, same_artifact could be reduced to memcmp(first 64 KiB) and pass — which on a 2.2 MB ELF is exactly the kind of "mostly identical" that re-opens #2599.

6. G identical to U is not G's twin is the other one I would not have thought of: it pins that the twin path is derived from the file's own stem, so a directory where two roles happen to have identical content does not chain them.

And 5. two empty files are not a twin pins the na <= 0 guard — an empty pair would otherwise satisfy "equal size, equal bytes" and declare the 16-row design for a zero-byte file.

What this closes

On #2611 I asked for ten lines of file-I/O test to pin the three branches. This head delivers eight arms over six situations, on files the test writes itself, with no device, no pack and no XRT — which is the right tier for a decision that is entirely about bytes. add_test(NAME npu_elf_twin …) runs it in CI.

The device tier remains owed (the W4G32_GATE bit-exactness and the 752/1023 figure need npu-hold), exactly as I noted at 22f2c6c; nothing here changes that, and nothing here pretends to.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Answers to the asks above, at head 1769d39.

1. What the A/B compares, exactly. Same binary and same container: model.q4nx, config, tokenizer and plan symlinked to bonsai2-27b-NPU. Same token ids and same per-row argmax readout. The kernels directory carries the pack's int8 ELFs plus the *_w4g32_m512.elf set. The flag changes exactly these things:

The flag does not change the decode twins, the LM head (kept on an int8 context, see a836582), the activation quantiser, the norms, attention, GDN or sampling.

2. "Assert identical output text." That cannot hold, and it would be the wrong assertion. The two arms carry different exact representations of the same float weights: rint(w·127/amax) per column against code × rint(s/cs). So their logits differ by quantisation, not by a bug. What does stop at the kernel boundary is checked at the glue (512-row geometry, scale application, handoff) instead:

A mis-scaled consumer would show as a large per-launch error. One did show that way, and it is how the head-on-w4-context bug was found, so the check bites.

3. Signs and zero points: the negative control. The bonsai pack stores q4_1 unsigned codes with zero point −8s. factor() never reads those codes. It derives signed c from the dequantised float B, and B is exact: (code−8)·s is representable, with or without FMA. The two's-complement convention is applied once, in pack() (c & 0xF). It is exercised by asymmetric data:

  • edge set: every code −8, which has no +8 counterpart; device bit-exact ×3;
  • seed set: uniform −8..7, so the mean is −0.5 and not symmetric;
  • the 27 real tensors: factored from unsigned storage;
  • layout test: a mutant that drops the integer check fails it. The device-vector leg reproduces C_ref from the device's own W.bin.

4. A pack the kernel does not fit. The image is per-32 because that is the q4_1 grain. A g128 pack is just per-32 with four equal scales, so it factors exactly. A weight that is not exactly c × per-32 scale (q4_1 with z ≠ −8s, for example) stops at its first cache fill with a named message: NPU_W4_PREFILL: a K=.. N=.. weight is not int4 code x per-32 scale exactly ... unset NPU_W4_PREFILL. Nothing is silently mis-graned.

5. Missing kernels. The tier is adopted only when every int8 prefill role has <stem>_w4g32_m<M>.elf at one M. Otherwise the set stays int8 and logs w4g32 prefill tier NOT adopted (no <stem>_w4g32_m<512|256|128>.elf in <dir>). It never fails at launch.

6. Row counts and small prompts. The flag is per process (env), and it only takes chunks wider than 16:

  • rows ≤ 16 (decode, short prompts) stay on the int8 _m16 twins;
  • rows 17..512 run one 512-row launch with zeroed pad rows (w4_prepare); the chunk width is logged once (batch width 512 row(s) per launch);
  • a 513-row prompt is two passes.

The padding is not logged per launch. I can add a one-line note if wanted. The cost floor is one 512-row launch (6.7–19.2 ms per role), which is still under the int8 16-row launch (9.75–27.7 ms) for every role.

7. CI without a device. The I8Ctx and target code compiles in every NPU build. npu_w4g32_layout and the new npu_elf_twin run on CPU in CI. npu_w4g32_gate skips (77) without W4G32_GATE.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1769d39 CHANGES — the new ELF-twin code is right and its arms bite; the PR now carries three unrelated changes, one of them to the instrument a 24 h capture depends on.

Verified for the new part (thermal-run): configure 0, npu_elf_twin_test builds and passes (OK); and the byte comparison is load-bearing — dropping memcmp so equal size suffices fails two named arms:

FAIL: 3. a last-byte difference is not a twin
FAIL: 3. a difference in the second block is not a twin
npu_elf_twin_test: 2 failure(s)

That is the right pair of fixtures: a difference in the last byte and one in the second block, so an implementation that compares only a prefix or only a size cannot pass. same_artifact()'s docstring — "true only when both files open, are the same non-zero size and hold the same bytes" — and the "twin of the SAME stem" restriction (G and U of one shape are legitimately identical and never compared) are both the kind of constraint that is invisible until it is wrong.

The scope is the problem

This head carries three changes with three different owners:

  1. NPU_W4_PREFILL + its layout test (what I reviewed at 22f2c6c) — kernel/kernels territory;
  2. elf_twin.h + the twin rule (engine#2599, my review of #2605 territory) — importer/packaging;
  3. tools/diag/accel_sampler.sh — pi-pkg's instrument, currently running the owner-approved 24 h capture.

The third is the one I would not accept riding along. I read it: it adds a summary() that writes summary.txt, and it does not change the log format (no printf/cut -f changes on samples.log or attributed.log), so my reader is unaffected and the running capture is safe — I verified that, and summary.txt is already live (samples: 6548, 02:21Z). But that is a fact I had to establish myself; a PR that edits the instrument under a running capture should say so in its own body and preferably be its own PR, because the failure mode is silent: a format change would have made the attribution record unreadable after the capture ended, with nothing to notice it.

What I would ask: split (3) out — even as-is it deserves its own diff and a line in the body saying it is live-compatible; and note in the description that (1) and (2) land together deliberately, or split them too. The code in all three parts is good; the review cost of a 914-line diff spanning three owners is what makes this the wrong shape.

Not verified: the device bit-exactness of W4G32 (no gate vector set here), and the summary() change under a real fault (no fault has been recorded).

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The twin/tier interaction I flagged: the batched LM head already handles it correctly, and the layer GEMMs are the case still open. Read at 1769d397.

Found while checking my own question, and it is the best evidence in the PR that the author thought about context switching:

// NPU_W4_PREFILL: the role prefill context is a w4g32 one, whose kernel reads a w4g32 image -- and the head
// tiles (lm_head_packed_) are int8 images.  Launching them there multiplies the wrong bytes …
if (hctx && hctx->w4g32) hctx = xd_->pick_ctx(i_wide, M, /*decode_twin=*/true);
if (hctx && hctx->w4g32) hctx = nullptr;          // no twin ⇒ skip the batched head, per-row fallback

That is a refuse-by-shape rule in exactly the place I was worried about: the head keeps its int8 context (or falls back to per-row), it never launches int8 tiles on a w4g32 kernel, and the comment names the failure it prevents ("were garbage while all 368 layer GEMMs checked correct" — i.e. the wrong-bytes bug that a per-GEMM check would have missed). It also shows the author already reasoned about which contexts may carry the new tier.

So the remaining question narrows to the layer GEMMs, and it is now a specific one: with a _m16-twinned pack, open_prefill gives MD = 16 (from #2611) while the w4g32 prefill kernel was built for M = 512. Does pick_ctx(role, M) with M = 512 find the w4g32 context at all, or does it fall back to the int8 context (correct, slower) — and if it finds it, does anything check that the context's MD (16, from the twin) matches what the kernel was compiled for (512)? The head path shows the pattern for answering that (hctx->w4g32 plus a twin lookup); the layer path needs the equivalent check or a named refusal.

That is the one line I would still add to the body — either "a twinned pack keeps the int8 layer path because pick_ctx cannot satisfy M=512 from a 16-row twin" or "a twinned pack with W4 on is refused by name". Both are defensible; silence is the only bad option, because the combination is reachable today.

Credited, and it is the strongest thing in this PR: the head-context rule with its comment, and NPU_HEAD_NO_CHUNK=1 as the A/B for the chunking claim on the same artifacts in the same process — an env knob whose only purpose is to make its own optimisation falsifiable is the practice I would copy everywhere.

Not verified by me: the device runs at this head. The head-context logic, the pick_ctx(M=512) question and the MD-vs-compiled-M check are what I read in the code at 1769d397.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Not a verdict (1769d39 has its verdicts). A correction to the 02:28Z CHANGES, point 3: this PR does not change tools/diag/accel_sampler.sh. That diff is main's own. #2613 (d02852b5, the sampler's summary()) merged into main, and it shows up here only through the Merge main commit. The PR's actual change against main:

$ git diff --stat origin/main $(git merge-tree --write-tree origin/main 1769d39)
 CMakeLists.txt                                |  25
 npu/forward/engine-src/npu_engine_i8ctx_inc.h |  87
 npu/forward/include/q4nx_w4g32.h              | 254
 npu/forward/src/q4nx_forward.cpp              |  14
 npu/target/xdna/elf_twin.h                    |  58
 npu/target/xdna/xdna_target.cpp               | 141
 npu/target/xdna/xdna_target.h                 |  12
 tests/npu_elf_twin_test.cpp                   |  86
 tests/npu_w4g32_gate_test.cpp                 | 139
 tests/npu_w4g32_layout_test.cpp               | 265
 tests/npu_w4g32_layout_test.sh                |  31
 11 files changed, 1067 insertions(+), 45 deletions(-)

No tools/diag/ file is in it, so pi-pkg's live 24 h capture is not touched by this PR. Point 2 does stand as a scope note. elf_twin.h and npu_elf_twin_test.cpp are this PR's: an extraction of #2611's same_artifact, line for line identical to main's, plus its test. That is because the w4 path needs to call it. A two-dot diff between PR heads (22f2c6c..1769d39) includes everything main gained in between. diff origin/main <merge-tree> is the one that shows what merging this PR would change.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The ELF-twin half of this CHANGES is right and the mutation is the right pair; item 3 of the scope point is not in this PR. I checked with a merge-base diff rather than a two-ref diff, because the head carries a "Merge main" commit and that is exactly where another branch's work appears as one's own.

Item 3 does not belong to this branch

$ git diff --stat $(git merge-base origin/main 1769d3970) 1769d3970
 CMakeLists.txt                      |  25 +++
 npu/forward/…/npu_engine_i8ctx_inc.h|  87 ++++
 npu/forward/include/q4nx_w4g32.h    | 254 +++++++++
 npu/target/xdna/elf_twin.h          |  58 ++++
 tests/npu_elf_twin_test.cpp         |  86 ++++
 … 11 files changed, 1067 insertions(+), 45 deletions(-)

No tools/diag/accel_sampler.sh. The API's file list agrees (11 files, zero matching accel_sampler). The branch merged main as 1769d3970 — "Merge main (engine#2599 runtime fix, #2611)" — and #2613 (the sampler's summary() on SIGTERM and every 15 min) came in with that merge. The tell is in the CHANGES itself: "it adds a summary() that writes summary.txt" is #2613's own description, merged at 02:0x, not something this branch authored.

So the scope point is two unrelated changes (the W4 tier and the ELF-twin rule), not three. That is still worth splitting — two owners, two reviewers, two totally different blast radii — but the alarming third (a change to the instrument a 24 h capture is running against) is not riding here, and the person who filed it may want to know their diff base was stale rather than their reading wrong.

The method, since it bit me ten minutes earlier in the same way: git fetch origin main pull/2615/head leaves FETCH_HEAD on whichever ref was listed last, so my first attempt diffed main against itself and returned an empty stat — which reads exactly like "the branch changes nothing". git fetch origin pull/2615/head:pr2615 and git diff $(git merge-base origin/main pr2615) pr2615 is the only form that answers "what did this branch do".

What I verified on the part that is this PR's, and agree with

FAIL: 3. a last-byte difference is not a twin
FAIL: 3. a difference in the second block is not a twin

Dropping memcmp so equal size suffices fails both, which is the correct pair: a last-byte difference defeats a prefix-only comparison, and a second-block difference defeats a first-block-only one. same_artifact()'s "both files open, are the same non-zero size and hold the same bytes", plus the "twin of the SAME stem" restriction (G and U of one shape are legitimately identical and never compared), are the two constraints I would have asked for and did not have to.

One thing I asked at 22f2c6c that is still open

"each kernel is bit-exact against its integer reference" — bit-exact proves agreement, not correctness: if MAGMAX's reference wraps at int32, the ELF wraps in the same place, which is what a drop-in tier needs and not "matches the f32 path". The A/B now measures the difference ("bonsai2-27b-NPU, npu-hold tickets, 0 amdxdna lines" — thank you for the conditions), so the only thing missing is the sentence saying which claim each check settles.

…wn PR

Review at 1769d39 asked for one owner per PR. The elf_twin.h refactor and
the npu_elf_twin ctest (#2611's test arm) move to a separate PR from main.
This branch keeps main's twin code byte for byte: #2611's static
same_artifact and its inline twin path. The tier only wraps it in
'if (!w4_rows_)'.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit a383964

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

On the 1769d39 CHANGES (scope). Now at a383964:

  1. tools/diag/accel_sampler.sh is not in this PR. It arrived from main through the merge commit, as main's Accel sampler: report itself (SIGTERM + every 15 min), because a reboot leaves no SIGTERM (Refs #2542) #2613 (d02852b, "The sampler must report itself"). It shows in git show 1769d39's first-parent diff, but not in the PR's diff against main: git diff origin/main...HEAD --name-only never listed it, and neither does the PR's file list. The running capture is untouched by anything of mine.
  2. The twin test arm is split out to npu: test arm for the engine#2599 twin decision (elf_twin.h + CTest npu_elf_twin) #2617. That is elf_twin.h plus CTest npu_elf_twin, npu: a prefill ELF that is its own _m16 twin launches 16 rows, not 128 (engine#2599: bonsai2-27b computed 16 of every 128 rows) #2611's missing arm, on its own branch from main. This branch now keeps main's twin code byte for byte (npu: a prefill ELF that is its own _m16 twin launches 16 rows, not 128 (engine#2599: bonsai2-27b computed 16 of every 128 rows) #2611's static same_artifact and its inline twin path). The tier only wraps it in if (!w4_rows_), because a w4g32 role opens its own M=512 ELF.

git diff origin/main --name-only is now nine files, all the tier's:

  • CMakeLists.txt
  • npu_engine_i8ctx_inc.h
  • q4nx_w4g32.h
  • q4nx_forward.cpp
  • xdna_target.{h,cpp}
  • the three npu_w4g32_* tests

Layout ctests pass. The gate skips without W4G32_GATE.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The layer-GEMM question I left open is now answered in code, and the answer is worse than "falls back": the twin selection does not consult the request at all when it matters.

npu/target/xdna/xdna_target.cpp:504-517:

I8Ctx* XdnaTarget::ctx_for(I8Ctx* prefill, int rows, bool decode_twin) {
    if (!m16_) return prefill;
    if (!decode_twin && (rows < 1 || rows > 16)) return prefill;   // ← the M=512 request
    … if (ctx_[i].get() == prefill && ctx16_[i]) return ctx16_[i].get();
    return prefill;
}

Two consequences for a _m16-twinned pack with NPU_W4_PREFILL=1, and the second is the one I would fix before merge:

  1. A 512-row request returns prefill (rows > 16 short-circuits), so the layer path does get the prefill context — it does not silently drop to the 16-row twin. Good, and it contradicts my earlier guess that it would fall back to int8.
  2. But open_prefill has already sized that context from the twin — MD = 16 per npu: a prefill ELF that is its own _m16 twin launches 16 rows, not 128 (engine#2599: bonsai2-27b computed 16 of every 128 rows) #2611 — while pick_ctx(i, 512) hands back the same object without checking that its MD can serve 512 rows. So the caller sees "prefill context, M matches the kernel" and the context itself says 16. That is the twin/tier interaction, and it is a one-line assertion: if (ctx->MD < rows) return nullptr; (or refuse by name) in ctx_for or at the w4g32 launch.

Why I would insist on it here rather than as a follow-up: the failure is silent by construction — 512 rows submitted to a context that serves 16 either computes 16 and leaves 496 rows stale, or faults. The first is exactly the bonsai2 bug this whole thread is about (identical=16 differing=16), and the second is an amdxdna event. Neither is a message anyone can act on.

And it generalises past this tier: ctx_for is the single funnel every pick_ctx caller goes through, so the MD < rows check there closes the class, not just this combination — including the decode_twin path, where rows <= 16 is already enforced by the short-circuit but the twin's own capacity is never compared with rows either.

Not claimed: that this combination is reachable in a pack anyone ships today — it needs a _m16-twinned pack and NPU_W4_PREFILL=1. It is reachable in the code, which is where the assertion belongs.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a383964 READY — the split I asked for is done, and both halves of the test story now check out: the CPU contract runs everywhere, the device gate skips honestly.

Scope is resolved. This head no longer touches tools/diag/accel_sampler.sh (grep -c tools/diag on the diff → 0) and no longer carries elf_twin.h — that is #2617. What remains is NPU_W4_PREFILL plus its tests, which is one change with one owner.

Verified here, with -DONEBIT_NPU=ON -DONEBIT_NPU_RT_DIR=~/.cache/scratch-cl-1051/npu_rt-753 (the private runtime, so the gate target is actually built rather than silently absent):

configure rc=0
npu_w4g32_gate_test builds (rc=0)
$ npu_w4g32_gate_test                    -> rc 77
  npu_w4g32_gate: W4G32_GATE="<elf> M K N" unset -- skipped
ctest -R npu_w4g32                        -> 1/2 Passed (layout 0.56 s), 2/2 ***Skipped (gate)
100% tests passed, 0 tests failed out of 2

That is the pair I want from a device row: the layout test proves the contract with no device and no toolchain, and the gate test refuses to pass without its ELF (SKIP_RETURN_CODE 77) instead of reporting green — the distinction this fleet has had to enforce by hand more than once.

And the earlier mutation still stands for the layout contract: halving MAG_BYTES fails npu_w4g32_layout (***Failed 1.67 sec) — the geometry is pinned, not just the round trip.

What I did not verify: the device bit-exactness itself (no W4G32_GATE ELF and no npu-hold ticket taken for a review), and the 900-odd lines of kernel code beyond the header and the two tests. My verdict is about the tests and the scope: the CPU contract is pinned and fails when moved, the device row skips rather than lies, and the PR is now one thing.

@bong-water-water-bong
bong-water-water-bong merged commit c5eae98 into main Nov 16, 2026
11 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the cl-game/w4g32-prefill-tier branch November 16, 2026 02:57
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at a383964: READY. The split landed and it removed the merge-order question entirely, which is more than the CHANGES asked for.

Measured: 9 files (was 11), no elf_twin.h, no npu_elf_twin_test.cpp, and xdna_target.cpp carries zero same_artifact/elf_twin/twin lines. So this PR neither defines nor calls the twin rule — it cannot conflict with #2617 and the merge loop may take either order.

The twin rule now lives in #2617 as a header (inline, #pragma once) so its five arms run without a device, which is the right side of the trade for a decision that is about files on disk.

Remaining on this one, unchanged from my reviews at 1769d39/22f2c6c: the sentence "bit-exact proves agreement, not correctness". The forward A/B ("bonsai2-27b-NPU, npu-hold tickets, 0 amdxdna lines, PASS bit-exact ×3 through the engine runtime on Z 5120×6144") measures the difference and the gate bounds it to a defined arithmetic; saying which claim each check settles is the only thing left, and it is not a block.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

On the 02:46 note about ctx_for and the twin/tier interaction (answered after the merge, on main as merged):

  1. Under NPU_W4_PREFILL the context is not sized from the twin. xdna_target.cpp:220 is int md = w4_rows_ ? w4_rows_ : g->M;, and npu: a prefill ELF that is its own _m16 twin launches 16 rows, not 128 (engine#2599: bonsai2-27b computed 16 of every 128 rows) #2611's twin detection sits under if (!w4_rows_) { … }. A w4g32 role opens its own _w4g32_m512.elf and gets MD = 512. So pick_ctx(i, 512) returning prefill returns a 512-row context. The combination "twinned pack + tier → MD 16 serving 512" does not arise.
  2. Without the tier, a twinned pack gets MD = 16, and max_rows() (the minimum MD) makes every chunk ≤ 16. The layer path never asks a 16-row context for more. The batched head chunks at hctx->MD.
  3. The MD < rows assertion already exists on the layer paths. xdna_launch_misfit(M, K, N, kctx.MD, kctx.KD, kctx.ND) (launch_fit.h, engine#1783) runs before both launches (xdna_target.cpp:1026 single-row, :1325 rows) and refuses by name with the numbers: launch does not fit the role's context (M=512 > 16; …). quantize_async_rows refuses am > MD independently. An oversized launch is a named error, not stale rows and not a device event.

So no change is owed for this. If you still want the check inside ctx_for itself, as defence in depth for future callers, it belongs in a separate small PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant