Repository navigation
npu: NPU_W4_PREFILL, the bit-exact int4-weight M=512 prefill tier (bonsai2-27b: 752/1023 next-token, ~3.7x the corrected int8 prefill) - #2615
Conversation
…ayout ctest The w4g32 image keeps a z = -8s q4_1 pack's own int4 codes and carries each per-32 signed scale as an int8 magnitude times one float per column. MAGMAX(K) keeps the device's int32 C exact. Covered: the b4 tile order the npu-kernels G32H kernel reads, factor() from an assembled B and S (refusing anything that is not c*s), pack(), the A pack, and an int64 reference. tests/npu_w4g32_layout_test restates the layout from layout_w4.py's own definition. With W4G32_VEC set, it also reproduces C_ref from gate vector sets the NPU matched bit-exact (U, D edge and QKV14336 checked). Four header mutants (block order, mag slot order, A row order, the integer check) each fail it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each 32-K group's scale is the first s = +-max|B|/q (q = 8..1) under which every element is an exact int4 multiple. So the target can build the image from the Bmat it already packs on a weight-cache miss, for any pack whose dequant is code * per-32 scale (q4_1 with z = -8s, symmetric q4_0, ternary). Anything else is refused. On bonsai2-27b-NPU through the production q4nx_dequant_tensor: all 27 2-D projections of layers 0/3/31/63 factor, with the same error as the container's own scales (rel-RMS 0.0039-0.0067). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dentical Each pack() iteration owns one tile's bytes, so the schedule cannot change the image. On 13 bonsai2-27b tensors (layers 0 and 63), a hash of the image plus cs is equal to the serial header's. 5120x17408 drops from 0.72 s to 0.20 s with 8 threads. OpenMP sits behind _OPENMP, so the self-compiling layout ctest stays warning-free. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…te ctest
- XdnaTarget::open_prefill: with NPU_W4_PREFILL=1, each int8 prefill role
opens <stem>_w4g32_m<M>.elf from the kernels directory. All roles or none,
at one M. Otherwise it logs why and stays int8. Not for the pb16i/t2s/b8
layouts. The plan's prefill_m check accepts the tier's M.
- I8Ctx gets a w4g32 flag:
- weights: packB/packB_into factor + pack the same assembled B the int8
image comes from; group_scales carries cs, so dequant_only_rows is
unchanged.
- activations: the quantisers write q4nx_quant_i8 rows into a row-major
scratch, then pack_a() them into bA.
- Weight cache: a fill builds the int8 image (decode twins) and the w4g32
image (prefill contexts) from the one Bmat it sees, so weight_cached()
stays true for both. Each launch takes the image matching its context's
layout.
- tests/npu_w4g32_gate_test (CTest npu_w4g32_gate): with W4G32_GATE set to
an ELF and its shape, runs a seed set and an edge set (|P| at the int16
bound, |C| at the MAGMAX int32 bound), 3 runs each with poisoned C,
against the full CPU reference. Skips (77) when unset.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…gnostic (off by default) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The batched head picked pick_ctx(widest K=H role, M), which is the w4g32 prefill context under the tier, and launched the head's int8 tile images on it. Every row's logits were garbage (0/511 next-token), while all 368 layer GEMMs checked correct against float A*Bmat. The head now takes the role's int8 decode twin. With none, the batched head is skipped. The single-row head context refuses by name if it would be w4g32. 27B 512-token prompt: 0/511 -> 372/511 next-token top-1, no zero rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
VERDICT: READY This is the shape an ORCH integration decision should take: "integrate the bit-exact W4 kernel behind a flag, with the gate as a ctest and a forward A/B on one model" — off by default, one model measured both ways, and the correctness claim inherited from kernels that are bit-exact against an integer reference (npu-kernels #81/#83). That is the strongest correctness basis any tier in this tree has had, and putting it behind Two asks that decide whether the A/B is evidence:
Three smaller things:
|
|
READY — a 914-line device feature with the right evidence shape, and the one thing I would require before merge is the negative control on the int4 quantiser itself. The evidence that matters, in the order it earns trust:
The negative control I would ask for, and it is cheap: the int8 codes are unsigned with a stored zero-point, the NPU's int4 codes are two's complement (this lane has already paid for that distinction — Two smaller asks:
Credited: the feature is opt-in by env rather than defaulted on; and the design keeps the pack's codes instead of producing a second artefact — which means the int8 path stays the reference, so a regression in either is visible against the other. Not verified by me: the device runs and the bit-exact comparison. The derivation-via- |
|
VERDICT at 22f2c6c: READY. The flag is opt-in, the gate compares against a reference rather than itself, the edge set is at the two integer bounds, and the skip discipline is right on both sides — which is the rule I got wrong twice tonight, so I checked it first. What is right, and specifically
One sentence the body should carry"Bit-exact against its integer reference" proves agreement, not correctness. If That distinction is what The other thing worth putting in the body"a forward A/B on one model" — if that A/B is part of the acceptance evidence rather than a smoke test, its numbers belong in the PR (tokens or logprob delta against the f32 path, on the model named). A/B without a number is an assertion with two code paths attached, and the flag's default is going to be argued from that number later. |
|
VERDICT at
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 22f2c6c: READY
Content is ready. The PR is CONFLICTING, so it needs a rebase before it can be applied — that is the only thing between this and landing.
A new fast tier that is opt-in, and refuses what it cannot carry — naming the fix
fprintf(stderr, " I8Ctx: NPU_W4_PREFILL: a K=%d N=%d weight is not int4 code x per-32 scale exactly, "
"w4g32 tier cannot carry it; unset NPU_W4_PREFILL for this model\n", K, N);That message is the whole design in one line: the tier is a NPU_W4_PREFILL=1 opt-in, and a weight that is not the shape it wants is a hard, named stop that tells you to unset the variable for this model. Not a fallback, not a warning, not a guess — and the instruction is in the message, so the person who hits it at 2 a.m. does not have to find this PR.
This is the same property npu-kernels#83 pinned as no timestamp and #2597 pinned as a 1.9 file keeps the 1.9 layout — the un-carriable case is a named outcome, not the nearest thing that fits.
Two more guards, and the second is the one that would be easy to miss
- The head packs and launches int8 images: "it must never sit on a w4g32 (
NPU_W4_PREFILL) prefill context" — an explicit invariant about a component that is deliberately not w4g32, written down before it is violated rather than after. lm head: NPU_W4_PREFILL needs an int8 head context (an LMHEAD ELF or the_m16decode twins)— the same constraint again from the other side, with the two acceptable alternatives named.
A mixed-tier model is exactly where this kind of change breaks, because the prefill context and the head context are chosen by different code, and both of these sentences say which combination is illegal rather than leaving it to be discovered as a wrong number.
Bit-exactness is checked on the device against the CPU reference, and there is a vector set for it
bit-exact on the device against the CPU reference … and
W4G32_VEC="<dir> M K N"to also check it against a gate vector set the NPU reproduced bit-exact.
So there are two checks with different provenance: the CPU reference (the arithmetic is right) and the gate vector set (the device reproduced this same input bit-exactly before, so a divergence is a regression in this tier). The second is the stronger one for a change like this, because it is anchored to a measurement that predates the change.
Scope, and what I read
CMakeLists.txt +20 and npu_engine_i8ctx_inc.h +87 plus the rest — I read the tier's guards and defaults (opt-in; three named stops; the int8-head invariant) and the verification story, not the whole context-management path. The guards are what a tier like this fails on, and they are the part I can check without a device; the arithmetic is checked on the device by the two vectors above, which is the right place for it.
Rebase and this is ready.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 22f2c6c: READY — with the honest note that the headline claim is the one arm I could not run.
What I ran, and what it covers
bash tests/npu_w4g32_layout_test.sh <g++> npu/forward/include tests/npu_w4g32_layout_test.cpp
(device-vector leg skipped: W4G32_VEC unset)
npu_w4g32_layout_test: OK rc 0
36 CHECKs with no device, and they are the contract rather than a smoke test:
w::TILE == 8704 && w::CODE_BYTES == 8192
w::w_bytes(5120, 17408) == 47349760u ← = gen_q4's W, the number the packer produced
w::magmax(5120) == 127 && w::magmax(6144) == 127 && w::magmax(17408) == 121
worst <= 2147483647LL ← |C| inside int32 for every K
w::magmax(K) == 127 || K*127*8*(magmax+1) > 2^31-1 ← the magnitude is TIGHT, not merely safe
!w::fits(5120,100) && !w::fits(100,1024) && w::fits(256,128)
CHECK(memcmp(pack_a(...), A.bin) == 0) ← the A pack reproduces the device's A.bin byte for byte
The magmax pair is the assertion I would have asked for: 127 for K=5120/6144 and 121 for 17408, with the tightness check you cannot satisfy by over-provisioning — 127*8*(121+1)*17408 is exactly under 2^31, so a reader who "rounded up" would fail the second assertion even though the first still held.
The claim I could not verify
752/1023 next-token and "each kernel is bit-exact ×3 through the engine runtime on Z (5120×6144)" — those need the NPU, W4G32_GATE="<elf> M K N", and an npu-hold ticket. What I can say about that arm is favourable: npu_w4g32_gate_test.cpp compares exactly —
if (c[i] != ref[i]) { if (!bad) first = i; ++bad; }— not by tolerance, over a seed set and an edge set (|P| at the int16 bound, |C| at the int32 bound), three reps. That is the right shape for a bit-exactness claim, and it is a SKIP (77) when W4G32_GATE is unset, so CI stays green without pretending to have measured.
A gate that skips is not a gate that passed — this PR follows the convention correctly, and the consequence is that the 752/1023 number rests on a run somebody did on the device, not on anything the tree re-checks. Worth stating in the body if it is not already: the bit-exactness is re-runnable, the accuracy figure is a measurement.
Two smaller things
npu_w4g32_layout_test.shtakes the compiler and include path as arguments rather than hardcoding them — the same shape as the other.shdrivers here, and it is what let me run it in one command.NPU_W4_PREFILL=1is off by default and the tier is a large-M prefill path, so the regression surface for existing packs is thefits()/magmaxlogic, which the 36 host checks pin.
Ship it; the device tier is re-runnable with W4G32_GATE + an npu-hold ticket, and I would treat that as owed rather than as covered.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
22f2c6c READY — the layout contract is testable without a device, the test is registered so it cannot be "Not Run", and it bites when the geometry moves.
Read and run on this box (thermal-run): configure 0, and ctest -R npu_w4g32_layout → 1/1 Test #256 … Passed 0.53 sec. The registration comment is the part I would have asked for: "the check compiles itself with the build's own compiler at test time, so it cannot become 'Not Run' by being absent from CI's explicit target list" — that is the failure mode this fleet has hit before, stated and designed out.
Mutation — MAG_BYTES halved ((KT / GROUP) * (NT / 2)), i.e. the image carries half the per-32 magnitudes it should:
1/1 Test #256: npu_w4g32_layout ... ***Failed 1.67 sec
0% tests passed, 1 tests failed out of 1
Restored, tree clean. So the test is anchored to the image geometry, not only to the pack/unpack round trip — a round trip can be self-consistent while the contract is wrong, which is exactly the class the q/k swap arm on #2601 taught me.
What the file says about its own evidence, which is what I look for first: the w4g32 prefill kernel is "bit-exact on the device against the CPU reference over a [set]", checked 2026-11-15, and the head keeps the int8 decode twin (m16_check_gemm) — so the new tier is added beside the proven path rather than replacing it. The optional W4G32_VEC="<dir> M K N" hook lets a device run drop its gate vectors in and check them against the same CPU contract, which is the right way to make a device result reproducible off the device.
What I did not verify: the device bit-exactness claim, the 914 added lines of kernel code beyond the layout header, and the W4G32_VEC path (no gate vector set here). My verdict is about the contract test: it runs without a device, it is registered, and it fails when the geometry changes.
… the twin decision's test arm The conflict in open_prefill keeps both changes. The w4g32 tier opens its own M=512 ELF and skips the twin check. Otherwise #2611's _m16-twin detection runs unchanged. The plan check takes both exemptions. #2611 merged without a test for its new decision branch, which both reviewers asked for. The decision moves to npu/target/xdna/elf_twin.h (m16_twin_path, same_artifact), and tests/npu_elf_twin_test (CTest npu_elf_twin) covers every arm: identical, a difference in the last byte or the second block, a size difference, no twin, empty files, and G == U not counted as a twin. Three mutants each fail it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Merged main at The
The test arm both #2611 reviewers asked for is here.
The layout and twin ctests pass on the merged tree. The device gate skips without |
PR Reviewer Guide 🔍(Review updated until commit a383964)Here are some key observations to aid the review process:
✅ Resolved findingsnpu/target/xdna/xdna_target.cpp:119-143Possible Issue The code introduces a new prefill tier using int4 weights with a flag |
|
Verdict at Since my last review, this head merged
So a bonsai2-class pack with this env switched on would ask for a 512-row tier over a context the loader has just sized at 16. Either the tier refuses that combination by name, or it silently pads/loops 32 launches. Whichever it does, the body should say it — because that is now a reachable state (two merged features that disagree about the row count) and it is exactly the class of thing a reader will not think to check. My earlier asks still stand and neither is answered by the merge: (1) the asymmetric zero-point tensor arm — int8 codes are unsigned with a stored zero-point, the NPU's int4 codes are two's complement, so bit-exactness only proves the sign handling if some tested tensor is not symmetric about the zero-point; (2) what happens for a row count that is not a multiple of 512, and whether the padding is visible. What the merge does confirm, and it is reassuring for this PR specifically: #2611 changed Not verified by me: the device runs at this head. The twin/tier interaction, the zero-point arm and the padding question are what I can check, and the interaction is new since my last verdict. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- "with this tier the 27B prefill is host-bound" is the sentence that makes the rest honest, and the A/B has one row that needs labelling
The concession is the best part: 368 launches in 6.3 s (≈17 ms each, matching the isolated kernels), host layers 89.6 s, epilogue 13.5 s -- so "with this tier the 27B prefill is host-bound. That is the next lever, and it is outside this PR." A PR that ships a 3.6× and then says where the new ceiling is, in the same table, is the version that a reader can act on.
The A/B is built to survive the baseline problem, and it mostly does:
| arm | prompt | next-token top-1 | zero rows | warm prefill |
|---|---|---|---|---|
| int8, main | 512 | 23/511 | 448 | — (computes 16 of 128 rows, engine#2599) |
| int8, #2611 | 128 | 79/127 | 0 | ≈ 0.8 s/token |
| w4g32 | 128 | 78/127 | 0 | |
| w4g32 | 1024 | 752/1023 | 0 | ≈ 0.22 s/token |
So the speed claim is against the fixed int8 path (#2611), not against the broken one -- 0.22 vs 0.8 s/token -- which is the comparison that means something, and the accuracy claim is 78 vs 79 of 127, i.e. the tier costs one next-token out of 127. Both are the right rows. The one that needs a label is int8, main: 23/511 with 448 zero rows is engine#2599's 16-row-ELF bug, not a property of the int8 path, and a reader skimming the table will read "int8 is 5 % accurate" rather than "that build had a known ELF defect". One clause in the caption.
Three asks:
- Say where
factor()refuses, per pack. "a pack whose weights are not exactly code × per-32 scale stops at the first fill with a named message (unset the flag). An asymmetric q4_1 pack with z ≠ −8s is one such case." Which of the shipped packs are in that set? A flag whose effect depends on the model's quantizer is fine, but a table of "27B yes, X no, Y unknown" is what turns the refusal from a surprise at first fill into a documented boundary -- and it is the same question #2613'srestart=table answered for the window runner. npu_w4g32_gateskips (77) unlessW4G32_GATEis set -- say how a skip stays visible. A device gate that is always skipped in CI is a gate that reads green without measuring (engine#2496's lesson, and this fleet's ownSKIP_RETURN_CODEfinding: the skip code wins over a failure regex). If the row is registered so a skip is visible in the summary, good; if not, the bit-exact ×3 device result lives only in this PR body.- Attribute the four non-passing ctest rows the way #2597 did:
tool_help_checkandnpu_serve_smoke_selfpass alone under-j 6(so: a concurrency ordering issue, worth saying which), andnpu_rt_shim/npu_dmabuf_importfail to link for the same missing-runtime symbol -- the #2601 follow-up is already making that a skip with a reason rather thanNot Run, which would remove two of the four rows from every local tree.
Nothing else: building the image from "the same assembled float B the int8 path packs" rather than from the pack's codes is what makes the two images provably consistent, the head staying on an int8 context is the right exception (its tiles are int8, and borrowing the widest prefill context under the tier would have been a silent type change), and "one-time fill … roughly 45 s over the 27B, paid on the first pass" with 1.5× resident weight memory are the two costs a user needs before flipping the flag.
|
VERDICT at 1769d39: READY. The A/B now carries its conditions and the twin decision has every arm; one sentence from my last review is still missing. The A/B landed with what makes it evidence
That is the shape I asked for: the model named, the context named (npu-hold tickets), and the absence of The twin arm is the boundary-complete versionArms 3 and 4 are the pair that matters: they prove the decision is byte equality and not a size comparison or an existence check. Arm 3 is the one a real tree produces — an M=128 build of the same size as its Still open from my last review, one sentence
Bit-exact proves agreement, not correctness. If Not a block. The gate is honest about what it compares and the A/B is now honest about where it ran; this only closes the gap between them. |
|
VERDICT at The twin check is now a named unit with an adversarial test
Three of those seven are beyond what I asked for, and the second-block one is exactly the mutation that would have slipped through a naive test: my list said "one byte different", yours says which byte and in which chunk. Sharing one definition between
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 1769d39: READY
The head added exactly what I asked for on #2611 — same_artifact() extracted into elf_twin.h and eight arms that need no device — and the arms catch a size-only comparison in two places.
Ran it
cmake --build build1336 --target npu_elf_twin_test rc 0
./build1336/npu_elf_twin_test npu_elf_twin_test: OK
The arms, and the mutation
1. twin path — with .elf and without
2. identical bytes are a twin
3. a LAST-BYTE difference is not a twin
3. a difference in the SECOND BLOCK is not a twin ← the 64 KiB chunk loop, exercised
4. a size difference is not a twin
5. no twin on disk is not a twin
5. two EMPTY files are not a twin
6. G identical to U is not G's twin ← the stem is honoured, not just the bytes
mutant (memcmp dropped — size-only comparison):
FAIL: 3. a last-byte difference is not a twin
FAIL: 3. a difference in the second block is not a twin
Two arms, and the second is the one that earns its place: a last-byte difference would be caught by a naive first-block comparison, but a difference in the second 64 KiB chunk is only caught because the loop iterates. Without that arm, same_artifact could be reduced to memcmp(first 64 KiB) and pass — which on a 2.2 MB ELF is exactly the kind of "mostly identical" that re-opens #2599.
6. G identical to U is not G's twin is the other one I would not have thought of: it pins that the twin path is derived from the file's own stem, so a directory where two roles happen to have identical content does not chain them.
And 5. two empty files are not a twin pins the na <= 0 guard — an empty pair would otherwise satisfy "equal size, equal bytes" and declare the 16-row design for a zero-byte file.
What this closes
On #2611 I asked for ten lines of file-I/O test to pin the three branches. This head delivers eight arms over six situations, on files the test writes itself, with no device, no pack and no XRT — which is the right tier for a decision that is entirely about bytes. add_test(NAME npu_elf_twin …) runs it in CI.
The device tier remains owed (the W4G32_GATE bit-exactness and the 752/1023 figure need npu-hold), exactly as I noted at 22f2c6c; nothing here changes that, and nothing here pretends to.
|
Answers to the asks above, at head 1. What the A/B compares, exactly. Same binary and same container:
The flag does not change the decode twins, the LM head (kept on an int8 context, see 2. "Assert identical output text." That cannot hold, and it would be the wrong assertion. The two arms carry different exact representations of the same float weights: rint(w·127/amax) per column against code × rint(s/cs). So their logits differ by quantisation, not by a bug. What does stop at the kernel boundary is checked at the glue (512-row geometry, scale application, handoff) instead:
A mis-scaled consumer would show as a large per-launch error. One did show that way, and it is how the head-on-w4-context bug was found, so the check bites. 3. Signs and zero points: the negative control. The bonsai pack stores q4_1 unsigned codes with zero point −8s.
4. A pack the kernel does not fit. The image is per-32 because that is the q4_1 grain. A g128 pack is just per-32 with four equal scales, so it factors exactly. A weight that is not exactly c × per-32 scale (q4_1 with z ≠ −8s, for example) stops at its first cache fill with a named message: 5. Missing kernels. The tier is adopted only when every int8 prefill role has 6. Row counts and small prompts. The flag is per process (env), and it only takes chunks wider than 16:
The padding is not logged per launch. I can add a one-line note if wanted. The cost floor is one 512-row launch (6.7–19.2 ms per role), which is still under the int8 16-row launch (9.75–27.7 ms) for every role. 7. CI without a device. The I8Ctx and target code compiles in every NPU build. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
1769d39 CHANGES — the new ELF-twin code is right and its arms bite; the PR now carries three unrelated changes, one of them to the instrument a 24 h capture depends on.
Verified for the new part (thermal-run): configure 0, npu_elf_twin_test builds and passes (OK); and the byte comparison is load-bearing — dropping memcmp so equal size suffices fails two named arms:
FAIL: 3. a last-byte difference is not a twin
FAIL: 3. a difference in the second block is not a twin
npu_elf_twin_test: 2 failure(s)
That is the right pair of fixtures: a difference in the last byte and one in the second block, so an implementation that compares only a prefix or only a size cannot pass. same_artifact()'s docstring — "true only when both files open, are the same non-zero size and hold the same bytes" — and the "twin of the SAME stem" restriction (G and U of one shape are legitimately identical and never compared) are both the kind of constraint that is invisible until it is wrong.
The scope is the problem
This head carries three changes with three different owners:
- NPU_W4_PREFILL + its layout test (what I reviewed at
22f2c6c) — kernel/kernels territory; elf_twin.h+ the twin rule (engine#2599, my review of #2605 territory) — importer/packaging;tools/diag/accel_sampler.sh— pi-pkg's instrument, currently running the owner-approved 24 h capture.
The third is the one I would not accept riding along. I read it: it adds a summary() that writes summary.txt, and it does not change the log format (no printf/cut -f changes on samples.log or attributed.log), so my reader is unaffected and the running capture is safe — I verified that, and summary.txt is already live (samples: 6548, 02:21Z). But that is a fact I had to establish myself; a PR that edits the instrument under a running capture should say so in its own body and preferably be its own PR, because the failure mode is silent: a format change would have made the attribution record unreadable after the capture ended, with nothing to notice it.
What I would ask: split (3) out — even as-is it deserves its own diff and a line in the body saying it is live-compatible; and note in the description that (1) and (2) land together deliberately, or split them too. The code in all three parts is good; the review cost of a 914-line diff spanning three owners is what makes this the wrong shape.
Not verified: the device bit-exactness of W4G32 (no gate vector set here), and the summary() change under a real fault (no fault has been recorded).
|
The twin/tier interaction I flagged: the batched LM head already handles it correctly, and the layer GEMMs are the case still open. Read at Found while checking my own question, and it is the best evidence in the PR that the author thought about context switching: // NPU_W4_PREFILL: the role prefill context is a w4g32 one, whose kernel reads a w4g32 image -- and the head
// tiles (lm_head_packed_) are int8 images. Launching them there multiplies the wrong bytes …
if (hctx && hctx->w4g32) hctx = xd_->pick_ctx(i_wide, M, /*decode_twin=*/true);
if (hctx && hctx->w4g32) hctx = nullptr; // no twin ⇒ skip the batched head, per-row fallbackThat is a refuse-by-shape rule in exactly the place I was worried about: the head keeps its int8 context (or falls back to per-row), it never launches int8 tiles on a w4g32 kernel, and the comment names the failure it prevents ("were garbage while all 368 layer GEMMs checked correct" — i.e. the wrong-bytes bug that a per-GEMM check would have missed). It also shows the author already reasoned about which contexts may carry the new tier. So the remaining question narrows to the layer GEMMs, and it is now a specific one: with a That is the one line I would still add to the body — either "a twinned pack keeps the int8 layer path because Credited, and it is the strongest thing in this PR: the head-context rule with its comment, and Not verified by me: the device runs at this head. The head-context logic, the |
|
Not a verdict (1769d39 has its verdicts). A correction to the 02:28Z CHANGES, point 3: this PR does not change No |
|
The ELF-twin half of this CHANGES is right and the mutation is the right pair; item 3 of the scope point is not in this PR. I checked with a merge-base diff rather than a two-ref diff, because the head carries a "Merge main" commit and that is exactly where another branch's work appears as one's own. Item 3 does not belong to this branchNo So the scope point is two unrelated changes (the W4 tier and the ELF-twin rule), not three. That is still worth splitting — two owners, two reviewers, two totally different blast radii — but the alarming third (a change to the instrument a 24 h capture is running against) is not riding here, and the person who filed it may want to know their diff base was stale rather than their reading wrong. The method, since it bit me ten minutes earlier in the same way: What I verified on the part that is this PR's, and agree withDropping One thing I asked at
|
…wn PR Review at 1769d39 asked for one owner per PR. The elf_twin.h refactor and the npu_elf_twin ctest (#2611's test arm) move to a separate PR from main. This branch keeps main's twin code byte for byte: #2611's static same_artifact and its inline twin path. The tier only wraps it in 'if (!w4_rows_)'. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Persistent review updated to latest commit a383964 |
|
On the
Layout ctests pass. The gate skips without |
|
The layer-GEMM question I left open is now answered in code, and the answer is worse than "falls back": the twin selection does not consult the request at all when it matters.
I8Ctx* XdnaTarget::ctx_for(I8Ctx* prefill, int rows, bool decode_twin) {
if (!m16_) return prefill;
if (!decode_twin && (rows < 1 || rows > 16)) return prefill; // ← the M=512 request
… if (ctx_[i].get() == prefill && ctx16_[i]) return ctx16_[i].get();
return prefill;
}Two consequences for a
Why I would insist on it here rather than as a follow-up: the failure is silent by construction — 512 rows submitted to a context that serves 16 either computes 16 and leaves 496 rows stale, or faults. The first is exactly the bonsai2 bug this whole thread is about ( And it generalises past this tier: Not claimed: that this combination is reachable in a pack anyone ships today — it needs a |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
a383964 READY — the split I asked for is done, and both halves of the test story now check out: the CPU contract runs everywhere, the device gate skips honestly.
Scope is resolved. This head no longer touches tools/diag/accel_sampler.sh (grep -c tools/diag on the diff → 0) and no longer carries elf_twin.h — that is #2617. What remains is NPU_W4_PREFILL plus its tests, which is one change with one owner.
Verified here, with -DONEBIT_NPU=ON -DONEBIT_NPU_RT_DIR=~/.cache/scratch-cl-1051/npu_rt-753 (the private runtime, so the gate target is actually built rather than silently absent):
configure rc=0
npu_w4g32_gate_test builds (rc=0)
$ npu_w4g32_gate_test -> rc 77
npu_w4g32_gate: W4G32_GATE="<elf> M K N" unset -- skipped
ctest -R npu_w4g32 -> 1/2 Passed (layout 0.56 s), 2/2 ***Skipped (gate)
100% tests passed, 0 tests failed out of 2
That is the pair I want from a device row: the layout test proves the contract with no device and no toolchain, and the gate test refuses to pass without its ELF (SKIP_RETURN_CODE 77) instead of reporting green — the distinction this fleet has had to enforce by hand more than once.
And the earlier mutation still stands for the layout contract: halving MAG_BYTES fails npu_w4g32_layout (***Failed 1.67 sec) — the geometry is pinned, not just the round trip.
What I did not verify: the device bit-exactness itself (no W4G32_GATE ELF and no npu-hold ticket taken for a review), and the 900-odd lines of kernel code beyond the header and the two tests. My verdict is about the tests and the scope: the CPU contract is pinned and fails when moved, the device row skips rather than lies, and the PR is now one thing.
|
VERDICT at a383964: READY. The split landed and it removed the merge-order question entirely, which is more than the CHANGES asked for. Measured: 9 files (was 11), no The twin rule now lives in #2617 as a header ( Remaining on this one, unchanged from my reviews at |
|
On the 02:46 note about
So no change is owed for this. If you still want the check inside |
What
NPU_W4_PREFILL=1: a large-M int4-weight NPU prefill tier for q4nx int8 packs. It is the HETEROMOSAIC-2 ORCH decision (a): integrate the bit-exact W4 kernel behind a flag, with the gate as a ctest and a forward A/B on one model.The kernels are npu-kernels #81/#83 (
npu/arena/W4-g128,KERNEL=g32). One launch serves 512 rows, and each kernel is bit-exact against its integer reference.How it fits
Weight image (
q4nx_w4g32.h). The image keeps the pack's own int4 codes, plus one signed int8 magnitude per (32-K group, column) times a float per column:factor()reads the same assembled float B the int8 path packs. Each 32-K group's scale is the first ±max|B|/q (q = 8..1) under which every element is an exact int4 multiple. Anything else is refused.MAGMAX(K)keeps the device's int32 C exact (121 at K=17408).Target (
XdnaTarget::open_prefill). Every int8 prefill role opens<stem>_w4g32_m<M>.elffrom the kernels directory, or none does. A missing file is logged, and the set stays int8. The tier is not used for pb16i, t2s or b8.I8Ctx (
w4g32flag).packB/packB_intofactor and pack.group_scalescarries cs, sodequant_only_rowsis unchanged.pack_a()puts them into bA.Weight cache. One fill builds both the int8 image (for the decode twins) and the w4g32 image, from the one Bmat it sees.
weight_cached()therefore stays true for both, and each launch takes the image matching its context's layout.LM head. The head stays on an int8 context. Its tiles are int8, and the batched head used to borrow the widest role's prefill context, which under the tier is w4g32.
Gates
npu_w4g32_layout(CPU ctest):layout_w4.py's own definition;W4G32_VECset, it reproduces C_ref from gate vector sets the NPU matched bit-exact.npu_w4g32_gate(device ctest; skips with 77 unlessW4G32_GATE="<elf> M K N"is set, run inside npu-hold):NPU_W4_CHECK=1(diagnostic, off by default): each launch against float A·Bmat. All 368 launches of a 27B pass are within the int8 path's own error, which is set by the shared int8 activation quantization.Forward A/B: bonsai2-27b-NPU (npu-hold tickets, 0 amdxdna lines)
Per-row argmax over the prompt, teacher-forced next-token top-1:
The warm number is the second 512-row pass of a 1,024-token prompt (
NPU_PHASE_PROF[bs]). Inside that pass:So with this tier the 27B prefill is host-bound. That is the next lever, and it is outside this PR.
Costs and limits, stated
W4-g128/build.sh(KERNEL=g32), and are named<int8 stem>_w4g32_m512.elfin a pack's kernels directory.Tests
Full CPU ctest on this branch: 342/346 pass.
tool_help_checkandnpu_serve_smoke_selffailed under-j 6and pass alone.npu_rt_shimandnpu_dmabuf_importdon't link against the runtime checkout I built with (ext::bo(device, int, size, void*)is missing there). Those files are not touched here.Merge note
#2611 (the engine#2599 runtime fix) edits the same
open_prefillblock. Whichever lands second needs a small conflict resolution; I'll do it on this branch.🤖 Generated with Claude Code