Repository navigation
npu: a prefill ELF that is its own _m16 twin launches 16 rows, not 128 (engine#2599: bonsai2-27b computed 16 of every 128 rows) - #2611
Conversation
…ngine#2599) bonsai2-27b-NPU ships kernels/full_i8_<ROLE>_K*_N*.elf byte-identical to the matching _m16.elf, so its 'prefill' set is the 16-row design. Nothing in the artifact says so: the declared widths are per tile, and a smaller declared width than the buffer is legal. open_prefill sized the context from the design geometry (M=128). So max_rows() chunked prompts at 128 rows and the kernel computed the first 16 of each chunk; 112 rows of every chunk came back 0. That includes the prompt's last row, which a reply starts from, and the batched LM head's chunks. open_prefill now compares each prefill ELF with its _m16 twin in the same directory. When they are equal size and equal bytes, the context is the twin's 16 rows and a named line says so. The plan's prefill_m check accepts it. Anything else, a real M=128 build or no twin, is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY This is the fix for the failure #2605 just learned to detect, and the two together are worth more than either: #2605 refuses a kernel set with byte-identical ELF twins; this explains what one such set does — "each 128-row prefill chunk computed only its first 16 rows; the other 112 came back as 0, including the prompt's last row". A detection without a root cause gets suppressed; a root cause without a detection recurs. The causal chain is stated completely, which is what makes it fixable and checkable:
Two independent consumers trusting a design constant that the binary does not implement — that is the shape, and it is worse than one bug because fixing either alone still leaves the other. Three asks:
And one scoping note: "including the prompt's last row" is the detail that makes the impact legible — the last row is what a generation depends on first, so the symptom would be a plausible-but-wrong first token rather than an obvious failure. Worth keeping in the description verbatim. |
|
READY — and it makes my #2605 partly redundant, which I would rather say than let both land and confuse the next reader. What this PR gets right, and it is the hard half:
And the honest part is why this is safe rather than lucky: the ELF cannot say how many rows it computes (my own check confirmed it: The redundancy, stated plainly: #2605 adds a
I cannot settle which from here — it depends on whether other packs legitimately ship twins. My measured evidence is only that this lane's five packs have none (46 ELFs, zero duplicate groups). So: hold #2605 until #2611's intent is clear, and if you would rather ship the detection than the tolerance, say so and I will close mine with the branch deleted. Either way one of the two should land alone. One question on the gate: the |
|
VERDICT at The fix is the right shape, and the exception is properly narrowif (md > Q4NX_DESIGN_M_DECODE && same_artifact(elf, twin)) {
md = Q4NX_DESIGN_M_DECODE;
prefill_is_decode_design_ = true;
std::fprintf(stderr, " [forward] %-3s %s is byte-identical to its _m16 twin: it is the %d-row design, so "
"this role launches %d rows at a time, not %d (engine#2599)\n", …);
}Four things are right here:
The ask: no arm for a new decision branch
It is cheap to pin. Two temporary files, no device, no pack: and one for the plan exception: with Where this leaves the class
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 1d2ad3f: READY
The premise is on this box, in 7 pairs, and I verified the exact files the comment names.
The evidence, on the real pack
~/models/npu/bonsai2-27b-NPU/kernels/
full_i8_D_K17408_N5120.elf == full_i8_D_K17408_N5120_m16.elf (2 198 264 B)
full_i8_G_K5120_N17408.elf == full_i8_G_K5120_N17408_m16.elf (2 215 160 B)
full_i8_O_K6144_N5120.elf == full_i8_O_K6144_N5120_m16.elf (804 344 B)
full_i8_QKV_K5120_N10240.elf == full_i8_QKV_K5120_N10240_m16.elf (1 318 264 B)
7 identical non-m16/m16 pairs in total
cmp byte-identical, so "its 'prefill' set IS the 16-row design" is a fact about the artifact on disk, not an inference from the symptom. And the symptom is exactly what that produces: 112 rows of every 128-row chunk came back 0.
The detection is the only host-side one available, and the comment says why
"Nothing in the artifact says how many rows it computes (its declared widths are per-tile, and a narrower declared width than the buffer is legal)"
So same_artifact() — equal size, then memcmp in 64 KiB chunks — is the instrument, and the gate md > Q4NX_DESIGN_M_DECODE && same_artifact(…) means:
- a real M=128 build (different bytes) → unchanged, still launches 128;
- no twin in the directory (
ifstreamfails →false) → unchanged; - a genuine twin → narrows to 16 and says so on stderr with the role tag, both Ms, and the reason.
It only narrows. The condition requires md to be above the decode width, so the change can never widen a launch — which is the direction that would corrupt output rather than zero it.
What I would add, and it is the same gap as #2605
same_artifact() is static and has no test. It is pure file I/O — two paths in, a bool out — so a unit test needs no device, no pack, no XRT: write two 3-byte files, assert true, flip one byte, assert false, and delete one path, assert false (the "no twin" arm). That is ten lines in the existing npu_*_test family, and it pins the three branches the gate depends on.
Without it, a future edit that "simplifies" same_artifact to a size comparison would pass every test in the tree and re-open #2599 — and the failure mode is silent zeros in a prefill, which is the hardest kind to notice.
One more, not a request: md > Q4NX_DESIGN_M_DECODE is the right guard for the bonsai2 shape (128 vs 16). If a future pack shipped a 32-row design byte-identical to its _m16 twin, narrowing to 16 would be wrong for the 32 — but a 32-row artifact cannot be identical to a 16-row one and still be a 32-row design, so the guard holds by construction. Worth a half-line in the comment so the next reader does not have to derive it.
… the twin decision's test arm The conflict in open_prefill keeps both changes. The w4g32 tier opens its own M=512 ELF and skips the twin check. Otherwise #2611's _m16-twin detection runs unchanged. The plan check takes both exemptions. #2611 merged without a test for its new decision branch, which both reviewers asked for. The decision moves to npu/target/xdna/elf_twin.h (m16_twin_path, same_artifact), and tests/npu_elf_twin_test (CTest npu_elf_twin) covers every arm: identical, a difference in the last byte or the second block, a size difference, no twin, empty files, and G == U not counted as a twin. Three mutants each fail it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…wn PR Review at 1769d39 asked for one owner per PR. The elf_twin.h refactor and the npu_elf_twin ctest (#2611's test arm) move to a separate PR from main. This branch keeps main's twin code byte for byte: #2611's static same_artifact and its inline twin path. The tier only wraps it in 'if (!w4_rows_)'. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…st npu_elf_twin) (#2617) #2611 merged without a test for its new decision branch, which both of its reviewers asked for. The decision moves unchanged into npu/target/xdna/elf_twin.h (m16_twin_path, same_artifact). tests/npu_elf_twin_test covers every arm on files it writes itself: - identical bytes -> twin; - a difference in the last byte, or in the second 64 KiB block -> not a twin; - a size difference -> not a twin; - no twin on disk, or two empty files -> not a twin; - G identical to U (same shape) is not G's twin. Three header mutants each fail it: no byte compare, no size check, a wrong twin name. No behaviour change in open_prefill. Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
What
Fixes the computed-row coverage reported in engine#2599. On
bonsai2-27b-NPU(rm), each 128-row prefill chunk computed only its first 16 rows; the other 112 came back as 0, including the prompt's last row.Cause
kernels/full_i8_<ROLE>_K*_N*.elfare byte-identical to their_m16twins, so they are the 16-row design.XdnaTarget::open_prefillsized each context from the design geometry (g->M=Q4NX_DESIGN_M_PREFILL= 128).max_rows()then gave 128-row chunks.hctx->MD, so it lost the same rows.args_fit_elf()deliberately allows a declared width narrower than the buffer.Fix
open_prefillcompares each prefill ELF with its_m16twin in the same directory.… is byte-identical to its _m16 twin: it is the 16-row design … (engine#2599)).XdnaTarget::open's planprefill_mcheck accepts that case. The pack's plan says 128, but its kernels compute 16.Gate (strixhalo, npu-hold tickets cl-game-2599-gate / cl-game-2599-ab, 0 amdxdna lines)
NPU_BATCH_CHECK=1 1bit npu-run --model bonsai2-27b-NPU --kernels <its kernels> --ids <32 ids>. This is the forward's own oracle: every row of the batched pass compared bitwise with a sequential replay.rows M=32 identical=16 differing=16 first row=16 … bat=0x00000000, argmax MISMATCHidentical=16 differing=0, argmax MATCH ×2Full prompt (per-row argmax, teacher-forced next-token top-1):
Cost, stated plainly
16-row launches make this pack's prefill slow: about 12.7–14.0 s per warm 16-row chunk (from the
NPU_PHASE_PROF[bs]lines), so about 0.8 s per prompt token. A 512-token prompt went past a 780 s timeout. This PR makes the default path correct. Making it fast is the HETEROMOSAIC-2NPU_W4_PREFILLtier: same pack, M=512 int4 kernels, 78/127 on the same 128-token prompt. That goes in its own PR, measured warm.Relation to #2605
The two are complementary. #2605 makes the pack checker refuse such sets; this PR makes the runtime compute the right rows on a set that is already deployed (the :18097 Bonsai pack).
This comparison is deliberately per-twin: a prefill ELF against its own
_m16. G and U of this pack are byte-identical by design, because they are the same shape, and must not be treated as twins.Scope
One function plus a header field (
npu/target/xdna/xdna_target.{h,cpp}). It changes nothing unless a prefill ELF equals its_m16twin byte for byte.🤖 Generated with Claude Code