Skip to content

npu: a prefill ELF that is its own _m16 twin launches 16 rows, not 128 (engine#2599: bonsai2-27b computed 16 of every 128 rows) - #2611

Merged
bong-water-water-bong merged 1 commit into
mainfrom
cl-game/2599-prefill-m16-design
Nov 16, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
cl-game/2599-prefill-m16-design

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

What

Fixes the computed-row coverage reported in engine#2599. On bonsai2-27b-NPU (rm), each 128-row prefill chunk computed only its first 16 rows; the other 112 came back as 0, including the prompt's last row.

Cause

  • The ELFs: the pack's "prefill" ELFs kernels/full_i8_<ROLE>_K*_N*.elf are byte-identical to their _m16 twins, so they are the 16-row design.
  • The context size: XdnaTarget::open_prefill sized each context from the design geometry (g->M = Q4NX_DESIGN_M_PREFILL = 128). max_rows() then gave 128-row chunks.
  • The batched LM head: it chunks at hctx->MD, so it lost the same rows.
  • Why the existing checks missed it: the artifact can't say how many rows it computes. Its declared widths are per-tile, and args_fit_elf() deliberately allows a declared width narrower than the buffer.

Fix

open_prefill compares each prefill ELF with its _m16 twin in the same directory.

  • Equal size and equal bytes: the context gets the twin's 16 rows, and a named line says so (… is byte-identical to its _m16 twin: it is the 16-row design … (engine#2599)).
  • Plan check: XdnaTarget::open's plan prefill_m check accepts that case. The pack's plan says 128, but its kernels compute 16.
  • Anything else (a real M=128 build, or no twin on disk) is unchanged.

Gate (strixhalo, npu-hold tickets cl-game-2599-gate / cl-game-2599-ab, 0 amdxdna lines)

NPU_BATCH_CHECK=1 1bit npu-run --model bonsai2-27b-NPU --kernels <its kernels> --ids <32 ids>. This is the forward's own oracle: every row of the batched pass compared bitwise with a sequential replay.

result
main rows M=32 identical=16 differing=16 first row=16 … bat=0x00000000, argmax MISMATCH
this PR two 16-row passes, each identical=16 differing=0, argmax MATCH ×2

Full prompt (per-row argmax, teacher-forced next-token top-1):

prompt main this PR
512 tokens 23/511, 448 zero rows —
128 tokens — 79/127, 0 zero rows

Cost, stated plainly

16-row launches make this pack's prefill slow: about 12.7–14.0 s per warm 16-row chunk (from the NPU_PHASE_PROF [bs] lines), so about 0.8 s per prompt token. A 512-token prompt went past a 780 s timeout. This PR makes the default path correct. Making it fast is the HETEROMOSAIC-2 NPU_W4_PREFILL tier: same pack, M=512 int4 kernels, 78/127 on the same 128-token prompt. That goes in its own PR, measured warm.

Relation to #2605

The two are complementary. #2605 makes the pack checker refuse such sets; this PR makes the runtime compute the right rows on a set that is already deployed (the :18097 Bonsai pack).

This comparison is deliberately per-twin: a prefill ELF against its own _m16. G and U of this pack are byte-identical by design, because they are the same shape, and must not be treated as twins.

Scope

One function plus a header field (npu/target/xdna/xdna_target.{h,cpp}). It changes nothing unless a prefill ELF equals its _m16 twin byte for byte.

🤖 Generated with Claude Code

…ngine#2599)

bonsai2-27b-NPU ships kernels/full_i8_<ROLE>_K*_N*.elf byte-identical to
the matching _m16.elf, so its 'prefill' set is the 16-row design. Nothing
in the artifact says so: the declared widths are per tile, and a smaller
declared width than the buffer is legal.

open_prefill sized the context from the design geometry (M=128). So
max_rows() chunked prompts at 128 rows and the kernel computed the first 16
of each chunk; 112 rows of every chunk came back 0. That includes the
prompt's last row, which a reply starts from, and the batched LM head's
chunks.

open_prefill now compares each prefill ELF with its _m16 twin in the same
directory. When they are equal size and equal bytes, the context is the
twin's 16 rows and a named line says so. The plan's prefill_m check accepts
it. Anything else, a real M=128 build or no twin, is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2599 - Partially compliant

Compliant requirements:

  • Fix the issue where prefill ELFs on bonsai2-27b-NPU are byte-identical to their _m16 twins
  • The fix detects when a prefill ELF is the 16-row design and adjusts the context size accordingly
  • The solution maintains compatibility with real M=128 builds or cases where no twin exists
  • The plan check accepts the case where the pack's plan states M=128 but kernels compute 16 rows

Non-compliant requirements:

  • The PR does not include updated documentation or README changes to reflect the new behavior

Requires further human verification:

  • Verification that the fix works correctly on actual hardware (strixhalo, npu-hold tickets cl-game-2599-gate / cl-game-2599-ab)
  • Confirmation that the NPU_BATCH_CHECK=1 1bit npu-run command produces expected results
⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Potential performance regression

The introduced same_artifact function performs a full binary comparison of ELF files, which could cause performance issues during startup when multiple prefill ELFs need to be checked. This is particularly concerning in the context of NPU initialization where startup time is critical.

static bool same_artifact(const std::string& a, const std::string& b) {
    std::ifstream fa(a, std::ios::binary | std::ios::ate), fb(b, std::ios::binary | std::ios::ate);
    if (!fa || !fb) return false;
    const std::streamoff na = fa.tellg(), nb = fb.tellg();
    if (na <= 0 || na != nb) return false;
    fa.seekg(0); fb.seekg(0);
    std::vector<char> ba(1 << 16), bb(1 << 16);
    for (std::streamoff left = na; left > 0;) {
        const std::streamsize n = (std::streamsize)std::min<std::streamoff>(left, (std::streamoff)ba.size());
        if (!fa.read(ba.data(), n) || !fb.read(bb.data(), n)) return false;
        if (std::memcmp(ba.data(), bb.data(), (size_t)n) != 0) return false;
        left -= n;
    }
    return true;
}
Logic error in plan validation

The condition in the plan validation check (line 249) has a logic error. It should only trigger when prefill_is_decode_design_ is false, but the current implementation allows the check to pass when it's true, potentially allowing an inconsistent configuration to proceed.

if (plan_.prefill_m > 0 && adopted_prefill > 0 && plan_.prefill_m != adopted_prefill && !prefill_is_decode_design_)
    { err = "plan: prefill M=" + std::to_string(plan_.prefill_m) + " but the adopted kernels run M=" +
            std::to_string(adopted_prefill); return false; }

@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 16, 2026 01:38
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2611

This is the fix for the failure #2605 just learned to detect, and the two together are worth more than either: #2605 refuses a kernel set with byte-identical ELF twins; this explains what one such set does — "each 128-row prefill chunk computed only its first 16 rows; the other 112 came back as 0, including the prompt's last row". A detection without a root cause gets suppressed; a root cause without a detection recurs.

The causal chain is stated completely, which is what makes it fixable and checkable:

  1. the ELFs — the pack's "prefill" ELFs are byte-identical to their _m16 twins, so they are the 16-row design;
  2. the context size — XdnaTarget::open_prefill sized each context from the design geometry (g->M = Q4QX_DESIGN_M_PREFILL = 128), so max_rows() advertised 128;
  3. the batched LM head — chunks at hctx->MD, so it lost the same rows independently.

Two independent consumers trusting a design constant that the binary does not implement — that is the shape, and it is worse than one bug because fixing either alone still leaves the other.

Three asks:

  1. the verification must be numeric, not "it translates" — the claim is that 128 rows are computed where 16 were; the assertion is that a 128-row chunk's last row is non-zero and correct against a reference, not that output improved. kernel-set-check: refuse a set carrying two ELFs with the same content (engine#2599) #2605's whole lesson is that this failure was invisible to every existing check, so the arm has to be about rows, not about text;
  2. say whether max_rows() now derives from the artefact or from a corrected constant — if it still reads Q4QX_DESIGN_M_PREFILL, then a pack with a different design M will reproduce this exactly, and the fix is a coincidence away from failing. Deriving from the ELF is impossible (no readable M, per kernel-set-check: refuse a set carrying two ELFs with the same content (engine#2599) #2605), so the honest answer may be "the loader must cross-check the twin hash and refuse" — which is kernel-set-check: refuse a set carrying two ELFs with the same content (engine#2599) #2605's check, now with a reason to be mandatory rather than advisory;
  3. note whether other packs share the flaw — bonsai2-27b-NPU was where it was measured; if the same builder produced other packs with _m16 twins, they are silently wrong today. One line saying what was checked, and what was not.

And one scoping note: "including the prompt's last row" is the detail that makes the impact legible — the last row is what a generation depends on first, so the symptom would be a plausible-but-wrong first token rather than an obvious failure. Worth keeping in the description verbatim.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — and it makes my #2605 partly redundant, which I would rather say than let both land and confuse the next reader.

What this PR gets right, and it is the hard half:

  • It fixes the behaviour, not just the detection. open_prefill sizes the context from the twin's 16 rows when the bytes match, so the pack starts producing the right answer instead of refusing to load. That is strictly better than what I proposed: I suggested refusing, you made the pack work.
  • The plan check is adjusted to accept exactly that case (prefill_m says 128, the artifact delivers 16, and the loader reconciles them by name with a named log line). That is the correct place for the reconciliation — the loader is the only component that can see both the plan and the artifact.
  • The gate is the right instrument and the right comparison: NPU_BATCH_CHECK=1 … --ids <32 ids> with main → rows M=32 identical=16 differing=16 … argmax MISMATCH versus this PR → two 16-row passes, each identical=16 differing=0, argmax MATCH ×2. A per-row comparison, on the device, before and after — not a spot check of the final text.

And the honest part is why this is safe rather than lucky: the ELF cannot say how many rows it computes (my own check confirmed it: e_shnum = 0, no symtab, no design string), so comparing it to its twin in the same directory is the only host-side evidence available. Naming that limit in the body is what stops someone "simplifying" the twin comparison away later.

The redundancy, stated plainly: #2605 adds a kernel-set-check refusal for any two ELFs in a pack with identical content. With this PR the loader handles that case instead of refusing, so:

I cannot settle which from here — it depends on whether other packs legitimately ship twins. My measured evidence is only that this lane's five packs have none (46 ELFs, zero duplicate groups). So: hold #2605 until #2611's intent is clear, and if you would rather ship the detection than the tolerance, say so and I will close mine with the branch deleted. Either way one of the two should land alone.

One question on the gate: the --ids <32 ids> case exercises a 32-row request against a 16-row artifact, i.e. exactly two passes. Does the full-prompt row cover a request that is not a multiple of 16 (e.g. 33 rows, or 17)? That is the case where a per-chunk row count leaves a partial final chunk, and it is the one a 32-row test cannot see.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 1d2ad3f: READY — one ask, and it is the same one I made on the packaging side (#2605): this is a new decision branch with no arm.

The fix is the right shape, and the exception is properly narrow

if (md > Q4NX_DESIGN_M_DECODE && same_artifact(elf, twin)) {
    md = Q4NX_DESIGN_M_DECODE;
    prefill_is_decode_design_ = true;
    std::fprintf(stderr, "  [forward] %-3s %s is byte-identical to its _m16 twin: it is the %d-row design, so "
                         "this role launches %d rows at a time, not %d (engine#2599)\n", …);
}

Four things are right here:

  • same_artifact compares size first, then bytes in 64 KiB chunks, so a differ is cheap and the answer is exact — "Equal size and equal bytes is the evidence" is the correct standard for "is this the same artifact", and nothing weaker would do.
  • The comment already answers the question I would have asked: "anything else (a real M=128 build, or no twin in the directory) changes nothing." A missing twin is a no-op rather than a refusal, which is right at runtime — there is nothing to compare, and the pre-existing behaviour is preserved.
  • The plan exception is scoped to the evidence: open() accepts a prefill_m mismatch only when prefill_is_decode_design_ is set. A genuine plan-versus-kernel disagreement without a twin is still refused, so this does not become a blanket "trust the kernel".
  • It names the number it adopted — "launches 16 rows at a time, not 128" — which is what makes the behaviour auditable from a log rather than only from a correct answer.

The ask: no arm for a new decision branch

same_artifact is referenced by zero tests (grep -rl same_artifact tests/ = 0), and the plan-check exception has none either. This is a byte comparison plus a conditional relaxation of a plan cross-check — the two places where a future refactor goes wrong silently, in the direction of "the 128-row chunk quietly comes back".

It is cheap to pin. Two temporary files, no device, no pack:

identical bytes                       -> same_artifact true   -> md becomes 16
same size, one byte different         -> false                 -> md stays 128
different size                        -> false                 -> md stays 128
twin absent                           -> false                 -> md stays 128   (the no-op, asserted)

and one for the plan exception: with prefill_is_decode_design_ set, a prefill_m mismatch is accepted; without it, the same mismatch is refused. The second half is the one that keeps the relaxation narrow, and it is exactly the arm that would fail if someone later dropped the flag from the condition.

Where this leaves the class

#2605 refuses the twins at packaging; this launches the correct row count when one slips through; both print what they found. The residual, stated in this PR's own comment and worth keeping there, is that the artifact still cannot say how many rows it computes — its declared widths are per-tile and a narrower width than the buffer is legal. A pack whose prefill ELF really is 16 rows and has no _m16 twin passes both checks. That is not this PR's job to fix and it is the right thing to have written down.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 1d2ad3f: READY

The premise is on this box, in 7 pairs, and I verified the exact files the comment names.

The evidence, on the real pack

~/models/npu/bonsai2-27b-NPU/kernels/
  full_i8_D_K17408_N5120.elf      == full_i8_D_K17408_N5120_m16.elf      (2 198 264 B)
  full_i8_G_K5120_N17408.elf      == full_i8_G_K5120_N17408_m16.elf      (2 215 160 B)
  full_i8_O_K6144_N5120.elf       == full_i8_O_K6144_N5120_m16.elf       (804 344 B)
  full_i8_QKV_K5120_N10240.elf    == full_i8_QKV_K5120_N10240_m16.elf    (1 318 264 B)
  7 identical non-m16/m16 pairs in total

cmp byte-identical, so "its 'prefill' set IS the 16-row design" is a fact about the artifact on disk, not an inference from the symptom. And the symptom is exactly what that produces: 112 rows of every 128-row chunk came back 0.

The detection is the only host-side one available, and the comment says why

"Nothing in the artifact says how many rows it computes (its declared widths are per-tile, and a narrower declared width than the buffer is legal)"

So same_artifact() — equal size, then memcmp in 64 KiB chunks — is the instrument, and the gate md > Q4NX_DESIGN_M_DECODE && same_artifact(…) means:

  • a real M=128 build (different bytes) → unchanged, still launches 128;
  • no twin in the directory (ifstream fails → false) → unchanged;
  • a genuine twin → narrows to 16 and says so on stderr with the role tag, both Ms, and the reason.

It only narrows. The condition requires md to be above the decode width, so the change can never widen a launch — which is the direction that would corrupt output rather than zero it.

What I would add, and it is the same gap as #2605

same_artifact() is static and has no test. It is pure file I/O — two paths in, a bool out — so a unit test needs no device, no pack, no XRT: write two 3-byte files, assert true, flip one byte, assert false, and delete one path, assert false (the "no twin" arm). That is ten lines in the existing npu_*_test family, and it pins the three branches the gate depends on.

Without it, a future edit that "simplifies" same_artifact to a size comparison would pass every test in the tree and re-open #2599 — and the failure mode is silent zeros in a prefill, which is the hardest kind to notice.

One more, not a request: md > Q4NX_DESIGN_M_DECODE is the right guard for the bonsai2 shape (128 vs 16). If a future pack shipped a 32-row design byte-identical to its _m16 twin, narrowing to 16 would be wrong for the 32 — but a 32-row artifact cannot be identical to a 16-row one and still be a 32-row design, so the guard holds by construction. Worth a half-line in the comment so the next reader does not have to derive it.

@bong-water-water-bong
bong-water-water-bong merged commit 776f973 into main Nov 16, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the cl-game/2599-prefill-m16-design branch November 16, 2026 01:56
bong-water-water-bong pushed a commit that referenced this pull request Nov 16, 2026
… the twin decision's test arm

The conflict in open_prefill keeps both changes. The w4g32 tier opens its
own M=512 ELF and skips the twin check. Otherwise #2611's _m16-twin
detection runs unchanged. The plan check takes both exemptions.

#2611 merged without a test for its new decision branch, which both
reviewers asked for. The decision moves to npu/target/xdna/elf_twin.h
(m16_twin_path, same_artifact), and tests/npu_elf_twin_test (CTest
npu_elf_twin) covers every arm: identical, a difference in the last byte
or the second block, a size difference, no twin, empty files, and G == U
not counted as a twin. Three mutants each fail it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bong-water-water-bong pushed a commit that referenced this pull request Nov 16, 2026
…wn PR

Review at 1769d39 asked for one owner per PR. The elf_twin.h refactor and
the npu_elf_twin ctest (#2611's test arm) move to a separate PR from main.
This branch keeps main's twin code byte for byte: #2611's static
same_artifact and its inline twin path. The tier only wraps it in
'if (!w4_rows_)'.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bong-water-water-bong added a commit that referenced this pull request Nov 16, 2026
…st npu_elf_twin) (#2617)

#2611 merged without a test for its new decision branch, which both of its
reviewers asked for. The decision moves unchanged into
npu/target/xdna/elf_twin.h (m16_twin_path, same_artifact).
tests/npu_elf_twin_test covers every arm on files it writes itself:
- identical bytes -> twin;
- a difference in the last byte, or in the second 64 KiB block -> not a twin;
- a size difference -> not a twin;
- no twin on disk, or two empty files -> not a twin;
- G identical to U (same shape) is not G's twin.

Three header mutants each fail it: no byte compare, no size check, a wrong
twin name. No behaviour change in open_prefill.

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant