Skip to content

npu: bonsai2-27b-NPU's rm prefill computes 16 of every 128 rows (its 'prefill' ELFs are the M=16 design; the other 112 rows come back 0) #2599

Closed

Description

@bong-water-water-bong

What

On bonsai2-27b-NPU (rm layout), each 128-row prefill chunk computes only its first 16 rows. The other 112 come back as zeros.

The cause, as far as the evidence goes:

  • The pack's "prefill" ELFs kernels/full_i8_*_K*_N*.elf are byte-identical (md5) to their _m16 twins. They are the 16-row design.
  • XdnaTarget::open_prefill sets ctx->MD = g->M = Q4NX_DESIGN_M_PREFILL (128). So max_rows() gives 128-row chunks, and quantize_async_rows writes 128 rows into a buffer the kernel reads 16 rows of.
  • args_fit_elf() cannot catch it. The declared widths are per-tile (512 / 1,024 / 4,096 B for U), and a declared width smaller than the buffer is allowed by design.

Measured (strixhalo, npu-hold ticket cl-game-w4ab-27b, 0 amdxdna lines)

1bit npu-run --model <bonsai2-27b-NPU> --kernels <its kernels> --ids <512 ids> prints the argmax of every row:

  • In all four 128-row chunks, exactly 16 rows are nonzero and 112 are token 0.
  • Teacher-forced next-token top-1 is 23 / 511, almost all of it from the computed rows.
  • The last prompt row (511) is a zeroed row, so a reply starts from a hidden state the layers never computed.

md5 groups of kernels/*.elf, each the unsuffixed file and its _m16:

role identical to its _m16?
D yes
G = U yes
O yes
QKV N10240 yes
QKV N14336 yes
Z yes

What would fix it

Either of these:

  • Ship real M=128 int8 prefill ELFs for this pack.
  • Make the forward refuse, or chunk at the design's real M, when a prefill ELF is the 16-row design. The pack or plan has to say which, because the artifact's declared widths don't.

The NPU_W4_PREFILL tier being built under HETEROMOSAIC-2 runs its own M=512 kernels and does not have this problem. It is opt-in, though, so the default path stays broken until this is fixed.

🤖 Generated with Claude Code

Activity

  1. bong-water-water-bong commented on Nov 16, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The cause named here is a name-based check meeting a content-based lie, and this lane has already built the pattern that catches it: refuse by name is not enough — refuse by content.

    The evidence chain says it precisely: the prefill ELFs are byte-identical (md5) to their _m16 twins, open_prefill sets MD = M = Q4NX_DESIGN_M_PREFILL (128) from the container's declaration, and args_fit_elf() cannot catch it because the declared widths are per-tile — a 16-row ELF and a 128-row ELF have identical argument widths. So every check in the path is asking the file's name (or the container's metadata) and none is asking the ELF's design M.

    That is the same class as the two rules this lane has already encoded:

    • staged() refuses a kernel name that is off the engine grid (2304→2560, 768→1024) by name rather than trusting the pack's folder labels;
    • Q4NX_DESIGN_M_PREFILL widening: the target refuses by name on an MD mismatch rather than accepting whatever the container claims.

    The fix this suggests, and it needs no device: at open_prefill, after npu_rt hands back the loaded ELF, compare the ELF's own design-M (whatever the XDNA loader exposes, or a value the design writes into a known location — q4nx_forward.cpp:549-617 already tracks designs16_[i] and the _m16 marker the arena uses) against the MD being requested. Equal ⇒ proceed; not equal ⇒ refuse, naming both numbers. That single assertion turns "an md5-identical prefill ELF that computes 16 rows" from a silent 8× wrong answer into a load-time error, and it is checkable on the box with the pack that reproduces it.

    And a cheap belt on the pack side, since the twins are byte-identical: npu-pack import can refuse a pack whose prefill ELF is byte-identical to its _m16 sibling — that is a property of the pack, cheap to compute (md5), and it catches the specific packaging mistake at the moment the pack is made rather than at the moment a user gets 16 rows of prefill out of 128. The pair (import refuses identical twins; loader refuses an MD mismatch) covers both a bad pack and a bad loader path.

    One question that decides whether the loader assertion is implementable: does the ELF carry the design M anywhere a host can read it before a launch — a design row/parameter block the loader could parse, or a symbol table entry? If it does not, the md5 twin check at import is the only cheap instrument, and that should be said in the issue rather than left as an assumption.

    Not verified by me: the reproduction and the md5 comparison of the twins. The per-tile-width argument, the name-vs-content distinction and the two refusal points above are what I can check in the code this lane has been working on, and the first is right.

  2. bong-water-water-bong commented on Nov 16, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Checked whether the md5-twin defect is systemic or specific to that pack: on the five kernel packs this lane owns, no ELF pair is byte-identical — 46 ELFs, zero duplicate md5 groups.

    kernels-whisper-small-768   elfs=8   duplicate-md5-groups=0
    kernels-synth768-run        elfs=12  duplicate-md5-groups=0
    kernels-synth768-i1536      elfs=8   duplicate-md5-groups=0
    kernels-whisper-lv3-1280    elfs=8   duplicate-md5-groups=0
    kernels-synth1280-run       elfs=10  duplicate-md5-groups=0
    

    Three things that follows, and the third is the useful one:

    1. The md5 twin check is cheap and specific — one md5sum over the pack's ELFs, no device, no ELF parsing. It would have caught the bonsai2 defect at import time and flags nothing on the packs that are correct, so it is safe to make it a refusal rather than a warning.
    2. It is also not sufficient on its own: a pack could ship a prefill ELF that is byte-identical to nothing and still be a 16-row design with a different build stamp. The md5 check catches the packaging mistake (copying the twins); only the design-M assertion at load catches the design mistake. That is why I suggested both rather than either.
    3. The absence of twins here is evidence about how the defect arose: these five packs were produced by the flow that names kernels on the engine grid (ONEBIT-side staged() refuses an off-grid name), while the offending pack appears to have had its prefill ELFs copied from the _m16 set. If that is what happened, the durable fix is in the packer — emit the prefill family from the design that was widened, and refuse to write a pack whose prefill file is identical to a decode twin — not in the loader alone.

    Still the open question from my last comment: does an ELF carry its design M anywhere a host can read before a launch? If the answer is no, the loader assertion I proposed is not implementable and the packer-side check becomes the only real defence — worth settling before someone writes code either way.

  3. bong-water-water-bong commented on Nov 16, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Checked whether the md5-twin defect is systemic or specific to that pack: on the five kernel packs this lane owns, no ELF pair is byte-identical — 46 ELFs, zero duplicate md5 groups.

    kernels-whisper-small-768   elfs=8   duplicate-md5-groups=0
    kernels-synth768-run        elfs=12  duplicate-md5-groups=0
    kernels-synth768-i1536      elfs=8   duplicate-md5-groups=0
    kernels-whisper-lv3-1280    elfs=8   duplicate-md5-groups=0
    kernels-synth1280-run       elfs=10  duplicate-md5-groups=0
    

    Three things follow, and the third is the useful one:

    1. The md5 twin check is cheap and specific — one md5sum over the pack's ELFs, no device, no ELF parsing. It would have caught the bonsai2 defect at import time and flags nothing on the packs that are correct, so it is safe to make it a refusal rather than a warning.
    2. It is not sufficient on its own: a pack could ship a prefill ELF that is byte-identical to nothing and still be a 16-row design with a different build stamp. The md5 check catches the packaging mistake (copying the twins); only the design-M assertion at load catches the design mistake. That is why I suggested both rather than either.
    3. The absence of twins here is evidence about how the defect arose: these five packs were produced by the flow that names kernels on the engine grid (staged() refuses an off-grid name), while the offending pack appears to have had its prefill ELFs copied from the _m16 set. If that is what happened, the durable fix is in the packer — emit the prefill family from the design that was widened, and refuse to write a pack whose prefill file is identical to a decode twin — not in the loader alone.

    Still the open question from my last comment: does an ELF carry its design M anywhere a host can read before a launch? If the answer is no, the loader assertion I proposed is not implementable and the packer-side check becomes the only real defence — worth settling before someone writes code either way.

  4. bong-water-water-bong commented on Nov 16, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Settled the question I left open: these AIE ELFs carry no readable design M — no section headers (e_shnum = 0), no symbols, no design string. So the loader-side assertion needs a value the packer has to write in, and that changes which fix is the cheap one.

    Checked on this lane's own packs (kernels-whisper-small-768):

    full_i8_D_K3072_N1024_b8.elf         315 400 B   e_shnum=0   (no sections)
    full_i8_D_K3072_N1024_b8_m16.elf     110 328 B   e_shnum=0
    strings: "IDPP", "aie_image", then compressed payload — no kernel name, no "m16", no K/N/M tags
    

    What that means, concretely:

    • A host cannot read the design M from the ELF before a launch — at least not by any means I can find: the file is a stripped AIE image with no section table, no symtab, and no design string. So my proposed open_prefill assertion is not implementable as stated, and I would rather say that now than have someone try.
    • The m16 tag that distinguishes the families lives only in the filename, which is the same name-based trust the bonsai2 pack defeated. q4nx_forward.cpp:606-617 copies designs16_[i].elf_name and the arena distinguishes the twins by that name — so if a pack ships a prefill file whose content is the m16 design under a prefill name, nothing on the host can tell.
    • Therefore the durable check is on the side that knows both facts — the packer. At pack time the design is available: refuse to write a pack whose prefill ELF is byte-identical to a decode twin (cheap, and it would have caught this one), and — stronger — write the design parameters into the pack as metadata (kernels/design.json with {M, K, N, bits} per design). Then the loader can assert metadata.M == MD and the assertion I wanted becomes possible, because the value comes from the producer rather than from the ELF.

    That is the shape I would recommend: the packer records what it knows; the loader checks the record against what it is about to do. A pack without the metadata still loads (compatibility), but a pack with it can never silently compute 16 rows out of 128 — and the failure names both numbers.

    And the minimal version, if nobody wants a new file in the format: md5-twin refusal at import only. It is one line, needs no new format, and catches exactly the packaging accident that produced this issue. It does not catch a 16-row design compiled under a fresh name, which is why the metadata is worth having eventually.

    Not verified by me: the bonsai2 pack itself (I do not have it); the ELF structure claim is from this lane's whisper/synth packs, and the two _m16/non-_m16 files above are the same family the issue describes.

  5. bong-water-water-bong commented on Nov 16, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Fixed on main by #2611 (merged).

    The fix: open_prefill gives a role 16 rows when its prefill ELF is byte-identical to the _m16 twin of its own stem, with a named line (engine#2599).

    Gate: NPU_BATCH_CHECK on bonsai2-27b-NPU, 32 ids:

    rows argmax
    before identical=16 differing=16, bat=0x00000000 MISMATCH
    after 32/32 bitwise identical MATCH ×2

    128-token prompt: 79/127 next-token, 0 zero rows.

    Follow-ups:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions