Repository navigation
npu-pack: read the gemma3 1.8 six-header layer block (engine#2604) - #2614
bong-water-water-bong wants to merge 1 commit into
Conversation
`npu-pack info` on amd/translategemma-4b-it_rai_1.8.0_npu_4K reported 340 gaps, every one
"the layer block has 6 epsilon headers, and this importer reads blocks of 2 ... or 4". The
block is Qwen3's four headers plus Gemma-3's pre- and post-feedforward norms.
Measured on the pack's own Token .fconst (2,189,399,392 B), by a scan for the envelope header
(eps followed by 62 zero bytes -- searching the eps pair alone gives ~47,800 false positives):
205 headers = 34 x 6 + the tail, and per block
h0->h1 q/k/v tiles | h1->h2 first per-head gain | h2->h3 second per-head gain [+32 B on the
29 SLIDING-window layers] then o_proj | h3->h4 post_attention | h4->h5 pre/post-feedforward,
with the MLP between h4 and h5.
Both spans reproduce the file exactly: 64 + 512 + 578*5120 = 2,959,936 on the 5 full-attention
layers and + 32 = 2,959,968 on the other 29; the 32 B is 16 bf16 = 2 x num_attention_heads and
the reader SKIPS it rather than interpreting it. The five short layers are the full_attention
ones (config layer_types puts them at 5, 11, 17, 23, 29, matching the scan).
Two changes:
* the 6-header attention section; `extra in {0,32}` is derived from the section span itself,
never from a version or a config;
* mlp_start_in_block(): on this form the MLP starts after the SECOND-TO-LAST header and the
block ends one envelope+weight after the MLP does. Reading the last header put the MLP one
envelope past its start ("ends at 51,741,088 B in a 51,746,272-byte layer block"), and
passing the full stride then made the section 5,184 B too generous.
Verified: `npu-pack info` on the real pack -> 0 gaps (was 340). Arm: test_mlp_start_in_block
pins the rule on the pack's own measured offsets plus the unchanged 4- and 2-header cases;
reverting the helper to last-header makes 4 assertions fail.
Not in this PR: the import still refuses, one layer later, on the HEAD -- "best-fit scale
50.325832, rms ratio 0.019792", i.e. the tail decodes ~50x too small (430,026,304 B over
262,208 x 2,560 = 0.6406 B/value, a q4_1 32-group tile at 0.625, so the scale/zero reading is
the suspect). No device row: a clean import is not proof (engine#2594).
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY "Qwen3's four headers plus Gemma-3's pre- and post-feedforward norms" is the answer #2612 left open, and it answers it in the right form: not "the count is 6" but what the two extra headers are. That distinction matters because a count is a fact about one pack and a composition is a fact about a family — and it makes the next pack's block predictable instead of discoverable. Three asks:
And one thing worth recording: this is the third layout variant handled in |
Superseded by #2612, which is the same fix plus the two things mine did not haveClosed without merging. #2612 reads the same six-header geometry and fixes the head, verifies against an independent ungated reference pack, and has a device row. Two things worth keeping from this attempt: 1. Our geometry measurements agree, independently. Their table ( 2. My head diagnosis was right, and their cause is the reason. I reported the import's own numbers -- "best-fit scale 50.325832, rms ratio 0.019792", i.e. the tail decodes ~50x too small -- and left it as "the scale/zero reading is the suspect". What mine had that theirs does not: nothing that matters -- Closing rather than asking them to absorb it, because a 105-line duplicate of a 359-line fix that also has the head and a device row is not worth a reviewer's time. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 6e18406: READY
The arm is real — three failures on the mutant — but only once I mutated the right function; my first attempt tested the parse path and the test calls a helper.
The rule, with measured offsets and a cross-check
h0->h1 q/k/v tiles | h1->h2 first per-head gain | h2->h3 second per-head gain [+32 B on the 29 SLIDING layers]
then o_proj | h3->h4 post_attention | h4->h5 pre/post-feedforward | MLP
64 + 512 + 578*5120 = 2,959,936 on the 5 full-attention layers, +32 = 2,959,968 on the other 29
The 32 B is 16 bf16 = 2 x num_attention_heads: a per-head scalar pair the reader has to SKIP, not interpret. That is the detail a reader gets wrong by trying to understand it — the correct action is to skip it and cross-check its presence against the span, which is what the code does: "nothing here is inferred from a version or a config."
The arms, and the two mutants
mutant `hdr[3]` → `last` (h5, the bug the PR describes), placed in `mlp_start_in_block()`:
FAIL mlp start: the 6-header form opens the MLP after h4, not h5 (51746272 vs 8825184)
FAIL mlp start: h5 is past the MLP, so it is not the start
FAIL mlp start: the section is 42,915,904 B, as the file has
The first mutant I wrote — mlp_off_6h = g.hdr[4] + … in the parse path — survived, because the test calls pi::mlp_start_in_block(), which is the shared helper. Testing a decision means mutating the function that makes it; a mutant in a caller that reaches the same answer is not a coverage gap, and I checked before writing one down.
Three independent arms fail on the real mutant: the offset, the "h5 is past the MLP" negative, and the section length — the last one is the assertion that would have caught the reported refusal ("the MLP section ends at 51,741,088 B in a 51,746,272-byte layer block"), because a wrong start makes the length wrong by exactly the difference.
And the 4-header control is there: s4 == 3000 + 64 + H * 2 && trim4 == 0 — "a 4-header block keeps the last header and trims nothing". That is what keeps the gemma3 branch from changing the other form's behaviour, which is the regression a hdr.size() == 5 branch invites.
One merge-order note
#2612 also edits npu/pack_import.cpp (+156 lines there, +1 067 overall because it carries several already-merged files). This PR's +59 lines in the same file are independent, but whichever lands second needs a rebase — and if #2612 is a branch containing merged work, its own diff against main is the only part worth reading.
Ship it.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
6e18406 CHANGES — the code and its arms are good; it cannot land beside #2612, which implements the same fix in the same functions.
Content verified: built and run here (thermal-run, -j3) — configure 0, build 0, npu_pack_import_test rc 0. The new helper and its arms are load-bearing; mutating mlp_start_in_block to use hdr.back() (h5) instead of hdr[3] (h4) fails three named checks with real numbers:
FAIL mlp start: the 6-header form opens the MLP after h4, not h5 (51746272 vs 8825184)
FAIL mlp start: h5 is past the MLP, so it is not the start
FAIL mlp start: the section is 42,915,904 B, as the file has
That third line is the one that matters: it closes on the file's own byte count, not only on a relative offset, so the rule is pinned to a measured pack (translategemma-4b layer 0: q/k/v | h1 5,854,272 | h2 5,854,848 | h3 8,814,816 | h4 8,820,000 | …) rather than to arithmetic that happens to be self-consistent.
The problem is scope, not quality: git merge-tree pr2612 pr2614 reports a conflict, and the two PRs implement the same idea — "per-layer starts from every header, not one stride" — in the same three files:
| #2612 | #2614 | |
|---|---|---|
| files | 7 (importer, pack header, docs, turn-end template probe, two test files) | 3 (importer, pack header, pack test) |
| importer lines added | 153 | 58 |
| mechanism | six_headers() + fconst_six_header_blocks(), refusal messages naming the form |
inline hdr.size() == 5 branch + the exported mlp_start_in_block() |
Both are right in different ways: #2612 gives the form a name and refuses unknown 6-header files loudly, and carries the template/docs work; #2614 carries the tighter arithmetic and the 5,184 B correction ("Passing the full stride made the decoder expect 5,184 B more than the file has"). They cannot both land, and whichever loses has to delete work the other does not contain.
What I would do — yours to decide, not mine: land #2614 as the importer change (its arm closes on a real file's byte count, and it is the smaller diff), then rebase #2612 onto it keeping only what it uniquely adds (turn-end template probe, docs, refusal messages if they still apply).
Not verified: the real translategemma-4b-it_rai_1.8.0_npu_4K / Gemma-3-4b_rai_1.8.0_npu_16K packs — the arms synthesise their headers from the measured offsets, which is right for determinism.
What
npu-pack infoonamd/translategemma-4b-it_rai_1.8.0_npu_4Kreported 340 gaps, every one of them:The block is Qwen3's four headers plus Gemma-3's pre- and post-feedforward norms.
Measured on the pack's own
.fconstfirstThe Token file is 2,189,399,392 B. A header is
eps(bf16 1e-6) followed by 62 zero bytes -- searching the eps pair alone gives ~47 800 false positives in that file, of which 205 are headers (34 × 6 + the tail). Per block:Both spans reproduce the file exactly:
The 32 B is 16 bf16 = 2 ×
num_attention_heads(config: hidden 2560, IM 10,240, 34 layers, 8 heads, 4 KV, head_dim 256, sliding_window 1024), i.e. a per-head scalar pair on the sliding layers -- and the five short layers are the full_attention ones, whose indices fromlayer_types(5, 11, 17, 23, 29) match the scan exactly. The reader skips the pair; it does not have to interpret it.Change
extra ∈ {0, 32}derived from the section span itself (after_gains % tile_o) -- never from a version or a config, and the tiles counted as 578 either way;mlp_start_in_block()(npu/pack_import.h): on this form the MLP opens after the second-to-last header and the block ends one envelope+weight after the MLP does. Reading the last header put the MLP one envelope past its start ("the MLP section ends at 51,741,088 B in a 51,746,272-byte layer block"), and passing the full stride then made the section exactly 5,184 B too generous;Verified
Arm:
test_mlp_start_in_blockpins the rule on the pack's own measured offsets (h3 8,814,816 · h4 8,820,000 · MLP 42,921,088 · h5 51,741,088, section = 42,915,904 B) plus the unchanged 4-header and 2-header cases. Reverting the helper to last-header makes 4 assertions fail; restored,npu_pack_import_testexits 0.Not in this PR, and worth knowing
262,208 × 2,560= 0.6406 B/value, and(16 codes + 4 B scale/zero)/32 = 0.625, so the tail is q4_1 32-group and the scale/zero reading is the suspect (an f16-vs-bf16 mismatch would give a factor in this range rather than garbage).kernels-*family on this box yet.npu-pack infoleaves a 1.9 canonical copy under~/.cache/1bit/fconst19/, and a refused import leaves a partial container in the output dir that must be deleted before retrying.