Skip to content

npu-pack: read the gemma3 1.8 six-header layer block (engine#2604) - #2614

Closed
bong-water-water-bong wants to merge 1 commit into
mainfrom
npu-pack-gemma3-six-header-block
Closed

bong-water-water-bong wants to merge 1 commit into
mainfrom
npu-pack-gemma3-six-header-block

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

What

npu-pack info on amd/translategemma-4b-it_rai_1.8.0_npu_4K reported 340 gaps, every one of them:

the layer block has 6 epsilon headers, and this importer reads blocks of 2 (q/k/v/o in one payload)
or 4 (q/k/v, two per-head QK norms, o_proj)

The block is Qwen3's four headers plus Gemma-3's pre- and post-feedforward norms.

Measured on the pack's own .fconst first

The Token file is 2,189,399,392 B. A header is eps (bf16 1e-6) followed by 62 zero bytes -- searching the eps pair alone gives ~47 800 false positives in that file, of which 205 are headers (34 × 6 + the tail). Per block:

h0→h1  5,854,272        q/k/v tiles
h1→h2        576        first per-head gain  (64 + bf16[head_dim 256])
h2→h3  2,959,968        second per-head gain [+32 B on the 29 sliding-window layers] + o_proj
         or 2,959,936   ... on the 5 full-attention layers
h3→h4      5,184        post_attention_layernorm
h4→h5      5,184        pre_feedforward_layernorm ... MLP 42,921,088 ... post_feedforward_layernorm

Both spans reproduce the file exactly:

full_attention (5):  64 + 512 +           578 × 5,120 = 2,959,936   EXACT
sliding        (29): 64 + 512 + 32 +      578 × 5,120 = 2,959,968   EXACT

The 32 B is 16 bf16 = 2 × num_attention_heads (config: hidden 2560, IM 10,240, 34 layers, 8 heads, 4 KV, head_dim 256, sliding_window 1024), i.e. a per-head scalar pair on the sliding layers -- and the five short layers are the full_attention ones, whose indices from layer_types (5, 11, 17, 23, 29) match the scan exactly. The reader skips the pair; it does not have to interpret it.

Change

  • the 6-header attention section, with extra ∈ {0, 32} derived from the section span itself (after_gains % tile_o) -- never from a version or a config, and the tiles counted as 578 either way;
  • mlp_start_in_block() (npu/pack_import.h): on this form the MLP opens after the second-to-last header and the block ends one envelope+weight after the MLP does. Reading the last header put the MLP one envelope past its start ("the MLP section ends at 51,741,088 B in a 51,746,272-byte layer block"), and passing the full stride then made the section exactly 5,184 B too generous;
  • the refusal message now names all three block forms (2, 4, 6).

Verified

npu-pack info <pack>   ->  gaps: 340   ...   gaps: 0

Arm: test_mlp_start_in_block pins the rule on the pack's own measured offsets (h3 8,814,816 · h4 8,820,000 · MLP 42,921,088 · h5 51,741,088, section = 42,915,904 B) plus the unchanged 4-header and 2-header cases. Reverting the helper to last-header makes 4 assertions fail; restored, npu_pack_import_test exits 0.

Not in this PR, and worth knowing

  • The import still refuses, one layer later, on the head: "best-fit scale 50.325832, rms ratio 0.019792" -- the tail decodes ~50× too small. The tail is 430,026,304 B over 262,208 × 2,560 = 0.6406 B/value, and (16 codes + 4 B scale/zero)/32 = 0.625, so the tail is q4_1 32-group and the scale/zero reading is the suspect (an f16-vs-bf16 mismatch would give a factor in this range rather than garbage).
  • No device row. A clean import is not proof (engine#2594), and this pack has no kernels-* family on this box yet.
  • Operational traps for the next person: npu-pack info leaves a 1.9 canonical copy under ~/.cache/1bit/fconst19/, and a refused import leaves a partial container in the output dir that must be deleted before retrying.

`npu-pack info` on amd/translategemma-4b-it_rai_1.8.0_npu_4K reported 340 gaps, every one
"the layer block has 6 epsilon headers, and this importer reads blocks of 2 ... or 4". The
block is Qwen3's four headers plus Gemma-3's pre- and post-feedforward norms.

Measured on the pack's own Token .fconst (2,189,399,392 B), by a scan for the envelope header
(eps followed by 62 zero bytes -- searching the eps pair alone gives ~47,800 false positives):
205 headers = 34 x 6 + the tail, and per block
  h0->h1 q/k/v tiles | h1->h2 first per-head gain | h2->h3 second per-head gain [+32 B on the
  29 SLIDING-window layers] then o_proj | h3->h4 post_attention | h4->h5 pre/post-feedforward,
  with the MLP between h4 and h5.

Both spans reproduce the file exactly: 64 + 512 + 578*5120 = 2,959,936 on the 5 full-attention
layers and + 32 = 2,959,968 on the other 29; the 32 B is 16 bf16 = 2 x num_attention_heads and
the reader SKIPS it rather than interpreting it. The five short layers are the full_attention
ones (config layer_types puts them at 5, 11, 17, 23, 29, matching the scan).

Two changes:
  * the 6-header attention section; `extra in {0,32}` is derived from the section span itself,
    never from a version or a config;
  * mlp_start_in_block(): on this form the MLP starts after the SECOND-TO-LAST header and the
    block ends one envelope+weight after the MLP does. Reading the last header put the MLP one
    envelope past its start ("ends at 51,741,088 B in a 51,746,272-byte layer block"), and
    passing the full stride then made the section 5,184 B too generous.

Verified: `npu-pack info` on the real pack -> 0 gaps (was 340). Arm: test_mlp_start_in_block
pins the rule on the pack's own measured offsets plus the unchanged 4- and 2-header cases;
reverting the helper to last-header makes 4 assertions fail.

Not in this PR: the import still refuses, one layer later, on the HEAD -- "best-fit scale
50.325832, rms ratio 0.019792", i.e. the tail decodes ~50x too small (430,026,304 B over
262,208 x 2,560 = 0.6406 B/value, a q4_1 32-group tile at 0.625, so the scale/zero reading is
the suspect). No device row: a clean import is not proof (engine#2594).
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Incorrect MLP offset calculation for 6-header blocks

The logic for calculating the MLP start offset in 6-header blocks appears to assume that the last header (hdr.back()) is always the post-feedforward norm, but in the 6-header form, the MLP starts after the second-to-last header (hdr[3]). This could lead to incorrect MLP section boundaries if the header structure is not properly validated or if the assumption about the header order is violated.

    mlp_off_6h = g.hdr[3] + kNormOff + H * 2;
}
Potential off-by-one error in header validation

In the validation logic for 6-header blocks, the check n_otiles != H / kTileRows || (extra != 0 && extra != 32) may be susceptible to off-by-one errors if the tile size or the extra bytes calculation is not precisely aligned with the actual memory layout. This could cause false positives or negatives in block validation.

else if (n_otiles != H / kTileRows || (extra != 0 && extra != 32))
    refuse("the o_proj section is " + std::to_string(after_gains) + " B after the gains; hidden " +
           std::to_string(H) + " rows at K = q_proj's " + std::to_string(q_out) + " are " +
           std::to_string(H / kTileRows) + " tiles of " + std::to_string(tile_o) +
           " B, optionally preceded by 32 B of per-head scalars on a sliding-window layer");

@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 16, 2026 02:02
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2614

"Qwen3's four headers plus Gemma-3's pre- and post-feedforward norms" is the answer #2612 left open, and it answers it in the right form: not "the count is 6" but what the two extra headers are. That distinction matters because a count is a fact about one pack and a composition is a fact about a family — and it makes the next pack's block predictable instead of discoverable.

Three asks:

  1. the layout should be described as a composition, in the code — 4 (q, k, v, o with QK norms) + 2 (pre/post-FFN norms), so a seventh header arrives as "this composition plus one" rather than as a new magic number. The refusal message quoted here (2 or 4 … 6) is already better than most; making it list named components is what would let the next variant be diagnosed from the message alone;
  2. say whether the two FFN norms are interleaved or appended — "6 headers" says nothing about order, and a reader that assumes appended when the pack appends and one that assumes interleaved produce two different (both plausible) wrong results. npu-pack: read the 1.8 all-block-32 Token .fconst: k before q, no sink field (Fixes #2594) #2597's lesson exactly: a numeric match can be fitted by two structural stories, and only a reconstruction separates them;
  3. verify numerically, per npu-pack: read the gemma3 1.8 Token .fconst (6 headers, two block sizes); serve stops on its turn end (Fixes #2604) #2612's ask — the epsilon values and norm weights must match the source blobs after import. A composition understood correctly and applied in the wrong order still imports and still translates.

And one thing worth recording: this is the third layout variant handled in npu/pack_import.cpp tonight (2-header blocks, the 1.8 k/q/v/o order in #2597, this composition). The file now has enough cases that a short table — block order, header count, composition, which pack version — would be worth more than the three separate comments. That is the same argument as #2589/#2595/#2592's three tools needing one page.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Superseded by #2612, which is the same fix plus the two things mine did not have

Closed without merging. #2612 reads the same six-header geometry and fixes the head, verifies against an independent ungated reference pack, and has a device row. Two things worth keeping from this attempt:

1. Our geometry measurements agree, independently. Their table (k, q, v after h0, head_dim gains after h1/h2, the 32 B field on sliding layers only, the MLP after h4) is what my header scan produced before I read their PR -- 205 headers = 34 x 6 + 1, h0→h1 5,854,272, h1→h2 576, h2→h3 2,959,968 / 2,959,936, and 64 + 512 + 578 x 5,120 reproducing both spans exactly. Two different methods landing on the same layout is the best evidence either of us can produce for it.

2. My head diagnosis was right, and their cause is the reason. I reported the import's own numbers -- "best-fit scale 50.325832, rms ratio 0.019792", i.e. the tail decodes ~50x too small -- and left it as "the scale/zero reading is the suspect". sqrt(2560) = 50.596: the pack's embedding is pre-scaled by sqrt(hidden) and the 4-bit head is the raw table, so the factor is not a decode bug at all. I had the symptom and the magnitude and not the cause; gemma_prescale_head() is the cause.

What mine had that theirs does not: nothing that matters -- mlp_start_in_block() is folded into their fconst_six_header_blocks() mapping, and their per-layer starts subsume my "second-to-last header" rule. The one small thing my PR stated that is worth carrying: on this form the block ends one envelope+weight after the MLP, so a caller passing the full stride to mlp_geom() asks for 5,184 B more than the file holds. If their mapping already excludes the trailing norm, nothing to do; if not, it is the next refusal.

Closing rather than asking them to absorb it, because a 105-line duplicate of a 359-line fix that also has the head and a device row is not worth a reviewer's time.

auto-merge was automatically disabled November 16, 2026 02:05

Pull request was closed

@bong-water-water-bong
bong-water-water-bong deleted the npu-pack-gemma3-six-header-block branch November 16, 2026 02:05

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 6e18406: READY

The arm is real — three failures on the mutant — but only once I mutated the right function; my first attempt tested the parse path and the test calls a helper.

The rule, with measured offsets and a cross-check

h0->h1 q/k/v tiles | h1->h2 first per-head gain | h2->h3 second per-head gain [+32 B on the 29 SLIDING layers]
then o_proj | h3->h4 post_attention | h4->h5 pre/post-feedforward | MLP
64 + 512 + 578*5120 = 2,959,936 on the 5 full-attention layers, +32 = 2,959,968 on the other 29

The 32 B is 16 bf16 = 2 x num_attention_heads: a per-head scalar pair the reader has to SKIP, not interpret. That is the detail a reader gets wrong by trying to understand it — the correct action is to skip it and cross-check its presence against the span, which is what the code does: "nothing here is inferred from a version or a config."

The arms, and the two mutants

mutant `hdr[3]` → `last` (h5, the bug the PR describes), placed in `mlp_start_in_block()`:
  FAIL mlp start: the 6-header form opens the MLP after h4, not h5 (51746272 vs 8825184)
  FAIL mlp start: h5 is past the MLP, so it is not the start
  FAIL mlp start: the section is 42,915,904 B, as the file has

The first mutant I wrote — mlp_off_6h = g.hdr[4] + … in the parse path — survived, because the test calls pi::mlp_start_in_block(), which is the shared helper. Testing a decision means mutating the function that makes it; a mutant in a caller that reaches the same answer is not a coverage gap, and I checked before writing one down.

Three independent arms fail on the real mutant: the offset, the "h5 is past the MLP" negative, and the section length — the last one is the assertion that would have caught the reported refusal ("the MLP section ends at 51,741,088 B in a 51,746,272-byte layer block"), because a wrong start makes the length wrong by exactly the difference.

And the 4-header control is there: s4 == 3000 + 64 + H * 2 && trim4 == 0 — "a 4-header block keeps the last header and trims nothing". That is what keeps the gemma3 branch from changing the other form's behaviour, which is the regression a hdr.size() == 5 branch invites.

One merge-order note

#2612 also edits npu/pack_import.cpp (+156 lines there, +1 067 overall because it carries several already-merged files). This PR's +59 lines in the same file are independent, but whichever lands second needs a rebase — and if #2612 is a branch containing merged work, its own diff against main is the only part worth reading.

Ship it.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

6e18406 CHANGES — the code and its arms are good; it cannot land beside #2612, which implements the same fix in the same functions.

Content verified: built and run here (thermal-run, -j3) — configure 0, build 0, npu_pack_import_test rc 0. The new helper and its arms are load-bearing; mutating mlp_start_in_block to use hdr.back() (h5) instead of hdr[3] (h4) fails three named checks with real numbers:

FAIL mlp start: the 6-header form opens the MLP after h4, not h5 (51746272 vs 8825184)
FAIL mlp start: h5 is past the MLP, so it is not the start
FAIL mlp start: the section is 42,915,904 B, as the file has

That third line is the one that matters: it closes on the file's own byte count, not only on a relative offset, so the rule is pinned to a measured pack (translategemma-4b layer 0: q/k/v | h1 5,854,272 | h2 5,854,848 | h3 8,814,816 | h4 8,820,000 | …) rather than to arithmetic that happens to be self-consistent.

The problem is scope, not quality: git merge-tree pr2612 pr2614 reports a conflict, and the two PRs implement the same idea — "per-layer starts from every header, not one stride" — in the same three files:

#2612 #2614
files 7 (importer, pack header, docs, turn-end template probe, two test files) 3 (importer, pack header, pack test)
importer lines added 153 58
mechanism six_headers() + fconst_six_header_blocks(), refusal messages naming the form inline hdr.size() == 5 branch + the exported mlp_start_in_block()

Both are right in different ways: #2612 gives the form a name and refuses unknown 6-header files loudly, and carries the template/docs work; #2614 carries the tighter arithmetic and the 5,184 B correction ("Passing the full stride made the decoder expect 5,184 B more than the file has"). They cannot both land, and whichever loses has to delete work the other does not contain.

What I would do — yours to decide, not mine: land #2614 as the importer change (its arm closes on a real file's byte count, and it is the smaller diff), then rebase #2612 onto it keeping only what it uniquely adds (turn-end template probe, docs, refusal messages if they still apply).

Not verified: the real translategemma-4b-it_rai_1.8.0_npu_4K / Gemma-3-4b_rai_1.8.0_npu_16K packs — the arms synthesise their headers from the measured offsets, which is right for determinism.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant