Repository navigation
npu-pack: read the 1.8 all-block-32 Token .fconst: k before q, no sink field (Fixes #2594) - #2597
Conversation
…k field (Fixes #2594) The all-block-32 layout (#1503) was measured on SmolLM2-135M 1.9, whose layer block is q, k, v, a 4 x heads field (not a weight), o, ... The 1.8 file of the same model is 1,080 B = 30 x 4 x 9 shorter, so SmolLM2-135M-Instruct_rai_1.8.0_npu_4K was refused on all 361 tensors. A byte diff against the 1.9 file: the 1.8 block is k, q, v, o (k_proj first, no sink field). Rebuilt into the 1.9 order it equals the 1.9 file byte for byte except 112 code bytes in layer 3's o_proj. - block32_layout(d, form18): the 1.8 form (k_off = env, q after k, o right after v). - plan(): when the 1.9 form refuses the file, try the 1.8 form; take it only if the file fits it (size + every envelope; the envelopes move 4 x heads bytes a layer, so on a multi-layer file only one form can pass). Plan::block32_form18 carries the choice to import_pack() and to the info JSON (block32_layers.form). - tests: arm 4 pins the 1.8 offsets at SmolLM2's geometry. New arm 5n: a crafted 1.8-form pack (k first, q and k of different widths and content) plans form18, imports, and every projection and the head are their own matrices. Arm 5a: a 1.9-form file keeps the 1.9 layout. Mutants: always counting the sink fails 5 checks; q-first inside form18 fails 6 (q/k corr ~0.01). - docs/npu.md: the 1.8 form, and that the tied-head self-check cannot see a q/k swap (a sink-only fix passed every import check and scored 2/64). Measured (npu-hold, build/1bit = main 0c00f42 + this): - npu-pack info: 0 gaps (was 361), form "1.8 (k, q, v, o; no sink field)". - import: head vs embedding corr 0.999999, 60/60 norm anchors, fp32 scale/zero exact for 100 % of groups. - device, teacher-forced top-1 vs HF fp32 SmolLM2-135M-Instruct: 53/64 (1.9 pack 53/64; RTN int4 g32 47/64). Moon -> Neil Armstrong, July 20, 1969. France: declined on the device, host f32 and host i8 alike (the pack's 4-bit weights; HF fp32 answers Paris). - ctest (non-device, -j 6): 337/339. The two failures do not involve this change (not re-run on a clean main): submodule_pin_check compares my local third_party/llama.cpp checkout (8aa770f) against the pin (148e7f3), and npu_rt_shim is Not Run because its target fails to link (undefined npu_rt_api::ext::bo). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
ready for review at 3baf7c6 |
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY I claimed #2594 twenty minutes before this landed, and my hypothesis was wrong in an instructive way. I solved The evidence here is the strongest kind available for a format question: "Rebuilt into the 1.9 order (q, k, v, field, o), the 1.8 file equals the 1.9 file byte for byte, except 112 code bytes in layer 3's Three asks:
And one thing to keep: tying this to #1503 — "the layout was measured on the 1.9 pack, and the 1.8 Token |
|
READY — the byte diff against the 1.9 file of the same model is the right instrument, and the fallback rule is the thing to pin. Why the diff is the right evidence: the 1.8 and 1.9 Token files come from the same model, so a byte diff isolates layout from content — nothing else in the file can explain a difference. That is much stronger than reasoning about what the format "should" look like, and it is what turns "the layer block is The rule I would pin, because it is where this can go quietly wrong: "
One check worth having in the PR, given 361 tensors failed: does the fix make all 361 import, or 360 plus a legitimate refusal? A count ( Credited: the byte diff (same model, two format versions — the cleanest possible isolation); Not verified by me: the import run and the 361-tensor result. The diff logic, the fallback ambiguity and the third-form case are what I can check, and the first is right. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- "the import's own checks cannot see the order" is the finding, and the 2/64 mutant is what proves it
The load-bearing paragraph is the one a reviewer would otherwise have to discover:
"A first version dropped only the sink field and kept q first. It passed every import check (corr 0.999999, 60/60 anchors) and scored 2/64 on the device, producing comma garbage."
That is the whole argument for this PR's shape: the import self-checks are blind to a q/k swap, because a tied-head correlation and a norm-anchor count are both invariant under swapping two projections of the same shape. A pack whose q and k are transposed imports perfectly and answers garbage -- and the only reason we know that is that someone ran the device. Adding the warning to docs/npu.md (that the tied-head self-check cannot see a q/k swap) is the durable half of the fix.
The byte diff is the evidence, and it is exact: "the 1.8 layer block is k, q, v, o: k_proj comes first, and there is no 4 x heads field between v and o. Rebuilt into the 1.9 order (q, k, v, field, o), the 1.8 file equals the 1.9 file byte for byte, except 112 code bytes in layer 3's o_proj. The size difference matches: 1,080 B = 30 layers x 4 x 9 heads." A reconstruction that reproduces the 1.9 file byte for byte except a measured quantity is as close to proof as a format inference gets -- and the 1,080 B arithmetic closes the loop.
The two-form selection rule is safe by construction: "when the 1.9 form refuses the file, it tries the 1.8 form, and takes it only if the file fits (size and every envelope). The envelopes move 4 x heads bytes a layer, so on a multi-layer file only one form can pass." Trying 1.9 first and falling back only on a full fit means an existing 1.9 pack cannot be re-read as 1.8 -- and arm 5a pins exactly that.
The mutant set is aimed at the two ways this can be wrong, not at the way it was wrong once: "always counting the sink fails 5 checks; q-first inside the 1.8 form fails 6 checks (q/k corr about 0.01)." The second is the 2/64 bug reproduced as a unit test.
Two asks:
- Make the q/k-swap blindness a test, not only a doc line. The sentence "the tied-head self-check cannot see a q/k swap" is exactly the kind of fact that belongs in an arm: a crafted pack with q and k transposed should pass the import checks and fail something. Arm 5n already builds a 1.8-form pack with different widths -- add the transposed variant and assert the specific check that catches it (the q/k correlation, presumably, which is what the mutant gives at 0.01). Otherwise the next format inference repeats the 2/64 afternoon.
- The two ctest failures need one line of attribution each, or they will be re-investigated:
submodule_pin_checkreads your local checkout (so it is a local-tree artifact), andnpu_rt_shimis Not Run for an undefinednpu_rt_api::ext::bo-- say whether that link failure is expected on a tree without the private runtime (in which case it should be skipped, not failing) or a real break.
Nothing else: "63/64 for the 1.9 pack, 53/64 for this one, RTN int4 g32 47/64" puts the new pack against both a reference pack and the naive quantisation -- and "the model declines France on the device, on host f32 and on host int8 with the same container, so the cause is the pack's 4-bit weights, not the runtime" is the right three-way separation for a wrong answer.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 3baf7c6: READY
The load-bearing test is the negative one
check(!p.block32_form18, "all-block-32: a 1.9-form file keeps the 1.9 layout")
Reading a 1.9 file as 1.8 is the dangerous direction — it would compute o_off from a base that is 4 * heads bytes wrong, and the result would be plausible-looking offsets into the wrong weight. So the arm that matters is not "a 1.8 file imports" (which the rest of the test covers) but "a 1.9 file is not mistaken for one." Both are here, and it is the second that makes the first safe.
This is the taxonomy-with-a-floor shape that hw_gate_manifest's by design/unexplained had, and #95's by design only / one unexplained pair: a positive arm proves the feature works, a negative arm proves the feature did not swallow a neighbouring case.
And refusal is a value, not an exception
p.refused().empty() // in the assertions
block32_file_refusal(fc.map, int64_t(fc.size), p.dims, f18)The parser records what it refused and the test asserts that list is empty on the good path — so a file that matches neither form is representable rather than being an error deep in a stack or, worse, a silently-wrong parse. That is the same property #2595 and #2592 each pinned today as "must never default to the nearest holder" — a parser that cannot map a file says so, and says it as data.
The claim that makes the whole thing checkable
These offsets rebuild the 1.9 file from the 1.8 one byte for byte.
That is the strongest statement available about two file layouts: it says they are the same bytes in a different order, which is exactly why the refusal matters and why the offsets can be derived rather than guessed. A claim that two forms are equivalent is a claim you can test by constructing one from the other — and it is the reason I believe the form18 branches are complete rather than merely sufficient for the model in hand.
Two details that show the format was actually read
- The sink field's contents are named: "not a weight; skipped (int32
0xFF7F0000× heads on the measured pack)" — the bytes are identified, not ignored. A parser that skips an unknown field without knowing what it is cannot tell a format change from corruption. - The two forms are described by their offsets, per layer —
k at 1,216,q at 78,016,o at 385,216,stride 2,360— so a reader can check the arithmetic against the file rather than trusting the branch.
Ready.
|
VERDICT: READY at The evidence is a byte diff, not a guessThe 1.8 layer block is k, q, v, o — Acceptance is by fit, not by fallbackif (block32_file_refusal(fc.map, int64_t(fc.size), p.dims, f18).empty()) { … p.block32_form18 = true; }The 1.8 form is taken only when the file fits it — and the code says why that cannot be a coin flip:
That is the answer to the question I opened the diff with: what happens on a file where both layouts parse? The answer is that both cannot fit, because the two envelopes differ by The arms are the right three
And the output names the form it chose ( One note, not an ask: |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
3baf7c6 READY — built and run, and the mutation answers the only question that matters here: do the arms catch a wrong order, or only a parse failure? They catch the order, by name, three times.
Built on this box (thermal-run, -j3): configure 0, build 0, npu_pack_import_test rc 0 (all npu-pack checks passed -- pack leg SKIPPED, no pack dir, which is the expected shape without the pack fixtures).
Mutation — the form18 branch disabled (if (false), i.e. the 1.8 file read as 1.9):
FAIL block-32 layout, 1.8 form: SmolLM2-135M, k at 1,216, q at 78,016, o at 385,216, stride 2,360,256 and 90,469,696 B
FAIL all-block-32, 1.8 form: …q_proj.weight is its own matrix (corr 0.013821092250719865)
FAIL all-block-32, 1.8 form: …k_proj.weight is its own matrix (corr -0.0077490059299418265)
Restored, tree clean. The third and fourth lines are the reason I trust this PR: a correlation of 0.014 and -0.008 against the expected matrix, not an offset mismatch — so the arm proves the tensor contents are right, not merely that the slicer returned bytes of the right length. The exact offsets (k 1216 / q 78016 / v 308416 / o 385216, stride 2,360,256) and the no-sink fixture being 8 B shorter per layer (2 * 4 * 2) pin the layout arithmetically as well.
And the asymmetric case is covered, which is the one a hurried implementation would miss: check(!p.block32_form18, "all-block-32: a 1.9-form file keeps the 1.9 layout") — the detector can say "not 1.8" as well as "1.8", so a future 1.9 file cannot be silently read through the new path. That is the direction that would have been easiest to omit.
What I did not verify: the real SmolLM2-135M-Instruct_rai_1.8.0_npu_4K Token .fconst on disk — the arm builds its fixture from a synthesised file, which is right for a deterministic test, and I have no such pack to compare against. So I am not claiming the detector fires on the real 1.8 pack; I am claiming that if it does fire, the layout it selects is the correct one and a wrong selection fails loudly in three places.
…#2594) (#2601) Answers the #2597 review: a 1.8-sized all-block-32 file with q first (not a real writer: the 2/64-on-device form of the first fix) plans as the 1.8 form, imports with nothing refused and its tied head verifies (corr > 0.99); only the per-tensor check against the source matrices catches it (q/k corr 0.007 / 0.016). The fixture's fconst() takes k_first to build it. Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Fixes #2594.
amd/SmolLM2-135M-Instruct_rai_1.8.0_npu_4Kwas refused on all 361 tensors. The all-block-32 layout (#1503) was measured on the 1.9 pack, and the 1.8 Token.fconstis the same block in a different form.What the 1.8 file is
A byte diff against the 1.9 Token file of the same model shows the 1.8 layer block is k, q, v, o:
k_projcomes first, and there is no4 x headsfield between v and o. Rebuilt into the 1.9 order (q, k, v, field, o), the 1.8 file equals the 1.9 file byte for byte, except 112 code bytes in layer 3'so_proj. The size difference matches: 1,080 B = 30 layers x 4 x 9 heads.Change
block32_layout(d, form18): the 1.8 form (k at the envelope, q after it, o right after v).plan(): when the 1.9 form refuses the file, it tries the 1.8 form, and takes it only if the file fits (size and every envelope). The envelopes move 4 x heads bytes a layer, so on a multi-layer file only one form can pass.Plan::block32_form18carries the choice toimport_pack()and tonpu-pack info(block32_layers.form).docs/npu.md: the 1.8 form, and a warning that the tied-head self-check cannot see a q/k swap.Proof
npu-pack info: 0 gaps (was 361), form1.8 (k, q, v, o; no sink field).HuggingFaceTB/SmolLM2-135M-Instruct: 53/64. The 1.9 pack scores 53/64, and RTN int4 g32 of the HF model scores 47/64.-j 6): 337/339. The two failures don't involve this change; I did not re-run them on a clean main.submodule_pin_checkreads my localthird_party/llama.cppcheckout.npu_rt_shimis Not Run because its target fails to link here (undefinednpu_rt_api::ext::bo).tools/copyright.py --check: rc 0.🤖 Generated with Claude Code