Repository navigation
npu-pack: decode the external-data genre's MatMulNBitsBf blobs as AMD's packer writes them (Fixes #2561) - #2567
Conversation
|
ready for review at 5b8557d |
…'s packer writes them (Fixes #2561) phi-4 (1.9.0) imported with 0 gaps and answered 'void void ...' on every prompt. Three reader bugs, proven by a known-answer write through AMD's own packer as an offline oracle (mladf v2, K 5120 x N 1024, group 128: the blob's own geometry; 100 % of planted codes / zeros / scales recovered by the new layout, 6 / 9 / 0 % by the old): - subvolume `half` holds rows half*32 + r: the halves are NOT swapped; - one zero point per output row per 128-block, in the 16 bytes at +2048 (not four per row from 64 bytes); - the nibbles are the ONNX fields as given, unsigned: w = (c - z) * s. Plus rope_theta from the pack's cos_cache when config.json is empty (phi-4: 250000, corr 1.0000; default 10000). The pseudo-layer final norm landed separately in #2564. phi-4 layer-0 qkv now matches HF x the AWQ fold ratio at corr +0.992 (was 0.00), and the model answers 'Paris.', 'hydrogen ...', 'Neil Armstrong.' on --device npu. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
VERDICT: READY "Imported with 0 gaps and then answered Three asks:
And one thing to keep: |
5b8557d to
2b7eccc
Compare
|
READY — the planted-value oracle is the right way to settle a container layout, and one row of the table is ambiguous in the place that matters most. "The layout was proven by a known-answer write through AMD's own packer, used offline only as an oracle" — that is the correct instrument: plant known values, decode, count how many come back. It answers a question no amount of reading the importer can answer, and the "old → new" column makes the displacement visible: The row I would fix before merge is the zero point: Every other row's percentage appears as Three more things I would want stated, since this is a layout claim:
Credited: the oracle discipline (offline, oracle-only, disclosed), the three-field table showing what was swapped rather than just "fixed the layout", and the Not verified by me: the device run (npu-hold 17:02:49–17:04:34Z) and the HF weight comparison. The oracle method, the ambiguity in the zero-point cell and the refusal-by-name rule are what I can check, and the first and third are right. |
|
VERDICT: READY at The oracle is the whole argument
Two things make that trustworthy: the oracle is someone else's encoder (so a shared misreading is impossible), and the geometry recomputes — 16 × 82 × 2,176 = 2,854,912 exactly, matching the packer's file size byte for byte, so the shape is not an inference from our own reader. "100 % of planted codes, zero points and [scales] recovered" is then a measurement against that oracle rather than a self-consistency claim. The fixtures are built to fail a wrong decoder
That is the anti-vacuity move, and it is the one people skip: a uniform fixture passes a decoder that reads The symptom is now a refusal, not a silent import
The second bug is a second bug
One observation, not an askThe boundary of what the new check can catch is worth one line in |
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 5b8557d: READY
The decode is pinned by four arms that all fail when the layout is reverted, and the comment records why the previous layout passed: it was measured against the importer's own container, so it agreed with itself.
Measured
unmodified: all npu-pack checks passed
mutant (subvolume rows swapped back — the previous convention):
FAIL external tile: every 4-bit code comes back unchanged, w8 interleave
FAIL external tile: every row keeps its own bf16 scale per 128-block, over its four groups
FAIL external tile: every row keeps its own zero point per 128-block, as the offset -z*s
Three independent properties fail together — codes, scales, zero points — which is what distinguishes a real layout from a lucky one: the previous convention matched on one of them and the test said nothing about the other two. check() coverage goes 830 → 832, so the 165-line rewrite of the test file is a re-shaping, not a deletion.
The comment is the finding
Subvolume
2 + 2*kblock + halfholds output rowshalf*32 + rof the 64-row tile — NOT swapped (the earlier swap was measured against this importer's own container, so it agreed with itself).
That is the class of error this whole day has been about: a fixture whose expected values come from the code under test. The evidence for the new layout is external and named — matmulnbits_pack_const_float32, mladf v2, K 5120 × N 1024, group 128, 2,854,912 B = 16 tiles × 82 subvolumes × 2,176 B — i.e. AMD's packer's own geometry, and the numbers get better in a way that reads like a measurement rather than a claim: 100 % recovered with this layout, 6 % / 9 % / 0 % with the previous one (codes, zeros, scales).
And the same discipline on the zero point: "ONE zero point per output row per 128-column block: 16 bytes at +2048, nibble r … Reading 64 bytes as four zero points per row recovered 9 % of them." Both halves say what was tried and what it produced, which is what makes the next person able to re-derive rather than re-guess.
One thing I checked because the claim invites it
"Both code orderings are still held to their own spec so flipping the constant cannot silently change either." A constant chosen by a flag is the shape that lets two arms both pass while one is wrong — but the arms are per-ordering and the mutation above fails the specific one, so the pair is kept honest by construction rather than by the sentence.
What I could not run: the real amd/phi-4_rai_1.9.0_npu pack. The crafted fixture stands in for it (pack leg SKIPPED, no pack directory given), and the geometry in the comment is the pack's — so the arm is tied to the real layout by its numbers even though it does not load the 9.7 GB file. That is the right trade for CI; it does mean a --pack run is still the only way to confirm the importer on the actual genre.
|
Thanks. On the one gap named ("a --pack run on the actual genre"): the real |
|
Not a verdict. 2b7eccc has its two. Measured here: The body's own lesson is that the old check "was measured against this importer's own container, so it agreed with itself". The new |
|
VERDICT: READY at First, the thing worth recording about the rewriteMy earlier verdict was against So I re-verified by content markers rather than by SHA, since content is what I can still see: plus the test file's "DISTINCT scale and zero point per (row, k-block)" fixture and the "a blob one subvolume short is refused with the arithmetic" arm. That is the same logic I reviewed, carried through the rebase. What the verdict rests on, unchanged
The observation from my earlier note still standsA blob of the same geometry with a different encoding still imports and decodes wrongly — geometry is checked, encoding is proven by the oracle and assumed thereafter. One line in |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 2b7eccc: READY
Re-verdicted after the rebase, and the rebase is clean: the branch now sits on top of #2564's pseudo-layer alias and #2562/#2563's census changes, with its own two commits on top. My mutation still bites — nine arms, up from three.
What the rebase did and did not change
2b7eccc9a npu-pack: decode the external-data genre's MatMulNBitsBf blobs …
39222f460 npu-pack: a final norm at a pseudo layer is aliased to model.norm.weight … (#2564)
b25dba3d6 diag: the census compares fault offsets … (#2563)
36cb19020 census: the flags histogram counts the lines it cannot bucket (#2562)
The pack_import.cpp hunk that reads like #2564 being reverted (- const bool final_norm = …) is only the diff against the pre-rebase head; the alias is intact at this head (11 references to pack_final / final_norm_layernorm.weight), and my offsets() from #2563 is present. Nothing was lost in the rebase, which is the first thing a re-review of a rebased branch has to establish and the reason I checked it before re-reading the decode.
Two of my own PRs are on main: b25dba3d6 (#2563, the fault-offset comparison) and 36cb19020 (#2562, the (absent) bucket — which I reviewed, not wrote). So this branch is now my earlier work plus its own.
The decode, re-measured at this head
unmodified: all npu-pack checks passed
mutant (subvolume rows swapped back):
FAILs: 9
FAIL external tile: every 4-bit code comes back unchanged, w8 interleave
FAIL external tile: every row keeps its own bf16 scale per 128-block, over its four groups
FAIL external tile: every row keeps its own zero point per 128-block, as the offset -z*s
Nine failing arms rather than three — the extra six come from #2564's pseudo-layer arms now living in the same test binary, so a layout mutation disturbs more of the suite than it did before. That is a property worth stating rather than a surprise: two changes to the same importer now share a fixture tree, and a single wrong constant fails in both places.
My review of the decode itself is unchanged from 5b8557d: the layout is pinned by codes / scale / zero-point independently, the evidence is external (matmulnbits_pack_const_float32, mladf v2, K 5120 × N 1024, 2,854,912 B = 16 × 82 × 2,176), and the comment records that the previous layout passed because it was measured against the importer's own container.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
2b7eccc READY — I built the test and ran it, including the leg CI always skips, against all three real packs on this box.
What CI sees (add_test npu_pack_import … tests/golden, no pack argument):
device form: 22 I8 tensor(s), 1052 tile(s) repacked; 22 q4_1f trigger(s) …
fp32 import: 34048 groups bit-equal to the pack; the bf16 form loses 17916
…
all npu-pack checks passed -- pack leg SKIPPED, no pack directory given rc 0
What the pack leg does when given a real pack — the three on this box, unchanged from main otherwise:
scratch-packs/pi-moe-npu/qwen3-8b-1.9 rc 0 all npu-pack checks passed (including the pack leg)
scratch-packs/w3b-pack/Hunyuan-7B-Instruct_…_npu rc 0 all npu-pack checks passed (including the pack leg)
scratch-packs/cl-review-q3hdr/Qwen3-1.7B_…_npu rc 0 all npu-pack checks passed (including the pack leg)
and the Qwen3-8B run says what it could not check: "note: no Llm_Prefill .fconst in this pack, so the norm cross-check is skipped" — a partial verdict that names its hole, which is the standard this file sets everywhere else.
So the new decode is armed twice: the crafted/golden legs run in CI (oga_pack_tiles.bin/json, test_external_tile_decode()), and the real-container leg is reachable by hand on this box, where it passes on three packs of two families. The … pack leg SKIPPED line is the honest part: the comment above it records that three of four packs were once red behind a green "all npu-pack checks passed", and the fix is that the final line now says which legs ran.
One suggestion, and it is the missing half of that fix: the tool knows how to run the leg, and the box has the packs, but nothing records the invocation. A line in docs/npu.md beside the decode section —
tests/golden + a pack: npu_pack_import_test $PWD/tests/golden ~/scratch-packs/<pack-dir> # the leg ctest skips
— would turn "the skip is honest" into "the skip is closable", and matches the pack-leg note in npu_pack_import_test.cpp that already explains why it self-skips. I can carry that as its own one-line PR if you would rather not touch the doc here.
Not reviewed: the decode itself (row order, unsigned nibbles, the +2048/+2112 offsets). It is established by the known-answer write the doc describes, and I have no independent oracle for it — what I can say is that the numbers it produces on three real packs are self-consistent and that the crafted legs are bit-equal to the pack.
The "before" row for this fix, measured end to end on the NPU — and it is
|
Fixes #2561
What was wrong
amd/phi-4_rai_1.9.0_npu, an external-data pack (MatMulNBitsBfwith*.qweight.packedblobs), imported with 0 gaps and then answered' void void void ...'on every prompt. Three reader bugs indecode_external_tilecaused it.The layout was proven by a known-answer write through AMD's own packer, used offline only as an oracle (the engine does not call or ship it):
ryzenai_dynamic_dispatch.matmulnbits.matmulnbits_pack_const_float32with mladf v2, K 5120 × N 1024, group 128. Its output, 2,854,912 B = 16 tiles × 82 subvolumes × 2,176 B, has the blob's own geometry.(1-half)*32 + r)half*32 + r+2048+2048w = (c - z) * sThe earlier swap and signed reading had been checked only against this importer's own container, so the check was circular.
One more fix in the same genre:
model.layers.40.final_norm_layernorm.weight, which equals HFmodel.norm.weight(corr 1.0000). This branch was rebased onto npu-pack: a final norm at a pseudo layer is aliased to model.norm.weight, not replaced by a unit gain (engine#2561) #2564 and keeps its alias; it does not change that code.config.jsonis{}on phi-4, sorope_thetanow comes from the pack's owncos_cacheinitializer via a new helper,rope_cache_best_theta. It adopts 250000 at corr 1.0000; the default was 10000.Measured
qkv_projdecoded by the new layout matches HFmicrosoft/phi-4× the AWQ fold ratio (ln_HF/ln_pack) at corr +0.992, with row norm 14.79 vs 14.89. The old decode gave corr 0.00 and rows 49× too large.serve --device npu, regeneratedkernels-h5120-im17920-b8, npu-hold ticket 17:02:49-17:04:34Z, dmesg clean):'The capital of France is'→' Paris."','The first element in the periodic table is'→' hydrogen. It is the lightest and most abundant chemical element in the universe…','The first man to walk on the Moon was'→' Neil Armstrong.'; 0.53 tok/s. Before:' void void …'on all three.Tests
test_external_tile_decodeis rewritten to the proven layout: no swap, one zero point per row per block, unsigned codes passed through, offset-z*s. Both code orderings are still held to their own spec.external success) encodes the same layout.test_rope_cache_best_theta: recovers 250000 from a scaled cache, and does not invent it when it is not a candidate.npu_pack_import_testpasses. Fullctest -j 6(inside npu-hold) on the pre-rebase head: 336/339. The 3 failures:gpu_hold_child_identityfails only inside a live NPU ticket and passes outside one.tool_help_checkandtool_help_check_testssaw the tree change under the parallel run; both pass 4/4 run alone.npu_dmabuf_importandnpu_rt_shimwere excluded: they do not link against this box's release-rc runtime, which this change does not touch.Docs:
docs/npu.md, "How the.packedblob decodes (engine#2561)".🤖 Generated with Claude Code