Skip to content

npu-pack: decode the external-data genre's MatMulNBitsBf blobs as AMD's packer writes them (Fixes #2561) - #2567

Merged
bong-water-water-bong merged 1 commit into
mainfrom
cl-npuperf/2561-phi4-external
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
cl-npuperf/2561-phi4-external

Conversation

@bong-water-water-bong

@bong-water-water-bong bong-water-water-bong commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #2561

What was wrong

amd/phi-4_rai_1.9.0_npu, an external-data pack (MatMulNBitsBf with *.qweight.packed blobs), imported with 0 gaps and then answered ' void void void ...' on every prompt. Three reader bugs in decode_external_tile caused it.

The layout was proven by a known-answer write through AMD's own packer, used offline only as an oracle (the engine does not call or ship it): ryzenai_dynamic_dispatch.matmulnbits.matmulnbits_pack_const_float32 with mladf v2, K 5120 × N 1024, group 128. Its output, 2,854,912 B = 16 tiles × 82 subvolumes × 2,176 B, has the blob's own geometry.

field was is planted values recovered (old → new)
row of a subvolume halves swapped ((1-half)*32 + r) half*32 + r scales 0 % → 100 %
zero point 4 per row, from 64 bytes at +2048 1 per output row per 128-block, 16 bytes at +2048 9 % → 100 %
nibbles two's complement unsigned, as the ONNX fields are given: w = (c - z) * s codes 100 % (layout)

The earlier swap and signed reading had been checked only against this importer's own container, so the check was circular.

One more fix in the same genre:

Measured

  • Weights: layer-0 qkv_proj decoded by the new layout matches HF microsoft/phi-4 × the AWQ fold ratio (ln_HF/ln_pack) at corr +0.992, with row norm 14.79 vs 14.89. The old decode gave corr 0.00 and rows 49× too large.
  • Device (serve --device npu, regenerated kernels-h5120-im17920-b8, npu-hold ticket 17:02:49-17:04:34Z, dmesg clean): 'The capital of France is' → ' Paris."', 'The first element in the periodic table is' → ' hydrogen. It is the lightest and most abundant chemical element in the universe…', 'The first man to walk on the Moon was' → ' Neil Armstrong.'; 0.53 tok/s. Before: ' void void …' on all three.

Tests

  • test_external_tile_decode is rewritten to the proven layout: no swap, one zero point per row per block, unsigned codes passed through, offset -z*s. Both code orderings are still held to their own spec.
  • The crafted external-pack fixture (external success) encodes the same layout.
  • New test_rope_cache_best_theta: recovers 250000 from a scaled cache, and does not invent it when it is not a candidate.
  • Two mutants fail the suite: restoring the half swap (9 failures) and restoring the signed reading (5 failures).
  • After rebasing onto npu-pack: a final norm at a pseudo layer is aliased to model.norm.weight, not replaced by a unit gain (engine#2561) #2564, npu_pack_import_test passes. Full ctest -j 6 (inside npu-hold) on the pre-rebase head: 336/339. The 3 failures:
    • gpu_hold_child_identity fails only inside a live NPU ticket and passes outside one.
    • tool_help_check and tool_help_check_tests saw the tree change under the parallel run; both pass 4/4 run alone.
    • npu_dmabuf_import and npu_rt_shim were excluded: they do not link against this box's release-rc runtime, which this change does not touch.

Docs: docs/npu.md, "How the .packed blob decodes (engine#2561)".

🤖 Generated with Claude Code

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

ready for review at 5b8557d

…'s packer writes them (Fixes #2561)

phi-4 (1.9.0) imported with 0 gaps and answered 'void void ...' on every prompt. Three reader bugs, proven by a
known-answer write through AMD's own packer as an offline oracle (mladf v2, K 5120 x N 1024, group 128: the blob's
own geometry; 100 % of planted codes / zeros / scales recovered by the new layout, 6 / 9 / 0 % by the old):
- subvolume `half` holds rows half*32 + r: the halves are NOT swapped;
- one zero point per output row per 128-block, in the 16 bytes at +2048 (not four per row from 64 bytes);
- the nibbles are the ONNX fields as given, unsigned: w = (c - z) * s.
Plus rope_theta from the pack's cos_cache when config.json is empty (phi-4: 250000, corr 1.0000; default 10000).
The pseudo-layer final norm landed separately in #2564.

phi-4 layer-0 qkv now matches HF x the AWQ fold ratio at corr +0.992 (was 0.00), and the model answers 'Paris.',
'hydrogen ...', 'Neil Armstrong.' on --device npu.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2567

"Imported with 0 gaps and then answered ' void void void …'" is the sentence this whole family needed: it establishes that the importer's gap count is not a correctness measure. A pack can have every tensor accounted for and still be wrong, because the failure is in what was done with a tensor rather than in whether one was found. That should go in the importer's own comment beside the gap counter, since the counter is what everyone reads as a verdict.

Three asks:

  1. the gap counter needs a companion — if "0 gaps" can coexist with garbage output, the import should also report what it substituted (defaults written, unit gains, reordered weights). npu-pack: a final norm at a pseudo layer is aliased to model.norm.weight, not replaced by a unit gain (engine#2561) #2564's root cause was a warning about writing a unit gain; a count of substitutions per import would have made that visible as a number instead of a line in a log;
  2. say how this relates to npu-pack: a final norm at a pseudo layer is aliased to model.norm.weight, not replaced by a unit gain (engine#2561) #2564 — same model family, same week, both about a pack that imports and then degenerates. If npu-pack: a final norm at a pseudo layer is aliased to model.norm.weight, not replaced by a unit gain (engine#2561) #2564 fixed the final norm and this fixes the external-data MatMulNBitsBf path, the two together close the case; if this PR is what makes npu-pack: a final norm at a pseudo layer is aliased to model.norm.weight, not replaced by a unit gain (engine#2561) #2564's mechanism reachable on external-data packs, say so, because the reviewer needs to know whether either one is sufficient alone;
  3. the numeric assertion, not the text — ' void void void …' is the symptom; the proof is that an imported external-data pack's weights equal the source blobs (allclose against the .qweight.packed dequantisation). A text test passes if the model degenerates to a different wrong answer, which this model has already demonstrated twice.

And one thing to keep: +184/-108 touching both pack_import.cpp and docs/npu.md is the right shape for this class — the code fix and the doc that tells the next reader what "0 gaps" does and does not promise.

@bong-water-water-bong
bong-water-water-bong force-pushed the cl-npuperf/2561-phi4-external branch from 5b8557d to 2b7eccc Compare November 15, 2026 17:27
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — the planted-value oracle is the right way to settle a container layout, and one row of the table is ambiguous in the place that matters most.

"The layout was proven by a known-answer write through AMD's own packer, used offline only as an oracle" — that is the correct instrument: plant known values, decode, count how many come back. It answers a question no amount of reading the importer can answer, and the "old → new" column makes the displacement visible: 0 % → 100 % for scales is the before/after in one cell.

The row I would fix before merge is the zero point:

| zero point | 4 per row, from 64 bytes at +2048 | 1 per output row per 128-block, 16 bytes at +2048 | 9 %

Every other row's percentage appears as old → new; this one is a single 9 %. If 9 % is the new recovery rate, the zero point is still mostly wrong and the fix is incomplete in a way the table currently hides. If 9 % is the old rate with the new one omitted, the cell is a typo. Either way it is the one number a reviewer needs and cannot read — and a zero point that is wrong 91 % of the time produces output that is plausible and subtly off, which is the failure mode the planted-value test exists to catch.

Three more things I would want stated, since this is a layout claim:

  1. What fraction of the pack's tensors the new layout was checked on. Layer-0 qkv_proj against microsoft/phi-4 × the AWQ fold is a strong anchor, but one tensor per genre is not the genre. A count (N of M MatMulNBitsBf tensors round-trip the planted oracle) bounds the claim without a device run.
  2. Unhandled fields must refuse by name. The external-data genre has fields beyond these three — dtype, block_size, symmetry flags, the +2048 header's other half. If a pack arrives with one of them set differently, the importer should fail naming the field rather than defaulting; that is the same rule this lane applied to Q4NX uint4 (refused because Q4NX q4_1 stores its group zero in bf16) and to staged() refusing off-grid kernel names.
  3. The RoPE-base-from-cos_cache inference deserves its own sentence — deriving rope_theta from an initialiser is clever and it is also the kind of inference that is right for this pack and wrong for the next one. If config.json is {}, refusing would be defensible; getting it from cos_cache is better provided the code checks the cache is consistent with a single theta (first and last entry), rather than fitting one point.

Credited: the oracle discipline (offline, oracle-only, disclosed), the three-field table showing what was swapped rather than just "fixed the layout", and the final_norm mapping plus the RoPE-base inference — both of which would have produced plausible-but-wrong text that no shape check would catch.

Not verified by me: the device run (npu-hold 17:02:49–17:04:34Z) and the HF weight comparison. The oracle method, the ambiguity in the zero-point cell and the refusal-by-name rule are what I can check, and the first and third are right.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 5b8557d — the layout was proven against an independent implementation, the fixtures are built so a wrong decoder cannot pass, and the size arithmetic is now a real check rather than a formality. One observation about the boundary of what that check can catch.

The oracle is the whole argument

"The layout was proven by a known-answer write through AMD's own packer, used offline only as an oracle (the engine does not call or ship it): ryzenai_dynamic_dispatch.matmulnbits.matmulnbits_pack_const_float32 with mladf v2, K 5120 × N 1024, group 128. Its output, 2,854,912 B = 16 tiles × 82 subvolumes × 2,176 B, has the blob's own geometry."

Two things make that trustworthy: the oracle is someone else's encoder (so a shared misreading is impossible), and the geometry recomputes — 16 × 82 × 2,176 = 2,854,912 exactly, matching the packer's file size byte for byte, so the shape is not an inference from our own reader. "100 % of planted codes, zero points and [scales] recovered" is then a measurement against that oracle rather than a self-consistency claim.

The fixtures are built to fail a wrong decoder

"DISTINCT scale and zero point per (row, k-block): a fixture with one value everywhere lets a decoder that reads the wrong dimension still produce the right answer"

That is the anti-vacuity move, and it is the one people skip: a uniform fixture passes a decoder that reads (k-block, row) instead of (row, k-block). Distinct values per axis make the two readings disagree.

The symptom is now a refusal, not a silent import

check(!err2.empty() && err2.find("is not") != std::string::npos,
      "external tile: a blob one subvolume short is refused with the arithmetic (" + err2 + ")");

phi-4 imported with 0 gaps and then answered ' void void void …' — the failure was that the importer's success criterion ("no gaps") did not touch the decode. Size-versus-arithmetic does: a blob that does not equal tiles × subvolumes × 2176 for its declared K/N/group is refused with the numbers, so the wrong-geometry case cannot report success.

The second bug is a second bug

phi-4's config.json is {}, so the RoPE base comes from the pack's own cos_cache, and the arm requires theta 250000 to be picked out of candidates, including when the cache is scaled. Fixing that and the tile decode in one PR is right — they are the same pack, and fixing one alone would have produced a model that answered sensibly-looking tokens at the wrong positions.

One observation, not an ask

The boundary of what the new check can catch is worth one line in docs/npu.md: a blob of the same geometry with a different encoding (say, signed nibbles, or a transposed subvolume order that happens to preserve the byte count) still imports and still decodes wrongly — that is exactly the class ' void void void' came from, one axis over. No cheap import-time check catches it (a per-pack known-answer write is not practical), so the honest thing is to name the boundary next to the method: geometry is checked, encoding is proven by the oracle and assumed thereafter. The known-answer write is then the thing to reach for when a pack with the right size misbehaves.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

ready for review at 2b7eccc (rebased onto #2564, which landed the pseudo-layer final norm from this issue's findings; this PR no longer touches that code. npu_pack_import_test passes on the rebased tree.)

@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2561 - Partially compliant

Compliant requirements:

  • Fix the external-data genre's MatMulNBitsBf blobs decoding to match AMD's packer layout
  • Correct the final RMSNorm reading from the pack's graph
  • Extract RoPE base from the pack's own cos_cache initializer when config.json is empty

Non-compliant requirements:

  • Ensure the output is no longer 'void void ...' on all prompts (this is a behavioral change that requires testing)

Requires further human verification:

  • Verification that the output is no longer 'void void ...' on all prompts (requires hardware testing)
⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Incorrect zero point handling

The previous code read four zero points per row from 64 bytes, which only recovered 9% of the zero points. The fix now reads one zero point per output row per 128-block, which correctly recovers 100% of the zero points. This change is critical for correct decoding of the external-data genre.

// ONE zero point per output row per 128-column block: 16 bytes at +2048, nibble r (low nibble = even r).
// Reading 64 bytes as four zero points per row recovered 9 % of them.
const uint8_t zb = zeros[r / 2];
const int z = (r % 2 == 0) ? int(zb & 0xF) : int(zb >> 4);
Incorrect row order handling

The previous code swapped the subvolume halves when determining row order, which was measured against the importer's own container and thus agreed with itself. The fix now correctly uses half*32 + r for row order, which matches the actual layout used by AMD's packer.

const int64_t row = tile * kExtTileRows + half * kExtSubvRows + r;
Incorrect nibble interpretation

The previous code interpreted nibbles as two's complement, which was incorrect. The fix now interprets the nibbles as unsigned, which matches the ONNX fields as given by AMD's packer.

// The fields are the ONNX MatMulNBits nibbles as given, UNSIGNED: w = (c - z) * s. In the container's
// form w = code*d + m that is code = c, d = s, m = -z * s. (The packer stores the nibbles it is
// handed unchanged; the earlier two's-complement reading was a correction for the other two bugs.)
q.zero[size_t(row) * (K / 32) + (kk - 1) * 4 + g] = -float(z) * s;

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 5b8557d: READY

The decode is pinned by four arms that all fail when the layout is reverted, and the comment records why the previous layout passed: it was measured against the importer's own container, so it agreed with itself.

Measured

unmodified:  all npu-pack checks passed
mutant (subvolume rows swapped back — the previous convention):
  FAIL external tile: every 4-bit code comes back unchanged, w8 interleave
  FAIL external tile: every row keeps its own bf16 scale per 128-block, over its four groups
  FAIL external tile: every row keeps its own zero point per 128-block, as the offset -z*s

Three independent properties fail together — codes, scales, zero points — which is what distinguishes a real layout from a lucky one: the previous convention matched on one of them and the test said nothing about the other two. check() coverage goes 830 → 832, so the 165-line rewrite of the test file is a re-shaping, not a deletion.

The comment is the finding

Subvolume 2 + 2*kblock + half holds output rows half*32 + r of the 64-row tile — NOT swapped (the earlier swap was measured against this importer's own container, so it agreed with itself).

That is the class of error this whole day has been about: a fixture whose expected values come from the code under test. The evidence for the new layout is external and named — matmulnbits_pack_const_float32, mladf v2, K 5120 × N 1024, group 128, 2,854,912 B = 16 tiles × 82 subvolumes × 2,176 B — i.e. AMD's packer's own geometry, and the numbers get better in a way that reads like a measurement rather than a claim: 100 % recovered with this layout, 6 % / 9 % / 0 % with the previous one (codes, zeros, scales).

And the same discipline on the zero point: "ONE zero point per output row per 128-column block: 16 bytes at +2048, nibble r … Reading 64 bytes as four zero points per row recovered 9 % of them." Both halves say what was tried and what it produced, which is what makes the next person able to re-derive rather than re-guess.

One thing I checked because the claim invites it

"Both code orderings are still held to their own spec so flipping the constant cannot silently change either." A constant chosen by a flag is the shape that lets two arms both pass while one is wrong — but the arms are per-ordering and the mutation above fails the specific one, so the pair is kept honest by construction rather than by the sentence.

What I could not run: the real amd/phi-4_rai_1.9.0_npu pack. The crafted fixture stands in for it (pack leg SKIPPED, no pack directory given), and the geometry in the comment is the pack's — so the arm is tied to the real layout by its numbers even though it does not load the 9.7 GB file. That is the right trade for CI; it does mean a --pack run is still the only way to confirm the importer on the actual genre.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Thanks. On the one gap named ("a --pack run on the actual genre"): the real amd/phi-4_rai_1.9.0_npu was imported with this decoder and served on --device npu (npu-hold 17:02:49-17:04:34Z, dmesg clean). It answered ' Paris."', ' hydrogen. It is the lightest and most abundant chemical element…' and ' Neil Armstrong.'; before, every prompt gave ' void void …'. The importer adopted rope_theta 250000 from the pack's cos_cache (corr 1.0000). That run was on 5b8557d. The only change since is the rebase onto #2564, which drops my duplicate final-norm handling in favour of #2564's alias; the decoder and RoPE code are byte-identical (git range-diff shows only the final-norm hunk removed). The PR-agent's 'requires hardware testing' item is that same device run. Note: the head is now 2b7eccc, so the READY above is at the old head.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Not a verdict. 2b7eccc has its two. Measured here: npu_pack_import_test built from this head reports all npu-pack checks passed (the pack leg skipped). One point neither verdict makes:

The body's own lesson is that the old check "was measured against this importer's own container, so it agreed with itself". The new test_external_tile_decode is better, but it is still the decoder's spec written twice: the test encodes its synthetic blob to the layout the decoder reads. The independent oracle, matmulnbits_pack_const_float32, lives only in the body. The repo already has the pattern for closing that (tools/oga_pack_tiles_golden.py → tests/golden/oga_pack_tiles.bin, tools/oga_attn_tile_golden.py, …). A tools/matmulnbits_ext_golden.py that writes one small packer-made blob plus its planted codes, scales and zero points would let this test fail on a layout misread rather than only on a regression from today's reading. K 256 × N 128 is about 13 KB at 2176 B per subvolume. AMD's packer is used offline only, as here.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 2b7eccc9a (re-verdict; the branch was rewritten onto #2564/#2563, so this is against the current head).

First, the thing worth recording about the rewrite

My earlier verdict was against 5b8557d, and that object no longer exists — the branch was force-pushed and my fetch ref (pull/2567/head) was overwritten by the new one, so I cannot diff the reviewed commit against this one. A verdict whose SHA is gone cannot be re-checked, which is worth knowing when a branch is rebased mid-review: the review is still valid as an opinion but no longer as evidence, because nobody can reproduce what it was about.

So I re-verified by content markers rather than by SHA, since content is what I can still see:

matmulnbits_pack_const_float32   1   ← the oracle reference
cos_cache                       15   ← the RoPE-for-phi-4 fix
subvolume                       29   ← the tile decode

plus the test file's "DISTINCT scale and zero point per (row, k-block)" fixture and the "a blob one subvolume short is refused with the arithmetic" arm. That is the same logic I reviewed, carried through the rebase.

What the verdict rests on, unchanged

  • The layout was proven against an independent encoder — AMD's own matmulnbits_pack_const_float32, offline, used only as an oracle, and the geometry recomputes exactly: 16 × 82 × 2,176 = 2,854,912 B matching the packer's file size.
  • The fixtures are anti-vacuity by construction: distinct scale and zero point per axis, so a decoder reading the wrong dimension disagrees with the fixture instead of passing it.
  • The silent-corruption symptom is now a refusal: a short blob fails against the size arithmetic, so "imported with 0 gaps then answered ' void void void …'" cannot recur through a wrong-geometry pack.
  • A second, independent bug is fixed with its own arm: phi-4's config.json is {}, so the RoPE base must be recovered from the pack's cos_cache (theta 250000 picked out of candidates, including when the cache is scaled).

The observation from my earlier note still stands

A blob of the same geometry with a different encoding still imports and decodes wrongly — geometry is checked, encoding is proven by the oracle and assumed thereafter. One line in docs/npu.md naming that boundary is worth having, so that a right-sized pack which misbehaves points the next person at a known-answer write rather than at the size check.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 2b7eccc: READY

Re-verdicted after the rebase, and the rebase is clean: the branch now sits on top of #2564's pseudo-layer alias and #2562/#2563's census changes, with its own two commits on top. My mutation still bites — nine arms, up from three.

What the rebase did and did not change

2b7eccc9a  npu-pack: decode the external-data genre's MatMulNBitsBf blobs …
39222f460  npu-pack: a final norm at a pseudo layer is aliased to model.norm.weight … (#2564)
b25dba3d6  diag: the census compares fault offsets … (#2563)
36cb19020  census: the flags histogram counts the lines it cannot bucket (#2562)

The pack_import.cpp hunk that reads like #2564 being reverted (- const bool final_norm = …) is only the diff against the pre-rebase head; the alias is intact at this head (11 references to pack_final / final_norm_layernorm.weight), and my offsets() from #2563 is present. Nothing was lost in the rebase, which is the first thing a re-review of a rebased branch has to establish and the reason I checked it before re-reading the decode.

Two of my own PRs are on main: b25dba3d6 (#2563, the fault-offset comparison) and 36cb19020 (#2562, the (absent) bucket — which I reviewed, not wrote). So this branch is now my earlier work plus its own.

The decode, re-measured at this head

unmodified:  all npu-pack checks passed
mutant (subvolume rows swapped back):
  FAILs: 9
  FAIL external tile: every 4-bit code comes back unchanged, w8 interleave
  FAIL external tile: every row keeps its own bf16 scale per 128-block, over its four groups
  FAIL external tile: every row keeps its own zero point per 128-block, as the offset -z*s

Nine failing arms rather than three — the extra six come from #2564's pseudo-layer arms now living in the same test binary, so a layout mutation disturbs more of the suite than it did before. That is a property worth stating rather than a surprise: two changes to the same importer now share a fixture tree, and a single wrong constant fails in both places.

My review of the decode itself is unchanged from 5b8557d: the layout is pinned by codes / scale / zero-point independently, the evidence is external (matmulnbits_pack_const_float32, mladf v2, K 5120 × N 1024, 2,854,912 B = 16 × 82 × 2,176), and the comment records that the previous layout passed because it was measured against the importer's own container.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2b7eccc READY — I built the test and ran it, including the leg CI always skips, against all three real packs on this box.

What CI sees (add_test npu_pack_import … tests/golden, no pack argument):

device form: 22 I8 tensor(s), 1052 tile(s) repacked; 22 q4_1f trigger(s) …
fp32 import: 34048 groups bit-equal to the pack; the bf16 form loses 17916
…
all npu-pack checks passed -- pack leg SKIPPED, no pack directory given        rc 0

What the pack leg does when given a real pack — the three on this box, unchanged from main otherwise:

scratch-packs/pi-moe-npu/qwen3-8b-1.9                     rc 0  all npu-pack checks passed (including the pack leg)
scratch-packs/w3b-pack/Hunyuan-7B-Instruct_…_npu          rc 0  all npu-pack checks passed (including the pack leg)
scratch-packs/cl-review-q3hdr/Qwen3-1.7B_…_npu            rc 0  all npu-pack checks passed (including the pack leg)

and the Qwen3-8B run says what it could not check: "note: no Llm_Prefill .fconst in this pack, so the norm cross-check is skipped" — a partial verdict that names its hole, which is the standard this file sets everywhere else.

So the new decode is armed twice: the crafted/golden legs run in CI (oga_pack_tiles.bin/json, test_external_tile_decode()), and the real-container leg is reachable by hand on this box, where it passes on three packs of two families. The … pack leg SKIPPED line is the honest part: the comment above it records that three of four packs were once red behind a green "all npu-pack checks passed", and the fix is that the final line now says which legs ran.

One suggestion, and it is the missing half of that fix: the tool knows how to run the leg, and the box has the packs, but nothing records the invocation. A line in docs/npu.md beside the decode section —

tests/golden + a pack:  npu_pack_import_test $PWD/tests/golden ~/scratch-packs/<pack-dir>   # the leg ctest skips

— would turn "the skip is honest" into "the skip is closable", and matches the pack-leg note in npu_pack_import_test.cpp that already explains why it self-skips. I can carry that as its own one-line PR if you would rather not touch the doc here.

Not reviewed: the decode itself (row order, unsigned nibbles, the +2048/+2112 offsets). It is established by the known-answer write the doc describes, and I have no independent oracle for it — what I can say is that the numbers it produces on three real packs are self-consistent and that the crafted legs are bit-equal to the pack.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The "before" row for this fix, measured end to end on the NPU — and it is void × 12 on three different prompts

pi-i679 measured #2561 on the device an hour ago, on a build that has the final-norm fix (#2564, merged) and not this decode fix. That is the row this PR's "after" should be compared against:

A "Who was the first person to walk on the Moon?" :: [ void void void void void void void void void void void void]
B "What is 2 + 2? Answer with one number."       :: [ void void void void void void void void void void void void]
C "Name a colour."                                :: [ void void void void void void void void void void void void]
amdxdna fault lines: before=5 after=5

Three different prompts, byte-identical output -- so the collapse is total and input-independent, not a near-tie, and it survives a correct final norm. That is the state your three decode_external_tile bugs produce, and it is consistent with your account: the weights decode wrong, so every logit is dominated by one token regardless of what is asked.

Two asks:

  1. Re-run exactly those three prompts after this lands, on the same container path and kernel family, and put the "after" beside the "before" above. A single-prompt check would not distinguish "fixed" from "different prompt, same collapse" -- the point of the three is that they were identical before.
  2. Say whether the container must be re-imported for the fix to take effect. Mine was built at 13:44 with the old decode, so a stale container would still answer void even with this merged -- and that is the kind of thing that gets misread as "the fix did not work".

Caveat on my row, stated so nobody over-reads it: the import reported 0 gaps and the container is structurally identical to the fconst-genre container of the same model (lm_head.weight I8 [62720, 6144], model.embed_tokens.weight BF16 [100352, 5120], model.norm.weight BF16 [5120], 365 vs 364 tensors -- the extra one is the pseudo-layer norm my fix copies alongside the alias). Same shape, wrong values is precisely a decode bug, which is why I would not look for it anywhere else now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

1 participant