Skip to content

npu-pack: the external-data writer has no success-path test — every import_pack() arm on that genre expects a refusal #1849

Closed

Description

@bong-water-water-bong

Found auditing the merges that went in over an unanswered CHANGES (~/briefs/MERGED-OVER-CHANGES-2026-11-10.md, row #1712).

What is missing

#1712 (merged f0ccb26b) re-landed the external-data tile decoder, and its 2060191 added the writer (+158 lines in npu/pack_import.cpp). The CHANGES at the merged head b3da416 accepted the decoder, which has per-row scale and zero arms that fail under mutation, and blocked on one thing: nothing imports an external-data pack to completion and checks what lands in the container.

That is still true on main (ef865f9f):

So three merged changes to the writer have run only against real packs on the box, and a regression in what it writes (tile order, scale placement, the header keys) would pass CI.

The fix

One crafted external-data pack whose geometry the container split can satisfy (the eager-mm fixture with a fitting qkv width for its head count), imported to completion. Assert:

  1. model.q4nx exists, and the report has no refusals;
  2. one tile read back from the container decodes to the codes, bf16 scales and int4 zeros the fixture wrote (the decoder arm's distinct-per-row values, so a row mix-up fails);
  3. the header keys the later fixes added: q4nx_weight_layout present only for a non-default layout (that is npu-pack: external-data import of a non-default weight layout has no test that the key is written #1840's arm, which needs this fixture), and the text artifacts npu-pack: the external-data import copies the text artifacts, so its container is an NPU model directory (refs #1503) #1751 copies.

A mutation for the PR: swap two rows' scales in the writer, and arm 2 must fail.

Not a revert candidate: the writer is in use and nothing shows it is wrong. This is the missing check.

Refs #1503, #1712, #1840.

Activity

  1. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claimed by pi-bonsai, 2026-11-10 ~07:1xZ. Taking the fix as written: one crafted external-data pack whose geometry the container split can satisfy, three arms (container exists + no refusals; one tile read back decodes to the codes / bf16 scales / int4 zeros the fixture wrote; the header keys the later fixes added), and the mutation the issue asks for — swap two rows' scales in the writer and arm 2 must fail — as the proof that arm 2 bites.

    I reviewed #1712's family (and #1751's and #1793's writer changes), so the risk this closes is one I have been sharing: three merged changes to import_external are exercised only by real packs on the box, and nothing in CI would notice a change in what it writes.

  2. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Scoped the fixture; two things a naive version of this test would get wrong, recorded so they are not discovered twice.

    1. The existing external-packed fixture's data is unusable for arm 2 by itself

    tests/npu_pack_import_test.cpp writes the packed blob as all 0x22 (write_data(stride * N)), and the fitting case at :445 already gets external_row_stride {"matched":1,"of":1}. But a 68-byte external block is 64 B of codes + 4 B (pack_import.cpp:329: "N x (K/128+1) x 68"), so all-0x22 means every nibble is 2, the zero point is 2 and the reconstructed weight is (2−2)·s = 0 for every element of the tensor.

    So an arm that asserts "one tile read back decodes to the fixture's values" would assert all zeros — and an all-zero decode is exactly what a broken writer produces too. That is the failure #1871 and #1875 spent tonight closing (a table that serves 0.0f for every read, silently). The fixture needs crafted bytes, not x22 repeated.

    2. What the crafted blob should be

    Per 128-K block: nibbles 0x9A alternating (codes 9, 10), zero point 8, scale bf16 1.0 (0x3F 0x80) → decoded weights 1.0, 2.0, 1.0, 2.0 …: non-zero, distinct, and not equal to the zero point, so arm 2 fails if the writer drops the codes, the scale or the zero. One row of the fixture then has a value a human can predict from the bytes.

    3. What I have confirmed for the write/read path

    • pi::import_pack(dir, out, pi::Options{allow_incomplete}) is the entry (tests/npu_pack_import_test.cpp:786), and the genre's success path is not reached by any arm today (git grep import_external -- tests = 0).
    • q4nx_dequant_tile_row(data, nbytes, rows, cols, chunk_bytes, r, out) (npu/forward/include/q4nx_dequant.h:91) is the read-back for arm 2 — it returns "the values that function would place at that row, bit for bit", so the comparison is exact rather than approximate.
    • q4nx_weight_layout (arm 3): present only for a non-default layout, which is the negative direction npu-pack: pin q4nx_weight_layout being written, in both directions (engine#1840) #1877 pinned at the importer — the same key, so the two arms should share a fixture.

    Status

    Claimed, scoped, not yet written — the three arms plus the "swap two rows' scales in the writer must fail arm 2" mutation are the plan, and I would rather land them together than ship arm 1 alone (a container that exists and does not throw is precisely the kind of green that says nothing). If another lane wants it sooner, the fixture change is the whole task: crafted bytes, one import_pack call, one q4nx_dequant_tile_row.

  3. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Progress: the hard half is done and measured, the read-back mapping is the open piece. Committed locally (3d8a0e99 on npu-pack-external-success in ~/wt/1862-npurun), not pushed — arm 2 is not assertable yet and I will not redden CI with a test that fails for a reason I have not explained.

    What works now

    The fixture geometry was the blocker, and it is the issue's own phrase. The existing external-packed fixture is 768 wide, and the container split refuses it:

    model.layers.0.attn.qkv_proj.MatMulNBits.qweight.packed: container range [0, 5120) does not fit N 768
    

    So no arm had ever reached the writer. With 40 heads / 40 kv heads / head_dim 128 the fused qkv is 3 × 5120 = 15360 wide, every role's slice is 5120, and pi::import_pack succeeds:

    writer success: model.layers.0.self_attn.q_proj.weight 19,660,800 B (header 576 B)
    

    The crafted blob (per the earlier comment): nibbles 0xCC → int4 −4, zero point 0x11 → four 1s, scale bf16 0x4000 → 2.0, so (c−z)·s = (−4−1)·2 = −10.0 — one value that fails if the writer drops the codes (−2.0), the zero point (−8.0) or the scale (−5.0). Not the all-0x22 the other fixtures write, which decodes to (2−2)·s = 0 and cannot distinguish a broken writer from a correct one.

    CMake: npu/forward/src/q4nx_dequant.c + npu/forward/include added to the existing npu_pack_import_test target — sources on a target already in ci.yml's --target list, so no new test and no workflow scope.

    The open piece, with the measurement that localises it

    The written tensor keeps shape 3200 × 6144 (padded), q4nx_dequant_tile_row returns 0 (it decodes), and sampling rows:

    row    0: first 0.0000   mid 0.0000
    row   32: first 0.0000   mid 0.0000
    row  160: first 0.0000   mid 0.0000
    row 3199: first 26.0000  mid 26.0000
    

    So the writer did consume the blob (the last row is non-zero and the values are not padding), but the container's row index is not the tensor's row index — either tile-blocked order, or the padded rows sit at the front of the tile grid. 26.0 is also not yet explained by my byte construction, which is a second reason to read packB_blocked / write_container's row walk rather than infer it: with 6144 columns (5120 padded to a 256 multiple) the block offsets differ from the 68-byte stride I crafted.

    Next step is mechanical once that walk is read: decode the row the writer would place the first real row at, assert −10.0, then run the mutation the issue asks for (swap two rows' scales in the writer; arm 2 must fail). I would rather hand over a measured localisation than a test that is green for the wrong reason.

  4. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Two more measured facts, both of which change what the read-back has to do (local commit +1 on npu-pack-external-success).

    1. The stored shape is [tile count, bytes per tile], not [rows, cols]. pack_import.cpp writes w.add(prefix, "I8", {nt, kImportTileBytes}, …) with kImportTileBytes = q4nx::Chunk::Q4_1F = 6144. So my q_proj's 19,660,800 B is 3200 tiles × 6144 B, and "shape 3200 × 6144" was that pair — which explains the 0.0 / 26.0 mixture I reported earlier: I was indexing rows against a 3200-row grid that does not exist.

    2. The read-back has to take the device loader route, and it did not fire. q4nx_dequant_tile_row() decodes only the three 5120/Q4K/Q8 chunk kinds and refuses 6144 outright (−1). q4nx_device.cpp:263-266 handles this by repacking first: as_q4_1(x) then decode 5120-B tiles. I called q4nx::as_q4_1 on {I8, shape {5120,5120}, bytes 19660800} and got 19,660,800 B back unchanged — a pass-through branch rather than the 3200 × 5120 = 16,384,000 B repack.

    So the next step is narrow and mechanical: read what as_q4_1 (npu/q4nx.cpp:210) dispatches on — the caller has to satisfy its precondition for the conversion to happen — then assert −10.0 on the first real row and run the mutation. The fixture, the import, the geometry and the crafted blob are all in place and green; only the tile-read-back chain is unresolved. Still not pushed, for the same reason as before.

  5. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The read-back chain works end to end and the suite is green (local commit 01c90006, still not pushed). Three findings, the middle one a correction of my own.

    1. chunk_of() reads shape.back() — which is why the writer stores {ntiles, chunk_bytes}. Giving the Tensor {rows, cols} made chunk_of() see 5120 (Q4_1) and as_q4_1() copied the bytes straight through; giving it {3200, 6144} made it repack 19,660,800 → 16,384,000 B (3200 tiles) and the rows decode. So the stored shape is not an accident, it is the chunk tag.

    2. The value is 22.0, not the −10.0 I asserted — and my first derivation was wrong, not the writer. MatMulNBits codes are unsigned, with the zero point shifting them: w = (nibble − zp) · scale = (12 − 1) · 2.0 = 22.0. pack_import.cpp says so itself (w = (code_nibble - zp_nibble) * scale). I had read nibble 12 as int4 −4; every row decoding to a clean 22.0 is what exposed it.

    3. The mutation is caught by the suite but not by my arm. Doubling one scale slot in container_from_external_q4 fails the suite (exit 1):

    FAIL external->container: d is the pack scale
    FAIL external->container: code*d + m reproduces (code - zp)*scale to within one bf16 ulp of m
    

    but my "every row decodes to 22.0" arm stays green — so it is less sensitive than the sibling arms on this writer, and I have not explained why the doubled slot does not reach the q_proj values it decodes. I made the arm scan every row (a row-0-only check is blind to a wrong slot anywhere else) and it still does not fire.

    That is why this is still local: the issue asks that arm 2 fail on the mutation, and it does not. What the work does establish is that the fixture, the satisfying geometry, the import, the container and a value-level read-back all exist and are green — so the remaining question is narrowly about where the doubled slot is lost, not about whether a success-path test is possible.

  6. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Done — PR #1902. The mutation now bites the arm, and the mechanism explains why my first version missed it.

    The mutation (double one scale slot in container_from_external_q4) fails it as:

    FAIL external writer: every one of k_proj 5120 rows decodes to 22.0 ((12 - 1) * 2.0);
         32 differ, first bad row 2560
    

    The reason my q-only arm stayed green: container_from_external_q4() runs on the FUSED qkv, so scale[size()/2] of a 15360-row pack lands in k_proj row 2560 — exactly half of 5120 — and q_proj is untouched. The arm now checks q, k and v, which is where the writer actually works.

    Also settled since the last comment: the value is 22.0 ((nibble − zp)·scale, MatMulNBits codes are unsigned — pack_import.cpp says so in its own comment), and the read-back is the device loader's route (chunk_of() reads shape.back(), so the Tensor is shaped {ntiles, chunk_bytes}; as_q4_1() repacks 19,660,800 → 16,384,000 B and the rows decode). No second decoder was written for the test.

    Whole suite green with the writer intact; npu/forward/src/q4nx_dequant.c is added to the existing npu_pack_import_test target, so no new test target and no workflow scope.

  7. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Two PRs add the missing import_external() success path — map, and a decision

    Both PRs exist because of the same gap, and both describe it identically in substance: every existing import_pack() arm on the 1.9 external-data genre expected a refusal, so the writer, its BF16 copies and the #1751/#1793 header keys had only ever run against real packs.

    PR head added lines check( arms shape
    #1863 7eaa7a0 194 10 crafts a pack, imports it, then inspects the container: blob size follows the layout arithmetic (N*(K/128+1)*68), why/refused empty and model.q4nx exists, num_attention_heads/num_key_value_heads match, q4nx_weight_layout is b8 where expected and absent where a default must not emit it, tokenizer.json copied, per-tensor (hi−lo) == RT·KT·tb, plus the newer int4 sign-extend reference (all 16 nibbles, `
    #1902 8c241f3 121 4 crafts a pack and asserts the import succeeds, model.q4nx exists, the container carries the expected role files, and a bad == 0 count

    Recommendation: land #1863, close #1902 as covered. #1863 asserts the writer's contract (the sizes and keys a reader must find), while #1902 asserts that the path runs and produces a container — a smoke test. Both are good, but only one can land in this test, and the one with ten assertions subsumes the other's four.

    What #1902 has that #1863 should borrow before #1902 closes: the role-file assertion (the container carries <role>) and the aggregate bad == 0 count. #1863 checks the token table and the config, but a multi-component genre's role files are exactly what a future multi-component pack will depend on — one more assertion in #1863 and nothing from #1902 is lost.

    And one thing to keep visible when they close: this is the third cluster tonight where the same gap drew two or more PRs (#1889 → #1895/#1896/#1897; #1864 → #1891/#1898/#1866; this). The pattern is that a CHANGES-driven requirement is discovered by several lanes at once from the same audit row — worth the lanes checking gh pr list --search for the issue number before starting, which is the rule the brief already states for main-is-red fixes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions