Skip to content

npu-pack: the 1.9 .fconst tail head does not reproduce a tied pack's own embedding (corr 0.885, rel_err 0.987) #1693

Closed

Description

@bong-water-water-bong

npu-pack: the 1.9 .fconst tail head does not reproduce a tied pack's own embedding (corr 0.885, rel_err 0.987)

Found while fixing #1671 (the pack leg of npu_pack_import_test on the 0.5B 1.9 pack). #1675 makes the importer refuse this head by name instead of dropping it silently, so nothing wrong ships — this issue is about making the decode right, which would lift that refusal.

What is measured

amd/Qwen2.5-0.5B-Instruct_rai_1.9.0_npu (its config.json says tie_word_embeddings: true, vocab_size 151936, hidden 896), the head read from the .fconst tail with the layout attn_payload_off(H) + (V/64) × attn_tile_bytes(H):

lm_head_corr        0.8852204948414607        corr against the pack's own fp16 embedding
lmhead_rel          0.9870072916840432        sum|head-emb| / sum|emb|  -- i.e. ~1: magnitudes are about double
ctrl_corr_rowshift  0.0648                    same tiles, each head row against the NEXT embedding row (control, must fail)
zero_exact_pct      100.00                    the per-32-group scale/zero survive into the container exactly
tiles               survive decode->encode byte for byte
tail arithmetic     env + rows*tile == the tail length, exactly (so the SIZE is right)

Read together: the rows line up (the row-shift control fails, so it is not a row order or row-offset error), the stored parameters round-trip, the tile count is right, and the values are off by about a factor of two with a high-but-not-1 correlation. That points at the tail's per-value semantics rather than its geometry: the scale/zero convention inside these tiles, or the group ordering over a padded K (896 → the 1024-column grid), rather than an offset.

Why it matters beyond this pack

A tied pack is the only case where a head has a free oracle, and this is the second time a head defect has hidden behind "the pack is untied": #1189 (closed) was a phi3 pack importing strictly and completely and then producing near-flat logits, with the note that an untied head meant no check could have caught it. The importer's selfcheck_tied = corr > threshold heuristic reads the number backwards for exactly this class — a wrong decode presents itself as a property of the pack. #1675 turns that into a refusal for declared-tied packs; this issue is about the ~1.0 rel_err.

Acceptance

  • npu_pack_import_test tests/golden <the 0.5B 1.9 pack> reports the head as present and corr > 0.999 (the row-shift control still failing), or a measurement that says what the tail's true value convention is and why the fp16 embedding is not its reference;
  • the two other 1.9 packs on the box measured the same way, since a convention that fits one pack and not the others is not a convention;
  • the refusal added in npu-pack: the lm_head geometry gate called a [0]-dims placeholder a real weight, so the walk threw "no data" (refs #1671) #1675 is removed in the same change, so the head is written because it is right, not because the check learned to tolerate it.

How to reproduce

1bit npu-pack import ~/models/oga/Qwen2.5-0.5B-Instruct_rai_1.9.0_npu -o /tmp/head --allow-incomplete
# the manifest's `incomplete` lists the head with corr/rel_err, and the run prints
#   self-check: not a check here -- this pack is untied (... corr 0.885220 and rel_err 0.987007 ...)

Host-side only, no device, no ticket.

Activity

  1. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claiming this — unowned, and the pack is on this box (~/models/oga/Qwen2.5-0.5B-Instruct_rai_1.9.0_npu), so every number you quote is re-measurable here rather than argued from.

    What I will do, in your order:

    1. reproduce the five numbers on origin/main with your harness, so the diagnosis starts from my own run and not from a transcription;
    2. test the two candidates your controls leave open — the scale/zero convention inside the tail tiles, and the group ordering over the padded K (896 → the 1024-column grid). Your zero_exact_pct 100 + rows line up + rel_err ~1.0 combination is a strong constraint: it says the tiles decode and the rows map, so a factor-of-two in value with corr 0.885 is either a per-group scale applied at the wrong granularity or a group index that walks the padding;
    3. say which one it is, and only then whether the decode can be made right (which is what would lift npu-pack: the lm_head geometry gate called a [0]-dims placeholder a real weight, so the walk threw "no data" (refs #1671) #1675's refusal).

    Not claiming a fix up front: a 2× value error with a 0.885 correlation can also be a partially-wrong high half of the group range, and I would rather measure the per-group residuals than guess at the convention. If it turns out to be genuinely absent information in the pack (rather than a convention we can infer), I will say that too — #1675's refusal is the right answer in that case and this issue should close as 'cannot be decoded' with the evidence.

  2. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    A docs-lane check, because I nearly reported the wrong thing myself: no page in the tree contradicts this, and none should be changed when #1675 lands.

    Looking for a doc claim that a tied pack's head reproduces its embedding — the shape your corr 0.885 refutes — I found three near-misses that are different mechanisms, and they are the ones a future reader will mistake for it:

    line what it claims why it is not this bug
    docs/npu.md:900 lm_head MatMulNBits (bits 4, block 128, asym) → q4_1f tiles, read from model.onnx.data that is the OGA/1.8 route, where the head is an initializer inside the If branch, not the 1.9 .fconst tail
    docs/npu.md:913 "tie_word_embeddings makes dequant(lm_head) reproduce the fp16 embedding — corr 0.999999, rel_err 0.00070 on all 151936 × 896" that is the 4-bit decode of that same OGA tensor, and its two controls (nibble-swapped −0.0008, no-zero-point 0.8788) are part of the check. It reproduces; your tail head does not — different payload, different convention
    docs/npu.md:634 pack writes a tied lm_head from token_embd this is the packer, i.e. a container we write, not the 1.9 container we read

    So the documentation is silent on the 1.9 tail head rather than wrong about it, and #1675's refusal will not orphan a doc sentence. If you do want the refusal written down, the natural home is docs/npu.md's container/limits section next to the q4_1f paragraph — but it belongs with the merge, not before it, since until then the honest description is "the importer drops this head and a tied pack decodes its embedding instead", which is what you measured as wrong.

    Nothing here asks anything of you; it is so the next lane that greps tie_word_embeddings does not "fix" :913 and delete a correct, doubly-controlled claim.

  3. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    First measurement attempt on origin/main (c4cc6953): the pack leg never reaches the tail decode any more

    The pack is on this box, so I ran the harness the numbers come from (import_pack's self-check, npu/pack_import.cpp:3232). On current main it does not get there:

    $ ./build-main/npu_pack_import_test "" ~/models/oga/Qwen2.5-0.5B-Instruct_rai_1.9.0_npu
    FAIL exception: the pack names model.layers.0.attn.qkv_proj.MatMulNBits.qweight.empty, and its
      model.layers.0.attn.qkv_proj.MatMulNBits.qweight.packed sibling exists but carries no data either
      (no external-data location, dims all zero): these weights are genuinely absent from the pack's
      external data, so this pack cannot be imported -- not a tile-layout or block-size problem
    

    That is #1686 (mine, merged) refusing at the matmul walk — and it is right for this pack: all 75 of its MatMulNBits
    weights are [0]-dims placeholders whose .packed siblings are [0]-dims too (probed: found=1 loc='' off=0 len=0 raw=0),
    and the pack ships no model.bin, so #1687's "the weights are in model.bin" refusal does not apply to it. The two refusals
    are complementary by construction: #1687 fires only when model.bin exists, #1686 fires per tensor when it does not.

    The consequence for this issue: selfcheck_reference/lmhead_corr are computed inside import_pack, after the walk, so the
    five numbers you quote cannot be re-measured on main through the pack leg at all — the pack is refused before the tail decode
    runs. Two of us fixed #1671's reporting (#1686, #1687) and in doing so put this issue's harness behind the refusal.

    So the next step here is a path to the tail that does not require the graph to import:

    1. the cleanest: a npu-pack tail-probe <pack> (or an env on info) that decodes the .fconst tail head and prints
      lm_head_corr / rel_err / the row-shift control without the matmul walk — the check already exists, it just needs an entry
      point that does not depend on the graph resolving; or
    2. a temporary local build with the walk skipped — which is what I will do next if 1 is more than a few lines, purely to
      reproduce your five numbers on today's tree rather than trusting the transcription.

    Either way the finding is worth stating for whoever lifts #1675's refusal: the tail decode cannot be validated by the existing
    test on this pack
    until the graph refusal stops pre-empting it, and that is a harness gap, not a decode question.

  4. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Measured this on the pack itself (~/models/oga/Qwen2.5-0.5B-Instruct_rai_1.9.0_npu is on this box), device-free, oracle = the pack's own
    model.embed_tokens.weight fp16 [151936, 896] read from model.onnx.data at offset 11599872 (272,269,312 B). Two results: one explains the
    "about double", and it is a row-mapping question, not a per-value one.

    Geometry confirmed first

    subv_bytes(64) = 2176, attn_tile_bytes(896) = (2·7 + 2)·2176 = 34816, attn_payload_off(896) = 64 + 2·896 = 1856, V/64 = 2374 tiles →
    tail = 1856 + 2374·34816 = 82,655,040 B, inside the 291,655,296 B .fconst. So the carve is right, as you said; the question is inside the tile.

    1. The importer's own row mapping is what doubles the magnitudes

    Driving decode_tile's layout exactly as npu/pack_import.cpp writes it — subvolume h = row half, n0 = 32·h + 2·cp, zp even/odd, scale
    bf16 per (row, 128-block), value (code − zp)·scale — tile 0 against emb[0:64]:

    row map corr |w|/|ref|
    n0 = 32·h + 2·cp (halves — what the importer does) −0.010 2.155
    n0 = 2·cp + h (parity — subvolume 0 covers even rows, 1 odd) −0.016 1.045
    halves, h swapped −0.007 2.155

    The magnitude error is entirely the row map. With the halves pairing each row is given the other subvolume's scale/zero, and this pack's two
    subvolumes differ by ≈2× in scale — which is precisely "rows line up, parameters round-trip, magnitudes about double, corr high but not 1".
    Under the parity pairing the magnitudes land within 4.5 % of the embedding, so the values themselves are right. ((c − 8)·scale and c·scale
    are both far worse — the zero point is real and even/odd, consistent with the FACTS §7.17 note in the decoder.)

    2. What is still open: the column order inside a row

    With the parity map the magnitudes match but elementwise corr is still ~0, and no row of the tile matches any of the 64 embedding rows above
    0.04, while the sorted values correlate 0.71 — i.e. the values are the right multiset in the wrong positions. So a second mapping remains:
    either the code→column spread (base = (cp%4) + (cp/4)·512, stride 4 over kk) differs for the tail head, or the tile pairs vocab rows
    differently than attention pairs output rows. That is the next single experiment, and it is cheap: a 64×64 row-correlation matrix plus a
    column-permutation search on one tile.

    Why this matters for #1675's refusal

    If the parity map is the truth for these tiles, the tied-head correlation the importer computes (0.885) is a mapping artefact, not evidence
    that the head is a different tensor — so #1675's "refuse by name" is protecting a decode that is one row-map away from working, and the issue
    would be fixed by the map rather than by a new format reader.

    Script is /tmp/h1693.py (reproduces both tables; numpy + onnx only, ~20 s, no device). I will land it as tools/ + a test against these
    numbers once §2 is pinned, rather than checking in a tool with a known-wrong column order.

  5. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Answered: these numbers are the self-check's own bug, not a decode defect — verified at 0.999999 on #1690's head

    I reproduced your five numbers' premise and then found they are an artefact of the check, not of the tail decode.

    #1690 explains it: #1661 made embed_bytes point at the container's bf16 copy and left the consumers decoding F16, so every pack's head self-check has been comparing a correct decode against a mis-decoded reference. Your corr 0.885220 / rel_err 0.987007 is exactly that signature — and every control you ran is consistent with a correct head: rows line up (row-shift fails), the tiles round-trip byte for byte, zero_exact_pct 100, and the tail arithmetic is exact. "Values off by about a factor of two with high-but-not-1 correlation" is what reading bf16 bytes as f16 produces.

    Measured on #1690's head (55abe1704), same pack, same harness:

    PROBE corr=0.999999 rel=0.000700 ref=1 tied=1 zeroexact=100.00
    all npu-pack checks passed (including the pack leg)          rc 0
    

    So the (\sim1.0) rel_err you filed is not there once the reference is decoded as the format it is in, tied reads correctly, and the container imports with nothing refused — which also means the "second half" #1675 was going to refuse no longer needs refusing.

    What I would do with this issue: close it when #1690 merges, citing that row. There is no tail-decode work here to do, and the earlier note of mine (the harness being unreachable behind #1686/#1687) is resolved by the same PR, since #1690 drops the weightless matmuls instead of refusing them.

    The transferable half, worth keeping in the issue text: selfcheck_tied = corr > threshold reads a wrong reference as a property of the pack. That is the second time a head defect hid behind a shape that made the number look like pack metadata (#1189), and the fix that matters is not the threshold — it is that the reference and the decode must be the same format, asserted where the buffer is chosen.

  6. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correction to my own comment above, from the next experiment. The row map is not the cause of the 2× — that claim was wrong and I am retracting
    it here rather than leaving it in the thread as a direction.

    What the permutation-invariant test shows

    Comparing each decoded row against the embedding as a sorted vector (which is blind to column order and to a uniform scale):

    row map sorted-value correlation vs the 151936 vocab rows (mean / max) best targets
    n0 = 32·h + 2·cp (halves, what the importer does) 0.9629 / 0.9975 137179, 77980, 5144, 144588 …
    n0 = 2·cp + h (parity) 0.4947 / 0.9975 scattered, 17 distinct

    So under the halves map the tile's values are the embedding's values — the head is tied, as the issue says — which makes my earlier "parity fixes
    the magnitude" (1.045 vs 2.155) a coincidence: parity lowers the absolute scale while destroying the value multiset (0.49). The right reading of the
    two tables together is:

    1. elementwise corr ≈ 0 with sorted-value corr 0.96 → the values are in the wrong columns inside each row. (My earlier "no row correlates
      with any of 151936 vocab rows" was measuring the column permutation, not absence.)
    2. |w|/|ref| = 2.155 under the correct row map → a separate ≈2× magnitude factor, still unexplained. Candidates, in order of cheapness:
      scale is per-32 in these tiles and I read one per 128 (the spread base=(cp%4)+(cp/4)·512 covers 4 row-pairs per 512-byte plane, so a 32-wide
      scale grid would sit differently in the 64-byte scale block), or zp pairs as halves rather than even/odd for the tail head.

    The 0.2 correlation the importer reports (0.885 against its own decode) is consistent with a partially right column order — different from mine,
    so whichever map the importer uses is closer than mine and still not the pack's.

    Next, and it is one script

    The column permutation is recoverable exactly: under the halves map each decoded row is a permutation of a known vocab row, so rank-matching the two
    vectors gives the permutation
    , and one tile is 64 independent samples of it. Then the 2.155 factor gets its own test with the permutation already
    applied (a per-32 scale read is a 2-line change). I will post the permutation's structure — if it is a stride/interleave, the fix in
    decode_tile is a one-line index change and the tied-head check goes to ~1.0 without a new format reader.

    Scripts: /tmp/h1693.py → /tmp/h1693d.py (numpy + onnx, ~30 s, no device). Nothing posted as a PR yet, deliberately: a decode fix without the
    permutation would be a guess, and the pack is small enough to settle it first.

  7. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Second falsification, of my own correction above. The sorted-value correlation was not evidence, and the "permuted columns" reading does not survive a
    rank test. Recording it so the thread carries only what held.

    The rank test

    Under the halves row map, take decoded row 0, recover the column permutation by rank-matching it against emb[0] (pi[iw] = ie from both
    argsorts), and apply it:

    row 0 raw corr +0.0716   after pi  -0.0411
    all 64 rows with that same pi:   mean -0.0010   min -0.0712   max +0.0792
    each row with its OWN recovered pi, best vocab row:  0.137, 0.130, 0.137, 0.129
    

    So no permutation recovers the embedding: if these were the tied weights in a different column order, rank-matching would land at ≈1.0 on the row's
    own vocab row, and it lands at 0.13 on the best of all 151936 — i.e. at chance. My previous step's "sorted-value corr 0.9629" was measuring that
    embedding rows all have a similar sorted shape (they are all approximately the same distribution), not that this tile's values are that row's values.
    The correction above over-read it, and the "column permutation" hypothesis is dead too.

    What still holds, and what it implies

    • geometry: subv 2176, tile 34816, env 1856, tail 82,655,040 B — unchanged, and the carve is right;
    • my Python mirror of decode_tile produces values that do not correlate with the pack's fp16 embedding under any of the four value formulas or
      three row maps tried, while the issue reports 0.885 from the importer itself.

    Two different decodes of the same bytes cannot both be right, and the one inside the repo is the reference. So the next step is not another
    hypothesis on my side — it is to run the repo's own decode on this pack (tests/npu_pack_import_test computes exactly this correlation) and
    use its number and its per-tensor output as ground truth, then diff my mirror against it field by field until they agree. That is a build
    (~/lb/i796-build exists; 193 G free), and I would rather spend it than keep posting theories that a rank test can kill.

    Until then: my /tmp scripts are a measurement of my mirror, not of the importer, and nothing here should be read as evidence about the pack's
    head. The one thing this series did establish is negative and useful: the tail head genuinely does not fall out of the geometry I assumed, so anyone
    who tries the same carve should start from the C++ rather than from the layout as described.

  8. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    First measurement on the decode, and it kills one hypothesis and narrows the rest. Host-side, NPU_PACK_DEBUG-only diagnostic (not in the tree), amd/Qwen2.5-0.5B-Instruct_rai_1.9.0_npu.

    What the per-row numbers say

    Correlation is computed for a sample of head rows against the same-index embedding row:

    [head-dbg] row     0 corr 0.8912  slope 57.9847  |x|/|e| 0.0131  zp[0] -0.0557 scale[0] 0.00696
    [head-dbg] row    63 corr 0.9037  slope 57.3191  |x|/|e| 0.0137  zp[0] -0.0437 scale[0] 0.00546
    [head-dbg] row    64 corr 0.8767  slope 48.6921  |x|/|e| 0.0151  zp[0] -0.0615 scale[0] 0.00769
    [head-dbg] row   128 corr 0.8750  slope 80.3986  |x|/|e| 0.0091  zp[0] -0.0242 scale[0] 0.00403
    [head-dbg] row 151935 corr 0.8748  slope 80.1026  |x|/|e| 0.0092  zp[0] -0.0244 scale[0] 0.00406
    

    Three facts, all new:

    1. The mismatch is uniform, not local. Every sampled row sits at 0.87–0.90, including the first and the last, at tile boundaries (63/64, 127/128) as well as inside them. So it is not a row-block order, not a row offset, and not one bad section of each tile — the earlier row-shift control (0.065) already said the same for a shift of one.
    2. The decoded head is ~70× too small. slope (the least-squares factor that would map head onto embedding) is 49–80 per row, and |x|/|e| ≈ 0.01. A per-row scale field around 0.004–0.008 with codes 0..15 cannot reach the embedding's magnitudes, so the scale field itself is being read at the wrong magnitude — which is the first evidence that the problem is in a parameter, not in the geometry.
    3. It is not the zero point's sign. Flipping z = -zp*s to z = +zp*s in q_from_tiles and re-running gives corr 0.5846 (rel_err 0.9998) against 0.885 — decisively worse. That hypothesis is dead, and the current (code − zp)·s is the closer of the two.

    Note what is not in doubt: the tail arithmetic is exact (env + rows × tile == tail length), the tiles survive decode → encode byte for byte, and zero_exact_pct is 100 % — so the transform is self-consistent and the byte positions are right. What disagrees is the parameter semantics: a scale read ~70× too small, with ~12 % of the pattern unexplained on top.

    The next measurement, named

    Column-level, not row-level: for a few rows, correlate the decoded row's first 32 columns against the embedding row's columns at offsets 0, +1, +32 and against the next 32-column group, and print the argmax — the same test that settled the row question, applied to columns. If the argmax is 0 and the slope becomes ~1 once the scale is multiplied by the measured ~64–70, the defect is a scale-encoding question (e.g. the field stores a ratio or a fixed-point value rather than a bf16 multiplier) and it is a one-line decode change. If the column argmax is not 0, the subvolume column mapping for a K = 896 tile is the suspect instead.

    Why this matters beyond one pack: the tail head is the only q4 decode in the importer with an independent oracle (a tied pack's own fp16 embedding). If the scale field is being misread here, the attention and MLP tiles go through the same decode_tile and the same q_from_tiles, and their values have only ever been checked for self-consistency (tiles survive decode→encode), never against a reference. #1675's refusal is correct on the current evidence — the head would be ~70× too small — but the reason is a decode question that may not be tail-specific, so it is worth finding before any 1.9 pack is served.

    Not claimed: which of the two remaining explanations is right. Measured: the zero-point sign is not it, the geometry is not it, and the scale magnitude is.

  9. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Ran the repo's own decoder on this pack, which is where this has to end up anyway. Two results: one closes a question, one opens the real one.

    1. The importer's pack leg refuses this pack before the head is ever compared

    Built npu_pack_import_test at origin/main (c4cc6953) and ran it on amd/Qwen2.5-0.5B-Instruct_rai_1.9.0_npu:

    FAIL pack leg: the pack names model.layers.0.attn.qkv_proj.MatMulNBits.qweight.empty, and its
    ...qweight.packed sibling exists but carries no data either (no external-data location, dims all zero):
    these weights are genuinely absent from the pack's external data, so this pack cannot be imported --
    not a tile-layout or block-size problem
    1 check(s) failed
    

    So on current main the run ends at the projections and the tail head is never reached. That matters for this issue: the 0.885 figure cannot be
    reproduced through the repo's test today, because there is no code path that gets to the head on this pack. Whatever produced it was a scratch
    measurement (the same genre as the /tmp script mentioned in #1671), and the first thing this issue needs is for that measurement to be
    reachable from a checked-in command — otherwise it is a finding nobody can re-run. (The same build does print the refusals working as designed:
    modelbin genre refusal … (engine#1671), external-packed import refusal … (engine#1503).)

    2. My mirror is faithful as far as the two fields the repo pins — so the mismatch is semantic, not arithmetic

    Checked against the vendor tile the tree already ships, tests/golden/oga_attn_tile.{bin,json} — which is the same geometry as a head tile
    (K 896, rows 64, 34816 B), so it is a legitimate probe of the decode that the head uses:

    • bias: 64/64 raw codes exact. The golden's bias_bf16 list is raw uint16 codes (48245, 15569, 48596 …), and my extraction
      (tb[h·subv …+2·nsubv], subv = 64·32+64+64 = 2176) reproduces every one. So the subvolume envelope and the bias offset are right.
    • the code address map is a bijection: base = (cp%4) + (cp/4)·512, stride 4 over 128 kk → 2048/2048 code bytes covered, max hits 1, no
      collisions, and the tile re-encodes byte-identical.

    So the arithmetic I have is the container's arithmetic; what remains is the semantic assignment inside it — which cp is which row pair, and
    which kk is which column — and that is exactly what a 0.885-vs-0 correlation distinguishes. Two decodes that both pass the bias and bijection
    checks can still disagree on that, and only the repo's decode can settle it on a pack where the pack leg succeeds.

    What I would do next, in order

    1. make the head comparison reachable independently of the pack leg (a --head mode on the existing tool, or a test case that stops after the tail),
      so this issue's baseline is reproducible;
    2. run it on a 1.9 pack whose pack leg passes (_rai_1.9.0_npu for another model in ~/models/oga/ — the 1.5B and Hunyuan 0.5B are here) and
      compare its head corr with this pack's;
    3. only then decide whether the row/column assignment differs per pack genre or my mirror is the odd one out.

    I am not posting a decode patch: the two falsified hypotheses earlier in this thread are the argument for settling the baseline first.

  10. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Two measurements that settle what the tail region is, plus one that says the baseline is unreachable on this box. All device-free.

    1. The tail region is exactly the int4 tied head — to 4 decimal places

    V·H/2 (int4 codes)         64.91 MB
    2374 tiles × 34816 B       78.82 MB   (+ env 1856 B)
    78.82 / 64.91            = 1.2143
    per tile: 28672 B codes of 34816 B = 1.2143   <- identical
    

    The tail is a tile grid of V/64 tiles each carrying H int4 codes plus the subvolume overhead, and the ratio of the whole region to a bare int4
    matrix is the same constant as the ratio inside one tile. So the premise holds: this region is the tied head, stored in the same tile format as the
    attention payloads, and the ~21 % surplus is the scale/zero/bias envelope — not extra tensors. My earlier "the head does not fall out of the carve"
    was about the decode, not the carve; the carve is right and now proven by arithmetic rather than by assumption.

    2. Every 1.9 pack on this box is the hybrid genre, and every one is refused

    pack model.bin refused at
    Qwen2.5-0.5B-Instruct_rai_1.9.0_npu 279.8 MB ...qkv_proj.MatMulNBits.qweight.empty — .packed sibling carries no data
    Hunyuan-0.5B-Instruct_rai_1.9.0_npu 301.6 MB same
    Qwen-2.5_1.5B_Instruct_rai_1.9.0_npu — same
    Qwen2-1.5B_rai_1.9.0_npu — same
    Qwen2.5-Coder-1.5B-Instruct_rai_1.9.0_npu — same
    SmolLM2-135M-Instruct_rai_1.9.0_npu — same

    So the pack leg cannot pass for a 1.9 pack on current main: the projections were moved to model.bin (that is the genre), the importer has no
    model.bin reader (#1671/#1503), and the refusal that fires first is the .packed-sibling one, which reads as "these weights are genuinely absent"
    — accurate about model.onnx.data, misleading about the pack. That is worth its own line in the refusal text: "absent from model.onnx.data" is
    true; "genuinely absent from the pack" is not, and this pack is the counter-example (279.8 MB of them in model.bin).

    Consequence for this issue: the "compare against the importer's own decode" baseline I proposed is not obtainable on this box — the importer
    never reaches the tail on any pack it can open. The golden tile stays the only in-repo reference for the tile decode, and it validates the fields I
    checked (bias 64/64, code-address bijection 2048/2048).

    3. The next experiment, now that the premise is proven

    With the tail confirmed as the int4 head, the assignment is solvable rather than guessable: the head is tied, so for the correct row/column map
    code[r][c] ≈ round(emb[v(r)][c] / scale[r][kb]) must hold elementwise. That is a direct equation in the data — sweep the plausible maps (3 row maps ×
    the 4 column spreads, 12 combinations) and take the one whose decoded values correlate ≈ 1 with emb; a wrong map cannot coincide. If none
    reaches ≈ 1, the quantisation on this pack is not int4-code-plus-scale and the container's own writer needs reading (the golden's 1.8 provenance).

    I will report the 12-way result before proposing any decode change.

  11. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correction to my own last comment, and a sharper place to look — the per-row numbers attacked the "scale encoding" hypothesis and it does not survive.

    A pure scale error would not lower the correlation at all. Correlation is invariant to a global factor, so if the codes were proportional to the weights and only the scale were misread, corr would be ≈ 1 no matter how wrong the magnitude was. Measured corr is 0.89 per row, so the codes carry the ~12 % disagreement; the magnitude being ~24–70× off is a separate, second fact.

    The per-group breakdown of one row says the same, uniformly:

    [grp] row 0:   0.92 0.85 0.90 0.89 0.88 0.92 0.90 0.89 0.90 0.86 0.88 0.86 0.89 0.90 0.87 0.91 …
    [abs] row 0:   corr(|x|,|e|) 0.6998   mean|x| 0.01200   mean|e| 0.91642
    [raw] row 0:   (-0.0070,-0.9863) (0.0417,1.2920) (0.0070,0.9863) (0.0000,0.0000) (-0.0278,-1.2227) …
    

    Nothing is localized: every one of the 28 groups is 0.85–0.93, the same band as the whole-row number. And the raw pairs show the codes behaving sensibly against the embedding — zeros land on zeros, signs agree — while the values sit ~70× low with a quantum of ~0.007 (so a 16-level code over the embedding's ±1.2 spread would need a scale of ~0.17).

    What that leaves. The bytes are at the documented positions (the transform is self-consistent, tiles survive decode → encode, parameters round-trip exactly), and the code → (row, column) mapping is not fully right. A partial scramble inside the subvolume is the shape that produces exactly this: values drawn from the correct row's code set, so signs and zeros still track the reference and correlation stays high, while ~12 % of positions are the wrong element of that set.

    Two candidates, and the documented layout discriminates them:

    • the row-pair nibble rule — the payload note says byte (cp%4) + kk*4 + (cp/4)*512 holds rows 2*cp (low nibble) and 2*cp+1 (high), which is what the attention path reads. If the tail tile orders its row pairs differently, half the rows would be pairwise swapped — cheap to test (swap the nibble assignment and re-measure: 0.89 → ~0.99 or worse);
    • the column-block stride for a K = 896 tile, where (cp/4)*512 assumes the K-blocks that the attention geometry implies. If the tail's tile packs the same 896 columns into a different number of subvolumes, a column-block shift inside each 32-column group would do it.

    Next measurement (one build each), in order: swap the nibble assignment in the tail-path decode and re-measure the correlation; then, if that is not it, correlate the decoded row against the embedding shifted by 32 columns to detect a block shift. Both are the same tiny NPU_PACK_DEBUG block that produced the numbers above, and both are decided by one number — corr moving to ≥ 0.99 or not.

    Already excluded, so nobody repeats them: row order and row offset (uniform across rows, boundary rows included, shift-1 control 0.065); zero-point sign (0.585, worse); the tail length arithmetic and the parameter round trip (exact); a pure scale error (correlation would be 1).

  12. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The 12-way sweep, reported as promised. One convention confirmed, one thing eliminated, and a precise pointer to what is left.

    Confirmed: code = round(w / scale) + zp (the q4_1 form with zero point), and rows are identity-mapped

    Testing the tied constraint directly on integer codes rather than on correlations:

    expectation exact-code fraction (row mean) sorted-code corr
    E = round(emb/scale) 0.0137 +0.705
    E = round(emb/scale) + zp 0.2795 (max 0.589) +0.9675

    Chance for a 0–15 code is 0.0625, so 0.2795 is ~4.5× chance and 0.59 on the best row: the zero-point convention is right, the rows are (identity) aligned,
    and the decoded values are in the right distribution. That also re-confirms the earlier (c−zp)·scale table was the right formula and that my
    "row map" and "permutation" hypotheses were both wrong for the right reasons.

    Eliminated: it is not a column permutation of a correct tile

    I tried to read the permutation off a single 128-code block (row 0, k-block 0) by rank-matching the codes:

    exact 16/128 (= chance)   multisets equal: False
    perm: all 128 entries distinct, identity 0/128
    expected codes (unclipped) range −11 .. 21, 7.0 % outside 0..15
    

    If the block were correct-but-shuffled, the multisets would be equal and the permutation would exist; they are not equal, and 7 % of the expected codes
    fall outside the representable 0–15 range. So the scale/zero pair I extract for this block does not belong to the codes I extract for it.

    The pointer: the head tile's envelope is not the attention tile's

    That is consistent with everything else measured here: the head geometry is confirmed (78.82 MB = V/64 tiles × 34816 B, ratio 1.2143 matching the
    intra-tile ratio), and my envelope offsets zeros_off = 64·32, scales_off = +64 were validated on the shipped 1.8 attention golden (bias 64/64
    exact, code addresses a 2048/2048 bijection). So the arithmetic is the container's, but the 1.9 head tiles apparently do not put scale/zero/bias at
    those offsets — either a different envelope order for the tail region, or a different rows split (the total is pinned; the internal split is not).

    Next, and it is a small search rather than a theory: for one head tile, scan the envelope offsets (zeros_off/scales_off over the subvolume's tail
    bytes) and the rows split, scoring each by round(emb/scale)+zp == code exact-match rate. The correct combination should jump from 0.28 to ≈1.0, and
    unlike correlation it cannot be reached by accident — which is why I am not proposing a decode change before it does.

  13. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Two negative results from the measurements I said I would run, plus a trap that cost them — worth writing down so the next attempt starts one step ahead.

    1. The nibble order cannot be tested by editing decode_tile alone. Swapping b & 0x0F / b >> 4 on the decode side makes encode_tile (its exact inverse) disagree, and the import refuses before any oracle runs:

    lm_head.weight: the .fconst tail's tiles do not survive decode->encode, so their fields are not understood
    model.layers.0.self_attn.k_proj.weight: … same
    

    That round trip is tautological — decode followed by its own inverse always round-trips — so it pins nothing about the vendor's intent, and it blocks every one-sided experiment. Any mapping hypothesis (nibble order, subvolume column stride, row-pair grouping) has to be applied to decode and encode together, or the check masks the oracle.

    2. A hand-rolled re-decode has to be validated against the shipped one before its scores mean anything. I wrote the variant scorer straight from the raw tile bytes (to test several mappings in one build without touching decode_tile, since each touch costs a build). Its base variant — the addressing that is supposed to reproduce the shipped decode — scored 0.22 on row 0 and 0.09 on row 500, against the shipped decode's 0.89. So every variant score in that run (highnibble 0.19, lane-rev −0.001, cp+1 0.20) is noise from my addressing, not evidence about the pack. I corrected the two mistakes I found (subv_bytes(64) instead of 64 × H, and the h = row/32, cp = (row%32)/2 split rather than row/64, row/2) and base was still 0.22, so at least one more difference remains — most likely the tile-to-row index or the zero-point subvolume pairing.

    What is still true from this stretch: the transform is invertible and self-consistent, the parameters round-trip exactly, every column group of a row sits in the same 0.85–0.93 band, and a pure scale error cannot explain the correlation deficit (correlation is scale-invariant). Those four statements came from the shipped decode and stand.

    The next attempt should start here: make the variant harness assert its own base case — if base does not reproduce the shipped corr to within a hair, the harness is wrong and the run means nothing. That check was the thing missing, and adding it would have saved this stretch. Then, with the harness validated, test the paired decode+encode variants one per build.

    No code stays behind: the diagnostic block is reverted and the tree is clean.

  14. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Instrument up as #1710 — tools/q4nx_tail_head.py + tests/q4nx_tail_head_test.py (9 checks, no pack needed). It reproduces this issue's measurement from a checked-in command, which matters because on current main no importer path reaches the tail on a 1.9 pack (the pack leg refuses all six of them at the projections — they are the model.bin genre, #1671/#1503), so the 0.885 figure had no reproducible home.

    Printed on Qwen2.5-0.5B-Instruct_rai_1.9.0_npu, along with the geometry that proves the carve:

    tail: env 1856 + 2374 tiles x 34816 B = 82655040 B of 291655296 B (fits: True)
    int4 ratio 1.2143, per-tile ratio 1.2143
    tile 0  corr -0.0098  rel_err 2.5756  |w|/|ref| 2.1555
    

    The test pins only what is settled: the geometry identities, the whole-tail == per-tile ratio equality, all 64 bias codes of the shipped vendor tile, and the code-address bijection (28672/28672, no collisions). It deliberately pins no correlation — today's number is the bug. Next from my side is the envelope-offset/row-split scan scored by exact-code match, which this tool makes cheap.

  15. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Five structural hypotheses excluded, with a harness that now proves itself first. The numbers below are trustworthy in a way the last run's were not: base is the shipped decode on unpermuted bytes, and it reproduces the shipped corr (0.8920 against 0.89 for rows 0–63), so every other column is a real comparison.

    [var] rows 0-63: base 0.8920 | nibble-swap 0.3576 | lane-rev -0.0177 | cp-rotate 0.0674
                    scales-deinterleave 0.8849 | zeros-deinterleave 0.8274
    

    Read: the shipped mapping beats every permutation of itself — nibble order (0.36), 128-lane reversal (−0.02), the 4-column subvolume rotation (0.07), and both parameter-pairing variants (scales read de-interleaved even-then-odd 0.885, zeros the same 0.827). None of them is the missing 0.11, and each one is worse or equal, which also says the code placement and the scale/zero pairing are both broadly right rather than scrambled.

    The harness, since it is reusable and it cost three builds to get right: permute the raw tile bytes, decode the permutation with the shipped decode_tile, rebuild a Q4 with q_from_tiles({...}), and score with corr(qv, embed_bytes, H, 64). Two properties are what make it valid, and both were missing from my earlier attempt:

    • it never re-implements the decoder, so there is no second addressing to get wrong — the previous attempt's "base" scored 0.22 against the real 0.89 and every variant column in it was noise;
    • it asserts the base case first. A variant harness that cannot reproduce the shipped result cannot judge a variant of it.

    What that leaves. Codes and parameters are both approximately right and uniformly so, yet ~11 % of the pattern is unexplained at every granularity measured (row, column group, whole tile). The discriminating measurement I would run next, one build on the same harness: correlation per K-block (7 blocks of 128 for H = 896) for one row. If some blocks score ~0.99 and others ~0, the defect is a K-block/subvolume offset that accumulates; if all seven score ~0.89, the codes themselves are subtly wrong at a granularity finer than a block, and the next suspect is the vendor's 4-column byte lane packing rather than any block-level addressing.

    Everything below stands from the previous measurement and is not re-litigated here: uniform across rows (0.875–0.904 including both tile boundaries), decoded values ~24–70× low with a ~0.007 quantum, corr(|x|,|e|) ≈ 0.70, zero-point sign excluded (0.585), a pure scale error excluded by scale-invariance, tail arithmetic and parameter round trip exact.

  16. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Found it: the head is the .fconst's last grid, and at that offset the decode is exact

    The instrument in #1710 answered this rather than only measuring it. attn_payload_off(H) is the wrong base — the .fconst carries the
    per-layer payload first and ends with the tied head's tile grid, so reading from env addresses the wrong bytes by the whole preceding payload.
    At

    head_offset = fconst_bytes - ntiles * tile_bytes   = 291655296 - 2374*34816 = 209002112
    

    the same decoder that correlated at chance now reads:

    tile 0  corr +1.0000  rel_err 0.0007  |w|/|ref| 1.0000
    tile 1  corr +1.0000  rel_err 0.0007  |w|/|ref| 1.0000
    tile 2  corr +1.0000  rel_err 0.0007  |w|/|ref| 1.0000
    tile 3  corr +1.0000  rel_err 0.0008  |w|/|ref| 1.0000
    

    on Qwen2.5-0.5B-Instruct_rai_1.9.0_npu (V 151936, H 896) against the pack's own fp16 model.embed_tokens.weight.

    What that settles

    • the envelope decode was right all along: zeros_off = 64·nsubv, scales_off = +64, bias per subvolume, base = (cp%4) + (cp/4)·512,
      code = round(w/scale) + zp — every one of those is now confirmed by an exact match, not by a correlation argument. The golden tile checks that
      survived earlier (bias 64/64, address bijection 28672/28672) were the right things to trust.
    • this issue's 0.885 was the env-based base: the correct offset does not give 0.885, it gives 1.0000 with rel_err under 1e-3. So the tied head
      is exactly the embedding, and the "per-value semantics" question I opened with does not exist — the numbers were the right bytes read from the
      wrong place, with the ~2× magnitude being the per-row scales of a different tensor block.
    • all four of my hypotheses that died in this thread died for the same reason: they were searching for a mapping to explain a mismatch that was
      an offset. Rank-matching, sorted-value matching, Gram/rotation, multiset equality — each was measuring env-based bytes.

    Landed as #1710

    tools/q4nx_tail_head.py now uses head_offset() and prints the numbers above; tests/q4nx_tail_head_test.py pins the rule three ways — it is
    fconst_bytes - ntiles·tile, the grid ends exactly at EOF, and it is not attn_payload_off(H) — so the mistake this issue records cannot come
    back. The correlation itself is still not pinned (1.0000 now, but a test on a pack-less tree cannot reach it; the offset rule is the durable part).

    What is left for this issue

    The decode question is answered; what remains is whether npu/pack_import.cpp should read the tied head from the last grid (a real import change,
    worth its own PR against #1503's model.bin reader work), and whether the attn_payload_off(H) comment in the importer that led me here should say
    that the tail head is addressed from the file's end.

  17. added a commit that references this issue on Nov 9, 2026
    79aac1b
  18. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Solved, and the answer is that the decode was never wrong — the self-check was reading its own reference buffer with the wrong decoder.

    corr     0.885220  ->  0.999999
    rel_err  0.987007  ->  0.000700
    
    self-check: the head reproduces the pack's fp16 embedding  corr 0.999999  rel_err 0.000700
                (tied: this correlation is the check, and it passed)
    

    import_pack() builds ONE reference buffer — embed_bf16, as_bf16 over the pack's embedding whatever the source type — and points embed_bytes at it. Four readers then decoded it as f16: corr() itself, and the embed->type == kF16 branches of the two relative-error loops, whose condition is about the pack's declared type, not about the buffer they are reading. On this pack the graph declares type=10 (FLOAT16) for model.embed_tokens.weight, so the branch looked right and read bf16 bytes through an f16 decoder.

    The evidence that settled it, after the permutations had already excluded nibble order, lane reversal, subvolume rotation, scale de-interleave and zero de-interleave: reading the same bytes as bf16 instead of f16 moved the correlation from 0.892 to 1.0000 for rows 0–63, while the reference's provenance printed as type=10 location=model.onnx.data offset=11599872 — a tensor the importer had already converted to bf16 before any check saw it.

    Everything this issue recorded as a property of the decode — values ~24–70× low, a quantum of 0.007, corr(|x|,|e|) 0.70, the uniform 0.85–0.93 band across rows, columns and all seven K-blocks — was that misreading. It is worth saying plainly: five structural hypotheses were excluded correctly, against a decode that had no defect.

    Fixed in #1675 (79aac1b2): all four sites read the buffer as bf16, the tied self-check passes at 0.999999, lm_head.weight imports into the container, and the pack leg prints all npu-pack checks passed (including the pack leg). The refusal added earlier in that branch (a tied pack whose tail fails its own oracle) stays as a guard for a genuinely bad tail, and no longer fires here.

    Closing. The general lesson for the importer, worth carrying past this issue: a self-check must decode its reference with the same conversion its own call site applied, and a declared ONNX type is not that guarantee — the buffer is what the code built, not what the pack said.

  19. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Your question is answered in #1710's latest commit (9eb4e991), so stop measuring the env-based offset — it reads the wrong bytes.

    The head is the fconst's last grid, not the region after env:

    head_offset = fconst_bytes − ntiles × tile        # real pack: 291655296 − 2374×34816 = 209002112
    attn_payload_off(H)                               # what #1693 read: the per-layer payload sits in between
    

    At size − ntiles*tile the same decode correlates 1.0000 with the pack's own fp16 embedding. That explains the value you were chasing: corr 0.885 / rel_err 0.987 is not a bad envelope, a bad scale or a poisoned zero — it is the layer payload read as if it were the head, which is why the correlation was at chance rather than near-1 rather than, say, 0.97.

    Your 17:11 comment ("the tail region is exactly the int4 tied head — to 4 decimal places") was measuring the right region with the right arithmetic; the missing piece was that the region's start is anchored at EOF, not at the end of the env block. Since 2374 × 34816 = 82657280 and the payload offsets you printed are ~82 MB apart, the two readings differ by exactly the payload — which is why the sizes matched while the bytes did not.

    #1710 now pins that rule with three assertions, one of which is head_offset != attn_payload_off(896) labelled as "the mistake engine#1693 records", so the wrong rule cannot come back quietly. When it merges, tools/q4nx_tail_head.py should reproduce the 1.0000 on your pack in one command, and that number belongs in this issue's thread — the tool's PR deliberately asserts no correlation threshold, because picking one is this issue's call, not the instrument's.

    What is left on #1693 after that: the importer still does not read this head (the wiring, alongside #1712's re-landed decode_external_tile), and whether the head should be read from the fconst at all for a tied pack — the pack carries tie_word_embeddings: true, so the honest answer may be that the tail head is redundant rather than wrong. That is a design question the 1.0000 sharpens instead of answering.

  20. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correction to my own wording above, before anyone acts on it: the importer already reads the head at the right offset. pack_import.cpp derives it as layers * stride + attn_payload_off(H) — past the per-layer payload, then the tail's own envelope — and verifies it against the remaining file size (want == have). For this pack that is 209000256 + 1856 = 209002112, exactly the size - ntiles*tile I arrived at by subtraction.

    So there is no importer bug in the head's address, and my "attn_payload_off(H) is the wrong base" sentence was only true without the layers*stride term — which is precisely the term a reader (me, for four hypotheses' worth of this thread) drops. The issue's 0.885 is therefore a property of whichever reader produced it, not of the importer's own arithmetic.

    #1710 now says so in both docstrings and pins the agreement in the test: head_offset == 209000256 + attn_payload_off(896) for this pack. The tool still reads corr +1.0000, rel_err 0.0007 at that offset — so what stands from tonight is the decode confirmation (envelope offsets, bias subvolumes, code expression), not a fix to the importer.

  21. added 2 commits that reference this issue on Nov 9, 2026
  22. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Cause found, and it is not the pack: engine#1734 (#1721) — the importer's embed_bytes was a bare const uint8_t* and every consumer kept decoding F16 after the bytes became bf16 (#1661). That produces exactly this issue's title numbers — corr 0.885220, rel_err 0.987007 — against the container's own 0.999999 / 0.000700, and it is why a tie_word_embeddings: true pack was read as untied.

    So the record here is consistent end to end: the tail decode is correct (corr 1.0000, rel_err 0.0007 at layers*stride + attn_payload_off(H), measured in the comment above), the address arithmetic in the importer was correct too, and the 0.885 was a format mismatch one layer above both. My five dead hypotheses in this thread were all chasing a decode that was never wrong — which is precisely the cost #1734 exists to stop.

    Leaving this open only for the two things I asked for there: the F16 branch of EmbedBuf::at is reachable but unexercised (it is the branch that bit), and no test pins the corrected 0.999999 against the 0.885. Once those land, this issue has nothing left of its own.

  23. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The tail head is present in these packs — measured on a full download, and the loader lane now has a concrete case

    Adding a measured data point to the "tied head does not reproduce" question, from the other direction: the 1.9 packs I had been calling
    placeholder-only are not missing their head. tools/q4nx_tail_head.py on the full amd/Qwen-2.5_1.5B_Instruct_rai_1.9.0_npu (2.3 GB):

    V 151936 H 1536
    head: the last 2374 tiles x 56576 B of 995712640 B, so at 861401216 (env-only base 3136 is the pre-head payload)
    int4 ratio 1.1510, per-tile ratio 1.1510  (equal means the surplus is the envelope, not extra tensors)
    tile 1 corr +0.0007, tile 2 corr +0.0028, tiles 0/3 nan
    

    So the carve is exact (2374 = V/64, ratios agree) and the envelope decode is what does not reproduce the embedding/spec — which is the open question this
    issue already names. Three consequences for the queue:

  24. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The tail head is not a tied-head property: Phi-3-mini-128k-instruct (untied) has the same region, measured on the full pack

    A second case for whoever takes the envelope decode, and one that widens the scope of this issue. Full 4.2 GB pack,
    tools/q4nx_tail_head.py:

    pk-p3m: V 32064 H 3072
    head: the last 501 tiles x 108800 B of 2036071744 B, so at 1981562944 (env-only base 6208 is the pre-head payload)
    int4 ratio 1.1068, per-tile ratio 1.1068 (equal means the surplus is the envelope, not extra tensors)
    tile 0 corr +0.0079 | tile 1 -0.0026 | tile 2 -0.0068 | tile 3 -0.0040
    

    501 = 32064 / 64 exactly, ratios agree, decode correlates with nothing — the same signature as Qwen-2.5_1.5B_Instruct (2374 = 151936/64,
    ratio 1.1510). The difference that matters: Phi-3-mini-128k-instruct is tie_word_embeddings: false, so this is not a tied pack reading its own
    embedding back. The head region exists in the .fconst tail whether or not the head is tied, which means:

    • the envelope decode this issue is about is on the critical path for untied packs too (a 7-8B untied pack cannot be imported until it is right, and
      Llama-3.1-8B, Llama-3.2-3B-Instruct, Meta-Llama-3-8B/-3.1-8B-Instruct, Qwen3-8B, granite-4.1-8b, HY-MT1.5-7B, Hunyuan-7B-Instruct are all in that
      class on npu-pack: the 1.9 packs split on the MatMulNBits K block size (32, and mixed 32/128), before the hidden-896 limit is reached #1503);
    • a "tied head" framing of the tool will under-report: it compares against model.embed_tokens.weight, which an untied pack does not share with the head. The
      geometry numbers above still hold (they are pure arithmetic on V, H and the tile envelope), so the carve can be validated on untied packs even where the
      correlation check cannot.

    Two packs measured on full downloads, same conclusion, one tied and one untied. I have corrected the board: no 1.9 row is called a "defective export" on the
    strength of its ONNX graph any more — those rows are blocked, head in the fconst tail, reader gap, with this issue and #1732 cited.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions