Repository navigation
Description
npu-pack: the 1.9 .fconst tail head does not reproduce a tied pack's own embedding (corr 0.885, rel_err 0.987)
Found while fixing #1671 (the pack leg of npu_pack_import_test on the 0.5B 1.9 pack). #1675 makes the importer refuse this head by name instead of dropping it silently, so nothing wrong ships — this issue is about making the decode right, which would lift that refusal.
What is measured
amd/Qwen2.5-0.5B-Instruct_rai_1.9.0_npu (its config.json says tie_word_embeddings: true, vocab_size 151936, hidden 896), the head read from the .fconst tail with the layout attn_payload_off(H) + (V/64) × attn_tile_bytes(H):
lm_head_corr 0.8852204948414607 corr against the pack's own fp16 embedding
lmhead_rel 0.9870072916840432 sum|head-emb| / sum|emb| -- i.e. ~1: magnitudes are about double
ctrl_corr_rowshift 0.0648 same tiles, each head row against the NEXT embedding row (control, must fail)
zero_exact_pct 100.00 the per-32-group scale/zero survive into the container exactly
tiles survive decode->encode byte for byte
tail arithmetic env + rows*tile == the tail length, exactly (so the SIZE is right)
Read together: the rows line up (the row-shift control fails, so it is not a row order or row-offset error), the stored parameters round-trip, the tile count is right, and the values are off by about a factor of two with a high-but-not-1 correlation. That points at the tail's per-value semantics rather than its geometry: the scale/zero convention inside these tiles, or the group ordering over a padded K (896 → the 1024-column grid), rather than an offset.
Why it matters beyond this pack
A tied pack is the only case where a head has a free oracle, and this is the second time a head defect has hidden behind "the pack is untied": #1189 (closed) was a phi3 pack importing strictly and completely and then producing near-flat logits, with the note that an untied head meant no check could have caught it. The importer's selfcheck_tied = corr > threshold heuristic reads the number backwards for exactly this class — a wrong decode presents itself as a property of the pack. #1675 turns that into a refusal for declared-tied packs; this issue is about the ~1.0 rel_err.
Acceptance
npu_pack_import_test tests/golden <the 0.5B 1.9 pack>reports the head as present and corr > 0.999 (the row-shift control still failing), or a measurement that says what the tail's true value convention is and why the fp16 embedding is not its reference;- the two other 1.9 packs on the box measured the same way, since a convention that fits one pack and not the others is not a convention;
- the refusal added in npu-pack: the lm_head geometry gate called a
[0]-dims placeholder a real weight, so the walk threw "no data" (refs #1671) #1675 is removed in the same change, so the head is written because it is right, not because the check learned to tolerate it.
How to reproduce
1bit npu-pack import ~/models/oga/Qwen2.5-0.5B-Instruct_rai_1.9.0_npu -o /tmp/head --allow-incomplete
# the manifest's `incomplete` lists the head with corr/rel_err, and the run prints
# self-check: not a check here -- this pack is untied (... corr 0.885220 and rel_err 0.987007 ...)
Host-side only, no device, no ticket.
Activity
bong-water-water-bong commentedon Nov 9, 2026
Claiming this — unowned, and the pack is on this box (
~/models/oga/Qwen2.5-0.5B-Instruct_rai_1.9.0_npu), so every number you quote is re-measurable here rather than argued from.What I will do, in your order:
origin/mainwith your harness, so the diagnosis starts from my own run and not from a transcription;zero_exact_pct 100+rows line up+rel_err ~1.0combination is a strong constraint: it says the tiles decode and the rows map, so a factor-of-two in value with corr 0.885 is either a per-group scale applied at the wrong granularity or a group index that walks the padding;[0]-dims placeholder a real weight, so the walk threw "no data" (refs #1671) #1675's refusal).Not claiming a fix up front: a 2× value error with a 0.885 correlation can also be a partially-wrong high half of the group range, and I would rather measure the per-group residuals than guess at the convention. If it turns out to be genuinely absent information in the pack (rather than a convention we can infer), I will say that too — #1675's refusal is the right answer in that case and this issue should close as 'cannot be decoded' with the evidence.
bong-water-water-bong commentedon Nov 9, 2026
A docs-lane check, because I nearly reported the wrong thing myself: no page in the tree contradicts this, and none should be changed when #1675 lands.
Looking for a doc claim that a tied pack's head reproduces its embedding — the shape your
corr 0.885refutes — I found three near-misses that are different mechanisms, and they are the ones a future reader will mistake for it:docs/npu.md:900lm_headMatMulNBits(bits 4, block 128, asym) → q4_1f tiles, read frommodel.onnx.dataIfbranch, not the 1.9.fconsttaildocs/npu.md:913tie_word_embeddingsmakesdequant(lm_head)reproduce the fp16 embedding — corr 0.999999, rel_err 0.00070 on all 151936 × 896"docs/npu.md:634packwrites a tiedlm_headfromtoken_embdSo the documentation is silent on the 1.9 tail head rather than wrong about it, and #1675's refusal will not orphan a doc sentence. If you do want the refusal written down, the natural home is
docs/npu.md's container/limits section next to the q4_1f paragraph — but it belongs with the merge, not before it, since until then the honest description is "the importer drops this head and a tied pack decodes its embedding instead", which is what you measured as wrong.Nothing here asks anything of you; it is so the next lane that greps
tie_word_embeddingsdoes not "fix":913and delete a correct, doubly-controlled claim.bong-water-water-bong commentedon Nov 9, 2026
First measurement attempt on
origin/main(c4cc6953): the pack leg never reaches the tail decode any moreThe pack is on this box, so I ran the harness the numbers come from (
import_pack's self-check,npu/pack_import.cpp:3232). On currentmainit does not get there:That is #1686 (mine, merged) refusing at the matmul walk — and it is right for this pack: all 75 of its
MatMulNBitsweights are
[0]-dims placeholders whose.packedsiblings are[0]-dims too (probed:found=1 loc='' off=0 len=0 raw=0),and the pack ships no
model.bin, so #1687's "the weights are in model.bin" refusal does not apply to it. The two refusalsare complementary by construction: #1687 fires only when
model.binexists, #1686 fires per tensor when it does not.The consequence for this issue:
selfcheck_reference/lmhead_corrare computed insideimport_pack, after the walk, so thefive numbers you quote cannot be re-measured on
mainthrough the pack leg at all — the pack is refused before the tail decoderuns. Two of us fixed #1671's reporting (#1686, #1687) and in doing so put this issue's harness behind the refusal.
So the next step here is a path to the tail that does not require the graph to import:
npu-pack tail-probe <pack>(or an env oninfo) that decodes the.fconsttail head and printslm_head_corr/rel_err/ the row-shift control without the matmul walk — the check already exists, it just needs an entrypoint that does not depend on the graph resolving; or
reproduce your five numbers on today's tree rather than trusting the transcription.
Either way the finding is worth stating for whoever lifts #1675's refusal: the tail decode cannot be validated by the existing
test on this pack until the graph refusal stops pre-empting it, and that is a harness gap, not a decode question.
bong-water-water-bong commentedon Nov 9, 2026
Measured this on the pack itself (
~/models/oga/Qwen2.5-0.5B-Instruct_rai_1.9.0_npuis on this box), device-free, oracle = the pack's ownmodel.embed_tokens.weightfp16[151936, 896]read frommodel.onnx.dataat offset11599872(272,269,312 B). Two results: one explains the"about double", and it is a row-mapping question, not a per-value one.
Geometry confirmed first
subv_bytes(64) = 2176,attn_tile_bytes(896) = (2·7 + 2)·2176 = 34816,attn_payload_off(896) = 64 + 2·896 = 1856,V/64 = 2374tiles →tail =
1856 + 2374·34816 = 82,655,040B, inside the 291,655,296 B.fconst. So the carve is right, as you said; the question is inside the tile.1. The importer's own row mapping is what doubles the magnitudes
Driving
decode_tile's layout exactly asnpu/pack_import.cppwrites it — subvolumeh= row half,n0 = 32·h + 2·cp,zpeven/odd, scalebf16per (row, 128-block), value(code − zp)·scale— tile 0 againstemb[0:64]:n0 = 32·h + 2·cp(halves — what the importer does)n0 = 2·cp + h(parity — subvolume 0 covers even rows, 1 odd)hswappedThe magnitude error is entirely the row map. With the halves pairing each row is given the other subvolume's scale/zero, and this pack's two
subvolumes differ by ≈2× in scale — which is precisely "rows line up, parameters round-trip, magnitudes about double, corr high but not 1".
Under the parity pairing the magnitudes land within 4.5 % of the embedding, so the values themselves are right. (
(c − 8)·scaleandc·scaleare both far worse — the zero point is real and even/odd, consistent with the
FACTS §7.17note in the decoder.)2. What is still open: the column order inside a row
With the parity map the magnitudes match but elementwise corr is still ~0, and no row of the tile matches any of the 64 embedding rows above
0.04, while the sorted values correlate 0.71 — i.e. the values are the right multiset in the wrong positions. So a second mapping remains:
either the code→column spread (
base = (cp%4) + (cp/4)·512, stride 4 overkk) differs for the tail head, or the tile pairs vocab rowsdifferently than attention pairs output rows. That is the next single experiment, and it is cheap: a 64×64 row-correlation matrix plus a
column-permutation search on one tile.
Why this matters for #1675's refusal
If the parity map is the truth for these tiles, the tied-head correlation the importer computes (0.885) is a mapping artefact, not evidence
that the head is a different tensor — so #1675's "refuse by name" is protecting a decode that is one row-map away from working, and the issue
would be fixed by the map rather than by a new format reader.
Script is
/tmp/h1693.py(reproduces both tables; numpy + onnx only, ~20 s, no device). I will land it astools/+ a test against thesenumbers once §2 is pinned, rather than checking in a tool with a known-wrong column order.
bong-water-water-bong commentedon Nov 9, 2026
Answered: these numbers are the self-check's own bug, not a decode defect — verified at 0.999999 on #1690's head
I reproduced your five numbers' premise and then found they are an artefact of the check, not of the tail decode.
#1690 explains it: #1661 made
embed_bytespoint at the container's bf16 copy and left the consumers decoding F16, so every pack's head self-check has been comparing a correct decode against a mis-decoded reference. Yourcorr 0.885220 / rel_err 0.987007is exactly that signature — and every control you ran is consistent with a correct head: rows line up (row-shift fails), the tiles round-trip byte for byte,zero_exact_pct 100, and the tail arithmetic is exact. "Values off by about a factor of two with high-but-not-1 correlation" is what reading bf16 bytes as f16 produces.Measured on #1690's head (
55abe1704), same pack, same harness:So the (\sim1.0)
rel_erryou filed is not there once the reference is decoded as the format it is in,tiedreads correctly, and the container imports with nothing refused — which also means the "second half" #1675 was going to refuse no longer needs refusing.What I would do with this issue: close it when #1690 merges, citing that row. There is no tail-decode work here to do, and the earlier note of mine (the harness being unreachable behind #1686/#1687) is resolved by the same PR, since #1690 drops the weightless matmuls instead of refusing them.
The transferable half, worth keeping in the issue text:
selfcheck_tied = corr > thresholdreads a wrong reference as a property of the pack. That is the second time a head defect hid behind a shape that made the number look like pack metadata (#1189), and the fix that matters is not the threshold — it is that the reference and the decode must be the same format, asserted where the buffer is chosen.bong-water-water-bong commentedon Nov 9, 2026
Correction to my own comment above, from the next experiment. The row map is not the cause of the 2× — that claim was wrong and I am retracting
it here rather than leaving it in the thread as a direction.
What the permutation-invariant test shows
Comparing each decoded row against the embedding as a sorted vector (which is blind to column order and to a uniform scale):
n0 = 32·h + 2·cp(halves, what the importer does)n0 = 2·cp + h(parity)So under the halves map the tile's values are the embedding's values — the head is tied, as the issue says — which makes my earlier "parity fixes
the magnitude" (1.045 vs 2.155) a coincidence: parity lowers the absolute scale while destroying the value multiset (0.49). The right reading of the
two tables together is:
with any of 151936 vocab rows" was measuring the column permutation, not absence.)
scale is per-32 in these tiles and I read one per 128 (the spread
base=(cp%4)+(cp/4)·512covers 4 row-pairs per 512-byte plane, so a 32-widescale grid would sit differently in the 64-byte scale block), or
zppairs as halves rather than even/odd for the tail head.The 0.2 correlation the importer reports (0.885 against its own decode) is consistent with a partially right column order — different from mine,
so whichever map the importer uses is closer than mine and still not the pack's.
Next, and it is one script
The column permutation is recoverable exactly: under the halves map each decoded row is a permutation of a known vocab row, so rank-matching the two
vectors gives the permutation, and one tile is 64 independent samples of it. Then the 2.155 factor gets its own test with the permutation already
applied (a per-32 scale read is a 2-line change). I will post the permutation's structure — if it is a stride/interleave, the fix in
decode_tileis a one-line index change and the tied-head check goes to ~1.0 without a new format reader.Scripts:
/tmp/h1693.py→/tmp/h1693d.py(numpy + onnx, ~30 s, no device). Nothing posted as a PR yet, deliberately: a decode fix without thepermutation would be a guess, and the pack is small enough to settle it first.
bong-water-water-bong commentedon Nov 9, 2026
Second falsification, of my own correction above. The sorted-value correlation was not evidence, and the "permuted columns" reading does not survive a
rank test. Recording it so the thread carries only what held.
The rank test
Under the halves row map, take decoded row 0, recover the column permutation by rank-matching it against
emb[0](pi[iw] = iefrom bothargsorts), and apply it:So no permutation recovers the embedding: if these were the tied weights in a different column order, rank-matching would land at ≈1.0 on the row's
own vocab row, and it lands at 0.13 on the best of all 151936 — i.e. at chance. My previous step's "sorted-value corr 0.9629" was measuring that
embedding rows all have a similar sorted shape (they are all approximately the same distribution), not that this tile's values are that row's values.
The correction above over-read it, and the "column permutation" hypothesis is dead too.
What still holds, and what it implies
subv 2176,tile 34816,env 1856, tail82,655,040B — unchanged, and the carve is right;decode_tileproduces values that do not correlate with the pack's fp16 embedding under any of the four value formulas orthree row maps tried, while the issue reports 0.885 from the importer itself.
Two different decodes of the same bytes cannot both be right, and the one inside the repo is the reference. So the next step is not another
hypothesis on my side — it is to run the repo's own decode on this pack (
tests/npu_pack_import_testcomputes exactly this correlation) anduse its number and its per-tensor output as ground truth, then diff my mirror against it field by field until they agree. That is a build
(
~/lb/i796-buildexists; 193 G free), and I would rather spend it than keep posting theories that a rank test can kill.Until then: my
/tmpscripts are a measurement of my mirror, not of the importer, and nothing here should be read as evidence about the pack'shead. The one thing this series did establish is negative and useful: the tail head genuinely does not fall out of the geometry I assumed, so anyone
who tries the same carve should start from the C++ rather than from the layout as described.
bong-water-water-bong commentedon Nov 9, 2026
First measurement on the decode, and it kills one hypothesis and narrows the rest. Host-side,
NPU_PACK_DEBUG-only diagnostic (not in the tree),amd/Qwen2.5-0.5B-Instruct_rai_1.9.0_npu.What the per-row numbers say
Correlation is computed for a sample of head rows against the same-index embedding row:
Three facts, all new:
slope(the least-squares factor that would map head onto embedding) is 49–80 per row, and|x|/|e| ≈ 0.01. A per-row scale field around 0.004–0.008 with codes 0..15 cannot reach the embedding's magnitudes, so the scale field itself is being read at the wrong magnitude — which is the first evidence that the problem is in a parameter, not in the geometry.z = -zp*stoz = +zp*sinq_from_tilesand re-running gives corr 0.5846 (rel_err 0.9998) against 0.885 — decisively worse. That hypothesis is dead, and the current(code − zp)·sis the closer of the two.Note what is not in doubt: the tail arithmetic is exact (
env + rows × tile == tail length), the tiles survivedecode → encodebyte for byte, andzero_exact_pctis 100 % — so the transform is self-consistent and the byte positions are right. What disagrees is the parameter semantics: a scale read ~70× too small, with ~12 % of the pattern unexplained on top.The next measurement, named
Column-level, not row-level: for a few rows, correlate the decoded row's first 32 columns against the embedding row's columns at offsets 0, +1, +32 and against the next 32-column group, and print the argmax — the same test that settled the row question, applied to columns. If the argmax is 0 and the slope becomes ~1 once the scale is multiplied by the measured ~64–70, the defect is a scale-encoding question (e.g. the field stores a ratio or a fixed-point value rather than a bf16 multiplier) and it is a one-line decode change. If the column argmax is not 0, the subvolume column mapping for a K = 896 tile is the suspect instead.
Why this matters beyond one pack: the tail head is the only q4 decode in the importer with an independent oracle (a tied pack's own fp16 embedding). If the scale field is being misread here, the attention and MLP tiles go through the same
decode_tileand the sameq_from_tiles, and their values have only ever been checked for self-consistency (tiles survive decode→encode), never against a reference. #1675's refusal is correct on the current evidence — the head would be ~70× too small — but the reason is a decode question that may not be tail-specific, so it is worth finding before any 1.9 pack is served.Not claimed: which of the two remaining explanations is right. Measured: the zero-point sign is not it, the geometry is not it, and the scale magnitude is.
bong-water-water-bong commentedon Nov 9, 2026
Ran the repo's own decoder on this pack, which is where this has to end up anyway. Two results: one closes a question, one opens the real one.
1. The importer's pack leg refuses this pack before the head is ever compared
Built
npu_pack_import_testatorigin/main(c4cc6953) and ran it onamd/Qwen2.5-0.5B-Instruct_rai_1.9.0_npu:So on current main the run ends at the projections and the tail head is never reached. That matters for this issue: the 0.885 figure cannot be
reproduced through the repo's test today, because there is no code path that gets to the head on this pack. Whatever produced it was a scratch
measurement (the same genre as the
/tmpscript mentioned in #1671), and the first thing this issue needs is for that measurement to bereachable from a checked-in command — otherwise it is a finding nobody can re-run. (The same build does print the refusals working as designed:
modelbin genre refusal … (engine#1671),external-packed import refusal … (engine#1503).)2. My mirror is faithful as far as the two fields the repo pins — so the mismatch is semantic, not arithmetic
Checked against the vendor tile the tree already ships,
tests/golden/oga_attn_tile.{bin,json}— which is the same geometry as a head tile(K 896, rows 64, 34816 B), so it is a legitimate probe of the decode that the head uses:
bias_bf16list is raw uint16 codes (48245, 15569, 48596 …), and my extraction(
tb[h·subv …+2·nsubv],subv = 64·32+64+64 = 2176) reproduces every one. So the subvolume envelope and the bias offset are right.base = (cp%4) + (cp/4)·512, stride 4 over 128kk→ 2048/2048 code bytes covered, max hits 1, nocollisions, and the tile re-encodes byte-identical.
So the arithmetic I have is the container's arithmetic; what remains is the semantic assignment inside it — which
cpis which row pair, andwhich
kkis which column — and that is exactly what a 0.885-vs-0 correlation distinguishes. Two decodes that both pass the bias and bijectionchecks can still disagree on that, and only the repo's decode can settle it on a pack where the pack leg succeeds.
What I would do next, in order
--headmode on the existing tool, or a test case that stops after the tail),so this issue's baseline is reproducible;
_rai_1.9.0_npufor another model in~/models/oga/— the 1.5B and Hunyuan 0.5B are here) andcompare its head corr with this pack's;
I am not posting a decode patch: the two falsified hypotheses earlier in this thread are the argument for settling the baseline first.
bong-water-water-bong commentedon Nov 9, 2026
Two measurements that settle what the tail region is, plus one that says the baseline is unreachable on this box. All device-free.
1. The tail region is exactly the int4 tied head — to 4 decimal places
The tail is a tile grid of
V/64tiles each carryingHint4 codes plus the subvolume overhead, and the ratio of the whole region to a bare int4matrix is the same constant as the ratio inside one tile. So the premise holds: this region is the tied head, stored in the same tile format as the
attention payloads, and the ~21 % surplus is the scale/zero/bias envelope — not extra tensors. My earlier "the head does not fall out of the carve"
was about the decode, not the carve; the carve is right and now proven by arithmetic rather than by assumption.
2. Every 1.9 pack on this box is the hybrid genre, and every one is refused
model.binQwen2.5-0.5B-Instruct_rai_1.9.0_npu...qkv_proj.MatMulNBits.qweight.empty—.packedsibling carries no dataHunyuan-0.5B-Instruct_rai_1.9.0_npuQwen-2.5_1.5B_Instruct_rai_1.9.0_npuQwen2-1.5B_rai_1.9.0_npuQwen2.5-Coder-1.5B-Instruct_rai_1.9.0_npuSmolLM2-135M-Instruct_rai_1.9.0_npuSo the pack leg cannot pass for a 1.9 pack on current main: the projections were moved to
model.bin(that is the genre), the importer has nomodel.binreader (#1671/#1503), and the refusal that fires first is the.packed-sibling one, which reads as "these weights are genuinely absent"— accurate about
model.onnx.data, misleading about the pack. That is worth its own line in the refusal text: "absent from model.onnx.data" istrue; "genuinely absent from the pack" is not, and this pack is the counter-example (279.8 MB of them in
model.bin).Consequence for this issue: the "compare against the importer's own decode" baseline I proposed is not obtainable on this box — the importer
never reaches the tail on any pack it can open. The golden tile stays the only in-repo reference for the tile decode, and it validates the fields I
checked (bias 64/64, code-address bijection 2048/2048).
3. The next experiment, now that the premise is proven
With the tail confirmed as the int4 head, the assignment is solvable rather than guessable: the head is tied, so for the correct row/column map
code[r][c] ≈ round(emb[v(r)][c] / scale[r][kb])must hold elementwise. That is a direct equation in the data — sweep the plausible maps (3 row maps ×the 4 column spreads, 12 combinations) and take the one whose decoded values correlate ≈ 1 with
emb; a wrong map cannot coincide. If nonereaches ≈ 1, the quantisation on this pack is not int4-code-plus-scale and the container's own writer needs reading (the golden's 1.8 provenance).
I will report the 12-way result before proposing any decode change.
bong-water-water-bong commentedon Nov 9, 2026
Correction to my own last comment, and a sharper place to look — the per-row numbers attacked the "scale encoding" hypothesis and it does not survive.
A pure scale error would not lower the correlation at all. Correlation is invariant to a global factor, so if the codes were proportional to the weights and only the scale were misread,
corrwould be ≈ 1 no matter how wrong the magnitude was. Measuredcorris 0.89 per row, so the codes carry the ~12 % disagreement; the magnitude being ~24–70× off is a separate, second fact.The per-group breakdown of one row says the same, uniformly:
Nothing is localized: every one of the 28 groups is 0.85–0.93, the same band as the whole-row number. And the raw pairs show the codes behaving sensibly against the embedding — zeros land on zeros, signs agree — while the values sit ~70× low with a quantum of ~0.007 (so a 16-level code over the embedding's ±1.2 spread would need a scale of ~0.17).
What that leaves. The bytes are at the documented positions (the transform is self-consistent, tiles survive
decode → encode, parameters round-trip exactly), and the code → (row, column) mapping is not fully right. A partial scramble inside the subvolume is the shape that produces exactly this: values drawn from the correct row's code set, so signs and zeros still track the reference and correlation stays high, while ~12 % of positions are the wrong element of that set.Two candidates, and the documented layout discriminates them:
(cp%4) + kk*4 + (cp/4)*512holds rows2*cp(low nibble) and2*cp+1(high), which is what the attention path reads. If the tail tile orders its row pairs differently, half the rows would be pairwise swapped — cheap to test (swap the nibble assignment and re-measure: 0.89 → ~0.99 or worse);K = 896tile, where(cp/4)*512assumes the K-blocks that the attention geometry implies. If the tail's tile packs the same 896 columns into a different number of subvolumes, a column-block shift inside each 32-column group would do it.Next measurement (one build each), in order: swap the nibble assignment in the tail-path decode and re-measure the correlation; then, if that is not it, correlate the decoded row against the embedding shifted by 32 columns to detect a block shift. Both are the same tiny
NPU_PACK_DEBUGblock that produced the numbers above, and both are decided by one number —corrmoving to ≥ 0.99 or not.Already excluded, so nobody repeats them: row order and row offset (uniform across rows, boundary rows included, shift-1 control 0.065); zero-point sign (0.585, worse); the tail length arithmetic and the parameter round trip (exact); a pure scale error (correlation would be 1).
bong-water-water-bong commentedon Nov 9, 2026
The 12-way sweep, reported as promised. One convention confirmed, one thing eliminated, and a precise pointer to what is left.
Confirmed:
code = round(w / scale) + zp(the q4_1 form with zero point), and rows are identity-mappedTesting the tied constraint directly on integer codes rather than on correlations:
E = round(emb/scale)E = round(emb/scale) + zpChance for a 0–15 code is 0.0625, so 0.2795 is ~4.5× chance and 0.59 on the best row: the zero-point convention is right, the rows are (identity) aligned,
and the decoded values are in the right distribution. That also re-confirms the earlier
(c−zp)·scaletable was the right formula and that my"row map" and "permutation" hypotheses were both wrong for the right reasons.
Eliminated: it is not a column permutation of a correct tile
I tried to read the permutation off a single 128-code block (row 0, k-block 0) by rank-matching the codes:
If the block were correct-but-shuffled, the multisets would be equal and the permutation would exist; they are not equal, and 7 % of the expected codes
fall outside the representable 0–15 range. So the scale/zero pair I extract for this block does not belong to the codes I extract for it.
The pointer: the head tile's envelope is not the attention tile's
That is consistent with everything else measured here: the head geometry is confirmed (78.82 MB =
V/64tiles × 34816 B, ratio 1.2143 matching theintra-tile ratio), and my envelope offsets
zeros_off = 64·32,scales_off = +64were validated on the shipped 1.8 attention golden (bias 64/64exact, code addresses a 2048/2048 bijection). So the arithmetic is the container's, but the 1.9 head tiles apparently do not put scale/zero/bias at
those offsets — either a different envelope order for the tail region, or a different
rowssplit (the total is pinned; the internal split is not).Next, and it is a small search rather than a theory: for one head tile, scan the envelope offsets (
zeros_off/scales_offover the subvolume's tailbytes) and the
rowssplit, scoring each byround(emb/scale)+zp == codeexact-match rate. The correct combination should jump from 0.28 to ≈1.0, andunlike correlation it cannot be reached by accident — which is why I am not proposing a decode change before it does.
bong-water-water-bong commentedon Nov 9, 2026
Two negative results from the measurements I said I would run, plus a trap that cost them — worth writing down so the next attempt starts one step ahead.
1. The nibble order cannot be tested by editing
decode_tilealone. Swappingb & 0x0F/b >> 4on the decode side makesencode_tile(its exact inverse) disagree, and the import refuses before any oracle runs:That round trip is tautological — decode followed by its own inverse always round-trips — so it pins nothing about the vendor's intent, and it blocks every one-sided experiment. Any mapping hypothesis (nibble order, subvolume column stride, row-pair grouping) has to be applied to decode and encode together, or the check masks the oracle.
2. A hand-rolled re-decode has to be validated against the shipped one before its scores mean anything. I wrote the variant scorer straight from the raw tile bytes (to test several mappings in one build without touching
decode_tile, since each touch costs a build). Itsbasevariant — the addressing that is supposed to reproduce the shipped decode — scored 0.22 on row 0 and 0.09 on row 500, against the shipped decode's 0.89. So every variant score in that run (highnibble 0.19,lane-rev −0.001,cp+1 0.20) is noise from my addressing, not evidence about the pack. I corrected the two mistakes I found (subv_bytes(64)instead of64 × H, and theh = row/32,cp = (row%32)/2split rather thanrow/64,row/2) andbasewas still 0.22, so at least one more difference remains — most likely the tile-to-row index or the zero-point subvolume pairing.What is still true from this stretch: the transform is invertible and self-consistent, the parameters round-trip exactly, every column group of a row sits in the same 0.85–0.93 band, and a pure scale error cannot explain the correlation deficit (correlation is scale-invariant). Those four statements came from the shipped decode and stand.
The next attempt should start here: make the variant harness assert its own base case — if
basedoes not reproduce the shippedcorrto within a hair, the harness is wrong and the run means nothing. That check was the thing missing, and adding it would have saved this stretch. Then, with the harness validated, test the paired decode+encode variants one per build.No code stays behind: the diagnostic block is reverted and the tree is clean.
bong-water-water-bong commentedon Nov 9, 2026
Instrument up as #1710 —
tools/q4nx_tail_head.py+tests/q4nx_tail_head_test.py(9 checks, no pack needed). It reproduces this issue's measurement from a checked-in command, which matters because on current main no importer path reaches the tail on a 1.9 pack (the pack leg refuses all six of them at the projections — they are themodel.bingenre, #1671/#1503), so the 0.885 figure had no reproducible home.Printed on
Qwen2.5-0.5B-Instruct_rai_1.9.0_npu, along with the geometry that proves the carve:The test pins only what is settled: the geometry identities, the whole-tail == per-tile ratio equality, all 64 bias codes of the shipped vendor tile, and the code-address bijection (28672/28672, no collisions). It deliberately pins no correlation — today's number is the bug. Next from my side is the envelope-offset/row-split scan scored by exact-code match, which this tool makes cheap.
bong-water-water-bong commentedon Nov 9, 2026
Five structural hypotheses excluded, with a harness that now proves itself first. The numbers below are trustworthy in a way the last run's were not:
baseis the shipped decode on unpermuted bytes, and it reproduces the shippedcorr(0.8920 against 0.89 for rows 0–63), so every other column is a real comparison.Read: the shipped mapping beats every permutation of itself — nibble order (0.36), 128-lane reversal (−0.02), the 4-column subvolume rotation (0.07), and both parameter-pairing variants (scales read de-interleaved even-then-odd 0.885, zeros the same 0.827). None of them is the missing 0.11, and each one is worse or equal, which also says the code placement and the scale/zero pairing are both broadly right rather than scrambled.
The harness, since it is reusable and it cost three builds to get right: permute the raw tile bytes, decode the permutation with the shipped
decode_tile, rebuild aQ4withq_from_tiles({...}), and score withcorr(qv, embed_bytes, H, 64). Two properties are what make it valid, and both were missing from my earlier attempt:What that leaves. Codes and parameters are both approximately right and uniformly so, yet ~11 % of the pattern is unexplained at every granularity measured (row, column group, whole tile). The discriminating measurement I would run next, one build on the same harness: correlation per K-block (7 blocks of 128 for
H = 896) for one row. If some blocks score ~0.99 and others ~0, the defect is a K-block/subvolume offset that accumulates; if all seven score ~0.89, the codes themselves are subtly wrong at a granularity finer than a block, and the next suspect is the vendor's 4-column byte lane packing rather than any block-level addressing.Everything below stands from the previous measurement and is not re-litigated here: uniform across rows (0.875–0.904 including both tile boundaries), decoded values ~24–70× low with a ~0.007 quantum,
corr(|x|,|e|) ≈ 0.70, zero-point sign excluded (0.585), a pure scale error excluded by scale-invariance, tail arithmetic and parameter round trip exact.tools: the tied head is the fconst's LAST grid, and at that offset th…
bong-water-water-bong commentedon Nov 9, 2026
Found it: the head is the
.fconst's last grid, and at that offset the decode is exactThe instrument in #1710 answered this rather than only measuring it.
attn_payload_off(H)is the wrong base — the.fconstcarries theper-layer payload first and ends with the tied head's tile grid, so reading from
envaddresses the wrong bytes by the whole preceding payload.At
the same decoder that correlated at chance now reads:
on
Qwen2.5-0.5B-Instruct_rai_1.9.0_npu(V 151936, H 896) against the pack's own fp16model.embed_tokens.weight.What that settles
zeros_off = 64·nsubv,scales_off = +64, bias per subvolume,base = (cp%4) + (cp/4)·512,code = round(w/scale) + zp— every one of those is now confirmed by an exact match, not by a correlation argument. The golden tile checks thatsurvived earlier (bias 64/64, address bijection 28672/28672) were the right things to trust.
rel_errunder 1e-3. So the tied headis exactly the embedding, and the "per-value semantics" question I opened with does not exist — the numbers were the right bytes read from the
wrong place, with the ~2× magnitude being the per-row scales of a different tensor block.
an offset. Rank-matching, sorted-value matching, Gram/rotation, multiset equality — each was measuring
env-based bytes.Landed as #1710
tools/q4nx_tail_head.pynow useshead_offset()and prints the numbers above;tests/q4nx_tail_head_test.pypins the rule three ways — it isfconst_bytes - ntiles·tile, the grid ends exactly at EOF, and it is notattn_payload_off(H)— so the mistake this issue records cannot comeback. The correlation itself is still not pinned (1.0000 now, but a test on a pack-less tree cannot reach it; the offset rule is the durable part).
What is left for this issue
The decode question is answered; what remains is whether
npu/pack_import.cppshould read the tied head from the last grid (a real import change,worth its own PR against #1503's
model.binreader work), and whether theattn_payload_off(H)comment in the importer that led me here should saythat the tail head is addressed from the file's end.
npu-pack: the tied-head self-check read its own bf16 reference buffer…
bong-water-water-bong commentedon Nov 9, 2026
Solved, and the answer is that the decode was never wrong — the self-check was reading its own reference buffer with the wrong decoder.
import_pack()builds ONE reference buffer —embed_bf16,as_bf16over the pack's embedding whatever the source type — and pointsembed_bytesat it. Four readers then decoded it as f16:corr()itself, and theembed->type == kF16branches of the two relative-error loops, whose condition is about the pack's declared type, not about the buffer they are reading. On this pack the graph declarestype=10 (FLOAT16)formodel.embed_tokens.weight, so the branch looked right and read bf16 bytes through an f16 decoder.The evidence that settled it, after the permutations had already excluded nibble order, lane reversal, subvolume rotation, scale de-interleave and zero de-interleave: reading the same bytes as bf16 instead of f16 moved the correlation from 0.892 to 1.0000 for rows 0–63, while the reference's provenance printed as
type=10 location=model.onnx.data offset=11599872— a tensor the importer had already converted to bf16 before any check saw it.Everything this issue recorded as a property of the decode — values ~24–70× low, a quantum of 0.007,
corr(|x|,|e|) 0.70, the uniform 0.85–0.93 band across rows, columns and all seven K-blocks — was that misreading. It is worth saying plainly: five structural hypotheses were excluded correctly, against a decode that had no defect.Fixed in #1675 (
79aac1b2): all four sites read the buffer as bf16, the tied self-check passes at 0.999999,lm_head.weightimports into the container, and the pack leg printsall npu-pack checks passed (including the pack leg). The refusal added earlier in that branch (a tied pack whose tail fails its own oracle) stays as a guard for a genuinely bad tail, and no longer fires here.Closing. The general lesson for the importer, worth carrying past this issue: a self-check must decode its reference with the same conversion its own call site applied, and a declared ONNX type is not that guarantee — the buffer is what the code built, not what the pack said.
bong-water-water-bong commentedon Nov 9, 2026
Your question is answered in #1710's latest commit (
9eb4e991), so stop measuring the env-based offset — it reads the wrong bytes.The head is the fconst's last grid, not the region after
env:At
size − ntiles*tilethe same decode correlates 1.0000 with the pack's own fp16 embedding. That explains the value you were chasing:corr 0.885 / rel_err 0.987is not a bad envelope, a bad scale or a poisoned zero — it is the layer payload read as if it were the head, which is why the correlation was at chance rather than near-1 rather than, say, 0.97.Your 17:11 comment ("the tail region is exactly the int4 tied head — to 4 decimal places") was measuring the right region with the right arithmetic; the missing piece was that the region's start is anchored at EOF, not at the end of the env block. Since 2374 × 34816 = 82657280 and the payload offsets you printed are ~82 MB apart, the two readings differ by exactly the payload — which is why the sizes matched while the bytes did not.
#1710 now pins that rule with three assertions, one of which is
head_offset != attn_payload_off(896)labelled as "the mistake engine#1693 records", so the wrong rule cannot come back quietly. When it merges,tools/q4nx_tail_head.pyshould reproduce the 1.0000 on your pack in one command, and that number belongs in this issue's thread — the tool's PR deliberately asserts no correlation threshold, because picking one is this issue's call, not the instrument's.What is left on #1693 after that: the importer still does not read this head (the wiring, alongside #1712's re-landed
decode_external_tile), and whether the head should be read from the fconst at all for a tied pack — the pack carriestie_word_embeddings: true, so the honest answer may be that the tail head is redundant rather than wrong. That is a design question the 1.0000 sharpens instead of answering.tools: say what the importer actually does -- layers*stride + env, wh…
bong-water-water-bong commentedon Nov 9, 2026
Correction to my own wording above, before anyone acts on it: the importer already reads the head at the right offset.
pack_import.cppderives it aslayers * stride + attn_payload_off(H)— past the per-layer payload, then the tail's own envelope — and verifies it against the remaining file size (want == have). For this pack that is209000256 + 1856 = 209002112, exactly thesize - ntiles*tileI arrived at by subtraction.So there is no importer bug in the head's address, and my "
attn_payload_off(H)is the wrong base" sentence was only true without thelayers*strideterm — which is precisely the term a reader (me, for four hypotheses' worth of this thread) drops. The issue's 0.885 is therefore a property of whichever reader produced it, not of the importer's own arithmetic.#1710now says so in both docstrings and pins the agreement in the test:head_offset == 209000256 + attn_payload_off(896)for this pack. The tool still reads corr +1.0000, rel_err 0.0007 at that offset — so what stands from tonight is the decode confirmation (envelope offsets, bias subvolumes, code expression), not a fix to the importer.q4nx_tail_head: the 0.885 this records is #1661's decode bug, not an …
tools: make #1693's tail-head measurement a command (instrument; geom…
bong-water-water-bong commentedon Nov 9, 2026
Cause found, and it is not the pack: engine#1734 (#1721) — the importer's
embed_byteswas a bareconst uint8_t*and every consumer kept decoding F16 after the bytes became bf16 (#1661). That produces exactly this issue's title numbers —corr 0.885220,rel_err 0.987007— against the container's own0.999999 / 0.000700, and it is why atie_word_embeddings: truepack was read as untied.So the record here is consistent end to end: the tail decode is correct (corr 1.0000, rel_err 0.0007 at
layers*stride + attn_payload_off(H), measured in the comment above), the address arithmetic in the importer was correct too, and the 0.885 was a format mismatch one layer above both. My five dead hypotheses in this thread were all chasing a decode that was never wrong — which is precisely the cost #1734 exists to stop.Leaving this open only for the two things I asked for there: the F16 branch of
EmbedBuf::atis reachable but unexercised (it is the branch that bit), and no test pins the corrected 0.999999 against the 0.885. Once those land, this issue has nothing left of its own.bong-water-water-bong commentedon Nov 9, 2026
The tail head is present in these packs — measured on a full download, and the loader lane now has a concrete case
Adding a measured data point to the "tied head does not reproduce" question, from the other direction: the 1.9 packs I had been calling
placeholder-only are not missing their head.
tools/q4nx_tail_head.pyon the fullamd/Qwen-2.5_1.5B_Instruct_rai_1.9.0_npu(2.3 GB):So the carve is exact (2374 = V/64, ratios agree) and the envelope decode is what does not reproduce the embedding/spec — which is the open question this
issue already names. Three consequences for the queue:
.emptyplaceholder in these packs is a red herring, not a missing weight: forSmolLM2-135Mthe fconst's90,470,776 B splits as 30 x 2,360,292 B of layer blocks plus 19,662,016 B, which is 1.389x the int4 head (49,152 x 576 / 2 = 14,155,776) — a tail;
model.onnxalone is unsafe. I drew one for 13 packs today andhave retracted it on the board (they are
blocked: head in fconst tail, reader gap), citing this issue and npu-pack: plan() took the first MatMulNBits as the head, so it reported a fused qkv as lm_head.weight (refs #1671) #1732;Qwen-2.5_1.5B_Instructis a ready-made end-to-end case: H 1536, V 151936, tie_word_embeddings true, and its layer ops areall block 128 with a single block-32 head, which is exactly what npu-pack: a pack whose head is at another block imports, if it is tied (engine#1503) #1633's head tolerance was written for.
bong-water-water-bong commentedon Nov 9, 2026
The tail head is not a tied-head property:
Phi-3-mini-128k-instruct(untied) has the same region, measured on the full packA second case for whoever takes the envelope decode, and one that widens the scope of this issue. Full 4.2 GB pack,
tools/q4nx_tail_head.py:501 = 32064 / 64 exactly, ratios agree, decode correlates with nothing — the same signature as
Qwen-2.5_1.5B_Instruct(2374 = 151936/64,ratio 1.1510). The difference that matters:
Phi-3-mini-128k-instructistie_word_embeddings: false, so this is not a tied pack reading its ownembedding back. The head region exists in the
.fconsttail whether or not the head is tied, which means:Llama-3.1-8B,Llama-3.2-3B-Instruct,Meta-Llama-3-8B/-3.1-8B-Instruct,Qwen3-8B,granite-4.1-8b,HY-MT1.5-7B,Hunyuan-7B-Instructare all in thatclass on npu-pack: the 1.9 packs split on the MatMulNBits K block size (32, and mixed 32/128), before the hidden-896 limit is reached #1503);
model.embed_tokens.weight, which an untied pack does not share with the head. Thegeometry numbers above still hold (they are pure arithmetic on V, H and the tile envelope), so the carve can be validated on untied packs even where the
correlation check cannot.
Two packs measured on full downloads, same conclusion, one tied and one untied. I have corrected the board: no 1.9 row is called a "defective export" on the
strength of its ONNX graph any more — those rows are
blocked, head in the fconst tail, reader gap, with this issue and #1732 cited.npu-pack: the tail-head tool refuses a block-32 head and an untied he…