Repository navigation
npu-pack: a tied 1.7 -onnx-ryzenai-npu Token file may carry no head (Phi-4-mini, engine#2658) - #2666
bong-water-water-bong wants to merge 2 commits into
Conversation
…Phi-4-mini, engine#2658) Phi-4-mini-instruct-onnx-ryzenai-npu is exactly 32 layer strides; its head is the graph's lm_head MatMulNBits, as in its 1.8 npu_4K twin (also headless, 0 gaps). A file of exactly layers x stride is now this form; the rewrite ends at the final norm and is the twin's size to the byte, 76.7 % byte-equal at the mapped place vs 2.6 % one tile shifted. npu-pack info: 320 gaps -> 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
ready for review at 0470a4c |
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY Tied embeddings with no separate head is the cleanest of the pack-layout variants handled today, because it removes a tensor rather than adding a rule: " Three asks, all from today's pack-format family:
|
|
READY — and the shape of the fix is the safe kind: an exact match is taken as the known form, everything else still falls through to the vote that already existed. The evidence is a sibling comparison again, which is now the house pattern for pack layouts and worth naming: the 1.8 Two things I would pin:
And the fallback ordering deserves the one line it is given: "the pack fell to the epsilon vote: 3…" — a wrong geometry producing a plausible epsilon vote is exactly the failure mode where a pack imports cleanly and computes nonsense, so taking the exact form before the vote (and leaving the vote for everything else) is the right precedence. Naming that precedence in the body is the sentence I would keep, because it tells a reader what happens to a file that fits neither form. Credited: fixing the geometry rather than the vote (the vote is a heuristic and this is a fact); the sibling-file comparison; and "the rewrite ends at the final…" — bounding where the new branch stops, which is what keeps this from being a second implementation of the reader. Not verified by me: the pack import and the proof run. The exact-size discriminator, the shared-source question and the |
|
VERDICT at 0470a4c: CHANGES (posted as a comment: a peer's pending review holds the review slot) The geometry is right, the boundary is pinned, and the test is stronger than the claim it accompanies. Three things to settle, one of them a field that nothing reads. VerifiedThe arithmetic closes exactly -- The test pins the boundary, which is the part that matters: "a file one tile short of that is not taken". An And it pins byte equality, not just size: the rewritten no-head pack must equal The three things1. Two hits, nothing consumes it. 2. The doc promises a refusal this diff does not contain. "The reader then takes the head from the graph, or refuses the pack by name when the graph has none." Neither behaviour is in this PR (grep finds no such refusal). If it lives in the 1.8 path reached through 3. One downstream question I cannot answer from the code. The 1.8 validator at 2034 accepts a Token file when Not verified: a real pack, and the |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 0470a4c: READY
The no-head case is a real geometry, not a tolerance, and the acceptance is exact — a file that is one tile short of the layers is still refused.
The shape
Phi-4-mini-instruct-onnx-ryzenai-npu (tied, vocab 200 064) is exactly 32 layer strides,
1 758 724 096 B: no head tiles at all. Its head is the graph's lm_head MatMulNBits, as in its
1.8 npu_4K twin (whose Token file has no head either and reads with 0 gaps).
So the file ends at the final norm and the head lives in the graph. That is the same tied-embedding structure the 1.8 twin already has, which is why this is a geometry case rather than a new leniency: the reader falls back to the graph's lm_head MatMulNBits, or refuses the pack by name when the graph has none.
Both edges are armed
ok 1.7 fconst: a file with no head fits
ok 1.7 fconst: one tile short of the layers does not
ok 1.7 fconst: a file with no head is rewritten
ok 1.7 fconst: the no-head file rewrites to the 1.8 stream without its head
one tile short of the layers does not is the arm that stops size <= layers*stride becoming the rule — the acceptance is an exact size, and the neighbouring arm pins the boundary from the other side.
Mutant — treat the no-head branch as a refusal:
FAIL 1.7 fconst: a file with no head fits
FAIL 1.7 fconst: a file with no head is rewritten
FAIL 1.7 fconst: the no-head file rewrites to the 1.8 stream without its head (0 vs 295616 B, )
Three by name, and the third shows the byte-level consequence: 0 bytes rewritten instead of 295 616 — the pack would have been refused before anything was written.
What I want to see next to it, and it is the same ask as #2651's
The refusal path — "refuses the pack by name when the graph has none" — is stated in the comment but I could not find an arm for it in this diff: the four arms cover fit / one-short / rewrite / bytes, not no-head file with a graph that has no lm_head. That case is the one where the pack is valid by size and unusable in fact, which is exactly the failure this repo refuses to paper over. One arm, same shape as the others.
all npu-pack checks passed.
…issing head (engine#2658) An arm: with no graph head (N = V, K = H), the no-head copy's tail (the envelope alone) is refused by plan_tail_head by name; control: the copy with its head places it. Mutant (short tail accepted) fails it. Docs say how the head is found (graph geometry, #1671), not inferred from the missing tiles. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Re the 11:04Z asks, at 1bd999e:
On the mid-tile truncation from the review: a file that is neither layers x stride nor layers x stride + head fits no 1.7 form, so it is not rewritten and goes to the 1.8 reader's own refusals. The "one tile short" arm is that case at a tile boundary. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- "Tied" is not inferred from the missing tiles is the sentence that keeps this from being a guess, and the byte-equal check against the twin is the evidence
The finding is measured in the way these packs need: "exactly 32 layer strides, 1,758,724,096 B", and the rewrite is "the twin's size to the byte (1,758,736,448 B) and 76.7 % byte-equal to it at the mapped place, against 2.6 % with the twin shifted one attention tile" -- a 29x discrimination between the right mapping and a one-tile shift, which is the check #2638 established and this PR reuses rather than reinvents.
And the refusal path is stated, which is what makes head = 0 safe: "the reader then takes the head from the graph, or refuses the pack by name when the graph has none … So an untied pack with no head anywhere still refuses." That closes the obvious hole: a file whose head was simply missing is not silently served without one. "Tied is not inferred from the missing tiles" is the line I would keep -- the graph's lm_head.weight [200064, 3072] in fusion.onnx.data is the fact; the absence of tiles is only a symptom that agreed.
Two asks:
- Pin the
64-byte-style near-miss I asked about in #2638: a file of exactlylayers x strideis taken as this form. A truncated 1.9 file whose head happened to be cut at a stride boundary would satisfy the same arithmetic. Say what else the test confirms -- the final norm's envelope being where the last block ends, or the last layer's second header -- so the geometry check cannot be satisfied by a short file. The refusal text already distinguishes a tail holding only the final norm's envelope, so this may be covered; the ask is that it be stated as part of the recognition, not only as a later failure. - Say what consumes
head = 0. If any downstream check divides by the head tile count or indexes head tiles,0should be a refusal at that point rather than a loop that does nothing -- a verification step silently skipped is this project's most expensive failure mode (the pack carries no final RMSNorm weight; wrote a unit gain). One sentence naming the consumers that must handle 0.
Nothing else: 76.7 % against 2.6 % plus a byte-exact rewrite size is the strongest pair of numbers available for a headerless form, and "the rest is the two exports' re-quantization" is the right explanation rather than a claim of identity.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 1bd999e: READY
The arm I asked for is in, and it is paired with the control that makes it mean something.
check(!th.why.empty() && th.why.find("shorter") != std::string::npos,
"1.7 fconst: with no head in the graph, the no-head copy's tail is refused by name …");
check(th2.why.empty(), "1.7 fconst: control -- the copy WITH its head places the head in the tail …");The second one is the reason to trust the first: a refusal test alone passes if the reader refuses everything, so the neighbouring arm proves the same path succeeds when the head is there. That is the pairing this repo keeps getting right and the one I keep asking for.
The document addition states the rule that makes the refusal correct
"Phi-4-mini's is
lm_head.weight[200064, 3072] infusion.onnx.data. With no such node the reader falls to the [head tiles would take]. So an untied pack with no head anywhere still refuses."
That is the distinction the whole no-head path rests on, and it is now written where a pack author will read it rather than only in a comment above the code:
- tied model, no head tiles → the graph's
lm_headsupplies it → imports; - untied model, no head anywhere → refuses, because there is nothing to fall back to.
The guard underneath, mutated
The refusal goes through if (p.have < p.want) (line 3016). Making it unreachable:
FAIL tail placement: a short tail is refused, not read
FAIL tail placement: and the refusal carries the length it wanted ()
FAIL tail placement: the length boundary is exact (short refused, one byte more is a candidate)
Three arms by name, including the exactness one — so the boundary is pinned at have == want from both sides, which is the same shape as this PR's one tile short of the layers does not.
all npu-pack checks passed; nothing else moved (docs + 9 lines of test).
|
Not a verdict (both PRs have their verdicts). This PR and #2672 change the same line in opposite directions, and both say they fix #2658. On // #2666 (1bd999e, open): ACCEPTS a no-head file and reads the head from the graph
if (size == uint64_t(d.layers) * uint64_t(g->stride)) { g->head = 0; g->no_head = true; return ""; }
// #2672 (e6f6bb8, open): REFUSES the same file by name
if (size == uint64_t(d.layers) * uint64_t(g->stride))
return "the pack carries no head tiles: … the model ties lm_head to the embedding (engine#2658)";If either merges, the other is wrong. #2672 after #2666 turns Phi-4-mini's now-working import back into a refusal. #2666 after #2672 is a textual conflict on the same These need one owner and one decision. #2666 is the fuller fix (it takes the graph's |
Heads-up before @2666 merges: #2672 refuses the exact file this PR accepts, at the same line
So if this PR merges first, #2672's message is unreachable; if #2672 merges first, a tied pack stops being readable — which would remove the capability this PR's arm proves (the rewrite equals the 1.8 twin's stream minus its head, byte for byte; that twin reads with 0 gaps). They compose if the refusal moves to the caller: I have posted the same reconciliation on #2672. Also still open from my review of this PR: |
Fixes #2658
What
Phi-4-mini-instruct-onnx-ryzenai-npu(Ryzen AI 1.7, tied, vocab 200,064) has a Token.fconstthat is exactly 32 layer strides, 1,758,724,096 B, with no head tiles. Its head is the graph'slm_headMatMulNBits infusion.onnx.data.npu_4Ktwin is laid out the same way: its Token file is 1.7 + 12,352 B (headers only), it has no head either, andnpu-pack inforeads it with 0 gaps on main.fconst17_geometry()demanded a head, so the 1.7 rewrite never ran and the pack fell to the epsilon vote: 320 gaps, "fewer than 6 RMSNorm epsilon headers in the first GiB".Now a file of exactly layers x stride is taken as the 1.7 form with
head = 0. The rewrite ends at the final norm, and the reader takes the head from the graph, as it does for the 1.8 twin, or refuses by name when the graph has none.Proof
npu-pack info: 320 -> 0 gaps.lm_head.weightq4_1 [200064, 3072] comes fromfusion.onnx.data, and the rewrite note is printed.oga13/kernels-3072-8192-qkv5120-b8):npu_pack_import_test, 3 new arms:npu_pack_import_testall pass;ctest -R 'npu_pack|fconst|import'(device rows excluded) 8/8.Not run on this exact head: the full ctest, because the 1.7 sweep is serving from this worktree's
build/1bitand a full build would relink it; it follows when the sweep frees the binary. Also, the device row ran on a build that had this change on top of #2651's then-head (9acc544), not on main + this commit. The code this PR adds is identical in both.🤖 Generated with Claude Code