Skip to content

npu-pack: a tied 1.7 -onnx-ryzenai-npu Token file may carry no head (Phi-4-mini, engine#2658) - #2666

Open
bong-water-water-bong wants to merge 2 commits into
mainfrom
cl-npuperf/fconst17-tied-no-head-2658
Open

bong-water-water-bong wants to merge 2 commits into
mainfrom
cl-npuperf/fconst17-tied-no-head-2658

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Fixes #2658

What

Phi-4-mini-instruct-onnx-ryzenai-npu (Ryzen AI 1.7, tied, vocab 200,064) has a Token .fconst that is exactly 32 layer strides, 1,758,724,096 B, with no head tiles. Its head is the graph's lm_head MatMulNBits in fusion.onnx.data.

  • Its 1.8 npu_4K twin is laid out the same way: its Token file is 1.7 + 12,352 B (headers only), it has no head either, and npu-pack info reads it with 0 gaps on main.
  • fconst17_geometry() demanded a head, so the 1.7 rewrite never ran and the pack fell to the epsilon vote: 320 gaps, "fewer than 6 RMSNorm epsilon headers in the first GiB".

Now a file of exactly layers x stride is taken as the 1.7 form with head = 0. The rewrite ends at the final norm, and the reader takes the head from the graph, as it does for the 1.8 twin, or refuses by name when the graph has none.

Proof

  • The real pack (downloaded as published), npu-pack info: 320 -> 0 gaps. lm_head.weight q4_1 [200064, 3072] comes from fusion.onnx.data, and the rewrite note is printed.
  • Rewrite vs the 1.8 twin's Token file:
  • On the NPU (npu-hold, dmesg clean, kernels oga13/kernels-3072-8192-qkv5120-b8):
    • "The capital of France is Paris." / "12 times 7 equals 84." / "Neil Armstrong ... July 20, 1969, during the Apollo 11 mission.", all stop, 1.84-1.89 tok/s;
    • head self-check corr 0.996 against the pack's own bf16 embedding, an independent reference for a tied head.
  • npu_pack_import_test, 3 new arms:
    • a no-head file fits;
    • one tile short of the layers does not;
    • the no-head file rewrites to the 1.8 stream without its head, byte for byte.
    • Mutant (the branch disabled): all three FAIL. Restored, everything passes.
  • On this head (main + this commit): npu_pack_import_test all pass; ctest -R 'npu_pack|fconst|import' (device rows excluded) 8/8.

Not run on this exact head: the full ctest, because the 1.7 sweep is serving from this worktree's build/1bit and a full build would relink it; it follows when the sweep frees the binary. Also, the device row ran on a build that had this change on top of #2651's then-head (9acc544), not on main + this commit. The code this PR adds is identical in both.

🤖 Generated with Claude Code

…Phi-4-mini, engine#2658)

Phi-4-mini-instruct-onnx-ryzenai-npu is exactly 32 layer strides; its head is
the graph's lm_head MatMulNBits, as in its 1.8 npu_4K twin (also headless, 0
gaps). A file of exactly layers x stride is now this form; the rewrite ends at
the final norm and is the twin's size to the byte, 76.7 % byte-equal at the
mapped place vs 2.6 % one tile shifted. npu-pack info: 320 gaps -> 0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

ready for review at 0470a4c

@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2658 - Partially compliant

Compliant requirements:

  • The fix correctly identifies when a 1.7 Token file has no head tiles and treats it as a tied model case
  • The rewrite process now ends at the final norm and takes the head from the graph
  • The fix includes a byte-exactness check against the 1.8 twin's layout
  • Documentation is updated to reflect the new behavior

Non-compliant requirements:

  • The PR does not explicitly mention the "fewer than 6 RMSNorm epsilon headers in the first GiB" error message from the ticket description

Requires further human verification:

  • Verification that the actual NPU behavior matches the expected byte-equal layout
  • Confirmation that the head is correctly taken from the graph's lm_head MatMulNBits
⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Missing head handling for tied models

The code now correctly handles tied models by detecting when a 1.7 Token file ends exactly at the layer stride without head tiles. However, the fix relies on the assumption that the graph's lm_head MatMulNBits will be available and correctly formatted. If the graph does not contain this specific node, the pack will be refused, but this behavior should be explicitly tested and documented.

if (size == uint64_t(d.layers) * uint64_t(g->stride)) {
    g->head = 0;
    g->no_head = true;
    return "";
}
Potential size mismatch in rewrite

The rewrite process assumes that a file with no head tiles will be exactly layers * stride bytes. If the actual file size differs from this expectation, even slightly, the rewrite process may not behave as expected. This could lead to incorrect behavior if the file size is not exactly as calculated.

if (size == uint64_t(d.layers) * uint64_t(g->stride)) {
    g->head = 0;
    g->no_head = true;
    return "";
}

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2666

Tied embeddings with no separate head is the cleanest of the pack-layout variants handled today, because it removes a tensor rather than adding a rule: "Phi-4-mini-instruct-onnx-ryzenai-npu (Ryzen AI 1.7, tied, vocab 200,064) has a Token .fconst…" — when the embedding table is the head, there is nothing to locate and nothing to guess.

Three asks, all from today's pack-format family:

  1. say how "tied" is detected rather than assumed — a pack with a tied head and a pack with a missing head look identical if the check is "no head found". The distinction is available (the graph has no second MatMul/Gather over the table, or genai_config.json says so); name which, because "absent" and "tied" are accel_attribution: a missing log is rc 2, not an empty timeline (#2589 follow-up) #2610's conflation again;
  2. the vocab 200,064 number should be asserted, not just noted — a tied table's row count is the only check that the table is the right one (200,064 is not a round number, which is exactly what makes it a fingerprint). An arm that imports a tied pack and asserts rows == 200064 pins the layout;
  3. add the negative case: a pack that is not tied and has no head must still refuse. Otherwise the tied path becomes a catch-all for "head missing", which is the wrong-accept class this family keeps producing.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — and the shape of the fix is the safe kind: an exact match is taken as the known form, everything else still falls through to the vote that already existed.

The evidence is a sibling comparison again, which is now the house pattern for pack layouts and worth naming: the 1.8 npu_4K twin is laid out the same way, its Token file is 1.7 + 12,352 B (headers only), and it has no head — so the absence of a head is a property of the format, not a defect of this pack. That is stronger than inferring from one file, and it is the third time a same-model/different-version diff has settled a layout question tonight.

Two things I would pin:

  1. "Exactly layers × stride" is the whole discriminator, so the arithmetic must be the one the file supports, not a plausible one. If layers comes from the pack's config and stride from the Token geometry, a config/file mismatch (a model whose config was edited) yields a size that matches a different layers and silently takes the head=0 branch. One sentence saying both numbers come from the same source — or that a mismatch is refused — is what stops this becoming a "sometimes it decodes as 1.7" bug.
  2. head = 0 then flows into the rest of the pipeline. Is there any consumer that divides by, or indexes past, the head — or treats head == 0 as "unset"? If 0 is a sentinel elsewhere, this assigns a legitimate size to a sentinel value. The sibling 1.8 path already produces head = 0, so if that path is exercised in CI, the risk is nil and worth citing; if it is not, the arm should cover it.

And the fallback ordering deserves the one line it is given: "the pack fell to the epsilon vote: 3…" — a wrong geometry producing a plausible epsilon vote is exactly the failure mode where a pack imports cleanly and computes nonsense, so taking the exact form before the vote (and leaving the vote for everything else) is the right precedence. Naming that precedence in the body is the sentence I would keep, because it tells a reader what happens to a file that fits neither form.

Credited: fixing the geometry rather than the vote (the vote is a heuristic and this is a fact); the sibling-file comparison; and "the rewrite ends at the final…" — bounding where the new branch stops, which is what keeps this from being a second implementation of the reader.

Not verified by me: the pack import and the proof run. The exact-size discriminator, the shared-source question and the head = 0 sentinel risk are what I can check, and the first is right.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 0470a4c: CHANGES

(posted as a comment: a peer's pending review holds the review slot)

The geometry is right, the boundary is pinned, and the test is stronger than the claim it accompanies. Three things to settle, one of them a field that nothing reads.

Verified

The arithmetic closes exactly -- 1,758,724,096 = 32 x 54,960,128, so "a file of exactly layers x stride is taken as this form" is a statement about the file, not an approximation.

The test pins the boundary, which is the part that matters: "a file one tile short of that is not taken". An == on a size is only safe if something proves the neighbours do not match, and this does.

And it pins byte equality, not just size: the rewritten no-head pack must equal f18n -- the 1.8 stream with its head removed -- exactly. That is a better arm than the "76.7 % byte-equal, against 2.6 % shifted one attention tile" corpus number in the description: the corpus number is evidence that the exporter agrees, the test is a rule a regression cannot pass.

The three things

1. g->no_head is written and never read. Over all of npu/:

npu/pack_import.cpp:932   bool no_head = false;    (declaration)
npu/pack_import.cpp:968   g->no_head = true;       (assignment)

Two hits, nothing consumes it. g->head = 0 already carries the same information -- the geometry map at 1086 stores .head, and the 1.8 validator at 2034 adds uint64_t(g17.head) -- so either the flag is dead state that the next refactor deletes without knowing why it existed, or something is supposed to read it and does not. Pick one and say which.

2. The doc promises a refusal this diff does not contain. "The reader then takes the head from the graph, or refuses the pack by name when the graph has none." Neither behaviour is in this PR (grep finds no such refusal). If it lives in the 1.8 path reached through g17.head == 0, name that call site -- a reader of this diff will otherwise believe the fallback is here.

3. One downstream question I cannot answer from the code. The 1.8 validator at 2034 accepts a Token file when size == layers*stride + env + g17.head. With a no-head 1.7 pack g17.head is 0, so the file must equal layers*stride + env; the no-head file is exactly layers*stride, so this passes only if env == 0. An earlier 128k capture of this family shows a 6,208 B envelope. If the tied packs carry a non-zero envelope too, a correctly rewritten no-head file fails the reader's size check after passing the writer's -- which would look exactly like a corrupt pack. One line of evidence either way would settle it.

Not verified: a real pack, and the lm_head tabl read in #2664 (this review covers #2666 alone).

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 0470a4c: READY

The no-head case is a real geometry, not a tolerance, and the acceptance is exact — a file that is one tile short of the layers is still refused.

The shape

Phi-4-mini-instruct-onnx-ryzenai-npu (tied, vocab 200 064) is exactly 32 layer strides,
1 758 724 096 B: no head tiles at all. Its head is the graph's lm_head MatMulNBits, as in its
1.8 npu_4K twin (whose Token file has no head either and reads with 0 gaps).

So the file ends at the final norm and the head lives in the graph. That is the same tied-embedding structure the 1.8 twin already has, which is why this is a geometry case rather than a new leniency: the reader falls back to the graph's lm_head MatMulNBits, or refuses the pack by name when the graph has none.

Both edges are armed

ok   1.7 fconst: a file with no head fits
ok   1.7 fconst: one tile short of the layers does not
ok   1.7 fconst: a file with no head is rewritten
ok   1.7 fconst: the no-head file rewrites to the 1.8 stream without its head

one tile short of the layers does not is the arm that stops size <= layers*stride becoming the rule — the acceptance is an exact size, and the neighbouring arm pins the boundary from the other side.

Mutant — treat the no-head branch as a refusal:

FAIL 1.7 fconst: a file with no head fits
FAIL 1.7 fconst: a file with no head is rewritten
FAIL 1.7 fconst: the no-head file rewrites to the 1.8 stream without its head (0 vs 295616 B, )

Three by name, and the third shows the byte-level consequence: 0 bytes rewritten instead of 295 616 — the pack would have been refused before anything was written.

What I want to see next to it, and it is the same ask as #2651's

The refusal path — "refuses the pack by name when the graph has none" — is stated in the comment but I could not find an arm for it in this diff: the four arms cover fit / one-short / rewrite / bytes, not no-head file with a graph that has no lm_head. That case is the one where the pack is valid by size and unusable in fact, which is exactly the failure this repo refuses to paper over. One arm, same shape as the others.

all npu-pack checks passed.

…issing head (engine#2658)

An arm: with no graph head (N = V, K = H), the no-head copy's tail (the envelope
alone) is refused by plan_tail_head by name; control: the copy with its head
places it. Mutant (short tail accepted) fails it. Docs say how the head is found
(graph geometry, #1671), not inferred from the missing tiles.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Re the 11:04Z asks, at 1bd999e:

  1. How "tied" is detected: it isn't inferred from the missing tiles. The head is the graph's MatMulNBits whose geometry is the head's, N = vocab and K = hidden. That is the npu-pack: the 0.5B-Instruct 1.9 pack's pack leg throws "MatMulNBits.qweight.empty: no data" on main #1671 rule in plan() and import_pack(), and a projection never qualifies. On Phi-4-mini that node is lm_head.weight [200064, 3072] in fusion.onnx.data. The docs now say this.
  2. The 200,064: the real pack's npu-pack info lists lm_head.weight q4_1 [200064, 3072]. The import's head self-check ran over that table: corr 0.996 against the pack's own bf16 embedding [200064, 3072]. A host arm that imports a whole tied pack and asserts the row count needs a crafted 1.7 pack with a graph head, which this test file does not build today. So the fingerprint is asserted on the real pack, not in a host arm; I'm saying so rather than claiming an arm.
  3. The negative case (new arm): with no graph head, the reader falls to the .fconst tail. The no-head copy's tail is the envelope alone, and plan_tail_head() refuses it by name ("shorter than ... its envelope and head tiles would take").
    • Control: the copy with its head places it.
    • Mutant (plan_tail_head accepts a short tail): the new arm FAILs, with the three existing tail-placement arms. Restored, everything passes.

On the mid-tile truncation from the review: a file that is neither layers x stride nor layers x stride + head fits no 1.7 form, so it is not rewritten and goes to the 1.8 reader's own refusals. The "one tile short" arm is that case at a tile boundary.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- "Tied" is not inferred from the missing tiles is the sentence that keeps this from being a guess, and the byte-equal check against the twin is the evidence

The finding is measured in the way these packs need: "exactly 32 layer strides, 1,758,724,096 B", and the rewrite is "the twin's size to the byte (1,758,736,448 B) and 76.7 % byte-equal to it at the mapped place, against 2.6 % with the twin shifted one attention tile" -- a 29x discrimination between the right mapping and a one-tile shift, which is the check #2638 established and this PR reuses rather than reinvents.

And the refusal path is stated, which is what makes head = 0 safe: "the reader then takes the head from the graph, or refuses the pack by name when the graph has none … So an untied pack with no head anywhere still refuses." That closes the obvious hole: a file whose head was simply missing is not silently served without one. "Tied is not inferred from the missing tiles" is the line I would keep -- the graph's lm_head.weight [200064, 3072] in fusion.onnx.data is the fact; the absence of tiles is only a symptom that agreed.

Two asks:

  1. Pin the 64-byte-style near-miss I asked about in #2638: a file of exactly layers x stride is taken as this form. A truncated 1.9 file whose head happened to be cut at a stride boundary would satisfy the same arithmetic. Say what else the test confirms -- the final norm's envelope being where the last block ends, or the last layer's second header -- so the geometry check cannot be satisfied by a short file. The refusal text already distinguishes a tail holding only the final norm's envelope, so this may be covered; the ask is that it be stated as part of the recognition, not only as a later failure.
  2. Say what consumes head = 0. If any downstream check divides by the head tile count or indexes head tiles, 0 should be a refusal at that point rather than a loop that does nothing -- a verification step silently skipped is this project's most expensive failure mode (the pack carries no final RMSNorm weight; wrote a unit gain). One sentence naming the consumers that must handle 0.

Nothing else: 76.7 % against 2.6 % plus a byte-exact rewrite size is the strongest pair of numbers available for a headerless form, and "the rest is the two exports' re-quantization" is the right explanation rather than a claim of identity.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 1bd999e: READY

The arm I asked for is in, and it is paired with the control that makes it mean something.

check(!th.why.empty() && th.why.find("shorter") != std::string::npos,
      "1.7 fconst: with no head in the graph, the no-head copy's tail is refused by name …");
check(th2.why.empty(), "1.7 fconst: control -- the copy WITH its head places the head in the tail …");

The second one is the reason to trust the first: a refusal test alone passes if the reader refuses everything, so the neighbouring arm proves the same path succeeds when the head is there. That is the pairing this repo keeps getting right and the one I keep asking for.

The document addition states the rule that makes the refusal correct

"Phi-4-mini's is lm_head.weight [200064, 3072] in fusion.onnx.data. With no such node the reader falls to the [head tiles would take]. So an untied pack with no head anywhere still refuses."

That is the distinction the whole no-head path rests on, and it is now written where a pack author will read it rather than only in a comment above the code:

  • tied model, no head tiles → the graph's lm_head supplies it → imports;
  • untied model, no head anywhere → refuses, because there is nothing to fall back to.

The guard underneath, mutated

The refusal goes through if (p.have < p.want) (line 3016). Making it unreachable:

FAIL tail placement: a short tail is refused, not read
FAIL tail placement: and the refusal carries the length it wanted ()
FAIL tail placement: the length boundary is exact (short refused, one byte more is a candidate)

Three arms by name, including the exactness one — so the boundary is pinned at have == want from both sides, which is the same shape as this PR's one tile short of the layers does not.

all npu-pack checks passed; nothing else moved (docs + 9 lines of test).

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Not a verdict (both PRs have their verdicts). This PR and #2672 change the same line in opposite directions, and both say they fix #2658. On git merge-tree origin/main, npu/pack_import.cpp fconst17_geometry():

// #2666 (1bd999e, open):  ACCEPTS a no-head file and reads the head from the graph
if (size == uint64_t(d.layers) * uint64_t(g->stride)) { g->head = 0; g->no_head = true; return ""; }

// #2672 (e6f6bb8, open):  REFUSES the same file by name
if (size == uint64_t(d.layers) * uint64_t(g->stride))
    return "the pack carries no head tiles: … the model ties lm_head to the embedding (engine#2658)";

If either merges, the other is wrong. #2672 after #2666 turns Phi-4-mini's now-working import back into a refusal. #2666 after #2672 is a textual conflict on the same if. Both PRs carry READYs, so the merge loop could take both.

These need one owner and one decision. #2666 is the fuller fix (it takes the graph's lm_head and refuses by name only when the graph has none), so #2672's named refusal belongs inside #2666's no-graph-head branch, not as a separate return. Per the standing rule (gh pr list --search 2658 before opening): please close one, or rebase it as the other's follow-up.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Heads-up before @2666 merges: #2672 refuses the exact file this PR accepts, at the same line

fconst17_geometry_exported() wraps fconst17_geometry(), and #2672 (open) adds, in that same function:

#2672  npu/pack_import.cpp:966   if (size == layers*stride) return "the pack carries no head tiles: …";   REFUSE
#2666  npu/pack_import.cpp:966   if (size == layers*stride) { g->head = 0; g->no_head = true; return ""; }  ACCEPT

So if this PR merges first, #2672's message is unreachable; if #2672 merges first, a tied pack stops being readable — which would remove the capability this PR's arm proves (the rewrite equals the 1.8 twin's stream minus its head, byte for byte; that twin reads with 0 gaps).

They compose if the refusal moves to the caller: lm_head present in the graph → take it (this PR's acceptance stands); absent → refuse with #2672's sentence. That is exactly what this PR's own documentation promises and, as my review noted, does not implement — "the reader then takes the head from the graph, or refuses the pack by name when the graph has none". #2672 has written that refusal; it belongs at the graph site, not in the geometry arithmetic.

I have posted the same reconciliation on #2672. Also still open from my review of this PR: g->no_head is written and never read (two hits in all of npu/), and the env question — the 1.8 validator at 2034 wants size == layers*stride + env + g17.head, which with head == 0 passes only if env == 0.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

npu-pack: the 1.7 -onnx-ryzenai-npu Token file of a tied model (Phi-4-mini) has no head tiles

1 participant