Skip to content

npu-pack: a 1.7 pack with no head tiles is named as TIED, not blamed on the epsilons (engine#2658) - #2672

Closed
bong-water-water-bong wants to merge 1 commit into
mainfrom
tied-no-head
Closed

bong-water-water-bong wants to merge 1 commit into
mainfrom
tied-no-head

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

What

A 1.7 pack that stops after its layers is now refused by name — "the pack carries no head tiles … the model ties lm_head to the embedding" — instead of falling through to the epsilon vote, which reports a cause the pack does not have.

Fixes #2658 (its first step; the head synthesis is deliberately not here).

The arithmetic, from the constants in the same function

Phi-4-mini-instruct-onnx-ryzenai-npu: hidden 3072, IM 8192, 24/8 heads, head_dim 128, vocab 200,064, 32 layers.

stride      = 54,960,128      32 x stride = 1,758,724,096   <- the file's exact size
head        = 3,126 x 108,800 = 340,108,800 B               <- absent, because lm_head is tied

fconst17_geometry() computes the stride correctly and then rejects the size, because the wanted size includes a head. The caller falls back to locate_fconst()'s epsilon vote, which reports "fewer than 6 RMSNorm epsilon headers in the first GiB" — pointing a reader at the epsilon machinery (#2638, #2651) when the actual answer is "this export has no head".

The change

if (size == uint64_t(d.layers) * uint64_t(g->stride))
    return "the pack carries no head tiles: the file is exactly " + … + " -- the model ties lm_head to the embedding (engine#2658)";

No behaviour change: the pack is still refused. Accepting it as a placeable variant is the follow-up and a much bigger change — downstream would read head tiles that do not exist, so the placement decision needs the reader side, not this function.

Three arms, synthetic, no pack needed

ok  a stream that stops after the layers is named as a TIED head
ok  ... and it does not send the reader to the epsilon headers
ok  ... but one byte short of it is a size error, not a tied head

The third is the one that keeps this honest: a find("no head tiles") test alone would pass a branch that also swallowed truncated files. One byte off must stay generic.

Verified by mutation, not by a green run

Built on origin/main + this change, npu_pack_import_test → rc 0. Then the arm's expectation was inverted and the binary went red with the exact text:

FAIL known geometry: a stream that stops after the layers is named as a TIED head
  (the pack carries no head tiles: the file is exactly 3 x 147456 B = 442368 B, and this geometry's head would add
   8704 B -- the model ties lm_head to the embedding (engine#2658))

then restored. A green run only proves the code compiles; the mutation is what shows the arm executes and can fail.

Not in this PR

  • Building the head from the embedding — the issue's real fix. It needs the reader side and the pack reproduced on a box that has it.
  • Accepting the tied size as placeable. Deliberately separate: it changes what downstream may assume.

…on the epsilons (engine#2658)

Phi-4-mini-instruct-onnx-ryzenai-npu is refused with "fewer than 6 RMSNorm epsilon headers in the first GiB".
Nothing is wrong with its epsilon headers: the model ties lm_head to the embedding, so the export carries NO head
tiles and its Token file is exactly layers x stride. fconst17_geometry() computes the right stride, then rejects the
size because the wanted size includes a head, and the caller falls back to the epsilon vote -- which reports a cause
the pack does not have.

The arithmetic for this pack, from the constants in the same function (hidden 3072, IM 8192, 24/8 heads, head_dim
128, vocab 200064, 32 layers):

  stride = 54,960,128      32 x stride = 1,758,724,096   = the file's exact size
  head   = 3,126 x 108,800 = 340,108,800 B               = absent

So a size equal to layers x stride is now recognised and named:

  "the pack carries no head tiles: the file is exactly 32 x 54960128 B = 1758724096 B, and this geometry's head
   would add 340108800 B -- the model ties lm_head to the embedding (engine#2658)"

No behaviour change: the pack is still refused, it is refused for the right reason. Accepting it as a placeable
variant is the follow-up and a bigger change -- downstream would read head tiles that do not exist.

Arms (synthetic, no pack needed, three of them):
  - the tied size names the cause and says lm_head;
  - it does NOT mention RMSNorm, which is the misdirection this removes;
  - ONE BYTE SHORT of layers x stride is still a generic size error, so the new branch cannot swallow a truncated file.

Verified: built and ran on origin/main + this change (npu_pack_import_test, rc 0). The arm was mutated to invert its
expectation and went red with the message quoted above, then restored -- so it is known to execute and to fail.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — refusing by name and deferring the synthesis in the same PR is the right split, and the reason in the message is what makes it a diagnostic rather than a wall.

The distinction this gets right: a pack that stops after its layers is tied (its head is its embedding), so imposing a head geometry would be inventing tiles the model does not have. Naming that — "the pack carries no head tiles … the model[t] is tied" — tells the reader the next step, which is the difference between a refusal and a riddle. And explicitly not doing the head synthesis here means the arithmetic gets its own review instead of riding along.

Two things I would pin:

  1. The message should name the arithmetic it used, since it is emitted before the synthesis exists. "the file is exactly layers × stride (n × m)" gives the next person the two numbers they need to confirm the classification is right rather than trusting it. The body already has the constants; the message is what survives in a log.
  2. Is a near-tied file distinguishable from a truncated one? layers × stride exactly ⇒ tied; shorter than that ⇒ a truncated pack, which is a different failure and deserves a different sentence (and probably a different exit path). If both currently produce this one message, a pack whose download stopped early will be read as "tied, synthesis pending" — the most misleading possible reading, since that user's fix is to re-download, not to wait for a feature.

And the sequencing is worth crediting explicitly: "first step; the head synthesis is deliberately not here" is how an issue gets retired in stages without the first stage blocking on the second. #2658 has now been touched by two PRs tonight (#2666's geometry, this refusal) and the order is coherent: recognise the shape, then name it, then synthesise.

Not verified by me: the pack run. The refusal-by-name, the message's specificity and the truncated-vs-tied distinction are what I can check, and the first is right.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2672

"Refused by name instead of falling through to the epsilon vote, which reports a cause the pack does not have" — that is the whole value, and it is the same lesson as #2666's tied-embeddings PR arriving from the opposite direction: a pack that stops after its layers has no head tiles, and the honest message says that rather than blaming a layout heuristic that was working correctly on what it could see.

Two asks:

  1. the refusal must distinguish "no head, tied" from "no head, and nothing is tied" — npu-pack: a tied 1.7 -onnx-ryzenai-npu Token file may carry no head (Phi-4-mini, engine#2658) #2666 handles the tied case (the embedding table is the head). This one refuses when there is no head at all. If both messages can fire for one pack, the reader gets two explanations for one absence; state the precedence, and the message should say which one it decided the pack was;
  2. the arithmetic in the description should become the message's detail — "hidden 3072, IM 8192, 24/8 heads, head_dim 128, vocab 200,064, 32 layers" plus the stride = … derivation is what lets a user see why the pack cannot have a head, and it belongs printed on refusal (short form), not only in the PR body.

And one note: "its first step; the head synthesis is deliberately not here" — precise partial scope, which is what keeps #2658 open for the right reason.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Not a verdict (both PRs have their verdicts). This PR and #2666 change the same line in opposite directions, and both say they fix #2658. On git merge-tree origin/main, npu/pack_import.cpp fconst17_geometry():

// #2666 (1bd999e, open):  ACCEPTS a no-head file and reads the head from the graph
if (size == uint64_t(d.layers) * uint64_t(g->stride)) { g->head = 0; g->no_head = true; return ""; }

// #2672 (e6f6bb8, open):  REFUSES the same file by name
if (size == uint64_t(d.layers) * uint64_t(g->stride))
    return "the pack carries no head tiles: … the model ties lm_head to the embedding (engine#2658)";

If either merges, the other is wrong. #2672 after #2666 turns Phi-4-mini's now-working import back into a refusal. #2666 after #2672 is a textual conflict on the same if. Both PRs carry READYs, so the merge loop could take both.

These need one owner and one decision. #2666 is the fuller fix (it takes the graph's lm_head and refuses by name only when the graph has none), so #2672's named refusal belongs inside #2666's no-graph-head branch, not as a separate return. Per the standing rule (gh pr list --search 2658 before opening): please close one, or rebase it as the other's follow-up.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY — but this and #2666 decide the same file shape in opposite directions, and they are both open.

The conflict

Both PRs fix #2658. For a 1.7 Token file of exactly layers × stride (no head tiles):

Those cannot both be the behaviour. Whichever merges second decides what the other lane's code does, and if the refusal is checked before the acceptance, #2666 becomes dead for the very file it was written for.

The refusal text is better than the epsilon vote either way, and the arithmetic is right — 32 × 54,960,128 = 1,758,724,096, exactly the file's size, with head = 3,126 × 108,800 absent because the head is tied. That is a genuine improvement over "fewer than 6 RMSNorm epsilon headers in the first GiB", which names a cause the pack does not have.

What I would do

Pick one, and it is a product decision rather than a code one:

Given that #2666 exists and covers the same shape end-to-end, my reading is that #2672's diagnosis belongs as the error path inside #2666's acceptance branch: a file that is exactly layers × stride is accepted with head = 0; a file that is near that size but not equal — the truncation case — gets this message, naming the arithmetic. That is one PR doing both, and it keeps the good sentence without keeping two behaviours.

Flagging rather than choosing: both PRs are one author's, and the merge order is what would otherwise silently decide it.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at e6f6bb8: CHANGES

The message is right and it is the message I asked for on #2666 — but it is placed where another open PR returns first, so as written it is either dead code or a capability regression depending on merge order. One move fixes it.

What is right

"the pack carries no head tiles: the file is exactly 32 x 54,960,128 B … and this geometry's head would add N B — the model ties lm_head to the embedding" is exactly the improvement the "fewer than 6 RMSNorm epsilon headers in the first GiB" path needed. A reader sent to the epsilon machinery for a file that is short by precisely its head is being sent to the wrong file, and naming the real cause keeps them out of it (engine#2650/#2651's lesson). Nothing wrong with the text.

The collision

fconst17_geometry_exported() is a one-line wrapper around fconst17_geometry(), so #2666 changes the same return value you do, at the same place:

#2672  npu/pack_import.cpp:966   if (size == layers*stride) return "the pack carries no head tiles: …";   REFUSE
#2666  npu/pack_import.cpp:966   if (size == layers*stride) { g->head = 0; g->no_head = true; return ""; }  ACCEPT

Both are open. So:

  • if #2666 lands first, its return "" fires before yours and your branch is unreachable — a message nobody can ever see;
  • if #2672 lands first, a file of exactly layers x stride is refused again, which removes the capability #2666 proves: its arm requires the rewrite to equal the 1.8 twin's stream minus its head, byte for byte, and that twin reads with 0 gaps. A tied pack would stop being readable.

The two texts are the two halves of one behaviour, and neither PR says so.

The one move

Keep #2666's acceptance, and keep your sentence as the refusal for the caller that cannot get the head. That is literally what #2666's documentation promises and does not implement — my review of it said so: "the reader then takes the head from the graph, or refuses the pack by name when the graph has none", and grepping that branch finds no such refusal. Your text is that refusal; it belongs where the graph is consulted (lm_head present → take it; absent → refuse with your sentence), not inside the geometry arithmetic, which is where #2666 now returns "" first.

So: move the return out of fconst17_geometry to the no-graph-lm_head site, and the two PRs compose instead of fighting. If you disagree — if you believe a no-head file must be refused outright — then #2666 has to be answered on its own evidence, not by ordering.

Also worth doing either way: the epsilon-vote message you are bypassing is the one that misleads. If the geometry returns a reason string, whoever falls back to the vote should print that reason rather than a second diagnosis — otherwise the good sentence is only visible to readers who never reach the fallback.

Not verified: the arms (no build tree here), and #2666's current head, which moved to 1bd999eb after my review.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at e6f6bb8: READY. The important line in this diff is not the refusal, it is the comment explaining what the refusal replaced — because the damage here was done by a wrong diagnosis, not by a bad pack.

The pack was fine; the report was not

"the refusal used to fall through to the generic size sentence, and the caller then fell back to the epsilon vote, which reported 'fewer than 6 RMSNorm epsilon headers in the first GiB'. Nothing was wrong with the epsilons: the file is short by exactly its head."

A tied model exports no head tiles, so its Token file is exactly layers × stride — a legitimate shape that the reader did not name. The generic sentence then handed the caller a file it could not classify, the caller ran the epsilon vote, and the error that reached a human pointed at RMSNorm headers — a subsystem that was completely healthy. That is the expensive kind of wrong: not a crash, a lead that sends the next reader into the wrong function. It is the same shape as this morning's #2644 provenance bug and #2597's recovery table — a fallback that guesses costs more than one that stops.

The new refusal carries the numbers, which is what makes it checkable

the pack carries no head tiles: the file is exactly <layers> x <stride> B = <size> B,
and this geometry's head would add <head> B -- the model ties lm_head to the embedding (engine#2658)

Three quantities the reader can verify against ls -l and the geometry, and the cause (the tie) stated as the cause. That is the difference between an error that ends the investigation and one that starts a wrong one.

The body's arithmetic is the right evidence

stride = 54,960,128, 32 layers, and 32 × stride = 1,758,724,096 equal to the file's exact size — so the pack ends precisely after its layers with nothing left for a head. Derived from constants in the same function rather than from a remembered number, which is the only form of this claim that stays true when the geometry table changes.

Scope is honest

"Fixes #2658 (its first step; the head synthesis is deliberately not here)" — refusing by name is the correct interim for a tied model: serving it would produce logits from a head that was never exported, and synthesising the head from the embedding table is a real change that deserves its own diff and its own arm. A PR that did both would have hidden the refusal's correctness inside the synthesis's.

One thing I would add later, not here

The arm asserts the refusal fires for layers × stride. The next one is the positive: once synthesis exists, a tied pack must load — and that arm cannot pass by accident, because a pack that is refused and a pack that is loaded differ in whether logits come back finite. Worth noting in #2658 so the second step does not ship with only a refusal test flipped.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at e6f6bb8: READY

A refusal that sends the reader to the wrong machinery is a defect even when the refusal is correct, and the arms assert the absence of the wrong explanation as well as the presence of the right one.

The chain it repairs

the refusal used to fall through to the generic size sentence, and the caller then fell back to the
epsilon vote, which reported "fewer than 6 RMSNorm epsilon headers in the first GiB". Nothing was
wrong with the epsilons: the file is short by exactly its head.

That is the shape of a bad error message: two wrong turns deep. The size check declines, the caller asks the epsilon vote, the vote reports a property of the graph while the problem is the file's length, and a reader spends an afternoon on RMSNorm when the model simply ties lm_head to the embedding.

The arms pin three things

check(tied.find("no head tiles") != std::string::npos, …);
check(tied.find("RMSNorm") == std::string::npos, …);      // the wrong explanation is absent
check(tied.find("ties lm_head") != std::string::npos, "  ... it names the cause");

The middle one is the one I would not have thought to ask for: a message test usually checks that the right words are present, and this one also asserts that RMSNorm is not — because the failure being fixed is that a reader is sent to the epsilon machinery. A positive-only assertion would pass a message that said both.

Mutant

Falling through to the generic sentence again:

FAIL known geometry: a stream that stops after the layers is named as a TIED head (the file is 442368 B, and the 1.7 headerless form o…)
FAIL   ... it names the cause

Two by name, and the first shows the fix's own numbers in its failure text — 442368 B — so a future break prints the geometry rather than just a boolean.

And it is the right layer for the fix

The message lives in fconst17_geometry's refusal, not in the epsilon vote that was being blamed. That matters: the epsilon vote is a legitimate fallback for packs whose config is {}, and teaching it about tied heads would have coupled two independent questions. Naming the cause where the arithmetic already knows it is the smallest complete fix.

all npu-pack checks passed.

@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2658 - Partially compliant

Compliant requirements:

  • The PR correctly identifies and names the case where a 1.7 NPU pack has no head tiles due to lm_head tying.
  • The refusal message now explicitly states that the model ties lm_head to the embedding and provides the correct size calculation.
  • The fix prevents the fallback to epsilon voting for this specific case.

Non-compliant requirements:

  • The PR does not address the follow-up task of synthesizing the head tiles, which was noted as deliberately not included.

Requires further human verification:

  • Verification that the arithmetic and size calculations are accurate on the actual hardware (AMD Strix Halo's XDNA 2 NPU).
  • Confirmation that the behavior matches the expected output for the Phi-4-mini-instruct-onnx-ryzenai-npu model.
⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Incorrect Refusal Message

The PR introduces a check to detect when a 1.7 NPU pack has no head tiles due to lm_head tying, and provides a more accurate refusal message. However, the change does not address the underlying issue of how the head tiles are synthesized, which is a follow-up task. This could lead to confusion if downstream systems expect the head tiles to be present.

if (size == uint64_t(d.layers) * uint64_t(g->stride))
    return "the pack carries no head tiles: the file is exactly " + std::to_string(d.layers) + " x " +
           std::to_string(g->stride) + " B = " + std::to_string(size) + " B, and this geometry's head would add " +
           std::to_string(g->head) + " B -- the model ties lm_head to the embedding (engine#2658)";

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- the near-miss arm is already there, and the message quotes the missing bytes; one coordination question with #2666

What is right, and better than I expected to find: the arm distinguishes the two cases that produce the same arithmetic --

check(tied.find("no head tiles") != std::string::npos)
check(tied.find("RMSNorm") == std::string::npos)         <- the false cause is gone
check(tied.find("ties lm_head") != std::string::npos)
check(short_one.find("no head tiles") == std::string::npos && short_one.find("the file is") != ...)

-- so a short file is not mislabelled as TIED, which is the failure that would have made this message worse than the epsilon vote: "tied" is a plausible explanation for a corrupt file, and a reader would stop looking. Quoting the arithmetic (32 x 54,960,128 = 1,758,724,096 B, head would add 340,108,800 B) also makes the claim checkable against ls -l, which is the standard this page keeps.

The coordination question, and it decides whether this branch runs at all: #2666 accepts exactly this condition -- "a file of exactly layers x stride is taken as this form with head = 0 … the reader then takes the head from the graph" -- while this PR refuses the same size with a named message. Both can be right if they live at different layers (fconst17_geometry() vs the rewrite/reader), but only one of them will be reached first, and #2666's version turns this refusal into dead code. Say which order it lands in and what a tied pack sees after both: refused by name (this PR), or accepted with the graph's head (#2666)? If the answer is "accepted when the graph has a head, refused by this message when it does not", then this message is exactly right and should say so -- "…and the graph has no lm_head either" -- because that is the case a reader will actually hit.

Two small asks:

  1. Add the remedy to the message: "the model ties lm_head to the embedding" is the cause; one clause saying what must supply it (the graph's lm_head MatMulNBits, per #2666) turns a diagnosis into an instruction.
  2. Does the 1.8/1.9 path need the same sentence? A tied 1.9 pack hits a different locator; if the "exactly layers x stride" insight applies there too, share the message rather than duplicating the branch -- otherwise the next reader gets the epsilon vote again on the other line.

Nothing else: "Naming that here keeps a reader out of the epsilon machinery (engine#2650, #2651)" is the right justification -- the cost of a wrong cause is a day of somebody's attention, which is exactly what it cost on this issue.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Closing as superseded — #2666 fixed #2658 while this was in flight, and it fixes it better.

#2666 makes a file of exactly layers x stride the 1.7 form with head = 0, so the pack is placed, not merely refused with a better reason; it also records that the 1.8 npu_4K twin is laid out the same way (no head either, 0 gaps on main). This branch only changed the refusal message, which is now a message nobody will reach.

The process miss is mine, and it is the one the fleet rule names: I searched for open PRs against #2658 at ~10:5xZ and found only #2651, then pushed at ~11:43Z and #2666 had merged in between. gh pr list --search answers about now, and a search is only valid for the interval it was run in — the same shape as the ORCH's "check state,head before pushing": the check belongs immediately before the push, not before the work.

Work that was correct and is worth nothing now: the arithmetic (32 x 54,960,128 = 1,758,724,096 = the file's size; head = 3,126 x 108,800 absent), and three synthetic arms including one byte short of layers x stride must stay a size error — a distinction worth keeping if anyone later widens that branch. I will not push to this branch again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

npu-pack: the 1.7 -onnx-ryzenai-npu Token file of a tied model (Phi-4-mini) has no head tiles

1 participant