Skip to content

npu-pack: the 1.7 -onnx-ryzenai-npu Token file of a tied model (Phi-4-mini) has no head tiles #2658

Closed

Description

@bong-water-water-bong

Phi-4-mini-instruct-onnx-ryzenai-npu (Ryzen AI 1.7) is refused with 320 gaps, "fewer than 6 RMSNorm epsilon headers in the first GiB". That happens even with #2651, which fixes the other 1.7 npu refusals.

Measured from its own size: the Token file dd_metastate_Llm_Token_MatMulNBits_2_0.fconst is 1,758,724,096 B, which is exactly 32 x 54,960,128 B layer strides of the 1.7 headerless form (#2638: hidden 3072, IM 8192, 24/8 heads, head_dim 128). Nothing follows the layers. The model ties lm_head to the embedding (vocab 200,064), so the export leaves out the 3,126 head tiles that every other 1.7 npu pack carries.

The 1.7 rewrite needs the head in the tile format. So the fix has to build it from the pack's embedding (or reuse it in whatever form the forward takes for a tied head), with a byte-exactness check against an independent decode. The 1.8 npu_4K twin's Token tail should show the target layout. Found in the TAMDCOL 1.7 sweep (cl-npuperf).

Activity

  1. bong-water-water-bong commented on Nov 16, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    I verified the arithmetic in this issue exactly, from #2651's own constants, and it names why the refusal happens — and why it is misleading.

    The numbers match to the byte

    Re-deriving the 1.7 headerless stride from fconst17_geometry()'s arithmetic (the one #2651 introduces), with this model's dims — hidden 3072, IM 8192, 24 heads / 8 kv, head_dim 128, vocab 200,064, 32 layers:

    tile_h = tile_o = 108,800      n_q = 48   n_kv = 16   n_o = 48
    stride      = 54,960,128        <- this issue says 54,960,128        MATCH
    32 x stride = 1,758,724,096     <- this issue says 1,758,724,096     MATCH
    head        = 3,126 x 108,800 = 340,108,800 B
    

    So the file is exactly layers × stride, with nothing after — and the 340,108,800 B that a sibling pack would carry is precisely the 3,126 head tiles this one omits. Independent confirmation of the issue's central claim, from constants rather than from the file.

    That tells us where the refusal actually comes from

    fconst17_geometry() computes

    want = layers x stride + head
    

    and refuses when size != want (modulo the whole-tile padding it tolerates after the head). For a tied pack, size == layers × stride — so the geometry path rejects a stream whose stride it has computed correctly, and the caller falls through to the epsilon-header vote, which is what emits "fewer than 6 RMSNorm epsilon headers in the first GiB".

    The message names the wrong cause. There is nothing wrong with the epsilon headers; the arithmetic declined because a head is missing, and the fallback then blamed an anchor it could not find. A reader with this pack is sent to the epsilon machinery — which is what #2638/#2651 are about — when the actual reason is "this export has no head".

    The change I would make, in the order I would make it

    1. fconst17_geometry() should accept size == layers × stride as a named variant — head absent because the model ties lm_head to the embedding — and set g->head = 0 plus a flag saying so. That places this pack without inventing bytes, and it is the same predicate shape the function already uses: it knows the stride, so it can tell "short by exactly a head" from "the wrong geometry entirely".
    2. The refusal should then name it: "the pack carries no head tiles (size == layers x stride; the model ties lm_head to the embedding)", not the epsilon message.
    3. Building the head from the embedding (the issue's own fix) is the real work and follows from (1): the geometry is placeable, one thing is missing, and the forward's tied-head path already exists at the container level (engine#1503's selfcheck_tied). Whether the 1.7 rewrite should synthesise 3,126 tiles of 108,800 B or the reader should reuse the embedding in place is a decision for whoever owns that path — but it can only be made once the stream is located, which is what (1) buys.

    Verified without the pack

    The pack is not on this box (~/models has only the amd-rai/sweep logs), so I cannot reproduce the refusal end to end. Everything above is arithmetic against the constants in #2651 — which is why I am posting it as a finding and not as a patch: step 1 is testable with a synthetic stream (a layers × stride file with no head), the same way test_fconst17_known_geometry builds one, but I would want the refusal reproduced before writing the head-synthesis half.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions