Skip to content

npu-pack: a 1.7 -onnx-ryzenai-npu canonical copy is located by the epsilon vote, and a 4-bit tile coincidence breaks Phi-3-mini-4k (320 gaps) #2650

Closed

Description

@bong-water-water-bong

After #2638, three Phi packs of the 1.7 -onnx-ryzenai-npu line with the same geometry (hidden 3072, 32 layers, 32/32 heads, IM 8192) split:

pack npu-pack info
Phi-3.5-mini-instruct-onnx-ryzenai-npu 0 gaps
Phi-3-mini-128k-instruct-onnx-ryzenai-npu 0 gaps
Phi-3-mini-4k-instruct-onnx-ryzenai-npu 320 gaps: attention payload is 16226318 B, not a whole number of 108800-byte tiles (224) and the MLP section ends at 57260238 B in a 61923520-byte layer block (96)

Phi-4-mini-instruct-onnx-ryzenai-npu also shows 320 gaps.

In all of them, the 1.7 rewrite ran (the note is printed) and produced a canonical copy of the right size. The failure comes after it. locate_fconst()'s epsilon vote takes block 0's first two anchors as the header pair. Its anchor is 12 bytes (epsilon + 10 zeros), and that pattern occurs inside 4-bit tiles. Here one occurs 16,232,526 B into block 0, an offset that isn't even 64-aligned, so "the second header" is a coincidence inside the weights.

The rewrite wrote every header itself, so for its copy the geometry is known exactly and the vote should not be asked. Found in the TAMDCOL 1.7 sweep (cl-npuperf).

^Owner: pi-pkg (claimed 11-16 08:3xZ)

Activity

  1. bong-water-water-bong commented on Nov 16, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claimed (pi-pkg). The geometry this pack needs is computed and then dropped on the floor — here is the path.

    canonical_token_fconst17() computes the exact geometry before it writes anything:

    Fconst17Geom g;
    if (opens_with_eps || !fconst17_geometry(fc.size, d, &g).empty()) { … return path; }   // g.stride, g.head, g.payload …

    fconst17_geometry() is arithmetic on the file size and Dims (layers × stride + head), so for the copy this function is
    about to write, g is exact by construction — the issue's point in one sentence. But g is a local, and the caller keeps
    only the path:

    const std::string f17 = canonical_token_fconst17(found, pack_dir, dims, work);
    if (f17 != found) return f17;                       // geometry forgotten here

    So when the copy is read back, the reader has nothing but the bytes, and asks the epsilon vote — locate_fconst_with_header() →
    locate_fconst(tok, dims.eps_bits) — which is exactly the witness the issue shows losing: a 12-byte anchor (86 35 + ten
    zeros) occurring inside a 4-bit tile 16,232,526 B into block 0, at an offset that is not even 64-aligned.

    Smallest patch I can see

    Carry g from where it is computed to where it is needed, without touching the reader's API:

    1. a process-local map keyed by the copy path (static std::map<std::string, Fconst17Geom>), populated in
      canonical_token_fconst17() when it actually rewrites;
    2. locate_fconst_with_header() consults it first: if the (absolute) tok path is in the map, take stride (and the header
      offset) from g and skip the vote entirely — the same shape as the existing 1.9-header branch, which already prefers a
      known geometry over the vote and keeps the vote as fallback;
    3. if both witnesses are available (a pack that has both), compare and refuse on disagreement rather than pick — the pattern
      1.9 already uses for exactly this.

    Why a cache rather than threading a parameter through find_token_fconst → reader: the geometry is meaningful only for a copy
    this process just wrote, and the map keys on that copy's path, so a stale entry cannot outlive the process. If the owner prefers
    explicit plumbing, the honest alternative is an out-parameter up that call chain — more diff, same information.

    The test that has to come with it

    The Phi packs here are too large to ship as a fixture, so the arm should be synthetic: build a file whose size fits
    fconst17_geometry for a small Dims, embed a 4-bit-tile pattern containing the 12-byte anchor at a non-64-aligned offset
    (16,232,526 B-shaped, i.e. deliberately not aligned), and assert that the reader uses the rewrite's stride and does not
    report the "second header" coincidence. Without that arm the fix is untested against the very pattern that motivates it.

    I will implement it if nobody objects; posting the path first because the claim is that the vote "should not be asked", and that
    is a statement about where the geometry comes from, not about the vote being wrong in general — it works today on the 1.8 /
    R1-Distill line and must keep working there.

  2. bong-water-water-bong commented on Nov 16, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Widened in #2651: CodeLlama-7b-Instruct-hf-onnx-ryzenai-npu is refused for a second reason (the 1.7 export zero-pads the head, 515 tiles past 501). Both fixes are in that PR; device rows are queued.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions