You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
npu-pack: a 1.7 -onnx-ryzenai-npu canonical copy is located by the epsilon vote, and a 4-bit tile coincidence breaks Phi-3-mini-4k (320 gaps) #2650
After #2638, three Phi packs of the 1.7 -onnx-ryzenai-npu line with the same geometry (hidden 3072, 32 layers, 32/32 heads, IM 8192) split:
pack
npu-pack info
Phi-3.5-mini-instruct-onnx-ryzenai-npu
0 gaps
Phi-3-mini-128k-instruct-onnx-ryzenai-npu
0 gaps
Phi-3-mini-4k-instruct-onnx-ryzenai-npu
320 gaps: attention payload is 16226318 B, not a whole number of 108800-byte tiles (224) and the MLP section ends at 57260238 B in a 61923520-byte layer block (96)
Phi-4-mini-instruct-onnx-ryzenai-npu also shows 320 gaps.
In all of them, the 1.7 rewrite ran (the note is printed) and produced a canonical copy of the right size. The failure comes after it. locate_fconst()'s epsilon vote takes block 0's first two anchors as the header pair. Its anchor is 12 bytes (epsilon + 10 zeros), and that pattern occurs inside 4-bit tiles. Here one occurs 16,232,526 B into block 0, an offset that isn't even 64-aligned, so "the second header" is a coincidence inside the weights.
The rewrite wrote every header itself, so for its copy the geometry is known exactly and the vote should not be asked. Found in the TAMDCOL 1.7 sweep (cl-npuperf).
fconst17_geometry() is arithmetic on the file size and Dims (layers × stride + head), so for the copy this function is
about to write, g is exact by construction — the issue's point in one sentence. But g is a local, and the caller keeps
only the path:
const std::string f17 = canonical_token_fconst17(found, pack_dir, dims, work);
if (f17 != found) return f17; // geometry forgotten here
So when the copy is read back, the reader has nothing but the bytes, and asks the epsilon vote — locate_fconst_with_header() → locate_fconst(tok, dims.eps_bits) — which is exactly the witness the issue shows losing: a 12-byte anchor (86 35 + ten
zeros) occurring inside a 4-bit tile 16,232,526 B into block 0, at an offset that is not even 64-aligned.
Smallest patch I can see
Carry g from where it is computed to where it is needed, without touching the reader's API:
a process-local map keyed by the copy path (static std::map<std::string, Fconst17Geom>), populated in canonical_token_fconst17() when it actually rewrites;
locate_fconst_with_header() consults it first: if the (absolute) tok path is in the map, take stride (and the header
offset) from g and skip the vote entirely — the same shape as the existing 1.9-header branch, which already prefers a
known geometry over the vote and keeps the vote as fallback;
if both witnesses are available (a pack that has both), compare and refuse on disagreement rather than pick — the pattern
1.9 already uses for exactly this.
Why a cache rather than threading a parameter through find_token_fconst → reader: the geometry is meaningful only for a copy
this process just wrote, and the map keys on that copy's path, so a stale entry cannot outlive the process. If the owner prefers
explicit plumbing, the honest alternative is an out-parameter up that call chain — more diff, same information.
The test that has to come with it
The Phi packs here are too large to ship as a fixture, so the arm should be synthetic: build a file whose size fits fconst17_geometry for a small Dims, embed a 4-bit-tile pattern containing the 12-byte anchor at a non-64-aligned offset
(16,232,526 B-shaped, i.e. deliberately not aligned), and assert that the reader uses the rewrite's stride and does not
report the "second header" coincidence. Without that arm the fix is untested against the very pattern that motivates it.
I will implement it if nobody objects; posting the path first because the claim is that the vote "should not be asked", and that
is a statement about where the geometry comes from, not about the vote being wrong in general — it works today on the 1.8 /
R1-Distill line and must keep working there.
Widened in #2651: CodeLlama-7b-Instruct-hf-onnx-ryzenai-npu is refused for a second reason (the 1.7 export zero-pads the head, 515 tiles past 501). Both fixes are in that PR; device rows are queued.
Activity
bong-water-water-bong commentedon Nov 16, 2026
Claimed (pi-pkg). The geometry this pack needs is computed and then dropped on the floor — here is the path.
canonical_token_fconst17()computes the exact geometry before it writes anything:fconst17_geometry()is arithmetic on the file size andDims(layers × stride + head), so for the copy this function isabout to write,
gis exact by construction — the issue's point in one sentence. Butgis a local, and the caller keepsonly the path:
So when the copy is read back, the reader has nothing but the bytes, and asks the epsilon vote —
locate_fconst_with_header()→locate_fconst(tok, dims.eps_bits)— which is exactly the witness the issue shows losing: a 12-byte anchor (86 35+ tenzeros) occurring inside a 4-bit tile 16,232,526 B into block 0, at an offset that is not even 64-aligned.
Smallest patch I can see
Carry
gfrom where it is computed to where it is needed, without touching the reader's API:static std::map<std::string, Fconst17Geom>), populated incanonical_token_fconst17()when it actually rewrites;locate_fconst_with_header()consults it first: if the (absolute)tokpath is in the map, takestride(and the headeroffset) from
gand skip the vote entirely — the same shape as the existing 1.9-header branch, which already prefers aknown geometry over the vote and keeps the vote as fallback;
1.9 already uses for exactly this.
Why a cache rather than threading a parameter through
find_token_fconst→ reader: the geometry is meaningful only for a copythis process just wrote, and the map keys on that copy's path, so a stale entry cannot outlive the process. If the owner prefers
explicit plumbing, the honest alternative is an out-parameter up that call chain — more diff, same information.
The test that has to come with it
The Phi packs here are too large to ship as a fixture, so the arm should be synthetic: build a file whose size fits
fconst17_geometryfor a smallDims, embed a 4-bit-tile pattern containing the 12-byte anchor at a non-64-aligned offset(
16,232,526 B-shaped, i.e. deliberately not aligned), and assert that the reader uses the rewrite's stride and does notreport the "second header" coincidence. Without that arm the fix is untested against the very pattern that motivates it.
I will implement it if nobody objects; posting the path first because the claim is that the vote "should not be asked", and that
is a statement about where the geometry comes from, not about the vote being wrong in general — it works today on the 1.8 /
R1-Distill line and must keep working there.
bong-water-water-bong commentedon Nov 16, 2026
Widened in #2651: CodeLlama-7b-Instruct-hf-onnx-ryzenai-npu is refused for a second reason (the 1.7 export zero-pads the head, 515 tiles past 501). Both fixes are in that PR; device rows are queued.
npu-pack: the 1.7 rewrite's own stride beats the epsilon vote (Refs #…