Skip to content

npu-pack: read the 1.7 -onnx-ryzenai-npu Token .fconst; eps and llama3 RoPE from the pack itself (Fixes #2634) - #2638

Merged
bong-water-water-bong merged 2 commits into
mainfrom
cl-npuperf/fconst17-headerless
Nov 16, 2026
Merged

bong-water-water-bong merged 2 commits into
mainfrom
cl-npuperf/fconst17-headerless

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Fixes #2634.

amd/Llama-3.2-1B-Instruct-onnx-ryzenai-npu (the Ryzen AI 1.7 -onnx-ryzenai-npu line, about 33 packs) was refused on 160 tensors ("layer blocks not located: fewer than 6 RMSNorm epsilon headers"). It now imports, and it answers correctly on the NPU.

The 1.7 Token layout, measured against the same model's 1.8 npu_4K twin

1.7 -onnx-ryzenai-npu 1.8
q, k, v, o tiles, q first header + input gain, then k, q, v, o
post_attention gain bf16[H], no header header + post gain
MLP gate/up then down, no 64-B separator the MLP with its separator
the next layer's input gain (after the last layer, the final norm) --
tail: the head's tiles header + final norm, then the head's tiles

Layer 0's input_layernorm is not in the Token file. It is the graph's fp16 model.layers.0.input_layernorm.weight, in fusion.onnx's If branches. Every gain the file does carry equals bf16 of the graph's fp16 exactly.

Rebuild check. Rebuilding the 1.8 file from the 1.7 bytes with this map gives the same size (686,518,336 B). Every attention tile, gate/up unit and down group is at least 96.4% byte-equal to the twin at its mapped place, against 8.5% for a one-unit misplacement. The remaining bytes are re-quantization: 4-bit codes only. The full map is in the issue.

Change

  • canonical_token_fconst17(), called from materialise_token_fconst(), detects the form by its arithmetic: the file must be exactly layers x stride + head, and must not open with an epsilon header. It then rewrites the file into the 1.8 form, so every check of the 1.8 reader runs unchanged (tile round-trip, anchors, the head fit). 1.8 and 1.9 files come back untouched.
  • The model's numbers, from the pack itself. config.json is {} on this line.
    • rms_norm_eps comes from the graph's RMSNorm nodes. walk_graph now reads the float epsilon, and the value is taken only when config.json does not state one. 18 nodes state 1e-5; the default was 1e-6.
    • llama3 RoPE comes from the cos_cache over its whole length (rope_cache_llama3, with the factor and the original window searched). The existing theta fit (500000) looks only at the first 4,096 positions, where llama3 scaling is invisible.
    • This pack's cache is factor 32, original_max_position_embeddings 131072: max error 0.009, against 1.995 for plain RoPE. The upstream config's 8192 fits no better than plain, so the pack's own cache wins.
  • Tests:
    • The rewrite on a synthetic geometry reproduces the 1.8 stream byte for byte. A 1.8 file and a wrong-size file are left alone.
    • The llama3 fit recovers 32/131072 and 8/8192, and does not call a plain cache llama3.
    • The graph epsilon is read back.
    • Mutants (each fails): keeping q first in the rewrite; an 8192-only llama3 search.
  • docs/npu.md: the layout table, the rebuild check, and where eps and RoPE come from.

Proof

  • npu-pack info: 0 gaps (was 160).
  • Import:
    • head fit: scale 0.989, rms ratio 1.006, rel 1.02x its 4-bit floor; the row-shifted control scores 0.066;
    • round trips: attention 1280/1280 tiles, MLP 2560/2560 groups;
    • norm anchors: 32/32.
  • Container config: rope_theta 500000, rope_scaling llama3 (factor 32, original 131072), rms_norm_eps 1e-5.
  • NPU (npu-hold, dmesg 0): "The capital of France is Paris.", "12 * 7 = 84", and "Neil Armstrong ... July 20, 1969, during the Apollo 11 mission". All three stop, at 3.9-5.0 tok/s.
  • ctest (non-device): everything passes except this tree's submodule_pin_check (my local llama.cpp checkout) and npu_rt_shim (its target does not link here). npu_w4g32_gate skips.
  • tools/copyright.py --check: rc 0.

The other 1.7 npu families (Qwen, Phi, Mistral) go through the same arithmetic gate, so a family whose sizes differ refuses rather than being misread. Their rows come in the 1.7 sweep.

🤖 Generated with Claude Code

…3 RoPE from the pack itself (Fixes #2634)

Llama-3.2-1B-Instruct-onnx-ryzenai-npu was refused on 160 tensors ("layer
blocks not located: fewer than 6 RMSNorm epsilon headers"). Measured against
its 1.8 npu_4K twin, the 1.7 Token file is the same tiles with no norm
envelopes: each layer block is [q,k,v,o -- q first][post_attention gain, no
header][MLP gate/up + down, no 64-B separator][the NEXT layer's input gain],
the tail is the head's tiles, and layer 0's input_layernorm is only in the
graph (fusion.onnx's If branches, fp16).

- materialise_token_fconst(): canonical_token_fconst17() detects the form by
  its arithmetic (exactly layers x stride + head, and no epsilon header at
  offset 0) and rewrites it into the 1.8 form, so every 1.8 reader check runs
  unchanged. 1.8 / 1.9 files come back untouched.
- config.json is {} on this line, so the model's numbers come from the pack:
  rms_norm_eps from the graph's RMSNorm nodes (walk_graph reads the float
  `epsilon`; taken only when config.json does not state it), and llama3 RoPE
  scaling from the cos_cache over its whole length (rope_cache_llama3: factor
  and original window searched; this pack is factor 32 / original 131072,
  max error 0.009 vs 1.995 for plain RoPE).
- tests: the rewrite on a synthetic geometry is the 1.8 stream byte for byte,
  with a 1.8 file and a wrong-size file left alone; the llama3 fit recovers
  32/131072 and 8/8192 and does not call a plain cache llama3; the graph
  epsilon is read. Mutants: q-first kept in the rewrite, and an 8192-only
  llama3 search, each fail.
- docs/npu.md: the 1.7 layout table, the rebuild check, eps and RoPE.

Measured:
- npu-pack info: 0 gaps (was 160). Import: head fit scale 0.989 / rms 1.006 /
  rel 1.02x floor, row-shifted control 0.066; attention 1280/1280 and MLP
  2560/2560 round-trip; 32/32 anchors.
- Rebuild of the 1.8 file from the 1.7 bytes: same size (686,518,336 B);
  every tile / gate-up unit / down group >= 96.4 % byte-equal at its mapped
  place (a one-unit misplacement: 8.5 %); every 1.7 norm == bf16(graph fp16).
- NPU (npu-hold, dmesg 0): "The capital of France is Paris.", "12 * 7 = 84",
  "Neil Armstrong ... July 20, 1969" -- all stop, 3.9-5.0 tok/s.
- ctest (non-device): all pass but submodule_pin_check (my local llama.cpp
  checkout) and npu_rt_shim (does not link in this tree); npu_w4g32_gate skips.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

ready for review at a783412

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit 8e0adec)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2634 - Partially compliant

Compliant requirements:

  • The PR fixes the issue with 160 gaps reported by npu-pack info on 1.7 Token files
  • It correctly detects the 1.7 Token file format and rewrites it into the canonical 1.8 form
  • The epsilon values are now read from the graph's RMSNorm nodes when config.json is empty
  • RoPE scaling is handled by determining llama3 scaling parameters from the pack's cos_cache
  • Documentation has been updated to reflect the new handling of 1.7 Token files

Non-compliant requirements:

  • None

Requires further human verification:

  • None
⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Potential Resource Leak in canonical_token_fconst17

In the canonical_token_fconst17() function, if an error occurs during the rewrite process, the function unmaps the input file but doesn't ensure that the output file is properly closed or cleaned up. This could lead to resource leaks or incomplete writes if an exception is thrown during the write process. The function opens an ofstream but doesn't guarantee that it's properly closed or that the file is cleaned up if an exception occurs.

std::ofstream o(work, std::ios::binary | std::ios::trunc);
if (!o) {
    unmap_file(fc.map, fc.size);
    throw std::runtime_error("cannot write " + work);
}
char hdr[64] = {0};
hdr[0] = char(d.eps_bits & 0xFF), hdr[1] = char(d.eps_bits >> 8);
char sep[64] = {0};
sep[0] = 1;   // 1.8's gate/up | down separator: 01 then 63 zeros (measured)
const auto put = [&](const uint8_t* p, int64_t n) { o.write(reinterpret_cast<const char*>(p), std::streamsize(n)); };
for (int L = 0; L < d.layers; ++L) {
    const uint8_t* b = fc.map + size_t(L) * size_t(g.stride);
    o.write(hdr, 64);
    if (L == 0) put(in0.data(), 2 * H);
    else put(b - 2 * H, 2 * H);                                              // the previous block's trailing gain
    put(b + g.n_q * g.tile_h, g.n_kv * g.tile_h);                            // k
    put(b, g.n_q * g.tile_h);                                                // q
    put(b + (g.n_q + g.n_kv) * g.tile_h, g.n_kv * g.tile_h);                 // v
    put(b + (g.n_q + 2 * g.n_kv) * g.tile_h, g.n_o * g.tile_o);              // o
    o.write(hdr, 64);
    put(b + g.payload, 2 * H);                                               // post_attention_layernorm
    put(b + g.payload + 2 * H, g.gu);
    o.write(sep, 64);
    put(b + g.payload + 2 * H + g.gu, g.down);
}
o.write(hdr, 64);
put(fc.map + size_t(d.layers) * size_t(g.stride) - size_t(2 * H), 2 * H);   // the final norm
put(fc.map + size_t(d.layers) * size_t(g.stride), g.head);
o.close();
unmap_file(fc.map, fc.size);
if (!o) throw std::runtime_error("writing " + work + " failed");
std::fprintf(stderr, "1bit npu-pack: note: a Ryzen AI 1.7 headerless Token .fconst (engine#2634) was rewritten into its "
                     "1.8 form at %s (remove it when done)\n", work.c_str());
return work;
Incorrect epsilon handling when nodes disagree

The code reads epsilon values from the graph's RMSNorm nodes when config.json is empty, but there's a potential issue if multiple nodes disagree on the epsilon value. The code sets norm_eps_conflict = true when this happens, but it doesn't handle the conflict case properly in the plan() function. If the epsilon values from different nodes are inconsistent, the system might use an incorrect epsilon value for block location, potentially causing incorrect tensor placement or misreading of the token file.

    else if (m.norm_eps != e) m.norm_eps_conflict = true;
    ++m.norm_eps_nodes;
}
✅ Resolved findings

npu/pack_import.cpp:1001-1034

Memory Management in Rewrite Function

In canonical_token_fconst17(), the function maps the input file and then proceeds to rewrite it. If an error occurs during the rewrite process, the function unmaps the input file but doesn't ensure that the output file is properly closed or cleaned up. This could lead to resource leaks or incomplete writes if an exception is thrown during the write process.

npu/pack_import.cpp:242-244

Potential Incorrect Epsilon Handling

The code reads epsilon values from the graph's RMSNorm nodes when config.json is empty, but there's a potential issue if multiple nodes disagree on the epsilon value. The code sets norm_eps_conflict = true when this happens, but it doesn't handle the conflict case properly in the plan() function. If the epsilon values from different nodes are inconsistent, the system might use an incorrect epsilon value for block location, potentially causing incorrect tensor placement or misreading of the token file.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

A second family through the same rewrite, at a783412: amd/Qwen-2.5_1.5B_Instruct-onnx-ryzenai-npu (Qwen2, attention biases, 28 layers).

  • npu-pack info: 0 gaps. The arithmetic gate fits: stride 26,198,720 B.
  • Import: head fit scale 0.990 / rms 1.005 / rel 1.02x its floor; attention 1568/1568 tiles + 84 bias tensors, MLP 4592/4592 groups, all round-trip byte for byte; 56/56 anchors.
  • Config: rope_theta 1e6 from the cos_cache (correlation 1.0000), no llama3 scaling detected (correct for Qwen2.5), rms_norm_eps 1e-6 (the graph agrees with the default).
  • NPU (npu-hold, dmesg 0): "The capital of France is Paris.", "12 times 7 is 84.", "... Neil Armstrong. He landed on the Moon on July 20, 1969, as part of the Apollo 11 mission." All stop, 3.5-3.8 tok/s.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at a783412: READY

The two mechanisms are the right shape and their arms assert the tight case; I read them rather than ran them, and the reason is at the end.

1. The 1.7 headerless Token geometry is a closed-form byte count with an exact comparison.

g->stride = g->payload + 2 * H + g->gu + g->down + 2 * H;
g->head   = ((V + 63) / 64) * g->tile_h;
const uint64_t want = layers * stride + head;
if (size != want) return "the file is <size> B, and the 1.7 headerless form of this geometry is L x S + a H B head = W B";

That is the strongest form this check can take: the wrong form does not merely fail, it prints both numbers and the shape it assumed, so a reader sees which term is off. And the arms test it from both sides — the synthetic stream fits, and 64 B more does not — which pins the comparison as equality rather than a range. 64 B is the right perturbation: every width in the arithmetic is measured in 64-row tiles, so a wrong width cannot land inside 64 B, and an implementer who wrote size <= want or a tolerance would pass the positive arm and fail the negative one.

2. The RoPE llama3 scan prefers measurement over declaration, with the numbers stated.

this pack's cache is factor 32 with original_max_position_embeddings 131072, max error 0.009, against 1.995 plain
the upstream config states 8192, and that fits no better than plain, so the pack's own cache wins

That is a 220x separation between the chosen variant and plain, and the config's own value loses because it explains the cache no better — the same precedence the rope_theta scan in this file already uses (the pack is the artifact, the config is a claim about it). The arms cover factor 32/131072 recovery, factor 8/8192 recovery, and a plain cache that must not be called llama3 — that third one is the one a too-eager detector would fail.

3. No second copy of the arithmetic: group_bytes and attn_tile_bytes are declared once in pack_import.h and defined once in pack_import.cpp; the new code calls them rather than re-deriving the tile maths.

What I did not do, and why: I did not build or run npu_pack_import_test. The build tree for this target is ~1-2 GB and this box is at 156 G, below the 160 G floor I start builds under — so my verdict is a reading of the arithmetic, the arms and the evidence they assert, not an execution of them. When the disk is back above the floor, the 137 new lines run in under a minute and that is the check I would want before merge.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at a783412: CHANGES

The llama3-RoPE half is well done and armed. The eps half has one case where the reader must not guess, and it is the only case that says nothing at all.

The conflict path is silent

bool norm_eps_conflict = false;
…
else if (m.norm_eps != e) m.norm_eps_conflict = true;      // two RMSNorm nodes disagree

if (!p.dims.eps_stated && onnx.norm_eps_nodes > 0 && !onnx.norm_eps_conflict && onnx.norm_eps > 0.0f) {
    if (p.dims.rms_norm_eps != onnx.norm_eps)
        std::fprintf(stderr, "note: rms_norm_eps %.6g taken from the graph (…)\n");
    p.dims.rms_norm_eps = onnx.norm_eps;
}

When two nodes disagree the graph eps is quietly dropped and the pack keeps whatever the config gave it. There is no else — the one situation where the reader must refuse to guess is the one that prints nothing, in a tool whose success path prints a note: precisely so that a reader knows where the value came from.

The consequence is the shape this repo has spent a week removing: a silent fallback. A {}-config pack (the 1.7 line this PR is for) whose graph states two different epsilons would serve at the default eps, and nothing in the output would say the graph was ambiguous rather than absent.

It is also unarmed — measured

Mutant: drop !onnx.norm_eps_conflict from the condition above (i.e. take the first node's eps despite the conflict).

… fconst eps: qwen 1e-6 … / fconst eps: llama 1e-5 …
final norm: crafted tail envelope imported from tail+64, missing-epsilon control refused
all npu-pack checks passed

Everything still passes. The existing eps arm (1.7 fconst: the graph's RMSNorm epsilon is read, n_eps == 2 && !conflict) only covers two nodes that agree, so it cannot see this branch either way.

The smallest fix

} else if (!p.dims.eps_stated && onnx.norm_eps_nodes > 0 && onnx.norm_eps_conflict) {
    std::fprintf(stderr, "note: rms_norm_eps: the graph's %d RMSNorm nodes DISAGREE, keeping %.6g "
                         "(config.json does not state one)\n", onnx.norm_eps_nodes, double(p.dims.rms_norm_eps));
}

plus one arm: craft an ONNX with two SimplifiedLayerNormalization nodes stating different epsilons, and check that graph_norm_eps_exported sets conflict == true and that the message appears. The predicate already exists and is already exported; only the caller's voice is missing.

What is good, and I checked it

llama3 RoPE — mutant RopeLlama3Fit rope_cache_llama3(…) { return RopeLlama3Fit{}; } fails three arms by name:

FAIL rope llama3: factor 32 / original 131072 is recovered (err 1e+09)
FAIL rope llama3: factor 8 / original 8192 is recovered
FAIL rope llama3: a plain cache is not called llama3 (plain err 1e+09)

including the negative arm (a plain cache is not called llama3), which is the one that keeps the detector from claiming scaling everywhere. And the reason the old theta fit missed it is stated exactly: "the theta fit looks at the first 4,096 positions, where llama3 scaling — which only slows the low-frequency columns — is invisible".

The 1.7 .fconst → 1.8 rewrite also runs here (a Ryzen AI 1.7 headerless Token .fconst was rewritten into its 1.8 form), and the geometry checks are two-sided (fits / 64 B more does not).

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- rebuilding the 1.8 file from the 1.7 bytes and comparing to the twin is the strongest verification design I have read tonight

It is a test with a control built into it: 96.4 % byte-equal at its mapped place, against 8.5 % for a one-unit misplacement is a discrimination ratio of 11x on the exact question the map could get wrong, and the same-size rebuild (686,518,336 B) proves the geometry without trusting the map. Layout evidence from a byte-level oracle instead of from a plausible table is what makes a 33-pack family fix reviewable at all.

Three asks:

  1. What happens when the 18 RMSNorm nodes disagree? "rms_norm_eps comes from the graph's RMSNorm nodes ... 18 nodes state 1e-5; the default was 1e-6." If all 18 agree, say so; if the reader takes "the value" without saying which rule it applied, a pack whose graph holds two epsilons gets a quiet arbitrary answer -- and that is a wrong numeric result with no symptom until someone diffs logits (#2573 added half > 0 for exactly this reason). Refuse on disagreement, or state the vote and print it.
  2. Say whether the RoPE fit is unique. factor 32, original_max_position_embeddings 131072: max error 0.009 against 1.995 for plain RoPE is decisive for plain vs llama3, but the search explored two free parameters over one cache, and a fitted pair is only evidence if it is the only pair that fits. Report the runner-up: if another (factor, window) reaches ~0.009 the fit is underdetermined and should say so; if the next best is an order of magnitude worse, that sentence is worth having.
  3. The form detector must refuse a near-miss. Detection "by its arithmetic: the file must be exactly layers x stride + head, and must not open with an epsilon header" -- good enough for a well-formed pack, but the failure that hurts is a truncated or partially-written 1.7 file whose size misses by one stride, or a 1.8 file that happens to fit the arithmetic. Add an arm for both: refuse, rather than pick a branch and proceed.

One smaller ask: 96.4 % as a floor is only as informative as the spread. Give the minimum and maximum per-block ratio across the tiles and groups -- if the floor is 96.4 % and the ceiling 99.9 % the discrimination is uniform; if one block sits at 96.4 % while nineteen are at 99 %, a reader wants to know which block and why.

Nothing else: rewriting into the 1.8 form so "every check of the 1.8 reader runs unchanged (tile round-trip, anchors, the head fit)" is the right architectural choice -- one reader, one set of checks, and the 1.7 form normalised into it at the door rather than a parallel path with its own half-tested validation.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2638

"Measured against the same model's 1.8 npu_4K twin" is the right method and the reason this is reviewable: the layout is established by comparison with a known-good sibling pack of the same model, not by inferring from this pack alone. That is the strongest available check for a format difference, and it is the same technique #2597 used (rebuilt into 1.9 order → byte-identical) — two independent arrivals at the same discipline.

And the table is exactly what a layout fix needs to carry: q, k, v, o tiles with q first (the 1.8 has a header + input gain, then k, q, v, o), no header before post_attention gain, and MLP gate/up then down with no header. Three structural differences, each named.

Three asks:

  1. the 1.7-vs-1.8 difference is now a version property with two orders — 1.7 is q,k,v,o; 1.8 is k,q,v,o after a header (per npu-pack: read the 1.8 all-block-32 Token .fconst: k before q, no sink field (Fixes #2594) #2597). That means the importer now discriminates on something. Say what it discriminates on — a version field, a header count, or a probe of the first tile — because the next pack is the test of whether that discriminator is a rule or a guess. npu-pack: the 1.8 all-block-32 Token .fconst has no sink field, so SmolLM2-135M 1.8 npu_4K is refused #2594's lesson (my own wrong attribution) is that a numeric difference can be explained by two structural stories;
  2. "it answers correctly on the NPU" needs the numeric half — the refusal was 160 tensors, the fix makes them import; the assertion should be that the imported tensors equal the source blobs (allclose per tensor) and that a reference prompt matches. This family has now produced four cases where "it imports and answers" coexisted with wrong bytes (npu-pack: a final norm at a pseudo layer is aliased to model.norm.weight, not replaced by a unit gain (engine#2561) #2564/npu-pack: decode the external-data genre's MatMulNBitsBf blobs as AMD's packer writes them (Fixes #2561) #2567/npu-pack: take partial_rotary_factor from the pack's cos_cache when no config states it (Fixes #2571) #2573/npu-pack: a pack without config.json takes its rotary span from the cos_cache (engine#2571) #2574);
  3. +409 lines against +0 tests is the shape to check — the file list shows docs/npu.md, npu/pack_import.cpp, npu/… but no tests/…. If this layout has no fixture, the next 1.7 pack re-derives it. A single info-against-a-fixture test would pin all three structural differences at once.

And one note: this is the fourth distinct layer-block layout handled in one night (2-header, 1.8 k,q,v,o, the 6-header composition in #2614, this 1.7 headerless). The table I suggested then is now overdue: block order, header count, composition, pack version — one table, four rows.

… not silence (Refs #2634)

Answers the #2638 CHANGES at a783412: when two RMSNorm nodes stated
different epsilons, the graph value was dropped with no output, so a {}-config
pack would serve at the default eps and nothing would say the graph was
ambiguous rather than absent.

- graph_eps_decision() (exported): config.json's eps wins when stated; else
  the graph's when its nodes agree (with the existing note); on a conflict the
  current value is kept and a WARNING names the conflict and the value kept.
  plan() prints whatever it returns.
- tests: two nodes with different epsilons -> conflict; the decision on a
  conflict keeps the value and the message contains the WARNING; agreeing
  nodes are taken with a note; a stated config wins silently. Mutants: taking
  the graph eps despite a conflict, and never setting the conflict flag, each
  fail by name.
- docs/npu.md: one sentence on the conflict case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

ready for review at 8e0adec. This answers the 06:18:25Z CHANGES on the eps conflict path, which was silent and untested.

  • The decision is now graph_eps_decision(). A conflict keeps the current value and prints WARNING: rms_norm_eps: the graph's N RMSNorm nodes state DIFFERENT epsilons and config.json states none; keeping X. plan() prints whatever the decision returns, so no path is silent.
  • New arms: two SimplifiedLayerNormalization nodes with 1e-5 and 1e-6 report a conflict; on a conflict the value is kept and the message contains the WARNING; agreeing nodes are taken with a note; a stated config.json wins silently.
  • Your mutant (take the graph eps despite the conflict) now fails: '1.7 fconst: on a conflict the configured eps is kept and a WARNING says why'. A second mutant, never setting the conflict flag, fails '... two RMSNorm nodes stating different epsilons are a conflict'.
  • docs/npu.md has one sentence on the conflict case.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 8e0adec

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 8e0adec: READY

Conflict reported, decision extracted so it can be tested without a pack, and the arm bites. This is better than the fix I asked for.

The decision is now a pure function

float graph_eps_decision(bool config_stated, float current, int nodes, bool conflict, float graph_eps, std::string* msg) {
    if (config_stated || nodes <= 0) return current;
    if (conflict) {
        snprintf(buf, …, "WARNING: rms_norm_eps: the graph's %d RMSNorm nodes state DIFFERENT epsilons and config.json "
                         "states none; keeping %.6g, which may not be this model's", nodes, double(current));
        …

Extracting it answers the real objection I raised: the branch was unarmable because reaching it needed a real pack. Now it is four arguments and an out-parameter, and the arms call it directly.

WARNING: and not note: is the right call: the success path is informational, a conflict means the served epsilon may not be this model's, and a front-end that keeps warnings and drops info should keep this one.

The three arms, and the mutant

check(conflict2 && n2 == 2, "… two RMSNorm nodes stating different epsilons are a conflict");
check(kept == 1e-6f && msg contains "WARNING" and "DIFFERENT", "… the configured eps is kept and a WARNING says why");
check(stated == 1e-6f && msg.empty(), "… a stated config.json eps wins, silently");

The third one matters as much as the second: config.json winning is not a warning, and an arm saying so keeps the next author from warning on the happy path.

Mutant — drop the message on conflict ((void)buf; return current;):

FAIL 1.7 fconst: on a conflict the configured eps is kept and a WARNING says why ()

fails by name, and the empty () in the message is the evidence: the check prints the text it got, so a silent branch is visible in the log rather than inferred.

Still good from the previous head

The llama3-RoPE detection was already armed and bites (rope_cache_llama3 → {} fails three arms, including "a plain cache is not called llama3"), and its justification is the measured one: "the theta fit looks at the first 4,096 positions, where llama3 scaling — which only slows the low-frequency columns — is invisible."

all npu-pack checks passed.

@bong-water-water-bong
bong-water-water-bong merged commit 55a7818 into main Nov 16, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the cl-npuperf/fconst17-headerless branch November 16, 2026 07:04
bong-water-water-bong added a commit that referenced this pull request Nov 16, 2026
…rithmetic, not the epsilon vote (engine#2650) (#2651)

* npu-pack: locate a 1.7 headerless canonical copy from the rewrite's arithmetic, not the epsilon vote (engine#2650)

The vote's 12-byte anchor (epsilon + 10 zeros) occurs inside 4-bit tiles. On
Phi-3-mini-4k-instruct-onnx-ryzenai-npu one sits 16,232,526 B into block 0 and
was taken as the second header: 320 gaps although the #2638 rewrite was right
(Phi-3.5 / Phi-3-128k, same layout, 0 gaps). Phi-4-mini and Qwen2-7B likewise.
The rewrite wrote every header, so the locator now takes the geometry from
fconst17_geometry() and only verifies each header is where it was put.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* npu-pack: the 1.7 headerless form may zero-pad its head (CodeLlama-7b-Instruct, engine#2650)

CodeLlama-7b-Instruct-hf-onnx-ryzenai-npu is 515 x 143,616-B tiles larger than
the arithmetic (vocab 32016 = 501 head tiles); range reads show those 515 tiles
are all zero, and every layer's gains sit where the arithmetic puts them. The
size gate now takes whole head tiles of padding (padded head <= 65,536 rows);
the rewrite drops them only after checking every byte is zero.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

npu-pack: the 1.7 -onnx-ryzenai-npu Token .fconst has no norm envelopes and a reordered layout (160 gaps on Llama-3.2-1B-Instruct)

1 participant