Repository navigation
npu-pack: read the 1.7 -onnx-ryzenai-npu Token .fconst; eps and llama3 RoPE from the pack itself (Fixes #2634) - #2638
Conversation
…3 RoPE from the pack itself (Fixes #2634) Llama-3.2-1B-Instruct-onnx-ryzenai-npu was refused on 160 tensors ("layer blocks not located: fewer than 6 RMSNorm epsilon headers"). Measured against its 1.8 npu_4K twin, the 1.7 Token file is the same tiles with no norm envelopes: each layer block is [q,k,v,o -- q first][post_attention gain, no header][MLP gate/up + down, no 64-B separator][the NEXT layer's input gain], the tail is the head's tiles, and layer 0's input_layernorm is only in the graph (fusion.onnx's If branches, fp16). - materialise_token_fconst(): canonical_token_fconst17() detects the form by its arithmetic (exactly layers x stride + head, and no epsilon header at offset 0) and rewrites it into the 1.8 form, so every 1.8 reader check runs unchanged. 1.8 / 1.9 files come back untouched. - config.json is {} on this line, so the model's numbers come from the pack: rms_norm_eps from the graph's RMSNorm nodes (walk_graph reads the float `epsilon`; taken only when config.json does not state it), and llama3 RoPE scaling from the cos_cache over its whole length (rope_cache_llama3: factor and original window searched; this pack is factor 32 / original 131072, max error 0.009 vs 1.995 for plain RoPE). - tests: the rewrite on a synthetic geometry is the 1.8 stream byte for byte, with a 1.8 file and a wrong-size file left alone; the llama3 fit recovers 32/131072 and 8/8192 and does not call a plain cache llama3; the graph epsilon is read. Mutants: q-first kept in the rewrite, and an 8192-only llama3 search, each fail. - docs/npu.md: the 1.7 layout table, the rebuild check, eps and RoPE. Measured: - npu-pack info: 0 gaps (was 160). Import: head fit scale 0.989 / rms 1.006 / rel 1.02x floor, row-shifted control 0.066; attention 1280/1280 and MLP 2560/2560 round-trip; 32/32 anchors. - Rebuild of the 1.8 file from the 1.7 bytes: same size (686,518,336 B); every tile / gate-up unit / down group >= 96.4 % byte-equal at its mapped place (a one-unit misplacement: 8.5 %); every 1.7 norm == bf16(graph fp16). - NPU (npu-hold, dmesg 0): "The capital of France is Paris.", "12 * 7 = 84", "Neil Armstrong ... July 20, 1969" -- all stop, 3.9-5.0 tok/s. - ctest (non-device): all pass but submodule_pin_check (my local llama.cpp checkout) and npu_rt_shim (does not link in this tree); npu_w4g32_gate skips. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
ready for review at a783412 |
PR Reviewer Guide 🔍(Review updated until commit 8e0adec)Here are some key observations to aid the review process:
✅ Resolved findingsnpu/pack_import.cpp:1001-1034Memory Management in Rewrite Function In npu/pack_import.cpp:242-244Potential Incorrect Epsilon Handling The code reads epsilon values from the graph's RMSNorm nodes when config.json is empty, but there's a potential issue if multiple nodes disagree on the epsilon value. The code sets |
|
A second family through the same rewrite, at a783412:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at a783412: READY
The two mechanisms are the right shape and their arms assert the tight case; I read them rather than ran them, and the reason is at the end.
1. The 1.7 headerless Token geometry is a closed-form byte count with an exact comparison.
g->stride = g->payload + 2 * H + g->gu + g->down + 2 * H;
g->head = ((V + 63) / 64) * g->tile_h;
const uint64_t want = layers * stride + head;
if (size != want) return "the file is <size> B, and the 1.7 headerless form of this geometry is L x S + a H B head = W B";That is the strongest form this check can take: the wrong form does not merely fail, it prints both numbers and the shape it assumed, so a reader sees which term is off. And the arms test it from both sides — the synthetic stream fits, and 64 B more does not — which pins the comparison as equality rather than a range. 64 B is the right perturbation: every width in the arithmetic is measured in 64-row tiles, so a wrong width cannot land inside 64 B, and an implementer who wrote size <= want or a tolerance would pass the positive arm and fail the negative one.
2. The RoPE llama3 scan prefers measurement over declaration, with the numbers stated.
this pack's cache is factor 32 with original_max_position_embeddings 131072, max error 0.009, against 1.995 plain
the upstream config states 8192, and that fits no better than plain, so the pack's own cache wins
That is a 220x separation between the chosen variant and plain, and the config's own value loses because it explains the cache no better — the same precedence the rope_theta scan in this file already uses (the pack is the artifact, the config is a claim about it). The arms cover factor 32/131072 recovery, factor 8/8192 recovery, and a plain cache that must not be called llama3 — that third one is the one a too-eager detector would fail.
3. No second copy of the arithmetic: group_bytes and attn_tile_bytes are declared once in pack_import.h and defined once in pack_import.cpp; the new code calls them rather than re-deriving the tile maths.
What I did not do, and why: I did not build or run npu_pack_import_test. The build tree for this target is ~1-2 GB and this box is at 156 G, below the 160 G floor I start builds under — so my verdict is a reading of the arithmetic, the arms and the evidence they assert, not an execution of them. When the disk is back above the floor, the 137 new lines run in under a minute and that is the check I would want before merge.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at a783412: CHANGES
The llama3-RoPE half is well done and armed. The eps half has one case where the reader must not guess, and it is the only case that says nothing at all.
The conflict path is silent
bool norm_eps_conflict = false;
…
else if (m.norm_eps != e) m.norm_eps_conflict = true; // two RMSNorm nodes disagree
if (!p.dims.eps_stated && onnx.norm_eps_nodes > 0 && !onnx.norm_eps_conflict && onnx.norm_eps > 0.0f) {
if (p.dims.rms_norm_eps != onnx.norm_eps)
std::fprintf(stderr, "note: rms_norm_eps %.6g taken from the graph (…)\n");
p.dims.rms_norm_eps = onnx.norm_eps;
}When two nodes disagree the graph eps is quietly dropped and the pack keeps whatever the config gave it. There is no else — the one situation where the reader must refuse to guess is the one that prints nothing, in a tool whose success path prints a note: precisely so that a reader knows where the value came from.
The consequence is the shape this repo has spent a week removing: a silent fallback. A {}-config pack (the 1.7 line this PR is for) whose graph states two different epsilons would serve at the default eps, and nothing in the output would say the graph was ambiguous rather than absent.
It is also unarmed — measured
Mutant: drop !onnx.norm_eps_conflict from the condition above (i.e. take the first node's eps despite the conflict).
… fconst eps: qwen 1e-6 … / fconst eps: llama 1e-5 …
final norm: crafted tail envelope imported from tail+64, missing-epsilon control refused
all npu-pack checks passed
Everything still passes. The existing eps arm (1.7 fconst: the graph's RMSNorm epsilon is read, n_eps == 2 && !conflict) only covers two nodes that agree, so it cannot see this branch either way.
The smallest fix
} else if (!p.dims.eps_stated && onnx.norm_eps_nodes > 0 && onnx.norm_eps_conflict) {
std::fprintf(stderr, "note: rms_norm_eps: the graph's %d RMSNorm nodes DISAGREE, keeping %.6g "
"(config.json does not state one)\n", onnx.norm_eps_nodes, double(p.dims.rms_norm_eps));
}plus one arm: craft an ONNX with two SimplifiedLayerNormalization nodes stating different epsilons, and check that graph_norm_eps_exported sets conflict == true and that the message appears. The predicate already exists and is already exported; only the caller's voice is missing.
What is good, and I checked it
llama3 RoPE — mutant RopeLlama3Fit rope_cache_llama3(…) { return RopeLlama3Fit{}; } fails three arms by name:
FAIL rope llama3: factor 32 / original 131072 is recovered (err 1e+09)
FAIL rope llama3: factor 8 / original 8192 is recovered
FAIL rope llama3: a plain cache is not called llama3 (plain err 1e+09)
including the negative arm (a plain cache is not called llama3), which is the one that keeps the detector from claiming scaling everywhere. And the reason the old theta fit missed it is stated exactly: "the theta fit looks at the first 4,096 positions, where llama3 scaling — which only slows the low-frequency columns — is invisible".
The 1.7 .fconst → 1.8 rewrite also runs here (a Ryzen AI 1.7 headerless Token .fconst was rewritten into its 1.8 form), and the geometry checks are two-sided (fits / 64 B more does not).
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- rebuilding the 1.8 file from the 1.7 bytes and comparing to the twin is the strongest verification design I have read tonight
It is a test with a control built into it: 96.4 % byte-equal at its mapped place, against 8.5 % for a one-unit misplacement is a discrimination ratio of 11x on the exact question the map could get wrong, and the same-size rebuild (686,518,336 B) proves the geometry without trusting the map. Layout evidence from a byte-level oracle instead of from a plausible table is what makes a 33-pack family fix reviewable at all.
Three asks:
- What happens when the 18 RMSNorm nodes disagree? "
rms_norm_epscomes from the graph's RMSNorm nodes ... 18 nodes state 1e-5; the default was 1e-6." If all 18 agree, say so; if the reader takes "the value" without saying which rule it applied, a pack whose graph holds two epsilons gets a quiet arbitrary answer -- and that is a wrong numeric result with no symptom until someone diffs logits (#2573 addedhalf > 0for exactly this reason). Refuse on disagreement, or state the vote and print it. - Say whether the RoPE fit is unique.
factor 32, original_max_position_embeddings 131072: max error 0.009against1.995for plain RoPE is decisive for plain vs llama3, but the search explored two free parameters over one cache, and a fitted pair is only evidence if it is the only pair that fits. Report the runner-up: if another(factor, window)reaches ~0.009 the fit is underdetermined and should say so; if the next best is an order of magnitude worse, that sentence is worth having. - The form detector must refuse a near-miss. Detection "by its arithmetic: the file must be exactly layers x stride + head, and must not open with an epsilon header" -- good enough for a well-formed pack, but the failure that hurts is a truncated or partially-written 1.7 file whose size misses by one stride, or a 1.8 file that happens to fit the arithmetic. Add an arm for both: refuse, rather than pick a branch and proceed.
One smaller ask: 96.4 % as a floor is only as informative as the spread. Give the minimum and maximum per-block ratio across the tiles and groups -- if the floor is 96.4 % and the ceiling 99.9 % the discrimination is uniform; if one block sits at 96.4 % while nineteen are at 99 %, a reader wants to know which block and why.
Nothing else: rewriting into the 1.8 form so "every check of the 1.8 reader runs unchanged (tile round-trip, anchors, the head fit)" is the right architectural choice -- one reader, one set of checks, and the 1.7 form normalised into it at the door rather than a parallel path with its own half-tested validation.
|
VERDICT: READY "Measured against the same model's 1.8 And the table is exactly what a layout fix needs to carry: Three asks:
And one note: this is the fourth distinct layer-block layout handled in one night (2-header, 1.8 |
… not silence (Refs #2634) Answers the #2638 CHANGES at a783412: when two RMSNorm nodes stated different epsilons, the graph value was dropped with no output, so a {}-config pack would serve at the default eps and nothing would say the graph was ambiguous rather than absent. - graph_eps_decision() (exported): config.json's eps wins when stated; else the graph's when its nodes agree (with the existing note); on a conflict the current value is kept and a WARNING names the conflict and the value kept. plan() prints whatever it returns. - tests: two nodes with different epsilons -> conflict; the decision on a conflict keeps the value and the message contains the WARNING; agreeing nodes are taken with a note; a stated config wins silently. Mutants: taking the graph eps despite a conflict, and never setting the conflict flag, each fail by name. - docs/npu.md: one sentence on the conflict case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
ready for review at 8e0adec. This answers the 06:18:25Z CHANGES on the eps conflict path, which was silent and untested.
|
|
Persistent review updated to latest commit 8e0adec |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 8e0adec: READY
Conflict reported, decision extracted so it can be tested without a pack, and the arm bites. This is better than the fix I asked for.
The decision is now a pure function
float graph_eps_decision(bool config_stated, float current, int nodes, bool conflict, float graph_eps, std::string* msg) {
if (config_stated || nodes <= 0) return current;
if (conflict) {
snprintf(buf, …, "WARNING: rms_norm_eps: the graph's %d RMSNorm nodes state DIFFERENT epsilons and config.json "
"states none; keeping %.6g, which may not be this model's", nodes, double(current));
…Extracting it answers the real objection I raised: the branch was unarmable because reaching it needed a real pack. Now it is four arguments and an out-parameter, and the arms call it directly.
WARNING: and not note: is the right call: the success path is informational, a conflict means the served epsilon may not be this model's, and a front-end that keeps warnings and drops info should keep this one.
The three arms, and the mutant
check(conflict2 && n2 == 2, "… two RMSNorm nodes stating different epsilons are a conflict");
check(kept == 1e-6f && msg contains "WARNING" and "DIFFERENT", "… the configured eps is kept and a WARNING says why");
check(stated == 1e-6f && msg.empty(), "… a stated config.json eps wins, silently");
The third one matters as much as the second: config.json winning is not a warning, and an arm saying so keeps the next author from warning on the happy path.
Mutant — drop the message on conflict ((void)buf; return current;):
FAIL 1.7 fconst: on a conflict the configured eps is kept and a WARNING says why ()
fails by name, and the empty () in the message is the evidence: the check prints the text it got, so a silent branch is visible in the log rather than inferred.
Still good from the previous head
The llama3-RoPE detection was already armed and bites (rope_cache_llama3 → {} fails three arms, including "a plain cache is not called llama3"), and its justification is the measured one: "the theta fit looks at the first 4,096 positions, where llama3 scaling — which only slows the low-frequency columns — is invisible."
all npu-pack checks passed.
…rithmetic, not the epsilon vote (engine#2650) (#2651) * npu-pack: locate a 1.7 headerless canonical copy from the rewrite's arithmetic, not the epsilon vote (engine#2650) The vote's 12-byte anchor (epsilon + 10 zeros) occurs inside 4-bit tiles. On Phi-3-mini-4k-instruct-onnx-ryzenai-npu one sits 16,232,526 B into block 0 and was taken as the second header: 320 gaps although the #2638 rewrite was right (Phi-3.5 / Phi-3-128k, same layout, 0 gaps). Phi-4-mini and Qwen2-7B likewise. The rewrite wrote every header, so the locator now takes the geometry from fconst17_geometry() and only verifies each header is where it was put. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * npu-pack: the 1.7 headerless form may zero-pad its head (CodeLlama-7b-Instruct, engine#2650) CodeLlama-7b-Instruct-hf-onnx-ryzenai-npu is 515 x 143,616-B tiles larger than the arithmetic (vocab 32016 = 501 head tiles); range reads show those 515 tiles are all zero, and every layer's gains sit where the arithmetic puts them. The size gate now takes whole head tiles of padding (padded head <= 65,536 rows); the rewrite drops them only after checking every byte is zero. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Fixes #2634.
amd/Llama-3.2-1B-Instruct-onnx-ryzenai-npu(the Ryzen AI 1.7-onnx-ryzenai-npuline, about 33 packs) was refused on 160 tensors ("layer blocks not located: fewer than 6 RMSNorm epsilon headers"). It now imports, and it answers correctly on the NPU.The 1.7 Token layout, measured against the same model's 1.8
npu_4Ktwin-onnx-ryzenai-npuLayer 0's input_layernorm is not in the Token file. It is the graph's fp16
model.layers.0.input_layernorm.weight, infusion.onnx'sIfbranches. Every gain the file does carry equals bf16 of the graph's fp16 exactly.Rebuild check. Rebuilding the 1.8 file from the 1.7 bytes with this map gives the same size (686,518,336 B). Every attention tile, gate/up unit and down group is at least 96.4% byte-equal to the twin at its mapped place, against 8.5% for a one-unit misplacement. The remaining bytes are re-quantization: 4-bit codes only. The full map is in the issue.
Change
canonical_token_fconst17(), called frommaterialise_token_fconst(), detects the form by its arithmetic: the file must be exactly layers x stride + head, and must not open with an epsilon header. It then rewrites the file into the 1.8 form, so every check of the 1.8 reader runs unchanged (tile round-trip, anchors, the head fit). 1.8 and 1.9 files come back untouched.config.jsonis{}on this line.rms_norm_epscomes from the graph's RMSNorm nodes.walk_graphnow reads the floatepsilon, and the value is taken only whenconfig.jsondoes not state one. 18 nodes state 1e-5; the default was 1e-6.cos_cacheover its whole length (rope_cache_llama3, with the factor and the original window searched). The existing theta fit (500000) looks only at the first 4,096 positions, where llama3 scaling is invisible.docs/npu.md: the layout table, the rebuild check, and where eps and RoPE come from.Proof
npu-pack info: 0 gaps (was 160).rope_theta500000,rope_scalingllama3 (factor 32, original 131072),rms_norm_eps1e-5.submodule_pin_check(my localllama.cppcheckout) andnpu_rt_shim(its target does not link here).npu_w4g32_gateskips.tools/copyright.py --check: rc 0.The other 1.7 npu families (Qwen, Phi, Mistral) go through the same arithmetic gate, so a family whose sizes differ refuses rather than being misread. Their rows come in the 1.7 sweep.
🤖 Generated with Claude Code