You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
npu-pack: phi-4 (external-data genre) answers 'void void ...': the final norm is dropped (named layers.40.final_norm_layernorm) and the rest is still wrong #2561
amd/phi-4_rai_1.9.0_npu is an external-data pack (eager-mm.onnx + .data, MatMulNBitsBf, GroupQueryAttention). It imports with 0 gaps and its container reads back exactly. On engine main (6a25f90 code) with a regenerated kernels-h5120-im17920-b8 family, though, serve --device npu answers every prompt with the same text: ' void void void void ...'. The output is input-independent, so something global is wrong, not a near-tie.
Found so far
The final RMSNorm is dropped (a definite reader bug). The importer says the pack carries no final RMSNorm weight; wrote a unit gain. The pack's graph does have one, attached to a pseudo layer 40:
Fetched by range, it is HF microsoft/phi-4model.norm.weight exactly: corr +1.0000, mean 1.97, range 0.03-4.47. A unit gain is wrong by about 2x per channel.
That is not the whole cause. With the true gain patched into the container, the output is still void ... on all three prompts. Two other things were ruled out:
RoPE base: with rope_theta set to 250000 (HF's value; the pack's config.json is {}, so the importer defaulted to 10000), the output is still void. That default is a second, smaller problem.
Layer norms: the container's layer-0 input_layernorm equals the pack's own initializer bit for bit (corr 1.000000). It differs from HF's (corr 0.36) because AMD folded smoothing scales into it, so it is not a reader error.
The old board note for this row ("residual grows 1.09x/layer; host-GEMM gives the same garbage") also points at the import maths rather than the kernels.
Next (cl-npuperf)
Check how the fused tensors are split, using the fold ratio s = ln_HF / ln_pack: for every projection fed by that norm, W_pack should equal W_HF · diag(s). This tests the q/k/v split out of the packed GQA input and the gate/up split of phi-3's fused gate_up_proj (order and halves). A wrong split shows up as a collapse of that correlation.
The fix PR will carry: the final norm read from layers.<L>.final_norm_layernorm, rope_theta taken from the pack (or refusing when it is unknown), and whatever the split check finds.
The projection weights themselves are wrong in the container. Measured; the final norm is a separate, smaller bug.
Our container's layer-0 q/k/v, decoded with the engine's own q4nx::dequant_tile (Q4_1F), compared with HF microsoft/phi-4qkv_proj:
Correlation ≈ 0 (q +0.0004, k -0.0002, v -0.0001). This holds with or without the AWQ fold ratio s = ln_HF / ln_pack, whose median is 2.15, and with row-major or column-major tile order.
Magnitude is far off: our rows have norm 171 and element std 2.8, against HF's 2.26 (39 with s) and about 0.03.
I re-decoded the pack's own blob independently: layer 0's qkv_proj.MatMulNBits.qweight.packed, 21,411,840 B, range-fetched at .data offset 2068905984. The size equals 120 tiles × 82 subvolumes × 2176 B, so the importer's envelope is right.
Single-field variants of decode_external_tile on its first 64-row tile all sit at the best-of-7680 noise floor, |sim| about 0.055. The variants were: no half swap, no kk == 0 skip, column-pair codes, unsigned fields, and scales/zeros swapped. So this is not one flipped field.
Spectrum test (rotation-invariant): the singular values of our decoded 64 rows, normalised, correlate 0.973 with HF's raw rows and 0.873 with HF×s. The largest singular value is 266.6 against 5.47, about 49×.
That is consistent with a pack transform (a rotation that keeps the spectrum, plus a global scale), but it is not proof: HF vs HF×s already scores 0.896. The pack's graph has no rotation op (only MatMulNBitsBf, SkipSimplifiedLayerNormalization, GroupQueryAttention, SiLU), and its layer norms are not unit (mean 0.0132), so a QuaRot-style folded rotation of the residual stream does not fit cleanly either.
What this needs: the RyzenAI eager MatMulNBitsBf weight transform, from the runtime's reference (REA on the provider, reference only), not more guessing. Until then the row stays loads-wrong-output.
The two small importer bugs are confirmed and will go in the same PR as the real fix, not on their own, because neither changes the output:
the final norm, named model.layers.40.final_norm_layernorm.weight, is not mapped to model.norm.weight (a unit gain is written instead);
rope_theta defaults to 10000 when the pack's config.json is {}; HF uses 250000.
Scratch on strixhalo: the container (with the true final norm patched in), eager-mm.onnx, the layer-0 blob, the HF reference tensors, and the numpy decoder at ~/.cache/scratch-cl-npuperf/amdcol/{phi4onnx,phi4ref,phi4dec}.
Ruled out, so nobody repeats these. All were run on layer 0's qkv_proj blob (first 64-row tile), scored by best |cosine| against HF qkv_proj rows (×s where noted); the noise floor is ~0.055.
hypothesis
result
single-field variants of decode_external_tile (half swap, kk == 0 skip, column-pair codes, unsigned fields, scale/zero order)
block-Hadamard on K (32-1024), with s, 1/s or no scale
peak 0.125 at 128 with s, all others ≤ 0.098
signed 128-Hadamard (W_pack = W_HF·diag(s)·D·H₁₂₈, row order kept): per-column corr after undoing H
0.08 (noise for 64 rows); whole-matrix +0.02 after the sign fix
So the codes are neither a plain layout of HF's weights nor a simple rotation of them. The transform lives in the provider: libonnxruntime_providers_ryzenai.so (ryzenai::onnx_utils::MatMulNBitsKernel, "Packed_Consts", "deferring packed-const BO allocation"). Next is an REA pass on that pre-packing (reference only). The 1.8 provider is on this box (~/ryzenai-1.8/venv/deployment/lib/); whether 1.9's eager packing matches it is part of the question.
Device row measured: the reader fix is in and the symptom persists — the final norm was not the whole cause
Build from merged main (39222f460, my #2564), the engine's pinned NPU runtime (config/npu-runtime.config: sha=8dca02ec, path=runtime/npu_rt), container imported with the fixed reader (--weight-layout b8, the import prints the pseudo-layer line), kernel family ~/ogaload/families/kernels-h5120-im17920-b8. One npu-hold ticket, -t 12, no other ticket of mine:
gpu-hold took NPU-TOKEN after 0s (die 59C)
1bit forward-serve: phi4 ready on http://127.0.0.1:18180 (full ELF) [npu-target=xdna]
health=200 after 37s
REPLY: {"content":" void void void void …" } 24 of 24 tokens
timings: prompt_ms 6721, predicted_ms 42973
amdxdna timeout/power-cycle lines: before=5 after=5
gpu-hold released NPU-TOKEN; command exited 0
So: container right, model loaded on the NPU with full ELFs, answer unchanged. The final norm in that container was verified bit-exact against the pack's model.layers.40.final_norm_layernorm.weight (max|diff| = 0.000000 after bf16), so the reader half of #2561 is real -- but it is not sufficient, and "Found so far 1." was right to be a list.
Where I would look next, from the code rather than a guess: the forward reads the final gain through mw_->norm_weight (q4nx_forward.cpp:2149, read_bf16(&mw_->norm_weight, …)) and applies it at :5344/:5358/:5363, while the container key the importer writes is model.norm.weight (pack_import.cpp:3826 role final_norm). Those agree -- so the next suspects are the ones that produce the same symptom (argmax collapsing to one token regardless of input) rather than the same cause: everything downstream of a correct norm. Two cheap discriminators before touching the device again:
1bit serve --dump-logits-style row (or two different prompts): if two prompts give the same 24 tokens, the collapse is input-independent as reported, and the cheapest next probe is the embedding (model.embed_tokens.weight) for this genre -- the importer's own message for external-data packs lists what it cannot represent, and a zero/garbage embedding produces exactly this.
Not retried, not claimed: one run, no amdxdna line, ticket released cleanly. I am not closing #2561: the fix that merged removes one confirmed defect; this row says a second one remains, and its evidence is now two-sided (before: unit gain → void; after: pack's norm → still void).
Fixed end to end: the same three prompts now answer correctly — Neil Armstrong, 4, Blue
The "before" row on this issue was three different prompts giving byte-identicalvoid × 12 (measured at 13:58Z on a build with #2564 only). Same harness, same pack, same kernel family, same container recipe, one npu-hold ticket:
The single variable between the two rows is #2567 — the "before" build already contained #2564 (the final-norm alias, my PR), so the decode fix is what moved the answer, which matches #2567's own analysis of decode_external_tile. That is also why my earlier comment's "the norm was not the whole cause" was right and this one is the completion of it: two defects, two fixes, and the row that separates them is the one above.
Container notes, for whoever repeats this: the import is npu-pack import <pack> -o <dir> --weight-layout b8 (~1m12s, 11.6 GB, prints the pseudo-layer line from #2564), and at first serve the container is repacked from q4nx-2 to the device's q4_1 form:
1bit q4nx: …/model.q4nx is a q4nx-2 container (fp32 scale/zero, engine#1185);
repacked 281 I8 tensor(s), 1,726,720 tile(s) to the device's 5,120-byte q4_1 at ~/.cache/1bit/q4nx/…
So a stale ~/.cache/1bit/q4nx/* from an earlier import is worth clearing before comparing rows — a cached repack of the old container would answer void with both fixes merged, which is exactly the misreading I flagged on #2567.
No amdxdna line in either run, so this was two 12-minute NPU tickets with nothing to stop for.
Activity
bong-water-water-bong commentedon Nov 15, 2026
Taking this (cl-npuperf): the fold-ratio split check is next, then the importer fix.
bong-water-water-bong commentedon Nov 15, 2026
The projection weights themselves are wrong in the container. Measured; the final norm is a separate, smaller bug.
Our container's layer-0
q/k/v, decoded with the engine's ownq4nx::dequant_tile(Q4_1F), compared with HFmicrosoft/phi-4qkv_proj:s = ln_HF / ln_pack, whose median is 2.15, and with row-major or column-major tile order.I re-decoded the pack's own blob independently: layer 0's
qkv_proj.MatMulNBits.qweight.packed, 21,411,840 B, range-fetched at.dataoffset 2068905984. The size equals 120 tiles × 82 subvolumes × 2176 B, so the importer's envelope is right.Single-field variants of
decode_external_tileon its first 64-row tile all sit at the best-of-7680 noise floor, |sim| about 0.055. The variants were: no half swap, nokk == 0skip, column-pair codes, unsigned fields, and scales/zeros swapped. So this is not one flipped field.Spectrum test (rotation-invariant): the singular values of our decoded 64 rows, normalised, correlate 0.973 with HF's raw rows and 0.873 with HF×s. The largest singular value is 266.6 against 5.47, about 49×.
That is consistent with a pack transform (a rotation that keeps the spectrum, plus a global scale), but it is not proof: HF vs HF×s already scores 0.896. The pack's graph has no rotation op (only
MatMulNBitsBf,SkipSimplifiedLayerNormalization,GroupQueryAttention, SiLU), and its layer norms are not unit (mean 0.0132), so a QuaRot-style folded rotation of the residual stream does not fit cleanly either.What this needs: the RyzenAI eager
MatMulNBitsBfweight transform, from the runtime's reference (REA on the provider, reference only), not more guessing. Until then the row staysloads-wrong-output.The two small importer bugs are confirmed and will go in the same PR as the real fix, not on their own, because neither changes the output:
model.layers.40.final_norm_layernorm.weight, is not mapped tomodel.norm.weight(a unit gain is written instead);rope_thetadefaults to 10000 when the pack'sconfig.jsonis{}; HF uses 250000.Scratch on strixhalo: the container (with the true final norm patched in),
eager-mm.onnx, the layer-0 blob, the HF reference tensors, and the numpy decoder at~/.cache/scratch-cl-npuperf/amdcol/{phi4onnx,phi4ref,phi4dec}.bong-water-water-bong commentedon Nov 15, 2026
Ruled out, so nobody repeats these. All were run on layer 0's
qkv_projblob (first 64-row tile), scored by best |cosine| against HFqkv_projrows (×s where noted); the noise floor is ~0.055.decode_external_tile(half swap,kk == 0skip, column-pair codes, unsigned fields, scale/zero order)So the codes are neither a plain layout of HF's weights nor a simple rotation of them. The transform lives in the provider:
libonnxruntime_providers_ryzenai.so(ryzenai::onnx_utils::MatMulNBitsKernel, "Packed_Consts", "deferring packed-const BO allocation"). Next is an REA pass on that pre-packing (reference only). The 1.8 provider is on this box (~/ryzenai-1.8/venv/deployment/lib/); whether 1.9's eager packing matches it is part of the question.npu-pack: decode the external-data genre's MatMulNBitsBf blobs as AMD…
bong-water-water-bong commentedon Nov 15, 2026
Device row measured: the reader fix is in and the symptom persists — the final norm was not the whole cause
Build from merged
main(39222f460, my #2564), the engine's pinned NPU runtime (config/npu-runtime.config:sha=8dca02ec,path=runtime/npu_rt), container imported with the fixed reader (--weight-layout b8, the import prints the pseudo-layer line), kernel family~/ogaload/families/kernels-h5120-im17920-b8. Onenpu-holdticket,-t 12, no other ticket of mine:So: container right, model loaded on the NPU with full ELFs, answer unchanged. The final norm in that container was verified bit-exact against the pack's
model.layers.40.final_norm_layernorm.weight(max|diff| = 0.000000after bf16), so the reader half of #2561 is real -- but it is not sufficient, and "Found so far 1." was right to be a list.Where I would look next, from the code rather than a guess: the forward reads the final gain through
mw_->norm_weight(q4nx_forward.cpp:2149,read_bf16(&mw_->norm_weight, …)) and applies it at:5344/:5358/:5363, while the container key the importer writes ismodel.norm.weight(pack_import.cpp:3826rolefinal_norm). Those agree -- so the next suspects are the ones that produce the same symptom (argmax collapsing to one token regardless of input) rather than the same cause: everything downstream of a correct norm. Two cheap discriminators before touching the device again:[mla] final xn: mean|x|=…-- the log line that npu-pack: phi-4 (external-data genre) answers 'void void ...': the final norm is dropped (named layers.40.final_norm_layernorm) and the rest is still wrong #2561 quotes from the earlier diagnosis. If it is now non-zero, the norm path is fixed and the collapse is in the head/embedding; if it is still 0, the gain is not reachingstep(). That is one line from the same serve log, no new build.1bit serve --dump-logits-style row (or two different prompts): if two prompts give the same 24 tokens, the collapse is input-independent as reported, and the cheapest next probe is the embedding (model.embed_tokens.weight) for this genre -- the importer's own message for external-data packs lists what it cannot represent, and a zero/garbage embedding produces exactly this.Not retried, not claimed: one run, no
amdxdnaline, ticket released cleanly. I am not closing #2561: the fix that merged removes one confirmed defect; this row says a second one remains, and its evidence is now two-sided (before: unit gain →void; after: pack's norm → stillvoid).bong-water-water-bong commentedon Nov 15, 2026
Fixed end to end: the same three prompts now answer correctly —
Neil Armstrong,4,BlueThe "before" row on this issue was three different prompts giving byte-identical
void × 12(measured at 13:58Z on a build with #2564 only). Same harness, same pack, same kernel family, same container recipe, onenpu-holdticket:39222f460, #2564 only)21a186d9c, #2564 + #2567)void × 12The first person to walk on the Moon was Neil Armstrong.void × 124void × 12Blue.health=200 after 35s,1bit forward-serve: phi4 ready … (full ELF) [npu-target=xdna],amdxdnafault linesbefore=5 after=5, ticket released, nothing left running.The single variable between the two rows is #2567 — the "before" build already contained #2564 (the final-norm alias, my PR), so the decode fix is what moved the answer, which matches #2567's own analysis of
decode_external_tile. That is also why my earlier comment's "the norm was not the whole cause" was right and this one is the completion of it: two defects, two fixes, and the row that separates them is the one above.Container notes, for whoever repeats this: the import is
npu-pack import <pack> -o <dir> --weight-layout b8(~1m12s, 11.6 GB, prints the pseudo-layer line from #2564), and at first serve the container is repacked from q4nx-2 to the device's q4_1 form:So a stale
~/.cache/1bit/q4nx/*from an earlier import is worth clearing before comparing rows — a cached repack of the old container would answervoidwith both fixes merged, which is exactly the misreading I flagged on #2567.No
amdxdnaline in either run, so this was two 12-minute NPU tickets with nothing to stop for.