Skip to content

npu-pack: phi-4 (external-data genre) answers 'void void ...': the final norm is dropped (named layers.40.final_norm_layernorm) and the rest is still wrong #2561

Closed

Description

@bong-water-water-bong

What

amd/phi-4_rai_1.9.0_npu is an external-data pack (eager-mm.onnx + .data, MatMulNBitsBf, GroupQueryAttention). It imports with 0 gaps and its container reads back exactly. On engine main (6a25f90 code) with a regenerated kernels-h5120-im17920-b8 family, though, serve --device npu answers every prompt with the same text: ' void void void void ...'. The output is input-independent, so something global is wrong, not a near-tie.

Found so far

  1. The final RMSNorm is dropped (a definite reader bug). The importer says the pack carries no final RMSNorm weight; wrote a unit gain. The pack's graph does have one, attached to a pseudo layer 40:

    SkipSimplifiedLayerNormalization /model/layers.40/final_norm_layernorm/SkipLayerNorm <- model.layers.40.final_norm_layernorm.weight (FLOAT [5120], .data offset 2068840448)

    Fetched by range, it is HF microsoft/phi-4 model.norm.weight exactly: corr +1.0000, mean 1.97, range 0.03-4.47. A unit gain is wrong by about 2x per channel.

  2. That is not the whole cause. With the true gain patched into the container, the output is still void ... on all three prompts. Two other things were ruled out:

    • RoPE base: with rope_theta set to 250000 (HF's value; the pack's config.json is {}, so the importer defaulted to 10000), the output is still void. That default is a second, smaller problem.
    • Layer norms: the container's layer-0 input_layernorm equals the pack's own initializer bit for bit (corr 1.000000). It differs from HF's (corr 0.36) because AMD folded smoothing scales into it, so it is not a reader error.

The old board note for this row ("residual grows 1.09x/layer; host-GEMM gives the same garbage") also points at the import maths rather than the kernels.

Next (cl-npuperf)

Check how the fused tensors are split, using the fold ratio s = ln_HF / ln_pack: for every projection fed by that norm, W_pack should equal W_HF · diag(s). This tests the q/k/v split out of the packed GQA input and the gate/up split of phi-3's fused gate_up_proj (order and halves). A wrong split shows up as a collapse of that correlation.

The fix PR will carry: the final norm read from layers.<L>.final_norm_layernorm, rope_theta taken from the pack (or refusing when it is unknown), and whatever the split check finds.

🤖 Generated with Claude Code

Activity

  1. bong-water-water-bong commented on Nov 15, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Taking this (cl-npuperf): the fold-ratio split check is next, then the importer fix.

  2. bong-water-water-bong commented on Nov 15, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The projection weights themselves are wrong in the container. Measured; the final norm is a separate, smaller bug.

    Our container's layer-0 q/k/v, decoded with the engine's own q4nx::dequant_tile (Q4_1F), compared with HF microsoft/phi-4 qkv_proj:

    • Correlation ≈ 0 (q +0.0004, k -0.0002, v -0.0001). This holds with or without the AWQ fold ratio s = ln_HF / ln_pack, whose median is 2.15, and with row-major or column-major tile order.
    • Magnitude is far off: our rows have norm 171 and element std 2.8, against HF's 2.26 (39 with s) and about 0.03.

    I re-decoded the pack's own blob independently: layer 0's qkv_proj.MatMulNBits.qweight.packed, 21,411,840 B, range-fetched at .data offset 2068905984. The size equals 120 tiles × 82 subvolumes × 2176 B, so the importer's envelope is right.

    Single-field variants of decode_external_tile on its first 64-row tile all sit at the best-of-7680 noise floor, |sim| about 0.055. The variants were: no half swap, no kk == 0 skip, column-pair codes, unsigned fields, and scales/zeros swapped. So this is not one flipped field.

    Spectrum test (rotation-invariant): the singular values of our decoded 64 rows, normalised, correlate 0.973 with HF's raw rows and 0.873 with HF×s. The largest singular value is 266.6 against 5.47, about 49×.

    That is consistent with a pack transform (a rotation that keeps the spectrum, plus a global scale), but it is not proof: HF vs HF×s already scores 0.896. The pack's graph has no rotation op (only MatMulNBitsBf, SkipSimplifiedLayerNormalization, GroupQueryAttention, SiLU), and its layer norms are not unit (mean 0.0132), so a QuaRot-style folded rotation of the residual stream does not fit cleanly either.

    What this needs: the RyzenAI eager MatMulNBitsBf weight transform, from the runtime's reference (REA on the provider, reference only), not more guessing. Until then the row stays loads-wrong-output.

    The two small importer bugs are confirmed and will go in the same PR as the real fix, not on their own, because neither changes the output:

    1. the final norm, named model.layers.40.final_norm_layernorm.weight, is not mapped to model.norm.weight (a unit gain is written instead);
    2. rope_theta defaults to 10000 when the pack's config.json is {}; HF uses 250000.

    Scratch on strixhalo: the container (with the true final norm patched in), eager-mm.onnx, the layer-0 blob, the HF reference tensors, and the numpy decoder at ~/.cache/scratch-cl-npuperf/amdcol/{phi4onnx,phi4ref,phi4dec}.

  3. bong-water-water-bong commented on Nov 15, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Ruled out, so nobody repeats these. All were run on layer 0's qkv_proj blob (first 64-row tile), scored by best |cosine| against HF qkv_proj rows (×s where noted); the noise floor is ~0.055.

    hypothesis result
    single-field variants of decode_external_tile (half swap, kk == 0 skip, column-pair codes, unsigned fields, scale/zero order) 0.052-0.060
    168 microtile layouts inside the 32×128 subvolume (blocks 1-32 × 2-128, row/column block order, row/column inner order) 0.052-0.061
    block-Hadamard on K (32-1024), with s, 1/s or no scale peak 0.125 at 128 with s, all others ≤ 0.098
    signed 128-Hadamard (W_pack = W_HF·diag(s)·D·H₁₂₈, row order kept): per-column corr after undoing H 0.08 (noise for 64 rows); whole-matrix +0.02 after the sign fix

    So the codes are neither a plain layout of HF's weights nor a simple rotation of them. The transform lives in the provider: libonnxruntime_providers_ryzenai.so (ryzenai::onnx_utils::MatMulNBitsKernel, "Packed_Consts", "deferring packed-const BO allocation"). Next is an REA pass on that pre-packing (reference only). The 1.8 provider is on this box (~/ryzenai-1.8/venv/deployment/lib/); whether 1.9's eager packing matches it is part of the question.

  4. added a commit that references this issue on Nov 15, 2026
    2b7eccc
  5. bong-water-water-bong commented on Nov 15, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Device row measured: the reader fix is in and the symptom persists — the final norm was not the whole cause

    Build from merged main (39222f460, my #2564), the engine's pinned NPU runtime (config/npu-runtime.config: sha=8dca02ec, path=runtime/npu_rt), container imported with the fixed reader (--weight-layout b8, the import prints the pseudo-layer line), kernel family ~/ogaload/families/kernels-h5120-im17920-b8. One npu-hold ticket, -t 12, no other ticket of mine:

    gpu-hold took NPU-TOKEN after 0s (die 59C)
    1bit forward-serve: phi4 ready on http://127.0.0.1:18180 (full ELF) [npu-target=xdna]
    health=200 after 37s
    REPLY: {"content":" void void void void …" }     24 of 24 tokens
    timings: prompt_ms 6721, predicted_ms 42973
    amdxdna timeout/power-cycle lines: before=5 after=5
    gpu-hold released NPU-TOKEN; command exited 0
    

    So: container right, model loaded on the NPU with full ELFs, answer unchanged. The final norm in that container was verified bit-exact against the pack's model.layers.40.final_norm_layernorm.weight (max|diff| = 0.000000 after bf16), so the reader half of #2561 is real -- but it is not sufficient, and "Found so far 1." was right to be a list.

    Where I would look next, from the code rather than a guess: the forward reads the final gain through mw_->norm_weight (q4nx_forward.cpp:2149, read_bf16(&mw_->norm_weight, …)) and applies it at :5344/:5358/:5363, while the container key the importer writes is model.norm.weight (pack_import.cpp:3826 role final_norm). Those agree -- so the next suspects are the ones that produce the same symptom (argmax collapsing to one token regardless of input) rather than the same cause: everything downstream of a correct norm. Two cheap discriminators before touching the device again:

    1. [mla] final xn: mean|x|=… -- the log line that npu-pack: phi-4 (external-data genre) answers 'void void ...': the final norm is dropped (named layers.40.final_norm_layernorm) and the rest is still wrong #2561 quotes from the earlier diagnosis. If it is now non-zero, the norm path is fixed and the collapse is in the head/embedding; if it is still 0, the gain is not reaching step(). That is one line from the same serve log, no new build.
    2. 1bit serve --dump-logits-style row (or two different prompts): if two prompts give the same 24 tokens, the collapse is input-independent as reported, and the cheapest next probe is the embedding (model.embed_tokens.weight) for this genre -- the importer's own message for external-data packs lists what it cannot represent, and a zero/garbage embedding produces exactly this.

    Not retried, not claimed: one run, no amdxdna line, ticket released cleanly. I am not closing #2561: the fix that merged removes one confirmed defect; this row says a second one remains, and its evidence is now two-sided (before: unit gain → void; after: pack's norm → still void).

  6. bong-water-water-bong commented on Nov 15, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Fixed end to end: the same three prompts now answer correctly — Neil Armstrong, 4, Blue

    The "before" row on this issue was three different prompts giving byte-identical void × 12 (measured at 13:58Z on a build with #2564 only). Same harness, same pack, same kernel family, same container recipe, one npu-hold ticket:

    prompt before (main @ 39222f460, #2564 only) after (main @ 21a186d9c, #2564 + #2567)
    "Who was the first person to walk on the Moon?" void × 12 The first person to walk on the Moon was Neil Armstrong.
    "What is 2 + 2? Answer with one number." void × 12 4
    "Name a colour." void × 12 Blue.

    health=200 after 35s, 1bit forward-serve: phi4 ready … (full ELF) [npu-target=xdna], amdxdna fault lines before=5 after=5, ticket released, nothing left running.

    The single variable between the two rows is #2567 — the "before" build already contained #2564 (the final-norm alias, my PR), so the decode fix is what moved the answer, which matches #2567's own analysis of decode_external_tile. That is also why my earlier comment's "the norm was not the whole cause" was right and this one is the completion of it: two defects, two fixes, and the row that separates them is the one above.

    Container notes, for whoever repeats this: the import is npu-pack import <pack> -o <dir> --weight-layout b8 (~1m12s, 11.6 GB, prints the pseudo-layer line from #2564), and at first serve the container is repacked from q4nx-2 to the device's q4_1 form:

    1bit q4nx: …/model.q4nx is a q4nx-2 container (fp32 scale/zero, engine#1185);
               repacked 281 I8 tensor(s), 1,726,720 tile(s) to the device's 5,120-byte q4_1 at ~/.cache/1bit/q4nx/…
    

    So a stale ~/.cache/1bit/q4nx/* from an earlier import is worth clearing before comparing rows — a cached repack of the old container would answer void with both fixes merged, which is exactly the misreading I flagged on #2567.

    No amdxdna line in either run, so this was two 12-minute NPU tickets with nothing to stop for.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions