Skip to content

npu: a wrong Q4NX converter version yields containers the engine answers with noise — silently, 200, nothing in the log #1109

Closed

Description

@bong-water-water-bong

What happens

The GGUF-on-NPU repack trusts whatever Q4NX converter checkout it finds, and a wrong one produces a container this engine
answers with noise — with no error anywhere in serve, and no line in the log
. Measured on this box today, same binary
(da845311…, origin/main bd7d451a), same GGUF, same prompt ("The capital of France is", 16 greedy tokens), host arm:

converter checkout repack Qwen3-0.6B answer
~/wt/npu-rt-753-pr/third_party/FLM_Q4NX_Converter ok " Paris. The capital of the country is a city. The capital of the country"
~/amd-oss/FLM_Q4NX_Converter (d1d5232, "feat: update converter for q4_k") ok "οοοοοοοοοοοοοοοο"
llama.cpp on the same GGUF (reference) — " Paris. The capital of France is also the capital of the Republic of France."

Both converters "succeed"; both containers load (311 tensors, 28 layers, ready on … (full ELF)); the only difference is
repack-stamp.txt's converter_commit. The tile-order difference is total, not subtle: decoding one weight from each —
layer-0 down_proj of ZR1-1.5B, packed from the same GGUF by our own tools/gguf_to_q4nx.py versus by the amd-oss
converter — gives corr +0.0325 (same values, permuted inside the 5120-byte chunk), while the npu-rt-753-pr converter's
container gives corr +0.9994.

So a converter checkout that is the wrong version is a silent correctness bug: the engine serves fluent-looking garbage
with a 200. On a 1.5B model it looks like a bad quantization; on a 0.6B model it is οοοο…, and on a different prompt nobody
may notice.

Why the resolver does not save us

scripts/repack_gguf.py resolves the converter in a documented order (its comment block explains which checkout is
"canonical" and why: the q/k interleaved→paired reorder, the MiniCPM4 scale fixes, LongRoPE). On this box that order lands on
nothing usable:

  • ~/np-model-generic — exists as an empty directory (no .git, no third_party/), so the canonical checkout the script
    prefers is absent;
  • ~/1bit-MONSTER/third_party/FLM_Q4NX_Converter — repack fails: ModuleNotFoundError: No module named 'q4nx.models'
    (partial checkout);
  • ~/1bit-MONSTER-iso-build/… and ~/1bit-MONSTER-build-2628/… — repack fails (rc=1);
  • the only checkout that repacks and is correct is ~/wt/npu-rt-753-pr/third_party/FLM_Q4NX_Converter, which is not in
    the resolver's candidate list at all — I only found it by sweeping every checkout on the box.

And when the resolver finds nothing, the documented fallback is "point ONEBIT_Q4NX_CONVERTER at a converter checkout"
(docs/npu.md:498), which is how the wrong one gets used: the doc names no validated commit, and repack-stamp.txt records
the commit but nothing compares it to a known-good one.

Fix options, cheapest first

  1. Fail loud on an unknown stamp at serve time. serve already reads repack-stamp.txt for --check-stamp; make it print
    one line when the stamp's converter_commit is not in a small pinned list (and --strict-repack refuse), the way an
    unreadable pinned model or a missing kernel directory already refuses. Costs a constant, catches every future drift.
  2. Name the validated checkout in docs/npu.md and in the resolver's candidate list (~/wt/npu-rt-753-pr/... is a build
    tree — the durable answer is to say which commit is validated and let the stamp check enforce it).
  3. Verify the repack against its source. scripts/repack_gguf.py could decode the first tile of one tensor and compare it
    with the GGUF it came from — the check that finds this class without trusting any commit: tools/q4nx_diff.py's decode
    subcommand already does exactly this comparison (--layout flm|ours), and this lane measured 0.0325 vs 0.9994 with it.
    A --verify-source step there would have refused the bad container before serve cached it.

Reproducer

~/lb/repack-converter-sweep.sh (pi-onebp lane): for each converter checkout on the box, clears
~/.cache/1bit/q4nx/<model>, repacks Qwen3-0.6B through 1bit serve -m <gguf> --device npu, posts one prompt, and prints the
answer next to llama.cpp's on the same GGUF. One ticket, ~10 min. The container cache is the trap as well: a stale good
container hides a bad converter and vice versa, so the script clears it per arm.

Found while fixing #1097 (padded weight grids, PR #1107); deliberately kept out of that PR — it is a different failure with a
different owner, and #1097's guard ("read only a grid that dominates the model's, else keep failing") already opts for
refusing over decoding garbage.

Activity

  1. bong-water-water-bong commented on Nov 5, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Measured the same two converters today from the other end (silent-noise container), and this is the repack-side half of a
    defect whose engine-side half is #1110. Numbers, because they decide what the guard can be:

    ZR1-1.5B container from decode with --layout ours decode with --layout flm
    ~/wt/npu-rt-753-pr/third_party/FLM_Q4NX_Converter 2.957e-02 / 2.313e-02 (requant scale) 2.831e+00 / 1.027e+00
    ~/amd-oss/FLM_Q4NX_Converter @ d1d5232 5.476e+00 / 6.084e-01 3.455e+00 / 7.680e-01 (and 2.313e-02 for the Q6_K-sourced tensor with unsigned nibbles)

    (two tensors: blk.14.attn_output Q4_K and blk.0.ffn_down Q6_K. Requant scale ≈ 2-3e-2; noise ≈ 1e+0.)

    What that means for a guard here: there is no single "the flm order" to detect — d1d5232's output mixes orders by source
    quant type, and one of its tensors is only correct with unsigned nibbles. So a guard in scripts/repack_gguf.py should not try
    to identify the converter; it should verify the container it just wrote against the GGUF it came from, which is exactly what
    tools/q4nx_diff.py decode --layout ours already does and which separates these two containers by ~2 orders of magnitude
    (2.9e-2 vs 5.5e0). Concretely, after the converter returns:

    python3 tools/q4nx_diff.py decode --gguf <gguf> --q4nx <out> --layout ours [--tensor blk.0]
      -> rc 0 when a layout matches within the requant tolerance, rc 1 when nothing does
    

    and the repack fails (or warns loudly, your call) on rc 1, naming repack-stamp.txt's converter_commit. That closes the hole
    the moment the wrong checkout is used, on any model, without a table of converter versions — and it is device-free (the same
    host arm your rows use).

    Two caveats worth stating before someone implements it:

    1. --tensor scoping matters: a padded container (IM 8960 → 9216) has tensors the checker cannot read until q4nx-diff: read a container whose weight is padded to a wider tile grid (part of #1110) #1119 lands, so
      scope the verify to non-MLP tensors (attn*) for now — those already separate the two converters by ~200x.
    2. The verify costs a GGUF pass (seconds), so it belongs after the write with a --skip-verify escape for the tight loops the
      payload builds run in.

    Engine-side, #1110/#1130 makes a container declare its tile order and refuses an unknown one — but that only helps containers
    from a converter that declares an id, so it does not replace this guard: the wrong checkout writes no id and would simply be
    trusted.

    Not taking it tonight — three of my PRs are open (#1119, #1130, #1133) and stacking a fourth unreviewed one is what this lane
    got wrong earlier in the week. Say the word and it is first up, or take it: the measurement and the checker are already there.

  2. bong-water-water-bong commented on Nov 6, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Resolved by #1234 (merged aed110c1): the repack/verify path this issue's last comment asks for is now the contract.

    • scripts/repack_gguf.py --verify-source <dir> <model.gguf> (reusable verify_source()) decodes the container against the GGUF it came from under both tile layouts — the source comparison, not a table of converter commits. Exit 0 engine-1, 1 another order or none, 2 nothing comparable.
    • The verdict names the worst source tensor and its correlation (corr +0.0325 for the wrong checkout, +0.9994 honest), and the refusal names the converter path @ commit.
    • --skip-verify is the tight-loop escape and stamps tile_order=unverified, never engine-1.
    • The reuse hole is closed: a directory the repack refused is now stale on the next --check-stamp instead of current, so a rejected container cannot be served after the refusal.
    • Test: tests/repack_verify_source_test.py (25 assertions, wired into ci.yml), plus q4nx_diff --self-test, q4nx_diff_grid, repack_tile_order, tile_check_no_shared_tensor, npu_plan_tile_order.

    The issue's option 2 (name a validated converter commit in docs/npu.md and the resolver) is deliberately not done: verifying the container against its source removes the need to trust any commit list. Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions