Repository navigation
Description
What happens
The GGUF-on-NPU repack trusts whatever Q4NX converter checkout it finds, and a wrong one produces a container this engine
answers with noise — with no error anywhere in serve, and no line in the log. Measured on this box today, same binary
(da845311…, origin/main bd7d451a), same GGUF, same prompt ("The capital of France is", 16 greedy tokens), host arm:
| converter checkout | repack | Qwen3-0.6B answer |
|---|---|---|
~/wt/npu-rt-753-pr/third_party/FLM_Q4NX_Converter |
ok | " Paris. The capital of the country is a city. The capital of the country" |
~/amd-oss/FLM_Q4NX_Converter (d1d5232, "feat: update converter for q4_k") |
ok | "οοοοοοοοοοοοοοοο" |
| llama.cpp on the same GGUF (reference) | — | " Paris. The capital of France is also the capital of the Republic of France." |
Both converters "succeed"; both containers load (311 tensors, 28 layers, ready on … (full ELF)); the only difference is
repack-stamp.txt's converter_commit. The tile-order difference is total, not subtle: decoding one weight from each —
layer-0 down_proj of ZR1-1.5B, packed from the same GGUF by our own tools/gguf_to_q4nx.py versus by the amd-oss
converter — gives corr +0.0325 (same values, permuted inside the 5120-byte chunk), while the npu-rt-753-pr converter's
container gives corr +0.9994.
So a converter checkout that is the wrong version is a silent correctness bug: the engine serves fluent-looking garbage
with a 200. On a 1.5B model it looks like a bad quantization; on a 0.6B model it is οοοο…, and on a different prompt nobody
may notice.
Why the resolver does not save us
scripts/repack_gguf.py resolves the converter in a documented order (its comment block explains which checkout is
"canonical" and why: the q/k interleaved→paired reorder, the MiniCPM4 scale fixes, LongRoPE). On this box that order lands on
nothing usable:
~/np-model-generic— exists as an empty directory (no.git, nothird_party/), so the canonical checkout the script
prefers is absent;~/1bit-MONSTER/third_party/FLM_Q4NX_Converter— repack fails:ModuleNotFoundError: No module named 'q4nx.models'
(partial checkout);~/1bit-MONSTER-iso-build/…and~/1bit-MONSTER-build-2628/…— repack fails (rc=1);- the only checkout that repacks and is correct is
~/wt/npu-rt-753-pr/third_party/FLM_Q4NX_Converter, which is not in
the resolver's candidate list at all — I only found it by sweeping every checkout on the box.
And when the resolver finds nothing, the documented fallback is "point ONEBIT_Q4NX_CONVERTER at a converter checkout"
(docs/npu.md:498), which is how the wrong one gets used: the doc names no validated commit, and repack-stamp.txt records
the commit but nothing compares it to a known-good one.
Fix options, cheapest first
- Fail loud on an unknown stamp at serve time.
servealready readsrepack-stamp.txtfor--check-stamp; make it print
one line when the stamp'sconverter_commitis not in a small pinned list (and--strict-repackrefuse), the way an
unreadable pinned model or a missing kernel directory already refuses. Costs a constant, catches every future drift. - Name the validated checkout in
docs/npu.mdand in the resolver's candidate list (~/wt/npu-rt-753-pr/...is a build
tree — the durable answer is to say which commit is validated and let the stamp check enforce it). - Verify the repack against its source.
scripts/repack_gguf.pycould decode the first tile of one tensor and compare it
with the GGUF it came from — the check that finds this class without trusting any commit:tools/q4nx_diff.py'sdecode
subcommand already does exactly this comparison (--layout flm|ours), and this lane measured 0.0325 vs 0.9994 with it.
A--verify-sourcestep there would have refused the bad container beforeservecached it.
Reproducer
~/lb/repack-converter-sweep.sh (pi-onebp lane): for each converter checkout on the box, clears
~/.cache/1bit/q4nx/<model>, repacks Qwen3-0.6B through 1bit serve -m <gguf> --device npu, posts one prompt, and prints the
answer next to llama.cpp's on the same GGUF. One ticket, ~10 min. The container cache is the trap as well: a stale good
container hides a bad converter and vice versa, so the script clears it per arm.
Found while fixing #1097 (padded weight grids, PR #1107); deliberately kept out of that PR — it is a different failure with a
different owner, and #1097's guard ("read only a grid that dominates the model's, else keep failing") already opts for
refusing over decoding garbage.
Activity
bong-water-water-bong commentedon Nov 5, 2026
Measured the same two converters today from the other end (silent-noise container), and this is the repack-side half of a
defect whose engine-side half is #1110. Numbers, because they decide what the guard can be:
--layout ours--layout flm~/wt/npu-rt-753-pr/third_party/FLM_Q4NX_Converter~/amd-oss/FLM_Q4NX_Converter@d1d5232(two tensors:
blk.14.attn_outputQ4_K andblk.0.ffn_downQ6_K. Requant scale ≈ 2-3e-2; noise ≈ 1e+0.)What that means for a guard here: there is no single "the flm order" to detect —
d1d5232's output mixes orders by sourcequant type, and one of its tensors is only correct with unsigned nibbles. So a guard in
scripts/repack_gguf.pyshould not tryto identify the converter; it should verify the container it just wrote against the GGUF it came from, which is exactly what
tools/q4nx_diff.py decode --layout oursalready does and which separates these two containers by ~2 orders of magnitude(2.9e-2 vs 5.5e0). Concretely, after the converter returns:
and the repack fails (or warns loudly, your call) on rc 1, naming
repack-stamp.txt'sconverter_commit. That closes the holethe moment the wrong checkout is used, on any model, without a table of converter versions — and it is device-free (the same
host arm your rows use).
Two caveats worth stating before someone implements it:
--tensorscoping matters: a padded container (IM 8960 → 9216) has tensors the checker cannot read until q4nx-diff: read a container whose weight is padded to a wider tile grid (part of #1110) #1119 lands, soscope the verify to non-MLP tensors (
attn*) for now — those already separate the two converters by ~200x.--skip-verifyescape for the tight loops thepayload builds run in.
Engine-side, #1110/#1130 makes a container declare its tile order and refuses an unknown one — but that only helps containers
from a converter that declares an id, so it does not replace this guard: the wrong checkout writes no id and would simply be
trusted.
Not taking it tonight — three of my PRs are open (#1119, #1130, #1133) and stacking a fourth unreviewed one is what this lane
got wrong earlier in the week. Say the word and it is first up, or take it: the measurement and the checker are already there.
repack: the container is verified against the GGUF it came from, not …
docs/npu: --verify-source and --skip-verify are the names a reader ne…
repack: the container is verified against the GGUF it came from, unde…
bong-water-water-bong commentedon Nov 6, 2026
Resolved by #1234 (merged
aed110c1): the repack/verify path this issue's last comment asks for is now the contract.scripts/repack_gguf.py --verify-source <dir> <model.gguf>(reusableverify_source()) decodes the container against the GGUF it came from under both tile layouts — the source comparison, not a table of converter commits. Exit 0 engine-1, 1 another order or none, 2 nothing comparable.corr +0.0325for the wrong checkout,+0.9994honest), and the refusal names the converterpath @ commit.--skip-verifyis the tight-loop escape and stampstile_order=unverified, neverengine-1.--check-stampinstead ofcurrent, so a rejected container cannot be served after the refusal.tests/repack_verify_source_test.py(25 assertions, wired intoci.yml), plusq4nx_diff --self-test,q4nx_diff_grid,repack_tile_order,tile_check_no_shared_tensor,npu_plan_tile_order.The issue's option 2 (name a validated converter commit in
docs/npu.mdand the resolver) is deliberately not done: verifying the container against its source removes the need to trust any commit list. Closing.