You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
npu: the pb16i control route emits garbage (original sidecars) or empty text (repacked sidecars) while the t2s route on the same container is coherent #1605
Two routes over one container, same source GGUF: the t2s route answers coherently, the pb16i route does not — with either of two different
image sets. Measured today on this box, one npu-hold ticket, amdxdna errors 0:
arm
weight images
text
decode
t2s v2 (bonsai2-27b-t2s2)
bonsai2-t2s-v2/*.t2s
coherent (" Paris.", " the invention of movable type in China during the 11")
the original 2026-10-31 set (bonsai2-pb16-sidecars)
garbage (' basводаILONイナーouroLOPTInterpolatorchnikerrated_locatorukeogia', and every prompt different)
543 ms/token
Both model dirs symlink model.q4nx → ~/models/npu/bonsai2-27b-NPU/model.q4nx (16,832,423,048 B), and both config.jsons name gguf: bonsai2-unrot-q4_0.gguf. The pb16i side is otherwise intact: kernels-m16/ has all 20 ELFs including the *_pb16i_m16 set, and pb16i_sealed.elf (425,112 B) resolves.
Why this matters
bonsai2-27b-pb16i is the control for the t2s route (engine#802's model-level row, and the 410 ms/token row it was compared against). With the
control reading garbage, no t2s-vs-pb16i text comparison can be made on this box — my gate (b) run had to report FAILED for a reason that says
nothing about the t2s path, which is what it was supposed to test.
Two candidate causes, both checkable
The container/sidecar pairing. The pb16i arm's config.json declares q4nx_weight_layout = pb16i, q4nx_weight_layout_version = 4, q4nx_pb16i_dir = ~/models/npu/bonsai2-pb16i-ternary — but that directory did not exist until I made it, and the container both dirs share is the
one the t2s route reads coherently. If the pre-11-02 pb16i container is gone, the pb16i conventions (e_ref windowing, v4 image layout) are being
applied to a container packed for a different one; garbage logits, or an immediate-EOS empty completion, are both consistent with that.
tools/q4nx_pb16i.py pack does not reproduce its own historical output, so "just repack the sidecar" is not a fix either: blk_6_ffn_up_weight.pb16i is 50,135,040 B in both, but md5 ae8248dabca6… (2026-10-31 21:40, original) vs f6875e38439f… (today, tool 88715dea). Same size, different bytes — a convention difference inside the tile, not a layout change.
What I did not do
I did not touch the pb16i model dir's config.json, its plan.json (npu_plan.py 1.1) or its kernels; the only change on disk is that ~/models/npu/bonsai2-pb16i-ternary is now a symlink to the original ~/models/npu/bonsai2-pb16-sidecars (832 files, 13,059 MB) instead of absent.
My pack is parked at ~/models/npu/bonsai2-pb16i-ternary.rebuild-mine for the md5 diff above and can be deleted.
Ask
Someone who knows which pack produced the pre-11-02 pb16i container: is the intended control still a pb16icontainer (~16.8 GB repack), or has
the control moved to something cheaper (host-GEMM on the same container)? The kernel-level identity control is unaffected and still passes
(tests/q4nx_t2s_test.py, 3/3 seeds byte-exact) — this issue is only about the whole-model text comparison.
Host-side diagnosis (no device, no NPU ticket) — the two failing arms fail structurally, and in different ways
I read this while looking for an unclaimed issue, and the two arms in your table separate cleanly from the files alone. Two facts, both reproducible in seconds, both on the box as I found it. I am deliberately not claiming this issue: the pb16i format and its kernels are private, and #1612 is already open in the same reader — but these numbers should save the owner a session.
Arm 3 (original 10-31 set → garbage): its manifest.json covers 14 of 416 tensors
$ ls ~/models/npu/bonsai2-pb16-sidecars/*.pb16i | wc -l → 416
$ python3 -c "import json;m=json.load(open('.../manifest.json'));print(len(m))" → 14
entries=14 blocks=6..7 missing_blocks=[0,1,2,3,4,5,8..13, …]
blobs_without_entry = 402 (e.g. blk_0_attn_gate_weight, blk_0_attn_qkv_weight — the first layers)
The .pb16i blobs themselves are untouched: all 416 still carry 10-31 21:40-21:42 mtimes. So the set is intact and the manifest is not — it describes blocks 6-7 only, with {K, N, tiles} per tensor. Whether the loader then skips the other 402, defaults their geometry, or misreads them is the next question, but note that no arm in the table could have caught this: nothing checks that the manifest covers the blobs the loader opens, so a 402-tensor hole loads silently.
Arm 2 (fresh pack → empty text, 1252 dispatches, tokens=0): same geometry, every byte different
Comparing the fresh pack (bonsai2-pb16i-ternary.rebuild-mine, 22:09) against the original set, per tensor:
tensor
size (both)
md5
blk_0_attn_gate_weight
17 694 720 B
orig 9c5ff0c42179 vs rebuild c79ec19d507f
blk_0_attn_qkv_weight
29 491 200 B
orig 105b11c40268 vs rebuild d7224b3423ad
blk_27_attn_q_h1_weight
17 694 720 B
orig 024b8b67ec21 vs rebuild 3d0b6b44d7f3
blk_63_ffn_up_weight
50 135 040 B
orig b751568ca3b2 vs rebuild 8ef183c9c82b
…and the same for the .ref companions and the manifest itself. Identical sizes, different content: the packer's convention changed (rotation/sign/scale order, one of those), and the format carries nothing that would distinguish the two. This is why the two arms disagree in character rather than degree — a complete-manifest-but-foreign-convention set (empty text, all-zero or NaN logits → forced EOS at tokens=0) versus a right-convention set with a 402-tensor manifest hole (garbage). They are one class of defect — silent acceptance — seen at two different levels.
The format already ships the data to check this, and the route does not use it
.ref is exactly 2*N + 1 bytes for 14/14 manifest entries I checked (12 289 = 2×6 144 + 1 for gate N=6144; 20 481 = 2×10 240 + 1 for qkv N=10 240) — i.e. one leading byte and then one fp16 value per output column. That is a per-column reference the loader could compare against the loaded blob, next to the manifest's K/N/tiles.
So the check that follows from this, and it is the one I would write if the format were mine:
manifest coverage — refuse when a blob the loader opens has no manifest entry (and name the tensor), and
content identity — the packer records something in the manifest that a foreign-convention set fails (a per-tensor digest, or reuse the .ref columns), so a re-pack with a changed convention cannot load silently.
Both are refusals with a named tensor, in the spirit of #1612 but on the load path rather than the test's skip path — which is why I am leaving them to whoever owns the pb16i route rather than opening a competing PR in a private-format area.
Independent confirmation from a different direction: on this route a request does not merely return bad text, it never returns.
I was trying to measure prefill throughput on bonsai2-27b-NPU (the pb16i arm of this issue) as the non-folded comparison for #1416/#1421, using 1bit forward-serve + NPU_PHASE_PROF=1 under npu-hold. Two kernel dirs, same model dir:
arm
kernels
server ready
8-token warm-up request
bonsai2-27b-NPU
kernels/
not ready in 90 s; last line decode family: 7 role(s) at M=16, 0 on M=128
http 000 (server never came up)
bonsai2-27b-NPU
kernels-stock/
51 s
http 000 after 420 s, ticket expired at 544 s
And the same script, same ticket size, same box, same ELF geometry, on the t2s arm (bonsai2-t2s2-perrole876): server ready in 61 s, a 130-token request completed inside 240 s with [prof] pre rows (pack=0.12 ms/launch warm, wait=3.2 ms).
So the pb16i route's failure is reachable through forward-serve too, and it presents as a stall rather than wrong tokens: the request is accepted ([prefix] slot=-1 cached=0 of 130 in the log) and never produces a first token. That is consistent with this issue's "1,252 decode dispatches, tokens=0" arm — the work is being done and the output is not coming out — and it is a third sighting rather than a new bug.
Two details that may help whoever picks this up:
decode family: 7 role(s) at M=16, 0 on M=128 appeared when --kernels …/kernels was passed, and that server then never reached "ready". The same pack with --kernels …/kernels-stock loads both families and comes up in 51 s. So one arm has a kernel-selection difference, and it is worth printing the loaded family count in the ready line rather than only in a debug log.
The stall survived a warm-up attempt, so it is not first-touch cost: 420 s for 8 tokens is ~50 s/token, orders away from the ~3 ms/launch device time the t2s arm shows on the same geometry.
I am not adding a theory of the cause here — the two arms in this issue already bracket it (repacked images → empty; original images → garbage), and my data only shows the same failure from forward-serve with timings attached. If a fix lands, the row I owe #1421 is the non-folded prefill throughput; until then that comparison is not measurable, which is itself worth knowing before anyone spends a cycle on it.
Correction to my comment above: the 0 on M=128 line is not a symptom, and I misread it.
I went to the printing site (q4nx_forward.cpp:1346):
fprintf(stderr, " [forward] decode family: %d role(s) at M=%d, %d on M=%d%s\n",
made, Q4NX_DESIGN_M_DECODE, skipped, Q4NX_DESIGN_M_PREFILL, ...);
So 7 role(s) at M=16, 0 on M=128 means seven decode twins were made and zero roles were skipped — i.e. every role got its M=16 twin, and nothing fell back to M=128. That is a success line, which I quoted as if it were a failure. Struck from the list of clues; it points at nothing.
What still stands from my run, with the misleading line removed:
bonsai2-27b-NPU + kernels-stock/: the server is ready in 51 s and an 8-token request does not return in 420 s. That is the observation, and it is independent of any log-reading;
the kernels/ arm was only "not ready within my 90 s wait", which is a bound on my patience, not on the engine — the stock dir alone needs 51 s to load, and kernels/ holds 14 ELFs rather than 7, so two minutes would not have been surprising. I have no evidence of a loading defect and should not have implied one;
the comparison that holds regardless: the t2s arm on the same box, same geometry, same script completes a 130-token request inside 240 s.
So the useful part of my earlier comment is the reproduction (a small request on the pb16i arm returns nothing) plus the contrast with t2s. The kernel-selection theory is withdrawn.
The 50 s/token figure also needs care: it is a bound from a timeout, not a measurement — the request may have been waiting on something other than compute (a stall, a retry, a queue). If someone picks this up, the first useful instrument is the profiler rows rather than wall time, because a wall-time timeout cannot distinguish "slow" from "stuck".
The stall is not stuck — it is launching, and each launch costs 100–1000× what the folded route pays. Profiler rows at last, from the fix in #1619.
Until #1619 the profiler could not dump for a prefill-only run (prof_dec_n_ && gated it), so this arm printed nothing and "no rows" was unreadable. With the guard gone, the same 8-token request that returns nothing in 150 s shows:
pre Z 128 13 launches pack=129.58 quant=20.03 wait=11.676 rb=0.006 ms/launch=161.29
pre QKV 128 5 launches pack=222.21 quant= 8.29 wait=26.154 rb=0.009 ms/launch=256.66
pre O 128 17 launches pack= 91.18 quant= 2.62 wait=11.698 rb=0.006 ms/launch=105.50
pre G 128 17 launches pack=246.66 quant= 3.18 wait=29.346 rb=0.011 ms/launch=279.19
pre U 128 18 launches pack=238.08 quant=12.89 wait=29.399 rb=0.011 ms/launch=280.37
pre D 128 18 launches pack=299.90 quant= 1.48 wait=28.910 rb=0.007 ms/launch=330.30
Three things this settles:
Work is happening. 13–18 launches per role in the first window, so the request is not blocked before dispatch — the earlier "no profiler rows" told us nothing, exactly as I flagged, and now it says the opposite.
The cost is host-side and enormous: pack 91–300 ms per launch against wait 12–29 ms. On the folded t2s2 pack, same model, same ELF geometry, the matching numbers are pack 0.10–0.30 ms and wait 2.9–3.8 ms. That is a 300–1000× gap in the host term on identical shapes, and it is the reason an 8-token request cannot finish: 112 launches × ~250 ms ≈ 28 s per 128-row block, several blocks, plus whatever the decode side does — a request measured in minutes.
quant is also non-trivial here (20 ms for Z, 12.9 ms for U) where the folded route shows 0.1–0.2 ms, so the activation path differs too, not only the weight side.
I have no theory of the cause to offer that the code does not support, and the two arms of this issue already point at the images rather than the dispatcher. But this changes the shape of the bug: it is not "the pb16i route emits the wrong tokens", it is "the pb16i route's host per-launch work is 300–1000× the folded route's on identical geometry, so it cannot complete" — and the wrong/empty tokens reported in the opening table may well be a consequence (a partially warmed or partially dispatched pass) rather than a separate defect.
Reproduction: npu-hold -t 9 -- ./run_stall.sh <1bit binary> with PACK=~/models/npu/bonsai2-27b-NPU KERNELS=$PACK/kernels-stock; it starts forward-serve on 18099 (never :18097), sends 8 then 130 tokens with a 150 s timeout, prints the profiler rows, and kills the server on every exit path.
Refinement after inspecting the container itself — this narrows the issue and retracts one assumption in my own report above.
The container is self-contained, so the sidecars are an override, not the only source
model.q4nx's own tensor table (99,456-byte header) holds 852 tensors: 498 I8, 257 BF16, 96 F32, under HF names — model.layers.N.mlp.gate_proj.weight, …linear_attn.qkv_proj.weight, …self_attn.gate_proj.weight, plus BF16 norms and ssm_a/bias. There are no blk.* keys (the sidecar naming). So the container carries its own int8-packed projections, and the q4nx_pb16i/q4nx_t2s sidecars replace them when config.json declares that layout. That explains why the container is shared by both model dirs without conflict.
My "the sidecars fail verify" measurement used the wrong oracle
Both config.jsons name the same source: q4nx_source.gguf = bonsai2-unrot-q4_0.gguf. I verified the Oct-31 sidecars against Ternary-Bonsai-2-27B-PTQ1_0.gguf, which is a different GGUF (5.95 GB ternary codes vs the 15.18 GB q4_0 the container was built from), so
"MISMATCH 14/14" says the two files disagree — not that the sidecars are wrong. The same applies to my own rebuild: it verified against the PTQ1_0
GGUF by construction, which is why pack self-checked ok while the engine served empty text from it.
So of the three arms in the original table, only one conclusion survives intact: the Oct-31 pb16i images produce garbage while the t2s images derived
from that same directory produce coherent text — that asymmetry is the actual defect, and it is not explained by provenance.
What that leaves, stated as the question to answer next
Are the Oct-31 .pb16i images consistent with bonsai2-unrot-q4_0.gguf (their declared source)? tools/q4nx_pb16i.py verify reads PTQ1_0 (ggml type
143) only, so it cannot answer this; a q4_0 → codes/dbits dequant followed by the same verify_image comparison would.
If they are consistent, the defect is in the reader path, not the data — and the cheapest scope test is whether any pb16i pack serves correctly on
this build (none of the other model dirs on this box declares q4nx_weight_layout = pb16i, so there is currently nothing to compare against).
I am not going to change the images, the reader or the config on the strength of the retracted reading; the corrected evidence is recorded here so the next
person starts from it rather than from my earlier claim.
Measured follow-up: the sidecar weights are not derivable from either GGUF on this box by any transform I tried — and the converter's role map has been
checked, so this is not a naming mistake.
What was tested, and against what
The converter (/home/bcloud/wt/bonsai-speed/conv/qwen35_to_q4nx.py) maps roles per layer type: FULL = {"attn_k": "self_attn.k_proj", …} when l % 4 == 3, and GDN = {"attn_qkv": "linear_attn.qkv_proj", …} (a fused [q|k|v], line 265) otherwise. qwen35.full_attention_interval = 4 is confirmed
in the GGUF metadata, so layer 3 is full-attention and blk.3.attn_k.weight ← self_attn.k_proj — i.e. the tensor pair I compared is the right pair.
hadamard_signs.bin is also decoded now, and it is a proper table, not noise:
i.e. one sign per k index, per K size — the diagonal of the block-diagonal Hadamard.
Results (all against t2s sidecar blk_3_attn_k_weight, decoded with the repo's own tested unpacker)
comparison with the sidecar weights
result
q4_0 GGUF (bonsai2-unrot-q4_0.gguf), unrotated
pearson r = 0.0004
q4_0 under H@diag(s)@W, diag(s)@H@W, H@W (block 1024)
r = 0.0155 / 0.0007 / 0.0006
q4_0, other dims orientation; transform along N
r ≤ 0.0006
best single row/column alignment over 12
|r| ≤ 0.068
PTQ1_0 ternary codes vs sidecar codes, same positions
34.36 % identical — chance for ~⅓ zeros
PTQ1_0 float weights rotated, then sign-quantised vs sidecar codes
50.00 % / 50.02 % / 49.97 % — chance for a sign
So: not the q4_0 values, not those values under the documented Hadamard, and not the ternary codes — every test lands on its chance level. The 34 % is
exactly what one gets when ~⅓ of ternary codes are zero and the rest agree half the time, and 50.0 % is what one gets for signs; neither is a weak signal, they
are nulls.
The unknown, stated precisely
The rotation convention is the missing piece. Either the transform is not FWHT over 1024-row blocks along K (radix-4 / butterfly variant, a different
block size, or applied along N), or the sidecars were built from a rotated export that is not on this box (both GGUFs here are unrot/PTQ1_0, and
neither is a rotated checkpoint). Reproducing the pack from anything on this machine therefore cannot be assumed to yield the frozen images — which is
consistent with q4nx_pb16i.py pack differing from the set on all 8 layer-3 tensors at identical sizes.
I am stopping here rather than trying more conventions blind: three "wrong oracle" readings on this issue have already looked like defects in the artifact, and
the next test needs the rotated export or the original pack command, not another guess.
A PR now exists for the part of this that was actionable — #1683 — and this comment says exactly what is still blocked, as the owner asked.
What #1683 fixes: the wall every attempt at answering this issue hit. tools/q4nx_pb16i.py's gguf_tensors sized a tensor as n // QK * BLOCK_BYTES if tt == PTQ1_0 else 0, so every other ggml type came back as b"" — a caller cannot tell "this tensor is empty" from "this
reader cannot size it", and for a check against a declared source those are different answers. The PR adds Q4_0 to the sized table, a read_q4_0 that returns
values plus per-32-row scales, and a strict=True refusal that names the tensor and the type. With it, the comparison I had to do in a scratch script
becomes a repo command, with a test whose decode assertion is proven non-vacuous (inverting it fails the suite).
What is still blocked, and on what, precisely:
The sidecars do not correlate with their declared source, and I cannot close that from anything on this box. Measured: q4_0 values r = 0.0004; under H@diag(s)@W, diag(s)@H@W, H@W (block 1024, signs decoded from hadamard_signs.bin) r ≤ 0.016; other dims orientation and a transform along N r ≤
0.0007; best single row/column |r| ≤ 0.068. Against the ternary source the codes agree 34.36 % (chance for ⅓ zeros) and rotated-then-quantised signs
agree 50.00/50.02/49.97 % (chance). hadamard_signs.bin is decoded (HAD1, n=3, (5120,0)(6144,5120)(17408,11264), 28,672 ±1 signs = 5120+6144+17408).
So the open question is the rotation convention, or a rotated export that is not here. Both GGUFs on this box are unrot/PTQ1_0; neither is a rotated
checkpoint. I need one of: the rotated export the sidecars were built from, the original pack command, or the qwen35_to_q4nx.py invocation that produced
them — and the converter is at ~/wt/bonsai-speed/conv/qwen35_to_q4nx.py with its role map verified (FULL/GDN, full_attention_interval = 4 confirmed
in the GGUF metadata, so layer 3 really is full-attention and the tensor pairing is right).
The garbage itself is therefore unexplained rather than fixed. What is established and reproducible: the frozen .pb16i images give garbage tokens
while the .t2s images derived from the same directory give coherent text, on the same container and binary — so the defect is in the pb16i read path or
its composition, and not in the data's provenance.
If the owner or another lane can supply (2), this issue becomes a measurement; without it, any further step is a third guess at a convention, and I have
already spent two wrong-oracle readings on this one (wrong GGUF, then unrotated basis) that each looked like a defect in the artifact.
The measurement is now a repo command, not a scratch file.#1683 merged (235126a5), so tools/q4nx_pb16i.py can read a q4_0 source; this reproduces the
sidecar-vs-source comparison on main as merged, with no --hardware and no hand-rolled GGUF parsing:
Same number my scratch script produced, now rerunnable by anyone — which was the point of the PR.
Two findings worth carrying forward from doing it this way:
strict=True is all-or-nothing per file. It refuses the whole GGUF at the first unsized tensor (output_norm.weight, ggml type 0), so a caller that
wants one q4_0 tensor out of a real model cannot use it. That is correct as a "prove this file is fully readable" mode and wrong as a "read this tensor"
mode; a per-tensor opt-out (strict=NAME) or a size_of(name, dims, type) helper would cover the second. Not a defect, just the shape the next caller will
want.
read_q4_0's scales come back (K/32, N) — same granularity as PTQ1_0's — which is what makes an element-for-element comparison possible. Worth knowing
for the next person who tries this: if a sidecar were built from a q4_0 source read straight, r would be ~1, and it is 0.0004.
So the block on this issue is unchanged and is not a tooling block: the sidecar does not correspond to bonsai2-unrot-q4_0.gguf's same-named tensor under any
transform I can apply (unrotated r = 0.0004; H@diag(s)@W, diag(s)@H@W, H@W at block 1024 with signs decoded from hadamard_signs.bin ≤ 0.016; other dims
orientation and a transform along N ≤ 0.0007; ternary codes 34.36 % and rotated-then-quantised signs 50.00/50.02/49.97 %, both chance). What is missing is the rotation convention, the rotated export the sidecar was built from, or the original pack command — both GGUFs on this box are unrot/PTQ1_0.
Activity
bong-water-water-bong commentedon Nov 9, 2026
Host-side diagnosis (no device, no NPU ticket) — the two failing arms fail structurally, and in different ways
I read this while looking for an unclaimed issue, and the two arms in your table separate cleanly from the files alone. Two facts, both reproducible in seconds, both on the box as I found it. I am deliberately not claiming this issue: the pb16i format and its kernels are private, and #1612 is already open in the same reader — but these numbers should save the owner a session.
Arm 3 (original 10-31 set → garbage): its
manifest.jsoncovers 14 of 416 tensorsThe
.pb16iblobs themselves are untouched: all 416 still carry 10-31 21:40-21:42 mtimes. So the set is intact and the manifest is not — it describes blocks 6-7 only, with{K, N, tiles}per tensor. Whether the loader then skips the other 402, defaults their geometry, or misreads them is the next question, but note that no arm in the table could have caught this: nothing checks that the manifest covers the blobs the loader opens, so a 402-tensor hole loads silently.Arm 2 (fresh pack → empty text, 1252 dispatches,
tokens=0): same geometry, every byte differentComparing the fresh pack (
bonsai2-pb16i-ternary.rebuild-mine, 22:09) against the original set, per tensor:blk_0_attn_gate_weight9c5ff0c42179vs rebuildc79ec19d507fblk_0_attn_qkv_weight105b11c40268vs rebuildd7224b3423adblk_27_attn_q_h1_weight024b8b67ec21vs rebuild3d0b6b44d7f3blk_63_ffn_up_weightb751568ca3b2vs rebuild8ef183c9c82b…and the same for the
.refcompanions and the manifest itself. Identical sizes, different content: the packer's convention changed (rotation/sign/scale order, one of those), and the format carries nothing that would distinguish the two. This is why the two arms disagree in character rather than degree — a complete-manifest-but-foreign-convention set (empty text, all-zero or NaN logits → forced EOS attokens=0) versus a right-convention set with a 402-tensor manifest hole (garbage). They are one class of defect — silent acceptance — seen at two different levels.The format already ships the data to check this, and the route does not use it
.refis exactly2*N + 1bytes for 14/14 manifest entries I checked (12 289 = 2×6 144 + 1 forgateN=6144; 20 481 = 2×10 240 + 1 forqkvN=10 240) — i.e. one leading byte and then one fp16 value per output column. That is a per-column reference the loader could compare against the loaded blob, next to the manifest'sK/N/tiles.So the check that follows from this, and it is the one I would write if the format were mine:
.refcolumns), so a re-pack with a changed convention cannot load silently.Both are refusals with a named tensor, in the spirit of #1612 but on the load path rather than the test's skip path — which is why I am leaving them to whoever owns the pb16i route rather than opening a competing PR in a private-format area.
bong-water-water-bong commentedon Nov 9, 2026
Independent confirmation from a different direction: on this route a request does not merely return bad text, it never returns.
I was trying to measure prefill throughput on
bonsai2-27b-NPU(the pb16i arm of this issue) as the non-folded comparison for #1416/#1421, using1bit forward-serve+NPU_PHASE_PROF=1undernpu-hold. Two kernel dirs, same model dir:bonsai2-27b-NPUkernels/decode family: 7 role(s) at M=16, 0 on M=128bonsai2-27b-NPUkernels-stock/And the same script, same ticket size, same box, same ELF geometry, on the t2s arm (
bonsai2-t2s2-perrole876): server ready in 61 s, a 130-token request completed inside 240 s with[prof] prerows (pack=0.12 ms/launch warm,wait=3.2 ms).So the pb16i route's failure is reachable through
forward-servetoo, and it presents as a stall rather than wrong tokens: the request is accepted ([prefix] slot=-1 cached=0 of 130in the log) and never produces a first token. That is consistent with this issue's "1,252 decode dispatches,tokens=0" arm — the work is being done and the output is not coming out — and it is a third sighting rather than a new bug.Two details that may help whoever picks this up:
decode family: 7 role(s) at M=16, 0 on M=128appeared when--kernels …/kernelswas passed, and that server then never reached "ready". The same pack with--kernels …/kernels-stockloads both families and comes up in 51 s. So one arm has a kernel-selection difference, and it is worth printing the loaded family count in the ready line rather than only in a debug log.I am not adding a theory of the cause here — the two arms in this issue already bracket it (repacked images → empty; original images → garbage), and my data only shows the same failure from
forward-servewith timings attached. If a fix lands, the row I owe #1421 is the non-folded prefill throughput; until then that comparison is not measurable, which is itself worth knowing before anyone spends a cycle on it.bong-water-water-bong commentedon Nov 9, 2026
Correction to my comment above: the
0 on M=128line is not a symptom, and I misread it.I went to the printing site (
q4nx_forward.cpp:1346):So
7 role(s) at M=16, 0 on M=128means seven decode twins were made and zero roles were skipped — i.e. every role got its M=16 twin, and nothing fell back to M=128. That is a success line, which I quoted as if it were a failure. Struck from the list of clues; it points at nothing.What still stands from my run, with the misleading line removed:
bonsai2-27b-NPU+kernels-stock/: the server is ready in 51 s and an 8-token request does not return in 420 s. That is the observation, and it is independent of any log-reading;kernels/arm was only "not ready within my 90 s wait", which is a bound on my patience, not on the engine — the stock dir alone needs 51 s to load, andkernels/holds 14 ELFs rather than 7, so two minutes would not have been surprising. I have no evidence of a loading defect and should not have implied one;So the useful part of my earlier comment is the reproduction (a small request on the pb16i arm returns nothing) plus the contrast with t2s. The kernel-selection theory is withdrawn.
The 50 s/token figure also needs care: it is a bound from a timeout, not a measurement — the request may have been waiting on something other than compute (a stall, a retry, a queue). If someone picks this up, the first useful instrument is the profiler rows rather than wall time, because a wall-time timeout cannot distinguish "slow" from "stuck".
bong-water-water-bong commentedon Nov 9, 2026
The stall is not stuck — it is launching, and each launch costs 100–1000× what the folded route pays. Profiler rows at last, from the fix in #1619.
Until #1619 the profiler could not dump for a prefill-only run (
prof_dec_n_ &&gated it), so this arm printed nothing and "no rows" was unreadable. With the guard gone, the same 8-token request that returns nothing in 150 s shows:Three things this settles:
pack91–300 ms per launch againstwait12–29 ms. On the folded t2s2 pack, same model, same ELF geometry, the matching numbers arepack0.10–0.30 ms andwait2.9–3.8 ms. That is a 300–1000× gap in the host term on identical shapes, and it is the reason an 8-token request cannot finish: 112 launches × ~250 ms ≈ 28 s per 128-row block, several blocks, plus whatever the decode side does — a request measured in minutes.quantis also non-trivial here (20 ms for Z, 12.9 ms for U) where the folded route shows 0.1–0.2 ms, so the activation path differs too, not only the weight side.I have no theory of the cause to offer that the code does not support, and the two arms of this issue already point at the images rather than the dispatcher. But this changes the shape of the bug: it is not "the pb16i route emits the wrong tokens", it is "the pb16i route's host per-launch work is 300–1000× the folded route's on identical geometry, so it cannot complete" — and the wrong/empty tokens reported in the opening table may well be a consequence (a partially warmed or partially dispatched pass) rather than a separate defect.
Reproduction:
npu-hold -t 9 -- ./run_stall.sh <1bit binary>withPACK=~/models/npu/bonsai2-27b-NPU KERNELS=$PACK/kernels-stock; it startsforward-serveon 18099 (never :18097), sends 8 then 130 tokens with a 150 s timeout, prints the profiler rows, and kills the server on every exit path.bong-water-water-bong commentedon Nov 9, 2026
Refinement after inspecting the container itself — this narrows the issue and retracts one assumption in my own report above.
The container is self-contained, so the sidecars are an override, not the only source
model.q4nx's own tensor table (99,456-byte header) holds 852 tensors: 498I8, 257BF16, 96F32, under HF names —model.layers.N.mlp.gate_proj.weight,…linear_attn.qkv_proj.weight,…self_attn.gate_proj.weight, plus BF16 norms andssm_a/bias. There are noblk.*keys (the sidecar naming). So the container carries its own int8-packed projections, and theq4nx_pb16i/q4nx_t2ssidecars replace them whenconfig.jsondeclares that layout. That explains why the container is shared by both model dirs without conflict.My "the sidecars fail verify" measurement used the wrong oracle
Both
config.jsons name the same source:q4nx_source.gguf = bonsai2-unrot-q4_0.gguf. I verified the Oct-31 sidecars againstTernary-Bonsai-2-27B-PTQ1_0.gguf, which is a different GGUF (5.95 GB ternary codes vs the 15.18 GBq4_0the container was built from), so"MISMATCH 14/14" says the two files disagree — not that the sidecars are wrong. The same applies to my own rebuild: it verified against the PTQ1_0
GGUF by construction, which is why
packself-checkedokwhile the engine served empty text from it.So of the three arms in the original table, only one conclusion survives intact: the Oct-31 pb16i images produce garbage while the t2s images derived
from that same directory produce coherent text — that asymmetry is the actual defect, and it is not explained by provenance.
What that leaves, stated as the question to answer next
.pb16iimages consistent withbonsai2-unrot-q4_0.gguf(their declared source)?tools/q4nx_pb16i.py verifyreads PTQ1_0 (ggml type143) only, so it cannot answer this; a q4_0 → codes/dbits dequant followed by the same
verify_imagecomparison would.this build (none of the other model dirs on this box declares
q4nx_weight_layout = pb16i, so there is currently nothing to compare against).I am not going to change the images, the reader or the config on the strength of the retracted reading; the corrected evidence is recorded here so the next
person starts from it rather than from my earlier claim.
bong-water-water-bong commentedon Nov 9, 2026
Measured follow-up: the sidecar weights are not derivable from either GGUF on this box by any transform I tried — and the converter's role map has been
checked, so this is not a naming mistake.
What was tested, and against what
The converter (
/home/bcloud/wt/bonsai-speed/conv/qwen35_to_q4nx.py) maps roles per layer type:FULL = {"attn_k": "self_attn.k_proj", …}whenl % 4 == 3, andGDN = {"attn_qkv": "linear_attn.qkv_proj", …}(a fused[q|k|v], line 265) otherwise.qwen35.full_attention_interval = 4is confirmedin the GGUF metadata, so layer 3 is full-attention and
blk.3.attn_k.weight←self_attn.k_proj— i.e. the tensor pair I compared is the right pair.hadamard_signs.binis also decoded now, and it is a proper table, not noise:i.e. one sign per k index, per K size — the diagonal of the block-diagonal Hadamard.
Results (all against
t2ssidecarblk_3_attn_k_weight, decoded with the repo's own tested unpacker)q4_0GGUF (bonsai2-unrot-q4_0.gguf), unrotatedq4_0underH@diag(s)@W,diag(s)@H@W,H@W(block 1024)q4_0, other dims orientation; transform along NSo: not the q4_0 values, not those values under the documented Hadamard, and not the ternary codes — every test lands on its chance level. The 34 % is
exactly what one gets when ~⅓ of ternary codes are zero and the rest agree half the time, and 50.0 % is what one gets for signs; neither is a weak signal, they
are nulls.
The unknown, stated precisely
The rotation convention is the missing piece. Either the transform is not
FWHT over 1024-row blocks along K(radix-4 / butterfly variant, a differentblock size, or applied along N), or the sidecars were built from a rotated export that is not on this box (both GGUFs here are
unrot/PTQ1_0, andneither is a rotated checkpoint). Reproducing the pack from anything on this machine therefore cannot be assumed to yield the frozen images — which is
consistent with
q4nx_pb16i.py packdiffering from the set on all 8 layer-3 tensors at identical sizes.I am stopping here rather than trying more conventions blind: three "wrong oracle" readings on this issue have already looked like defects in the artifact, and
the next test needs the rotated export or the original pack command, not another guess.
bong-water-water-bong commentedon Nov 9, 2026
A PR now exists for the part of this that was actionable — #1683 — and this comment says exactly what is still blocked, as the owner asked.
What #1683 fixes: the wall every attempt at answering this issue hit.
tools/q4nx_pb16i.py'sgguf_tensorssized a tensor asn // QK * BLOCK_BYTES if tt == PTQ1_0 else 0, so every other ggml type came back asb""— a caller cannot tell "this tensor is empty" from "thisreader cannot size it", and for a check against a declared source those are different answers. The PR adds Q4_0 to the sized table, a
read_q4_0that returnsvalues plus per-32-row scales, and a
strict=Truerefusal that names the tensor and the type. With it, the comparison I had to do in a scratch scriptbecomes a repo command, with a test whose decode assertion is proven non-vacuous (inverting it fails the suite).
What is still blocked, and on what, precisely:
q4_0values r = 0.0004; underH@diag(s)@W,diag(s)@H@W,H@W(block 1024, signs decoded fromhadamard_signs.bin) r ≤ 0.016; other dims orientation and a transform along N r ≤0.0007; best single row/column |r| ≤ 0.068. Against the ternary source the codes agree 34.36 % (chance for ⅓ zeros) and rotated-then-quantised signs
agree 50.00/50.02/49.97 % (chance).
hadamard_signs.binis decoded (HAD1, n=3,(5120,0)(6144,5120)(17408,11264), 28,672 ±1 signs = 5120+6144+17408).unrot/PTQ1_0; neither is a rotatedcheckpoint. I need one of: the rotated export the sidecars were built from, the original pack command, or the
qwen35_to_q4nx.pyinvocation that producedthem — and the converter is at
~/wt/bonsai-speed/conv/qwen35_to_q4nx.pywith its role map verified (FULL/GDN,full_attention_interval = 4confirmedin the GGUF metadata, so layer 3 really is full-attention and the tensor pairing is right).
.pb16iimages give garbage tokenswhile the
.t2simages derived from the same directory give coherent text, on the same container and binary — so the defect is in the pb16i read path orits composition, and not in the data's provenance.
If the owner or another lane can supply (2), this issue becomes a measurement; without it, any further step is a third guess at a convention, and I have
already spent two wrong-oracle readings on this one (wrong GGUF, then unrotated basis) that each looked like a defect in the artifact.
bong-water-water-bong commentedon Nov 9, 2026
The measurement is now a repo command, not a scratch file. #1683 merged (
235126a5), sotools/q4nx_pb16i.pycan read a q4_0 source; this reproduces thesidecar-vs-source comparison on
mainas merged, with no--hardwareand no hand-rolled GGUF parsing:Same number my scratch script produced, now rerunnable by anyone — which was the point of the PR.
Two findings worth carrying forward from doing it this way:
strict=Trueis all-or-nothing per file. It refuses the whole GGUF at the first unsized tensor (output_norm.weight, ggml type 0), so a caller thatwants one q4_0 tensor out of a real model cannot use it. That is correct as a "prove this file is fully readable" mode and wrong as a "read this tensor"
mode; a per-tensor opt-out (
strict=NAME) or asize_of(name, dims, type)helper would cover the second. Not a defect, just the shape the next caller willwant.
read_q4_0's scales come back(K/32, N)— same granularity as PTQ1_0's — which is what makes an element-for-element comparison possible. Worth knowingfor the next person who tries this: if a sidecar were built from a q4_0 source read straight, r would be ~1, and it is 0.0004.
So the block on this issue is unchanged and is not a tooling block: the sidecar does not correspond to
bonsai2-unrot-q4_0.gguf's same-named tensor under anytransform I can apply (unrotated r = 0.0004;
H@diag(s)@W,diag(s)@H@W,H@Wat block 1024 with signs decoded fromhadamard_signs.bin≤ 0.016; other dimsorientation and a transform along N ≤ 0.0007; ternary codes 34.36 % and rotated-then-quantised signs 50.00/50.02/49.97 %, both chance). What is missing is the
rotation convention, the rotated export the sidecar was built from, or the original pack command — both GGUFs on this box are
unrot/PTQ1_0.serve: forward-serve warms the weight cache before it says ready (44.…
npu-pack: tensor(path, name) reads ONE tensor, because strict=True re…
npu: the profiler dumps for a prefill-only run (it required a decode …
npu: the residual dump says when it cannot write, instead of reportin…