Skip to content

npu: the pb16i control route emits garbage (original sidecars) or empty text (repacked sidecars) while the t2s route on the same container is coherent #1605

Closed

Description

@bong-water-water-bong

What happens

Two routes over one container, same source GGUF: the t2s route answers coherently, the pb16i route does not — with either of two different
image sets. Measured today on this box, one npu-hold ticket, amdxdna errors 0:

arm weight images text decode
t2s v2 (bonsai2-27b-t2s2) bonsai2-t2s-v2/*.t2s coherent (" Paris.", " the invention of movable type in China during the 11") 293 ms/token
pb16i (bonsai2-27b-pb16i) freshly packed with tools/q4nx_pb16i.py empty (0 chars; log shows 1,252 decode dispatches, tokens=0) —
pb16i (bonsai2-27b-pb16i) the original 2026-10-31 set (bonsai2-pb16-sidecars) garbage (' basводаILONイナーouroLOPTInterpolatorchnikerrated_locatorukeogia', and every prompt different) 543 ms/token

Both model dirs symlink model.q4nx → ~/models/npu/bonsai2-27b-NPU/model.q4nx (16,832,423,048 B), and both config.jsons name
gguf: bonsai2-unrot-q4_0.gguf. The pb16i side is otherwise intact: kernels-m16/ has all 20 ELFs including the *_pb16i_m16 set, and
pb16i_sealed.elf (425,112 B) resolves.

Why this matters

bonsai2-27b-pb16i is the control for the t2s route (engine#802's model-level row, and the 410 ms/token row it was compared against). With the
control reading garbage, no t2s-vs-pb16i text comparison can be made on this box — my gate (b) run had to report FAILED for a reason that says
nothing about the t2s path, which is what it was supposed to test.

Two candidate causes, both checkable

  1. The container/sidecar pairing. The pb16i arm's config.json declares q4nx_weight_layout = pb16i, q4nx_weight_layout_version = 4,
    q4nx_pb16i_dir = ~/models/npu/bonsai2-pb16i-ternary — but that directory did not exist until I made it, and the container both dirs share is the
    one the t2s route reads coherently. If the pre-11-02 pb16i container is gone, the pb16i conventions (e_ref windowing, v4 image layout) are being
    applied to a container packed for a different one; garbage logits, or an immediate-EOS empty completion, are both consistent with that.
  2. tools/q4nx_pb16i.py pack does not reproduce its own historical output, so "just repack the sidecar" is not a fix either:
    blk_6_ffn_up_weight.pb16i is 50,135,040 B in both, but md5 ae8248dabca6… (2026-10-31 21:40, original) vs f6875e38439f… (today, tool
    88715dea). Same size, different bytes — a convention difference inside the tile, not a layout change.

What I did not do

I did not touch the pb16i model dir's config.json, its plan.json (npu_plan.py 1.1) or its kernels; the only change on disk is that
~/models/npu/bonsai2-pb16i-ternary is now a symlink to the original ~/models/npu/bonsai2-pb16-sidecars (832 files, 13,059 MB) instead of absent.
My pack is parked at ~/models/npu/bonsai2-pb16i-ternary.rebuild-mine for the md5 diff above and can be deleted.

Ask

Someone who knows which pack produced the pre-11-02 pb16i container: is the intended control still a pb16i container (~16.8 GB repack), or has
the control moved to something cheaper (host-GEMM on the same container)? The kernel-level identity control is unaffected and still passes
(tests/q4nx_t2s_test.py, 3/3 seeds byte-exact) — this issue is only about the whole-model text comparison.

Activity

  1. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Host-side diagnosis (no device, no NPU ticket) — the two failing arms fail structurally, and in different ways

    I read this while looking for an unclaimed issue, and the two arms in your table separate cleanly from the files alone. Two facts, both reproducible in seconds, both on the box as I found it. I am deliberately not claiming this issue: the pb16i format and its kernels are private, and #1612 is already open in the same reader — but these numbers should save the owner a session.

    Arm 3 (original 10-31 set → garbage): its manifest.json covers 14 of 416 tensors

    $ ls ~/models/npu/bonsai2-pb16-sidecars/*.pb16i | wc -l          → 416
    $ python3 -c "import json;m=json.load(open('.../manifest.json'));print(len(m))"  → 14
      entries=14 blocks=6..7 missing_blocks=[0,1,2,3,4,5,8..13, …]
      blobs_without_entry = 402   (e.g. blk_0_attn_gate_weight, blk_0_attn_qkv_weight — the first layers)
    

    The .pb16i blobs themselves are untouched: all 416 still carry 10-31 21:40-21:42 mtimes. So the set is intact and the manifest is not — it describes blocks 6-7 only, with {K, N, tiles} per tensor. Whether the loader then skips the other 402, defaults their geometry, or misreads them is the next question, but note that no arm in the table could have caught this: nothing checks that the manifest covers the blobs the loader opens, so a 402-tensor hole loads silently.

    Arm 2 (fresh pack → empty text, 1252 dispatches, tokens=0): same geometry, every byte different

    Comparing the fresh pack (bonsai2-pb16i-ternary.rebuild-mine, 22:09) against the original set, per tensor:

    tensor size (both) md5
    blk_0_attn_gate_weight 17 694 720 B orig 9c5ff0c42179 vs rebuild c79ec19d507f
    blk_0_attn_qkv_weight 29 491 200 B orig 105b11c40268 vs rebuild d7224b3423ad
    blk_27_attn_q_h1_weight 17 694 720 B orig 024b8b67ec21 vs rebuild 3d0b6b44d7f3
    blk_63_ffn_up_weight 50 135 040 B orig b751568ca3b2 vs rebuild 8ef183c9c82b

    …and the same for the .ref companions and the manifest itself. Identical sizes, different content: the packer's convention changed (rotation/sign/scale order, one of those), and the format carries nothing that would distinguish the two. This is why the two arms disagree in character rather than degree — a complete-manifest-but-foreign-convention set (empty text, all-zero or NaN logits → forced EOS at tokens=0) versus a right-convention set with a 402-tensor manifest hole (garbage). They are one class of defect — silent acceptance — seen at two different levels.

    The format already ships the data to check this, and the route does not use it

    .ref is exactly 2*N + 1 bytes for 14/14 manifest entries I checked (12 289 = 2×6 144 + 1 for gate N=6144; 20 481 = 2×10 240 + 1 for qkv N=10 240) — i.e. one leading byte and then one fp16 value per output column. That is a per-column reference the loader could compare against the loaded blob, next to the manifest's K/N/tiles.

    So the check that follows from this, and it is the one I would write if the format were mine:

    1. manifest coverage — refuse when a blob the loader opens has no manifest entry (and name the tensor), and
    2. content identity — the packer records something in the manifest that a foreign-convention set fails (a per-tensor digest, or reuse the .ref columns), so a re-pack with a changed convention cannot load silently.

    Both are refusals with a named tensor, in the spirit of #1612 but on the load path rather than the test's skip path — which is why I am leaving them to whoever owns the pb16i route rather than opening a competing PR in a private-format area.

  2. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Independent confirmation from a different direction: on this route a request does not merely return bad text, it never returns.

    I was trying to measure prefill throughput on bonsai2-27b-NPU (the pb16i arm of this issue) as the non-folded comparison for #1416/#1421, using 1bit forward-serve + NPU_PHASE_PROF=1 under npu-hold. Two kernel dirs, same model dir:

    arm kernels server ready 8-token warm-up request
    bonsai2-27b-NPU kernels/ not ready in 90 s; last line decode family: 7 role(s) at M=16, 0 on M=128 http 000 (server never came up)
    bonsai2-27b-NPU kernels-stock/ 51 s http 000 after 420 s, ticket expired at 544 s

    And the same script, same ticket size, same box, same ELF geometry, on the t2s arm (bonsai2-t2s2-perrole876): server ready in 61 s, a 130-token request completed inside 240 s with [prof] pre rows (pack=0.12 ms/launch warm, wait=3.2 ms).

    So the pb16i route's failure is reachable through forward-serve too, and it presents as a stall rather than wrong tokens: the request is accepted ([prefix] slot=-1 cached=0 of 130 in the log) and never produces a first token. That is consistent with this issue's "1,252 decode dispatches, tokens=0" arm — the work is being done and the output is not coming out — and it is a third sighting rather than a new bug.

    Two details that may help whoever picks this up:

    1. decode family: 7 role(s) at M=16, 0 on M=128 appeared when --kernels …/kernels was passed, and that server then never reached "ready". The same pack with --kernels …/kernels-stock loads both families and comes up in 51 s. So one arm has a kernel-selection difference, and it is worth printing the loaded family count in the ready line rather than only in a debug log.
    2. The stall survived a warm-up attempt, so it is not first-touch cost: 420 s for 8 tokens is ~50 s/token, orders away from the ~3 ms/launch device time the t2s arm shows on the same geometry.

    I am not adding a theory of the cause here — the two arms in this issue already bracket it (repacked images → empty; original images → garbage), and my data only shows the same failure from forward-serve with timings attached. If a fix lands, the row I owe #1421 is the non-folded prefill throughput; until then that comparison is not measurable, which is itself worth knowing before anyone spends a cycle on it.

  3. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correction to my comment above: the 0 on M=128 line is not a symptom, and I misread it.

    I went to the printing site (q4nx_forward.cpp:1346):

    fprintf(stderr, "  [forward] decode family: %d role(s) at M=%d, %d on M=%d%s\n",
            made, Q4NX_DESIGN_M_DECODE, skipped, Q4NX_DESIGN_M_PREFILL, ...);

    So 7 role(s) at M=16, 0 on M=128 means seven decode twins were made and zero roles were skipped — i.e. every role got its M=16 twin, and nothing fell back to M=128. That is a success line, which I quoted as if it were a failure. Struck from the list of clues; it points at nothing.

    What still stands from my run, with the misleading line removed:

    • bonsai2-27b-NPU + kernels-stock/: the server is ready in 51 s and an 8-token request does not return in 420 s. That is the observation, and it is independent of any log-reading;
    • the kernels/ arm was only "not ready within my 90 s wait", which is a bound on my patience, not on the engine — the stock dir alone needs 51 s to load, and kernels/ holds 14 ELFs rather than 7, so two minutes would not have been surprising. I have no evidence of a loading defect and should not have implied one;
    • the comparison that holds regardless: the t2s arm on the same box, same geometry, same script completes a 130-token request inside 240 s.

    So the useful part of my earlier comment is the reproduction (a small request on the pb16i arm returns nothing) plus the contrast with t2s. The kernel-selection theory is withdrawn.

    The 50 s/token figure also needs care: it is a bound from a timeout, not a measurement — the request may have been waiting on something other than compute (a stall, a retry, a queue). If someone picks this up, the first useful instrument is the profiler rows rather than wall time, because a wall-time timeout cannot distinguish "slow" from "stuck".

  4. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The stall is not stuck — it is launching, and each launch costs 100–1000× what the folded route pays. Profiler rows at last, from the fix in #1619.

    Until #1619 the profiler could not dump for a prefill-only run (prof_dec_n_ && gated it), so this arm printed nothing and "no rows" was unreadable. With the guard gone, the same 8-token request that returns nothing in 150 s shows:

    pre Z    128   13 launches   pack=129.58  quant=20.03  wait=11.676  rb=0.006  ms/launch=161.29
    pre QKV  128    5 launches   pack=222.21  quant= 8.29  wait=26.154  rb=0.009  ms/launch=256.66
    pre O    128   17 launches   pack= 91.18  quant= 2.62  wait=11.698  rb=0.006  ms/launch=105.50
    pre G    128   17 launches   pack=246.66  quant= 3.18  wait=29.346  rb=0.011  ms/launch=279.19
    pre U    128   18 launches   pack=238.08  quant=12.89  wait=29.399  rb=0.011  ms/launch=280.37
    pre D    128   18 launches   pack=299.90  quant= 1.48  wait=28.910  rb=0.007  ms/launch=330.30
    

    Three things this settles:

    1. Work is happening. 13–18 launches per role in the first window, so the request is not blocked before dispatch — the earlier "no profiler rows" told us nothing, exactly as I flagged, and now it says the opposite.
    2. The cost is host-side and enormous: pack 91–300 ms per launch against wait 12–29 ms. On the folded t2s2 pack, same model, same ELF geometry, the matching numbers are pack 0.10–0.30 ms and wait 2.9–3.8 ms. That is a 300–1000× gap in the host term on identical shapes, and it is the reason an 8-token request cannot finish: 112 launches × ~250 ms ≈ 28 s per 128-row block, several blocks, plus whatever the decode side does — a request measured in minutes.
    3. quant is also non-trivial here (20 ms for Z, 12.9 ms for U) where the folded route shows 0.1–0.2 ms, so the activation path differs too, not only the weight side.

    I have no theory of the cause to offer that the code does not support, and the two arms of this issue already point at the images rather than the dispatcher. But this changes the shape of the bug: it is not "the pb16i route emits the wrong tokens", it is "the pb16i route's host per-launch work is 300–1000× the folded route's on identical geometry, so it cannot complete" — and the wrong/empty tokens reported in the opening table may well be a consequence (a partially warmed or partially dispatched pass) rather than a separate defect.

    Reproduction: npu-hold -t 9 -- ./run_stall.sh <1bit binary> with PACK=~/models/npu/bonsai2-27b-NPU KERNELS=$PACK/kernels-stock; it starts forward-serve on 18099 (never :18097), sends 8 then 130 tokens with a 150 s timeout, prints the profiler rows, and kills the server on every exit path.

  5. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Refinement after inspecting the container itself — this narrows the issue and retracts one assumption in my own report above.

    The container is self-contained, so the sidecars are an override, not the only source

    model.q4nx's own tensor table (99,456-byte header) holds 852 tensors: 498 I8, 257 BF16, 96 F32, under HF names —
    model.layers.N.mlp.gate_proj.weight, …linear_attn.qkv_proj.weight, …self_attn.gate_proj.weight, plus BF16 norms and ssm_a/bias. There are no
    blk.* keys
    (the sidecar naming). So the container carries its own int8-packed projections, and the q4nx_pb16i/q4nx_t2s sidecars replace them when
    config.json declares that layout. That explains why the container is shared by both model dirs without conflict.

    My "the sidecars fail verify" measurement used the wrong oracle

    Both config.jsons name the same source: q4nx_source.gguf = bonsai2-unrot-q4_0.gguf. I verified the Oct-31 sidecars against
    Ternary-Bonsai-2-27B-PTQ1_0.gguf, which is a different GGUF (5.95 GB ternary codes vs the 15.18 GB q4_0 the container was built from), so
    "MISMATCH 14/14" says the two files disagree — not that the sidecars are wrong. The same applies to my own rebuild: it verified against the PTQ1_0
    GGUF by construction, which is why pack self-checked ok while the engine served empty text from it.

    So of the three arms in the original table, only one conclusion survives intact: the Oct-31 pb16i images produce garbage while the t2s images derived
    from that same directory produce coherent text
    — that asymmetry is the actual defect, and it is not explained by provenance.

    What that leaves, stated as the question to answer next

    1. Are the Oct-31 .pb16i images consistent with bonsai2-unrot-q4_0.gguf (their declared source)? tools/q4nx_pb16i.py verify reads PTQ1_0 (ggml type
      143) only, so it cannot answer this; a q4_0 → codes/dbits dequant followed by the same verify_image comparison would.
    2. If they are consistent, the defect is in the reader path, not the data — and the cheapest scope test is whether any pb16i pack serves correctly on
      this build (none of the other model dirs on this box declares q4nx_weight_layout = pb16i, so there is currently nothing to compare against).

    I am not going to change the images, the reader or the config on the strength of the retracted reading; the corrected evidence is recorded here so the next
    person starts from it rather than from my earlier claim.

  6. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Measured follow-up: the sidecar weights are not derivable from either GGUF on this box by any transform I tried — and the converter's role map has been
    checked, so this is not a naming mistake.

    What was tested, and against what

    The converter (/home/bcloud/wt/bonsai-speed/conv/qwen35_to_q4nx.py) maps roles per layer type: FULL = {"attn_k": "self_attn.k_proj", …} when
    l % 4 == 3, and GDN = {"attn_qkv": "linear_attn.qkv_proj", …} (a fused [q|k|v], line 265) otherwise. qwen35.full_attention_interval = 4 is confirmed
    in the GGUF metadata
    , so layer 3 is full-attention and blk.3.attn_k.weight ← self_attn.k_proj — i.e. the tensor pair I compared is the right pair.

    hadamard_signs.bin is also decoded now, and it is a proper table, not noise:

    HAD1 | n=3 | (K,offset) = (5120,0) (6144,5120) (17408,11264) | 28,672 float32 ±1 signs
                                                            = 5120 + 6144 + 17408 exactly
    

    i.e. one sign per k index, per K size — the diagonal of the block-diagonal Hadamard.

    Results (all against t2s sidecar blk_3_attn_k_weight, decoded with the repo's own tested unpacker)

    comparison with the sidecar weights result
    q4_0 GGUF (bonsai2-unrot-q4_0.gguf), unrotated pearson r = 0.0004
    q4_0 under H@diag(s)@W, diag(s)@H@W, H@W (block 1024) r = 0.0155 / 0.0007 / 0.0006
    q4_0, other dims orientation; transform along N r ≤ 0.0006
    best single row/column alignment over 12 |r| ≤ 0.068
    PTQ1_0 ternary codes vs sidecar codes, same positions 34.36 % identical — chance for ~⅓ zeros
    PTQ1_0 float weights rotated, then sign-quantised vs sidecar codes 50.00 % / 50.02 % / 49.97 % — chance for a sign

    So: not the q4_0 values, not those values under the documented Hadamard, and not the ternary codes — every test lands on its chance level. The 34 % is
    exactly what one gets when ~⅓ of ternary codes are zero and the rest agree half the time, and 50.0 % is what one gets for signs; neither is a weak signal, they
    are nulls.

    The unknown, stated precisely

    The rotation convention is the missing piece. Either the transform is not FWHT over 1024-row blocks along K (radix-4 / butterfly variant, a different
    block size, or applied along N), or the sidecars were built from a rotated export that is not on this box (both GGUFs here are unrot/PTQ1_0, and
    neither is a rotated checkpoint). Reproducing the pack from anything on this machine therefore cannot be assumed to yield the frozen images — which is
    consistent with q4nx_pb16i.py pack differing from the set on all 8 layer-3 tensors at identical sizes.

    I am stopping here rather than trying more conventions blind: three "wrong oracle" readings on this issue have already looked like defects in the artifact, and
    the next test needs the rotated export or the original pack command, not another guess.

  7. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    A PR now exists for the part of this that was actionable — #1683 — and this comment says exactly what is still blocked, as the owner asked.

    What #1683 fixes: the wall every attempt at answering this issue hit. tools/q4nx_pb16i.py's gguf_tensors sized a tensor as
    n // QK * BLOCK_BYTES if tt == PTQ1_0 else 0, so every other ggml type came back as b"" — a caller cannot tell "this tensor is empty" from "this
    reader cannot size it", and for a check against a declared source those are different answers. The PR adds Q4_0 to the sized table, a read_q4_0 that returns
    values plus per-32-row scales, and a strict=True refusal that names the tensor and the type. With it, the comparison I had to do in a scratch script
    becomes a repo command, with a test whose decode assertion is proven non-vacuous (inverting it fails the suite).

    What is still blocked, and on what, precisely:

    1. The sidecars do not correlate with their declared source, and I cannot close that from anything on this box. Measured: q4_0 values r = 0.0004; under
      H@diag(s)@W, diag(s)@H@W, H@W (block 1024, signs decoded from hadamard_signs.bin) r ≤ 0.016; other dims orientation and a transform along N r ≤
      0.0007; best single row/column |r| ≤ 0.068. Against the ternary source the codes agree 34.36 % (chance for ⅓ zeros) and rotated-then-quantised signs
      agree 50.00/50.02/49.97 % (chance). hadamard_signs.bin is decoded (HAD1, n=3, (5120,0)(6144,5120)(17408,11264), 28,672 ±1 signs = 5120+6144+17408).
    2. So the open question is the rotation convention, or a rotated export that is not here. Both GGUFs on this box are unrot/PTQ1_0; neither is a rotated
      checkpoint. I need one of: the rotated export the sidecars were built from, the original pack command, or the qwen35_to_q4nx.py invocation that produced
      them — and the converter is at ~/wt/bonsai-speed/conv/qwen35_to_q4nx.py with its role map verified (FULL/GDN, full_attention_interval = 4 confirmed
      in the GGUF metadata, so layer 3 really is full-attention and the tensor pairing is right).
    3. The garbage itself is therefore unexplained rather than fixed. What is established and reproducible: the frozen .pb16i images give garbage tokens
      while the .t2s images derived from the same directory give coherent text, on the same container and binary — so the defect is in the pb16i read path or
      its composition, and not in the data's provenance.

    If the owner or another lane can supply (2), this issue becomes a measurement; without it, any further step is a third guess at a convention, and I have
    already spent two wrong-oracle readings on this one (wrong GGUF, then unrotated basis) that each looked like a defect in the artifact.

  8. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The measurement is now a repo command, not a scratch file. #1683 merged (235126a5), so tools/q4nx_pb16i.py can read a q4_0 source; this reproduces the
    sidecar-vs-source comparison on main as merged, with no --hardware and no hand-rolled GGUF parsing:

    import sys; sys.path.insert(0, 'tools')
    import q4nx_pb16i as Q
    SRC = '/home/bcloud/models/bonsai2-27b/bonsai2-unrot-q4_0.gguf'   # the q4nx_source both pb16i configs name
    NAME = 'blk.3.attn_k.weight'                                      # full-attention: full_attention_interval = 4
    raw = next(r for n, d, t, r in Q.gguf_tensors(SRC) if n == NAME)
    vals, d = Q.read_q4_0(raw, 5120, 1024)                            # (K, N) values + (K/32, N) scales
    source: blk.3.attn_k.weight q4_0 -> (5120, 1024), mean|w| 0.01119, scales (160, 1024)
    sidecar: (5120, 1024), ternary [-1 0 1]
    pearson r = 0.0004   (5,242,880 elements)
    

    Same number my scratch script produced, now rerunnable by anyone — which was the point of the PR.

    Two findings worth carrying forward from doing it this way:

    1. strict=True is all-or-nothing per file. It refuses the whole GGUF at the first unsized tensor (output_norm.weight, ggml type 0), so a caller that
      wants one q4_0 tensor out of a real model cannot use it. That is correct as a "prove this file is fully readable" mode and wrong as a "read this tensor"
      mode; a per-tensor opt-out (strict=NAME) or a size_of(name, dims, type) helper would cover the second. Not a defect, just the shape the next caller will
      want.
    2. read_q4_0's scales come back (K/32, N) — same granularity as PTQ1_0's — which is what makes an element-for-element comparison possible. Worth knowing
      for the next person who tries this: if a sidecar were built from a q4_0 source read straight, r would be ~1, and it is 0.0004.

    So the block on this issue is unchanged and is not a tooling block: the sidecar does not correspond to bonsai2-unrot-q4_0.gguf's same-named tensor under any
    transform I can apply (unrotated r = 0.0004; H@diag(s)@W, diag(s)@H@W, H@W at block 1024 with signs decoded from hadamard_signs.bin ≤ 0.016; other dims
    orientation and a transform along N ≤ 0.0007; ternary codes 34.36 % and rotated-then-quantised signs 50.00/50.02/49.97 %, both chance). What is missing is the
    rotation convention, the rotated export the sidecar was built from, or the original pack command — both GGUFs on this box are unrot/PTQ1_0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions