Skip to content

F6: MiniCPM-V 4.7 35B-A3B (MoE vision) on our own NPU runtime — images, tools, prefix cache, measured row #1788

Closed

Description

@bong-water-water-bong

F6: MiniCPM-V 4.7 35B-A3B (MoE vision) on our own NPU runtime

Owner 2026-11-09: "support all of it" — every FLM 1.0.12 behaviour on this chip, on our own runtime and kernels. This is
item F6: the first MoE vision model in that catalogue, and the heaviest single row in the plan (~22 GB).

Scope. openbmb/MiniCPM-V-4.7-35B-A3B — 35B parameters, ~3B active per token, text + images, thinking available (off by
default, as the model card recommends), tool calling, and a prefix cache that must also cover pictures. Context up to
256k with a 32k default. Memory ~22 GB, which is the constraint that dictates the order of work below.

Starting point, as the owner directed: we already serve a 35B-A3B MoE decode on the NPU, so the text route is a
starting point rather than new work. What is new is the vision tower, which ships as its own component inside the pack
(a separate vision_weight.q4nx next to model.q4nx), and the ingest path that feeds it.

Done = the plan's five conditions on the pack's own files (TNPU-ALL §1), plus the three behaviours this model adds:

  1. imports with zero gaps and passes its self-check against a named reference;
  2. serves on --device npu;
  3. answers a known-answer prompt through the model's own chat template, 3 identical requests returning identical text;
  4. a measured decode tok/s and time to first token under npu-hold;
  5. the row names the kernel set and the engine commit that produced it;
  6. images: one known image, one known question, an answer that is only obtainable from the picture — not "it accepted
    the request";
  7. tools: a tool-call round trip parsed and answered;
  8. prefix cache: a repeated prefix (text and a picture) hits the cache and the second TTFT is measurably lower.

Order of work — the 22 GB pack is not the place to iterate:

  1. vision ingest on the small siblings first. Two 1B-class vision packs ship in the same catalogue with the same
    vision_weight.q4nx shape, and they are the cheap place to get the image path, the preprocessor and the join right
    before touching a 22 GB model. A wrong answer there costs a minute, not a memory gate.
  2. the text route, from our existing 35B-A3B MoE decode lane — confirm the pack imports and decodes before any image
    is involved, so a failure is attributable to one half.
  3. the join: image embeddings into the text model at the right positions, then the known-image question.
  4. tools + prefix cache, then the measured row (decode tok/s, TTFT, cache-hit TTFT) under npu-hold.

Constraints being honoured (TNPU-ALL §2): own runtime and own kernels, full ELFs only, npu-hold tickets of ≤9
minutes, one pack >8 GB at a time with free -g ≥ 40 GiB first, packs on disk, nothing redistributed, and the pack's
licence recorded on the row. Kernel and runtime internals stay in the private repos; this issue and any PR stay neutral.

Not in this issue: the text-only speed comparison against another build (that is F3/F9), and the other new 1.0.12
sizes (F7).

Found while working the catalogue item; no existing issue covers this model (searched MiniCPM, vision tower — only
census mentions and the unrelated CLIP-graph row).

Activity

  1. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Pack facts, read from the published packs (no download — the box is at its memory and disk limits right now)

    The three models are on HF ungated: FastFlowLM/MiniCPM-V-4.7-35B-A3B-NPU2, MiniCPM-V-4.7-1B-NPU2, MiniCPM-OCR-1.0-NPU2. What the files say:

    pack files text component vision component
    MiniCPM-V-4.7-35B-A3B-NPU2 9 (diff. MANIFEST.sha256) model.q4nx 20,068.93 MiB vision_weight.q4nx 1,091.56 MiB
    MiniCPM-V-4.7-1B-NPU2 8 model.q4nx 1,019.32 MiB vision_weight.q4nx 1,082.56 MiB

    Two things fall out of that table:

    1. The vision tower is its own component in both packs, and it is the same size in each (1,082.56 vs 1,091.56 MiB) while the text model differs by 20×. So the tower is one fixed piece of work — building it on the 1B pack gives the same artifact for the 35B, which is exactly why the order below starts there.
    2. ~22 GB is 20.07 GiB of text + 1.07 GiB of vision, not a rounded-up estimate.

    config.json of the 35B (flm_version: 1.0.12, so the pack is the version this item targets):

    architecture          MiniCPMV4_7ForConditionalGeneration (model_type minicpmv4_7)
    text backbone         hidden 2048, 40 layers, head_dim 256, vocab 248,144, attn_output_gate
    attention             hybrid: layer_types = 3× linear_attention + 1× full_attention, full_attention_interval 4, linear_conv_kernel_dim
    MoE                   256 experts, 8 per token, moe_intermediate_size 512
    context               max_position_embeddings 262,144
    image                 image_token_id present
    

    The 1B sibling is the same family at hidden 1024 / 35 layers, and its card declares flm_version: 1.0.13 — so the catalogue has already moved past 1.0.12 for the small pack, which matters only if we pin packs by version.

    The image contract, from the model's own chat template

    The template renders an image as the literal <|image_pad|> (and a video as <|video_pad|>), and it raises on an image inside a system message. So the ingest path is: render → tokenise → find the <|image_pad|> positions → replace those token embeddings with the tower's output. That is the shape the row's condition 6 tests.

    Blocked on the box, not on the work — recorded rather than worked around

    TNPU-ALL §2 requires free -g ≥ 40 GiB before taking a pack over 8 GB, and a 150 GB disk floor. Measured just now:

    free -g   →  total 122, used 116, available 3–5
    df /      →  151 G free   (floor 150)
    

    So taking the 22 GB pack would put the box below its disk floor and it would not fit in 3 GiB of available RAM anyway, and even the 2.07 GiB 1B pack would cross the disk floor. I am not going to start a download that ends with a full disk or an OOM kill of whatever else is running. The item stays open and the order of work below is what happens as soon as either limit relaxes — I will re-check rather than assume.

    Next concrete step when it does: npu-pack info on the 1B pack (device-free, host-side import) to see what our importer says about a minicpmv4_7 container with a separate vision component — that is the first gap to name, and it costs one 2 GiB download instead of 21.

  2. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Where the gap actually is, from our importer's code (no pack needed)

    npu/pack_import.cpp has no mention of a vision component at all — grep -in 'vision\|clip\|vit\|minicpm' over it returns nothing — so this is new work, as the catalogue delta predicted. But the seam is narrower than "a new container format", and that changes the size of the job:

    Our importer already knows about two-component packs, but in a different shape. find_token_fconst() (around npu/pack_import.cpp:553) looks for the Token .fconst at the pack root and then, per the compositemodel rule, in the component directory that graph_filename() names:

    // A multi-component pack (`type: compositemodel`) keeps its LLM graph AND its Token .fconst in the component
    // `genai_config.json` names (`decoder/`), while the config files that state the geometry stay at the pack root.

    The MiniCPM-V packs are not shaped like that. Their file list is flat:

    model.q4nx   20,068.93 MiB     ← the text model
    vision_weight.q4nx  1,091.56 MiB     ← the vision tower, a SIBLING FILE at the pack root
    

    So the difference is not "a new container": vision_weight.q4nx is the same q4nx container we already read for model.q4nx, holding a different kind of content — a vision transformer instead of a decoder. What is genuinely new:

    1. a second q4nx in the same pack, discovered by name rather than by a compositemodel component path, and imported as its own graph;
    2. that graph's op set — patch embedding, a ViT attention stack and MLP, no KV cache and no token loop, which is a different execution profile from everything the NPU lane runs today (it is prefill-shaped and runs once per image, not per token);
    3. the join: <|image_pad|> positions in the rendered prompt (the model's own chat template writes them) must be replaced by the tower's output embeddings before the text prefill consumes them.

    That is what makes the 1B sibling the right first target: it is the same flat pair at 1/20th the text size, so items 1 and 3 can be got right for one 2.07 GiB download instead of a 21 GiB one.

    Still blocked on the box

    Measured after freeing 4 GB of my own re-creatable model files: RAM available 1 GiB (of 122), disk 152 G (floor 150). The 1B pack needs ~2.07 GiB on disk and the import needs headroom in RAM; neither is available. Nothing has been downloaded, and I will not start a transfer that ends in a full disk or an OOM kill of another lane's work. Next step the moment either relaxes: npu-pack info on MiniCPM-V-4.7-1B-NPU2 — device-free, host-side, and it names the first refusal.

  3. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    First measured step: the pack is downloaded and our loader names the first gap

    MiniCPM-V-4.7-1B-NPU2 fetched into ~/scratch-packs (2.1 GiB, both components complete — packs go on disk per TNPU-ALL §2, never /tmp). Ran our own loader against it, device-free, under a 45 s timeout so nothing is left running:

    $ 1bit unified -m scratch-packs/MiniCPM-V-4.7-1B-NPU2 -p 18199 --host 127.0.0.1
    1bit unified: loading scratch-packs/MiniCPM-V-4.7-1B-NPU2 on the NPU
    1bit unified: scratch-packs/MiniCPM-V-4.7-1B-NPU2 is not an NPU model directory (the fast lane needs npu/)
    

    That is the text half's first gap, and it is not the model. unified takes model.q4nx + config.json + tokenizer.json — all present in this pack — plus npu/, our kernel set. The pack ships FLM's own shims instead, so nothing about minicpmv4_7 has been rejected yet; the route simply refuses a directory with no kernels in it.

    Two consequences for the order of work:

    1. The text row needs a kernel set for this geometry, built from our own compilers (Loom AIE2P / Peano), not a container fix. For the 35B sibling there is already a lane — 1bit npu-lax --model <dir> --kernels <dir>, which is documented as "Qwen3.6-35B-A3B Q4NX directory" — but MiniCPM-V 4.7 35B-A3B is a different model at a similar active-parameter count (256 experts, 8 per token, 40 layers, hidden 2048, hybrid 3-linear/1-full attention), so whether that kernel set accepts this geometry is the next question, not an assumption.
    2. vision_weight.q4nx still has no consumer at all — grep -in 'vision\|vit\|clip\|minicpm' over npu/ finds nothing, so on the 1B pack the tower is 1,082.56 MiB of container nobody opens. That half is unaffected by the kernel question and can be developed on this pack alone.

    And a disk consequence worth recording before anyone tries the big one: the 35B pack is 20.68 GiB, and the box is at 154 G free against a 150 GB floor — taking it would land at ~133 G, below the floor. So the large pack needs ~20 GB released first, or a lane that is allowed to run closer to the floor than this plan permits. The 1B pack was the right first move for that reason as well.

    Next: npu-lax --model <1B dir> --kernels <built set> will tell us whether the text container maps without an importer in the path at all — and if it does, F6's text half is a kernel-geometry question rather than a format one.

  4. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The vision tower's full geometry, from the pack's own config.json (device-free, no container parsing yet)

    The pack carries everything needed to size the work — and it is arithmetically self-consistent, which is the check that it is the tower's real geometry and not a default:

    vision_config:
      MINICPM_V_VISION_NUM_LAYERS            27
      MINICPM_V_VISION_EMBED_DIM             1152
      MINICPM_V_VISION_NUM_HEADS             16
      MINICPM_V_VISION_HEAD_DIM              72      → 16 × 72 = 1152 ✓
      MINICPM_V_VISION_MLP_INTERMEDIATE_SIZE 4304    → 3.74× embed
      MINICPM_V_VISION_LAYER_NORM_EPSILON    1e-06
      MINICPM_V_PATCH_SIZE                   14
      MINICPM_V_IMAGE_SIZE                   980
      MINICPM_V_SPATIAL_MERGE_SIZE           2
      MINICPM_V_TEMPORAL_PATCH_SIZE          1
      MINICPM_V_DOWNSAMPLE_FACTOR            16
      MINICPM_V_MERGER_HIDDEN_SIZE           4608    (= 4 × 1152)
      MINICPM_V_MAX_SLICE_NUMS               9
      MINICPM_V_INSERT_LAYER_ID              6
    vision_model_weight: vision_weight.q4nx
    image_token_id: 248056
    

    The token arithmetic that sizes the job:

    980 / 14                      = 70 patches per side → 4,900 patches per slice
    4900 after spatial merge /2   = 35 × 35 = 1,225 vision tokens per slice
    × MAX_SLICE_NUMS 9            = 11,025 vision tokens worst case
    

    So one image is a 4,900-row patch sequence through 27 ViT layers at embed 1152 — an 11k-token attention workload in the worst case, and it runs once per image, not per decoded token. That is a prefill-shaped profile, and it is the reason this is not "the decode kernel with a different weight matrix": the shapes (4,900 × 4,900 attention, no KV cache, no autoregressive step) are unlike anything the decode lane runs.

    INSERT_LAYER_ID = 6 is the join point named by the pack: the image embeddings enter the text model at layer 6, which is consistent with the <|image_pad|> substitution the chat template implies. DOWNSAMPLE_FACTOR 16 and MERGER_HIDDEN_SIZE 4608 are the merger between the tower and the text embedding width (1024 on this pack) — so the join is a projected, not a raw, splice.

    Two fields worth flagging for other items rather than this one: mtp_num_hidden_layers (multi-token-prediction heads — F2's speculative decode territory, not F6's), and addr_kv / addr_qk (address maps the pack states explicitly, which is unusual and worth reading before any layout assumption is made).

    Next device-free step: open vision_weight.q4nx as a container and list its tensors — if it is the same q4nx format as model.q4nx (which our loader already maps), the ViT's weights are reachable without any new reader, and the remaining work is purely the graph plus kernels.

  5. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The vision container opened: same format, a metadata preamble, and 347 bf16 tensors that cross-check against the config

    Read with a 20-line parser (8-byte little-endian length + JSON record per tensor — the same container our loader already maps for model.q4nx), no vendor tooling:

    records    1 metadata + 347 tensor records
    dtypes     BF16 × 347          ← every tensor
    structure  model.visual.blocks.*      324     (27 blocks × 12)
               model.visual.vit_merger     12
               model.visual.merger          8
               model.visual.patch_embed     2
               model.visual.pos_embed       1
    max block index 26 → 27 blocks  == config's MINICPM_V_VISION_NUM_LAYERS 27  ✓
    

    Three things follow, and the first is a kernel-level good-news result:

    1. The tower is bf16 end to end. XDNA2's native multiply modes are int8×int4, int8×int8 and bf16×bf16 — so the vision tower needs no quantization and no dequant path: it is a bf16 ViT, which is the one mode the hardware does natively. That is a materially better starting point than the text side, where the weights are q4nx-quantized.
    2. The container differs from model.q4nx in exactly one way: vision_weight.q4nx's first record is __metadata__, while model.q4nx starts with tensor records directly. A reader that assumes "offset 0 is a tensor" will either fail or, worse, treat the metadata as a tensor. That is the concrete reader change, and it is one branch.
    3. The metadata names the provenance, which is useful for the bit-exactness bar:
      source_file    mmproj-model-f16.gguf
      converter      minicpm_v vision (bf16)
      source_dtype   BF16 (+233 F32 tensors, bf16-exact)
      source_sha256  fb9cc047a3cdad5317fd8433a4aeda2d22f00f02ca4bd93b34ad84165b25a900
      
      The source is an mmproj (multimodal projector) GGUF — the same family our llama.cpp tree already loads for vision models — and it also gives a sha256 of the source weights, which is exactly what a reference-correlation gate needs.

    One discrepancy worth recording rather than smoothing over: the metadata claims "BF16 (+233 F32 tensors, bf16-exact)", but every one of the 347 records carries dtype: BF16. The consistent reading is that the 233 were F32 in the source and were stored as bf16 because they are bf16-exact (the metadata says so), so no precision was lost — but a reader should not take "all BF16" as proof that nothing was rounded. If it matters for the parity bar, the check is to re-read those 233 names from the source mmproj and compare bit patterns; that is the kind of thing to do before claiming bit-exactness, not after.

    Next: the graph. patch_embed (2 tensors) + pos_embed + 27 blocks of qkv/proj/mlp at embed 1152, 16 heads × 72, over 4,900 patches per slice — prefill-shaped, no KV cache, one pass per image. And the merger pair (vit_merger 12 + merger 8) projects to MERGER_HIDDEN_SIZE 4608 before the text model's width.

  6. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The tower's tensor shapes — and one place the config is wrong

    Every shape read from the container (35 sampled, then the fc1 across all blocks):

    patch_embed.proj.weight        [1152, 3, 14, 14]     Conv2d, RGB, 14×14 patches → 1152
    blocks.{0..26}.attn.qkv.weight [3456, 1152]          3456 = 3 × 1152 ✓
    blocks.{0..26}.attn.proj.weight[1152, 1152]
    blocks.{0..26}.mlp.fc1.weight  [4608, 1152]          × 27 blocks, every one
    blocks.{0..26}.mlp.fc2.weight  [1152, 4608]
    blocks.{0..26}.norm{1,2}       [1152] weight + BIAS   LayerNorm, not RMSNorm
    vit_merger.*                   one transformer block at embed 4608 (qkv/proj/mlp, mlp fc1 [17280, 4608])
    merger.fc1                     [4608, 4608]
    merger.fc2                     [1024, 4608]          → the TEXT model's hidden (1024) ✓
    

    The config is wrong about one shape, and it is the one a loader would allocate from. config.json declares MINICPM_V_VISION_MLP_INTERMEDIATE_SIZE = 4304; the weights are 4608, uniformly across all 27 blocks. 4608 is also MERGER_HIDDEN_SIZE and exactly 4 × embed (1152), so the value in the container is the coherent one — but a reader that trusts the config and sizes an activation buffer from it would be 304 elements short of the truth on every block. Trust the tensor record, not the config, for anything allocated. That is worth stating in the code that reads this pack, because the failure mode is an overflow or a truncated row, not a refusal.

    Three consequences for the kernels, all now measured rather than assumed:

    1. patch_embed is a Conv2d — [1152, 3, 14, 14] — so the vision path starts with a strided convolution, not a matrix multiply over pre-extracted patches. Either a real conv kernel or an im2col-style layout change, and the weight is bf16 like everything else.
    2. The blocks are LayerNorm with bias, where the text side uses RMSNorm. Different normalisation → a different kernel, and a place where reusing the text norm would silently produce plausible numbers.
    3. The join is merger.fc2: [1024, 4608] — it projects to exactly this pack's text hidden size (1024), which confirms the <|image_pad|> substitution is a projected splice at INSERT_LAYER_ID 6, not a raw embedding copy.

    So the ViT is: Conv2d patch embed → 27 × (LayerNorm → QKV → proj → LayerNorm → MLP 4608) at 4,900 patches → spatial merge (1,225 tokens) → 1 extra transformer block at 4608 → project to 1024. All bf16, one pass per image, no KV cache.

    Next device-free step: the arithmetic against the source. The metadata carries source_sha256 and the source is an mmproj GGUF, so the tower's weights can be re-read from a source we already know how to load and compared tensor-by-tensor — which is the reference the bit-exactness gate needs, before any kernel is written.

  7. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Decisive: MiniCPM-V 4.7 35B-A3B's text backbone is Qwen3.6-35B-A3B, field for field — so the text half is the existing lane plus a vocab delta

    Compared the pack's config.json against Qwen/Qwen3.6-35B-A3B's text_config — the model our npu-lax lane is documented for:

    field MiniCPM-V 4.7 35B-A3B Qwen3.6-35B-A3B
    hidden_size 2048 2048 ✓
    num_hidden_layers 40 40 ✓
    num_experts 256 256 ✓
    num_experts_per_tok 8 8 ✓
    moe_intermediate_size 512 512 ✓
    head_dim 256 256 ✓
    num_attention_heads 16 16 ✓
    num_key_value_heads 2 2 ✓
    full_attention_interval 4 4 ✓
    attn_output_gate true true ✓
    linear_conv_kernel_dim 4 4 ✓
    layer_types (40) identical list identical list ✓
    vocab_size 248,144 248,320 DIFFER (176)

    Every architectural field matches, including the full 40-entry layer_types list (3 linear-attention + 1 full-attention repeating). The only delta is the vocabulary, 176 tokens smaller, which shows up in exactly two tensors: the token embedding table and the lm-head. So F6's text half is not a new kernel set — it is our existing 35B-A3B MoE decode lane, with the embedding/lm-head row count taken from the pack (248,144) rather than from the geometry (248,320). A kernel set sized from the geometry would either read 176 rows past the table or mis-index the head; a kernel set sized from the container needs nothing else.

    That also explains the 22 GB: 20.07 GiB of text weights is a 256-expert MoE at this geometry, and the tower adds 1.07 GiB.

    What is left, and it is now two separable pieces

    1. Text: run the existing 35B-A3B lane against this pack with the container's vocab, its own tokenizer, and MiniCPM's chat template. The refusal I get today is only is not an NPU model directory (the fast lane needs npu/) — the kernel directory, not the model — so the question is whether the existing set's lm_head and embedding kernels can be pointed at 248,144 rows, which is a parameter, not a redesign.
    2. Vision: the tower is identical between both MiniCPM packs (27 layers, embed 1152, 16 heads × 72, merger 4608, 980 input, 9 slices) — 1,082.56 MiB on the 1B and 1,091.56 MiB on the 35B, i.e. the same weights with alignment rounding. So one tower implementation serves both, and it can be developed on the small pack.

    Pack integrity, verified

    The 35B pack ships a MANIFEST.sha256 and both components match it:

    model.q4nx          08e922a9e570…  MATCH
    vision_weight.q4nx  a229ba8c2c80…  MATCH
    

    That is the provenance a row needs, and it means the 21 GiB on disk is the pack as published rather than a partial fetch.

    The tower's geometry is identical across the two packs

    1B 35B
    vision layers / embed / heads×dim 27 / 1152 / 16×72 27 / 1152 / 16×72
    image / patch / spatial merge / slices 980 / 14 / 2 / 9 980 / 14 / 2 / 9
    merger hidden 4608 4608
    text hidden / layers / experts 1024 / 24 / dense 2048 / 40 / 256

    So the 1B is the dense sibling with the same tower — the cheapest place to get the tower right, exactly as planned, and its result transfers unchanged.

    Next: point the existing 35B-A3B lane at the 35B pack with --model <pack> and a kernel set, and see whether the vocab delta is the only thing that has to change. That needs an npu-hold ticket, so it is the first step that touches the device.

  8. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Vision tower inventory, read from the pack's own manifest (1B pack), and two config fields that are stale

    The container is one JSON manifest + a payload region, not a record stream: an 8-byte LE length, that many bytes of JSON whose top-level keys are the tensor names (plus __metadata__), then the payload at data_offsets relative to the payload start. My earlier note that "the first record is __metadata__" was a truncated reading — json.loads succeeded on the 38,808-byte first block and I stopped there. Correcting it here because the manifest is the whole tensor table.

    347 tensors, all BF16. The vision tower is two merger stages, which is what I had wrong:

    stage tensors shape
    patch embed proj.weight / .bias [1152,3,14,14] Conv2d / [1152]
    27 blocks attn.qkv [3456,1152], attn.proj [1152,1152], mlp.linear_fc1 [4608,1152], linear_fc2 [1152,4608], norm1/norm2 [1152] 12 tensors × 27
    vit_merger attn.qkv [3456,1152], attn.proj, mlp.linear_fc1 [17280,4608], linear_fc2 [1152,17280], pre_norm [4608], norm1 [1152] an attention block + MLP at 4608
    merger pre_norm [4608], linear_fc1 [4608,4608], linear_fc2 [1024,4608], norm [1152] 4608 → 1024 (LLM hidden)

    So MINICPM_V_MERGER_HIDDEN_SIZE: 4608 is 4 × 1152 — the 2×2 spatial merge feeding the merger, not a ViT MLP width. patch_embed.proj carries a bias, and there is no gate tensor anywhere in the MLP, so it is a plain fc1 → SiLU (hidden_act: silu) → fc2 two-layer MLP, not SwiGLU.

    The part that matters for a runtime

    Two config fields disagree with the tensors in the same pack:

    config pack tensor delta
    MINICPM_V_VISION_MLP_INTERMEDIATE_SIZE = 4304 blocks.*.mlp.linear_fc1 = 4608 rows (bias [4608]) +304
    MINICPM_V_VIT_MERGER_INTERMEDIATE_SIZE = 17216 vit_merger.mlp.linear_fc1 = 17280 rows +64

    Both are stale for 4.7 (4608 = 4 × 1152 is the round number; 4304 is the 4.5-era value). A runtime that sizes the ViT MLP from the config allocates 4304 × 1152 and reads 4608 × 1152 — a 304-row overrun per weight, which lands in the neighbouring tensor's bytes and returns plausible numbers rather than faulting. That is engine#1883's finding arriving from the pack side, and it is why the rule for this tower is:

    size every buffer from the tensor's own shape, never from config.json. The config remains useful for what the tensors cannot say — patch_size 14, image_size 980, spatial_merge 2, max_slice_nums 9, insert_layer_id 6, num_position_embeddings 4900, layer_norm_eps 1e-6, rescale_factor 1/255 with mean/std 0.5 — and the token count follows: 980/14 = 70 → 4,900 patches → 1,225 after merge, ×9 slices = 11,025 worst case.

    Next step on my side is unchanged (needs a kernel set for this geometry), but the vision tower no longer needs one to be specified: 27 such blocks, one attention+MLP at 4608, one 4608→1024 projection, all BF16, all shapes above read from the manifest.

  9. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Text side: the embedding matches vocab_size exactly in both packs, and the head's blocked layout is derivable from its own bytes

    Measured from each pack's manifest (data_offsets, so the byte extent is the pack's, not my fit of the shape):

    pack config.json vocab model.embed_tokens lm_head
    1B 248,094 [248094, 1024] BF16 — exact, no padding [7753, 4, 8704] I8
    35B-A3B 248,144 [248144, 2048] BF16 — exact, no padding [7755, 8, 8704] I8

    So the #1878 direction (shortfall refused, padding tolerated) is satisfied trivially here: both embedding tables are exactly [V, H], unquantised BF16. Worth stating because it means vocab × H × 2 is the right size for the device read in both packs — a check that would refuse a padded table would have been wrong, and a check that assumed equality would have been right for these two and wrong in general.

    The head's blocked layout, from its own extents. prod(shape) == data_offsets[1] - data_offsets[0] exactly in both, and dividing by the tile count each shape encodes gives the same number:

    • 1B: 7753 × 4 = 31,012 tiles → 269,928,448 / 31,012 = 8704.0 bytes per tile
    • 35B: 7755 × 8 = 62,040 tiles → 539,996,160 / 62,040 = 8704.0 bytes per tile

    8704 is a constant of the layout, not of either model: r × (H + H/16) with r = 8192/H.

    rows/tile r codes multipliers total
    H = 1024 8 8 × 1024 = 8192 8 × 64 = 512 8704
    H = 2048 4 4 × 2048 = 8192 4 × 128 = 512 8704

    i.e. every tile is 8 KiB of int8 codes plus 512 B of multipliers, one multiplier per 16 input columns — which is why the tile size does not move with H while its row count halves. Two independent packs agreeing on 8192 + 512 to the byte is the strongest form this kind of evidence takes.

    Row padding exists and must be handled: r × tiles is 248,096 for the 1B (2 rows past 248,094) and 248,160 for the 35B (16 rows past 248,144). Those rows are not tokens; emitting them or letting them contribute to a softmax is a silent wrong-answer path with no fault.

    And the two variants do not share a vocabulary: 248,094 versus 248,144 — a 50-row difference inside one family. Anything that builds a token table from one pack and applies it to the other is off by 50 ids at the tail, which lands exactly on the merge/special tokens an image prompt uses.

    Next: the vision side is specified (previous comment) and the text side's head geometry is now measured, so the remaining gap for a device run is still a kernel set for this geometry, not a missing fact.

  10. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The kernel-set blocker is not "no kernel set" — it is two shapes, and both are siblings of ones already on this box

    My earlier note that the pack "ships FLM shims only and no npu/ kernel dir" is still true of the pack, but there are kernel sets on this host, and one of them — ~/models/npu/qwen3-coder-elfs/ + qwen3-coder-kernels/ — shares this model's attention geometry exactly. My earlier sweep missed them because I grepped for the part names (lax_l, lax_a, ln, lm_head_q8) instead of for ELFs; the names are full_i8_<op>_K<K>_N<N>.elf.

    Mapping this model's shapes (hidden 2048, 16 q heads × 256, 2 kv × 256, moe_intermediate_size 512) against what exists:

    op needed on box
    QKV K2048 N5120 full_i8_QKV_K2048_N5120.elf exact ✓
    attn out K4096 N2048 full_i8_O_K4096_N2048.elf exact ✓
    expert gate+up K2048 N1024 G_K2048_N1536, GU_K2048_N1536, U_K2048_N1536 missing — 1536 is Qwen3-Coder's moe_inter 768 × 2
    expert down K512 N2048 D_K768_N2048 (K wrong), KB_K512_N256 (N wrong) missing
    lm head K2048 N≈248,144 none of that N separate head path, expected

    So the attention half of a device run is already available on this host, and the gap is exactly two expert shapes, each one parameter away from an existing sibling: GU_K2048_N1536 → N1024 (512 instead of 768) and D_K768_N2048 → K512. Both of the dimensions involved (K=512, N=1024) are already exercised by other kernels in the same directory (KB_K512_N256, D_K3072_N1024), so neither is a new code path — they are the same kernels at a different tiling.

    Two caveats before anyone treats this as an unblock. Kernel existence is not kernel correctness for this model: the set was generated for a different family, so num_layers (this model has 40, with full_attention_interval 4 — so 10 full-attention and 30 linear-attention layers), the RoPE variant and the eps must all agree with what the loader reads, and the linear-attention layers have no counterpart in a Qwen3-Coder set at all. And the vision tower needs no such set only if it runs on pi-npufeat, which is not the path I have measured.

    Next step I can take without a device: confirm the loader reads the layer kinds per layer from config.json (layer_types, which I have already verified is field-identical to Qwen3.6-35B-A3B) rather than from the kernel set's naming, since a 40-layer model with 10/30 attention split cannot be served by a set built for a uniform family.

  11. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Second pair of hands on this issue (lane cl-npurt, owner 11-10 21:40Z). The slice I am taking, so it does not overlap the claiming lane:

    The thread above has established the facts about vision_weight.q4nx by hand (Python, 20-line parser). Nothing in the engine opens that file yet. I am taking the device-free vision half, in this order:

    1. The vision component reader (first PR): open vision_weight.q4nx with the engine's own code, account for every tensor (zero gaps is done-condition 1), derive the tower geometry from the tensors' own shapes, and refuse by name what it cannot place. The two stale config fields found above (MLP_INTERMEDIATE_SIZE 4304 vs 4608, VIT_MERGER_INTERMEDIATE_SIZE 17216 vs 17280) become a tested rule in code rather than a note in a comment. Surfaced as a 1bit npu-pack subcommand so the result is a command anyone can run on both packs.
    2. A host float32 reference for this tower on top of the npu: assemble the vision tower — patch embed, N blocks, merger, in merge-block token order (engine#1503 W4) #1930 building blocks (block, merger), which need two additions this tower has and the Qwen3-VL one does not: a learned [4900, 1152] position table used directly, and the vit_merger stage. That one needs a named oracle before I claim anything about its numbers.

    I am not touching: the text route and its kernel set (the two missing expert shapes in the last comment), anything on the device, the chat-template/<|image_pad|> join, tools, or the prefix cache. Those stay with the claiming lane.

    One unblock already done for step 2's base: #1930 had its READY verdicts but conflicted with main again, so the queue could not merge it. I pushed a merge of main (d26d52d, one conflict in a test's main(), both sides kept, 5 of 5 tests pass) and it is mergeable again.

  12. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    First PR of the vision slice is up: #2040 (npu/vision_pack.h, 1bit npu-pack vision <pack dir>), ready for review at d51f4ef. The engine now opens vision_weight.q4nx itself.

    Measured with it on both published packs, device-free:

    1B 35B-A3B
    tensors, all BF16, every one placed 347 347
    payload bytes outside a tensor 0 0
    blocks / width / heads 27 / 1152 / 16 × 72 27 / 1152 / 16 × 72
    block MLP (config says 4304) 4608 4608
    vit_merger MLP (config says 17216) 17280 17280
    merger output = text hidden 1024 2048
    __metadata__ provenance source file + sha256 none

    One correction to the thread above. The 05:25Z comment says the tower is "identical between both MiniCPM packs … the same weights with alignment rounding". The geometry is identical; the weights are not. I hashed every tensor in both containers: 0 of 347 are byte-identical, including the blocks, which have the same shapes in both. The size difference (1,082.56 vs 1,091.56 MiB) is the last projection, [1024, 4608] vs [2048, 4608]. So one implementation serves both packs, but a numerical reference built for the 1B says nothing about the 35B's numbers, and the 35B's container names no source to build one from.

    What remains of the vision half, in order:

    1. A host float32 forward for this tower. It can reuse npu: assemble the vision tower — patch embed, N blocks, merger, in merge-block token order (engine#1503 W4) #1930's block and merger math, and needs two things that tower does not have: the [4900, 1152] position table and the vit_merger stage (attention at 1152, then an MLP over 2×2 groups at 4608). The exact order of operations (where vit_merger sits relative to block insert_layer_id 6, how positions are assigned for a non-square slice, which activation) has to come from the model's own reference code, not from the tensor names. I have not established it and am not guessing it.
    2. A numerical oracle for that forward. The 1B container names mmproj-model-f16.gguf with a SHA-256, which is the candidate; it is not on this box.
    3. Image preprocessing and slicing (980 px, up to 9 slices, mean/std 0.5), then the <|image_pad|> join. The join belongs with the text route.

    Unchanged and not mine: the text route's kernel set (two expert shapes missing), tools, prefix cache, the measured row.

  13. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Done-condition 1 for the vision component: the self-check against a named reference, bit for bit, on both packs. And a correction to my own comment above and to the thread.

    The 1B container's __metadata__ names mmproj-model-f16.gguf and a SHA-256. That file is in openbmb/MiniCPM-V-4.7-1B-gguf; downloaded, sha256sum = fb9cc047a3cd…165b25a900, the pack's stated hash. I rebuilt every pack tensor from the GGUF and compared BF16 bit patterns:

    identical identical + zero padding different
    1B pack vs its source 263 84 0
    35B-A3B pack vs openbmb/MiniCPM-V-4.7-35B-A3B-gguf mmproj (bff3b688…5d20f1) 263 84 0
    35B-A3B pack vs the 1B's source (control) 0 0 347

    All 347 of 347 in both packs. Every F32 source tensor is exactly representable in BF16 (0 exceptions), which settles the "+233 F32 tensors, bf16-exact" question raised at 03:31Z.

    The correction: config.json is not stale. The thread (04:10Z, 09:34Z) and my first commit said the config was wrong about the MLP widths (4304 vs 4608, 17216 vs 17280). The source GGUF's ffn tensors are 4304 and 17216 wide. The container stores them padded to the next multiple of 128, and the 84 "padding" tensors above are exactly those: the source's values in the leading extent, +0.0 everywhere past it (fc1 rows, fc1 bias entries, fc2 columns, in all 27 blocks and the vit_merger). So the rule "size every buffer from the tensor" stands, but the reason is padding, and a forward can stop at 4304. #2040 now verifies the zero tail and carries both widths (head a8c38f4).

    Name mapping, established by the comparison:

    pack source GGUF
    blocks.N.attn.qkv v.blk.N.attn_q, attn_k, attn_v stacked in that order
    blocks.N.mlp.linear_fc1 / linear_fc2 v.blk.N.ffn_up / ffn_down (padded)
    vit_merger.pre_norm, mlp.linear_fc1/2 v.vit_merger.ds_ln, ds_ffn_up/down (padded)
    merger.norm v.post_ln
    merger.pre_norm, linear_fc1, linear_fc2 mm.input_norm, mm.up, mm.down

    Two facts for whoever does the join and the forward:

    • One source tensor has no counterpart in either vision container: v.tok_embd_sep, [4, hidden] (four embedding rows at the text model's width). I have not looked for it in model.q4nx.
    • Both sources declare clip.projector_type minicpmv4_7, clip.vision.wa_layer_indexes [6], clip.use_gelu True, clip.vision.image_size 448, projector.scale_factor 4. Our llama.cpp fork (pin 5514ebe) has a minicpmv4_6 projector with a vit_merger graph and no minicpmv4_7, so there is no forward oracle for this tower on the box today. That is the blocker for a host reference of the tower's output; I am not writing a forward against an oracle that does not exist.

    The comparison was a script run by hand against 2.2 GB of source files in my lane scratch; it is described in the PR and the docs, not shipped as a test.

  14. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The host reference for the tower exists and matches the model's own implementation: #2056 (ready for review at 0b634fd). #2040 (the reader) merged at 22:42Z.

    I said two comments up that there was no forward oracle on the box. There is one now: the tower is in the transformers library itself (transformers.models.minicpmv4_7), so I loaded its MiniCPMV4_7VisionModel and merger with each pack's weights (strict load, every key matched) and compared vision::tower_forward() with it on the same input:

    pack patch grid tower before the merger, max |d| output rows, max |d| corr
    1B 8 × 12 2.9e-5 1.1e-6 1.000000000
    1B 16 × 20 1.1e-5 2.4e-6 1.000000000
    35B-A3B 16 × 20 9.5e-6 9.5e-7 1.000000000

    What the tower actually is, since the thread guessed at parts of it:

    • positions are a learned 70 × 70 table indexed nearest (floor(y·70/gh), floor(x·70/gw)): no interpolation, no rotary term;
    • blocks use the tanh GELU (the config's hidden_act: silu is the text model's, not the tower's);
    • the vit_merger runs once, after block 6: attention inside each 2 × 2 window, then an MLP over the four tokens plus their mean, halving the grid. Blocks 7–26 run on the halved grid;
    • the merger halves it again with the exact (erf) GELU. So a slice of gh × gw patches becomes (gh/4) × (gw/4) rows: a full 980-pixel slice is 70 × 70 patches... which is not a multiple of 4. The reference requires divisibility at each merge, so the preprocessor must pick grids that are multiples of 4; the "1,225 tokens per slice" arithmetic earlier in this thread (35 × 35) cannot be right for this model. I have not read the preprocessor yet and am not stating what it picks.

    What remains of the vision half:

    1. Image preprocessing: resize and slicing to grids the two merges accept, rescale and normalise. processing_minicpmv4_7.py in the same library is the reference.
    2. The join (<|image_pad|> rows, the slice separator tokens, positions on the text side). With the text route's owner.
    3. The tower on the device. This PR is what that will be compared with.

    Not started by me: 1. I will take it next unless the claiming lane says it has it.

  15. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The image path before the tower is written and matches the reference end to end: #2066 (stacked on #2056, ready for review at 44b5438).

    With #2040 (merged), #2056 and #2066 the device-free vision half goes from RGB bytes to the rows the text model receives, each step checked against the model's own implementation:

    step checked against result
    container: all 347 tensors the source GGUF named by the pack bit-identical (84 with zero padding), both packs
    image plan: sizes, slice grid, rows transformers' image processor, 3,481 image sizes identical
    resize (bicubic, antialias, bytes) torch interpolate on uint8, 11 cases every byte identical
    tower, float32 host transformers' tower with the pack's weights max |d| ~1e-6 on the output rows
    bytes → rows, 460 × 470 image, 1B pack all of the above chained 3 pieces, 196 rows, max |d| 7.5e-6, corr 1.000000000

    The token arithmetic for this model, corrected (the thread had 1,225 per slice and 11,025 worst case):

    • every piece is a multiple of 56 pixels (4 patches) per side and becomes (h/56) × (w/56) rows;
    • an image up to 448 × 448 is scaled up to one 448 × 448 view: 64 rows;
    • 1080 × 1920 is a 336 × 616 view plus 2 × 4 slices of 448 × 448: 66 + 8 × 64 = 578 rows;
    • over the 3,481 sizes I walked (1 to 8000 per side) the most is 716 rows.

    That matters for the text route: the prefill for one picture is a few hundred rows, not eleven thousand.

    Not done, and where each sits:

    • the text side of an image (the <image>/<slice> markers, the placeholder rows, v.tok_embd_sep, positions): the join, with the text route's lane;
    • an image decoder (PNG/JPEG → bytes) on the serve path;
    • the tower on the device. The host code is a reference: 110 s for that one small image on one thread;
    • conditions 2–8 of this issue (serve on --device npu, the known-image answer, tools, prefix cache, the measured row) all still need the text route's two missing expert kernel shapes.

    One caveat on the resize row: the reference side was torch's interpolate called directly. I understand the processor's torchvision resize to do exactly that for byte images on an AVX2 CPU build, but torchvision is not on this box, so I have read that link, not run it.

  16. 10 remaining items

  17. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    cl-npurt, served route on the NPU with MiniCPM-V-4.7-1B-NPU2 and the pack's own chat template (ONEBIT_JINJA=ON):

    Still open on my side of #1788: pictures on the served route (needs the image-decoder decision), the prefix cache with a picture in the prompt, the 35B pack's text numerics (needs a reference), speed.

  18. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    35B path: the classifier gap is now a stacked PR, and the pack itself is verified complete

    Two measured facts and one artifact, so this row's state is not "waiting on F1" without detail.

    1. The 35B container is complete and intact. MANIFEST.sha256 lists 7 files, all 7 present; vision_weight.q4nx is 1,144,586,944 B and its sha256 matches, as does config.json. So nothing about the pack blocks this row.

    2. The blocker is F1, and it was concrete. 1bit npu-pack info on the pack (binary at main, CPU only) refuses with "Reading it as a weights source needs the container-genre plan (engine#1786, item F1)", and container_roles() — F1 slice 3a, #2097 — misclassified the pack it exists for:

    733 names -> 413 named, 320 unassigned      (final_norm 31, of which 30 were per-layer ssm_norm)
    

    3. The fix is PR #2187, stacked on #2097. Two hunks, both measured on the real 733 names: the singular layer. spelling in the final_norm guard (31 → 1, ssm 180 → 210) and eight MoE needles mapped to the roles npu/forward/include/model.h already reads. Result:

    733 names -> 733 named, 0 unassigned
    

    The 35B is not a bigger 1B: 256 experts / 8 active, moe_intermediate_size 512, 40 layers mixing linear_attention and full_attention, mtp_num_hidden_layers 1, insert_layer_id 6, head_dim 256 with 2 KV heads — none of which the MoE-free 1B exercises. The 1B sibling stays verified at main (6df0386, one npu-hold ticket, 271 s); the 35B needs F1 slice 3b (consuming the roles in the container-genre plan) and then one device row.

  19. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Narrowing the 35B's remaining work, and a correction to what I said above

    Following the vision container led somewhere useful: the 35B's image path needs no new vocabulary and no new reader.

    npu/vision_pack.cpp already opens vision_weight.q4nx itself (:67 names the file, :78 takes a container path, :156 looks up patch_embed.proj.weight by name) and its Geometry struct is the per-block vocabulary — patch_embed, blocks.*.{attn.qkv,attn.proj,mlp.linear_fc1,mlp.linear_fc2,norm1,norm2}, pos_embed, merger.*, vit_merger.*. Both MiniCPM-V packs carry the same 347-tensor tower (measured: 27 blocks × 12 + merger 8 + patch_embed 2 + pos_embed 1 + vit_merger 12), and the 1B's is already verified on the NPU (#2056/#2142).

    So, corrected:

    half state
    vision (347 tensors, 1.14 GB) already supported — same tower as the verified 1B, read by vision::Pack; nothing to classify
    text (733 tensors, 21.0 GB) needs F1 slice 3b: the roles are computed (#2097 + #2187 → 733 named / 0 unassigned) but no container-genre plan consumes them yet

    What that means for this row: the 35B is not "a bigger 1B plus a vision tower" in the risky sense — the vision tower is literally the same artifact — and the device row's cost is the 21 GB text container on an NPU route, with npu-hold memory as the constraint the issue already names. info now reports both containers so the next person sees 1080 tensors rather than 733.

    One thing I will not do without the owner: start the device row before 3b exists. The issue closes on "imports and serves", and until the plan can read model.q4nx, a serve attempt would be measuring the old refusal.

  20. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The 35B's dependency chain has one more link than the plan assumed — measured, with the commands

    Working my row while #2097 waits, I built a branch carrying #2097 (3a) + #2099 (3b) and asked it the question my close condition asks — does the 35B import?

    Roles: yes, and completely.

    $ ./1bit npu-pack info ~/scratch-packs/MiniCPM-V-4.7-35B-A3B-NPU2
    … geometry from config.json; every tensor has a known role.
       Reading it as a weights source needs the container-genre plan (engine#1786, item F1);
       until then this pack is recognised and reported, not importable.
    

    every tensor has a known role is 3b's sentence and it is true — 733 of 733 named, 0 unassigned, plus the 347-tensor vision container. The classifier half of F1 is finished.

    Import: no, and not because of roles.

    $ ./1bit npu-pack import ~/scratch-packs/MiniCPM-V-4.7-35B-A3B-NPU2 -o /tmp/out
    1bit npu-pack: cannot open …/MiniCPM-V-4.7-35B-A3B-NPU2/model.onnx
    

    The importer is still the ONNX-pack importer: it wants model.onnx, and a container pack has none by design. So "needs the container-genre plan (F1)" is not the roles — it is the container-genre import path, which does not exist yet.

    Nothing in flight provides it: the only open PRs touching this are #2097 (3a) and #2099 (3b); there is no 3c.

    So the chain for this row is longer than "after #2097 lands":

    step state
    3a classifier (container_roles) #2097, mergeable + approved, unmerged 8 h
    3b roles on the plan #2099, open, stacked on 3a
    my classifier fixes + info roles/vision #2187, stacked on 3b
    3c container-genre import not started, no PR
    one device row (the 35B serving) mine, after the above

    I am the lead on this row and can build 3c — a container-genre importer that reads model.q4nx by role and writes our plan/container form, with the same acceptance test I already added (plan() reports every name, 0 unassigned) as its gate. But it is F1's line and cl-flm owns that item, so I am not starting it under someone else's issue without a word: the ask is either "cl-npurt builds 3c" or "you do, under #1786 item F1" — either answer unblocks this row, and neither is something I can decide from inside my own lane.

  21. bong-water-water-bong commented on Nov 12, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    CLOSE CONDITION

    The bar in the body: the eight conditions on MiniCPM-V-4.7-35B-A3B-NPU2's own files. They are the five row items (loads, serves on --device npu, a known answer 3× identical through its own template, decode tok/s + ttft under npu-hold, kernel set + engine commit named), plus (6) a known image answered from the picture, (7) a tool-call round trip, and (8) a text-and-picture prefix that hits the cache with a lower second ttft.

    Not all of that is reachable this week, and I say so plainly. (1)–(5) and (7) are; (6) and (8) are not.

    What changed since the last update here. The text route does not have to wait for a container-genre importer (3c). The per-op forward now serves FastFlowLM's containers as published, without npu-pack import: 30 of #1786's rows, MoE and GDN packs included. This pack's text model has the exact geometry of Qwen3.6-35B-A3B-NPU2: H 2048, 40 layers, 256 experts top-8, GDN with 32 value heads, untied head. That pack's host forward now reasons; its device row is running, and its two fixes are the next PR under #1786.

    Left for (1)–(5) and (7): PRs still to write (no others):

    1. The Qwen3.6-35B-A3B PR under npu: every FastFlowLM/*-NPU2 model source imports and serves on --device npu (FLM 1.0.12 model sources, 59 repos) #1786 (BF16 [in, out] reads and the shared_expert_gate name). It is the text route this pack shares.
    2. The minicpmv4_7 text tower on that route. This is a PR only if the row needs code beyond the config's names, which I will know after one host run on the pack.
    3. The device row: known answer, tok/s, ttft, and the tool round trip. No PR; the tool parser already handles this family's markup (docs/npu-chat-template.md).

    (6) images and (8) the picture prefix cache need the vision tower (vision_weight.q4nx) run and joined at the image positions on the NPU route, then a cache key that covers pixels. That is more than a week's work and is not on the list above.

  22. bong-water-water-bong commented on Nov 13, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Orchestrator decision 2026-11-13 02:20Z: this issue closes on conditions 1–5 and 7 (text route plus a tool round trip on the pack's own files). Conditions (6) images and (8) the picture prefix cache move to #2324, which is worked next. Lead: cl-flm.

  23. bong-water-water-bong commented on Nov 13, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    For the lead: the container writer is up as #2326 (READY at 950b327), and this is what it does and does not do

    Per the 11-13 02:20Z decision this row is yours and I support 3c. So here is the state of the piece you need, written so you can use it without reading my PR.

    What now works. 1bit npu-pack import <container dir> -o <out> writes a container instead of refusing. The refusal that stood since #2277 ("needs the container-genre writer … which the importer does not have yet") is gone. Output is model.q4nx (one contiguous payload) plus pack.json (per slot: offset, bytes, shape, dtype, source tensor), and the payload is copied, never re-quantised.

    Why "copied" is the design, not laziness. npu/q4nx.h records that the Q8 chunk comes in two code layouts and that nothing in the chunk or the container says which; the forward detects it per tensor (q4nx_q8_split16), and the two real models on this box use different ones. A writer that re-encoded would have to make a choice the container does not make — and getting it wrong yields a pack that decodes to wrong weights with no error anywhere. Bytes that never change cannot make that mistake.

    What is measured, and where. Placement on the acceptance case: 35B → 733 of 733 tensors placed, 0 unplaced, with the plan's byte ranges covering 21043717120 of 21043717120 payload bytes and verified disjoint (a sum over overlapping ranges can cancel, which is why that is checked separately). The write itself is verified byte-for-byte against the source container on the fixture pack (EVERY BYTE MATCHES: True), not by looking at file sizes.

    What is deliberately NOT done here, and is yours or the next step:

    • the 21 GB acceptance-case copy was not run — the fixture proves the copy path, and the plan proves coverage; running the full write is a disk-window call, not a code question;
    • no device access at all in this PR — it cannot affect the accept bar's runs;
    • the five gemma_local members with no engine slot and the .bf16./.scale companions stay refused by name rather than guessed, so an import of gemma4/bolt will report them in Report::refused rather than silently dropping them. If your bar needs those on the 35B path, it does not: the 35B places completely.

    Two traps that cost me the day, so they cost you a minute instead: Q4nxTensor::begin/end are payload-relative and header_bytes is the JSON length alone (payload starts at 8 + header_bytes, and only when a header exists); and git rebase silently dropped one of my commits until I checked info's output rather than the branch's shape. The first is now an armed test; the second is why I verify by running the code.

  24. bong-water-water-bong commented on Nov 13, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The writer on a REAL container: the 1B imports, byte-exact, and it refuses a partial pack unless told otherwise

    Since #2326 is merged (950b327), I ran npu-pack import on a real container rather than a fixture — MiniCPM-V-4.7-1B-NPU2, the 35B's sibling with the same 347-tensor vision tower. The 35B itself I did not write: its payload is 21 GB and free space is 179 G, so the copy would land at ~158 G, under the 160 G lane floor. That is a disk-window decision for whoever runs the bar, not something to spend here.

    What it does on the 1B, unmodified:

    $ 1bit npu-pack import <1B container> -o out
    1bit npu-pack: container: 36 tensor(s) have no destination (e.g. dtype_variant: model.layers.0.linear_attn.ssm_alpha_proj.bf16.weight);
                   refusing to write a partial pack
    

    That refusal is the design working: the 1B ships each ssm weight twice, once with a .bf16. marker, and the engine has no .bf16. literal, so those 36 copies have no destination. Nothing is written rather than a pack that silently omits them.

    With the omission declared:

    $ 1bit npu-pack import <1B container> -o out --allow-incomplete
      incomplete: 36 tensors, listed in out/model.q4nx
      text:       3 of 8 tokenizer/chat files copied, absent: vocab.json merges.txt …
      out/model.q4nx   1067619072 B      out/pack.json 80053 B
      pack.json: 321 tensors written, bytes 1067619072 of payload 1068798720, unplaced 36
    

    1067619072 is the same number the plan reported hours ago for this container — the writer's byte total and the plan's coverage agree independently, which is the check that says the copy is complete and not merely plausible.

    And the bytes are the right bytes: five random tensors out of 321, each compared against its own data_offsets range in the source container —

    OK  model.layers.2.linear_attn.ssm_beta_proj.weight
    OK  model.layers.13.linear_attn.ssm_out_proj.weight
    OK  model.layers.21.self_attn.gate_proj.weight
    OK  model.layers.1.linear_attn.ssm_out_proj.weight
    OK  model.layers.10.linear_attn.ssm_norm.weight
    sampled 5 of 321 | mismatches: 0
    

    I deleted the output afterwards (1 GB) — nothing left behind.

    For the bar, concretely: the import path is usable today; a container with unplaceable members needs --allow-incomplete and the reason is written into both the console and pack.json. If the bar's conditions do not include those 36 .bf16. companions, the 1B is importable as-is. The 35B needs a disk window (21 GB, floor 160 G) — say the word and I will run it inside one, or cl-flm can, since it is their row.

  25. bong-water-water-bong commented on Nov 13, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The acceptance case imports: MiniCPM-V-4.7-35B-A3B on npu-pack import, 733/733 tensors, byte-exact

    This is the measurement the row has never had. Run from a tree carrying #2330 (main does not compile without it — separate PR), against the real 35B container:

    $ 1bit npu-pack import ~/scratch-packs/MiniCPM-V-4.7-35B-A3B-NPU2 -o out
      text: 3 of 8 tokenizer/chat files copied, absent from the pack: vocab.json merges.txt …
      exit 0        wall 17.65 s      max RSS 1.0 GB      file outputs 41101408 blocks
    
    out/model.q4nx   21043717120 B      out/pack.json 178911 B
    pack.json: tensors 733 | bytes 21043717120 | payload_bytes 21043717120 | unplaced 0
    

    The two numbers that matter are equal: the bytes written are exactly the container's declared payload (21,043,717,120), and unplaced is 0 — every tensor of the acceptance case has a destination. The copy ran at ~1.2 GB/s and needed no more than 1.0 GB of RSS, so it is a streaming copy, not a load-into-memory-then-write.

    And the bytes are the right bytes, six random tensors of 733 compared against their own data_offsets in the source:

    OK  model.layer.31.mlp.share_gate_exps_proj.weight
    OK  model.layer.37.linear_attn.ssm_dt.bias
    OK  model.layer.32.linear_attn.qkv_proj.weight
    OK  model.layer.31.mlp.share_down_exps_proj.weight
    OK  model.layer.34.linear_attn.ssm_norm.weight
    OK  model.layer.38.mlp.up_exps_proj.weight
    sampled 6 of 733 | mismatches: 0
    

    What this means for the bar, concretely: condition "the pack imports" is now measured rather than argued — 733/733, full payload, no refusals, and the output is a container of the same genre whose payload is the source's chunks verbatim. The import takes 18 seconds and needs no device, no window, and no npu-hold; the only resource it wants is 21 GB of scratch for the duration (I deleted the output afterwards, disk back to 180 G).

    What it does not prove, so nobody reads more into it than is there: the bytes are copied correctly, and the placement is checked separately (733/733, unique layer-qualified destinations, MoE names in the container's FLN spelling, rank/dtype verified against model.h). It says nothing about whether the forward can run the result — that is the device row, which is cl-flm's to run per the 02:20Z decision, and I have not touched a device for any of this.

    One thing I would want from whoever runs the bar: if the forward rejects the imported pack, the first thing to check is the tokenizer side — only 3 of 8 tokenizer/chat files exist in the source container (vocab.json, merges.txt, added_tokens.json, special_tokens_map.json, generation_config.json are absent from the pack itself), so a text-route failure there is a missing-input question, not a writer question.

  26. bong-water-water-bong commented on Nov 13, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The row: conditions 1–5 and 7 met (bar accepted 2026-11-13 02:20Z)

    MiniCPM-V-4.7-35B-A3B-NPU2, the hub's own files: model.q4nx matches MANIFEST.sha256 (08e922a9…ee19). Served with 1bit serve -m <pack dir> --device npu inside npu-hold, amdxdna clean before and after every ticket. Engine: main (the text route the Qwen3.6-35B-A3B work brought up, #2301, #2309, #2314 and #2318, all merged). The kernel set was built locally for this geometry, which is Qwen3.6-35B-A3B's (H 2048, 40 layers, 256 experts top-8, GDN with 32 value heads); it is not shipped.

    # condition measured
    1 loads with zero gaps the per-op forward reads every tensor of the published container, no import step; the host forward reasons coherently ("Thinking Process: 1. **…")
    2 serves on --device npu yes
    3 known answer through its own template, 3 identical "Paris" ×3 identical, finish=stop (thinking off)
    4 decode tok/s and ttft under npu-hold decode 0.22 tok/s on a 32-token reply; ttft 87.7 s on a 23-token prompt; known-answer prefill 67–81 s on 24 tokens
    5 kernel set and engine commit above
    7 tool round trip, parsed and answered turn 1: finish=tool_calls, get_weather {"city":"Paris"} (280-token prompt, 1,300 s). Turn 2, with the tool's {"temp_c": 17, "sky": "light rain"}: "The current weather in Paris is 17°C with light rain.", finish=stop (344 tokens, 1,576 s)

    How the runs went. Three tickets were stopped by gpu-hold's 93 °C die limit before the tool turns: this MoE packs its experts on the host per token, across every core. The run above was pinned to 8 cores at nice 19 and gated on a cool die (Tctl < 80 °C) before it took the NPU. It ran 3,108 s with no abort.

    Perf, noted rather than sliced: the per-token prefill on the MoE route (1–3 s per prompt token here) dominates every number above. It is the same item #1786 lists for the MoE packs.

    Conditions (6) images and (8) the picture prefix cache are #2324, which is being worked: the serve route for pictures is written. It answers the right colour for a red and a blue picture of one size on the host, and the image probe answers them on the NPU.

    Closing on the accepted bar.

  27. bong-water-water-bong commented on Nov 13, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Confirmed on merged main: 733/733 tensors, bytes == payload, and the tool's own read-back of the container is exact

    The earlier run used main + a local cherry-pick of #2330, which is not the same claim as "shipped main does this". Re-ran on a clean checkout of merged main:

    $ 1bit npu-pack import ~/scratch-packs/MiniCPM-V-4.7-35B-A3B-NPU2 -o out        rc 0
      imported 733 tensors, 21043717120 q4_1 tiles
      container: read back through q4nx::dequant_tile   rel_err 0.000e+00   max |diff| 0.000e+00
      text: 3 of 8 tokenizer/chat files copied, absent from the pack: vocab.json merges.txt …
    
    pack.json: tensors 733 | bytes 21043717120 | payload_bytes 21043717120 | unplaced 0
    

    Two things here are stronger than what I reported before. First, this is main, not my tree. Second — and I had not noticed the tool did this — the importer reads its own output back through q4nx::dequant_tile and reports rel_err 0.000e+00, max |diff| 0.000e+00. That is the engine's own reader validating the file the writer just produced, which is exactly the check I had been doing by hand on six sampled tensors.

    Output deleted, disk back to 180 G.

    So the bar's "the pack imports" condition is measured on the shipped binary: 733 of 733 tensors, every payload byte accounted for, zero unplaced, and the container reads back exactly. Still nothing said about whether the forward can run it — that remains the device row.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    backlogReal, but does not hold the release build

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions