Repository navigation
Description
F6: MiniCPM-V 4.7 35B-A3B (MoE vision) on our own NPU runtime
Owner 2026-11-09: "support all of it" — every FLM 1.0.12 behaviour on this chip, on our own runtime and kernels. This is
item F6: the first MoE vision model in that catalogue, and the heaviest single row in the plan (~22 GB).
Scope. openbmb/MiniCPM-V-4.7-35B-A3B — 35B parameters, ~3B active per token, text + images, thinking available (off by
default, as the model card recommends), tool calling, and a prefix cache that must also cover pictures. Context up to
256k with a 32k default. Memory ~22 GB, which is the constraint that dictates the order of work below.
Starting point, as the owner directed: we already serve a 35B-A3B MoE decode on the NPU, so the text route is a
starting point rather than new work. What is new is the vision tower, which ships as its own component inside the pack
(a separate vision_weight.q4nx next to model.q4nx), and the ingest path that feeds it.
Done = the plan's five conditions on the pack's own files (TNPU-ALL §1), plus the three behaviours this model adds:
- imports with zero gaps and passes its self-check against a named reference;
- serves on
--device npu; - answers a known-answer prompt through the model's own chat template, 3 identical requests returning identical text;
- a measured decode tok/s and time to first token under
npu-hold; - the row names the kernel set and the engine commit that produced it;
- images: one known image, one known question, an answer that is only obtainable from the picture — not "it accepted
the request"; - tools: a tool-call round trip parsed and answered;
- prefix cache: a repeated prefix (text and a picture) hits the cache and the second TTFT is measurably lower.
Order of work — the 22 GB pack is not the place to iterate:
- vision ingest on the small siblings first. Two 1B-class vision packs ship in the same catalogue with the same
vision_weight.q4nxshape, and they are the cheap place to get the image path, the preprocessor and the join right
before touching a 22 GB model. A wrong answer there costs a minute, not a memory gate. - the text route, from our existing 35B-A3B MoE decode lane — confirm the pack imports and decodes before any image
is involved, so a failure is attributable to one half. - the join: image embeddings into the text model at the right positions, then the known-image question.
- tools + prefix cache, then the measured row (decode tok/s, TTFT, cache-hit TTFT) under
npu-hold.
Constraints being honoured (TNPU-ALL §2): own runtime and own kernels, full ELFs only, npu-hold tickets of ≤9
minutes, one pack >8 GB at a time with free -g ≥ 40 GiB first, packs on disk, nothing redistributed, and the pack's
licence recorded on the row. Kernel and runtime internals stay in the private repos; this issue and any PR stay neutral.
Not in this issue: the text-only speed comparison against another build (that is F3/F9), and the other new 1.0.12
sizes (F7).
Found while working the catalogue item; no existing issue covers this model (searched MiniCPM, vision tower — only
census mentions and the unrelated CLIP-graph row).
Activity
bong-water-water-bong commentedon Nov 10, 2026
Pack facts, read from the published packs (no download — the box is at its memory and disk limits right now)
The three models are on HF ungated:
FastFlowLM/MiniCPM-V-4.7-35B-A3B-NPU2,MiniCPM-V-4.7-1B-NPU2,MiniCPM-OCR-1.0-NPU2. What the files say:MiniCPM-V-4.7-35B-A3B-NPU2MANIFEST.sha256)model.q4nx20,068.93 MiBvision_weight.q4nx1,091.56 MiBMiniCPM-V-4.7-1B-NPU2model.q4nx1,019.32 MiBvision_weight.q4nx1,082.56 MiBTwo things fall out of that table:
~22 GBis 20.07 GiB of text + 1.07 GiB of vision, not a rounded-up estimate.config.jsonof the 35B (flm_version: 1.0.12, so the pack is the version this item targets):The 1B sibling is the same family at hidden 1024 / 35 layers, and its card declares
flm_version: 1.0.13— so the catalogue has already moved past 1.0.12 for the small pack, which matters only if we pin packs by version.The image contract, from the model's own chat template
The template renders an image as the literal
<|image_pad|>(and a video as<|video_pad|>), and it raises on an image inside a system message. So the ingest path is: render → tokenise → find the<|image_pad|>positions → replace those token embeddings with the tower's output. That is the shape the row's condition 6 tests.Blocked on the box, not on the work — recorded rather than worked around
TNPU-ALL §2 requires
free -g≥ 40 GiB before taking a pack over 8 GB, and a 150 GB disk floor. Measured just now:So taking the 22 GB pack would put the box below its disk floor and it would not fit in 3 GiB of available RAM anyway, and even the 2.07 GiB 1B pack would cross the disk floor. I am not going to start a download that ends with a full disk or an OOM kill of whatever else is running. The item stays open and the order of work below is what happens as soon as either limit relaxes — I will re-check rather than assume.
Next concrete step when it does:
npu-pack infoon the 1B pack (device-free, host-side import) to see what our importer says about aminicpmv4_7container with a separate vision component — that is the first gap to name, and it costs one 2 GiB download instead of 21.bong-water-water-bong commentedon Nov 10, 2026
Where the gap actually is, from our importer's code (no pack needed)
npu/pack_import.cpphas no mention of a vision component at all —grep -in 'vision\|clip\|vit\|minicpm'over it returns nothing — so this is new work, as the catalogue delta predicted. But the seam is narrower than "a new container format", and that changes the size of the job:Our importer already knows about two-component packs, but in a different shape.
find_token_fconst()(aroundnpu/pack_import.cpp:553) looks for the Token.fconstat the pack root and then, per thecompositemodelrule, in the component directory thatgraph_filename()names:The MiniCPM-V packs are not shaped like that. Their file list is flat:
So the difference is not "a new container":
vision_weight.q4nxis the same q4nx container we already read formodel.q4nx, holding a different kind of content — a vision transformer instead of a decoder. What is genuinely new:compositemodelcomponent path, and imported as its own graph;<|image_pad|>positions in the rendered prompt (the model's own chat template writes them) must be replaced by the tower's output embeddings before the text prefill consumes them.That is what makes the 1B sibling the right first target: it is the same flat pair at 1/20th the text size, so items 1 and 3 can be got right for one 2.07 GiB download instead of a 21 GiB one.
Still blocked on the box
Measured after freeing 4 GB of my own re-creatable model files: RAM available 1 GiB (of 122), disk 152 G (floor 150). The 1B pack needs ~2.07 GiB on disk and the import needs headroom in RAM; neither is available. Nothing has been downloaded, and I will not start a transfer that ends in a full disk or an OOM kill of another lane's work. Next step the moment either relaxes:
npu-pack infoonMiniCPM-V-4.7-1B-NPU2— device-free, host-side, and it names the first refusal.bong-water-water-bong commentedon Nov 10, 2026
First measured step: the pack is downloaded and our loader names the first gap
MiniCPM-V-4.7-1B-NPU2fetched into~/scratch-packs(2.1 GiB, both components complete — packs go on disk per TNPU-ALL §2, never/tmp). Ran our own loader against it, device-free, under a 45 s timeout so nothing is left running:That is the text half's first gap, and it is not the model.
unifiedtakesmodel.q4nx+config.json+tokenizer.json— all present in this pack — plusnpu/, our kernel set. The pack ships FLM's own shims instead, so nothing aboutminicpmv4_7has been rejected yet; the route simply refuses a directory with no kernels in it.Two consequences for the order of work:
1bit npu-lax --model <dir> --kernels <dir>, which is documented as "Qwen3.6-35B-A3B Q4NX directory" — but MiniCPM-V 4.7 35B-A3B is a different model at a similar active-parameter count (256 experts, 8 per token, 40 layers, hidden 2048, hybrid 3-linear/1-full attention), so whether that kernel set accepts this geometry is the next question, not an assumption.vision_weight.q4nxstill has no consumer at all —grep -in 'vision\|vit\|clip\|minicpm'overnpu/finds nothing, so on the 1B pack the tower is 1,082.56 MiB of container nobody opens. That half is unaffected by the kernel question and can be developed on this pack alone.And a disk consequence worth recording before anyone tries the big one: the 35B pack is 20.68 GiB, and the box is at 154 G free against a 150 GB floor — taking it would land at ~133 G, below the floor. So the large pack needs ~20 GB released first, or a lane that is allowed to run closer to the floor than this plan permits. The 1B pack was the right first move for that reason as well.
Next:
npu-lax --model <1B dir> --kernels <built set>will tell us whether the text container maps without an importer in the path at all — and if it does, F6's text half is a kernel-geometry question rather than a format one.bong-water-water-bong commentedon Nov 10, 2026
The vision tower's full geometry, from the pack's own
config.json(device-free, no container parsing yet)The pack carries everything needed to size the work — and it is arithmetically self-consistent, which is the check that it is the tower's real geometry and not a default:
The token arithmetic that sizes the job:
So one image is a 4,900-row patch sequence through 27 ViT layers at embed 1152 — an 11k-token attention workload in the worst case, and it runs once per image, not per decoded token. That is a prefill-shaped profile, and it is the reason this is not "the decode kernel with a different weight matrix": the shapes (4,900 × 4,900 attention, no KV cache, no autoregressive step) are unlike anything the decode lane runs.
INSERT_LAYER_ID = 6is the join point named by the pack: the image embeddings enter the text model at layer 6, which is consistent with the<|image_pad|>substitution the chat template implies.DOWNSAMPLE_FACTOR 16andMERGER_HIDDEN_SIZE 4608are the merger between the tower and the text embedding width (1024 on this pack) — so the join is a projected, not a raw, splice.Two fields worth flagging for other items rather than this one:
mtp_num_hidden_layers(multi-token-prediction heads — F2's speculative decode territory, not F6's), andaddr_kv/addr_qk(address maps the pack states explicitly, which is unusual and worth reading before any layout assumption is made).Next device-free step: open
vision_weight.q4nxas a container and list its tensors — if it is the same q4nx format asmodel.q4nx(which our loader already maps), the ViT's weights are reachable without any new reader, and the remaining work is purely the graph plus kernels.bong-water-water-bong commentedon Nov 10, 2026
The vision container opened: same format, a metadata preamble, and 347 bf16 tensors that cross-check against the config
Read with a 20-line parser (8-byte little-endian length + JSON record per tensor — the same container our loader already maps for
model.q4nx), no vendor tooling:Three things follow, and the first is a kernel-level good-news result:
model.q4nxin exactly one way:vision_weight.q4nx's first record is__metadata__, whilemodel.q4nxstarts with tensor records directly. A reader that assumes "offset 0 is a tensor" will either fail or, worse, treat the metadata as a tensor. That is the concrete reader change, and it is one branch.One discrepancy worth recording rather than smoothing over: the metadata claims "BF16 (+233 F32 tensors, bf16-exact)", but every one of the 347 records carries
dtype: BF16. The consistent reading is that the 233 were F32 in the source and were stored as bf16 because they are bf16-exact (the metadata says so), so no precision was lost — but a reader should not take "all BF16" as proof that nothing was rounded. If it matters for the parity bar, the check is to re-read those 233 names from the source mmproj and compare bit patterns; that is the kind of thing to do before claiming bit-exactness, not after.Next: the graph.
patch_embed(2 tensors) +pos_embed+ 27 blocks ofqkv/proj/mlpat embed 1152, 16 heads × 72, over 4,900 patches per slice — prefill-shaped, no KV cache, one pass per image. And the merger pair (vit_merger12 +merger8) projects toMERGER_HIDDEN_SIZE 4608before the text model's width.bong-water-water-bong commentedon Nov 10, 2026
The tower's tensor shapes — and one place the config is wrong
Every shape read from the container (35 sampled, then the fc1 across all blocks):
The config is wrong about one shape, and it is the one a loader would allocate from.
config.jsondeclaresMINICPM_V_VISION_MLP_INTERMEDIATE_SIZE = 4304; the weights are 4608, uniformly across all 27 blocks. 4608 is alsoMERGER_HIDDEN_SIZEand exactly 4 × embed (1152), so the value in the container is the coherent one — but a reader that trusts the config and sizes an activation buffer from it would be 304 elements short of the truth on every block. Trust the tensor record, not the config, for anything allocated. That is worth stating in the code that reads this pack, because the failure mode is an overflow or a truncated row, not a refusal.Three consequences for the kernels, all now measured rather than assumed:
patch_embedis a Conv2d —[1152, 3, 14, 14]— so the vision path starts with a strided convolution, not a matrix multiply over pre-extracted patches. Either a real conv kernel or an im2col-style layout change, and the weight is bf16 like everything else.LayerNormwith bias, where the text side uses RMSNorm. Different normalisation → a different kernel, and a place where reusing the text norm would silently produce plausible numbers.merger.fc2: [1024, 4608]— it projects to exactly this pack's text hidden size (1024), which confirms the<|image_pad|>substitution is a projected splice atINSERT_LAYER_ID 6, not a raw embedding copy.So the ViT is: Conv2d patch embed → 27 × (LayerNorm → QKV → proj → LayerNorm → MLP 4608) at 4,900 patches → spatial merge (1,225 tokens) → 1 extra transformer block at 4608 → project to 1024. All bf16, one pass per image, no KV cache.
Next device-free step: the arithmetic against the source. The metadata carries
source_sha256and the source is an mmproj GGUF, so the tower's weights can be re-read from a source we already know how to load and compared tensor-by-tensor — which is the reference the bit-exactness gate needs, before any kernel is written.bong-water-water-bong commentedon Nov 10, 2026
Decisive: MiniCPM-V 4.7 35B-A3B's text backbone is Qwen3.6-35B-A3B, field for field — so the text half is the existing lane plus a vocab delta
Compared the pack's
config.jsonagainstQwen/Qwen3.6-35B-A3B'stext_config— the model ournpu-laxlane is documented for:Every architectural field matches, including the full 40-entry
layer_typeslist (3 linear-attention + 1 full-attention repeating). The only delta is the vocabulary, 176 tokens smaller, which shows up in exactly two tensors: the token embedding table and the lm-head. So F6's text half is not a new kernel set — it is our existing 35B-A3B MoE decode lane, with the embedding/lm-head row count taken from the pack (248,144) rather than from the geometry (248,320). A kernel set sized from the geometry would either read 176 rows past the table or mis-index the head; a kernel set sized from the container needs nothing else.That also explains the 22 GB: 20.07 GiB of text weights is a 256-expert MoE at this geometry, and the tower adds 1.07 GiB.
What is left, and it is now two separable pieces
is not an NPU model directory (the fast lane needs npu/)— the kernel directory, not the model — so the question is whether the existing set'slm_headand embedding kernels can be pointed at 248,144 rows, which is a parameter, not a redesign.Pack integrity, verified
The 35B pack ships a
MANIFEST.sha256and both components match it:That is the provenance a row needs, and it means the 21 GiB on disk is the pack as published rather than a partial fetch.
The tower's geometry is identical across the two packs
So the 1B is the dense sibling with the same tower — the cheapest place to get the tower right, exactly as planned, and its result transfers unchanged.
Next: point the existing 35B-A3B lane at the 35B pack with
--model <pack>and a kernel set, and see whether the vocab delta is the only thing that has to change. That needs annpu-holdticket, so it is the first step that touches the device.bong-water-water-bong commentedon Nov 10, 2026
Vision tower inventory, read from the pack's own manifest (1B pack), and two config fields that are stale
The container is one JSON manifest + a payload region, not a record stream: an 8-byte LE length, that many bytes of JSON whose top-level keys are the tensor names (plus
__metadata__), then the payload atdata_offsetsrelative to the payload start. My earlier note that "the first record is__metadata__" was a truncated reading —json.loadssucceeded on the 38,808-byte first block and I stopped there. Correcting it here because the manifest is the whole tensor table.347 tensors, all BF16. The vision tower is two merger stages, which is what I had wrong:
proj.weight/.bias[1152,3,14,14]Conv2d /[1152]attn.qkv[3456,1152],attn.proj[1152,1152],mlp.linear_fc1[4608,1152],linear_fc2[1152,4608],norm1/norm2[1152]vit_mergerattn.qkv[3456,1152],attn.proj,mlp.linear_fc1[17280,4608],linear_fc2[1152,17280],pre_norm[4608],norm1[1152]mergerpre_norm[4608],linear_fc1[4608,4608],linear_fc2[1024,4608],norm[1152]So
MINICPM_V_MERGER_HIDDEN_SIZE: 4608is 4 × 1152 — the 2×2 spatial merge feeding the merger, not a ViT MLP width.patch_embed.projcarries a bias, and there is no gate tensor anywhere in the MLP, so it is a plain fc1 → SiLU (hidden_act: silu) → fc2 two-layer MLP, not SwiGLU.The part that matters for a runtime
Two config fields disagree with the tensors in the same pack:
MINICPM_V_VISION_MLP_INTERMEDIATE_SIZE= 4304blocks.*.mlp.linear_fc1= 4608 rows (bias[4608])MINICPM_V_VIT_MERGER_INTERMEDIATE_SIZE= 17216vit_merger.mlp.linear_fc1= 17280 rowsBoth are stale for 4.7 (4608 = 4 × 1152 is the round number; 4304 is the 4.5-era value). A runtime that sizes the ViT MLP from the config allocates 4304 × 1152 and reads 4608 × 1152 — a 304-row overrun per weight, which lands in the neighbouring tensor's bytes and returns plausible numbers rather than faulting. That is engine#1883's finding arriving from the pack side, and it is why the rule for this tower is:
size every buffer from the tensor's own shape, never from
config.json. The config remains useful for what the tensors cannot say —patch_size 14,image_size 980,spatial_merge 2,max_slice_nums 9,insert_layer_id 6,num_position_embeddings 4900,layer_norm_eps 1e-6,rescale_factor 1/255with mean/std 0.5 — and the token count follows:980/14 = 70 → 4,900 patches → 1,225 after merge, ×9 slices = 11,025 worst case.Next step on my side is unchanged (needs a kernel set for this geometry), but the vision tower no longer needs one to be specified: 27 such blocks, one attention+MLP at 4608, one 4608→1024 projection, all BF16, all shapes above read from the manifest.
bong-water-water-bong commentedon Nov 10, 2026
Text side: the embedding matches
vocab_sizeexactly in both packs, and the head's blocked layout is derivable from its own bytesMeasured from each pack's manifest (
data_offsets, so the byte extent is the pack's, not my fit of the shape):config.jsonvocabmodel.embed_tokenslm_head[248094, 1024]BF16 — exact, no padding[7753, 4, 8704]I8[248144, 2048]BF16 — exact, no padding[7755, 8, 8704]I8So the #1878 direction (shortfall refused, padding tolerated) is satisfied trivially here: both embedding tables are exactly
[V, H], unquantised BF16. Worth stating because it meansvocab × H × 2is the right size for the device read in both packs — a check that would refuse a padded table would have been wrong, and a check that assumed equality would have been right for these two and wrong in general.The head's blocked layout, from its own extents.
prod(shape) == data_offsets[1] - data_offsets[0]exactly in both, and dividing by the tile count each shape encodes gives the same number:7753 × 4 = 31,012tiles →269,928,448 / 31,012 = 8704.0bytes per tile7755 × 8 = 62,040tiles →539,996,160 / 62,040 = 8704.0bytes per tile8704 is a constant of the layout, not of either model:
r × (H + H/16)withr = 8192/H.ri.e. every tile is 8 KiB of int8 codes plus 512 B of multipliers, one multiplier per 16 input columns — which is why the tile size does not move with H while its row count halves. Two independent packs agreeing on 8192 + 512 to the byte is the strongest form this kind of evidence takes.
Row padding exists and must be handled:
r × tilesis 248,096 for the 1B (2 rows past 248,094) and 248,160 for the 35B (16 rows past 248,144). Those rows are not tokens; emitting them or letting them contribute to a softmax is a silent wrong-answer path with no fault.And the two variants do not share a vocabulary: 248,094 versus 248,144 — a 50-row difference inside one family. Anything that builds a token table from one pack and applies it to the other is off by 50 ids at the tail, which lands exactly on the merge/special tokens an image prompt uses.
Next: the vision side is specified (previous comment) and the text side's head geometry is now measured, so the remaining gap for a device run is still a kernel set for this geometry, not a missing fact.
bong-water-water-bong commentedon Nov 10, 2026
The kernel-set blocker is not "no kernel set" — it is two shapes, and both are siblings of ones already on this box
My earlier note that the pack "ships FLM shims only and no
npu/kernel dir" is still true of the pack, but there are kernel sets on this host, and one of them —~/models/npu/qwen3-coder-elfs/+qwen3-coder-kernels/— shares this model's attention geometry exactly. My earlier sweep missed them because I grepped for the part names (lax_l,lax_a,ln,lm_head_q8) instead of for ELFs; the names arefull_i8_<op>_K<K>_N<N>.elf.Mapping this model's shapes (hidden 2048, 16 q heads × 256, 2 kv × 256,
moe_intermediate_size512) against what exists:K2048 N5120full_i8_QKV_K2048_N5120.elfK4096 N2048full_i8_O_K4096_N2048.elfK2048 N1024G_K2048_N1536,GU_K2048_N1536,U_K2048_N1536moe_inter768 × 2K512 N2048D_K768_N2048(K wrong),KB_K512_N256(N wrong)K2048 N≈248,144So the attention half of a device run is already available on this host, and the gap is exactly two expert shapes, each one parameter away from an existing sibling:
GU_K2048_N1536→N1024(512 instead of 768) andD_K768_N2048→K512. Both of the dimensions involved (K=512,N=1024) are already exercised by other kernels in the same directory (KB_K512_N256,D_K3072_N1024), so neither is a new code path — they are the same kernels at a different tiling.Two caveats before anyone treats this as an unblock. Kernel existence is not kernel correctness for this model: the set was generated for a different family, so
num_layers(this model has 40, withfull_attention_interval 4— so 10 full-attention and 30 linear-attention layers), the RoPE variant and the eps must all agree with what the loader reads, and the linear-attention layers have no counterpart in a Qwen3-Coder set at all. And the vision tower needs no such set only if it runs on pi-npufeat, which is not the path I have measured.Next step I can take without a device: confirm the loader reads the layer kinds per layer from
config.json(layer_types, which I have already verified is field-identical to Qwen3.6-35B-A3B) rather than from the kernel set's naming, since a 40-layer model with 10/30 attention split cannot be served by a set built for a uniform family.bong-water-water-bong commentedon Nov 10, 2026
Second pair of hands on this issue (lane cl-npurt, owner 11-10 21:40Z). The slice I am taking, so it does not overlap the claiming lane:
The thread above has established the facts about
vision_weight.q4nxby hand (Python, 20-line parser). Nothing in the engine opens that file yet. I am taking the device-free vision half, in this order:vision_weight.q4nxwith the engine's own code, account for every tensor (zero gaps is done-condition 1), derive the tower geometry from the tensors' own shapes, and refuse by name what it cannot place. The two stale config fields found above (MLP_INTERMEDIATE_SIZE4304 vs 4608,VIT_MERGER_INTERMEDIATE_SIZE17216 vs 17280) become a tested rule in code rather than a note in a comment. Surfaced as a1bit npu-packsubcommand so the result is a command anyone can run on both packs.[4900, 1152]position table used directly, and thevit_mergerstage. That one needs a named oracle before I claim anything about its numbers.I am not touching: the text route and its kernel set (the two missing expert shapes in the last comment), anything on the device, the chat-template/
<|image_pad|>join, tools, or the prefix cache. Those stay with the claiming lane.One unblock already done for step 2's base: #1930 had its READY verdicts but conflicted with main again, so the queue could not merge it. I pushed a merge of main (
d26d52d, one conflict in a test'smain(), both sides kept, 5 of 5 tests pass) and it is mergeable again.bong-water-water-bong commentedon Nov 10, 2026
First PR of the vision slice is up: #2040 (
npu/vision_pack.h,1bit npu-pack vision <pack dir>), ready for review atd51f4ef. The engine now opensvision_weight.q4nxitself.Measured with it on both published packs, device-free:
__metadata__provenanceOne correction to the thread above. The 05:25Z comment says the tower is "identical between both MiniCPM packs … the same weights with alignment rounding". The geometry is identical; the weights are not. I hashed every tensor in both containers: 0 of 347 are byte-identical, including the blocks, which have the same shapes in both. The size difference (1,082.56 vs 1,091.56 MiB) is the last projection,
[1024, 4608]vs[2048, 4608]. So one implementation serves both packs, but a numerical reference built for the 1B says nothing about the 35B's numbers, and the 35B's container names no source to build one from.What remains of the vision half, in order:
[4900, 1152]position table and thevit_mergerstage (attention at 1152, then an MLP over 2×2 groups at 4608). The exact order of operations (wherevit_mergersits relative to blockinsert_layer_id6, how positions are assigned for a non-square slice, which activation) has to come from the model's own reference code, not from the tensor names. I have not established it and am not guessing it.mmproj-model-f16.ggufwith a SHA-256, which is the candidate; it is not on this box.<|image_pad|>join. The join belongs with the text route.Unchanged and not mine: the text route's kernel set (two expert shapes missing), tools, prefix cache, the measured row.
bong-water-water-bong commentedon Nov 10, 2026
Done-condition 1 for the vision component: the self-check against a named reference, bit for bit, on both packs. And a correction to my own comment above and to the thread.
The 1B container's
__metadata__namesmmproj-model-f16.ggufand a SHA-256. That file is inopenbmb/MiniCPM-V-4.7-1B-gguf; downloaded,sha256sum=fb9cc047a3cd…165b25a900, the pack's stated hash. I rebuilt every pack tensor from the GGUF and compared BF16 bit patterns:openbmb/MiniCPM-V-4.7-35B-A3B-ggufmmproj (bff3b688…5d20f1)All 347 of 347 in both packs. Every F32 source tensor is exactly representable in BF16 (0 exceptions), which settles the "+233 F32 tensors, bf16-exact" question raised at 03:31Z.
The correction:
config.jsonis not stale. The thread (04:10Z, 09:34Z) and my first commit said the config was wrong about the MLP widths (4304 vs 4608, 17216 vs 17280). The source GGUF's ffn tensors are 4304 and 17216 wide. The container stores them padded to the next multiple of 128, and the 84 "padding" tensors above are exactly those: the source's values in the leading extent,+0.0everywhere past it (fc1 rows, fc1 bias entries, fc2 columns, in all 27 blocks and thevit_merger). So the rule "size every buffer from the tensor" stands, but the reason is padding, and a forward can stop at 4304. #2040 now verifies the zero tail and carries both widths (heada8c38f4).Name mapping, established by the comparison:
blocks.N.attn.qkvv.blk.N.attn_q,attn_k,attn_vstacked in that orderblocks.N.mlp.linear_fc1/linear_fc2v.blk.N.ffn_up/ffn_down(padded)vit_merger.pre_norm,mlp.linear_fc1/2v.vit_merger.ds_ln,ds_ffn_up/down(padded)merger.normv.post_lnmerger.pre_norm,linear_fc1,linear_fc2mm.input_norm,mm.up,mm.downTwo facts for whoever does the join and the forward:
v.tok_embd_sep,[4, hidden](four embedding rows at the text model's width). I have not looked for it inmodel.q4nx.clip.projector_type minicpmv4_7,clip.vision.wa_layer_indexes [6],clip.use_gelu True,clip.vision.image_size 448,projector.scale_factor 4. Our llama.cpp fork (pin5514ebe) has aminicpmv4_6projector with avit_mergergraph and nominicpmv4_7, so there is no forward oracle for this tower on the box today. That is the blocker for a host reference of the tower's output; I am not writing a forward against an oracle that does not exist.The comparison was a script run by hand against 2.2 GB of source files in my lane scratch; it is described in the PR and the docs, not shipped as a test.
bong-water-water-bong commentedon Nov 10, 2026
The host reference for the tower exists and matches the model's own implementation: #2056 (ready for review at
0b634fd). #2040 (the reader) merged at 22:42Z.I said two comments up that there was no forward oracle on the box. There is one now: the tower is in the transformers library itself (
transformers.models.minicpmv4_7), so I loaded itsMiniCPMV4_7VisionModeland merger with each pack's weights (strict load, every key matched) and comparedvision::tower_forward()with it on the same input:What the tower actually is, since the thread guessed at parts of it:
floor(y·70/gh),floor(x·70/gw)): no interpolation, no rotary term;hidden_act: siluis the text model's, not the tower's);vit_mergerruns once, after block 6: attention inside each 2 × 2 window, then an MLP over the four tokens plus their mean, halving the grid. Blocks 7–26 run on the halved grid;gh × gwpatches becomes(gh/4) × (gw/4)rows: a full 980-pixel slice is 70 × 70 patches... which is not a multiple of 4. The reference requires divisibility at each merge, so the preprocessor must pick grids that are multiples of 4; the "1,225 tokens per slice" arithmetic earlier in this thread (35 × 35) cannot be right for this model. I have not read the preprocessor yet and am not stating what it picks.What remains of the vision half:
processing_minicpmv4_7.pyin the same library is the reference.<|image_pad|>rows, the slice separator tokens, positions on the text side). With the text route's owner.Not started by me: 1. I will take it next unless the claiming lane says it has it.
bong-water-water-bong commentedon Nov 10, 2026
The image path before the tower is written and matches the reference end to end: #2066 (stacked on #2056, ready for review at
44b5438).With #2040 (merged), #2056 and #2066 the device-free vision half goes from RGB bytes to the rows the text model receives, each step checked against the model's own implementation:
interpolateon uint8, 11 casesThe token arithmetic for this model, corrected (the thread had 1,225 per slice and 11,025 worst case):
(h/56) × (w/56)rows;That matters for the text route: the prefill for one picture is a few hundred rows, not eleven thousand.
Not done, and where each sits:
<image>/<slice>markers, the placeholder rows,v.tok_embd_sep, positions): the join, with the text route's lane;--device npu, the known-image answer, tools, prefix cache, the measured row) all still need the text route's two missing expert kernel shapes.One caveat on the resize row: the reference side was torch's
interpolatecalled directly. I understand the processor's torchvision resize to do exactly that for byte images on an AVX2 CPU build, but torchvision is not on this box, so I have read that link, not run it.10 remaining items
bong-water-water-bong commentedon Nov 11, 2026
cl-npurt, served route on the NPU with
MiniCPM-V-4.7-1B-NPU2and the pack's own chat template (ONEBIT_JINJA=ON):Paris,finish_reason: stop, three identical requests, 24 prompt tokens, prefill 2.0-2.3 s.toolsnever reached the template (25 prompt tokens where the reference renders 292, so the model answered in prose); the call form this template teaches (<function=…><parameter=…>) was not parsed; an earlier call's arguments string made the second turn a 400. With forward-serve: a request's tools reach the chat template — MiniCPM-V 4.7 answered a tool request in prose, now a call and a round trip (Refs #1788) #2172: first replyfinish_reason: tool_calls,get_weather {"city":"Paris"}; with the tool result appended: "The weather in Paris is 17 degrees Celsius with light rain.", prefix cache 291 of 376 tokens.Still open on my side of #1788: pictures on the served route (needs the image-decoder decision), the prefix cache with a picture in the prompt, the 35B pack's text numerics (needs a reference), speed.
forward-serve: a request's tools reach the chat template, and the fun…
bong-water-water-bong commentedon Nov 11, 2026
35B path: the classifier gap is now a stacked PR, and the pack itself is verified complete
Two measured facts and one artifact, so this row's state is not "waiting on F1" without detail.
1. The 35B container is complete and intact.
MANIFEST.sha256lists 7 files, all 7 present;vision_weight.q4nxis 1,144,586,944 B and itssha256matches, as doesconfig.json. So nothing about the pack blocks this row.2. The blocker is F1, and it was concrete.
1bit npu-pack infoon the pack (binary at main, CPU only) refuses with "Reading it as a weights source needs the container-genre plan (engine#1786, item F1)", andcontainer_roles()— F1 slice 3a, #2097 — misclassified the pack it exists for:3. The fix is PR #2187, stacked on #2097. Two hunks, both measured on the real 733 names: the singular
layer.spelling in thefinal_normguard (31 → 1,ssm180 → 210) and eight MoE needles mapped to the rolesnpu/forward/include/model.halready reads. Result:The 35B is not a bigger 1B: 256 experts / 8 active,
moe_intermediate_size512, 40 layers mixinglinear_attentionandfull_attention,mtp_num_hidden_layers1,insert_layer_id6,head_dim256 with 2 KV heads — none of which the MoE-free 1B exercises. The 1B sibling stays verified at main (6df0386, onenpu-holdticket, 271 s); the 35B needs F1 slice 3b (consuming the roles in the container-genre plan) and then one device row.bong-water-water-bong commentedon Nov 11, 2026
Narrowing the 35B's remaining work, and a correction to what I said above
Following the vision container led somewhere useful: the 35B's image path needs no new vocabulary and no new reader.
npu/vision_pack.cppalready opensvision_weight.q4nxitself (:67names the file,:78takes a container path,:156looks uppatch_embed.proj.weightby name) and itsGeometrystruct is the per-block vocabulary —patch_embed,blocks.*.{attn.qkv,attn.proj,mlp.linear_fc1,mlp.linear_fc2,norm1,norm2},pos_embed,merger.*,vit_merger.*. Both MiniCPM-V packs carry the same 347-tensor tower (measured: 27 blocks × 12 + merger 8 + patch_embed 2 + pos_embed 1 + vit_merger 12), and the 1B's is already verified on the NPU (#2056/#2142).So, corrected:
vision::Pack; nothing to classify#2097+ #2187 →733 named / 0 unassigned) but no container-genre plan consumes them yetWhat that means for this row: the 35B is not "a bigger 1B plus a vision tower" in the risky sense — the vision tower is literally the same artifact — and the device row's cost is the 21 GB text container on an NPU route, with
npu-holdmemory as the constraint the issue already names.infonow reports both containers so the next person sees 1080 tensors rather than 733.One thing I will not do without the owner: start the device row before 3b exists. The issue closes on "imports and serves", and until the plan can read
model.q4nx, a serve attempt would be measuring the old refusal.flm_answers: an answer also goes when its tracking issue closes unser…
census: an answered FastFlowLM gap stays answered, and the d3 pin wat…
bong-water-water-bong commentedon Nov 11, 2026
The 35B's dependency chain has one more link than the plan assumed — measured, with the commands
Working my row while #2097 waits, I built a branch carrying #2097 (3a) + #2099 (3b) and asked it the question my close condition asks — does the 35B import?
Roles: yes, and completely.
every tensor has a known roleis 3b's sentence and it is true — 733 of 733 named, 0 unassigned, plus the 347-tensor vision container. The classifier half of F1 is finished.Import: no, and not because of roles.
The importer is still the ONNX-pack importer: it wants
model.onnx, and a container pack has none by design. So "needs the container-genre plan (F1)" is not the roles — it is the container-genre import path, which does not exist yet.Nothing in flight provides it: the only open PRs touching this are #2097 (3a) and #2099 (3b); there is no 3c.
So the chain for this row is longer than "after #2097 lands":
container_roles)inforoles/visionI am the lead on this row and can build 3c — a container-genre importer that reads
model.q4nxby role and writes our plan/container form, with the same acceptance test I already added (plan()reports every name, 0 unassigned) as its gate. But it is F1's line and cl-flm owns that item, so I am not starting it under someone else's issue without a word: the ask is either "cl-npurt builds 3c" or "you do, under #1786 item F1" — either answer unblocks this row, and neither is something I can decide from inside my own lane.bong-water-water-bong commentedon Nov 12, 2026
CLOSE CONDITION
The bar in the body: the eight conditions on
MiniCPM-V-4.7-35B-A3B-NPU2's own files. They are the five row items (loads, serves on--device npu, a known answer 3× identical through its own template, decode tok/s + ttft undernpu-hold, kernel set + engine commit named), plus (6) a known image answered from the picture, (7) a tool-call round trip, and (8) a text-and-picture prefix that hits the cache with a lower second ttft.Not all of that is reachable this week, and I say so plainly. (1)–(5) and (7) are; (6) and (8) are not.
What changed since the last update here. The text route does not have to wait for a container-genre importer (3c). The per-op forward now serves FastFlowLM's containers as published, without
npu-pack import: 30 of #1786's rows, MoE and GDN packs included. This pack's text model has the exact geometry ofQwen3.6-35B-A3B-NPU2: H 2048, 40 layers, 256 experts top-8, GDN with 32 value heads, untied head. That pack's host forward now reasons; its device row is running, and its two fixes are the next PR under #1786.Left for (1)–(5) and (7): PRs still to write (no others):
Qwen3.6-35B-A3BPR under npu: every FastFlowLM/*-NPU2 model source imports and serves on --device npu (FLM 1.0.12 model sources, 59 repos) #1786 (BF16[in, out]reads and theshared_expert_gatename). It is the text route this pack shares.minicpmv4_7text tower on that route. This is a PR only if the row needs code beyond the config's names, which I will know after one host run on the pack.docs/npu-chat-template.md).(6) images and (8) the picture prefix cache need the vision tower (
vision_weight.q4nx) run and joined at the image positions on the NPU route, then a cache key that covers pixels. That is more than a week's work and is not on the list above.bong-water-water-bong commentedon Nov 13, 2026
Orchestrator decision 2026-11-13 02:20Z: this issue closes on conditions 1–5 and 7 (text route plus a tool round trip on the pack's own files). Conditions (6) images and (8) the picture prefix cache move to #2324, which is worked next. Lead: cl-flm.
bong-water-water-bong commentedon Nov 13, 2026
For the lead: the container writer is up as #2326 (READY at
950b327), and this is what it does and does not doPer the 11-13 02:20Z decision this row is yours and I support 3c. So here is the state of the piece you need, written so you can use it without reading my PR.
What now works.
1bit npu-pack import <container dir> -o <out>writes a container instead of refusing. The refusal that stood since #2277 ("needs the container-genre writer … which the importer does not have yet") is gone. Output ismodel.q4nx(one contiguous payload) pluspack.json(per slot: offset, bytes, shape, dtype, source tensor), and the payload is copied, never re-quantised.Why "copied" is the design, not laziness.
npu/q4nx.hrecords that the Q8 chunk comes in two code layouts and that nothing in the chunk or the container says which; the forward detects it per tensor (q4nx_q8_split16), and the two real models on this box use different ones. A writer that re-encoded would have to make a choice the container does not make — and getting it wrong yields a pack that decodes to wrong weights with no error anywhere. Bytes that never change cannot make that mistake.What is measured, and where. Placement on the acceptance case: 35B → 733 of 733 tensors placed, 0 unplaced, with the plan's byte ranges covering
21043717120 of 21043717120payload bytes and verified disjoint (a sum over overlapping ranges can cancel, which is why that is checked separately). The write itself is verified byte-for-byte against the source container on the fixture pack (EVERY BYTE MATCHES: True), not by looking at file sizes.What is deliberately NOT done here, and is yours or the next step:
gemma_localmembers with no engine slot and the.bf16./.scalecompanions stay refused by name rather than guessed, so an import of gemma4/bolt will report them inReport::refusedrather than silently dropping them. If your bar needs those on the 35B path, it does not: the 35B places completely.Two traps that cost me the day, so they cost you a minute instead:
Q4nxTensor::begin/endare payload-relative andheader_bytesis the JSON length alone (payload starts at8 + header_bytes, and only when a header exists); andgit rebasesilently dropped one of my commits until I checkedinfo's output rather than the branch's shape. The first is now an armed test; the second is why I verify by running the code.bong-water-water-bong commentedon Nov 13, 2026
The writer on a REAL container: the 1B imports, byte-exact, and it refuses a partial pack unless told otherwise
Since #2326 is merged (
950b327), I rannpu-pack importon a real container rather than a fixture — MiniCPM-V-4.7-1B-NPU2, the 35B's sibling with the same 347-tensor vision tower. The 35B itself I did not write: its payload is 21 GB and free space is 179 G, so the copy would land at ~158 G, under the 160 G lane floor. That is a disk-window decision for whoever runs the bar, not something to spend here.What it does on the 1B, unmodified:
That refusal is the design working: the 1B ships each ssm weight twice, once with a
.bf16.marker, and the engine has no.bf16.literal, so those 36 copies have no destination. Nothing is written rather than a pack that silently omits them.With the omission declared:
1067619072is the same number the plan reported hours ago for this container — the writer's byte total and the plan's coverage agree independently, which is the check that says the copy is complete and not merely plausible.And the bytes are the right bytes: five random tensors out of 321, each compared against its own
data_offsetsrange in the source container —I deleted the output afterwards (1 GB) — nothing left behind.
For the bar, concretely: the import path is usable today; a container with unplaceable members needs
--allow-incompleteand the reason is written into both the console andpack.json. If the bar's conditions do not include those 36.bf16.companions, the 1B is importable as-is. The 35B needs a disk window (21 GB, floor 160 G) — say the word and I will run it inside one, or cl-flm can, since it is their row.bong-water-water-bong commentedon Nov 13, 2026
The acceptance case imports: MiniCPM-V-4.7-35B-A3B on
npu-pack import, 733/733 tensors, byte-exactThis is the measurement the row has never had. Run from a tree carrying #2330 (main does not compile without it — separate PR), against the real 35B container:
The two numbers that matter are equal: the bytes written are exactly the container's declared payload (21,043,717,120), and
unplacedis 0 — every tensor of the acceptance case has a destination. The copy ran at ~1.2 GB/s and needed no more than 1.0 GB of RSS, so it is a streaming copy, not a load-into-memory-then-write.And the bytes are the right bytes, six random tensors of 733 compared against their own
data_offsetsin the source:What this means for the bar, concretely: condition "the pack imports" is now measured rather than argued — 733/733, full payload, no refusals, and the output is a container of the same genre whose payload is the source's chunks verbatim. The import takes 18 seconds and needs no device, no window, and no
npu-hold; the only resource it wants is 21 GB of scratch for the duration (I deleted the output afterwards, disk back to 180 G).What it does not prove, so nobody reads more into it than is there: the bytes are copied correctly, and the placement is checked separately (733/733, unique layer-qualified destinations, MoE names in the container's FLN spelling, rank/dtype verified against
model.h). It says nothing about whether the forward can run the result — that is the device row, which is cl-flm's to run per the 02:20Z decision, and I have not touched a device for any of this.One thing I would want from whoever runs the bar: if the forward rejects the imported pack, the first thing to check is the tokenizer side — only 3 of 8 tokenizer/chat files exist in the source container (
vocab.json,merges.txt,added_tokens.json,special_tokens_map.json,generation_config.jsonare absent from the pack itself), so a text-route failure there is a missing-input question, not a writer question.bong-water-water-bong commentedon Nov 13, 2026
The row: conditions 1–5 and 7 met (bar accepted 2026-11-13 02:20Z)
MiniCPM-V-4.7-35B-A3B-NPU2, the hub's own files:model.q4nxmatchesMANIFEST.sha256(08e922a9…ee19). Served with1bit serve -m <pack dir> --device npuinsidenpu-hold, amdxdna clean before and after every ticket. Engine:main(the text route the Qwen3.6-35B-A3B work brought up, #2301, #2309, #2314 and #2318, all merged). The kernel set was built locally for this geometry, which is Qwen3.6-35B-A3B's (H 2048, 40 layers, 256 experts top-8, GDN with 32 value heads); it is not shipped.--device npunpu-holdfinish=tool_calls,get_weather {"city":"Paris"}(280-token prompt, 1,300 s). Turn 2, with the tool's{"temp_c": 17, "sky": "light rain"}: "The current weather in Paris is 17°C with light rain.", finish=stop (344 tokens, 1,576 s)How the runs went. Three tickets were stopped by gpu-hold's 93 °C die limit before the tool turns: this MoE packs its experts on the host per token, across every core. The run above was pinned to 8 cores at nice 19 and gated on a cool die (Tctl < 80 °C) before it took the NPU. It ran 3,108 s with no abort.
Perf, noted rather than sliced: the per-token prefill on the MoE route (1–3 s per prompt token here) dominates every number above. It is the same item #1786 lists for the MoE packs.
Conditions (6) images and (8) the picture prefix cache are #2324, which is being worked: the serve route for pictures is written. It answers the right colour for a red and a blue picture of one size on the host, and the image probe answers them on the NPU.
Closing on the accepted bar.
bong-water-water-bong commentedon Nov 13, 2026
Confirmed on merged main: 733/733 tensors, bytes == payload, and the tool's own read-back of the container is exact
The earlier run used
main + a local cherry-pickof #2330, which is not the same claim as "shipped main does this". Re-ran on a clean checkout of merged main:Two things here are stronger than what I reported before. First, this is main, not my tree. Second — and I had not noticed the tool did this — the importer reads its own output back through
q4nx::dequant_tileand reportsrel_err 0.000e+00,max |diff| 0.000e+00. That is the engine's own reader validating the file the writer just produced, which is exactly the check I had been doing by hand on six sampled tensors.Output deleted, disk back to 180 G.
So the bar's "the pack imports" condition is measured on the shipped binary: 733 of 733 tensors, every payload byte accounted for, zero unplaced, and the container reads back exactly. Still nothing said about whether the forward can run it — that remains the device row.