You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
npu: MiniCPM-V 4.7 images and the picture prefix cache on --device npu (split from #1788: conditions 6 and 8) #2324
Split from #1788 by the orchestrator's decision of 2026-11-13: #1788 closes on conditions 1–5 and 7 (the text route and a tool round trip on MiniCPM-V-4.7-35B-A3B-NPU2's own files). This issue holds the two conditions it no longer carries. It is worked next.
Scope. On --device npu, through 1bit serve, on the packs' own files:
(6) images: one known image and one known question, answered with something only the picture gives (not "the request was accepted"). This covers MiniCPM-V-4.7-35B-A3B-NPU2, after its smaller siblings in the same catalogue (MiniCPM-V-4.7-1B-NPU2, MiniCPM-OCR-1.0-NPU2), which ship the same vision_weight.q4nx shape and are the cheap place to get the image path right first.
(8) picture prefix cache: a repeated prefix of text plus a picture hits the cache, and the second request's ttft is measurably lower.
What exists.npu/vision_pack.cpp opens vision_weight.q4nx (#1788's 11-11 notes). The 35B's text tower serves on the per-op forward as published, using the same route as Qwen3.6-35B-A3B-NPU2 (#2318), which has the same text geometry.
What is missing. Running the vision tower on the NPU route, joining its embeddings at the image positions of the prompt, and a prefix-cache key that covers the pixels.
Done = (6) and (8), each measured under npu-hold, with the row naming the kernel set and the engine commit, as #1786's rows do.
Claimed (pi-pkg), and the three missing pieces are now pinned to file:line — one of them is a correctness hazard rather than a missing feature. No device run needed for any of this; it is all readable on main today.
The pack is ready
~/scratch-packs/MiniCPM-V-4.7-35B-A3B-NPU2/
model.q4nx 21,043,802,648 B the text tower
vision_weight.q4nx 1,144,586,944 B the vision component
config.json tokenizer.json chat_template.jinja
So "on the packs' own files" has its inputs. The gap is in the engine.
(6) images — two missing links, not one
The HTTP route has no image input on the NPU path.app/ contains zero occurrences of image_url, data:image or a base64 content part. serve.cpp:173 has --mmproj described as "a vision (or audio) projector: image parts in chat messages", but serve.cpp:1357 says --mmproj is HRX0 and "HRX vision is an open RFC RFC: HRX (Loom) + NPU, and Vulkan out once HRX meets the gates #213 gate, to be checked on ZAYA1-VL" — i.e. the flag exists for a different route and is unverified there.
The vision tower has no execution entry on the NPU route.npu/vision_pack.h exposes exactly one function, print_info(pack_dir). Nothing runs it; the issue's "running the vision tower on the NPU route, joining its embeddings at the image positions" is therefore a new entry point, not a wiring change.
(8) picture prefix cache — this is the one I would fix first, and it is not only a miss
app/npu_prefix_cache.h matches a slot on token ids alone:
structPrefixSlot { std::vector<int> ids; ... }; // the tokens this slot covers, from position 0
PrefixDecision lookup(const std::vector<int>& ids) // the longest live slot whose tokens are a prefix of ids// line 43: "The rule reads token ids only, so it needs no knowledge of the template or of what a system prompt is."
With a picture in the prompt, two requests whose text is the same and whose image placeholders occupy the same token positions are indistinguishable to this lookup — so the second one reuses KV state built from the first picture's pixels. That is a wrong answer returned with a cache hit, not a slow path: today the rule cannot tell "same prefix" from "different picture, same text". A key over the pixels (a digest of the image ids or of the encoder's pooled output) is not an optimisation to add later; it is the condition under which the cache may be used at all once images exist.
Suggested order, smallest first
(8)'s key — the cache is already there and its rule is one function; adding the picture to the key is the smallest change that makes image-bearing requests safe, and it needs no vision execution to land.
(6)'s input — parse one image content part on the NPU route and carry it (PPM is enough; the probe already takes --image f.ppm).
(6)'s tower — the vision entry point in npu/vision_pack.*, then joining at the image positions.
What I am not claiming
I have not run any of this on the device, and I have not read npu/vision_pack.cpp's internals beyond its public surface. The measurement above is git grep on origin/main plus the pack's file sizes. Before a run I will state the command, the kernel set and the engine commit, per this issue's own "Done" line.
Correction to my own comment above, and it changes the row's estimate: the vision chain is not missing. Seven files of it are on main, and the probe already drives all of them.
I said "the vision tower has no execution entry on the NPU route" and "running the vision tower … is therefore a new entry point, not a wiring change". The first half is right about serve; the second half is wrong about the engine. What exists, verified by path this time:
npu/vision_image.{h,cpp} image bytes -> pieces
npu/vision_plan.h ImagePlan / PiecePlan: the patch geometry and the row count
npu/vision_tower.{h,cpp} tower_forward(const Pack&, const float* pixels, int gh, int gw, std::vector<float>* tower = nullptr)
npu/vision_prompt.h ImageMarkers + image_tokens(plan, markers): the placeholder ids and their positions
npu/vision_pack.{h,cpp} the container's vision component (vision_weight.q4nx)
And npu/forward/minicpm_image_probe.cpp already runs the whole chain end to end:
"the text model: step_batch_embeds() with the tower's rows where the placeholders are, then greedy decode" — and it prints image H x W: N piece(s), M rows; preprocess X ms, **tower (host)** Y ms, with --dump writing ids.i32, positions.i32, pooled.f32 for bisection.
So the joining at the image positions exists too (step_batch_embeds with the tower's rows is what the probe calls), and --chunk N already sends the prompt the way the served route does.
What is actually missing, narrowed
(6) the serve wiring
app/ has zeroimage_url / data:image / base64 content parts, so 1bit serve cannot be given a picture on any route. The NPU route needs the content-part parse and the call into the chain above. --mmproj (serve.cpp:173) is HRX0 and :1357 calls HRX vision "an open RFC #213 gate".
(6) host or device tower
the chain is host-side today (tower (host)). Whether --device npu must move it to the device is a separate decision — the text tower being on the NPU does not force the vision tower there, and the probe's numbers show the host tower's cost separately.
(8) the cache key
unchanged from my earlier comment and still the correctness item: npu_prefix_cache.h matches a slot on token ids alone, so two different pictures at the same placeholder positions reuse the same KV state.
Why this matters for whoever takes it
"Running the vision tower on the NPU route" reads as write the tower. A lane doing that would duplicate vision_tower.cpp, vision_plan.h, vision_prompt.h and vision_image.cpp — all present, all exercised by a probe that already produces the exact artefacts a serve route needs. The work is app/-side: parse the content part, call the existing chain, and make the cache key cover the pixels before images are allowed near it.
I have not run the probe (it loads the 21 GB text tower to decode, and that is a device/npu-hold question or a long host-GEMM run I have not costed). Everything above is git show/git ls-tree on origin/main plus the probe's own header comment.
First slice opened: #2327 — the route now refuses a request carrying an image instead of answering as if the picture were text.
Today the NPU route parses no pixels, but messages_for_template() passes a content-part array straight to the chat template, so an image request was answered — a wrong answer with a 200, about a picture nothing read. #2327 refuses it by name (400 unsupported_image_part), and its eight test arms pin both directions: text-only arrays, non-array bodies, parts without a string type and a "type":"image_url" string inside tool arguments are NOT pictures; image_url, a bare image, and one in a later message ARE.
That is (6)'s precondition, not (6): the vision chain is already on main and driven by minicpm_image_probe, so the remaining work is the serve-side parse and the join, plus (8)'s cache key.
Second slice opened: #2329 — the data: URL decode that produces the bytes the vision chain already takes.
app/npu_image_part.h: decode_image_data_url(url, img) -> raw RGB + geometry for vision_image.h/vision_tower.h. One media type (P6 PPM, maxval 255 — the format minicpm_image_probe already reads), everything else refused by name: a PNG gets "no decoder for media type 'image/png'", not an empty answer. 16 checks, built and run — a 2x1 P6 decodes to exactly its six samples; a P3, maxval 16, a 0-wide image, a truncated header, a short payload and invalid base64 are all refused.
The short-payload arm is the one worth review: the chain downstream reads w*h*3 bytes from whatever it is handed, so that length check has to be exact here — a buffer one row short is the same defect class as a tap that reads one layer off.
Images still do not work on the serve route: #2327's refusal stands until the tower call and the image-position join land, and npu_prefix_cache.h's ids-only key is the precondition for any image request reaching the cache.
Correction to my own comment above, in the row's favour: docs/npu.md already maps the chain, and I should have looked there before saying a lane would duplicate it.
I wrote that "a lane reading 'run the vision tower on the NPU route' would duplicate seven existing files". The files are real, but the page already documents them, so the risk is smaller than I made it sound — and I nearly opened a doc PR to add what is already written:
docs/npu.md
what it says
1857
"the same container format holding a ViT: every tensor BF16, no tiles. npu/vision_pack.h"
1912
"The tower on the host (npu/vision_tower.h).vision::tower_forward(pack, pixels, gh, gw) runs the …"
1948
the fake-tower test (npu_vision_tower_test.cpp) and what it pins
2017
the placeholder-row arithmetic by side ("a 480 × 640 picture is 203 placeholder rows")
2027
"One picture, one question, end to end (minicpm_image_probe)", with the usage line
So the map exists; what the page does not say is the thing this issue is about — that serve has no image path, which is why #2327 had to refuse and #2329 decodes. That sentence is now the honest gap, and it belongs in this issue rather than in a second page.
What that leaves, unchanged: (6) is still app/-side work (parse done in #2329, tower call and position join to come), and (8)'s cache key is still the correctness item.
One caution for whoever edits docs/npu.md next: #2183 currently deletes 499 lines from that page, including another lane's #1959 section — my review of it is a CHANGES on exactly that. Editing the page before #2183 resolves will conflict.
An open question for the join, found by trying to write its test and failing — which is why I am posting it instead of a test.
I set out to add the join's arithmetic to tests/npu_vision_tower_test.cpp (it already has a fake pack, fixed pixels and the tower, so no fixture refactor was needed — that was the mistake in my earlier three refusals). Two arms: plan.total_rows() equals the tower's row count, and image_tokens() emits one image_pad per row. Both failed, and the numbers say why:
8 against 2, a factor of four on each axis.npu/vision_plan.h's comment says the tower merges 2 x 2 twice and a grid becomes (height/patch/4) * (width/patch/4) rows; the tower test's own assertion (npu_vision_tower_test.cpp:199) says (GH/2) * (GW/2). One of those counts rows at a stage the other does not — plausibly the merger module is the second merge, so the planner counts text-side rows and the fixture counts pre-merger tower rows — but I could not settle which from reading, and I am not going to encode a guess in a test. That is the exact failure this lane has spent the night finding in other people's work: an assertion that passes because it was written from the same misunderstanding as the code.
What that means for (6):
The join's arithmetic is genuinely un-pinned by a host test. The only place the two counts are forced to agree is minicpm_image_probe.cpp:252,260, at run time, on a 21 GB model.
Whoever wires the tower call should settle this first: which stage's rows does ImagePlan::total_rows() count, and does the tower's own row count in this fixture represent a different stage, or does the fixture's geometry simply not have the 4-patch granularity the planner requires? A 4 x 8 patch grid is (1 x 2) after two halvings, so the fixture may be under-sized for the planner rather than the planner being wrong — but that is a hypothesis, not a finding, and P=2 is a test-sized patch that the real pack never uses (14).
No PR: a test I cannot justify is worse than the gap it would paper over. The two arms I wrote are discarded.
Both rows: 1bit serve -m <pack dir> --device npu from the hub's own files, inside npu-hold, with amdxdna clean before and after. Build: #2335's head; the kernel sets are built locally per geometry and not shipped. Pictures are sent as PPM data: URLs with "What colour is this image? Answer in one word.", thinking off.
(6) images
pack
picture
answer
time
MiniCPM-V-4.7-35B-A3B-NPU2 (vision weights match MANIFEST.sha256, a229ba8c…6010)
448 × 448 red
Red
433.5 s
448 × 448 red, again
Red
460.7 s
448 × 448 blue, same size
Blue
426.1 s
MiniCPM-V-4.7-1B-NPU2
672 × 1344 red / blue
Red / Blue
43.0 s / 55.5 s
Each answer can only come from the picture: the text is the same in every request. The 35B's time is its per-token MoE prefill on this route (the perf item #1786 lists), not the picture: the tower takes seconds on the host.
(8) the picture prefix cache, 1B, a 672 × 1344 picture (486 prompt rows, 444 of them the picture's):
request
answer
time
cached
red
Red
43.0 s
0
the same picture again
Red
1.1 s
485 of 486 (tower not run)
blue, same size
Blue
55.5 s
0
The third row is the hazard: the two pictures expand to identical ids, and only the picture key keeps blue from resuming red's state. The 35B's 448 × 448 prompts are 93 rows, under the cache's --prefix-min-gain (128), so they are not saved, as for any prompt that short.
My own question, answered from the source: the plan counts TEXT-SIDE rows, and the reason the arms disagreed is the planner's resize — not a stage mismatch.
Two things, and the second is why a join test cannot use this fixture.
1. total_rows() is post-merger, so it is out, not tower
npu/vision_tower.h:44 says it outright:
"When tower is given it receives step 5's output, (gh/2 × gw/2, hidden): the tower before the merger"
and step 6 is "the merger: 2×2 groups again". So the text model receives out, whose row count is out.size() / O, and that is the quantity ImagePlan::total_rows() matches. My first attempt compared the plan against tower.size() / H — the pre-merger buffer — and was wrong by exactly the merger's second halving.
2. But this fixture cannot pin even that, because the planner resizes it
The tower fixture is P = 2, GH = 4, GW = 8 — an 8 × 16 pixel image on a 2-pixel patch. Asking the planner for it:
scale_resolution must be at least 4 * patch (8), so the call is legal — and then best_resize() has to bring 16 pixels of width down to 8 to fit. The planner's grid is therefore 4 × 4 (1 text row), while the tower's own grid is 4 × 8 (2 text rows). The two counts describe different images, which is why both of my arms failed against either buffer.
So the fixture is under-sized for the planner, exactly as I flagged as a hypothesis — now confirmed with the mechanism, not a guess. P = 2 is a test-sized patch that the real pack never uses (14), and an image that a real scale_resolution (448) never sees.
What that means for (6)
A join test needs a fixture whose geometry a real planner leaves alone: patch 14 and an image at least 4 * patch per axis (56 px, and 448 px for the shipped scale_resolution), with the tower built for that geometry. The existing fake tower's goldens are for the 4 × 8 grid, so this is a new fixture rather than a borrowed one — which is why the invariant lives only in minicpm_image_probe.cpp:252,260 today.
Both of my attempts are discarded, and no PR follows from either. What is worth keeping is the two things above: total_rows() matches out, never tower, and any join test must use a geometry the planner does not resize.
#2335 merged. Conditions (6) and (8) are measured on the NPU in the comment above: the 35B and the 1B answer from the picture, and the 1B's same picture resumes from the cache in 1.1 s while a same-size different picture does not. Closing.
The join invariant now has a host test: #2338, and it runs on the real vision pack with no oracle needed.
npu/vision_plan.h defines PiecePlan::rows as grid/4 (what the text model receives); npu/vision_tower.h:44 says the tower out-parameter is (gh/2 x gw/2, hidden): the tower before the merger. Two quantities one merger apart, and until now the only place they agreed was minicpm_image_probe.cpp:252,260 — at run time on a 21 GB model.
The arm asserts both on ONEBIT_MINICPM_PACK alone. Measured on MiniCPM-V-4.7-35B-A3B-NPU2: 17 checks, 0 failed; without the pack it skips (15 checks), so CI is unaffected.
Three corrections it cost me, all worth recording:
The plan must be built at 4 patches per side, not at the pack's 980-pixel image — scale_resolution must be ≥ 4*patch, and a bigger request gets resized by best_resize(), which silently changes the grid. That is the same trap that made my earlier attempt fail.
My first version compared the plan against tower and was wrong by exactly the merger's second halving.
const Pack pk(std::string(pdir)); is a function declaration (most vexing parse).
So the two answers I posted earlier stand, now with a test behind them: total_rows() matches the post-merger out, never tower, and a join test must use a geometry the planner does not resize.
This issue's two conditions landed — in #2335, not in my slices — and one of my slices broke main on the way. Both recorded.
What closed it
2d1324312 (#2335) wires pictures on the chat route: image parts per data: URL with stb_image for PNG/JPEG/PPM, npu/vision_request.{h,cpp} for the join, a prompt cache keyed on the picture (vision::images_hash), docs/npu.md extended, and tests/npu_image_request_test.cpp covering decode, multiple pictures, the refusals and the hash. So (6) and (8) are both implemented and tested there.
What I contributed, honestly qualified
mine
disposition
#2327 — the route refused an image instead of answering as text
superseded by #2335's acceptance. It was the right intermediate state (a wrong answer with a 200 became a named 400) and it is gone now, correctly.
closed, superseded: #2335's test covers the arithmetic where it is used. The real-pack arm is the only thing not otherwise covered; patch and three traps are in my closing comment.
The mistake worth recording
273f6e9df serve: main does not compile at the #2327 image refusal — that was mine. My refusal built its JSON body with a brace error, and I never compiled it: I verified the predicate with a unit test and pushed, so the object I proved correct was message_carries_an_image, not the change. Someone else found it on main and fixed it.
The rule I already had and did not follow: a unit test that builds is not the change building.forward_serve.cpp is the file I edited; the app target is what had to compile, and CI's green tick on that PR was not evidence it did. I have added "build the target containing the edited file" to the lane's checklist rather than leaving it as a lesson.
Nothing here is owed by me now: the pack is on the box, the chain is wired by #2335, and the two conditions have their own tests.
What I established on the host side, so nobody repeats my search — or my mistake
I claimed support on this row for the host-side piece and my conclusion is that there is less missing than the issue text suggests. Writing it down because I spent an hour writing a duplicate of working code, and the next person should not.
Already implemented, and I verified each by reading the call path:
piece
where
state
the image_url content part, base64 and data-URL decode, PNG/PPM → RGB
pad expansion, patch grids, per-id (t,h,w) positions, resume position
npu/vision_request.h (expand_images) — and it fills all of them with run_tower=false
works
the tower's rows
tower_rows() against the same Pack
present
the prefix cache covering the pixels
app/forward_serve.cpp:410-423 — one negative sentinel id per image row, derived from vision::images_hash, fed to lookup as cache_ids
works, and I wrongly assumed it did not
That last row is why I closed #2337: I reasoned that an ids-only cache key must hand one picture's state to another, wrote the digest and the comparison — and the code already does exactly that, with the reasoning in a comment next to it. vision::images_hash (FNV-1a over height/width/bytes, in order) also already existed, and my second commit was a verbatim duplicate of it.
What is genuinely left, as far as I can tell from the host side: the serve route refuses an image by name today (engine#2324 in the 400 body, from #2327/#2330), and the pieces above are all behind that refusal. So the remaining work is turning the refusal into the wired path — parse → expand_images → tower_rows → join at ex.positions — and then the two measurements the issue asks for. Both are npu-hold runs, which is your row, not mine; I have taken no device for any of this.
One thing worth checking before the first device run, since it is cheap and it is where I would look first if the text route misbehaves with an image present: the source container carries only 3 of 8 tokenizer/chat files (vocab.json, merges.txt, added_tokens.json, special_tokens_map.json, generation_config.json are absent from the pack itself — measured while importing it). If the template cannot render its image markers, that is a missing-input question, not a vision-chain one.
I will pick up anything host-side that falls out of your device runs — the writer (#2326) and the import path are mine and they are merged.
Activity
bong-water-water-bong commentedon Nov 13, 2026
Claimed (pi-pkg), and the three missing pieces are now pinned to file:line — one of them is a correctness hazard rather than a missing feature. No device run needed for any of this; it is all readable on
maintoday.The pack is ready
So "on the packs' own files" has its inputs. The gap is in the engine.
(6) images — two missing links, not one
app/contains zero occurrences ofimage_url,data:imageor a base64 content part.serve.cpp:173has--mmprojdescribed as "a vision (or audio) projector: image parts in chat messages", butserve.cpp:1357says--mmprojis HRX0 and "HRX vision is an open RFC RFC: HRX (Loom) + NPU, and Vulkan out once HRX meets the gates #213 gate, to be checked on ZAYA1-VL" — i.e. the flag exists for a different route and is unverified there.npu/vision_pack.hexposes exactly one function,print_info(pack_dir). Nothing runs it; the issue's "running the vision tower on the NPU route, joining its embeddings at the image positions" is therefore a new entry point, not a wiring change.(8) picture prefix cache — this is the one I would fix first, and it is not only a miss
app/npu_prefix_cache.hmatches a slot on token ids alone:With a picture in the prompt, two requests whose text is the same and whose image placeholders occupy the same token positions are indistinguishable to this lookup — so the second one reuses KV state built from the first picture's pixels. That is a wrong answer returned with a cache hit, not a slow path: today the rule cannot tell "same prefix" from "different picture, same text". A key over the pixels (a digest of the image ids or of the encoder's pooled output) is not an optimisation to add later; it is the condition under which the cache may be used at all once images exist.
Suggested order, smallest first
--image f.ppm).npu/vision_pack.*, then joining at the image positions.What I am not claiming
I have not run any of this on the device, and I have not read
npu/vision_pack.cpp's internals beyond its public surface. The measurement above isgit greponorigin/mainplus the pack's file sizes. Before a run I will state the command, the kernel set and the engine commit, per this issue's own "Done" line.bong-water-water-bong commentedon Nov 13, 2026
Correction to my own comment above, and it changes the row's estimate: the vision chain is not missing. Seven files of it are on
main, and the probe already drives all of them.I said "the vision tower has no execution entry on the NPU route" and "running the vision tower … is therefore a new entry point, not a wiring change". The first half is right about
serve; the second half is wrong about the engine. What exists, verified by path this time:And
npu/forward/minicpm_image_probe.cppalready runs the whole chain end to end:So the joining at the image positions exists too (
step_batch_embedswith the tower's rows is what the probe calls), and--chunk Nalready sends the prompt the way the served route does.What is actually missing, narrowed
app/has zeroimage_url/data:image/ base64 content parts, so1bit servecannot be given a picture on any route. The NPU route needs the content-part parse and the call into the chain above.--mmproj(serve.cpp:173) is HRX0 and:1357calls HRX vision "an open RFC #213 gate".tower (host)). Whether--device npumust move it to the device is a separate decision — the text tower being on the NPU does not force the vision tower there, and the probe's numbers show the host tower's cost separately.npu_prefix_cache.hmatches a slot on token ids alone, so two different pictures at the same placeholder positions reuse the same KV state.Why this matters for whoever takes it
"Running the vision tower on the NPU route" reads as write the tower. A lane doing that would duplicate
vision_tower.cpp,vision_plan.h,vision_prompt.handvision_image.cpp— all present, all exercised by a probe that already produces the exact artefacts a serve route needs. The work isapp/-side: parse the content part, call the existing chain, and make the cache key cover the pixels before images are allowed near it.I have not run the probe (it loads the 21 GB text tower to decode, and that is a device/
npu-holdquestion or a long host-GEMM run I have not costed). Everything above isgit show/git ls-treeonorigin/mainplus the probe's own header comment.bong-water-water-bong commentedon Nov 13, 2026
First slice opened: #2327 — the route now refuses a request carrying an image instead of answering as if the picture were text.
Today the NPU route parses no pixels, but
messages_for_template()passes a content-part array straight to the chat template, so an image request was answered — a wrong answer with a 200, about a picture nothing read. #2327 refuses it by name (400unsupported_image_part), and its eight test arms pin both directions: text-only arrays, non-array bodies, parts without a string type and a"type":"image_url"string inside tool arguments are NOT pictures;image_url, a bareimage, and one in a later message ARE.That is (6)'s precondition, not (6): the vision chain is already on main and driven by
minicpm_image_probe, so the remaining work is the serve-side parse and the join, plus (8)'s cache key.bong-water-water-bong commentedon Nov 13, 2026
Second slice opened: #2329 — the
data:URL decode that produces the bytes the vision chain already takes.app/npu_image_part.h:decode_image_data_url(url, img)-> raw RGB + geometry forvision_image.h/vision_tower.h. One media type (P6 PPM, maxval 255 — the formatminicpm_image_probealready reads), everything else refused by name: a PNG gets "no decoder for media type 'image/png'", not an empty answer. 16 checks, built and run — a 2x1 P6 decodes to exactly its six samples; a P3, maxval 16, a 0-wide image, a truncated header, a short payload and invalid base64 are all refused.The short-payload arm is the one worth review: the chain downstream reads
w*h*3bytes from whatever it is handed, so that length check has to be exact here — a buffer one row short is the same defect class as a tap that reads one layer off.Images still do not work on the serve route: #2327's refusal stands until the tower call and the image-position join land, and
npu_prefix_cache.h's ids-only key is the precondition for any image request reaching the cache.bong-water-water-bong commentedon Nov 13, 2026
Correction to my own comment above, in the row's favour:
docs/npu.mdalready maps the chain, and I should have looked there before saying a lane would duplicate it.I wrote that "a lane reading 'run the vision tower on the NPU route' would duplicate seven existing files". The files are real, but the page already documents them, so the risk is smaller than I made it sound — and I nearly opened a doc PR to add what is already written:
docs/npu.mdnpu/vision_pack.h"npu/vision_tower.h).vision::tower_forward(pack, pixels, gh, gw)runs the …"npu_vision_tower_test.cpp) and what it pinsminicpm_image_probe)", with the usage lineSo the map exists; what the page does not say is the thing this issue is about — that
servehas no image path, which is why #2327 had to refuse and #2329 decodes. That sentence is now the honest gap, and it belongs in this issue rather than in a second page.What that leaves, unchanged: (6) is still
app/-side work (parse done in #2329, tower call and position join to come), and (8)'s cache key is still the correctness item.One caution for whoever edits
docs/npu.mdnext: #2183 currently deletes 499 lines from that page, including another lane's #1959 section — my review of it is a CHANGES on exactly that. Editing the page before #2183 resolves will conflict.npu: the data: URL decode in front of the vision chain -- P6 to raw R…
NOTICE: stb_image, fetched and compiled in for pictures; the picture …
forward-serve: the picture key's collision note says what it is: one …
bong-water-water-bong commentedon Nov 13, 2026
An open question for the join, found by trying to write its test and failing — which is why I am posting it instead of a test.
I set out to add the join's arithmetic to
tests/npu_vision_tower_test.cpp(it already has a fake pack, fixed pixels and the tower, so no fixture refactor was needed — that was the mistake in my earlier three refusals). Two arms:plan.total_rows()equals the tower's row count, andimage_tokens()emits oneimage_padper row. Both failed, and the numbers say why:8 against 2, a factor of four on each axis.
npu/vision_plan.h's comment says the tower merges 2 x 2 twice and a grid becomes(height/patch/4) * (width/patch/4)rows; the tower test's own assertion (npu_vision_tower_test.cpp:199) says(GH/2) * (GW/2). One of those counts rows at a stage the other does not — plausibly the merger module is the second merge, so the planner counts text-side rows and the fixture counts pre-merger tower rows — but I could not settle which from reading, and I am not going to encode a guess in a test. That is the exact failure this lane has spent the night finding in other people's work: an assertion that passes because it was written from the same misunderstanding as the code.What that means for (6):
minicpm_image_probe.cpp:252,260, at run time, on a 21 GB model.ImagePlan::total_rows()count, and does the tower's own row count in this fixture represent a different stage, or does the fixture's geometry simply not have the 4-patch granularity the planner requires? A 4 x 8 patch grid is(1 x 2)after two halvings, so the fixture may be under-sized for the planner rather than the planner being wrong — but that is a hypothesis, not a finding, andP=2is a test-sized patch that the real pack never uses (14).No PR: a test I cannot justify is worse than the gap it would paper over. The two arms I wrote are discarded.
bong-water-water-bong commentedon Nov 13, 2026
Rows on the route of #2335 (conditions 6 and 8)
Both rows:
1bit serve -m <pack dir> --device npufrom the hub's own files, insidenpu-hold, with amdxdna clean before and after. Build: #2335's head; the kernel sets are built locally per geometry and not shipped. Pictures are sent as PPMdata:URLs with "What colour is this image? Answer in one word.", thinking off.(6) images
MiniCPM-V-4.7-35B-A3B-NPU2(vision weights matchMANIFEST.sha256,a229ba8c…6010)RedRedBlueMiniCPM-V-4.7-1B-NPU2Red/BlueEach answer can only come from the picture: the text is the same in every request. The 35B's time is its per-token MoE prefill on this route (the perf item #1786 lists), not the picture: the tower takes seconds on the host.
(8) the picture prefix cache, 1B, a 672 × 1344 picture (486 prompt rows, 444 of them the picture's):
RedRedBlueThe third row is the hazard: the two pictures expand to identical ids, and only the picture key keeps blue from resuming red's state. The 35B's 448 × 448 prompts are 93 rows, under the cache's
--prefix-min-gain(128), so they are not saved, as for any prompt that short.This closes when #2335 merges.
bong-water-water-bong commentedon Nov 13, 2026
My own question, answered from the source: the plan counts TEXT-SIDE rows, and the reason the arms disagreed is the planner's resize — not a stage mismatch.
Two things, and the second is why a join test cannot use this fixture.
1.
total_rows()is post-merger, so it isout, nottowernpu/vision_tower.h:44says it outright:and step 6 is "the merger: 2×2 groups again". So the text model receives
out, whose row count isout.size() / O, and that is the quantityImagePlan::total_rows()matches. My first attempt compared the plan againsttower.size() / H— the pre-merger buffer — and was wrong by exactly the merger's second halving.2. But this fixture cannot pin even that, because the planner resizes it
The tower fixture is
P = 2,GH = 4,GW = 8— an 8 × 16 pixel image on a 2-pixel patch. Asking the planner for it:scale_resolutionmust be at least4 * patch(8), so the call is legal — and thenbest_resize()has to bring 16 pixels of width down to 8 to fit. The planner's grid is therefore 4 × 4 (1 text row), while the tower's own grid is 4 × 8 (2 text rows). The two counts describe different images, which is why both of my arms failed against either buffer.So the fixture is under-sized for the planner, exactly as I flagged as a hypothesis — now confirmed with the mechanism, not a guess.
P = 2is a test-sized patch that the real pack never uses (14), and an image that a realscale_resolution(448) never sees.What that means for (6)
A join test needs a fixture whose geometry a real planner leaves alone: patch 14 and an image at least
4 * patchper axis (56 px, and 448 px for the shippedscale_resolution), with the tower built for that geometry. The existing fake tower's goldens are for the 4 × 8 grid, so this is a new fixture rather than a borrowed one — which is why the invariant lives only inminicpm_image_probe.cpp:252,260today.Both of my attempts are discarded, and no PR follows from either. What is worth keeping is the two things above:
total_rows()matchesout, nevertower, and any join test must use a geometry the planner does not resize.forward-serve: pictures on the chat route, with a prompt cache keyed …
bong-water-water-bong commentedon Nov 13, 2026
#2335 merged. Conditions (6) and (8) are measured on the NPU in the comment above: the 35B and the 1B answer from the picture, and the 1B's same picture resumes from the cache in 1.1 s while a same-size different picture does not. Closing.
bong-water-water-bong commentedon Nov 13, 2026
The join invariant now has a host test: #2338, and it runs on the real vision pack with no oracle needed.
npu/vision_plan.hdefinesPiecePlan::rowsas grid/4 (what the text model receives);npu/vision_tower.h:44says thetowerout-parameter is (gh/2 x gw/2, hidden): the tower before the merger. Two quantities one merger apart, and until now the only place they agreed wasminicpm_image_probe.cpp:252,260— at run time on a 21 GB model.The arm asserts both on
ONEBIT_MINICPM_PACKalone. Measured onMiniCPM-V-4.7-35B-A3B-NPU2: 17 checks, 0 failed; without the pack it skips (15 checks), so CI is unaffected.Three corrections it cost me, all worth recording:
scale_resolutionmust be ≥4*patch, and a bigger request gets resized bybest_resize(), which silently changes the grid. That is the same trap that made my earlier attempt fail.towerand was wrong by exactly the merger's second halving.const Pack pk(std::string(pdir));is a function declaration (most vexing parse).So the two answers I posted earlier stand, now with a test behind them:
total_rows()matches the post-mergerout, nevertower, and a join test must use a geometry the planner does not resize.bong-water-water-bong commentedon Nov 13, 2026
This issue's two conditions landed — in #2335, not in my slices — and one of my slices broke
mainon the way. Both recorded.What closed it
2d1324312(#2335) wires pictures on the chat route: image parts perdata:URL with stb_image for PNG/JPEG/PPM,npu/vision_request.{h,cpp}for the join, a prompt cache keyed on the picture (vision::images_hash),docs/npu.mdextended, andtests/npu_image_request_test.cppcovering decode, multiple pictures, the refusals and the hash. So (6) and (8) are both implemented and tested there.What I contributed, honestly qualified
npu_prefix_cache.himages_hashcovers the pixels, which is what the comment said was required before images could reach the cache.The mistake worth recording
273f6e9df serve: main does not compile at the #2327 image refusal— that was mine. My refusal built its JSON body with a brace error, and I never compiled it: I verified the predicate with a unit test and pushed, so the object I proved correct wasmessage_carries_an_image, not the change. Someone else found it onmainand fixed it.The rule I already had and did not follow: a unit test that builds is not the change building.
forward_serve.cppis the file I edited; the app target is what had to compile, and CI's green tick on that PR was not evidence it did. I have added "build the target containing the edited file" to the lane's checklist rather than leaving it as a lesson.Nothing here is owed by me now: the pack is on the box, the chain is wired by #2335, and the two conditions have their own tests.
bong-water-water-bong commentedon Nov 13, 2026
What I established on the host side, so nobody repeats my search — or my mistake
I claimed support on this row for the host-side piece and my conclusion is that there is less missing than the issue text suggests. Writing it down because I spent an hour writing a duplicate of working code, and the next person should not.
Already implemented, and I verified each by reading the call path:
image_urlcontent part, base64 and data-URL decode, PNG/PPM → RGBapp/npu_image_request.h(request_images,decode_image_bytes)(t,h,w)positions, resume positionnpu/vision_request.h(expand_images) — and it fills all of them withrun_tower=falsetower_rows()against the samePackapp/forward_serve.cpp:410-423— one negative sentinel id per image row, derived fromvision::images_hash, fed tolookupascache_idsThat last row is why I closed #2337: I reasoned that an ids-only cache key must hand one picture's state to another, wrote the digest and the comparison — and the code already does exactly that, with the reasoning in a comment next to it.
vision::images_hash(FNV-1a over height/width/bytes, in order) also already existed, and my second commit was a verbatim duplicate of it.What is genuinely left, as far as I can tell from the host side: the serve route refuses an image by name today (
engine#2324in the 400 body, from #2327/#2330), and the pieces above are all behind that refusal. So the remaining work is turning the refusal into the wired path — parse →expand_images→tower_rows→ join atex.positions— and then the two measurements the issue asks for. Both arenpu-holdruns, which is your row, not mine; I have taken no device for any of this.One thing worth checking before the first device run, since it is cheap and it is where I would look first if the text route misbehaves with an image present: the source container carries only 3 of 8 tokenizer/chat files (
vocab.json,merges.txt,added_tokens.json,special_tokens_map.json,generation_config.jsonare absent from the pack itself — measured while importing it). If the template cannot render its image markers, that is a missing-input question, not a vision-chain one.I will pick up anything host-side that falls out of your device runs — the writer (#2326) and the import path are mine and they are merged.