Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -692,7 +692,7 @@ option(ONEBIT_NPU_HOST "Build the NPU host library and its tests" ${_onebit_npu_
if(ONEBIT_NPU_HOST)
find_package(PkgConfig REQUIRED)
pkg_check_modules(PCRE2 REQUIRED IMPORTED_TARGET libpcre2-8)
add_library(onebit_npu STATIC npu/file_map.cpp npu/flm_head_import.cpp npu/full_elf.cpp npu/model.cpp npu/vision_pack.cpp
add_library(onebit_npu STATIC npu/file_map.cpp npu/flm_head_import.cpp npu/full_elf.cpp npu/model.cpp npu/vision_pack.cpp npu/vision_tower.cpp
npu/moe.cpp npu/pack.cpp npu/pack_import.cpp npu/private_route.cpp
npu/q4nx.cpp npu/q4nx_device.cpp npu/tokenizer.cpp)
target_include_directories(onebit_npu PUBLIC npu)
Expand Down Expand Up @@ -1213,6 +1213,11 @@ add_executable(npu_vision_pack_test tests/npu_vision_pack_test.cpp)
target_link_libraries(npu_vision_pack_test PRIVATE onebit_npu)
add_test(NAME npu_vision_pack COMMAND npu_vision_pack_test)
add_dependencies(onebit npu_vision_pack_test)
# The host float32 tower against goldens from the transformers implementation (see the test's header).
add_executable(npu_vision_tower_test tests/npu_vision_tower_test.cpp)
target_link_libraries(npu_vision_tower_test PRIVATE onebit_npu)
add_test(NAME npu_vision_tower COMMAND npu_vision_tower_test)
add_dependencies(onebit npu_vision_tower_test)
add_executable(npu_q4nx_test tests/npu_q4nx_test.cpp)
target_include_directories(npu_q4nx_test PRIVATE app)
target_link_libraries(npu_q4nx_test PRIVATE onebit_npu)
Expand Down
40 changes: 37 additions & 3 deletions docs/npu.md
Original file line number Diff line number Diff line change
Expand Up @@ -1462,16 +1462,15 @@ tensors 347, all BF16, 1135109120 B of payload, 0 B unaccounted
patch embed 14 x 14 pixels x 3 channels -> 1152
positions 4900 (a 70 x 70 grid; image 980)
blocks 27 at width 1152, 16 heads x 72, MLP 4304 (4608 stored)
vit_merger attention at 1152, MLP 4608 -> 17216 (17280 stored) -> 1152
vit_merger after block 6; attention at 1152, MLP 4608 -> 17216 (17280 stored) -> 1152
merger 4608 -> 4608 -> 1024 (the text model's hidden size)
source_file mmproj-model-f16.gguf
source_sha256 fb9cc047a3cdad5317fd8433a4aeda2d22f00f02ca4bd93b34ad84165b25a900
note config.json states MINICPM_V_VISION_MLP_INTERMEDIATE_SIZE 4304; model.visual.blocks.*.mlp.linear_fc1.weight has 4608 rows, the last 304 all zero: the container pads the model's width, and sizes come from the tensor
note config.json states MINICPM_V_VIT_MERGER_INTERMEDIATE_SIZE 17216; model.visual.vit_merger.mlp.linear_fc1.weight has 17280 rows, the last 64 all zero: the container pads the model's width, and sizes come from the tensor
```

This is a reader, not a forward: nothing runs the tower yet. What it settles is what a forward may
rely on.
What the reader settles is what a forward may rely on.

- **Sizes come from the tensors.** The geometry is derived from the tensors' own shapes, and every
tensor must have the shape the others imply.
Expand Down Expand Up @@ -1508,6 +1507,41 @@ Pinned by `tests/npu_vision_pack_test.cpp` on crafted containers; `ONEBIT_MINICP
the published pack's own numbers. The source comparison above was run by hand and is not a test in the
tree.

**The tower on the host (`npu/vision_tower.h`).** `vision::tower_forward(pack, pixels, gh, gw)` runs the
tower in float32 on the CPU: normalised pixels for a `gh × gw` patch grid in, `(gh/4) × (gw/4)` rows at
the text model's width out. It is a reference for whatever runs the tower on the NPU later, not a
serving path; nothing calls it from a serve, and it does no image decoding, resizing or slicing. The
order of operations is the transformers implementation's (`transformers.models.minicpmv4_7`):

1. patch embedding (a Conv2d whose kernel is its stride), patches row-major;
2. plus one learned position row per patch, picked **nearest** on the 70 × 70 table: row
`floor(y·70/gh)`, column `floor(x·70/gw)`. No interpolation and no rotary term;
3. blocks 0–6: LayerNorm (eps 1e-6), full attention, MLP with the tanh GELU;
4. the `vit_merger`, once, after block `insert_layer_id` (6): attention inside each 2 × 2 window only,
then per window an MLP over the four tokens side by side plus their mean. The grid halves;
5. blocks 7–26 on the halved grid, then `merger.norm`;
6. the merger: 2 × 2 groups again, `pre_norm`, `linear_fc1`, the exact (erf) GELU, `linear_fc2`.

So both grid extents must be multiples of 4, and anything else is refused. The padded MLPs are read at
the tensor's stride and the model's width.

Measured against that implementation (transformers 5.22.2, torch 2.13 CPU, float32, eager attention)
loaded with each pack's own weights, same input (`pixel(c, y, x) = sin(0.37c + 0.11y + 0.05x +
0.003xy)`), 2026-11-10:

| pack | grid | tower before the merger: max \|d\|, corr | output: max \|d\|, corr |
|---|---|---|---|
| 1B | 8 × 12 | 2.9e-5, 1.000000000 | 1.1e-6, 1.000000000 |
| 1B | 16 × 20 | 1.1e-5, 1.000000000 | 2.4e-6, 1.000000000 |
| 35B-A3B | 16 × 20 | 9.5e-6, 1.000000000 | 9.5e-7, 1.000000000 |

The 16 × 20 grid took 5.5 s and 1.25 GB of memory for this code on one thread; a full 70 × 70 slice
has 61 times the tokens and its attention grows with the square, which is why this is a reference.
`tests/npu_vision_tower_test.cpp` carries a small tower with LCG weights and that implementation's
output for it as goldens, so the arithmetic is checked in CI without a pack; with
`ONEBIT_MINICPM_PACK` and `ONEBIT_MINICPM_ORACLE` set it repeats the comparison above. The reference
run itself needs transformers and torch and is not in the tree.

## Serving

The 1bit NPU engine runs the model on the XDNA 2 NPU. With a release package that carries the NPU
Expand Down
18 changes: 17 additions & 1 deletion npu/vision_pack.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -299,6 +299,22 @@ void Pack::open(const std::string& container_path, const std::string& config_jso
fail("config.json states the text model's hidden_size " + std::to_string(c["hidden_size"].get<int64_t>()) +
" but the merger projects to " + std::to_string(geom_.out_hidden) + ": this tower is not this text model's");

{
// top level first (the model's own config key), then the vision_config spelling; both present must agree
int top = -1, inner = -1;
const bool have_top = c.contains("insert_layer_id") && c["insert_layer_id"].is_number_integer();
if (have_top) top = int(c["insert_layer_id"].get<int64_t>());
const bool have_inner = vc.contains("MINICPM_V_INSERT_LAYER_ID") && vc["MINICPM_V_INSERT_LAYER_ID"].is_number_integer();
if (have_inner) inner = int(vc["MINICPM_V_INSERT_LAYER_ID"].get<int64_t>());
if (have_top && have_inner && top != inner)
fail("config.json states insert_layer_id " + std::to_string(top) + " and MINICPM_V_INSERT_LAYER_ID " +
std::to_string(inner) + ": one tower, two merge points");
if (have_top || have_inner) geom_.insert_layer = have_top ? top : inner;
if ((have_top || have_inner) && (geom_.insert_layer < 0 || geom_.insert_layer >= geom_.layers))
fail("config.json states insert_layer_id " + std::to_string(geom_.insert_layer) + " but the container holds " +
std::to_string(geom_.layers) + " blocks");
}

// An MLP's stated width against its tensors. `stems` are the MLPs that share the width (every block, or the
// vit_merger alone): fc1 is [width, in] with a [width] bias, fc2 is [out, width].
const auto zero_from = [&](const std::string& name, int64_t rows, int64_t cols, int64_t from, bool by_row) {
Expand Down Expand Up @@ -382,7 +398,7 @@ int print_info(const std::string& pack_dir) {
std::printf("positions %d (a %d x %d grid; image %d)\n", g.pos_rows, g.grid, g.grid, g.grid * g.patch);
std::printf("blocks %d at width %d, %d heads x %d, MLP %d (%d stored)\n", g.layers, g.hidden, g.heads, g.head_dim,
g.model_intermediate, g.intermediate);
std::printf("vit_merger attention at %d, MLP %d -> %d (%d stored) -> %d\n", g.hidden, g.merge_hidden,
std::printf("vit_merger after block %d; attention at %d, MLP %d -> %d (%d stored) -> %d\n", g.insert_layer, g.hidden, g.merge_hidden,
g.model_vit_merger_intermediate, g.vit_merger_intermediate, g.hidden);
std::printf("merger %d -> %d -> %d (the text model's hidden size)\n", g.merge_hidden, g.merge_hidden, g.out_hidden);
for (const char* k : {"source_file", "source_sha256", "source_dtype", "converter"})
Expand Down
3 changes: 3 additions & 0 deletions npu/vision_pack.h
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,9 @@ struct Geometry {
// and every row past it is zero in every block, else the tensors' own. A forward may stop at these; it must
// still index the tensors at `intermediate` / `vit_merger_intermediate`.
int model_intermediate = 0, model_vit_merger_intermediate = 0;
// The block after which the vit_merger runs: config.json's `insert_layer_id` (the reference's default, 6, when the
// config does not say). A value that is not a block of this tower is refused.
int insert_layer = 6;
};

class Pack {
Expand Down
Loading
Loading