Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -702,7 +702,7 @@ option(ONEBIT_NPU_HOST "Build the NPU host library and its tests" ${_onebit_npu_
if(ONEBIT_NPU_HOST)
find_package(PkgConfig REQUIRED)
pkg_check_modules(PCRE2 REQUIRED IMPORTED_TARGET libpcre2-8)
add_library(onebit_npu STATIC npu/file_map.cpp npu/flm_head_import.cpp npu/full_elf.cpp npu/model.cpp npu/vision_pack.cpp npu/vision_tower.cpp
add_library(onebit_npu STATIC npu/file_map.cpp npu/flm_head_import.cpp npu/full_elf.cpp npu/model.cpp npu/vision_pack.cpp npu/vision_tower.cpp npu/vision_image.cpp
npu/moe.cpp npu/pack.cpp npu/pack_import.cpp npu/private_route.cpp
npu/q4nx.cpp npu/q4nx_device.cpp npu/tokenizer.cpp)
target_include_directories(onebit_npu PUBLIC npu)
Expand Down Expand Up @@ -1228,6 +1228,16 @@ add_executable(npu_vision_tower_test tests/npu_vision_tower_test.cpp)
target_link_libraries(npu_vision_tower_test PRIVATE onebit_npu)
add_test(NAME npu_vision_tower COMMAND npu_vision_tower_test)
add_dependencies(onebit npu_vision_tower_test)
# The image plan (sizes, slice grid, text rows) against the reference image processor's answers for 3,481 sizes.
add_executable(npu_vision_plan_test tests/npu_vision_plan_test.cpp)
target_include_directories(npu_vision_plan_test PRIVATE npu)
add_test(NAME npu_vision_plan COMMAND npu_vision_plan_test)
add_dependencies(onebit npu_vision_plan_test)
# The resize (bit for bit against torch's uint8 antialiased bicubic) and the pieces handed to the tower.
add_executable(npu_vision_image_test tests/npu_vision_image_test.cpp)
target_link_libraries(npu_vision_image_test PRIVATE onebit_npu)
add_test(NAME npu_vision_image COMMAND npu_vision_image_test)
add_dependencies(onebit npu_vision_image_test)
add_executable(npu_q4nx_test tests/npu_q4nx_test.cpp)
target_include_directories(npu_q4nx_test PRIVATE app)
target_link_libraries(npu_q4nx_test PRIVATE onebit_npu)
Expand Down
52 changes: 52 additions & 0 deletions docs/npu.md
Original file line number Diff line number Diff line change
Expand Up @@ -1580,6 +1580,58 @@ output for it as goldens, so the arithmetic is checked in CI without a pack; wit
`ONEBIT_MINICPM_PACK` and `ONEBIT_MINICPM_ORACLE` set it repeats the comparison above. The reference
run itself needs transformers and torch and is not in the tree.

**Where an image goes before the tower (`npu/vision_plan.h`).** `vision::plan_image(height, width)` is
the arithmetic of the model's image processor (transformers' `MiniCPMV4_6ImageProcessor`, which 4.7
loads), with no pixel touched: the model looks at an image whole, resized to about 448 × 448 pixels,
and, when the image is larger than that, again as a grid of up to 9 slices cut from a second, larger
resize. Every piece is a multiple of 56 pixels (4 patches) in both extents, because the tower merges
2 × 2 twice, and becomes `(height/56) × (width/56)` rows on the text side.

| image | whole view | slices | text rows |
|---|---|---|---|
| 448 × 448 or smaller (scaled up) | 448 × 448 | none | 64 |
| 480 × 640 | 392 × 504 | 1 × 2 of 560 × 392 | 63 + 2 × 70 |
| 1080 × 1920 | 336 × 616 | 2 × 4 of 448 × 448 | 66 + 8 × 64 |
| 3024 × 4032 | 392 × 504 | 3 × 3 of 392 × 504 | 63 + 9 × 63 |

(heights first.) So one 448-pixel slice is 64 rows, not the 1,225 that 980 / 14 / 2 squared suggests: the
position table is 70 × 70, but no piece the processor produces is that large unless the image is a very
long strip. The plan is checked against the reference's own answers for 3,481 image sizes in
`tests/npu_vision_plan_test.cpp`. Nothing is cropped or padded and no size is refused for not being a
multiple of anything: every image is resized whole (385 × 384 becomes 448 × 448, so the aspect ratio moves
a little, as it does in the model's own processor), and the slices tile the refine size exactly. The plan
does not bound its result: a 1,048,576 × 1 strip gives the reference's answer, a piece 458,752 pixels
tall, and it is the caller that decides how many rows to accept. `plan_image()` reads no file and touches
no pixel, which is what lets it be checked anywhere; keep it that way.

**From bytes to the tower's input (`npu/vision_image.h`).** `vision::preprocess(rgb, height, width, plan)`
resizes an RGB byte image to the plan's whole view and, when it is sliced, to the refine size, cuts the
slices, and returns each piece as `(3, height, width)` float32 with value `(byte − 127.5) / 127.5`
(rescale by 1/255, then mean 0.5 and std 0.5, the same for each channel; R, G, B order in and out; byte
0 is exactly −1 and byte 255 exactly +1): the whole view first, then the slices row by row. The resize is bicubic with antialiasing on bytes and
reproduces `torch.nn.functional.interpolate(uint8, mode="bicubic", antialias=True)` (torch 2.13) bit
for bit: the a = −0.5 cubic stretched by the scale when shrinking, integer weights at the largest
precision that still fits int16 (PIL's fixed 22 bits differs by one in about 1 % of bytes), width
first and then height. `tests/npu_vision_image_test.cpp` holds the reference's bytes for eleven
resizes as hashes.

End to end on the 1B pack, 2026-11-10: a 460 × 470 synthetic image through plan, resize, pieces and
`tower_forward()`, against the reference's plan, torch's resize and the transformers tower on the same
image:

| piece | patches | rows | max \|d\| | corr |
|---|---|---|---|---|
| whole view 448 × 448 | 32 × 32 | 64 | 6.4e-6 | 1.000000000 |
| slice 0, 616 × 336 | 44 × 24 | 66 | 7.5e-6 | 1.000000000 |
| slice 1, 616 × 336 | 44 × 24 | 66 | 6.7e-6 | 1.000000000 |

That run took 110 s on one thread, which is the host reference's cost and the reason it is not a
serving path. Two things are not established: the reference side called torch's `interpolate`
directly, on the understanding that this is what the processor's torchvision resize does for byte
images on a CPU build with AVX2 (torchvision is not on this box, so that link is read, not run); and
no real photograph or image decoder has been through it. The slice separators and placeholder rows on
the text side are not written.

## Serving

The 1bit NPU engine runs the model on the XDNA 2 NPU. With a release package that carries the NPU
Expand Down
154 changes: 154 additions & 0 deletions npu/vision_image.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,154 @@
// Copyright 2026 bong-water-water-bong
// SPDX-License-Identifier: Apache-2.0
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#include "vision_image.h"

#include <cmath>
#include <stdexcept>
#include <string>

namespace onebit::npu::vision {

namespace {

double cubic(double x) {
const double a = -0.5;
if (x < 0) x = -x;
if (x < 1.0) return ((a + 2.0) * x - (a + 3.0)) * x * x + 1.0;
if (x < 2.0) return (((x - 5.0) * x + 8.0) * x - 4.0) * a;
return 0.0;
}

struct Taps {
std::vector<int> first; // per output index: the first input index
std::vector<int> count; // and how many taps
std::vector<int32_t> weight; // count taps each, `stride` apart
int stride = 0, precision = 0;
};

Taps taps_for(int in, int out) {
const double scale = double(in) / double(out);
const double fs = scale > 1.0 ? scale : 1.0, support = 2.0 * fs, inv = 1.0 / fs;
Taps t;
t.stride = int(std::ceil(support)) * 2 + 1;
t.first.resize(size_t(out));
t.count.resize(size_t(out));
std::vector<double> w(size_t(out) * t.stride, 0.0);
double wmax = 0.0;
for (int i = 0; i < out; ++i) {
const double center = scale * (i + 0.5);
int lo = int(center - support + 0.5);
if (lo < 0) lo = 0;
int hi = int(center + support + 0.5);
if (hi > in) hi = in;
const int n = hi - lo;
double total = 0.0;
for (int j = 0; j < n; ++j) {
w[size_t(i) * t.stride + j] = cubic((j + lo - center + 0.5) * inv);
total += w[size_t(i) * t.stride + j];
}
for (int j = 0; j < n; ++j) {
if (total != 0.0) w[size_t(i) * t.stride + j] /= total;
if (w[size_t(i) * t.stride + j] > wmax) wmax = w[size_t(i) * t.stride + j];
}
t.first[size_t(i)] = lo;
t.count[size_t(i)] = n;
}
// the largest precision at which the biggest weight still fits a signed 16-bit integer
for (int p = 0; p < 22; ++p) {
if (int64_t(0.5 + wmax * double(int64_t(1) << (p + 1))) >= (1 << 15)) break;
t.precision = p + 1;
}
t.weight.resize(w.size());
for (size_t k = 0; k < w.size(); ++k) {
const double v = w[k] * double(int64_t(1) << t.precision);
t.weight[k] = int32_t(v < 0 ? -0.5 + v : 0.5 + v);
}
return t;
}

inline uint8_t clip8(int64_t v) { return uint8_t(v < 0 ? 0 : (v > 255 ? 255 : v)); }

} // namespace

std::vector<uint8_t> resize_bicubic_u8(const uint8_t* rgb, int height, int width, int out_height, int out_width) {
if (!rgb || height <= 0 || width <= 0 || out_height <= 0 || out_width <= 0 || out_height > (1 << 16) || out_width > (1 << 16))
throw std::runtime_error("cannot resize a " + std::to_string(height) + " x " + std::to_string(width) + " image to " +
std::to_string(out_height) + " x " + std::to_string(out_width));
// width first
std::vector<uint8_t> mid;
const uint8_t* src = rgb;
if (out_width != width) {
const Taps t = taps_for(width, out_width);
const int64_t half = t.precision > 0 ? int64_t(1) << (t.precision - 1) : 0;
mid.resize(size_t(height) * out_width * 3);
for (int y = 0; y < height; ++y)
for (int x = 0; x < out_width; ++x) {
const int32_t* w = t.weight.data() + size_t(x) * t.stride;
const uint8_t* p = rgb + (size_t(y) * width + size_t(t.first[size_t(x)])) * 3;
int64_t acc[3] = {half, half, half};
for (int j = 0; j < t.count[size_t(x)]; ++j)
for (int c = 0; c < 3; ++c) acc[c] += int64_t(p[size_t(j) * 3 + c]) * w[j];
for (int c = 0; c < 3; ++c) mid[(size_t(y) * out_width + x) * 3 + c] = clip8(acc[c] >> t.precision);
}
src = mid.data();
}
if (out_height == height) return src == rgb ? std::vector<uint8_t>(rgb, rgb + size_t(height) * width * 3) : mid;
// then height
const Taps t = taps_for(height, out_height);
const int64_t half = t.precision > 0 ? int64_t(1) << (t.precision - 1) : 0;
std::vector<uint8_t> out(size_t(out_height) * out_width * 3);
const size_t row = size_t(out_width) * 3;
for (int y = 0; y < out_height; ++y) {
const int32_t* w = t.weight.data() + size_t(y) * t.stride;
for (size_t k = 0; k < row; ++k) {
int64_t acc = half;
for (int j = 0; j < t.count[size_t(y)]; ++j) acc += int64_t(src[(size_t(t.first[size_t(y)]) + j) * row + k]) * w[j];
out[size_t(y) * row + k] = clip8(acc >> t.precision);
}
}
return out;
}

std::vector<Piece> preprocess(const uint8_t* rgb, int height, int width, const ImagePlan& plan) {
if (plan.source.height <= 0 || plan.source.width <= 0 || plan.source.grid_h <= 0 || plan.source.grid_w <= 0)
throw std::runtime_error("the image plan is empty");
// one piece: the (h, w) window of an interleaved image at (y0, x0), as normalised planes
const auto planes = [](const std::vector<uint8_t>& img, int img_w, int y0, int x0, const PiecePlan& q) {
Piece p;
p.grid_h = q.grid_h;
p.grid_w = q.grid_w;
p.pixels.resize(size_t(3) * q.height * q.width);
for (int c = 0; c < 3; ++c)
for (int y = 0; y < q.height; ++y)
for (int x = 0; x < q.width; ++x)
p.pixels[(size_t(c) * q.height + y) * q.width + x] =
(float(img[(size_t(y0 + y) * img_w + size_t(x0 + x)) * 3 + c]) - 127.5f) / 127.5f;
return p;
};
std::vector<Piece> out;
out.push_back(planes(resize_bicubic_u8(rgb, height, width, plan.source.height, plan.source.width), plan.source.width, 0, 0,
plan.source));
if (plan.slice_rows > 0) {
if (plan.refine_height != plan.slice.height * plan.slice_rows || plan.refine_width != plan.slice.width * plan.slice_cols)
throw std::runtime_error("the image plan's slices do not tile its refine size");
const std::vector<uint8_t> refine = resize_bicubic_u8(rgb, height, width, plan.refine_height, plan.refine_width);
for (int r = 0; r < plan.slice_rows; ++r)
for (int c = 0; c < plan.slice_cols; ++c)
out.push_back(planes(refine, plan.refine_width, r * plan.slice.height, c * plan.slice.width, plan.slice));
}
return out;
}

} // namespace onebit::npu::vision
51 changes: 51 additions & 0 deletions npu/vision_image.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
// Copyright 2026 bong-water-water-bong
// SPDX-License-Identifier: Apache-2.0
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//
// A MiniCPM-V 4.7 image from RGB bytes to what the vision tower takes (engine#1788): resize, slice, normalise.
//
// plan_image() (vision_plan.h) says which sizes; this does the pixels. The pieces come out in the order the reference
// processor feeds the tower: the whole view first, then the slices row by row. Each is (3, height, width) float32,
// value (byte - 127.5) / 127.5, i.e. rescale by 1/255 then mean 0.5 and std 0.5 (preprocessor_config.json), the same
// for all three channels. Input and planes are in R, G, B order: byte 0 is exactly -1, byte 255 exactly +1.
//
// The resize is bicubic with antialiasing on BYTES, and it is torch's, bit for bit: torch.nn.functional.interpolate(
// uint8, mode="bicubic", antialias=True) at torch 2.13. That is integer arithmetic, so it can be exact, and three things
// about it are load bearing (tests/npu_vision_image_test.cpp has the reference's bytes):
// - the kernel is the a = -0.5 cubic, stretched by the scale when shrinking, normalised per output pixel;
// - weights become integers at the largest precision whose biggest weight still fits int16 (NOT PIL's fixed 22 bits:
// that differs from torch by one in about 1 % of bytes);
// - width first, then height, with the intermediate image rounded to bytes in between. A pass whose size does not
// change is skipped.
#pragma once

#include "vision_plan.h"

#include <cstdint>
#include <vector>

namespace onebit::npu::vision {

// `rgb` is (height, width, 3) interleaved bytes; the result is (out_height, out_width, 3). Throws on empty sizes.
std::vector<uint8_t> resize_bicubic_u8(const uint8_t* rgb, int height, int width, int out_height, int out_width);

struct Piece {
int grid_h = 0, grid_w = 0; // patches: what tower_forward() takes as gh, gw
std::vector<float> pixels; // (3, grid_h * patch, grid_w * patch)
};

// Every piece of `plan` (which must be plan_image(height, width, ...) for this image), source first.
std::vector<Piece> preprocess(const uint8_t* rgb, int height, int width, const ImagePlan& plan);

} // namespace onebit::npu::vision
Loading
Loading