Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -1213,6 +1213,14 @@ target_link_libraries(npu_pack_test PRIVATE onebit_npu)
# golden fixture that pins them.
target_include_directories(npu_pack_test PRIVATE npu/forward/include ${CMAKE_SOURCE_DIR}/tests)

# NPU_INFER_DUMP_OPS (npu/forward/include/ops-dump.h, engine#1503): the on/off rule, the mean|x|/absmax arithmetic and
# the line's text. Header-only, no device. `add_dependencies(onebit ...)` is what gets it built in CI, whose target
# list is explicit (tools/ci/ci_targets_check.py).
add_executable(ops_dump_test tests/ops_dump_test.cpp)
target_include_directories(ops_dump_test PRIVATE npu/forward/include)
add_test(NAME ops_dump COMMAND ops_dump_test)
add_dependencies(onebit ops_dump_test)

# The device weight layout contract (q4nx_weight_layout): pure index math, no device
# and no runtime. CI builds an explicit target list, so the check compiles itself with
# the build's compiler at test time (tests/npu_blocked_layout_test.sh) rather than being
Expand Down
28 changes: 28 additions & 0 deletions docs/npu.md
Original file line number Diff line number Diff line change
Expand Up @@ -2004,6 +2004,34 @@ reaps or signals the holder -- killing another lane's live serve is the box-cras
Qwen3-4B, whose kernel sets are on this box): each model solo, then both at once, three reps, every
number stamped with the build that produced it.

### Finding where a forward diverges: `NPU_INFER_DUMP_DIR` and `NPU_INFER_DUMP_OPS`

Two diagnostic switches of the NPU forward. Both are off by default, neither changes a result, and neither
can fail a run.

| variable | what it writes | answers |
|---|---|---|
| `NPU_INFER_DUMP_DIR=<dir>` | `<dir>/layer_<ll>.f32` per layer: the residual stream after that layer, M x H little-endian f32, row-major. The directory must exist; a file that cannot be written is reported on stderr and the run goes on. | which **layer** two runs part at (per-layer cosine against a reference trace) |
| `NPU_INFER_DUMP_OPS=1` | one stderr line per projection and per weighted RMSNorm: `[op] L<layer> <tag> rows=<M> \|x\| mean <m> absmax <a>` | which **op** inside the layer |

`NPU_INFER_DUMP_OPS` is on for any value other than empty or `0`, and is read once per process.

```
[op] L00 rmsnorm rows=1 |x| mean 0.0143171 absmax 3.01935
[op] L00 ffn_gate_up rows=1 |x| mean 0.0258482 absmax 1.25728
[op] L00 ffn_down rows=1 |x| mean 0.00315921 absmax 0.673885
```

- The tag is the one the projection's call site already passes (`attn_qkv`, `attn_output`, `ffn_gate_up`,
`ffn_down`, `ssm_out`, ...). Layer norms are `rmsnorm`; on a family with per-head q/k norms those print as
`q_norm` and `k_norm`, one line per head.
- The numbers are over row 0 of the result (its first N values). A batched projection prints one line with
`rows=M`; a norm is called per row, so a batched pass prints one norm line per row.
- A NaN or Inf is counted (`nonfinite 2/4096` at the end of the line) and left out of the mean and the maximum.
- The same lines come out on the host arms (`NPU_INFER_HOST_GEMM=1` or `=i8`) and on the device arm, in the same
order and number, so the two logs of one prompt can be compared with `diff`. Start there: the first line whose
numbers differ names the op.

### When a load follows an evict

On the appliance the NPU model is swapped by evicting it and starting a fresh `1bit` for the next
Expand Down
112 changes: 112 additions & 0 deletions npu/forward/include/ops-dump.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
// Copyright 2026 bong-water-water-bong
// SPDX-License-Identifier: Apache-2.0
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//
// ops-dump.h -- NPU_INFER_DUMP_OPS=1 prints mean|x| and absmax after every projection and every weighted RMSNorm,
// tagged with the call site's own name:
//
// [op] L00 rmsnorm rows=1 |x| mean 0.0105 absmax 0.0598
// [op] L00 attn_qkv rows=1 |x| mean 1.3877 absmax 31.8909
//
// Why it exists: NPU_INFER_DUMP_DIR (residual-dump.h) writes the residual stream at LAYER boundaries, which says
// that a model diverges and not which op did it (engine#1503: a residual already 1e4 too large at layer 0 while
// the container matched its pack tensor for tensor). The insertion points are `rmsnorm()`, `npu_gemm()` and
// `npu_gemm_rows()`; the tag each projection already carries is the label, so there is no second name table.
// The line is printed on the host arms and on the device arm alike, so the same run under NPU_INFER_HOST_GEMM=1
// and without it can be compared line for line: a batched projection prints one line with rows=M on every arm,
// also when the device arm walks it in blocks. The per-head q and k norms of a qk-norm family are tagged
// "q_norm" and "k_norm" (one line per head), so they cannot be mistaken for the layer norms.
//
// Row 0 only (the first N values of a batched result): one line per call, which is what localises a divergence.
// A norm is called per row, so a batched pass prints one norm line per row.
//
// Off unless the variable is set to something other than "" or "0". The variable is read ONCE per process
// (ops_dump_enabled() is called per projection per token, and off has to cost a load and a branch, not a
// getenv). It can never fail a run: no allocation, no file, stderr only.
#pragma once

#include <cmath>
#include <cstdio>
#include <cstdlib>
#include <cstring>

namespace onebit::npu {

// The rule itself, on a value: what the tests pin (the cached read below cannot be re-armed inside one process).
inline bool ops_dump_value_on(const char* v) { return v && *v && std::strcmp(v, "0") != 0; }

inline bool ops_dump_enabled() {
static const bool on = ops_dump_value_on(std::getenv("NPU_INFER_DUMP_OPS"));
return on;
}

// A projection that is carried out as several inner calls (the device arm's blocked launches) holds one of these
// across them, so the inner calls stay quiet and the outer call prints the one line. Per thread.
inline int& ops_dump_mute_depth() {
thread_local int depth = 0;
return depth;
}
struct OpsDumpMute {
OpsDumpMute() { ++ops_dump_mute_depth(); }
~OpsDumpMute() { --ops_dump_mute_depth(); }
OpsDumpMute(const OpsDumpMute&) = delete;
OpsDumpMute& operator=(const OpsDumpMute&) = delete;
};
inline bool ops_dump_muted() { return ops_dump_mute_depth() > 0; }

struct OpsStats {
double mean = 0.0; // mean of |x[i]|
float absmax = 0.0f; // max of |x[i]|
int nonfinite = 0; // NaN/Inf count: they are counted, and left out of mean and absmax
};

// The whole content of the diagnostic. A NaN would otherwise vanish from absmax (`a > mx` is false) and turn the
// mean into "nan" with no count, so non-finite values are reported as a number of their own.
inline OpsStats ops_stats(const float* x, int n) {
OpsStats s;
if (!x || n <= 0) return s;
double sum = 0.0;
int finite = 0;
for (int i = 0; i < n; ++i) {
if (!std::isfinite(x[i])) { ++s.nonfinite; continue; }
const float a = std::fabs(x[i]);
sum += a;
if (a > s.absmax) s.absmax = a;
++finite;
}
if (finite > 0) s.mean = sum / finite;
return s;
}

// Formats the line into `buf` (always NUL-terminated when cap > 0) and returns snprintf's count; 0 when there is
// nothing to say. Separate from the print so a test can match the text.
inline int ops_dump_line(char* buf, size_t cap, const char* tag, int layer, int rows, const float* x, int n) {
if (!buf || cap == 0) return 0;
buf[0] = '\0';
if (!x || n <= 0) return 0;
const OpsStats s = ops_stats(x, n);
if (s.nonfinite)
return std::snprintf(buf, cap, " [op] L%02d %-12s rows=%d |x| mean %.6g absmax %.6g nonfinite %d/%d\n", layer,
tag ? tag : "?", rows, s.mean, (double)s.absmax, s.nonfinite, n);
return std::snprintf(buf, cap, " [op] L%02d %-12s rows=%d |x| mean %.6g absmax %.6g\n", layer, tag ? tag : "?",
rows, s.mean, (double)s.absmax);
}

inline void ops_dump(const char* tag, int layer, int rows, const float* x, int n) {
if (ops_dump_muted()) return;
char buf[192];
if (ops_dump_line(buf, sizeof buf, tag, layer, rows, x, n) > 0) std::fputs(buf, stderr);
}

} // namespace onebit::npu
3 changes: 2 additions & 1 deletion npu/forward/include/q4nx_forward.h
Original file line number Diff line number Diff line change
Expand Up @@ -283,7 +283,8 @@ class Q4nxNpuForward {
// The same elementwise gate*up on caller memory, so a batched pass can read its ACT row and write its HH
// row where they already are instead of staging both through a vector (a copy on each side).
bool mlp(const float* gu, float* h, int im);
float rmsnorm(const float* x, const float* w, int n, float* out) const;
// `tag` is the NPU_INFER_DUMP_OPS label (ops-dump.h); the per-head q/k norms pass their own.
float rmsnorm(const float* x, const float* w, int n, float* out, const char* tag = "rmsnorm") const;
float rmsnorm_noweight(const float* x, int n, float* out) const;
bool run_lm_head(const std::vector<float>& h, std::vector<float>& logits);
// M rows of hidden state -> M x NV logits, row-major. Same head, same packed tiles: the rows ride
Expand Down
36 changes: 31 additions & 5 deletions npu/forward/src/q4nx_forward.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@
#include <cmath>
#include <cstdio>
#include "batch_check_compare.h"
#include "ops-dump.h"
#include "residual-dump.h"
#include <cstdlib>
#include <cstring>
Expand Down Expand Up @@ -3211,6 +3212,14 @@ void Q4nxNpuForward::host_gemm_i8(const float* A, int a_k, int M, int K, int N,
}
}

// NPU_INFER_DUMP_OPS (ops-dump.h): one line per projection, tagged with the tag its caller already passes
// ("attn_qkv", "ssm_out", ...). Called where a projection's result is final -- both host arms and the device arm --
// so a host run and a device run print the same lines. Row 0 of the result; off is one cached bool.
static inline void op_line(const char* tag, int layer, int rows, const std::vector<float>& out, int n) {
if (onebit::npu::ops_dump_enabled() && n > 0 && out.size() >= (size_t)n)
onebit::npu::ops_dump(tag, layer, rows, out.data(), n);
}

void Q4nxNpuForward::npu_gemm(int role, const float* A, int K, int N, const float* Bmat,
std::vector<float>& out, int wkey, const char* sc, int a_k) {
if (host_gemm_) {
Expand All @@ -3221,9 +3230,11 @@ void Q4nxNpuForward::npu_gemm(int role, const float* A, int K, int N, const floa
out.assign((size_t)N, 0.0f);
if (host_gemm_i8_) {
host_gemm_i8(A, a_k, 1, K, N, Bmat, wkey, out.data());
op_line(sc, cur_layer_, 1, out, N);
return;
}
q4nx_host_gemm_f32_padded(A, a_k, 1, K, N, Bmat, out.data());
op_line(sc, cur_layer_, 1, out, N);
return;
}
npu::target::GemvArgs a;
Expand All @@ -3232,7 +3243,8 @@ void Q4nxNpuForward::npu_gemm(int role, const float* A, int K, int N, const floa
a.a_k = a_k;
a.tiles = &Q4nxNpuForward::gemv_tile_cb; a.tiles_self = this;
std::string err;
if (!xd_ || !xd_->gemv(a, out, err)) err_ = err.empty() ? "npu_gemm: no target" : err;
if (!xd_ || !xd_->gemv(a, out, err)) { err_ = err.empty() ? "npu_gemm: no target" : err; return; }
op_line(sc, cur_layer_, 1, out, N);
}

void Q4nxNpuForward::npu_gemm_rows(int role, const float* A, int M, int K, int N,
Expand All @@ -3244,9 +3256,11 @@ void Q4nxNpuForward::npu_gemm_rows(int role, const float* A, int M, int K, int N
out.assign((size_t)M * N, 0.0f);
if (host_gemm_i8_) {
host_gemm_i8(A, a_k, M, K, N, Bmat, wkey, out.data());
op_line(sc, cur_layer_, M, out, N);
return;
}
q4nx_host_gemm_f32_padded(A, a_k, M, K, N, Bmat, out.data());
op_line(sc, cur_layer_, M, out, N);
return;
}
// The device's activation BO holds exactly max_batch_rows() rows (the M=128 prefill design), and a wider
Expand All @@ -3266,6 +3280,9 @@ void Q4nxNpuForward::npu_gemm_rows(int role, const float* A, int M, int K, int N
if (note && note[0]) { std::fprintf(stderr, " [batch] %s\n", note); note_said = true; }
}
out.assign((size_t)M * N, 0.0f);
// NPU_INFER_DUMP_OPS: the blocks are one projection, so they print one line (rows=M, row 0), as the host
// arms do; the inner launches are muted for the length of the walk.
onebit::npu::OpsDumpMute ops_mute;
for (int b0 = 0; b0 < M && err_.empty(); b0 += blk) {
const int rows = std::min(blk, M - b0);
std::vector<float> part;
Expand All @@ -3277,6 +3294,12 @@ void Q4nxNpuForward::npu_gemm_rows(int role, const float* A, int M, int K, int N
}
std::copy(part.begin(), part.begin() + (size_t)rows * N, out.begin() + (size_t)b0 * N);
}
// Printed by hand: op_line() would be muted by this scope's own guard. A nested walk stays quiet.
if (err_.empty() && onebit::npu::ops_dump_enabled() && onebit::npu::ops_dump_mute_depth() == 1) {
char line[192];
if (onebit::npu::ops_dump_line(line, sizeof line, sc, cur_layer_, M, out.data(), N) > 0)
std::fputs(line, stderr);
}
return;
}
npu::target::GemvArgs a;
Expand All @@ -3286,7 +3309,8 @@ void Q4nxNpuForward::npu_gemm_rows(int role, const float* A, int M, int K, int N
a.rows = true; a.decode_twin = decode_twin;
a.tiles = &Q4nxNpuForward::gemv_tile_cb; a.tiles_self = this;
std::string err;
if (!xd_ || !xd_->gemv(a, out, err)) err_ = err.empty() ? "npu_gemm_rows: no target" : err;
if (!xd_ || !xd_->gemv(a, out, err)) { err_ = err.empty() ? "npu_gemm_rows: no target" : err; return; }
op_line(sc, cur_layer_, M, out, N);
}

// The folded container's sign table: magic "HAD1", u32 count, count x (u32 width, u32 offset),
Expand Down Expand Up @@ -4021,7 +4045,7 @@ int Q4nxNpuForward::max_batch_rows() const {
// --------------------------------------------------------------------------
// host helpers (float32, matching the Python reference)
// --------------------------------------------------------------------------
float Q4nxNpuForward::rmsnorm(const float* x, const float* w, int n, float* out) const {
float Q4nxNpuForward::rmsnorm(const float* x, const float* w, int n, float* out, const char* tag) const {
LhScope _tlh(LH_RMS);
double ss = 0.0;
for (int i = 0; i < n; i++) ss += (double)x[i] * (double)x[i];
Expand All @@ -4032,6 +4056,8 @@ float Q4nxNpuForward::rmsnorm(const float* x, const float* w, int n, float* out)
} else {
for (int i = 0; i < n; i++) out[i] = x[i] * inv * w[i];
}
// NPU_INFER_DUMP_OPS: the norm's own output, so a divergence can be put on the norm and not on its layer.
if (onebit::npu::ops_dump_enabled()) onebit::npu::ops_dump(tag, cur_layer_, 1, out, n);
return inv;
}

Expand Down Expand Up @@ -4475,12 +4501,12 @@ bool Q4nxNpuForward::attention(const std::vector<float>& qkv, int pos, const int
if (sem_.qk_norm) {
std::vector<float> t(HD);
for (int h = 0; h < NH; h++) {
rmsnorm(&q[(size_t)h * HD], q_norm_[cur_layer_].data(), HD, t.data());
rmsnorm(&q[(size_t)h * HD], q_norm_[cur_layer_].data(), HD, t.data(), "q_norm");
memcpy(&q[(size_t)h * HD], t.data(), sizeof(float) * HD);
}
if (append_kv) {
for (int h = 0; h < NKV; h++) {
rmsnorm(&k[(size_t)h * HD], k_norm_[cur_layer_].data(), HD, t.data());
rmsnorm(&k[(size_t)h * HD], k_norm_[cur_layer_].data(), HD, t.data(), "k_norm");
memcpy(&k[(size_t)h * HD], t.data(), sizeof(float) * HD);
}
}
Expand Down
Loading
Loading