From 117e82c4ad0ba9b447685e1fc6d3059a3fb21cb4 Mon Sep 17 00:00:00 2001 From: t Date: Mon, 9 Nov 2026 23:54:41 -0400 Subject: [PATCH] npu ep: the probe path never declared the `_b8` layout, so every b8 design it loaded was packed row-major and measured wrong MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `I8Ctx::blocked_b` defaults to false and only two callers ever set it — the forward (`q4nx_forward.cpp:1303`, from the container's `q4nx_weight_layout`) and the xdna target (`xdna_target.cpp:164`). The EP/probe path (`onebit_npu_gemm.cpp`, the code behind `onebit_npu_gemm_probe` and `ONEBIT_EP_NPU_ELF(S)`) creates its `I8Ctx` and calls `init_elf` without touching it, so a `_b8` ELF got row-major-packed B on every call. That is not a wrong answer you can see as wrong. Measured on the device, one npu-hold ticket each: SHIPPED, device-proven 4B set full_i8_O_K4096_N2560_b8_m16 max_abs=5.763e+01 max_rel=4.195e+04 FAIL newly built 3584 family full_i8_QKV_K3584_N4608_b8_m16 max_abs=5.879e+01 max_rel=4.622e+04 FAIL full_i8_O_K3584_N3584_b8_m16 max_abs=6.336e+01 max_rel=5.560e+04 FAIL A set that already serves NPU rows (the 4B family) fails the same way as a set built minutes earlier, at the same magnitude — which is what exposed this: the harness, not the kernels. `fit got~scale*ref` came out 0.0002–0.0078, i.e. no scale relationship at all, which is the layout end rather than a quantisation end. The fix reads the layout off the **file name**, which is already the family contract (the forward derives the same `_b8` from the container key), with an explicit `ONEBIT_EP_NPU_BLOCKED_B` winning over it so a mis-named file stays testable: kernel->ctx->blocked_b = elf_declares_b8(elf); // before init_elf, at both creation sites Verified: the translation unit compiles clean (`g++ -fsyntax-only -std=c++20 -DONEBIT_NPU_RUNTIME_OWN=1` with the private runtime headers). **The device re-run is not done, and I will not claim it**: every NPU-enabled build tree on this box is either 121 commits stale (`wt/tprs-engine-s`) or points at deleted sources (`lb/onebp-ort-npu`), so an after-fix PASS needs a fresh configure. Whoever has one can settle it in one ticket: the shipped 4B `O` shape should go from `max_abs 5.76e1` to within the 2 % bound. Why this matters beyond my row: **any `_b8` family "verified" through this probe since the b8 layout landed was verified by a harness that could not speak it.** A `READY` on a kernel PR citing `onebit_npu_gemm_probe` for a `_b8` design should be re-read. Side observation from the same ticket: `~/ogaload/oga13/kernels-05b-hyb-b8/full_i8_O_K1024_N1024_b8_m16.elf` fails to *load* at all ("npu runtime: invalid program file (code 401)") while the 4B set loads — two generations of ELF in the same cache directory, and the older one is not readable by the current runtime. --- npu/ort/onebit_npu_gemm.cpp | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/npu/ort/onebit_npu_gemm.cpp b/npu/ort/onebit_npu_gemm.cpp index dfdbb0b65..68419e6f0 100644 --- a/npu/ort/onebit_npu_gemm.cpp +++ b/npu/ort/onebit_npu_gemm.cpp @@ -119,6 +119,19 @@ bool env_int(const char* name, int* out) { return true; } +// Whether the configured ELF is a `_b8` (device-blocked B) design. THIS WAS MISSING and it made every `_b8` design +// unjudgeable through the probe path: `I8Ctx::blocked_b` defaults to false and only the forward and the xdna target ever set +// it, so ONEBIT_EP_NPU_ELF loading a `_b8` kernel packed B row-major and returned a plausible wrong answer at every shape. +// Measured 2026-11-10 with onebit_npu_gemm_probe on the device: a *device-proven* 4B family set (full_i8_O_K4096_N2560_b8_m16) +// and a freshly built 3584 set both FAILed with max_abs 5.8-6.3e1 and max_rel ~4-5e4 -- i.e. the probe was measuring its own +// packing, not the kernel. The suffix IS the family contract (the forward derives the same `_b8` from the container's +// q4nx_weight_layout), and an explicit ONEBIT_EP_NPU_BLOCKED_B wins over the name so a mis-named file stays testable. +bool elf_declares_b8(const std::string& elf) { + if (const char* v = env("ONEBIT_EP_NPU_BLOCKED_B")) return strtol(v, nullptr, 10) != 0; + const std::string stem = elf.substr(elf.find_last_of('/') == std::string::npos ? 0 : elf.find_last_of('/') + 1); + return stem.find("_b8") != std::string::npos; +} + bool fail(State& s, const std::string& why) { s.why = why; s.ok = false; @@ -206,6 +219,7 @@ bool init(State& s) { kernel->n = n; kernel->elf = elf; kernel->ctx = std::make_unique(); + kernel->ctx->blocked_b = elf_declares_b8(elf); if (!kernel->ctx->init_elf(*s.dev, elf.c_str(), m, k, n, 1)) { // One unloadable design must cost that design, not the device. This used to fail the whole State, // which meant a single bad entry in ONEBIT_EP_NPU_ELFS (an ELF from another kernel family, a @@ -276,6 +290,7 @@ Q8Kernel* q8_kernel(State& s, int64_t k, int64_t n) { kern->n = nn; kern->elf = elf; kern->ctx = std::make_unique(); + kern->ctx->blocked_b = elf_declares_b8(elf); if (!kern->ctx->init_elf(*s.dev, elf.c_str(), m, kk, nn, 1)) { s.q8_unavailable.push_back( {k, n, "I8Ctx init_elf failed for " + elf + " (" + dims + ") -- check the design M"});