From 8ebfc05dffe673a4bcedb769c577218dfd9dadfe Mon Sep 17 00:00:00 2001 From: t Date: Tue, 10 Nov 2026 07:48:34 -0400 Subject: [PATCH] npu: the batch oracle restores the prefix instead of resetting, so a chunked pass stops reporting a false MISMATCH (fixes #1905) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `NPU_BATCH_CHECK` runs the tokens sequentially, then `reset()`s and lets the batched pass run. The reset threw away the state the batched pass **continues from**, so for every chunk at `start_pos > 0` the two sides ran from different states — the replay from an empty cache at position `start`, the batch from the caller's prefix — and the oracle reported a divergence that was not in the kernels. #1900's chunking (width 1 on a sealed pack) made it fire on four of four ids; it is latent for any prompt wider than the M=128 design. The replay now snapshots the prefix at `start_pos` and restores it, so both sides start from the same state and the comparison stays exact for a chunked pass instead of being disabled for one. `start_pos == 0` has no prefix (and `snapshot_state(0, …)` is defined to refuse), and a snapshot that did not take is not a licence to compare against the replay's own state: both fall back to `reset()`. ## Measured before/after on identical inputs, device-free `~/scratch-packs/imp1519b` (host GEMM, so the width is 1 and every id is its own chunk — i.e. every `start_pos > 0` case), `--ids "1 2 3 4 5"`, `NPU_BATCH_CHECK=1`: | | before | after | |---|---|---| | `start=0` | MATCH | MATCH | | `start=1` | **MISMATCH** (`0x411bf754` vs `0x409d210c`) | MATCH | | `start=2` | MATCH | MATCH | | `start=3` | MATCH | MATCH | | `start=4` | MATCH | MATCH | The same two binaries, the same ids, the same pack. Note that only `start=1` diverged: `start=2` and `3` "matched" because the argmax happened to coincide at id 0 — which is why the oracle keeps the whole row and compares it bitwise, and why a green line for one chunk says nothing about the next. Fixes #1905. Refs #1900, #1904, #1781. --- npu/forward/src/q4nx_forward.cpp | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/npu/forward/src/q4nx_forward.cpp b/npu/forward/src/q4nx_forward.cpp index a8e503162..0c1b8d235 100644 --- a/npu/forward/src/q4nx_forward.cpp +++ b/npu/forward/src/q4nx_forward.cpp @@ -3371,6 +3371,12 @@ bool Q4nxNpuForward::step_batch(const std::vector& tokens, int start_pos, if (batch_check && !inject_embeds_) { std::vector seq; if (c_.NV > 0) seq_rows.reserve((size_t)M * (size_t)c_.NV); + // engine#1905: keep the state the batched pass continues from. This used to `reset()`, which emptied the cache the + // replay had just filled AND dropped the caller's prefix -- so for every chunk at start_pos > 0 the two sides ran + // from different states ("one token at position start from scratch" vs "position start continuing the previous + // chunk") and the oracle reported a MISMATCH that was not in the kernels. Snapshot the prefix, replay, restore. + StateSnapshot oracle_prefix; + const bool had_prefix = start_pos > 0 && snapshot_state(start_pos, oracle_prefix); for (int m = 0; m < M; m++) { if (!step(tokens[m], start_pos + m, seq)) return false; seq_rows.insert(seq_rows.end(), seq.begin(), seq.end()); // every row, not only the last @@ -3378,7 +3384,9 @@ bool Q4nxNpuForward::step_batch(const std::vector& tokens, int start_pos, for (size_t i = 0; i < seq.size(); i++) if (seq_argmax < 0 || seq[i] > seq[(size_t)seq_argmax]) seq_argmax = (int)i; } - reset(); + // start_pos == 0 has no prefix to keep, and a snapshot that did not take is not a reason to compare against the + // replay's own state: both fall back to reset(), and the row comparison below still applies. + if (!(had_prefix && restore_state(oracle_prefix, start_pos))) reset(); } // Batch phase attribution: the same partition step() prints, for an M-row verify pass, so the