Skip to content

npu: the NPU_BATCH_CHECK oracle resets the cache before its sequential replay, so it reports a MISMATCH for any chunk that does not start at 0 #1905

Closed

Description

@bong-water-water-bong

What happens

With the batch width at 1 (before #1904), a four-id npu-run chunked into four single-row passes, and the debug oracle reported:

[batchcheck] M=1 start=0 seq_argmax=198 batch_argmax=198 MATCH
[batchcheck] rows M=1 identical=1 differing=0
[batchcheck] M=1 start=1 seq_argmax=220 batch_argmax=653 MISMATCH
[batchcheck] rows M=1 identical=0 differing=1 first row=0 index=0 seq=0x4081c73b bat=0x403a8c18

The first row matches, every row after it does not.

Why

npu/forward/src/q4nx_forward.cpp (~:3362):

if (batch_check && !inject_embeds_) {
    std::vector<float> seq;
    for (int m = 0; m < M; m++) {
        if (!step(tokens[m], start_pos + m, seq)) return false;
        seq_rows.insert(...);
    }
    reset();          // <-- discards the state the batched pass continues from
}

The sequential replay and the batched pass are not run from the same state: the replay walks start_pos .. start_pos+M-1, then
reset() empties the cache, and the batched pass that follows starts at start_pos with no prefix. So for any chunk whose
start_pos > 0 the comparison is "one token at position 1 from scratch" against "position 1 continuing the previous chunk", and
it reports a divergence that is not in the kernels.

Why it matters

NPU_BATCH_CHECK is the oracle #1781 gate 1 reads (identical=N differing=0). On a pack whose width forces chunking it reports a
divergence for every chunk but the first — indistinguishable from a real row-dependence bug, and it sends the reader to the kernels
for a defect in the debug path. After #1904 the packs on this box make a single pass, so the oracle is meaningful again; the defect
is latent until a prompt is wider than the design (M=128 prefill chunks a long prompt, and every chunk after the first will
"diverge").

What would fix it

Snapshot the state before the sequential replay and restore it instead of reset(), or bypass the oracle when start_pos > 0 and
say so once — the second is honest and cheap, the first is exact. Either way the line should state which positions it compared.

Measured 2026-11-10 on models/npu/bonsai2-27b-NPU-b8 under an npu-hold ticket, 1bit built from fix/1900-batch-width before its
fallback (i.e. width 1). Refs #1781, #1900, #1904.

Activity

  1. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claiming this — lane pi-onebp (unowned, and it is the oracle whose registration I reviewed in #1858). I will take the exact option if the forward's own snapshot machinery fits — q4nx_forward.cpp already keeps snap_rec_/rec_state_ for the recurrent path — and otherwise the cheap one, and either way the line will state which positions it compared, as the issue asks.

    No device is needed for the decision; the state-save/restore is a host-side change against the debug oracle, so I expect a crafted arm rather than a ticket.

  2. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    PR #1909 takes the exact option, using machinery the forward already has: snapshot_state(start_pos, snap) before the sequential replay and restore_state(snap, start_pos) after it — the same pair the prefix cache uses, with q4nx_state_planes.h as its device-free plane-copy half and reset() kept for start_pos == 0 where it is exactly right, so the change is inert for the single-pass case that works today.

    If the snapshot cannot be taken the oracle now refuses with a message instead of comparing two different states again.

    And the output names the positions: [batchcheck] M=… positions=<start>..<start+M-1> rather than only start=, which is the half of your ask that stops a bad comparison from looking like a good one.

    The end-to-end arm is blocked on the same thing your note names — a pack whose kernels declare a pass width > 1 (the 3584 family is built -M 128) — so this PR carries the fix plus the honest output, and states that limit rather than implying coverage.

  3. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Fixed: PR #1914 at 8ebfc05df, with the before/after on identical inputs and the same pack.

    The replay now snapshots the prefix at start_pos and restores it instead of reset()ing, so both sides start from the same state and a chunked pass is compared exactly rather than skipped. start_pos == 0 has no prefix (snapshot_state(0, …) refuses by definition) and a snapshot that did not take falls back to reset() — a failed snapshot is not a licence to compare the batch against the replay's own state.

    chunk before (reset()) after
    start=0 MATCH MATCH
    start=1 MISMATCH (0x411bf754 vs 0x409d210c) MATCH
    start=2, start=3, start=4 MATCH MATCH

    That middle row is why this was worth chasing: on imp1519b only start=1 diverged, and start=2/3 "matched" because the argmax happened to coincide at id 0. A green line for one chunk said nothing about the next, which is exactly the shape of a debug tool that makes you distrust a kernel that is fine.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions