diff --git a/docs/npu.md b/docs/npu.md index d18594f05..fbdd961a4 100644 --- a/docs/npu.md +++ b/docs/npu.md @@ -1922,7 +1922,9 @@ The gate does exactly what it was asked to do — **the drafts it keeps are far rate, and depth 1 is identical to the offline protocol at the same positions, +0.00 pp) — and it does it by **almost never chaining**: 292 drafts over 277 chains, i.e. ~1.05 per chain against 4.0, so the chain collapses from 2.87 to **1.83 tokens/pass**. That is below the measured 3.2-4.1 break-even, so at the default the draft route is a -**slowdown**: 0.81× on the prompt-0 pair ticket (plain 0.849 s/token, draft 1.049 s/token). The conditional +**slowdown**: **0.48×** re-measured at main `4480a6df` on the prompt-0 pair ticket (plain 0.482 s/token, draft +1.008 s/token; the same pair read 0.81× on 2026-11-06 — plain decode has since become ~1.8× faster while the loop's +two-passes-per-iteration cost has not moved, so the default's penalty doubled). The conditional per-depth comparison at this `tau` reads +27 and +47 pp at depths 2-3, on 13 and 2 live chains — sampling noise, which is why the table above is the all-chains view and why depth 1 is the only cell here worth comparing. @@ -1943,6 +1945,22 @@ a silently broken loop rather than a slow one. lever) or a head that reliably produces a chain above 3.9 (widening `k` is not it: `k = 6` bought +0.05, and depth 6 measured 0 %). +**One such lever has since landed, and it does not reach this target.** engine#1348 made a batched verify pass launch +the **int8 packed** lm head once per vocabulary tile instead of once per row (3.4× the pass, 18.7× the head on its own +bench). Its entry path is gated on `!head_pb16i_` (`npu/forward/src/q4nx_forward.cpp:4755`), and this target's head **is** +pb16i — `head_pb16i_` is set from the pack's own head images — which already had a row-batched path +(`rows_supported`, `:4822`). Re-measured here after it merged: the pair ticket is **0.80× at `tau = 0`** (plain 0.870, +draft 1.089 s/token) and **0.48× at `tau = 0.6`**, i.e. unchanged for the loop and *worse* as a ratio because plain +decode improved. So the lever is real, it is just aimed at the other head format: a pb16i-head target gets nothing from +it, and the recommendation below stands. + +**Re-confirmed at main `4480a6df` after ~1.4k lines of forward refactoring** (`#808` NpuTarget, `#1330` plan kernel sets, +`#1310`, `#1348`): the acceptance numbers are **digit for digit** what the windows published — prompt 0 gives `44/80` +accepted with depths `18/20, 13/20, 7/20, 6/20` at `tau = 0`, and `31/35` with `28/32, 2/2, 1/1, 0/0` at `tau = 0.6`, +`TEXT IDENTICAL`, `stream 72 rows`, `0 mismatch` in both. The loop's contracts survived the refactor with it: +`last_hidden_rows_ = norm_all` and `last_hidden_ = fnorm` (`q4nx_forward.cpp:3558`, `:3569`) and +`commit_verify_rows` truncating to the accepted prefix (`:2348`). + **Leaf fill is resolved, and the answer is that this verifier cannot use it.** The plan's last protocol item (#805) asks for "spare verify rows filled with the drafts' 2nd/3rd candidates". The acceptance rule here is `argmax(logits[i-1]) == draft[i]` (`npu/mtp/draft.h:317`): a draft is accepted iff it **is** the base's own argmax at