Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 19 additions & 1 deletion docs/npu.md
Original file line number Diff line number Diff line change
Expand Up @@ -1922,7 +1922,9 @@ The gate does exactly what it was asked to do — **the drafts it keeps are far
rate, and depth 1 is identical to the offline protocol at the same positions, +0.00 pp) — and it does it by **almost
never chaining**: 292 drafts over 277 chains, i.e. ~1.05 per chain against 4.0, so the chain collapses from 2.87 to
**1.83 tokens/pass**. That is below the measured 3.2-4.1 break-even, so at the default the draft route is a
**slowdown**: 0.81× on the prompt-0 pair ticket (plain 0.849 s/token, draft 1.049 s/token). The conditional
**slowdown**: **0.48×** re-measured at main `4480a6df` on the prompt-0 pair ticket (plain 0.482 s/token, draft
1.008 s/token; the same pair read 0.81× on 2026-11-06 — plain decode has since become ~1.8× faster while the loop's
two-passes-per-iteration cost has not moved, so the default's penalty doubled). The conditional
per-depth comparison at this `tau` reads +27 and +47 pp at depths 2-3, on 13 and 2 live chains — sampling noise, which
is why the table above is the all-chains view and why depth 1 is the only cell here worth comparing.

Expand All @@ -1943,6 +1945,22 @@ a silently broken loop rather than a slow one.
lever) or a head that reliably produces a chain above 3.9 (widening `k` is not it: `k = 6` bought +0.05, and depth
6 measured 0 %).

**One such lever has since landed, and it does not reach this target.** engine#1348 made a batched verify pass launch
the **int8 packed** lm head once per vocabulary tile instead of once per row (3.4× the pass, 18.7× the head on its own
bench). Its entry path is gated on `!head_pb16i_` (`npu/forward/src/q4nx_forward.cpp:4755`), and this target's head **is**
pb16i — `head_pb16i_` is set from the pack's own head images — which already had a row-batched path
(`rows_supported`, `:4822`). Re-measured here after it merged: the pair ticket is **0.80× at `tau = 0`** (plain 0.870,
draft 1.089 s/token) and **0.48× at `tau = 0.6`**, i.e. unchanged for the loop and *worse* as a ratio because plain
decode improved. So the lever is real, it is just aimed at the other head format: a pb16i-head target gets nothing from
it, and the recommendation below stands.

**Re-confirmed at main `4480a6df` after ~1.4k lines of forward refactoring** (`#808` NpuTarget, `#1330` plan kernel sets,
`#1310`, `#1348`): the acceptance numbers are **digit for digit** what the windows published — prompt 0 gives `44/80`
accepted with depths `18/20, 13/20, 7/20, 6/20` at `tau = 0`, and `31/35` with `28/32, 2/2, 1/1, 0/0` at `tau = 0.6`,
`TEXT IDENTICAL`, `stream 72 rows`, `0 mismatch` in both. The loop's contracts survived the refactor with it:
`last_hidden_rows_ = norm_all` and `last_hidden_ = fnorm` (`q4nx_forward.cpp:3558`, `:3569`) and
`commit_verify_rows` truncating to the accepted prefix (`:2348`).

**Leaf fill is resolved, and the answer is that this verifier cannot use it.** The plan's last protocol item (#805) asks
for "spare verify rows filled with the drafts' 2nd/3rd candidates". The acceptance rule here is
`argmax(logits[i-1]) == draft[i]` (`npu/mtp/draft.h:317`): a draft is accepted iff it **is** the base's own argmax at
Expand Down
Loading