From 9eb84e3492fb2fc338fa3b2624da19ed0df836ad Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Sat, 7 Nov 2026 18:21:17 -0400 Subject: [PATCH] =?UTF-8?q?mtp:=20the=20draft=20route=20at=20main=20?= =?UTF-8?q?=E2=80=94=20acceptance=20re-confirmed=20digit=20for=20digit,=20?= =?UTF-8?q?its=20cost=20ratio=20halved,=20and=20#1348=20does=20not=20reach?= =?UTF-8?q?=20this=20head?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Re-measured the two arms at `main` `4480a6df` after ~1.4k lines of forward refactoring landed under this lane's deliverable (#808 NpuTarget extraction, #1330 plan kernel sets, #1310, #1348). One `npu-hold` ticket per arm, prompt 0, 64 tokens, k=4, `--live` (plain and draft measured in the same process, so the pair is comparable), `amdxdna` errors 0, text identical and 0 stream mismatches in both. Acceptance (the determinism control) is EXACTLY what the windows published: tau=0 44/80 accepted, depths 18/20, 13/20, 7/20, 6/20, hit 0.550, chain 3.20 tau=0.6 31/35 accepted, depths 28/32, 2/2, 1/1, 0/0, hit 0.886, chain 1.97 and the loop's contracts survived the refactor with it: `last_hidden_rows_ = norm_all` (:3558), `last_hidden_ = fnorm` (:3569), `commit_verify_rows` truncating to the accepted prefix (:2348). Cost moved, in the direction that matters for the serve decision: tau=0 plain 0.870 s/token, draft 1.089 s/token -> 0.80x tau=0.6 plain 0.482 s/token, draft 1.008 s/token -> 0.48x (the same pair read 0.81x on 2026-11-06) Plain decode is ~1.8x faster than when 0.81x was recorded while the loop's two-passes-per-iteration cost has not moved, so the default tau's penalty doubled. The docs' 0.81x was stale at main; they now carry the measured pair and both figures. Why #1348 did not help, stated from its own gate: it batches the INT8 packed lm head for M>1, guarded by `!head_pb16i_` (npu/forward/src/q4nx_forward.cpp:4755). This target's head IS pb16i (`head_pb16i_`, set from the pack's own head images) and already had a row-batched path (`rows_supported`, :4822), so the 3.4x pass / 18.7x head win is aimed at the other head format. That is now one paragraph in docs/npu.md's "what would change the answer", because it is the first lever from that list to land and it silently does not apply here. The recommendation does not change, it hardens: do not enable the draft route on a serve at these chain lengths (and at the default tau it is now ~2x slower, not 1.2x). refs #1348 (whose reach this measures), refs #805 (whose measured windows these are). --- docs/npu.md | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/docs/npu.md b/docs/npu.md index d18594f05..fbdd961a4 100644 --- a/docs/npu.md +++ b/docs/npu.md @@ -1922,7 +1922,9 @@ The gate does exactly what it was asked to do — **the drafts it keeps are far rate, and depth 1 is identical to the offline protocol at the same positions, +0.00 pp) — and it does it by **almost never chaining**: 292 drafts over 277 chains, i.e. ~1.05 per chain against 4.0, so the chain collapses from 2.87 to **1.83 tokens/pass**. That is below the measured 3.2-4.1 break-even, so at the default the draft route is a -**slowdown**: 0.81× on the prompt-0 pair ticket (plain 0.849 s/token, draft 1.049 s/token). The conditional +**slowdown**: **0.48×** re-measured at main `4480a6df` on the prompt-0 pair ticket (plain 0.482 s/token, draft +1.008 s/token; the same pair read 0.81× on 2026-11-06 — plain decode has since become ~1.8× faster while the loop's +two-passes-per-iteration cost has not moved, so the default's penalty doubled). The conditional per-depth comparison at this `tau` reads +27 and +47 pp at depths 2-3, on 13 and 2 live chains — sampling noise, which is why the table above is the all-chains view and why depth 1 is the only cell here worth comparing. @@ -1943,6 +1945,22 @@ a silently broken loop rather than a slow one. lever) or a head that reliably produces a chain above 3.9 (widening `k` is not it: `k = 6` bought +0.05, and depth 6 measured 0 %). +**One such lever has since landed, and it does not reach this target.** engine#1348 made a batched verify pass launch +the **int8 packed** lm head once per vocabulary tile instead of once per row (3.4× the pass, 18.7× the head on its own +bench). Its entry path is gated on `!head_pb16i_` (`npu/forward/src/q4nx_forward.cpp:4755`), and this target's head **is** +pb16i — `head_pb16i_` is set from the pack's own head images — which already had a row-batched path +(`rows_supported`, `:4822`). Re-measured here after it merged: the pair ticket is **0.80× at `tau = 0`** (plain 0.870, +draft 1.089 s/token) and **0.48× at `tau = 0.6`**, i.e. unchanged for the loop and *worse* as a ratio because plain +decode improved. So the lever is real, it is just aimed at the other head format: a pb16i-head target gets nothing from +it, and the recommendation below stands. + +**Re-confirmed at main `4480a6df` after ~1.4k lines of forward refactoring** (`#808` NpuTarget, `#1330` plan kernel sets, +`#1310`, `#1348`): the acceptance numbers are **digit for digit** what the windows published — prompt 0 gives `44/80` +accepted with depths `18/20, 13/20, 7/20, 6/20` at `tau = 0`, and `31/35` with `28/32, 2/2, 1/1, 0/0` at `tau = 0.6`, +`TEXT IDENTICAL`, `stream 72 rows`, `0 mismatch` in both. The loop's contracts survived the refactor with it: +`last_hidden_rows_ = norm_all` and `last_hidden_ = fnorm` (`q4nx_forward.cpp:3558`, `:3569`) and +`commit_verify_rows` truncating to the accepted prefix (`:2348`). + **Leaf fill is resolved, and the answer is that this verifier cannot use it.** The plan's last protocol item (#805) asks for "spare verify rows filled with the drafts' 2nd/3rd candidates". The acceptance rule here is `argmax(logits[i-1]) == draft[i]` (`npu/mtp/draft.h:317`): a draft is accepted iff it **is** the base's own argmax at