From cc42e81467c3147ba594f915731a5f6e261e1560 Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Fri, 6 Nov 2026 16:00:31 -0400 Subject: [PATCH] npu: the t2s leaderboard rows say one run, and what the pair did not test Two corrections to the rows merged in #1276, both raised in review: - 'repeats': 7 read as seven repeats of the measurement, but the 7 is the number of steady-state tokens inside ONE request whose median is the recorded value. repeats is now 1 and the instrument says 'one run per arm ... median of its 7 steady-state tokens', which is what was done. (Same mismatch pi-prs flagged on #1206's pooled MTP rows.) - the t2s2 row's notes claimed 'greedy text identical on the three-prompt window'. That window was never run for this pair: what was measured is role-level byte equality with the pb16i path (6/6 roles at layer 0, 8/8 at layer 3, three seeds) plus text equality on the one prompt both arms were asked. The 2026-11-06 re-take that would have closed it was blocked by an unrelated serve holding the engine's device lock, so the notes now name that as the open acceptance step instead of implying it was done. Numbers unchanged: 313 vs 410 ms/token, 1.31x, 1.42x on the device wait, 2.12x on bytes. docs/npu-leaderboard.md regenerated (21 records valid). --- bench/npu-leaderboard/bonsai2-27b-pb16i-20261102.json | 4 ++-- bench/npu-leaderboard/bonsai2-27b-t2s2-20261102.json | 6 +++--- docs/npu-leaderboard.md | 6 +++--- 3 files changed, 8 insertions(+), 8 deletions(-) diff --git a/bench/npu-leaderboard/bonsai2-27b-pb16i-20261102.json b/bench/npu-leaderboard/bonsai2-27b-pb16i-20261102.json index 9ef228af2..89172ff29 100644 --- a/bench/npu-leaderboard/bonsai2-27b-pb16i-20261102.json +++ b/bench/npu-leaderboard/bonsai2-27b-pb16i-20261102.json @@ -17,8 +17,8 @@ "stream_per_token_gb": 12.93, "tokens": 7 }, - "instrument": "NPU_PHASE_PROF=1, the serve's own [ph] tok wall= line, median of the 7 steady-state tokens (first token dropped); request walls are not this number, they pay the per-serve image fill once", - "repeats": 7, + "instrument": "NPU_PHASE_PROF=1, one run per arm: the serve's own [ph] tok wall= line, median of its 7 steady-state tokens (first token dropped); request walls are not this number, they pay the per-serve image fill once", + "repeats": 1, "timestamp": "2026-11-02T23:50:00Z", "source": "~/briefs/TBONSAI-SPEED-STATE.md, 2026-11-02 20:05 AST entry: three npu-hold tickets (19:47, 19:48, 19:49 AST) on :18099, amdxdna counter 0 in each, spread 409-412 ms/token", "notes": "control arm of the P3 (#802) pair: the same container file, the same binary and the same box one minute from bonsai2-27b-t2s2-20261102.json, so the ratio is a weight-format difference and nothing else. This is the 4.5 bpw path the 2-bit stream replaced; it is not a candidate, it is the denominator. What would make this wrong: measuring it in a different window than the t2s arm (the pair is only valid adjacent)." diff --git a/bench/npu-leaderboard/bonsai2-27b-t2s2-20261102.json b/bench/npu-leaderboard/bonsai2-27b-t2s2-20261102.json index 65e15b552..5a3b73cea 100644 --- a/bench/npu-leaderboard/bonsai2-27b-t2s2-20261102.json +++ b/bench/npu-leaderboard/bonsai2-27b-t2s2-20261102.json @@ -17,9 +17,9 @@ "stream_per_token_gb": 6.11, "tokens": 7 }, - "instrument": "NPU_PHASE_PROF=1, the serve's own [ph] tok wall= line, median of the 7 steady-state tokens (first token dropped); request walls are not this number, they pay the per-serve image fill once", - "repeats": 7, + "instrument": "NPU_PHASE_PROF=1, one run per arm: the serve's own [ph] tok wall= line, median of its 7 steady-state tokens (first token dropped); request walls are not this number, they pay the per-serve image fill once", + "repeats": 1, "timestamp": "2026-11-02T23:50:00Z", "source": "~/briefs/TBONSAI-SPEED-STATE.md, 2026-11-02 20:05 AST entry: three npu-hold tickets (19:47, 19:48, 19:49 AST) on :18099, amdxdna counter 0 in each, spread 311-317 ms/token", - "notes": "P3 (#802) 'after' arm against bonsai2-27b-pb16i-20261102.json: 1.31x per token and 1.42x on the device wait (209 vs 296 ms). Accept: every decode role byte-identical to the pb16i path (6/6 at layer 0, 8/8 at layer 3, seeds 1/2/3) and greedy text identical on the three-prompt window. Stream is 2.12x fewer bytes (431 images, 14.44 -> 6.82 GB). What would make this wrong: quoting a request wall instead of [ph] tok (a fresh serve adds the ~10.6 s weight-image fill), or comparing across boxes - the pb16i arm is the only valid control." + "notes": "P3 (#802) 'after' arm against bonsai2-27b-pb16i-20261102.json: 1.31x per token and 1.42x on the device wait (209 vs 296 ms). Accept is byte-exactness, not a text diff: every decode role matched the pb16i path byte for byte at layer 0 (6/6) and layer 3 (8/8) on three seeds, and greedy text is equal to the pb16i arm's on the prompt both were asked (' Paris.\\nThe capital of Germany is'). A three-prompt full-model text window on this pair was NOT run - the 2026-11-06 re-take was blocked by an unrelated serve holding the engine's device lock - so that remains the open acceptance step. Stream is 2.12x fewer bytes (431 images, 14.44 -> 6.82 GB). What would make this wrong: quoting a request wall instead of [ph] tok (a fresh serve adds the ~10.6 s weight-image fill), or comparing across boxes - the pb16i arm is the only valid control." } diff --git a/docs/npu-leaderboard.md b/docs/npu-leaderboard.md index 631aa22a5..92f9a254f 100644 --- a/docs/npu-leaderboard.md +++ b/docs/npu-leaderboard.md @@ -24,8 +24,8 @@ One row is one measured run of this engine. A record is only ranked if it names | model | route | quant | metric | value | settings | repeats | instrument | | --- | --- | --- | --- | --- | --- | --- | --- | -| Bonsai-2-27B | npu | PTQ1_0 ternary, t2s 2-bit stream (2.125 bpw) | decode | 313.0 ms/token | device=accel0, port=18099, container=~/models/npu/bonsai2-27b-t2s2 (q4nx_weight_layout=t2s, version 6, sealed SEALED-T2S.elf, M=16), binary=~/wt/bonsai-speed-build2/1bit (2026-11-02, engine 916e90d + the Hadamard SIMD host path), stream_per_token_gb=6.11, tokens=7 | 7 | NPU_PHASE_PROF=1, the serve's own [ph] tok wall= line, median of the 7 steady-state tokens (first token dropped); request walls are not this number, they pay the per-serve image fill once | -| Bonsai-2-27B | npu | pb16i (int4 codes + int16 magnitudes, 4.5 bpw) | decode | 410.0 ms/token | device=accel0, port=18099, container=~/models/npu/bonsai2-27b-pb16i (q4nx_weight_layout=pb16i, version 4, M=16), binary=~/wt/bonsai-speed-build2/1bit (2026-11-02, engine 916e90d + the Hadamard SIMD host path), stream_per_token_gb=12.93, tokens=7 | 7 | NPU_PHASE_PROF=1, the serve's own [ph] tok wall= line, median of the 7 steady-state tokens (first token dropped); request walls are not this number, they pay the per-serve image fill once | +| Bonsai-2-27B | npu | PTQ1_0 ternary, t2s 2-bit stream (2.125 bpw) | decode | 313.0 ms/token | device=accel0, port=18099, container=~/models/npu/bonsai2-27b-t2s2 (q4nx_weight_layout=t2s, version 6, sealed SEALED-T2S.elf, M=16), binary=~/wt/bonsai-speed-build2/1bit (2026-11-02, engine 916e90d + the Hadamard SIMD host path), stream_per_token_gb=6.11, tokens=7 | 1 | NPU_PHASE_PROF=1, one run per arm: the serve's own [ph] tok wall= line, median of its 7 steady-state tokens (first token dropped); request walls are not this number, they pay the per-serve image fill once | +| Bonsai-2-27B | npu | pb16i (int4 codes + int16 magnitudes, 4.5 bpw) | decode | 410.0 ms/token | device=accel0, port=18099, container=~/models/npu/bonsai2-27b-pb16i (q4nx_weight_layout=pb16i, version 4, M=16), binary=~/wt/bonsai-speed-build2/1bit (2026-11-02, engine 916e90d + the Hadamard SIMD host path), stream_per_token_gb=12.93, tokens=7 | 1 | NPU_PHASE_PROF=1, one run per arm: the serve's own [ph] tok wall= line, median of its 7 steady-state tokens (first token dropped); request walls are not this number, they pay the per-serve image fill once | | Flash-Next (IQ4_XS 4.25 bpw) | hrx | IQ4_XS | prefill pp512 | 286.63 tok/s | ngl=99, fa=1, ub=512, engine=0d0c846 | 3 | llama-bench -p 512, 3 repeats, s.d. 0.94 tok/s | | Flash-Next (IQ4_XS 4.25 bpw) | hrx | IQ4_XS | prefill p1024 ub576 | 269.34 tok/s | ngl=99, fa=1, ub=576, engine=0d0c846 | 1 | llama-bench -p 1024, 1 repeat | | Flash-Next (IQ4_XS 4.25 bpw) | hrx | IQ4_XS | prefill pp4096 | 260.59 tok/s | ngl=99, fa=1, ub=512, engine=0d0c846 | 3 | llama-bench -p 4096, 3 repeats, s.d. 4.51 tok/s | @@ -46,7 +46,7 @@ One row is one measured run of this engine. A record is only ranked if it names | Qwen3.8-27B-NPU2 | npu | Q4NX (int8 GEMM) | first token (weight pack included) | 145.7 s | device=accel0, commits=800cf6f, threads=32 | 1 | request wall and driver trace span | | Qwen3.8-27B-NPU2 | npu | Q4NX (int8 GEMM) | first token (weight pack included) | 435.9 s | device=accel0, commits=pre-#622, threads=32 | 1 | driver trace span, gated on ~/npu.lock | -- `bonsai2-27b-t2s2-20261102` — 2026-11-02T23:50:00Z; source: ~/briefs/TBONSAI-SPEED-STATE.md, 2026-11-02 20:05 AST entry: three npu-hold tickets (19:47, 19:48, 19:49 AST) on :18099, amdxdna counter 0 in each, spread 311-317 ms/token; P3 (#802) 'after' arm against bonsai2-27b-pb16i-20261102.json: 1.31x per token and 1.42x on the device wait (209 vs 296 ms). Accept: every decode role byte-identical to the pb16i path (6/6 at layer 0, 8/8 at layer 3, seeds 1/2/3) and greedy text identical on the three-prompt window. Stream is 2.12x fewer bytes (431 images, 14.44 -> 6.82 GB). What would make this wrong: quoting a request wall instead of [ph] tok (a fresh serve adds the ~10.6 s weight-image fill), or comparing across boxes - the pb16i arm is the only valid control. +- `bonsai2-27b-t2s2-20261102` — 2026-11-02T23:50:00Z; source: ~/briefs/TBONSAI-SPEED-STATE.md, 2026-11-02 20:05 AST entry: three npu-hold tickets (19:47, 19:48, 19:49 AST) on :18099, amdxdna counter 0 in each, spread 311-317 ms/token; P3 (#802) 'after' arm against bonsai2-27b-pb16i-20261102.json: 1.31x per token and 1.42x on the device wait (209 vs 296 ms). Accept is byte-exactness, not a text diff: every decode role matched the pb16i path byte for byte at layer 0 (6/6) and layer 3 (8/8) on three seeds, and greedy text is equal to the pb16i arm's on the prompt both were asked (' Paris.\nThe capital of Germany is'). A three-prompt full-model text window on this pair was NOT run - the 2026-11-06 re-take was blocked by an unrelated serve holding the engine's device lock - so that remains the open acceptance step. Stream is 2.12x fewer bytes (431 images, 14.44 -> 6.82 GB). What would make this wrong: quoting a request wall instead of [ph] tok (a fresh serve adds the ~10.6 s weight-image fill), or comparing across boxes - the pb16i arm is the only valid control. - `bonsai2-27b-pb16i-20261102` — 2026-11-02T23:50:00Z; source: ~/briefs/TBONSAI-SPEED-STATE.md, 2026-11-02 20:05 AST entry: three npu-hold tickets (19:47, 19:48, 19:49 AST) on :18099, amdxdna counter 0 in each, spread 409-412 ms/token; control arm of the P3 (#802) pair: the same container file, the same binary and the same box one minute from bonsai2-27b-t2s2-20261102.json, so the ratio is a weight-format difference and nothing else. This is the 4.5 bpw path the 2-bit stream replaced; it is not a candidate, it is the denominator. What would make this wrong: measuring it in a different window than the t2s arm (the pair is only valid adjacent). - `flashnext-pp512-20261027` — 2026-10-27T02:38:52Z; source: ~/lb/572/run-20261027T015408Z/a1-timing.out (strixhalo); gufo reference on the same box is 1679 tok/s; the analysis is in docs/hrx.md - `flashnext-p1024-ub576-20261027` — 2026-10-27T02:50:00Z; source: ~/lb/572/run-20261027T015408Z/a4-ub576.out (strixhalo); shows the ubatch bound above 512 no longer refuses the run; single repeat, do not read as a tuned number