You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
npu: decode on the served 27B route is 70 % device wait (211 of 302 ms per token), and every launch runs a 16-row kernel for one row #2073
A decode step is 416 launches, and device wait is 211 of 302 ms (70 %). Host work is 42 ms. So decode on this route is bound by the launches, unlike prefill (#1990), where the host side was the larger half.
Every one of those launches runs a kernel built for 16 rows and carries one row. The device time of a launch does not depend on how many of its rows are filled, so a one-row launch pays for 16.
Proposal
The mirror of #2042's wide family: a sealed set built for fewer rows, taken by launches of 1..N rows, with the 16-row set kept for everything else. Same rules as the wide family: all roles or none, one hw_context, named by the pack or the environment, a file that does not load is logged and decode stays as it is.
Acceptance
decode tok/s on the served route, arms 16-row / narrow / 16-row / narrow in one ticket, text identical;
the sequential oracle (NPU_BATCH_CHECK=1) matches on the short chunks that take the narrow kernel;
the selection rules pinned by a host test.
Lane: cl-npuperf. A first measurement and PR follow in a comment.
First measurement and PR: #2074 (draft, stacked on #2042). With a 4-row sealed set, decode on the served route goes 3.26-3.33 -> 3.69-4.21 tok/s in one ticket and 2.85-3.32 -> 3.90-4.19 in a second, device wait per step 210 -> 151-158 ms, generated text identical, oracle MATCH. What remains after it: a plan.json field for the narrow set, and shipping the ELF in the served pack.
Activity
bong-water-water-bong commentedon Nov 10, 2026
First measurement and PR: #2074 (draft, stacked on #2042). With a 4-row sealed set, decode on the served route goes 3.26-3.33 -> 3.69-4.21 tok/s in one ticket and 2.85-3.32 -> 3.90-4.19 in a second, device wait per step 210 -> 151-158 ms, generated text identical, oracle MATCH. What remains after it: a plan.json field for the narrow set, and shipping the ELF in the served pack.
npu-plan: plan.json can name the wide and narrow sealed families (pre…
npu-plan: plan.json can name the wide and narrow sealed families (pre…
npu-plan: plan.json can name the wide and narrow sealed families (pre…