Skip to content

npu: decode on the served 27B route is 70 % device wait (211 of 302 ms per token), and every launch runs a 16-row kernel for one row #2073

Closed

Description

@bong-water-water-bong

Measured on this box, bonsai2-27b-t2s2 (the served 27B route), one npu-hold ticket, NPU_PHASE_PROF=1:

[bs] M= 1 ... wall=  301.6 | ... (gemm pack=   4.2 ql= 29.8 wait=  211.1 ... host=   41.5 n=416)     (medians of 6 steps)

A decode step is 416 launches, and device wait is 211 of 302 ms (70 %). Host work is 42 ms. So decode on this route is bound by the launches, unlike prefill (#1990), where the host side was the larger half.

Every one of those launches runs a kernel built for 16 rows and carries one row. The device time of a launch does not depend on how many of its rows are filled, so a one-row launch pays for 16.

Proposal

The mirror of #2042's wide family: a sealed set built for fewer rows, taken by launches of 1..N rows, with the 16-row set kept for everything else. Same rules as the wide family: all roles or none, one hw_context, named by the pack or the environment, a file that does not load is logged and decode stays as it is.

Acceptance

  • decode tok/s on the served route, arms 16-row / narrow / 16-row / narrow in one ticket, text identical;
  • the sequential oracle (NPU_BATCH_CHECK=1) matches on the short chunks that take the narrow kernel;
  • the selection rules pinned by a host test.

Lane: cl-npuperf. A first measurement and PR follow in a comment.

Activity

  1. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    First measurement and PR: #2074 (draft, stacked on #2042). With a 4-row sealed set, decode on the served route goes 3.26-3.33 -> 3.69-4.21 tok/s in one ticket and 2.85-3.32 -> 3.90-4.19 in a second, device wait per step 210 -> 151-158 ms, generated text identical, oracle MATCH. What remains after it: a plan.json field for the narrow set, and shipping the ELF in the served pack.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions