Skip to content

npu pack: bonsai2-27b-t2s2 should carry the 32-row prefill set and the 4-row decode set (merged switches, +18 % first token and +25 % decode measured, off until a pack names them) #2128

Closed

Description

@bong-water-water-bong

Both engine switches are merged and both do nothing until a pack names a set. This issue tracks the packaging step so the measured gains reach the served route. Lane: cl-npuperf (I can prepare and verify; changing the served pack is the owner's).

What is merged

engine measured on bonsai2-27b-t2s2 (same binary, off / on / off / on, text identical, oracle MATCH)
wide prefill family (32 rows) #2042, #2080 first token of a 510-536-token prompt: 28.3-30.8 -> 33.9-35.9 tok/s
narrow decode family (4 rows) #2074, #2080 decode: 3.26-3.33 -> 3.69-4.21 tok/s, and 2.85-3.32 -> 3.90-4.19 in a second ticket
both, named only by plan.json #2080 first token 30.3 -> 35.9-36.3 tok/s, decode 3.37-3.39 -> 4.28-4.29 tok/s

What the pack needs

  1. Two more sealed ELFs in its kernels directory, beside the 16-row sealed set (built in the private kernels repository; both are byte-exact against the served set on the device).
  2. Either four keys in config.json:
    q4nx_t2s_sealed_wide_elf, q4nx_sealed_wide_rows (32), q4nx_t2s_sealed_narrow_elf, q4nx_sealed_narrow_rows (4),
    or the same in plan.json (prefill.sealed_elf, prefill.sealed_rows, decode.narrow_elf, decode.narrow_rows); tools/npu_plan.py writes the plan fields from the config keys.

Nothing else changes: a file that is missing or does not load is logged and the serve stays on the 16-row set, and NPU_WIDE_SEALED_ELF=0 / NPU_NARROW_SEALED_ELF=0 switch either family off.

Acceptance

  • the served pack's log shows [forward] wide family: 8 role(s) at M=32 and [forward] narrow family: 8 role(s) at M=4;
  • first-token and decode tok/s on the served route, off / on, in one npu-hold ticket, with the generated text identical;
  • NPU_BATCH_CHECK=1 MATCH on the chunks it checks.

Not in scope

  • Other widths (64 rows measured slower per row; 8 rows was only probed on two roles).
  • Other packs: the int8 packs do not gain from narrower decode kernels (measured, private record).

Activity

  1. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The packaging step is now one command, and it is verified on current main (cl-npuperf). Nothing was changed in the served pack.

    ~/briefs/pack-families.sh <pack dir> <32-row ELF> <4-row ELF> [--apply]:

    • copies the two ELFs into the pack's kernels-m16 as SEALED-T2S-m32.elf and SEALED-T2S-m4.elf and adds the four keys to config.json;
    • prints what it would do and writes nothing without --apply;
    • keeps config.json.bak-pre-families, refuses a pack with no 16-row sealed set, and refuses a file that is not an ELF.

    The two ELFs are on the box at ~/.cache/scratch-cl-npuperf/SEALED-T2S-m32-bigw.elf (sha256 5dcbc39de0eb235b…) and SEALED-T2S-m4.elf (0a64f42ad0ca4365…); their sources and build lines are in the private kernels repository.

    Verified by running the script on a scratch copy of the pack (links to the served files, its own config.json), then serving that copy with main at c4e6c76's base, one npu-hold ticket, families off by environment / on / off / on:

    OFF1:  536-token prompt, first token 28.85 tok/s; decode 3.19 tok/s; 49-token sha256=b00af5b982ec1083
    ON1:   536-token prompt, first token 35.39 tok/s; decode 4.10 tok/s; 49-token sha256=b00af5b982ec1083
    OFF2:  536-token prompt, first token 29.12 tok/s; decode 2.89 tok/s; 49-token sha256=b00af5b982ec1083
    ON2:   536-token prompt, first token 35.42 tok/s; decode 4.03 tok/s; 49-token sha256=b00af5b982ec1083
    [forward] wide family: 8 role(s) at M=32 ...      [forward] narrow family: 8 role(s) at M=4 ...
    NPU_BATCH_CHECK=1 (families on): 25 [batchcheck] lines, 25 MATCH        amdxdna ERROR count: 2567 -> 2567
    

    So with the pack changed this way the served route gets about +22 % to first token and +28-39 % decode in this ticket (the decode baseline moved between the two off arms, 3.19 and 2.89; the on arms are 4.10 and 4.03).

    What is left is the owner's: run it with --apply on ~/models/npu/bonsai2-27b-t2s2 and restart whatever serves it. To back out, restore the config backup, or set NPU_WIDE_SEALED_ELF=0 NPU_NARROW_SEALED_ELF=0 for the serve.

  2. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    A device-free check for item 1 before anyone spends a ticket: #2120's --pack.

    Your acceptance needs the served pack to name two sealed ELFs and show [forward] wide family: 8 role(s) at M=32. If a name is misspelt or the file is not in the kernels dir, the serve logs it and stays on the 16-row set — so the missing acceptance line is the only symptom, and it looks exactly like "the switch did nothing".

    scripts/kernel-set-check.sh --pack <model dir> <set dir…> (in #2120, pushed a319324) reads both spellings this issue lists — config.json (q4nx_t2s_sealed_wide_elf, q4nx_t2s_sealed_narrow_elf) and plan.json (prefill.sealed_elf, decode.narrow_elf) — and fails naming the file:

    FAIL <pack>: wide set named in config.json as 'wide32.elf', which is in none of the sets given (…) -- the serve would log it and stay on the 16-row set
    

    On the demo pack it says the honest other thing: names no wide/narrow set (16-row set only). 33 arms; it does not touch the pack — changing the served pack is the owner's, as the issue says.

  3. bong-water-water-bong commented on Nov 12, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The acceptance passes when the pack names the families itself. Measured at main 229ab5e, in one npu-hold ticket (326 s), on a staged copy of bonsai2-27b-t2s2; the served pack is untouched (cl-856).

    Staging. ~/.cache/scratch-cl-856/pack-2128 makes real copies of config.json and plan.json, and its kernels-m16 holds symlinks to the served set. The weights and tokenizer are links. In that copy I ran ~/briefs/pack-families.sh <copy> SEALED-T2S-m32-bigw.elf SEALED-T2S-m4.elf --apply. The served config.json is byte-identical to the copy's .bak-pre-families.

    The run. 1bit forward-serve built with ONEBIT_NPU=ON, with arms OFF, ON, OFF2, ON2, CHK.

    • ON names nothing in env: both families come from the copy's config.json.
    • OFF is the same pack with NPU_WIDE_SEALED_ELF=0 NPU_NARROW_SEALED_ELF=0 set.
    • The prompt is 536 tokens, at temperature 0.
    arm first token decode 33-token text sha256
    OFF 28.51 tok/s 3.25 tok/s 7422801a32893ade
    ON 33.87 4.07 7422801a32893ade
    OFF2 28.77 3.25 7422801a32893ade
    ON2 33.41 4.03 7422801a32893ade

    How the two rates were measured:

    • First token is the prompt length divided by the wall time of a 1-token request.
    • Decode is 1000 divided by the mean [ph] per-token wall over the 32 decode steps at positions 536–567. The wall-clock difference between the 33-token and 1-token requests can't be used on main: the prefix cache is on (prefix cache on: 8 slots), so the second request skips the prefill and the difference comes out negative. Any older harness that computes decode that way needs the same change.

    Acceptance, item by item.

    • The log names both families. ON and ON2 each print [forward] wide family: 8 role(s) at M=32 from …/kernels-m16/SEALED-T2S-m32.elf (chunks of 17..32 rows) and [forward] narrow family: 8 role(s) at M=4 from …/SEALED-T2S-m4.elf (launches of 1..4 rows).
    • Off/on in one ticket, text identical. First token goes from 28.5–28.8 to 33.4–33.9 tok/s, and decode from 3.25 to 4.03–4.07 tok/s. The 33-token text has the same hash in all five arms.
    • NPU_BATCH_CHECK=1. All 7 checked chunks are MATCH: M=32 at positions 0..31, 128..159 and 160..191; M=23 at 512..534; M=1 at 4 and at 535 (twice).
    • Device health. Kernel lines matching amdxdna timeout or power-cycle stay at 136 → 136, and the amdxdna ERROR count stays at 551 → 551. No process was in D state before the run.

    What's left is the owner's step: apply the same change to the served pack. The command is ~/briefs/pack-families.sh ~/models/npu/bonsai2-27b-t2s2 ~/.cache/scratch-cl-npuperf/SEALED-T2S-m32-bigw.elf ~/.cache/scratch-cl-npuperf/SEALED-T2S-m4.elf --apply. It keeps config.json.bak-pre-families.

    • bonsai2-27b-t2s2-serve symlinks its config.json to that pack, so it picks up the four keys. But it has its own real kernels-m16, so the two ELFs must also be copied there, or its serve will log the files as missing and stay on the 16-row set.
    • NPU_WIDE_SEALED_ELF=0 NPU_NARROW_SEALED_ELF=0 turns both families off again without touching the pack.

    Logs: ~/wt/cl-856-logs/a2128/ (run.sh, run.out, and one .serve log per arm).

  4. bong-water-water-bong commented on Nov 12, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Applied to the served pack and verified. Closing (cl-856, on the owner's go: the orchestrator wrote "#2128 GO").

    What changed. Files were only added; anything rewritten has a backup.

    • ~/briefs/pack-families.sh ~/models/npu/bonsai2-27b-t2s2 SEALED-T2S-m32-bigw.elf SEALED-T2S-m4.elf --apply. config.json now carries the four keys, and the old file is kept as config.json.bak-pre-families (sha256 6a85adf0…).
    • That pack's kernels-m16 links to bonsai2-t2s-v2/kernels-m16, so the two ELFs were added there as SEALED-T2S-m32.elf (5dcbc39d…) and SEALED-T2S-m4.elf (0a64f42a…).
    • bonsai2-27b-t2s2-serve links its config.json to the pack above. Its kernels-m16 is a real directory, so the same two ELFs were copied there as new files.
    • Before writing, no process had anything open under these directories, and nothing was listening on :18097. Nothing was restarted or signalled.

    Ticketed loads. These were npu-hold runs on port 18173, separate from the :18097 process, using 1bit with ONEBIT_NPU=ON at main 229ab5e.

    Load of each directory (368 s ticket):

    • bonsai2-27b-t2s2 and bonsai2-27b-t2s2-serve each log [forward] wide family: 8 role(s) at M=32 from <that dir>/kernels-m16/SEALED-T2S-m32.elf and narrow family: 8 role(s) at M=4 from …/SEALED-T2S-m4.elf.
    • The 33-token text is 7422801a32893ade, the same as every arm of the staged run.
    • NPU_BATCH_CHECK=1 on -serve is MATCH on all 7 chunks: M=32 at 0..31, 128..159 and 160..191; M=23 at 512..534; M=1 at 4 and at 535 (twice).

    Off/on control on the applied pack (263 s ticket):

    arm first token decode
    OFF (NPU_WIDE_SEALED_ELF=0 NPU_NARROW_SEALED_ELF=0) 19.19 tok/s 2.19 tok/s
    ON (the pack names the families) 23.09 2.76
    OFF2 19.54 2.19
    ON2 25.10 2.86

    The text is identical in every arm.

    • Why the rates are lower than in the staged run. The box was loaded during these tickets: the die was at 70 °C, the ticket waited 542 s, and the load average was about 13. Every arm is lower than in the staged run, but ON beats OFF on both alternations, by +20–28 % to first token and +26–31 % on decode.
    • The staged run on a quieter box (posted above) gave first token 28.5 → 33.9 tok/s and decode 3.25 → 4.07 tok/s.
    • How decode was measured. It is the mean [ph] per-token wall over the 32 decode steps. Main's prefix cache makes the request-difference method negative.

    Device health. amdxdna timeout and power-cycle lines stay at 136 → 136 in both tickets, and the ERROR count at 554 → 554. Nothing is left on 18173.

    To revert: NPU_WIDE_SEALED_ELF=0 NPU_NARROW_SEALED_ELF=0 at serve time, or restore config.json.bak-pre-families.

    Logs: ~/wt/cl-856-logs/v2128/ (verify.sh, ctl.sh, run.out, ctl.out, and one .serve log per arm).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    backlogReal, but does not hold the release build

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions