You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
npu pack: bonsai2-27b-t2s2 should carry the 32-row prefill set and the 4-row decode set (merged switches, +18 % first token and +25 % decode measured, off until a pack names them) #2128
Both engine switches are merged and both do nothing until a pack names a set. This issue tracks the packaging step so the measured gains reach the served route. Lane: cl-npuperf (I can prepare and verify; changing the served pack is the owner's).
What is merged
engine
measured on bonsai2-27b-t2s2 (same binary, off / on / off / on, text identical, oracle MATCH)
Two more sealed ELFs in its kernels directory, beside the 16-row sealed set (built in the private kernels repository; both are byte-exact against the served set on the device).
Either four keys in config.json: q4nx_t2s_sealed_wide_elf, q4nx_sealed_wide_rows (32), q4nx_t2s_sealed_narrow_elf, q4nx_sealed_narrow_rows (4),
or the same in plan.json (prefill.sealed_elf, prefill.sealed_rows, decode.narrow_elf, decode.narrow_rows); tools/npu_plan.py writes the plan fields from the config keys.
Nothing else changes: a file that is missing or does not load is logged and the serve stays on the 16-row set, and NPU_WIDE_SEALED_ELF=0 / NPU_NARROW_SEALED_ELF=0 switch either family off.
Acceptance
the served pack's log shows [forward] wide family: 8 role(s) at M=32 and [forward] narrow family: 8 role(s) at M=4;
first-token and decode tok/s on the served route, off / on, in one npu-hold ticket, with the generated text identical;
NPU_BATCH_CHECK=1 MATCH on the chunks it checks.
Not in scope
Other widths (64 rows measured slower per row; 8 rows was only probed on two roles).
Other packs: the int8 packs do not gain from narrower decode kernels (measured, private record).
copies the two ELFs into the pack's kernels-m16 as SEALED-T2S-m32.elf and SEALED-T2S-m4.elf and adds the four keys to config.json;
prints what it would do and writes nothing without --apply;
keeps config.json.bak-pre-families, refuses a pack with no 16-row sealed set, and refuses a file that is not an ELF.
The two ELFs are on the box at ~/.cache/scratch-cl-npuperf/SEALED-T2S-m32-bigw.elf (sha256 5dcbc39de0eb235b…) and SEALED-T2S-m4.elf (0a64f42ad0ca4365…); their sources and build lines are in the private kernels repository.
Verified by running the script on a scratch copy of the pack (links to the served files, its own config.json), then serving that copy with main at c4e6c76's base, one npu-hold ticket, families off by environment / on / off / on:
So with the pack changed this way the served route gets about +22 % to first token and +28-39 % decode in this ticket (the decode baseline moved between the two off arms, 3.19 and 2.89; the on arms are 4.10 and 4.03).
What is left is the owner's: run it with --apply on ~/models/npu/bonsai2-27b-t2s2 and restart whatever serves it. To back out, restore the config backup, or set NPU_WIDE_SEALED_ELF=0 NPU_NARROW_SEALED_ELF=0 for the serve.
A device-free check for item 1 before anyone spends a ticket: #2120's --pack.
Your acceptance needs the served pack to name two sealed ELFs and show [forward] wide family: 8 role(s) at M=32. If a name is misspelt or the file is not in the kernels dir, the serve logs it and stays on the 16-row set — so the missing acceptance line is the only symptom, and it looks exactly like "the switch did nothing".
scripts/kernel-set-check.sh --pack <model dir> <set dir…> (in #2120, pushed a319324) reads both spellings this issue lists — config.json (q4nx_t2s_sealed_wide_elf, q4nx_t2s_sealed_narrow_elf) and plan.json (prefill.sealed_elf, decode.narrow_elf) — and fails naming the file:
FAIL <pack>: wide set named in config.json as 'wide32.elf', which is in none of the sets given (…) -- the serve would log it and stay on the 16-row set
On the demo pack it says the honest other thing: names no wide/narrow set (16-row set only). 33 arms; it does not touch the pack — changing the served pack is the owner's, as the issue says.
The acceptance passes when the pack names the families itself. Measured at main 229ab5e, in one npu-hold ticket (326 s), on a staged copy of bonsai2-27b-t2s2; the served pack is untouched (cl-856).
Staging.~/.cache/scratch-cl-856/pack-2128 makes real copies of config.json and plan.json, and its kernels-m16 holds symlinks to the served set. The weights and tokenizer are links. In that copy I ran ~/briefs/pack-families.sh <copy> SEALED-T2S-m32-bigw.elf SEALED-T2S-m4.elf --apply. The served config.json is byte-identical to the copy's .bak-pre-families.
The run.1bit forward-serve built with ONEBIT_NPU=ON, with arms OFF, ON, OFF2, ON2, CHK.
ON names nothing in env: both families come from the copy's config.json.
OFF is the same pack with NPU_WIDE_SEALED_ELF=0 NPU_NARROW_SEALED_ELF=0 set.
The prompt is 536 tokens, at temperature 0.
arm
first token
decode
33-token text sha256
OFF
28.51 tok/s
3.25 tok/s
7422801a32893ade
ON
33.87
4.07
7422801a32893ade
OFF2
28.77
3.25
7422801a32893ade
ON2
33.41
4.03
7422801a32893ade
How the two rates were measured:
First token is the prompt length divided by the wall time of a 1-token request.
Decode is 1000 divided by the mean [ph] per-token wall over the 32 decode steps at positions 536–567. The wall-clock difference between the 33-token and 1-token requests can't be used on main: the prefix cache is on (prefix cache on: 8 slots), so the second request skips the prefill and the difference comes out negative. Any older harness that computes decode that way needs the same change.
Acceptance, item by item.
The log names both families. ON and ON2 each print [forward] wide family: 8 role(s) at M=32 from …/kernels-m16/SEALED-T2S-m32.elf (chunks of 17..32 rows) and [forward] narrow family: 8 role(s) at M=4 from …/SEALED-T2S-m4.elf (launches of 1..4 rows).
Off/on in one ticket, text identical. First token goes from 28.5–28.8 to 33.4–33.9 tok/s, and decode from 3.25 to 4.03–4.07 tok/s. The 33-token text has the same hash in all five arms.
NPU_BATCH_CHECK=1. All 7 checked chunks are MATCH: M=32 at positions 0..31, 128..159 and 160..191; M=23 at 512..534; M=1 at 4 and at 535 (twice).
Device health. Kernel lines matching amdxdna timeout or power-cycle stay at 136 → 136, and the amdxdna ERROR count stays at 551 → 551. No process was in D state before the run.
What's left is the owner's step: apply the same change to the served pack. The command is ~/briefs/pack-families.sh ~/models/npu/bonsai2-27b-t2s2 ~/.cache/scratch-cl-npuperf/SEALED-T2S-m32-bigw.elf ~/.cache/scratch-cl-npuperf/SEALED-T2S-m4.elf --apply. It keeps config.json.bak-pre-families.
bonsai2-27b-t2s2-serve symlinks its config.json to that pack, so it picks up the four keys. But it has its own real kernels-m16, so the two ELFs must also be copied there, or its serve will log the files as missing and stay on the 16-row set.
NPU_WIDE_SEALED_ELF=0 NPU_NARROW_SEALED_ELF=0 turns both families off again without touching the pack.
Logs: ~/wt/cl-856-logs/a2128/ (run.sh, run.out, and one .serve log per arm).
Applied to the served pack and verified. Closing (cl-856, on the owner's go: the orchestrator wrote "#2128 GO").
What changed. Files were only added; anything rewritten has a backup.
~/briefs/pack-families.sh ~/models/npu/bonsai2-27b-t2s2 SEALED-T2S-m32-bigw.elf SEALED-T2S-m4.elf --apply. config.json now carries the four keys, and the old file is kept as config.json.bak-pre-families (sha256 6a85adf0…).
That pack's kernels-m16 links to bonsai2-t2s-v2/kernels-m16, so the two ELFs were added there as SEALED-T2S-m32.elf (5dcbc39d…) and SEALED-T2S-m4.elf (0a64f42a…).
bonsai2-27b-t2s2-serve links its config.json to the pack above. Its kernels-m16 is a real directory, so the same two ELFs were copied there as new files.
Before writing, no process had anything open under these directories, and nothing was listening on :18097. Nothing was restarted or signalled.
Ticketed loads. These were npu-hold runs on port 18173, separate from the :18097 process, using 1bit with ONEBIT_NPU=ON at main 229ab5e.
Load of each directory (368 s ticket):
bonsai2-27b-t2s2 and bonsai2-27b-t2s2-serve each log [forward] wide family: 8 role(s) at M=32 from <that dir>/kernels-m16/SEALED-T2S-m32.elf and narrow family: 8 role(s) at M=4 from …/SEALED-T2S-m4.elf.
The 33-token text is 7422801a32893ade, the same as every arm of the staged run.
NPU_BATCH_CHECK=1 on -serve is MATCH on all 7 chunks: M=32 at 0..31, 128..159 and 160..191; M=23 at 512..534; M=1 at 4 and at 535 (twice).
Off/on control on the applied pack (263 s ticket):
arm
first token
decode
OFF (NPU_WIDE_SEALED_ELF=0 NPU_NARROW_SEALED_ELF=0)
19.19 tok/s
2.19 tok/s
ON (the pack names the families)
23.09
2.76
OFF2
19.54
2.19
ON2
25.10
2.86
The text is identical in every arm.
Why the rates are lower than in the staged run. The box was loaded during these tickets: the die was at 70 °C, the ticket waited 542 s, and the load average was about 13. Every arm is lower than in the staged run, but ON beats OFF on both alternations, by +20–28 % to first token and +26–31 % on decode.
The staged run on a quieter box (posted above) gave first token 28.5 → 33.9 tok/s and decode 3.25 → 4.07 tok/s.
How decode was measured. It is the mean [ph] per-token wall over the 32 decode steps. Main's prefix cache makes the request-difference method negative.
Device health. amdxdna timeout and power-cycle lines stay at 136 → 136 in both tickets, and the ERROR count at 554 → 554. Nothing is left on 18173.
To revert:NPU_WIDE_SEALED_ELF=0 NPU_NARROW_SEALED_ELF=0 at serve time, or restore config.json.bak-pre-families.
Logs: ~/wt/cl-856-logs/v2128/ (verify.sh, ctl.sh, run.out, ctl.out, and one .serve log per arm).
Activity
bong-water-water-bong commentedon Nov 11, 2026
The packaging step is now one command, and it is verified on current main (cl-npuperf). Nothing was changed in the served pack.
~/briefs/pack-families.sh <pack dir> <32-row ELF> <4-row ELF> [--apply]:kernels-m16asSEALED-T2S-m32.elfandSEALED-T2S-m4.elfand adds the four keys toconfig.json;--apply;config.json.bak-pre-families, refuses a pack with no 16-row sealed set, and refuses a file that is not an ELF.The two ELFs are on the box at
~/.cache/scratch-cl-npuperf/SEALED-T2S-m32-bigw.elf(sha2565dcbc39de0eb235b…) andSEALED-T2S-m4.elf(0a64f42ad0ca4365…); their sources and build lines are in the private kernels repository.Verified by running the script on a scratch copy of the pack (links to the served files, its own
config.json), then serving that copy with main atc4e6c76's base, onenpu-holdticket, families off by environment / on / off / on:So with the pack changed this way the served route gets about +22 % to first token and +28-39 % decode in this ticket (the decode baseline moved between the two off arms, 3.19 and 2.89; the on arms are 4.10 and 4.03).
What is left is the owner's: run it with
--applyon~/models/npu/bonsai2-27b-t2s2and restart whatever serves it. To back out, restore the config backup, or setNPU_WIDE_SEALED_ELF=0 NPU_NARROW_SEALED_ELF=0for the serve.bong-water-water-bong commentedon Nov 11, 2026
A device-free check for item 1 before anyone spends a ticket: #2120's
--pack.Your acceptance needs the served pack to name two sealed ELFs and show
[forward] wide family: 8 role(s) at M=32. If a name is misspelt or the file is not in the kernels dir, the serve logs it and stays on the 16-row set — so the missing acceptance line is the only symptom, and it looks exactly like "the switch did nothing".scripts/kernel-set-check.sh --pack <model dir> <set dir…>(in #2120, pusheda319324) reads both spellings this issue lists —config.json(q4nx_t2s_sealed_wide_elf,q4nx_t2s_sealed_narrow_elf) andplan.json(prefill.sealed_elf,decode.narrow_elf) — and fails naming the file:On the demo pack it says the honest other thing:
names no wide/narrow set (16-row set only). 33 arms; it does not touch the pack — changing the served pack is the owner's, as the issue says.bong-water-water-bong commentedon Nov 12, 2026
The acceptance passes when the pack names the families itself. Measured at main
229ab5e, in onenpu-holdticket (326 s), on a staged copy ofbonsai2-27b-t2s2; the served pack is untouched (cl-856).Staging.
~/.cache/scratch-cl-856/pack-2128makes real copies ofconfig.jsonandplan.json, and itskernels-m16holds symlinks to the served set. The weights and tokenizer are links. In that copy I ran~/briefs/pack-families.sh <copy> SEALED-T2S-m32-bigw.elf SEALED-T2S-m4.elf --apply. The servedconfig.jsonis byte-identical to the copy's.bak-pre-families.The run.
1bit forward-servebuilt withONEBIT_NPU=ON, with arms OFF, ON, OFF2, ON2, CHK.config.json.NPU_WIDE_SEALED_ELF=0 NPU_NARROW_SEALED_ELF=0set.7422801a32893ade7422801a32893ade7422801a32893ade7422801a32893adeHow the two rates were measured:
[ph]per-token wall over the 32 decode steps at positions 536–567. The wall-clock difference between the 33-token and 1-token requests can't be used on main: the prefix cache is on (prefix cache on: 8 slots), so the second request skips the prefill and the difference comes out negative. Any older harness that computes decode that way needs the same change.Acceptance, item by item.
[forward] wide family: 8 role(s) at M=32 from …/kernels-m16/SEALED-T2S-m32.elf (chunks of 17..32 rows)and[forward] narrow family: 8 role(s) at M=4 from …/SEALED-T2S-m4.elf (launches of 1..4 rows).NPU_BATCH_CHECK=1. All 7 checked chunks are MATCH: M=32 at positions 0..31, 128..159 and 160..191; M=23 at 512..534; M=1 at 4 and at 535 (twice).What's left is the owner's step: apply the same change to the served pack. The command is
~/briefs/pack-families.sh ~/models/npu/bonsai2-27b-t2s2 ~/.cache/scratch-cl-npuperf/SEALED-T2S-m32-bigw.elf ~/.cache/scratch-cl-npuperf/SEALED-T2S-m4.elf --apply. It keepsconfig.json.bak-pre-families.bonsai2-27b-t2s2-servesymlinks itsconfig.jsonto that pack, so it picks up the four keys. But it has its own realkernels-m16, so the two ELFs must also be copied there, or its serve will log the files as missing and stay on the 16-row set.NPU_WIDE_SEALED_ELF=0 NPU_NARROW_SEALED_ELF=0turns both families off again without touching the pack.Logs:
~/wt/cl-856-logs/a2128/(run.sh, run.out, and one.servelog per arm).bong-water-water-bong commentedon Nov 12, 2026
Applied to the served pack and verified. Closing (cl-856, on the owner's go: the orchestrator wrote "#2128 GO").
What changed. Files were only added; anything rewritten has a backup.
~/briefs/pack-families.sh ~/models/npu/bonsai2-27b-t2s2 SEALED-T2S-m32-bigw.elf SEALED-T2S-m4.elf --apply.config.jsonnow carries the four keys, and the old file is kept asconfig.json.bak-pre-families(sha2566a85adf0…).kernels-m16links tobonsai2-t2s-v2/kernels-m16, so the two ELFs were added there asSEALED-T2S-m32.elf(5dcbc39d…) andSEALED-T2S-m4.elf(0a64f42a…).bonsai2-27b-t2s2-servelinks itsconfig.jsonto the pack above. Itskernels-m16is a real directory, so the same two ELFs were copied there as new files.Ticketed loads. These were
npu-holdruns on port 18173, separate from the :18097 process, using1bitwithONEBIT_NPU=ONat main229ab5e.Load of each directory (368 s ticket):
bonsai2-27b-t2s2andbonsai2-27b-t2s2-serveeach log[forward] wide family: 8 role(s) at M=32 from <that dir>/kernels-m16/SEALED-T2S-m32.elfandnarrow family: 8 role(s) at M=4 from …/SEALED-T2S-m4.elf.7422801a32893ade, the same as every arm of the staged run.NPU_BATCH_CHECK=1on-serveis MATCH on all 7 chunks: M=32 at 0..31, 128..159 and 160..191; M=23 at 512..534; M=1 at 4 and at 535 (twice).Off/on control on the applied pack (263 s ticket):
NPU_WIDE_SEALED_ELF=0 NPU_NARROW_SEALED_ELF=0)The text is identical in every arm.
[ph]per-token wall over the 32 decode steps. Main's prefix cache makes the request-difference method negative.Device health. amdxdna timeout and power-cycle lines stay at 136 → 136 in both tickets, and the ERROR count at 554 → 554. Nothing is left on 18173.
To revert:
NPU_WIDE_SEALED_ELF=0 NPU_NARROW_SEALED_ELF=0at serve time, or restoreconfig.json.bak-pre-families.Logs:
~/wt/cl-856-logs/v2128/(verify.sh, ctl.sh, run.out, ctl.out, and one.servelog per arm).