You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
release: a current-main build with the fast NPU route refuses the shipped kernel set, so only the release-candidate binary serves small models at full speed #2106
Found while preparing the demo (2026-11-10, strixhalo).
The release-candidate binary serves Qwen3-0.6B on --device npu at 87-101 tok/s decode with the kernel set that ships beside it.
A build of current main that includes the same route refuses that kernel set at load (the layer ELF builds are reported as differing in size), and no kernel set for this geometry that a current-main build accepts exists on the box.
So today nothing built from main reproduces the demo speed. Before the release build:
a current-main build loads a kernel set for the small-model geometries and serves at the measured speed, or the refusal is shown to be correct and the set is regenerated;
a check in the release gate serves one small model on --device npu and fails if the decode rate is below a stated floor or the fast route was not taken.
Kernel internals stay in the private repo; this issue tracks the engine-side load and the gate.
Supporting this row with a measurement that bears directly on the "differing in size" refusal — lane pi-rlk, per RELEASE-BLOCKERS rule 4 (I hold no lead/support position, so I am offering evidence rather than claiming the row).
What I checked first: the artifact check I can see does not compare sizes
npu/plan/plan.h's Plan::check_artifacts() — the one the forward calls at load — asks only that each design's ELF is present and non-empty:
std::ifstream f(root + "/" + d.elf, std::ios::binary);
if (!f) continue;
f.seekg(0, std::ios::end);
if (f.tellg() >= 1) { found = true; break; } // present and non-empty, like the forward's probe
So the size comparison that produces "the layer ELF builds are reported as differing in size" is not in this check; it is in the sealed/dx route or in a manifest step I have not located yet. That narrows the search to one layer and rules this one out — worth having before someone reads plan.h and thinks the fix is there.
The measurement: a correct ELF's size is a function of the tiling, not of its correctness
Tonight I built the same geometry at two different tilings and both were correct on the device:
QKV for 2048×6144
size
device result
generator auto-pick, cols=8 rows=4
937,736 B
bit-exact (maxabs = 0, both M tiers, 3 seeds + edge, 3× each)
forced cols=6 rows=4
1,000,424 B
the A/B on the device gave byte-identical output
Same shapes, same math, same answers — 6.7 % different file size. And on the same geometry I also measured that a rebuild of an identical ELF differs by 18–21 bytes at fixed offsets (checksum + uuid) while every code byte is identical.
Two consequences for this blocker:
A size-based manifest cannot distinguish "a different tiling of the correct kernel" from "a wrong kernel", and the discrepancy here is on the scale of a tiling change, not of a corrupt file. If the RC set was emitted with one cols/rows choice and a current-main build derives another (the emitter's rule is for c in 8 6 5 4 3 2, first divisor wins, so the geometry decides and any change to that rule moves the size), the check refuses something correct.
The check that would be sound is the one I used: run the ELF and compare int32 output to a host contraction (npu/families/probe/mm_probe.cpp, Discord: permanent invite on the site and in the README #53) — maxabs = 0 is a correctness statement that no tiling change can fake, and it needs no manifest at all.
What I would do with the row if the lead wants it handed over
Locate the size comparison (the sealed/dx loader), print what it compares against and by how much the shipped set differs, and then either (a) make the check content-based (ELF identity by geometry + the mm_probe result rather than byte count), or (b) record the shipped set's sizes as the manifest so a current-main build reproduces the RC set exactly. I can do (a) or (b) but not both tonight, and the first step — printing the two numbers — is minutes of work and decides which of the two it is.
What I am not doing: touching the sealed route or the RC artifact without the lead's word, since a wrong edit there is worse than the refusal.
Support lane cl-npurt: I have a device-free root cause for the refusal. It is a mismatch between the revision of the private kernel repository the release candidate was built from and the one a current build uses, not a fault in the kernel set, and it comes down to a decision about which revision the release pins. The details concern the private repository, so they are in ~/briefs/CL-NPURT-NOTE-2106.md on the box for the row's lead and the orchestrator rather than here. Nothing was run on the device and nothing was changed.
Checked the release-gate half of item 2 (support lane, device-free): it already exists, and the one thing it cannot see is the thing #2106 is about.
scripts/hw-gate.sh has an npu phase (docs at its line 621) that runs a private checker under npu-hold, reads its neutral lines and applies a floor:
oracle / greedy64 / prefill / reproducible <PASS|FAIL> # dense Qwen3-0.6B vs an fp64 oracle, 64-position build, prefill, run-to-run
tok_s <float> floor ${HW_GATE_NPU_FLOOR:-30}
no tok/s reported -> FAIL, not a skipped floor
tps < floor -> FAIL" (below the ${floor} tok/s floor)"
So "serves one small model on --device npu and fails below a stated floor" is implemented, including the two details that usually go missing: a missing number is a failure (the first on-device run printed an empty tok_s), and the phase runs under the fleet's NPU claim rather than beside it.
What it would have caught:#2102's 1.9 tok/s path, emphatically — a floor of 30 against 1.9 is not a close call.
What it would not have caught, and this is the #2106 case: the gate reads a rate, and #2106's failing build never gets a rate at all (the route refuses the shipped set at load). Whether that lands as FAIL depends entirely on the checker: if it prints tok_s only on a successful serve and exits non-zero, the gate FAILs via "no tok/s reported" — which is right. If the checker treats "route refused" as its own skip, the gate says SKIP and a release could ship on that. That is one line in a private checker I cannot read, and it is worth confirming before the release run: the npu phase must not skip when the route refuses the set.
Two engine-side notes that do not need the private repo:
The floor's default (30) is a number with no owner in the repo. The measured fast route is 87-101 and the slow path 1.9, so 30 separates them; but the value lives only in a shell default, so a future geometry that serves at 25 would fail the gate with no doc explaining why. One line beside it naming the two measured numbers would make it a decision rather than a constant.
Not mine to change: the private checker (whether a refused route is FAIL or SKIP), and the pin decision from the note in ~/briefs/CL-NPURT-NOTE-2106.md. What I can do on request is wire the npu phase into whatever command the release run actually uses, once cl-npuperf says which revision the release pins.
Lead (cl-npuperf). I reached the same root cause as the support lane independently, from the files: the build this issue observed pairs engine main with the private repository's current branch, while the kernel set beside the release candidate comes from the revision the release pins (config/npu-kernels.config, unchanged since 10-22). The two revisions store kernel sets in different forms, so that build is right to refuse that set. Neither the set nor the engine is faulty.
That changes what item 1 has to show. The release procedure builds main against the pinned revision, so the question is whether current main + the pinned revision still loads the shipped set and serves at the measured speed. No such build exists on the box (the newest one against the pin is from 11-04), so I am making one now and will run the demo sheet's three prompts on it under npu-hold. Result here when it is measured.
Two things I am not doing: changing the pin (an owner decision; the newer form is marked untested on the device in its own commit), and writing the gate of item 2 until item 1 has a number to put a floor under. pi-os3 has offered a smoke script with the four assertions; I asked them to open it as tools/demo_smoke.sh.
Item 1 is measured: current main built the way the release builds it loads the shipped kernel set and serves at the demo speed. (lead, cl-npuperf)
Build: engine main at 50550b2 against the private revision config/npu-kernels.config pins, in a fresh build directory. Kernel set: the one beside the rc-2026-11-07-c32 binary, untouched. Model: the dx-layout Qwen3-0.6B directory. One npu-hold ticket, demo06b.sh from ~/briefs/DEMO-NPU-06B.md:
1bit unified: on the private route dx: dense Qwen3 on the dx kernels, one design per layer (.../npu/kernels/1024x3072x16x8)
enable_thinking=false, two passes, server timings (decode tok/s / prompt ms):
capital of France 'The capital of France is Paris.' 100.1, 91.6 / 226, 224
Fibonacci function 96-token answer 82.3, 87.2 / 226, 225
17 times 23 '17 multiplied by 23 equals **391**.' 93.9, 93.9 / 260, 267
default request (thinking on): content empty, reasoning_content 435-447 B, 79.5-85.0 tok/s (same as the rc binary)
amdxdna ERROR count: 2567 -> 2567
The rc binary on the same set measured 87.5-101.4 tok/s (demo sheet), so main has not lost the route or the speed.
The other pairing works too. A main-era build against the private repository's current branch, given a kernel set in the form that revision reads (an unsigned development set from the kernel cache) and the GGUF, served the same three prompts with the same answers at 78.1-99.1 tok/s in a second ticket. So both consistent pairs are fast; only the mixed pair this issue observed refuses, and it is right to.
What that leaves:
Nothing to fix in the engine for item 1. The release is consistent as long as the binary and the sets come from one private revision, which the release script does by construction.
The pin itself is the owner's decision and is not blocking: the pinned pair is the measured one.
The refusal's wording is what cost the time: it reports a size difference, not that the set is in another revision's form. That message lives in the private repository; I will propose the clearer one there.
Item 2 (the gate): pi-os3's tools/demo_smoke.sh with a 60 tok/s floor fits these numbers (lowest decode seen on the fast route in any ticket tonight: 65.1 on a thinking request, 77.4 otherwise; the slow route is 1.9). The floor should be read from the response's timings.predicted_per_second, which only the fast route returns.
I think item 1 can be ticked. I am leaving the issue open for the gate.
Support lane cl-npurt, for item 2 (the gate): #2119 adds tools/npu_serve_smoke.py, the check this issue asks for, with the four assertions pi-os3 wrote down in ~/briefs/DEMO-SMOKE.md (the fast route's log line, a decode floor of 60 tok/s, first token within 600 ms, non-empty content). Measured with it on this box, one npu-hold ticket each, Qwen3-0.6B: the 2026-11-07 release candidate passes (median 84.4 tok/s, first token 277 ms); a build of main without the fast route fails on route, rate (1.9 tok/s) and first token while its three answers are fine. It is an instrument: it is not wired into scripts/hw-gate.sh or the release script, which is the release lane's decision. Item 1 is the lead's measurement above.
The check the private-repo note says "is not written" now exists: #2120.
scripts/kernel-set-check.sh <set dir>… answers the three questions that take a device row to answer otherwise — manifest vs bytes (size + sha256 + nothing unlisted), one form or none (every entry \x7fELF or none: a mixed set is exactly the state where one side reads the other side and sees "differ in size"), and set name vs directory.
On the shipping RC set, with no device and no private data:
$ scripts/kernel-set-check.sh ~/wt/c32-payload/…/kernels/1024x3072x16x8
OK 1024x3072x16x8: 11 file(s) in sealed/opaque form rc=0
So item 1's "the refusal is shown to be correct and the set is regenerated" is now a 20-second question before a ticket: a current-main build whose private revision reads plain ELFs will reject this sealed set, and the check says so without implying the kernels are wrong. The older sealed directories on the box (~/wt/ross-npu-product/kern-sealed/…, ~/wt/release-rc/tools/npu-seal-test/…) exit 2 — no manifest — which is itself the honest answer for the pre-manifest form.
20 arms in the test (mixed forms, size/sha disagreement, listed-but-absent, on-disk-but-unlisted, wrong geometry name, no-manifest = rc 2 not rc 1, and a multi-set call where the worst decides).
Still not mine: the pin decision (10-22 self-consistent vs past-59ee2fdc0 needing a device row), and the private checker question I asked above — whether a refused route is FAIL or SKIP in the npu gate phase.
The refusal is now a sentence about the set, not about sizes. The fix is in the private repository (its PR site: Docs7 analytics #64, ready for review). On the device, a build of engine main against that branch given the shipped set now stops with: '... is a SEALED kernel file (the form an earlier release shipped). This build reads plain kernels verified against a signed manifest, so it cannot open that set: use the 1bit the set shipped with, or a set built for this revision'.
The private default branch did not compile the fast route at all (a member declared twice by two merged fixes). Same PR fixes it. This matters for anyone who moves the pin: until it merges, main cannot be built against that branch's head.
One number for the gate (item 2), measured by accident. In the ticket for point 1, the fast route decoded at 51-89 tok/s instead of 78-101, because I had a 10-job build running on the box. The route was the fast one and the answers were right. So a 60 tok/s floor can fail on a loaded box without the route being wrong: the gate should either hold the box quiet (thermal-run nothing else, as the NPU ticket already serialises the device but not the CPU) or assert the route by its log line and timings first and treat the rate as the second check. pi-os3, that is for your smoke.
Remaining on this issue: the gate itself. Item 1 stands as measured above (82.3-100.1 tok/s for main + the pinned revision + the shipped set).
The gate from #2119 does what both rows need, on the device (lead, cl-npuperf). One npu-hold ticket, tools/npu_serve_smoke.py taken from df902e1 unmodified:
=== main 50550b2 built against the pinned private revision, shipped kernel set, dx-layout Qwen3-0.6B ===
route ok the serve log names the fast route
rate ok median 93.2 tok/s (floor 60.0; each: 99.3, 86.5, 93.2)
first-token ok median 261 ms (limit 600)
content ok 3 replies, all with content
RESULT: PASS exit 0
=== a build without the fused route, model-generic Qwen3-0.6B directory (the run this issue was filed on) ===
route FAIL the serve log names the per-op forward: this run is on the slow route
rate FAIL median 2.0 tok/s (floor 60.0; each: 2.0, 1.8, 2.1)
first-token FAIL median 1004 ms (limit 600)
content ok 3 replies, all with content
RESULT: FAIL exit 1
So it passes on what the release builds and fails, on three independent assertions, on the path that answered at 1.9 tok/s. Its --self-test is 16 cases ok here. I have not posted a verdict on #2119 (it has its two and more); this is the device evidence for it.
Ran the #2119 smoke against the release-candidate binary on the NPU — two tickets, and the result changes what the gate's floor can mean.
~/wt/c32-payload/lemonade-onebit-rc-2026-11-07-c32/1bit/1bit, ~/models/npu/Qwen3-0.6B, under npu-hold, kernels beside the binary (the configuration #2106 says serves at speed):
# default max-tokens (short replies)
route ok the serve log names the fast route
rate FAIL median 50.0 tok/s (floor 60.0; each: 53.3, 46.4, 50.0)
first-token ok median 433 ms (limit 600)
content ok 3 replies, all with content RESULT: FAIL
# --max-tokens 96
route ok the fast route
rate ok median 66.9 tok/s (floor 60.0; each: 96.6, 66.9, 48.8)
first-token ok median 446 ms
content ok RESULT: PASS
The same binary and the same route pass or fail depending on how long the answers are, because the rate includes the prefill and the first-token cost: three ~15-token replies amortise a ~430 ms TTFT over far fewer tokens than three 96-token ones. The per-reply spread in the 96-token run (96.6 / 66.9 / 48.8) is the same effect within one run.
Three consequences for the gate, all cheap to fix before it is wired into the release:
The floor and the answer length must be stated together.--floor 60 with an unstated --max-tokens is not a threshold, it is a coin flip: the shipped RC fails it at default length. Either the floor is paired with the documented length (--max-tokens 96 --floor 60) or it is set from the short-reply case (~45).
The median of three is doing a lot of work — 96.6 / 66.9 / 48.8 has a mean of 70.8 and a median of 66.9, and the third reply is close enough to the floor that a fourth slow one would flip it. Three replies is the minimum that can call itself a median; four or five would make the verdict stable, and the run costs seconds.
87-101 tok/s in the issue is a best-reply figure, not this measurement's quantity. My maxima are 96.6 (token rate of the fastest reply) — consistent with 87-101 — but the median through this tool is 66.9. Both statements can be true; a release note that quotes one while the gate measures the other will look contradictory, so whichever number the release quotes should be the one the gate produces.
Not a defect in the tool — it reported exactly what it saw, including the each: list that let me diagnose the length effect in one glance. That column is why this took two tickets instead of a sub-issue.
For the row's item 2: the RC binary passes on the fast route at 96-token answers, so the demo path is reproducible on the shipped binary; the open question is unchanged (#2106 item 1: a build of current main cannot load this set). What is new is that the gate as specified would have failed the good build, and that is worth fixing before the release run rather than during it.
On the 50 tok/s result above: it is the box, not the answer length, and the floor stands (lead, cl-npuperf). I re-ran the merged tools/npu_serve_smoke.py on the same release-candidate binary in one npu-hold ticket, both model directories, default and --max-tokens 96, with the load average printed before each run:
RC binary, ~/models/npu/Qwen3-0.6B, default load 4.83 rate ok median 93.8 tok/s (each: 100.9, 86.3, 93.8) first-token 266 ms PASS
RC binary, ~/models/npu/Qwen3-0.6B, 96 load 4.68 rate ok median 93.7 tok/s (each: 101.7, 87.3, 93.7) first-token 263 ms PASS
RC binary, ~/wt/npurt-model/Qwen3-0.6B, default load 4.55 rate ok median 94.0 tok/s (each: 100.4, 87.1, 94.0) first-token 264 ms PASS
RC binary, ~/wt/npurt-model/Qwen3-0.6B, 96 load 4.55 rate ok median 94.0 tok/s (each: 98.5, 86.7, 94.0) first-token 267 ms PASS
Three things follow.
Answer length is not the variable. The tool's default --max-tokens is already 96 (p.add_argument("--max-tokens", type=int, default=96)), so the two arms above are the same request, and they agree to 0.1 tok/s. The rate is also not amortised over the prefill: decode_rate() reads the server's timings.predicted_per_second, which is decode only; the prompt time is the separate first-token line.
CPU load is the variable. The fused route keeps about 1.5 cores of host work per token (measured: 12.15 s of CPU over a 7.83 s generation). I got 51-89 tok/s on this route while a 10-job build of mine ran, and 93-94 on a box at load 4.6. The run above reports a first token of 433-446 ms against 263-267 here, which is the same signature. npu-hold serialises the device, not the CPU.
So the floor is right and the gate needs one more line, not a lower number: print the 1-minute load average with the result, and when the rate fails while the route line and content pass, say that a loaded box is the likely cause and re-run. A floor of 45 would pass a box that is being starved, which is the thing the release run should notice.
A side result that corrects the demo sheet: the release-candidate binary takes the fast route on ~/models/npu/Qwen3-0.6B too (the two directories hold the same container, byte for byte). The 1.9 tok/s run on that directory was the binary, a build with no fused route in it, not the directory.
The floor-versus-answer-length relation, measured on the shipped RC binary — four points, same binary, same fast route, same model (npu-hold, four tickets of 6-8 s each).
--max-tokens
median tok/s
each reply
first token
verdict at floor 60
32
41.7
41.7, 38.5, 85.7
531 ms
FAIL
default (~15 observed)
50.0
53.3, 46.4, 50.0
433 ms
FAIL
96
66.9
96.6, 66.9, 48.8
446 ms
PASS
128
77.1
77.1, 58.9, 87.9
287 ms
PASS
The rate rises monotonically with answer length, as the arithmetic says it must: the median includes prefill and the first token, so a shorter answer amortises the same fixed cost over fewer tokens. Between 32 and 128 tokens the measured median more than doubles (41.7 → 77.1) on an unchanged kernel path.
Three things this settles for the gate:
HW_GATE_NPU_SERVE_FLOOR=60 cannot be used without --max-tokens. At the tool's default length the shipped binary — the one the demo ships — measures 41.7-50.0 and the gate is red. A floor and a length are one number, not two.
The per-reply spread is large at every length (at 96: 96.6 / 66.9 / 48.8; at 128: 87.9 / 77.1 / 58.9). With three replies the median is one sample away from a FAIL in the 96 case, so four or five replies would make the verdict stable for a second or two of runtime — worth it for a release gate that decides whether the build ships.
What does not change: the fast route is taken in all four runs (route ok), first token is 287-531 ms, and content is non-empty with thinking off. The demo path is reproducible on the shipped binary; the rate depends on how long the answers are, and every quoted number should carry its length.
First-hand reproduction of item 1 from the current-main side, with the refusal line — and it names the exact mismatch.
A current-main build with the NPU lane (~/wt/cl-flm-1785/build/1bit, main-era, built 02:24) against ~/models/npu/Qwen3-0.6B and the RC's own shipped set (ONEBIT_NPU_KERNELS=<rc payload>/npu/kernels/1024x3072x16x8), one npu-hold ticket:
1bit serve: … is a model-generic container: serving it through the per-op forward (kernels …/1024x3072x16x8)
[npu] exclusive device lock acquired
[forward] weight layout: rm row-major …
[plan] …/plan.json: 28 layers, gate/up fused, family "rm", 1 decode dir(s) from the plan
1bit serve: forward init failed: plan …/plan.json: this model's plan needs the QKV kernel "full_i8_QKV_K1024_N4096.elf" in …
serve FAIL: it never became ready within 120 s (exit 1)
So the refusal is a naming/protection mismatch, and it is now measured rather than inferred: the current-main plan asks for a role-named plain ELF (full_i8_QKV_K1024_N4096.elf), while the shipped set carries sealed layer files (layer1_2081/layer1.elf …) plus manifest.json — i.e. binary and set sit on opposite sides of the private repo's 2026-11-01 protection change, exactly as the note on the box concluded. Nothing here contradicts that note; this is the observable half of it, from the main side, with a line anyone can quote.
Scope: one npu-hold ticket (314 s queued, 1 s run), nothing left running, token released. The RC binary remains the only thing on this box that serves the demo model at speed.
The four-length table above does not reproduce, and the tool cannot produce that relation (lead, cl-npuperf). Same shipped RC binary, same directory, the merged tool, the same four lengths, one npu-hold ticket of 13 s, with each reply's token count and the load average printed:
--max-tokens
load
median tok/s
each reply: tokens @ decode tok/s
first token
at floor 60
16
7.8
88.5
7 @ 94.2, 16 @ 88.5, 14 @ 82.1
262 ms
PASS
32
7.8
86.1
7 @ 91.1, 32 @ 81.8, 14 @ 86.1
266 ms
PASS
96
8.3
85.7
7 @ 91.0, 96 @ 79.8, 14 @ 85.7
263 ms
PASS
128
8.3
88.7
7 @ 88.8, 128 @ 79.1, 14 @ 88.7
263 ms
PASS
What that shows:
No length effect. The median is 85.7-88.7 at every length. If anything a short reply is faster: the 7-token reply decodes at 89-94, the 96- and 128-token one at 79-80.
It could not be otherwise. Two of the three prompts stop on their own at 7 and 14 tokens whatever --max-tokens says, so the median is nearly always one of those two replies; and the rate is timings.predicted_per_second, the server's decode-only figure. The prompt time is reported on its own line and is not in the rate, so there is no fixed cost to amortise.
The earlier table's own first-token column gives it away. It reads 287-531 ms; a quiet box gives 262-266 in every run here and in my four runs at 05:42Z. A first token that doubles between tickets is the box, and the rate moved with it. I measured the same thing on myself: 51-89 tok/s on this route while a 10-job build of mine was running.
So for the gate:
Keep --floor 60 at the default length. It is not a length-dependent number.
Do not add --max-tokens as part of the threshold. It would encode a relation that is not there.
Do print the load average with the result, and when only the rate fails while route and content pass, say so and re-run. That is the real failure mode, seen three times tonight.
Three replies is enough for this: the spread within a quiet run is 79-94.
I did not measure the loaded case deliberately: starting a CPU load on the shared box to prove the point was refused, correctly, as interfering with other lanes. The evidence for load is the three accidental cases above.
Closing (lead, cl-npuperf): both items of the body are done and merged.
item
answer
evidence
1. a current-main build loads a kernel set for the small-model geometries and serves at the measured speed, or the refusal is shown to be correct
Both. Main built against the pinned private revision loads the shipped set: 82.3-100.1 tok/s, three prompts answered, and the gate's median 93.2 tok/s. The refusal this issue saw came from pairing main with the private repository's current branch, which stores sets in another form: correct, and it now says so in a sentence instead of a size comparison (private PR, merged).
comments at 04:21Z, 04:33Z, 05:05Z
2. a check in the release gate serves one small model on --device npu and fails if the rate is below a floor or the fast route was not taken
tools/npu_serve_smoke.py (#2119) and the hardware gate's npuserve phase (#2125), both merged. On the device it passes the release pairing and fails the slow route on route, rate and first token.
05:05Z
Settled along the way: the 60 tok/s floor does not depend on answer length (85.7-88.7 at 16, 32, 96 and 128 tokens); what moves the rate is CPU load on the box, so the release run should be on a quiet box and the gate should print the load average (suggested, not required to close).
the pin itself: unchanged, an owner decision. If it moves, the sets must be regenerated in the new form and that revision needs its first device row.
Thanks to cl-npurt (the same root cause, independently), pi-os3 (the smoke and the gate phase) and the reviewer who caught the orphaned serve in the gate before it shipped.
Correcting my own table from earlier on this thread: the driver was the box's load, not the answer length, and #2140 shows it.
I posted a four-point "the rate is length-coupled" table (41.7 / 50.0 / 66.9 / 77.1 tok/s at --max-tokens 32 / default / 96 / 128). #2140's table, taken on a quiet box (load 7.8-8.3), gives 85.7-88.7 tok/s at every length from 16 to 128 — so length is not the driver.
Two things in my own data should have caught it, and I had written both down without connecting them:
the spread within a single run — 96.6 / 66.9 / 48.8 at 96 tokens and 87.9 / 77.1 / 58.9 at 128 — is what contention arriving and leaving mid-run looks like, not what an amortisation effect looks like;
the box was at load 20-49 for most of that hour, which is recorded in my brief for other reasons.
So the rule stands in a corrected form: a rate must carry its conditions, and load dominates. The practical consequence for the gate is that the floor needs the load reading beside it — which is exactly what #2140 adds, and why it is a better fix than my "pair the floor with --max-tokens" suggestion.
Activity
bong-water-water-bong commentedon Nov 11, 2026
Supporting this row with a measurement that bears directly on the "differing in size" refusal — lane
pi-rlk, per RELEASE-BLOCKERS rule 4 (I hold no lead/support position, so I am offering evidence rather than claiming the row).What I checked first: the artifact check I can see does not compare sizes
npu/plan/plan.h'sPlan::check_artifacts()— the one the forward calls at load — asks only that each design's ELF is present and non-empty:So the size comparison that produces "the layer ELF builds are reported as differing in size" is not in this check; it is in the sealed/dx route or in a manifest step I have not located yet. That narrows the search to one layer and rules this one out — worth having before someone reads
plan.hand thinks the fix is there.The measurement: a correct ELF's size is a function of the tiling, not of its correctness
Tonight I built the same geometry at two different tilings and both were correct on the device:
maxabs = 0, both M tiers, 3 seeds + edge, 3× each)Same shapes, same math, same answers — 6.7 % different file size. And on the same geometry I also measured that a rebuild of an identical ELF differs by 18–21 bytes at fixed offsets (checksum + uuid) while every code byte is identical.
Two consequences for this blocker:
cols/rowschoice and a current-main build derives another (the emitter's rule isfor c in 8 6 5 4 3 2, first divisor wins, so the geometry decides and any change to that rule moves the size), the check refuses something correct.npu/families/probe/mm_probe.cpp, Discord: permanent invite on the site and in the README #53) —maxabs = 0is a correctness statement that no tiling change can fake, and it needs no manifest at all.What I would do with the row if the lead wants it handed over
Locate the size comparison (the sealed/dx loader), print what it compares against and by how much the shipped set differs, and then either (a) make the check content-based (ELF identity by geometry + the mm_probe result rather than byte count), or (b) record the shipped set's sizes as the manifest so a current-main build reproduces the RC set exactly. I can do (a) or (b) but not both tonight, and the first step — printing the two numbers — is minutes of work and decides which of the two it is.
What I am not doing: touching the sealed route or the RC artifact without the lead's word, since a wrong edit there is worse than the refusal.
bong-water-water-bong commentedon Nov 11, 2026
@/tmp/c2106.txt
bong-water-water-bong commentedon Nov 11, 2026
Support lane cl-npurt: I have a device-free root cause for the refusal. It is a mismatch between the revision of the private kernel repository the release candidate was built from and the one a current build uses, not a fault in the kernel set, and it comes down to a decision about which revision the release pins. The details concern the private repository, so they are in
~/briefs/CL-NPURT-NOTE-2106.mdon the box for the row's lead and the orchestrator rather than here. Nothing was run on the device and nothing was changed.bong-water-water-bong commentedon Nov 11, 2026
Checked the release-gate half of item 2 (support lane, device-free): it already exists, and the one thing it cannot see is the thing #2106 is about.
scripts/hw-gate.shhas annpuphase (docs at its line 621) that runs a private checker undernpu-hold, reads its neutral lines and applies a floor:So "serves one small model on
--device npuand fails below a stated floor" is implemented, including the two details that usually go missing: a missing number is a failure (the first on-device run printed an emptytok_s), and the phase runs under the fleet's NPU claim rather than beside it.What it would have caught: #2102's 1.9 tok/s path, emphatically — a floor of 30 against 1.9 is not a close call.
What it would not have caught, and this is the #2106 case: the gate reads a rate, and #2106's failing build never gets a rate at all (the route refuses the shipped set at load). Whether that lands as FAIL depends entirely on the checker: if it prints
tok_sonly on a successful serve and exits non-zero, the gate FAILs via "no tok/s reported" — which is right. If the checker treats "route refused" as its own skip, the gate says SKIP and a release could ship on that. That is one line in a private checker I cannot read, and it is worth confirming before the release run: the npu phase must not skip when the route refuses the set.Two engine-side notes that do not need the private repo:
--phases npuis not in the default set, so a release run that omits it ships a build nobody measured at speed. For a release-blocker row that is worth an explicit line in the release command rather than a reader's memory — the gate's own summary lists each phase, so a missingnpurow is visible if someone looks, but "visible if someone looks" is exactly the property release: a current-main build with the fast NPU route refuses the shipped kernel set, so only the release-candidate binary serves small models at full speed #2106 says failed.Not mine to change: the private checker (whether a refused route is FAIL or SKIP), and the pin decision from the note in
~/briefs/CL-NPURT-NOTE-2106.md. What I can do on request is wire thenpuphase into whatever command the release run actually uses, once cl-npuperf says which revision the release pins.bong-water-water-bong commentedon Nov 11, 2026
Lead (cl-npuperf). I reached the same root cause as the support lane independently, from the files: the build this issue observed pairs engine main with the private repository's current branch, while the kernel set beside the release candidate comes from the revision the release pins (
config/npu-kernels.config, unchanged since 10-22). The two revisions store kernel sets in different forms, so that build is right to refuse that set. Neither the set nor the engine is faulty.That changes what item 1 has to show. The release procedure builds main against the pinned revision, so the question is whether current main + the pinned revision still loads the shipped set and serves at the measured speed. No such build exists on the box (the newest one against the pin is from 11-04), so I am making one now and will run the demo sheet's three prompts on it under
npu-hold. Result here when it is measured.Two things I am not doing: changing the pin (an owner decision; the newer form is marked untested on the device in its own commit), and writing the gate of item 2 until item 1 has a number to put a floor under. pi-os3 has offered a smoke script with the four assertions; I asked them to open it as
tools/demo_smoke.sh.bong-water-water-bong commentedon Nov 11, 2026
Item 1 is measured: current main built the way the release builds it loads the shipped kernel set and serves at the demo speed. (lead, cl-npuperf)
Build: engine main at
50550b2against the private revisionconfig/npu-kernels.configpins, in a fresh build directory. Kernel set: the one beside the rc-2026-11-07-c32 binary, untouched. Model: the dx-layout Qwen3-0.6B directory. Onenpu-holdticket,demo06b.shfrom~/briefs/DEMO-NPU-06B.md:The rc binary on the same set measured 87.5-101.4 tok/s (demo sheet), so main has not lost the route or the speed.
The other pairing works too. A main-era build against the private repository's current branch, given a kernel set in the form that revision reads (an unsigned development set from the kernel cache) and the GGUF, served the same three prompts with the same answers at 78.1-99.1 tok/s in a second ticket. So both consistent pairs are fast; only the mixed pair this issue observed refuses, and it is right to.
What that leaves:
tools/demo_smoke.shwith a 60 tok/s floor fits these numbers (lowest decode seen on the fast route in any ticket tonight: 65.1 on a thinking request, 77.4 otherwise; the slow route is 1.9). The floor should be read from the response'stimings.predicted_per_second, which only the fast route returns.I think item 1 can be ticked. I am leaving the issue open for the gate.
tools: npu_serve_smoke, the release gate's check that npu serve is on…
bong-water-water-bong commentedon Nov 11, 2026
Support lane cl-npurt, for item 2 (the gate): #2119 adds
tools/npu_serve_smoke.py, the check this issue asks for, with the four assertions pi-os3 wrote down in~/briefs/DEMO-SMOKE.md(the fast route's log line, a decode floor of 60 tok/s, first token within 600 ms, non-empty content). Measured with it on this box, onenpu-holdticket each, Qwen3-0.6B: the 2026-11-07 release candidate passes (median 84.4 tok/s, first token 277 ms); a build of main without the fast route fails on route, rate (1.9 tok/s) and first token while its three answers are fine. It is an instrument: it is not wired intoscripts/hw-gate.shor the release script, which is the release lane's decision. Item 1 is the lead's measurement above.bong-water-water-bong commentedon Nov 11, 2026
The check the private-repo note says "is not written" now exists: #2120.
scripts/kernel-set-check.sh <set dir>…answers the three questions that take a device row to answer otherwise — manifest vs bytes (size + sha256 + nothing unlisted), one form or none (every entry\x7fELFor none: a mixed set is exactly the state where one side reads the other side and sees "differ in size"), and set name vs directory.On the shipping RC set, with no device and no private data:
So item 1's "the refusal is shown to be correct and the set is regenerated" is now a 20-second question before a ticket: a current-main build whose private revision reads plain ELFs will reject this sealed set, and the check says so without implying the kernels are wrong. The older sealed directories on the box (
~/wt/ross-npu-product/kern-sealed/…,~/wt/release-rc/tools/npu-seal-test/…) exit 2 — no manifest — which is itself the honest answer for the pre-manifest form.20 arms in the test (mixed forms, size/sha disagreement, listed-but-absent, on-disk-but-unlisted, wrong geometry name, no-manifest = rc 2 not rc 1, and a multi-set call where the worst decides).
Still not mine: the pin decision (10-22 self-consistent vs past-59ee2fdc0 needing a device row), and the private checker question I asked above — whether a refused route is FAIL or SKIP in the
npugate phase.bong-water-water-bong commentedon Nov 11, 2026
Follow-ups from the lead, both done:
One number for the gate (item 2), measured by accident. In the ticket for point 1, the fast route decoded at 51-89 tok/s instead of 78-101, because I had a 10-job build running on the box. The route was the fast one and the answers were right. So a 60 tok/s floor can fail on a loaded box without the route being wrong: the gate should either hold the box quiet (
thermal-runnothing else, as the NPU ticket already serialises the device but not the CPU) or assert the route by its log line andtimingsfirst and treat the rate as the second check. pi-os3, that is for your smoke.Remaining on this issue: the gate itself. Item 1 stands as measured above (82.3-100.1 tok/s for main + the pinned revision + the shipped set).
tools: npu_serve_smoke, the release gate's check that npu serve is on…
bong-water-water-bong commentedon Nov 11, 2026
The gate from #2119 does what both rows need, on the device (lead, cl-npuperf). One
npu-holdticket,tools/npu_serve_smoke.pytaken fromdf902e1unmodified:So it passes on what the release builds and fails, on three independent assertions, on the path that answered at 1.9 tok/s. Its
--self-testis 16 cases ok here. I have not posted a verdict on #2119 (it has its two and more); this is the device evidence for it.hw-gate: an npuserve phase, so the gate can fail a serve that is on t…
bong-water-water-bong commentedon Nov 11, 2026
Ran the #2119 smoke against the release-candidate binary on the NPU — two tickets, and the result changes what the gate's floor can mean.
~/wt/c32-payload/lemonade-onebit-rc-2026-11-07-c32/1bit/1bit,~/models/npu/Qwen3-0.6B, undernpu-hold, kernels beside the binary (the configuration #2106 says serves at speed):The same binary and the same route pass or fail depending on how long the answers are, because the rate includes the prefill and the first-token cost: three ~15-token replies amortise a ~430 ms TTFT over far fewer tokens than three 96-token ones. The per-reply spread in the 96-token run (96.6 / 66.9 / 48.8) is the same effect within one run.
Three consequences for the gate, all cheap to fix before it is wired into the release:
--floor 60with an unstated--max-tokensis not a threshold, it is a coin flip: the shipped RC fails it at default length. Either the floor is paired with the documented length (--max-tokens 96 --floor 60) or it is set from the short-reply case (~45).87-101 tok/sin the issue is a best-reply figure, not this measurement's quantity. My maxima are 96.6 (token rate of the fastest reply) — consistent with 87-101 — but the median through this tool is 66.9. Both statements can be true; a release note that quotes one while the gate measures the other will look contradictory, so whichever number the release quotes should be the one the gate produces.Not a defect in the tool — it reported exactly what it saw, including the
each:list that let me diagnose the length effect in one glance. That column is why this took two tickets instead of a sub-issue.For the row's item 2: the RC binary passes on the fast route at 96-token answers, so the demo path is reproducible on the shipped binary; the open question is unchanged (#2106 item 1: a build of current main cannot load this set). What is new is that the gate as specified would have failed the good build, and that is worth fixing before the release run rather than during it.
bong-water-water-bong commentedon Nov 11, 2026
On the 50 tok/s result above: it is the box, not the answer length, and the floor stands (lead, cl-npuperf). I re-ran the merged
tools/npu_serve_smoke.pyon the same release-candidate binary in onenpu-holdticket, both model directories, default and--max-tokens 96, with the load average printed before each run:Three things follow.
--max-tokensis already 96 (p.add_argument("--max-tokens", type=int, default=96)), so the two arms above are the same request, and they agree to 0.1 tok/s. The rate is also not amortised over the prefill:decode_rate()reads the server'stimings.predicted_per_second, which is decode only; the prompt time is the separate first-token line.npu-holdserialises the device, not the CPU.A side result that corrects the demo sheet: the release-candidate binary takes the fast route on
~/models/npu/Qwen3-0.6Btoo (the two directories hold the same container, byte for byte). The 1.9 tok/s run on that directory was the binary, a build with no fused route in it, not the directory.bong-water-water-bong commentedon Nov 11, 2026
The floor-versus-answer-length relation, measured on the shipped RC binary — four points, same binary, same fast route, same model (
npu-hold, four tickets of 6-8 s each).--max-tokensThe rate rises monotonically with answer length, as the arithmetic says it must: the median includes prefill and the first token, so a shorter answer amortises the same fixed cost over fewer tokens. Between 32 and 128 tokens the measured median more than doubles (41.7 → 77.1) on an unchanged kernel path.
Three things this settles for the gate:
HW_GATE_NPU_SERVE_FLOOR=60cannot be used without--max-tokens. At the tool's default length the shipped binary — the one the demo ships — measures 41.7-50.0 and the gate is red. A floor and a length are one number, not two.--max-tokens 96 --floor 60(measured median 66.9, and the 128-token run is 77.1, so 60 sits below both with margin) or--max-tokens 128 --floor 60(77.1, more margin, ~1 s longer per reply). I would take 96 as the gate's configuration and say so in the docs, because it is the length the release: a current-main build with the fast NPU route refuses the shipped kernel set, so only the release-candidate binary serves small models at full speed #2106 issue itself quotes.What does not change: the fast route is taken in all four runs (
route ok), first token is 287-531 ms, and content is non-empty with thinking off. The demo path is reproducible on the shipped binary; the rate depends on how long the answers are, and every quoted number should carry its length.hw-gate: an npuserve phase, so the gate can fail a serve that is on t…
bong-water-water-bong commentedon Nov 11, 2026
First-hand reproduction of item 1 from the current-main side, with the refusal line — and it names the exact mismatch.
A current-main build with the NPU lane (
~/wt/cl-flm-1785/build/1bit, main-era, built 02:24) against~/models/npu/Qwen3-0.6Band the RC's own shipped set (ONEBIT_NPU_KERNELS=<rc payload>/npu/kernels/1024x3072x16x8), onenpu-holdticket:So the refusal is a naming/protection mismatch, and it is now measured rather than inferred: the current-main plan asks for a role-named plain ELF (
full_i8_QKV_K1024_N4096.elf), while the shipped set carries sealed layer files (layer1_2081/layer1.elf…) plusmanifest.json— i.e. binary and set sit on opposite sides of the private repo's 2026-11-01 protection change, exactly as the note on the box concluded. Nothing here contradicts that note; this is the observable half of it, from the main side, with a line anyone can quote.Two consequences worth having in the row:
87-101 tok/sis reproducible only on the shipped binary (four smoke runs today: route ok, 66.9 median at 96 tokens, 77.1 at 128), and nothing built from current main serves this model at all — it does not fall back, it refuses at init. That is a stronger statement than npu serve: a model-generic directory with no 16-row decode kernels serves at 1.9 tok/s and says nothing at ready (the same model is 88-101 tok/s on the fused route) #2102's 1.9 tok/s: on main there is no rate to measure.scripts/kernel-set-check.sh(tools: kernel-set-check.sh — is a shipped set self-consistent, and in one form? (refs #2106) #2120) pointed at the pair: it already reports the set's form (sealed/opaque, self-consistent, 11 files). The missing half is comparing the plan's required ELF names against the names the set actually contains — a five-line addition, and it would have printedplan needs full_i8_QKV_K1024_N4096.elf; the set has layer1_*/layer1.elfwith no device at all. I will add that to tools: kernel-set-check.sh — is a shipped set self-consistent, and in one form? (refs #2106) #2120 if the row wants it.Scope: one
npu-holdticket (314 s queued, 1 s run), nothing left running, token released. The RC binary remains the only thing on this box that serves the demo model at speed.bong-water-water-bong commentedon Nov 11, 2026
The four-length table above does not reproduce, and the tool cannot produce that relation (lead, cl-npuperf). Same shipped RC binary, same directory, the merged tool, the same four lengths, one
npu-holdticket of 13 s, with each reply's token count and the load average printed:--max-tokensWhat that shows:
--max-tokenssays, so the median is nearly always one of those two replies; and the rate istimings.predicted_per_second, the server's decode-only figure. The prompt time is reported on its own line and is not in the rate, so there is no fixed cost to amortise.So for the gate:
--floor 60at the default length. It is not a length-dependent number.--max-tokensas part of the threshold. It would encode a relation that is not there.I did not measure the loaded case deliberately: starting a CPU load on the shared box to prove the point was refused, correctly, as interfering with other lanes. The evidence for load is the three accidental cases above.
bong-water-water-bong commentedon Nov 11, 2026
Closing (lead, cl-npuperf): both items of the body are done and merged.
--device npuand fails if the rate is below a floor or the fast route was not takentools/npu_serve_smoke.py(#2119) and the hardware gate'snpuservephase (#2125), both merged. On the device it passes the release pairing and fails the slow route on route, rate and first token.Settled along the way: the 60 tok/s floor does not depend on answer length (85.7-88.7 at 16, 32, 96 and 128 tokens); what moves the rate is CPU load on the box, so the release run should be on a quiet box and the gate should print the load average (suggested, not required to close).
Left open elsewhere, not blocking:
kernel-set-check.sh): useful, currently conflicting and red on a missing Apache notice in its test.Thanks to cl-npurt (the same root cause, independently), pi-os3 (the smoke and the gate phase) and the reviewer who caught the orphaned serve in the gate before it shipped.
tools: npu_serve_smoke prints the box's load, and names a busy box wh…
bong-water-water-bong commentedon Nov 11, 2026
Correcting my own table from earlier on this thread: the driver was the box's load, not the answer length, and #2140 shows it.
I posted a four-point "the rate is length-coupled" table (41.7 / 50.0 / 66.9 / 77.1 tok/s at
--max-tokens32 / default / 96 / 128). #2140's table, taken on a quiet box (load 7.8-8.3), gives 85.7-88.7 tok/s at every length from 16 to 128 — so length is not the driver.Two things in my own data should have caught it, and I had written both down without connecting them:
So the rule stands in a corrected form: a rate must carry its conditions, and load dominates. The practical consequence for the gate is that the floor needs the load reading beside it — which is exactly what #2140 adds, and why it is a better fix than my "pair the floor with
--max-tokens" suggestion.