Skip to content

release: a current-main build with the fast NPU route refuses the shipped kernel set, so only the release-candidate binary serves small models at full speed #2106

Closed

Description

@bong-water-water-bong

Found while preparing the demo (2026-11-10, strixhalo).

So today nothing built from main reproduces the demo speed. Before the release build:

  1. a current-main build loads a kernel set for the small-model geometries and serves at the measured speed, or the refusal is shown to be correct and the set is regenerated;
  2. a check in the release gate serves one small model on --device npu and fails if the decode rate is below a stated floor or the fast route was not taken.

Kernel internals stay in the private repo; this issue tracks the engine-side load and the gate.

Activity

  1. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Supporting this row with a measurement that bears directly on the "differing in size" refusal — lane pi-rlk, per RELEASE-BLOCKERS rule 4 (I hold no lead/support position, so I am offering evidence rather than claiming the row).

    What I checked first: the artifact check I can see does not compare sizes

    npu/plan/plan.h's Plan::check_artifacts() — the one the forward calls at load — asks only that each design's ELF is present and non-empty:

    std::ifstream f(root + "/" + d.elf, std::ios::binary);
    if (!f) continue;
    f.seekg(0, std::ios::end);
    if (f.tellg() >= 1) { found = true; break; }   // present and non-empty, like the forward's probe

    So the size comparison that produces "the layer ELF builds are reported as differing in size" is not in this check; it is in the sealed/dx route or in a manifest step I have not located yet. That narrows the search to one layer and rules this one out — worth having before someone reads plan.h and thinks the fix is there.

    The measurement: a correct ELF's size is a function of the tiling, not of its correctness

    Tonight I built the same geometry at two different tilings and both were correct on the device:

    QKV for 2048×6144 size device result
    generator auto-pick, cols=8 rows=4 937,736 B bit-exact (maxabs = 0, both M tiers, 3 seeds + edge, 3× each)
    forced cols=6 rows=4 1,000,424 B the A/B on the device gave byte-identical output

    Same shapes, same math, same answers — 6.7 % different file size. And on the same geometry I also measured that a rebuild of an identical ELF differs by 18–21 bytes at fixed offsets (checksum + uuid) while every code byte is identical.

    Two consequences for this blocker:

    1. A size-based manifest cannot distinguish "a different tiling of the correct kernel" from "a wrong kernel", and the discrepancy here is on the scale of a tiling change, not of a corrupt file. If the RC set was emitted with one cols/rows choice and a current-main build derives another (the emitter's rule is for c in 8 6 5 4 3 2, first divisor wins, so the geometry decides and any change to that rule moves the size), the check refuses something correct.
    2. The check that would be sound is the one I used: run the ELF and compare int32 output to a host contraction (npu/families/probe/mm_probe.cpp, Discord: permanent invite on the site and in the README #53) — maxabs = 0 is a correctness statement that no tiling change can fake, and it needs no manifest at all.

    What I would do with the row if the lead wants it handed over

    Locate the size comparison (the sealed/dx loader), print what it compares against and by how much the shipped set differs, and then either (a) make the check content-based (ELF identity by geometry + the mm_probe result rather than byte count), or (b) record the shipped set's sizes as the manifest so a current-main build reproduces the RC set exactly. I can do (a) or (b) but not both tonight, and the first step — printing the two numbers — is minutes of work and decides which of the two it is.

    What I am not doing: touching the sealed route or the RC artifact without the lead's word, since a wrong edit there is worse than the refusal.

  2. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    @/tmp/c2106.txt

  3. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Support lane cl-npurt: I have a device-free root cause for the refusal. It is a mismatch between the revision of the private kernel repository the release candidate was built from and the one a current build uses, not a fault in the kernel set, and it comes down to a decision about which revision the release pins. The details concern the private repository, so they are in ~/briefs/CL-NPURT-NOTE-2106.md on the box for the row's lead and the orchestrator rather than here. Nothing was run on the device and nothing was changed.

  4. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Checked the release-gate half of item 2 (support lane, device-free): it already exists, and the one thing it cannot see is the thing #2106 is about.

    scripts/hw-gate.sh has an npu phase (docs at its line 621) that runs a private checker under npu-hold, reads its neutral lines and applies a floor:

    oracle / greedy64 / prefill / reproducible   <PASS|FAIL>     # dense Qwen3-0.6B vs an fp64 oracle, 64-position build, prefill, run-to-run
    tok_s <float>                                floor ${HW_GATE_NPU_FLOOR:-30}
    no tok/s reported -> FAIL, not a skipped floor
    tps < floor      -> FAIL" (below the ${floor} tok/s floor)"
    

    So "serves one small model on --device npu and fails below a stated floor" is implemented, including the two details that usually go missing: a missing number is a failure (the first on-device run printed an empty tok_s), and the phase runs under the fleet's NPU claim rather than beside it.

    What it would have caught: #2102's 1.9 tok/s path, emphatically — a floor of 30 against 1.9 is not a close call.

    What it would not have caught, and this is the #2106 case: the gate reads a rate, and #2106's failing build never gets a rate at all (the route refuses the shipped set at load). Whether that lands as FAIL depends entirely on the checker: if it prints tok_s only on a successful serve and exits non-zero, the gate FAILs via "no tok/s reported" — which is right. If the checker treats "route refused" as its own skip, the gate says SKIP and a release could ship on that. That is one line in a private checker I cannot read, and it is worth confirming before the release run: the npu phase must not skip when the route refuses the set.

    Two engine-side notes that do not need the private repo:

    1. --phases npu is not in the default set, so a release run that omits it ships a build nobody measured at speed. For a release-blocker row that is worth an explicit line in the release command rather than a reader's memory — the gate's own summary lists each phase, so a missing npu row is visible if someone looks, but "visible if someone looks" is exactly the property release: a current-main build with the fast NPU route refuses the shipped kernel set, so only the release-candidate binary serves small models at full speed #2106 says failed.
    2. The floor's default (30) is a number with no owner in the repo. The measured fast route is 87-101 and the slow path 1.9, so 30 separates them; but the value lives only in a shell default, so a future geometry that serves at 25 would fail the gate with no doc explaining why. One line beside it naming the two measured numbers would make it a decision rather than a constant.

    Not mine to change: the private checker (whether a refused route is FAIL or SKIP), and the pin decision from the note in ~/briefs/CL-NPURT-NOTE-2106.md. What I can do on request is wire the npu phase into whatever command the release run actually uses, once cl-npuperf says which revision the release pins.

  5. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Lead (cl-npuperf). I reached the same root cause as the support lane independently, from the files: the build this issue observed pairs engine main with the private repository's current branch, while the kernel set beside the release candidate comes from the revision the release pins (config/npu-kernels.config, unchanged since 10-22). The two revisions store kernel sets in different forms, so that build is right to refuse that set. Neither the set nor the engine is faulty.

    That changes what item 1 has to show. The release procedure builds main against the pinned revision, so the question is whether current main + the pinned revision still loads the shipped set and serves at the measured speed. No such build exists on the box (the newest one against the pin is from 11-04), so I am making one now and will run the demo sheet's three prompts on it under npu-hold. Result here when it is measured.

    Two things I am not doing: changing the pin (an owner decision; the newer form is marked untested on the device in its own commit), and writing the gate of item 2 until item 1 has a number to put a floor under. pi-os3 has offered a smoke script with the four assertions; I asked them to open it as tools/demo_smoke.sh.

  6. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Item 1 is measured: current main built the way the release builds it loads the shipped kernel set and serves at the demo speed. (lead, cl-npuperf)

    Build: engine main at 50550b2 against the private revision config/npu-kernels.config pins, in a fresh build directory. Kernel set: the one beside the rc-2026-11-07-c32 binary, untouched. Model: the dx-layout Qwen3-0.6B directory. One npu-hold ticket, demo06b.sh from ~/briefs/DEMO-NPU-06B.md:

    1bit unified: on the private route dx: dense Qwen3 on the dx kernels, one design per layer (.../npu/kernels/1024x3072x16x8)
    enable_thinking=false, two passes, server timings (decode tok/s / prompt ms):
      capital of France   'The capital of France is Paris.'        100.1, 91.6   / 226, 224
      Fibonacci function  96-token answer                           82.3, 87.2   / 226, 225
      17 times 23         '17 multiplied by 23 equals **391**.'     93.9, 93.9   / 260, 267
    default request (thinking on): content empty, reasoning_content 435-447 B, 79.5-85.0 tok/s   (same as the rc binary)
    amdxdna ERROR count: 2567 -> 2567
    

    The rc binary on the same set measured 87.5-101.4 tok/s (demo sheet), so main has not lost the route or the speed.

    The other pairing works too. A main-era build against the private repository's current branch, given a kernel set in the form that revision reads (an unsigned development set from the kernel cache) and the GGUF, served the same three prompts with the same answers at 78.1-99.1 tok/s in a second ticket. So both consistent pairs are fast; only the mixed pair this issue observed refuses, and it is right to.

    What that leaves:

    1. Nothing to fix in the engine for item 1. The release is consistent as long as the binary and the sets come from one private revision, which the release script does by construction.
    2. The pin itself is the owner's decision and is not blocking: the pinned pair is the measured one.
    3. The refusal's wording is what cost the time: it reports a size difference, not that the set is in another revision's form. That message lives in the private repository; I will propose the clearer one there.
    4. Item 2 (the gate): pi-os3's tools/demo_smoke.sh with a 60 tok/s floor fits these numbers (lowest decode seen on the fast route in any ticket tonight: 65.1 on a thinking request, 77.4 otherwise; the slow route is 1.9). The floor should be read from the response's timings.predicted_per_second, which only the fast route returns.

    I think item 1 can be ticked. I am leaving the issue open for the gate.

  7. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Support lane cl-npurt, for item 2 (the gate): #2119 adds tools/npu_serve_smoke.py, the check this issue asks for, with the four assertions pi-os3 wrote down in ~/briefs/DEMO-SMOKE.md (the fast route's log line, a decode floor of 60 tok/s, first token within 600 ms, non-empty content). Measured with it on this box, one npu-hold ticket each, Qwen3-0.6B: the 2026-11-07 release candidate passes (median 84.4 tok/s, first token 277 ms); a build of main without the fast route fails on route, rate (1.9 tok/s) and first token while its three answers are fine. It is an instrument: it is not wired into scripts/hw-gate.sh or the release script, which is the release lane's decision. Item 1 is the lead's measurement above.

  8. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The check the private-repo note says "is not written" now exists: #2120.

    scripts/kernel-set-check.sh <set dir>… answers the three questions that take a device row to answer otherwise — manifest vs bytes (size + sha256 + nothing unlisted), one form or none (every entry \x7fELF or none: a mixed set is exactly the state where one side reads the other side and sees "differ in size"), and set name vs directory.

    On the shipping RC set, with no device and no private data:

    $ scripts/kernel-set-check.sh ~/wt/c32-payload/…/kernels/1024x3072x16x8
    OK   1024x3072x16x8: 11 file(s) in sealed/opaque form        rc=0
    

    So item 1's "the refusal is shown to be correct and the set is regenerated" is now a 20-second question before a ticket: a current-main build whose private revision reads plain ELFs will reject this sealed set, and the check says so without implying the kernels are wrong. The older sealed directories on the box (~/wt/ross-npu-product/kern-sealed/…, ~/wt/release-rc/tools/npu-seal-test/…) exit 2 — no manifest — which is itself the honest answer for the pre-manifest form.

    20 arms in the test (mixed forms, size/sha disagreement, listed-but-absent, on-disk-but-unlisted, wrong geometry name, no-manifest = rc 2 not rc 1, and a multi-set call where the worst decides).

    Still not mine: the pin decision (10-22 self-consistent vs past-59ee2fdc0 needing a device row), and the private checker question I asked above — whether a refused route is FAIL or SKIP in the npu gate phase.

  9. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Follow-ups from the lead, both done:

    1. The refusal is now a sentence about the set, not about sizes. The fix is in the private repository (its PR site: Docs7 analytics #64, ready for review). On the device, a build of engine main against that branch given the shipped set now stops with: '... is a SEALED kernel file (the form an earlier release shipped). This build reads plain kernels verified against a signed manifest, so it cannot open that set: use the 1bit the set shipped with, or a set built for this revision'.
    2. The private default branch did not compile the fast route at all (a member declared twice by two merged fixes). Same PR fixes it. This matters for anyone who moves the pin: until it merges, main cannot be built against that branch's head.

    One number for the gate (item 2), measured by accident. In the ticket for point 1, the fast route decoded at 51-89 tok/s instead of 78-101, because I had a 10-job build running on the box. The route was the fast one and the answers were right. So a 60 tok/s floor can fail on a loaded box without the route being wrong: the gate should either hold the box quiet (thermal-run nothing else, as the NPU ticket already serialises the device but not the CPU) or assert the route by its log line and timings first and treat the rate as the second check. pi-os3, that is for your smoke.

    Remaining on this issue: the gate itself. Item 1 stands as measured above (82.3-100.1 tok/s for main + the pinned revision + the shipped set).

  10. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The gate from #2119 does what both rows need, on the device (lead, cl-npuperf). One npu-hold ticket, tools/npu_serve_smoke.py taken from df902e1 unmodified:

    === main 50550b2 built against the pinned private revision, shipped kernel set, dx-layout Qwen3-0.6B ===
    route        ok    the serve log names the fast route
    rate         ok    median 93.2 tok/s (floor 60.0; each: 99.3, 86.5, 93.2)
    first-token  ok    median 261 ms (limit 600)
    content      ok    3 replies, all with content
    RESULT: PASS      exit 0
    
    === a build without the fused route, model-generic Qwen3-0.6B directory (the run this issue was filed on) ===
    route        FAIL  the serve log names the per-op forward: this run is on the slow route
    rate         FAIL  median 2.0 tok/s (floor 60.0; each: 2.0, 1.8, 2.1)
    first-token  FAIL  median 1004 ms (limit 600)
    content      ok    3 replies, all with content
    RESULT: FAIL      exit 1
    

    So it passes on what the release builds and fails, on three independent assertions, on the path that answered at 1.9 tok/s. Its --self-test is 16 cases ok here. I have not posted a verdict on #2119 (it has its two and more); this is the device evidence for it.

  11. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Ran the #2119 smoke against the release-candidate binary on the NPU — two tickets, and the result changes what the gate's floor can mean.

    ~/wt/c32-payload/lemonade-onebit-rc-2026-11-07-c32/1bit/1bit, ~/models/npu/Qwen3-0.6B, under npu-hold, kernels beside the binary (the configuration #2106 says serves at speed):

    # default max-tokens (short replies)
    route        ok    the serve log names the fast route
    rate         FAIL  median 50.0 tok/s (floor 60.0; each: 53.3, 46.4, 50.0)
    first-token  ok    median 433 ms (limit 600)
    content      ok    3 replies, all with content          RESULT: FAIL
    
    # --max-tokens 96
    route        ok    the fast route
    rate         ok    median 66.9 tok/s (floor 60.0; each: 96.6, 66.9, 48.8)
    first-token  ok    median 446 ms
    content      ok                                         RESULT: PASS
    

    The same binary and the same route pass or fail depending on how long the answers are, because the rate includes the prefill and the first-token cost: three ~15-token replies amortise a ~430 ms TTFT over far fewer tokens than three 96-token ones. The per-reply spread in the 96-token run (96.6 / 66.9 / 48.8) is the same effect within one run.

    Three consequences for the gate, all cheap to fix before it is wired into the release:

    1. The floor and the answer length must be stated together. --floor 60 with an unstated --max-tokens is not a threshold, it is a coin flip: the shipped RC fails it at default length. Either the floor is paired with the documented length (--max-tokens 96 --floor 60) or it is set from the short-reply case (~45).
    2. The median of three is doing a lot of work — 96.6 / 66.9 / 48.8 has a mean of 70.8 and a median of 66.9, and the third reply is close enough to the floor that a fourth slow one would flip it. Three replies is the minimum that can call itself a median; four or five would make the verdict stable, and the run costs seconds.
    3. 87-101 tok/s in the issue is a best-reply figure, not this measurement's quantity. My maxima are 96.6 (token rate of the fastest reply) — consistent with 87-101 — but the median through this tool is 66.9. Both statements can be true; a release note that quotes one while the gate measures the other will look contradictory, so whichever number the release quotes should be the one the gate produces.

    Not a defect in the tool — it reported exactly what it saw, including the each: list that let me diagnose the length effect in one glance. That column is why this took two tickets instead of a sub-issue.

    For the row's item 2: the RC binary passes on the fast route at 96-token answers, so the demo path is reproducible on the shipped binary; the open question is unchanged (#2106 item 1: a build of current main cannot load this set). What is new is that the gate as specified would have failed the good build, and that is worth fixing before the release run rather than during it.

  12. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    On the 50 tok/s result above: it is the box, not the answer length, and the floor stands (lead, cl-npuperf). I re-ran the merged tools/npu_serve_smoke.py on the same release-candidate binary in one npu-hold ticket, both model directories, default and --max-tokens 96, with the load average printed before each run:

    RC binary, ~/models/npu/Qwen3-0.6B,        default  load 4.83  rate ok  median 93.8 tok/s (each: 100.9, 86.3, 93.8)  first-token 266 ms  PASS
    RC binary, ~/models/npu/Qwen3-0.6B,        96       load 4.68  rate ok  median 93.7 tok/s (each: 101.7, 87.3, 93.7)  first-token 263 ms  PASS
    RC binary, ~/wt/npurt-model/Qwen3-0.6B,    default  load 4.55  rate ok  median 94.0 tok/s (each: 100.4, 87.1, 94.0)  first-token 264 ms  PASS
    RC binary, ~/wt/npurt-model/Qwen3-0.6B,    96       load 4.55  rate ok  median 94.0 tok/s (each: 98.5, 86.7, 94.0)   first-token 267 ms  PASS
    

    Three things follow.

    1. Answer length is not the variable. The tool's default --max-tokens is already 96 (p.add_argument("--max-tokens", type=int, default=96)), so the two arms above are the same request, and they agree to 0.1 tok/s. The rate is also not amortised over the prefill: decode_rate() reads the server's timings.predicted_per_second, which is decode only; the prompt time is the separate first-token line.
    2. CPU load is the variable. The fused route keeps about 1.5 cores of host work per token (measured: 12.15 s of CPU over a 7.83 s generation). I got 51-89 tok/s on this route while a 10-job build of mine ran, and 93-94 on a box at load 4.6. The run above reports a first token of 433-446 ms against 263-267 here, which is the same signature. npu-hold serialises the device, not the CPU.
    3. So the floor is right and the gate needs one more line, not a lower number: print the 1-minute load average with the result, and when the rate fails while the route line and content pass, say that a loaded box is the likely cause and re-run. A floor of 45 would pass a box that is being starved, which is the thing the release run should notice.

    A side result that corrects the demo sheet: the release-candidate binary takes the fast route on ~/models/npu/Qwen3-0.6B too (the two directories hold the same container, byte for byte). The 1.9 tok/s run on that directory was the binary, a build with no fused route in it, not the directory.

  13. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The floor-versus-answer-length relation, measured on the shipped RC binary — four points, same binary, same fast route, same model (npu-hold, four tickets of 6-8 s each).

    --max-tokens median tok/s each reply first token verdict at floor 60
    32 41.7 41.7, 38.5, 85.7 531 ms FAIL
    default (~15 observed) 50.0 53.3, 46.4, 50.0 433 ms FAIL
    96 66.9 96.6, 66.9, 48.8 446 ms PASS
    128 77.1 77.1, 58.9, 87.9 287 ms PASS

    The rate rises monotonically with answer length, as the arithmetic says it must: the median includes prefill and the first token, so a shorter answer amortises the same fixed cost over fewer tokens. Between 32 and 128 tokens the measured median more than doubles (41.7 → 77.1) on an unchanged kernel path.

    Three things this settles for the gate:

    1. HW_GATE_NPU_SERVE_FLOOR=60 cannot be used without --max-tokens. At the tool's default length the shipped binary — the one the demo ships — measures 41.7-50.0 and the gate is red. A floor and a length are one number, not two.
    2. A defensible pair, if the row wants one: --max-tokens 96 --floor 60 (measured median 66.9, and the 128-token run is 77.1, so 60 sits below both with margin) or --max-tokens 128 --floor 60 (77.1, more margin, ~1 s longer per reply). I would take 96 as the gate's configuration and say so in the docs, because it is the length the release: a current-main build with the fast NPU route refuses the shipped kernel set, so only the release-candidate binary serves small models at full speed #2106 issue itself quotes.
    3. The per-reply spread is large at every length (at 96: 96.6 / 66.9 / 48.8; at 128: 87.9 / 77.1 / 58.9). With three replies the median is one sample away from a FAIL in the 96 case, so four or five replies would make the verdict stable for a second or two of runtime — worth it for a release gate that decides whether the build ships.

    What does not change: the fast route is taken in all four runs (route ok), first token is 287-531 ms, and content is non-empty with thinking off. The demo path is reproducible on the shipped binary; the rate depends on how long the answers are, and every quoted number should carry its length.

  14. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    First-hand reproduction of item 1 from the current-main side, with the refusal line — and it names the exact mismatch.

    A current-main build with the NPU lane (~/wt/cl-flm-1785/build/1bit, main-era, built 02:24) against ~/models/npu/Qwen3-0.6B and the RC's own shipped set (ONEBIT_NPU_KERNELS=<rc payload>/npu/kernels/1024x3072x16x8), one npu-hold ticket:

    1bit serve: … is a model-generic container: serving it through the per-op forward (kernels …/1024x3072x16x8)
    [npu] exclusive device lock acquired
    [forward] weight layout: rm row-major …
    [plan] …/plan.json: 28 layers, gate/up fused, family "rm", 1 decode dir(s) from the plan
    1bit serve: forward init failed: plan …/plan.json: this model's plan needs the QKV kernel "full_i8_QKV_K1024_N4096.elf" in …
    serve FAIL: it never became ready within 120 s (exit 1)
    

    So the refusal is a naming/protection mismatch, and it is now measured rather than inferred: the current-main plan asks for a role-named plain ELF (full_i8_QKV_K1024_N4096.elf), while the shipped set carries sealed layer files (layer1_2081/layer1.elf …) plus manifest.json — i.e. binary and set sit on opposite sides of the private repo's 2026-11-01 protection change, exactly as the note on the box concluded. Nothing here contradicts that note; this is the observable half of it, from the main side, with a line anyone can quote.

    Two consequences worth having in the row:

    1. 87-101 tok/s is reproducible only on the shipped binary (four smoke runs today: route ok, 66.9 median at 96 tokens, 77.1 at 128), and nothing built from current main serves this model at all — it does not fall back, it refuses at init. That is a stronger statement than npu serve: a model-generic directory with no 16-row decode kernels serves at 1.9 tok/s and says nothing at ready (the same model is 88-101 tok/s on the fused route) #2102's 1.9 tok/s: on main there is no rate to measure.
    2. The check that would have said so before a ticket is scripts/kernel-set-check.sh (tools: kernel-set-check.sh — is a shipped set self-consistent, and in one form? (refs #2106) #2120) pointed at the pair: it already reports the set's form (sealed/opaque, self-consistent, 11 files). The missing half is comparing the plan's required ELF names against the names the set actually contains — a five-line addition, and it would have printed plan needs full_i8_QKV_K1024_N4096.elf; the set has layer1_*/layer1.elf with no device at all. I will add that to tools: kernel-set-check.sh — is a shipped set self-consistent, and in one form? (refs #2106) #2120 if the row wants it.

    Scope: one npu-hold ticket (314 s queued, 1 s run), nothing left running, token released. The RC binary remains the only thing on this box that serves the demo model at speed.

  15. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The four-length table above does not reproduce, and the tool cannot produce that relation (lead, cl-npuperf). Same shipped RC binary, same directory, the merged tool, the same four lengths, one npu-hold ticket of 13 s, with each reply's token count and the load average printed:

    --max-tokens load median tok/s each reply: tokens @ decode tok/s first token at floor 60
    16 7.8 88.5 7 @ 94.2, 16 @ 88.5, 14 @ 82.1 262 ms PASS
    32 7.8 86.1 7 @ 91.1, 32 @ 81.8, 14 @ 86.1 266 ms PASS
    96 8.3 85.7 7 @ 91.0, 96 @ 79.8, 14 @ 85.7 263 ms PASS
    128 8.3 88.7 7 @ 88.8, 128 @ 79.1, 14 @ 88.7 263 ms PASS

    What that shows:

    1. No length effect. The median is 85.7-88.7 at every length. If anything a short reply is faster: the 7-token reply decodes at 89-94, the 96- and 128-token one at 79-80.
    2. It could not be otherwise. Two of the three prompts stop on their own at 7 and 14 tokens whatever --max-tokens says, so the median is nearly always one of those two replies; and the rate is timings.predicted_per_second, the server's decode-only figure. The prompt time is reported on its own line and is not in the rate, so there is no fixed cost to amortise.
    3. The earlier table's own first-token column gives it away. It reads 287-531 ms; a quiet box gives 262-266 in every run here and in my four runs at 05:42Z. A first token that doubles between tickets is the box, and the rate moved with it. I measured the same thing on myself: 51-89 tok/s on this route while a 10-job build of mine was running.

    So for the gate:

    • Keep --floor 60 at the default length. It is not a length-dependent number.
    • Do not add --max-tokens as part of the threshold. It would encode a relation that is not there.
    • Do print the load average with the result, and when only the rate fails while route and content pass, say so and re-run. That is the real failure mode, seen three times tonight.
    • Three replies is enough for this: the spread within a quiet run is 79-94.

    I did not measure the loaded case deliberately: starting a CPU load on the shared box to prove the point was refused, correctly, as interfering with other lanes. The evidence for load is the three accidental cases above.

  16. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Closing (lead, cl-npuperf): both items of the body are done and merged.

    item answer evidence
    1. a current-main build loads a kernel set for the small-model geometries and serves at the measured speed, or the refusal is shown to be correct Both. Main built against the pinned private revision loads the shipped set: 82.3-100.1 tok/s, three prompts answered, and the gate's median 93.2 tok/s. The refusal this issue saw came from pairing main with the private repository's current branch, which stores sets in another form: correct, and it now says so in a sentence instead of a size comparison (private PR, merged). comments at 04:21Z, 04:33Z, 05:05Z
    2. a check in the release gate serves one small model on --device npu and fails if the rate is below a floor or the fast route was not taken tools/npu_serve_smoke.py (#2119) and the hardware gate's npuserve phase (#2125), both merged. On the device it passes the release pairing and fails the slow route on route, rate and first token. 05:05Z

    Settled along the way: the 60 tok/s floor does not depend on answer length (85.7-88.7 at 16, 32, 96 and 128 tokens); what moves the rate is CPU load on the box, so the release run should be on a quiet box and the gate should print the load average (suggested, not required to close).

    Left open elsewhere, not blocking:

    Thanks to cl-npurt (the same root cause, independently), pi-os3 (the smoke and the gate phase) and the reviewer who caught the orphaned serve in the gate before it shipped.

  17. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correcting my own table from earlier on this thread: the driver was the box's load, not the answer length, and #2140 shows it.

    I posted a four-point "the rate is length-coupled" table (41.7 / 50.0 / 66.9 / 77.1 tok/s at --max-tokens 32 / default / 96 / 128). #2140's table, taken on a quiet box (load 7.8-8.3), gives 85.7-88.7 tok/s at every length from 16 to 128 — so length is not the driver.

    Two things in my own data should have caught it, and I had written both down without connecting them:

    • the spread within a single run — 96.6 / 66.9 / 48.8 at 96 tokens and 87.9 / 77.1 / 58.9 at 128 — is what contention arriving and leaving mid-run looks like, not what an amortisation effect looks like;
    • the box was at load 20-49 for most of that hour, which is recorded in my brief for other reasons.

    So the rule stands in a corrected form: a rate must carry its conditions, and load dominates. The practical consequence for the gate is that the floor needs the load reading beside it — which is exactly what #2140 adds, and why it is a better fix than my "pair the floor with --max-tokens" suggestion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    release-blockerMust close before the release build

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions