Repository navigation
Description
F5 — a single-turn "flash" serving mode on the NPU route (W8 / FLM 1.0.12 parity)
Owner item F5 of ~/briefs/TNPU-ALL.md §9 (2026-11-09 23:2x "support all of it"), assigned to pi-fnprefill. Definition of done is §1's five
conditions plus a measured row; the rules are §2 (own code only, FLM as reference/model source, private kernels, bit-exact, measured numbers only,
npu-hold, packs on disk).
What "supported" means here: 1bit serve gains a single-turn mode with a fixed 1k context, clean state per request (no KV, recurrent or
scratch carry-over between requests), and the system prompt prefilled once and reused — the shape FLM ships as its flash profile.
Bar (FLM 1.0.12, qwen3.5-flash:2b): TTFT 0.49 s at 1k tokens, 33 tok/s decode. We measure ours before the change and after, same pack, same
hour, under npu-hold; the row names the kernel set and the engine commit.
What already exists, and must not be rebuilt
- Prefix/prompt cache: merged. npu: no multi-prompt prefix cache — every request prefills from scratch (FLM v1.0.11 has 8 slots) #650 ("no multi-prompt prefix cache — every request prefills from scratch") closed on PR npu: multi-prompt prefix cache (#650) — re-landing #738, which merged into its base branch and never reached main #745 →
91eadcde5
(the re-landing of npu: multi-prompt prefix cache, resumes from the longest saved prompt (#650) #738). So "system prompt saved once and reused" has a mechanism in tree; this item is the flash packaging of it (fixed 1k,
clean per-request state, the TTFT/decode contract), not a second cache. 1bit servealready has the single-flight,-c-configured route used by the 1.9/1.8 rows; the question is whether its per-request reset is
complete (KV + recurrent state + scratch), which is the part that has to be demonstrated rather than assumed.
Facts I verified for this issue
- FLM 1.0.12's public release notes (
repos/FastFlowLM/FastFlowLM/releases/latest, tag v1.0.12) listqwen3.5-flash:2bamong the new
models, context "up to 256k tokens (32k by default)", and a block of server fixes — including a server that stopped accepting connections after
clients hung up and "Qwen3.5 answering//////after hitting the context limit". That last one is directly adjacent to a fixed 1k mode: the
context-limit behaviour is part of what a flash profile must get right. - The fact base named in the brief is not on disk yet:
~/amd-oss/npu-re/docs/FACTS-fastflowlm-v1012-delta.mddoes not exist (the directory has
FACTS-fastflowlm-v107-artifacts.md,FACTS-fastflowlm-head-prefill-moe.md, and others). So the numbers above are the release notes' public claims
only; the pi-verify digest still has to land before any vendor-side mechanism is described here.
Plan, in the order the rules require
- Before-row, measured.
qwen3.5-flash:2bpack (FastFlowLM Q4NX on disk) on--device npuundernpu-hold: TTFT at 1k prompt tokens and
decode tok/s, 3 identical requests returning identical bytes (rules §2). If the pack does not import, that is the first gap and the row says so. - State-reset proof before the feature. Same request twice in one process with a different system prompt between them: identical output for
identical inputs, and no leakage of the first prompt's state. That is the "clean state per request" claim, and it is testable without the mode. - The mode itself:
--flash(name to be settled with the CLI owner) or a documented flag pair = fixed 1k ctx + reset-per-request + system
prompt prefilled once. Reuse npu: multi-prompt prefix cache (#650) — re-landing #738, which merged into its base branch and never reached main #745's cache; do not add a second one. - After-row, same pack/hour/kernel set, then the comparison against the 0.49 s / 33 tok/s bar.
Open questions for the owner
- Is
qwen3.5-flash:2balready in our pack scope (i.e. is its Q4NX repo in the F1 list and pulled to disk), or does F1 have to land it first? F5's
bar is stated on that model, so the row depends on the pack existing. - Is the flag surface mine to choose, or is there a
serveCLI convention (pi-tmplsec owns F10's server behaviours) it must match?
Refs: §9 row F5; #650 / PR #745 (91eadcde5) for the cache this must reuse.
Activity
bong-water-water-bong commentedon Nov 10, 2026
Before-row precondition: what
--device npuactually needs, measured on three packs in ten minutesAttempted the before-row today and hit a chain of prerequisites that is worth writing down, because each one costs a
npu-holdticket+queue wait todiscover and none of them is specific to the flash mode:
oga/SmolLM2-135M-Instruct_rai_1.9.0_npuis not an NPU model directory (model.q4nx, config.json, tokenizer.json)— the AMD oga bundle is the wrong shape forservenpu/Qwen3-0.6B(proper container)is a model-generic Q4NX container … which the per-op forward serves, but **no NPU kernel (full ELF) directory resolves for it**: set ONEBIT_NPU_KERNELS to a directory of full_i8_*.elf designs … or put one in <model>/npuscratch-packs/imp1519/npu/full_i8_QKV_K1536_N2048_b8*.elf, andmodels/npu/bonsai2-*/kernels-m16for the bonsai2 familySo the precondition chain for any NPU before-row is: a pack in container shape (
model.q4nx+config.json+tokenizer.json), plus akernel directory whose
full_i8_*ELFs match that model's geometry.Qwen3-0.6Bhas the container and no matching kernels; thebonsai2-*packs haveboth. This is the same prerequisite F1's rows carry (import with zero gaps → kernels exist → serve), which is why F5's bar on
qwen3.5-flash:2balsoneeds that model's kernels built, not just its pack downloaded.
What I will do next rather than guess: run the reset probe against a pack that is known servable (
bonsai2-*with itskernels-m16viaONEBIT_NPU_KERNELS), because the state-reset proof is model-agnostic — it asks whether KV/recurrent/scratch state leaks between requests, and anyservable 1.9 pack answers that. The bar's row stays blocked until
qwen3.5-flash:2b(pack and kernels) exists, which is a different lane'sprerequisite.
Probe:
~/lb/f5-reset-probe.sh <pack-dir> <port> <ctx>— sends A → B → A2 → B2 with B differing only in the system prompt, and assertsA1==A2,B1==B2, and cruciallyA1!=B1(a server that ignores the system prompt would pass both equality checks and print as INCONCLUSIVE, notPASS). Two runs so far produced exit 3 from the precondition check above, with zero tokens generated and nothing left running.
bong-water-water-bong commentedon Nov 10, 2026
First measured result: state is clean and the prefix cache never engages —
cached_tokens: 0, "no slot shares a prefix"on all four requestsRan the state-reset probe on a known-servable pack (
bonsai2-t2s2-perrole876, itskernels-m16viaONEBIT_NPU_KERNELS,npu-hold -t 10, ctx1024, port 18262). Two findings, one of them the thing F5 exists for.
1. State reset holds (the good half)
'\n\nParis''''\n\nParis'''A1 == A2exactly andB1 == B2exactly, after a differently-prompted request in between — so no KV/recurrent/scratch state leaks between requests onthis route. That is F5's "clean state per request" established before any flash-mode work, which is what I wanted from step 2.
Caveat on the third assertion, stated plainly: the probe's
A1 != B1check passed only because B is empty, and B's emptiness is a decode outcome(
finish_reason: lengthafter 24 completion tokens with no visible text), not evidence about the system prompt. So "the system prompt is honoured" isnot established by this run — it is unproven, and the honest reading is "identical inputs give identical bytes; the differently-prompted arm produced
nothing to compare".
2. The prefix cache never engages — this is the F5 gap, measured
Every response carries the server's own accounting, and all four say the same thing:
That includes A2, which repeats A1's prompt byte for byte. So the mechanism merged for #650 (PR #745 →
91eadcde5) is not being used on this route:"the system prompt saved once and reused" is currently not happening at all, and no amount of flash-mode packaging will produce TTFT 0.49 s while the
cache reports zero cached tokens on an identical repeat.
That reframes the item: F5 is not "add a flash mode", it is "why does the prefix cache never match, and make the fixed-1k single-turn path actually hit it".
The next measurement is the discriminating one: the same two identical requests with
--parallel 1vs--parallel 2(slot count is the obvious variablefor
slot: -1), and the same pair on a pack whose prompt exceeds one slot's capacity at 1k ctx.Timing note for the record: ~36 s per 24-token request on this pack, which makes it a semantics instrument, not a performance one — the TTFT/decode row
still needs
qwen3.5-flash:2bwith its kernels.bong-water-water-bong commentedon Nov 10, 2026
Correcting my own previous comment: the cache is not broken — the insert gate refused a 36-token prompt by design, and the real F5 gap is that
servecannot tune itI wrote above that "the prefix cache never engages — this is the F5 gap". That was wrong, and the code says why.
app/npu_prefix_cache.h:lookup()is a straight prefix match, socached_tokens: 0 / "no slot shares a prefix"on an identical repeat means the slot was never inserted — andwith
min_gain = 128, my probe's prompts (36 and 39 tokens) were refused at the gate:gain 36 < 128. Nothing was stored, so the repeat hadnothing to match. The cache is enabled (
slots = 8 > 0) and working as designed on the numbers it was given.So the F5-relevant conclusion changes, and it is more useful than the wrong one:
min_gain = 128, so the discriminatingmeasurement is one repeat of a >128-token prompt (ideally the flash mode's own 1k system prompt + a short user turn), expecting
cached_tokens > 0. That is one ticket, not an investigation.--prefix-slots,--prefix-cache-mband--prefix-min-gainare parsed inapp/forward_serve.cpp:808-810only.app/serve.cpphas a single mention of "prefix" (a comment about--cache-reuse 256for the llama-serverproxy) — so on the
--device npuserve route the cache runs at defaults and cannot be tuned, and the flash mode's contract ("system prompt savedonce and reused") has no way to pin a specific prompt or to lower the threshold for a short one. That is the piece F5 actually has to build: an
explicit "keep this system prompt" path (and/or pass the prefix options through from
serve), not a new cache.Correcting the record matters here more than the original claim did: "the cache never engages" would have sent someone to debug a working mechanism, while
"serve cannot configure it and there is no way to pin a system prompt" is the actual work item.
bong-water-water-bong commentedon Nov 10, 2026
Measured: the prefix cache does hit —
cached_tokens: 0 → 355on an identical 356-token prompt~/lb/f5-cache-probe.sh,bonsai2-t2s2-perrole876+kernels-m16,--device npu --ctx-size 1024, port 18263,npu-hold -t 10, one server, twoidentical requests with a 347-token system prompt (1390 chars, 14 rules — verified above
min_gain = 128before launch):So both halves of F5's mechanism are now measured on real hardware, and both hold:
A1 == A2andB1 == B2exactly, with a differently-prompted request between themcached_tokens 0 → 355of 356,reason: 'hit'And it closes the correction I posted an hour ago: my first probe's 36/39-token prompts were refused by
min_gain = 128— the gate working as designed,not a broken cache.
355/356is the number to quote: the cache reuses everything except the one token the request itself adds.What that leaves as F5's actual work, and it is now small and specific:
serveroute —--prefix-slots/--prefix-cache-mb/--prefix-min-gainare parsed inforward_serve.cpp:808-810only, and
app/serve.cppmentions "prefix" once (a--cache-reuse 256comment). Without that, the flash mode cannot tune the gate or the budget.cache does not offer, since entries age out by budget and are replaced by prompts that cover more tokens (
insert()'sextendedrule).qwen3.5-flash:2b, which still needs that pack and its kernels on disk (unchanged prerequisite fromthe earlier comment;
bonsai2-t2s2-perrole876is a semantics instrument at ~36 s per 24 tokens, not a speed one).Note on the text column: both responses returned empty text, so this run says nothing about answer quality — the two claims above are about state and
cache accounting, which is what they were designed to measure.
bong-water-water-bong commentedon Nov 10, 2026
The
serveknobs are in and verified as far as a non-NPU build can verify themCommitted locally as
2c410101b(WIP, not pushed — it needs the behavioural half before it deserves a PR):app/serve.cppnow carries--prefix-slots,--prefix-cache-mb,--prefix-min-gain, in the same style the file already uses for--npu-target("serve only carries theoperator's choice through, so there is one place that says which names exist"). Each defaults to −1 = unset, so
app/npu_prefix_cache.hstays thesingle source of truth for what "unset" means (8 slots / 4096 MB / min-gain 128).
What is verified, on a real build (
-DONEBIT_NPU=OFF,cmake --build --target onebit -j4underthermal-run.sh, 11 link steps,rc=0):fargspass-through.1bit serve --helpprints[--prefix-slots N] [--prefix-cache-mb N] [--prefix-min-gain N] the prompt prefix cache.serve --prefix-min-gain 16 --prefix-slots 4 --model /nonexistentreaches themodel check (
expected an NPU model directory, a .gguf file or a 1BP package) instead of an unknown-option error.What is NOT verified, and must not be claimed from the above: that the values actually arrive in
forward-serve. That needs an NPU-enabled build —and the configure for one refuses without the private runtime, correctly:
The value a working build on this box used is now known, from
~/wt/q1416/build/CMakeCache.txt:ONEBIT_NPU_RT_DIR=/home/bcloud/wt/release-rc/work/npu-runtime/engine/npu/npu_rt(verified present,npu_rt.hincluded). So the remaining sequence is oneconfigure + one run:
The probe prints the server's own
usage.flm_prefix_cache, so a wrong pass-through shows up as the old refusal at the old length — the test cannotpass by accident.
bong-water-water-bong commentedon Nov 10, 2026
Behavioural proof:
--prefix-min-gain 16reaches the forward and changes the cache's decision (cached_tokens 0 → 32)The wiring is now verified end to end on hardware, and the test could only pass by demonstrating the flag crossed the process boundary:
cached=0, 'no slot shares a prefix'cached=355 of 356, 'hit''gain below prefix-min-gain'--prefix-min-gain 16cached=0cached=32 of 33, 'hit'Same pack (
bonsai2-t2s2-perrole876+kernels-m16), ctx 1024,npu-hold -t 10, one server, two identical requests, reading the server's ownusage.flm_prefix_cache. The 33-token prompt is below the defaultmin_gain = 128and was refused before this change; with the flag it is inserted andthe repeat hits. The probe prints
serve-extra=--prefix-min-gain 16and the prompt length in its own log line, so the artifact carries the flags thatproduced it.
What that establishes, and it is three claims at once: the flag is parsed by
serve; it is forwarded into the forward's argv; and it changes thecache's decision on a prompt the default gate refuses. A broken pass-through would have reproduced the old refusal at the old length — the asymmetry is
why this was worth a ticket rather than an assertion.
Build:
-DONEBIT_NPU=ONwithONEBIT_NPU_RT_DIR=~/wt/release-rc/work/npu-runtime/engine/npu/npu_rt(the value the box's working build used),--target onebit -j4underthermal-run.sh; the non-NPU configuration compiles the same change as well. Commit is local (5f9404ba3, WIP on the lanebranch, not pushed — kernel/runtime material stays private and this needs its PR written against a clean base).
F5's remaining work is now only the part that was always the item: the flash profile itself — fixed 1k context + a pinned system prompt, which the
cache does not offer (entries age out by
cache_mbbudget and are replaced by prompts covering more tokens), plus the TTFT/decode row against 0.49 s /33 tok/s, which still needs
qwen3.5-flash:2band its kernels on disk.npu: serve carries the prefix cache's knobs (engine #1785, F5 flash s…
bong-water-water-bong commentedon Nov 10, 2026
Taking the flash-profile half (cl-flm): no push or comment here since 05:26Z and the owning lane's state file lists this as blocked on the pack. First slice is PR #1941.
A finding that changes what "system prompt reused" needs. The measured hits on this issue were on an identical prompt (
cached_tokens 0 → 355). A flash request repeats the system prompt and changes the user turn, and for that shape the table reuses nothing, because a slot holds a whole prompt and a hit needs the slot to be a prefix of the request:That is a row in #1941's test, run on main's table. No
--prefix-*setting changes it.#1941 adds the slot kind for the shared part (
pin_point,pin; token ids only, so no template knowledge), with tests and mutants. It is the table only.What remains, in order:
forward-servekeeps the previous request's ids, and whenpin_point()is non-zero cuts the prefill one token short of it, snapshots and pins. Needs an NPU build and a device run; the box is under the 175 G build floor (137 G free at 14:50Z);--flashonserve=--ctx-size 1024plus the pin on;bong-water-water-bong commentedon Nov 10, 2026
Forward half is up as #1982, measured on the NPU. With
--prefix-pin, the third question under one system prompt resumed from 341 of 355 tokens (pinned slot; whole call 23.4 s against 61.5 s with the cache off), and all four answers were identical to the cache-off arm. The default cache on the same requests reused nothing on request 3. Pack:bonsai2-t2s2-perrole876+kernels-m16, ctx 1024, onenpu-holdticket; times are whole HTTP calls with 16 generated tokens, one run per arm, not TTFT.Left:
--flashas a profile (ctx 1024 + the pin), and the row against 0.49 s / 33 tok/s onqwen3.5-flash:2b, which still needs that pack and its kernels.tests: pin_point arms for a shared ending and the common-prefix bound…
serve: --flash, the single-turn NPU profile; --ctx-size reaches the f…
forward-serve: --prefix-pin, the pinned shared prefix on the device p…
serve: --flash, the single-turn NPU profile; --ctx-size reaches the f…
tests: serve --flash refuses what contradicts the profile, with no mo…
bong-water-water-bong commentedon Nov 10, 2026
(cl-flm) #1982 (
--prefix-pinon the device path) merged at 21:04Z. The third slice is up as #2029:serve --flash(=--ctx-size 1024+--prefix-pin, refusing what contradicts it),--ctx-sizenow actually reaching the forward route, and an over-context request answered 400 instead of 500. Measured on the NPU with the flash profile: third question under a shared 341-token system prompt resumes with 341 of 355 tokens cached, 13.9 s against 56.2 s for the first.What remains after #2029: the bar row on
qwen3.5-flash:2b, which needs that model's pack and kernels on the box.15 remaining items
forward-serve: the tokenizer's eos_token and the ChatML turn end are …
bong-water-water-bong commentedon Nov 11, 2026
Current state, measured on
maintoday: the pin shipped (#2085 merged) and it is unreachable for a short system turn under the default gateTwo facts from
origin/mainafter #2085 merged at9fc1e0a:app/npu_prefix_cache.h:200haspin_at(ids, n)pluspin(prefix, bytes)(one pinned slot, untouched by the budget's eviction andby a longer prompt's replacement), and
forward-serve --prefix-pinrenders the system turn to find where it ends. That is the piece this issue was missing,and the
usage.flm_prefix_cacheaccounting I measured (355/356, and32/33at--prefix-min-gain 16) is what made it verifiable.pin_at()still gates onprefix-min-gain, and the default is unchanged:So a system turn of fewer than 128 tokens returns 0 — no pin — and nothing in the response distinguishes that from "pinned". Most chat system turns are 20–60
tokens, which means
--prefix-pinis a no-op for the ordinary case the flash profile exists to serve. I raised this on #2085 before it merged (the review asked foreither a separate floor for
pin_ator a documented interaction, and for a ~40-token test case alongside the 900-token one); the merged version kept the gate andthe test still uses a 900-token pin, which is comfortably above it.
Why this matters for the bar rather than being a nit: F5's target is TTFT 0.49 s at 1k tokens on
qwen3.5-flash:2b. With a short system turn the first requestprefills, the pin does not take, and the second request prefills again — i.e. the mode looks enabled and behaves cold. That is the same failure I initially
misread in my own probe (
cached_tokens: 0, 'gain below prefix-min-gain'on 36 tokens read as "the cache never engages").Fix shape, either is small:
pin_at()its own floor (--prefix-pin-min, default 0): an explicit request about a point the caller already knows is a different job frommin_gain's speculative retention heuristic, and conflating them makes the explicit request silently optional;min_gainitself (it knows it is pinning a system turn, so the gain heuristic is the wrong test), and say so in--help.Not claimed: I have not measured the 2B case —
qwen3.5-flash:2band its kernels are still not on disk. This is a reading of the merged code plus the twomeasurements I took earlier on a servable pack, and it is stated as exactly that.
bong-water-water-bong commentedon Nov 11, 2026
Correcting my previous comment: the
min_gaingate onpin_at()is deliberate and tested, not a defect — the gap is in the flash profileI posted above that
--prefix-pinis "unreachable for a short system turn" and offered to givepin_at()its own floor. That was wrong, and I found out by tryingit: I added a
pin_min_gain(default 0), wired--prefix-pin-min-gain, added a 40-token case — and the existing suite failed seven assertions:Reverted, unlanded. The docstring says it plainly, and I had read past it:
So "is this opening worth a KV snapshot" is a policy, applied uniformly to speculative retention and to explicit pins, with
pin_at(q1, 127) == 0 && pin_at(q1, 128) == 128asserted as the boundary. Sharing the knob is the design, not an accident.What that means for this issue, correctly stated: F5's "system prompt saved once and reused" does not need a cache change at all. It needs the flash profile to
set
min_gainto what its own system turn is worth — which is already possible (--prefix-min-gain, onforward-servesince #650, and onservesince my #1812) —and to say so where the profile is defined. A 40-token system turn is then pinned by configuring the threshold, not by changing what "worth pinning" means.
Why the correction matters more than the original claim: landing my patch would have broken a documented, tested policy to solve a problem that configuration
already solves. The suite caught it in one run, which is exactly what the boundary assertion is for — and it is the third time tonight that a defect I "found by
reading" turned out to be a design decision (the others: the cache's
min_gainrefusal, and #1830's role-keyed cache).npu: the dense-tail cache is on by default from a 128 MiB tail, the s…
bong-water-water-bong commentedon Nov 11, 2026
cl-1077 took a small docs and help slice, separate from the speed work on this issue: #2152 (2c6ea21). It does what the 03:04Z comment concluded: the
--flashparagraph andserve --helpnow say that a system turn shorter than--prefix-min-gainis not pinned, and how to set the threshold. It also moves a--npu-streamshelp line that had drifted under--flashback where it belongs. No cache code changes.serve --help: --npu-streams' default back under it; --flash says a sh…
bong-water-water-bong commentedon Nov 11, 2026
cl-npurt (lead on this row per ISSUE-OWNERS 12:00Z). Slice I am taking: the lm head during decode, on the host. No one has it and no open PR touches it.
Why, from numbers already on the box plus one new one:
Qwen3.5-2B-NPU2(q35-prof-235433): one decode token is 135 launches; the layers are about 233 ms and the head is 21 launches x 3.1 ms = 65 ms, of a 300 to 360 ms token. A launch costs the same for 1 row as for 128 (prefill rows in the same log: 3 to 4 ms per 128-row launch), so one-row decode pays the full weight pass per launch.-O3 -march=native, under thermal-run at load 11: 6.6 ms with 4 threads, 5.9 ms with 16.So a host head would take roughly 58 ms off each decode token on this model (about 3.3 to 4.0 tok/s), and the sums are integers, so the logits can be required byte-identical to the device head's (
--dump-logits, the harness used for #803's gate today). It does not close the gap to 33 tok/s: the layer launches are the rest, and that is kernel work outside this slice. Next from me: the PR with the before/after and the byte comparison.npu: NPU_HEAD_HOST=alt alternates host and device head per token and …
bong-water-water-bong commentedon Nov 11, 2026
cl-npurt (lead). Closing on the measured ceiling, per the orchestrator's decision today.
What the issue asked for and where each part is
serve --flashon main (#2029)--flash: same bytes before and after a different system prompt (cl-flm, 11-10 22:58Z)The host/device split of one decode token (
Qwen3.5-2B-NPU2,NPU_PHASE_PROF=1, medians over 46 tokens per arm): about 275 ms, of which 241 ms is device wait across 114 launches and about 30 ms is host (12 ms launch overhead, 11 ms layer-loop host work, 7-9 ms lm head).That 30 ms is after the two host-side PRs from this lane, both with byte-identical logits against the device path:
NPU_HEAD_HOST=1), 51-78 ms to 8-18 ms per token; net 52 ms (13 %) per token measured inside one serve. It stays opt-in until the window11 row-set comparison accepts itsHEAD M=1profile row.load_layer12-13 ms to 0.04 ms per token.Both are open (READY on #2191; #2201 has a red
buildfrom the CI runners' DNS outage, not from the diff).The bar is 30 ms per token in total, so the host side alone already equals it and the remaining 241 ms cannot be reached from the serve or the host. That part is now #2208 (label
perf): the per-token launch count and the cost of a one-row launch.npu: a hybrid model's full-attention layers no longer re-dequantise q…
npu: the decode lm head on the host, bit-identical logits — head time…
npu: four #1785/#803/#799 follow-ups in one PR — window11 accepts the…
npu: the decode lm head runs on the host by default; NPU_HEAD_HOST=0 …