Skip to content

npu: F5 — a single-turn flash serving mode (fixed 1k ctx, clean state per request, system prompt reused), measured against 0.49 s TTFT / 33 tok/s #1785

Closed

Description

@bong-water-water-bong

F5 — a single-turn "flash" serving mode on the NPU route (W8 / FLM 1.0.12 parity)

Owner item F5 of ~/briefs/TNPU-ALL.md §9 (2026-11-09 23:2x "support all of it"), assigned to pi-fnprefill. Definition of done is §1's five
conditions plus a measured row; the rules are §2 (own code only, FLM as reference/model source, private kernels, bit-exact, measured numbers only,
npu-hold, packs on disk).

What "supported" means here: 1bit serve gains a single-turn mode with a fixed 1k context, clean state per request (no KV, recurrent or
scratch carry-over between requests), and the system prompt prefilled once and reused — the shape FLM ships as its flash profile.

Bar (FLM 1.0.12, qwen3.5-flash:2b): TTFT 0.49 s at 1k tokens, 33 tok/s decode. We measure ours before the change and after, same pack, same
hour, under npu-hold; the row names the kernel set and the engine commit.

What already exists, and must not be rebuilt

Facts I verified for this issue

  • FLM 1.0.12's public release notes (repos/FastFlowLM/FastFlowLM/releases/latest, tag v1.0.12) list qwen3.5-flash:2b among the new
    models, context "up to 256k tokens (32k by default)", and a block of server fixes — including a server that stopped accepting connections after
    clients hung up and "Qwen3.5 answering ////// after hitting the context limit". That last one is directly adjacent to a fixed 1k mode: the
    context-limit behaviour is part of what a flash profile must get right.
  • The fact base named in the brief is not on disk yet: ~/amd-oss/npu-re/docs/FACTS-fastflowlm-v1012-delta.md does not exist (the directory has
    FACTS-fastflowlm-v107-artifacts.md, FACTS-fastflowlm-head-prefill-moe.md, and others). So the numbers above are the release notes' public claims
    only; the pi-verify digest still has to land before any vendor-side mechanism is described here.

Plan, in the order the rules require

  1. Before-row, measured. qwen3.5-flash:2b pack (FastFlowLM Q4NX on disk) on --device npu under npu-hold: TTFT at 1k prompt tokens and
    decode tok/s, 3 identical requests returning identical bytes (rules §2). If the pack does not import, that is the first gap and the row says so.
  2. State-reset proof before the feature. Same request twice in one process with a different system prompt between them: identical output for
    identical inputs, and no leakage of the first prompt's state. That is the "clean state per request" claim, and it is testable without the mode.
  3. The mode itself: --flash (name to be settled with the CLI owner) or a documented flag pair = fixed 1k ctx + reset-per-request + system
    prompt prefilled once. Reuse npu: multi-prompt prefix cache (#650) — re-landing #738, which merged into its base branch and never reached main #745's cache; do not add a second one.
  4. After-row, same pack/hour/kernel set, then the comparison against the 0.49 s / 33 tok/s bar.

Open questions for the owner

  • Is qwen3.5-flash:2b already in our pack scope (i.e. is its Q4NX repo in the F1 list and pulled to disk), or does F1 have to land it first? F5's
    bar is stated on that model, so the row depends on the pack existing.
  • Is the flag surface mine to choose, or is there a serve CLI convention (pi-tmplsec owns F10's server behaviours) it must match?

Refs: §9 row F5; #650 / PR #745 (91eadcde5) for the cache this must reuse.

Activity

  1. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Before-row precondition: what --device npu actually needs, measured on three packs in ten minutes

    Attempted the before-row today and hit a chain of prerequisites that is worth writing down, because each one costs a npu-hold ticket+queue wait to
    discover and none of them is specific to the flash mode:

    attempt pack result
    1 oga/SmolLM2-135M-Instruct_rai_1.9.0_npu is not an NPU model directory (model.q4nx, config.json, tokenizer.json) — the AMD oga bundle is the wrong shape for serve
    2 npu/Qwen3-0.6B (proper container) is a model-generic Q4NX container … which the per-op forward serves, but **no NPU kernel (full ELF) directory resolves for it**: set ONEBIT_NPU_KERNELS to a directory of full_i8_*.elf designs … or put one in <model>/npu
    3 searched for kernel dirs the dirs that exist are per-model geometry: scratch-packs/imp1519/npu/full_i8_QKV_K1536_N2048_b8*.elf, and models/npu/bonsai2-*/kernels-m16 for the bonsai2 family

    So the precondition chain for any NPU before-row is: a pack in container shape (model.q4nx + config.json + tokenizer.json), plus a
    kernel directory whose full_i8_* ELFs match that model's geometry. Qwen3-0.6B has the container and no matching kernels; the bonsai2-* packs have
    both. This is the same prerequisite F1's rows carry (import with zero gaps → kernels exist → serve), which is why F5's bar on qwen3.5-flash:2b also
    needs that model's kernels built, not just its pack downloaded.

    What I will do next rather than guess: run the reset probe against a pack that is known servable (bonsai2-* with its kernels-m16 via
    ONEBIT_NPU_KERNELS), because the state-reset proof is model-agnostic — it asks whether KV/recurrent/scratch state leaks between requests, and any
    servable 1.9 pack answers that. The bar's row stays blocked until qwen3.5-flash:2b (pack and kernels) exists, which is a different lane's
    prerequisite.

    Probe: ~/lb/f5-reset-probe.sh <pack-dir> <port> <ctx> — sends A → B → A2 → B2 with B differing only in the system prompt, and asserts
    A1==A2, B1==B2, and crucially A1!=B1 (a server that ignores the system prompt would pass both equality checks and print as INCONCLUSIVE, not
    PASS). Two runs so far produced exit 3 from the precondition check above, with zero tokens generated and nothing left running.

  2. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    First measured result: state is clean and the prefix cache never engages — cached_tokens: 0, "no slot shares a prefix" on all four requests

    Ran the state-reset probe on a known-servable pack (bonsai2-t2s2-perrole876, its kernels-m16 via ONEBIT_NPU_KERNELS, npu-hold -t 10, ctx
    1024, port 18262). Two findings, one of them the thing F5 exists for.

    1. State reset holds (the good half)

    req system prompt prompt tok completion finish text
    A1 terse assistant 36 23 stop '\n\nParis'
    B1 pirate 39 24 length ''
    A2 terse assistant 36 23 stop '\n\nParis'
    B2 pirate 39 24 length ''

    A1 == A2 exactly and B1 == B2 exactly, after a differently-prompted request in between — so no KV/recurrent/scratch state leaks between requests on
    this route. That is F5's "clean state per request" established before any flash-mode work, which is what I wanted from step 2.

    Caveat on the third assertion, stated plainly: the probe's A1 != B1 check passed only because B is empty, and B's emptiness is a decode outcome
    (finish_reason: length after 24 completion tokens with no visible text), not evidence about the system prompt. So "the system prompt is honoured" is
    not established by this run — it is unproven, and the honest reading is "identical inputs give identical bytes; the differently-prompted arm produced
    nothing to compare".

    2. The prefix cache never engages — this is the F5 gap, measured

    Every response carries the server's own accounting, and all four say the same thing:

    cached_tokens: 0   reason: 'no slot shares a prefix'   slot: -1
    

    That includes A2, which repeats A1's prompt byte for byte. So the mechanism merged for #650 (PR #745 → 91eadcde5) is not being used on this route:
    "the system prompt saved once and reused" is currently not happening at all, and no amount of flash-mode packaging will produce TTFT 0.49 s while the
    cache reports zero cached tokens on an identical repeat.

    That reframes the item: F5 is not "add a flash mode", it is "why does the prefix cache never match, and make the fixed-1k single-turn path actually hit it".
    The next measurement is the discriminating one: the same two identical requests with --parallel 1 vs --parallel 2 (slot count is the obvious variable
    for slot: -1), and the same pair on a pack whose prompt exceeds one slot's capacity at 1k ctx.

    Timing note for the record: ~36 s per 24-token request on this pack, which makes it a semantics instrument, not a performance one — the TTFT/decode row
    still needs qwen3.5-flash:2b with its kernels.

  3. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correcting my own previous comment: the cache is not broken — the insert gate refused a 36-token prompt by design, and the real F5 gap is that serve cannot tune it

    I wrote above that "the prefix cache never engages — this is the F5 gap". That was wrong, and the code says why. app/npu_prefix_cache.h:

    struct PrefixOptions {
        int slots = 8;            // 0 = off
        size_t cache_mb = 4096;
        int min_gain = 128;       // tokens a new prompt must add over the slot it would extend
    };
    ...
    const size_t gain = ids.size() > covered ? ids.size() - covered : 0;
    if (gain < size_t(opt_.min_gain)) return refuse("gain below prefix-min-gain");

    lookup() is a straight prefix match, so cached_tokens: 0 / "no slot shares a prefix" on an identical repeat means the slot was never inserted — and
    with min_gain = 128, my probe's prompts (36 and 39 tokens) were refused at the gate: gain 36 < 128. Nothing was stored, so the repeat had
    nothing to match. The cache is enabled (slots = 8 > 0) and working as designed on the numbers it was given.

    So the F5-relevant conclusion changes, and it is more useful than the wrong one:

    1. The reset half stands unchanged — A1 == A2 and B1 == B2 exactly, with a differently-prompted request in between.
    2. The cache question is now a threshold question, not a defect: F5's system prompt is ~1k tokens, well above min_gain = 128, so the discriminating
      measurement is one repeat of a >128-token prompt (ideally the flash mode's own 1k system prompt + a short user turn), expecting
      cached_tokens > 0. That is one ticket, not an investigation.
    3. The real gap for F5, found by reading instead of measuring: --prefix-slots, --prefix-cache-mb and --prefix-min-gain are parsed in
      app/forward_serve.cpp:808-810 only. app/serve.cpp has a single mention of "prefix" (a comment about --cache-reuse 256 for the llama-server
      proxy) — so on the --device npu serve route the cache runs at defaults and cannot be tuned, and the flash mode's contract ("system prompt saved
      once and reused") has no way to pin a specific prompt or to lower the threshold for a short one. That is the piece F5 actually has to build: an
      explicit "keep this system prompt" path (and/or pass the prefix options through from serve), not a new cache.

    Correcting the record matters here more than the original claim did: "the cache never engages" would have sent someone to debug a working mechanism, while
    "serve cannot configure it and there is no way to pin a system prompt" is the actual work item.

  4. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Measured: the prefix cache does hit — cached_tokens: 0 → 355 on an identical 356-token prompt

    ~/lb/f5-cache-probe.sh, bonsai2-t2s2-perrole876 + kernels-m16, --device npu --ctx-size 1024, port 18263, npu-hold -t 10, one server, two
    identical requests with a 347-token system prompt (1390 chars, 14 rules — verified above min_gain = 128 before launch):

    L1: prompt=356  cached=0    reason='no slot shares a prefix'   text=''
    L2: prompt=356  cached=355  reason='hit'                       text=''
    PASS: the identical long prompt hits the cache (cached_tokens 0 -> 355)
    

    So both halves of F5's mechanism are now measured on real hardware, and both hold:

    claim evidence
    clean state per request run 3: A1 == A2 and B1 == B2 exactly, with a differently-prompted request between them
    system prompt saved once, reused this run: identical prompt, cached_tokens 0 → 355 of 356, reason: 'hit'

    And it closes the correction I posted an hour ago: my first probe's 36/39-token prompts were refused by min_gain = 128 — the gate working as designed,
    not a broken cache. 355/356 is the number to quote: the cache reuses everything except the one token the request itself adds.

    What that leaves as F5's actual work, and it is now small and specific:

    1. Expose the prefix options on the serve route — --prefix-slots / --prefix-cache-mb / --prefix-min-gain are parsed in forward_serve.cpp:808-810
      only, and app/serve.cpp mentions "prefix" once (a --cache-reuse 256 comment). Without that, the flash mode cannot tune the gate or the budget.
    2. The flash packaging itself: fixed 1k context, per-request reset (already proven), and a system prompt pinned once — a pin being the one thing the
      cache does not offer, since entries age out by budget and are replaced by prompts that cover more tokens (insert()'s extended rule).
    3. Then the bar — TTFT 0.49 s / 33 tok/s on qwen3.5-flash:2b, which still needs that pack and its kernels on disk (unchanged prerequisite from
      the earlier comment; bonsai2-t2s2-perrole876 is a semantics instrument at ~36 s per 24 tokens, not a speed one).

    Note on the text column: both responses returned empty text, so this run says nothing about answer quality — the two claims above are about state and
    cache accounting, which is what they were designed to measure.

  5. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The serve knobs are in and verified as far as a non-NPU build can verify them

    Committed locally as 2c410101b (WIP, not pushed — it needs the behavioural half before it deserves a PR): app/serve.cpp now carries
    --prefix-slots, --prefix-cache-mb, --prefix-min-gain, in the same style the file already uses for --npu-target ("serve only carries the
    operator's choice through, so there is one place that says which names exist"). Each defaults to −1 = unset, so app/npu_prefix_cache.h stays the
    single source of truth for what "unset" means (8 slots / 4096 MB / min-gain 128).

    What is verified, on a real build (-DONEBIT_NPU=OFF, cmake --build --target onebit -j4 under thermal-run.sh, 11 link steps, rc=0):

    1. It compiles — the struct, the parsing branch, the usage text and the fargs pass-through.
    2. The flags are in the interface: 1bit serve --help prints
      [--prefix-slots N] [--prefix-cache-mb N] [--prefix-min-gain N] the prompt prefix cache.
    3. The parser consumes them rather than rejecting them as unknown: serve --prefix-min-gain 16 --prefix-slots 4 --model /nonexistent reaches the
      model check (expected an NPU model directory, a .gguf file or a 1BP package) instead of an unknown-option error.

    What is NOT verified, and must not be claimed from the above: that the values actually arrive in forward-serve. That needs an NPU-enabled build —
    and the configure for one refuses without the private runtime, correctly:

    CMake Error at CMakeLists.txt:1369: ONEBIT_NPU needs the NPU runtime: … (ONEBIT_NPU_RT_DIR=''). The runtime is private.
    

    The value a working build on this box used is now known, from ~/wt/q1416/build/CMakeCache.txt:
    ONEBIT_NPU_RT_DIR=/home/bcloud/wt/release-rc/work/npu-runtime/engine/npu/npu_rt (verified present, npu_rt.h included). So the remaining sequence is one
    configure + one run:

    cmake -B build-f5 -DONEBIT_NPU=ON -DONEBIT_NPU_RT_DIR=/home/bcloud/wt/release-rc/work/npu-runtime/engine/npu/npu_rt …
    cmake --build build-f5 --target onebit -- -j4        # under thermal-run.sh
    OUT=… ./f5-cache-probe.sh <pack> 18263 1024 --prefix-min-gain 16   # expect: 'hit' at 36 tokens instead of 'gain below prefix-min-gain'
    

    The probe prints the server's own usage.flm_prefix_cache, so a wrong pass-through shows up as the old refusal at the old length — the test cannot
    pass by accident.

  6. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Behavioural proof: --prefix-min-gain 16 reaches the forward and changes the cache's decision (cached_tokens 0 → 32)

    The wiring is now verified end to end on hardware, and the test could only pass by demonstrating the flag crossed the process boundary:

    run prompt serve flags L1 L2
    earlier, default gate 347 tok — cached=0, 'no slot shares a prefix' cached=355 of 356, 'hit'
    earlier, default gate 36 tok — — refused, 'gain below prefix-min-gain'
    this run 33 tok --prefix-min-gain 16 cached=0 cached=32 of 33, 'hit'

    Same pack (bonsai2-t2s2-perrole876 + kernels-m16), ctx 1024, npu-hold -t 10, one server, two identical requests, reading the server's own
    usage.flm_prefix_cache. The 33-token prompt is below the default min_gain = 128 and was refused before this change; with the flag it is inserted and
    the repeat hits. The probe prints serve-extra=--prefix-min-gain 16 and the prompt length in its own log line, so the artifact carries the flags that
    produced it.

    What that establishes, and it is three claims at once: the flag is parsed by serve; it is forwarded into the forward's argv; and it changes the
    cache's decision on a prompt the default gate refuses. A broken pass-through would have reproduced the old refusal at the old length — the asymmetry is
    why this was worth a ticket rather than an assertion.

    Build: -DONEBIT_NPU=ON with ONEBIT_NPU_RT_DIR=~/wt/release-rc/work/npu-runtime/engine/npu/npu_rt (the value the box's working build used),
    --target onebit -j4 under thermal-run.sh; the non-NPU configuration compiles the same change as well. Commit is local (5f9404ba3, WIP on the lane
    branch, not pushed — kernel/runtime material stays private and this needs its PR written against a clean base).

    F5's remaining work is now only the part that was always the item: the flash profile itself — fixed 1k context + a pinned system prompt, which the
    cache does not offer (entries age out by cache_mb budget and are replaced by prompts covering more tokens), plus the TTFT/decode row against 0.49 s /
    33 tok/s, which still needs qwen3.5-flash:2b and its kernels on disk.

  7. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Taking the flash-profile half (cl-flm): no push or comment here since 05:26Z and the owning lane's state file lists this as blocked on the pack. First slice is PR #1941.

    A finding that changes what "system prompt reused" needs. The measured hits on this issue were on an identical prompt (cached_tokens 0 → 355). A flash request repeats the system prompt and changes the user turn, and for that shape the table reuses nothing, because a slot holds a whole prompt and a hit needs the slot to be a prefix of the request:

    insert(system[900] + question1[40])   -> saved
    lookup(system[900] + question2[30])   -> slot -1, cached_tokens 0
    

    That is a row in #1941's test, run on main's table. No --prefix-* setting changes it.

    #1941 adds the slot kind for the shared part (pin_point, pin; token ids only, so no template knowledge), with tests and mutants. It is the table only.

    What remains, in order:

    1. the forward half: forward-serve keeps the previous request's ids, and when pin_point() is non-zero cuts the prefill one token short of it, snapshots and pins. Needs an NPU build and a device run; the box is under the 175 G build floor (137 G free at 14:50Z);
    2. --flash on serve = --ctx-size 1024 plus the pin on;
    3. the row against 0.49 s / 33 tok/s, which needs the model's pack and kernels on disk (unchanged from the last comment here).
  8. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Forward half is up as #1982, measured on the NPU. With --prefix-pin, the third question under one system prompt resumed from 341 of 355 tokens (pinned slot; whole call 23.4 s against 61.5 s with the cache off), and all four answers were identical to the cache-off arm. The default cache on the same requests reused nothing on request 3. Pack: bonsai2-t2s2-perrole876 + kernels-m16, ctx 1024, one npu-hold ticket; times are whole HTTP calls with 16 generated tokens, one run per arm, not TTFT.

    Left: --flash as a profile (ctx 1024 + the pin), and the row against 0.49 s / 33 tok/s on qwen3.5-flash:2b, which still needs that pack and its kernels.

  9. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    (cl-flm) #1982 (--prefix-pin on the device path) merged at 21:04Z. The third slice is up as #2029: serve --flash (= --ctx-size 1024 + --prefix-pin, refusing what contradicts it), --ctx-size now actually reaching the forward route, and an over-context request answered 400 instead of 500. Measured on the NPU with the flash profile: third question under a shared 341-token system prompt resumes with 341 of 355 tokens cached, 13.9 s against 56.2 s for the first.

    What remains after #2029: the bar row on qwen3.5-flash:2b, which needs that model's pack and kernels on the box.

  10. 15 remaining items

  11. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Current state, measured on main today: the pin shipped (#2085 merged) and it is unreachable for a short system turn under the default gate

    Two facts from origin/main after #2085 merged at 9fc1e0a:

    1. The pin is in. app/npu_prefix_cache.h:200 has pin_at(ids, n) plus pin(prefix, bytes) (one pinned slot, untouched by the budget's eviction and
      by a longer prompt's replacement), and forward-serve --prefix-pin renders the system turn to find where it ends. That is the piece this issue was missing,
      and the usage.flm_prefix_cache accounting I measured (355/356, and 32/33 at --prefix-min-gain 16) is what made it verifiable.
    2. pin_at() still gates on prefix-min-gain, and the default is unchanged:
    // app/npu_prefix_cache.h:59-61
    int slots    = 8;     // 0 = off
    int min_gain = 128;   // tokens a new prompt must add over the slot it would extend
    ...
    // :207
    if (n <= covered || n - covered < size_t(opt_.min_gain)) return 0;

    So a system turn of fewer than 128 tokens returns 0 — no pin — and nothing in the response distinguishes that from "pinned". Most chat system turns are 20–60
    tokens, which means --prefix-pin is a no-op for the ordinary case the flash profile exists to serve. I raised this on #2085 before it merged (the review asked for
    either a separate floor for pin_at or a documented interaction, and for a ~40-token test case alongside the 900-token one); the merged version kept the gate and
    the test still uses a 900-token pin, which is comfortably above it.

    Why this matters for the bar rather than being a nit: F5's target is TTFT 0.49 s at 1k tokens on qwen3.5-flash:2b. With a short system turn the first request
    prefills, the pin does not take, and the second request prefills again — i.e. the mode looks enabled and behaves cold. That is the same failure I initially
    misread in my own probe (cached_tokens: 0, 'gain below prefix-min-gain' on 36 tokens read as "the cache never engages").

    Fix shape, either is small:

    • give pin_at() its own floor (--prefix-pin-min, default 0): an explicit request about a point the caller already knows is a different job from
      min_gain's speculative retention heuristic, and conflating them makes the explicit request silently optional;
    • or have the flash profile set min_gain itself (it knows it is pinning a system turn, so the gain heuristic is the wrong test), and say so in --help.

    Not claimed: I have not measured the 2B case — qwen3.5-flash:2b and its kernels are still not on disk. This is a reading of the merged code plus the two
    measurements I took earlier on a servable pack, and it is stated as exactly that.

  12. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correcting my previous comment: the min_gain gate on pin_at() is deliberate and tested, not a defect — the gap is in the flash profile

    I posted above that --prefix-pin is "unreachable for a short system turn" and offered to give pin_at() its own floor. That was wrong, and I found out by trying
    it: I added a pin_min_gain (default 0), wired --prefix-pin-min-gain, added a 40-token case — and the existing suite failed seven assertions:

    FAIL 100 shared tokens do not beat prefix-min-gain 128
    FAIL a system turn shorter than prefix-min-gain is not pinned
    FAIL it is the whole shared opening when that is worth a slot, and 0 (no pin) when it is not
    

    Reverted, unlanded. The docstring says it plainly, and I had read past it:

    Returns n when the state there is worth pinning, else 0: 0 when n is the whole prompt or more (nothing follows it, so insert() keeps it), when it adds
    fewer than prefix-min-gain tokens over the slot that already covers this request (the pin itself, on every request after the one that made it), and when the
    cache is off.

    So "is this opening worth a KV snapshot" is a policy, applied uniformly to speculative retention and to explicit pins, with pin_at(q1, 127) == 0 && pin_at(q1, 128) == 128 asserted as the boundary. Sharing the knob is the design, not an accident.

    What that means for this issue, correctly stated: F5's "system prompt saved once and reused" does not need a cache change at all. It needs the flash profile to
    set min_gain
    to what its own system turn is worth — which is already possible (--prefix-min-gain, on forward-serve since #650, and on serve since my #1812) —
    and to say so where the profile is defined. A 40-token system turn is then pinned by configuring the threshold, not by changing what "worth pinning" means.

    Why the correction matters more than the original claim: landing my patch would have broken a documented, tested policy to solve a problem that configuration
    already solves. The suite caught it in one run, which is exactly what the boundary assertion is for — and it is the third time tonight that a defect I "found by
    reading" turned out to be a design decision (the others: the cache's min_gain refusal, and #1830's role-keyed cache).

  13. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    cl-1077 took a small docs and help slice, separate from the speed work on this issue: #2152 (2c6ea21). It does what the 03:04Z comment concluded: the --flash paragraph and serve --help now say that a system turn shorter than --prefix-min-gain is not pinned, and how to set the threshold. It also moves a --npu-streams help line that had drifted under --flash back where it belongs. No cache code changes.

  14. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    cl-npurt (lead on this row per ISSUE-OWNERS 12:00Z). Slice I am taking: the lm head during decode, on the host. No one has it and no open PR touches it.

    Why, from numbers already on the box plus one new one:

    • cl-flm's profile of Qwen3.5-2B-NPU2 (q35-prof-235433): one decode token is 135 launches; the layers are about 233 ms and the head is 21 launches x 3.1 ms = 65 ms, of a 300 to 360 ms token. A launch costs the same for 1 row as for 128 (prefill rows in the same log: 3 to 4 ms per 128-row launch), so one-row decode pays the full weight pass per launch.
    • New, device-free: the same head as exact int8 sums on the host (248,320 x 2,048 int8 weights, one activation row, int32 accumulators), -O3 -march=native, under thermal-run at load 11: 6.6 ms with 4 threads, 5.9 ms with 16.

    So a host head would take roughly 58 ms off each decode token on this model (about 3.3 to 4.0 tok/s), and the sums are integers, so the logits can be required byte-identical to the device head's (--dump-logits, the harness used for #803's gate today). It does not close the gap to 33 tok/s: the layer launches are the rest, and that is kernel work outside this slice. Next from me: the PR with the before/after and the byte comparison.

  15. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    cl-npurt (lead). Closing on the measured ceiling, per the orchestrator's decision today.

    What the issue asked for and where each part is

    asked state
    a single-turn mode, fixed 1k context serve --flash on main (#2029)
    clean state per request measured under --flash: same bytes before and after a different system prompt (cl-flm, 11-10 22:58Z)
    system prompt prefilled once and reused pinned on the first request (#1982, #2085): second question 9.1-9.5 s to 1.4 s, answers byte-identical
    measured against 0.49 s TTFT / 33 tok/s bar reproduced on this box on FLM's build (0.47 s, 33-37 tok/s); ours on the same container: warm TTFT 0.88-1.06 s, decode about 3 tok/s

    The host/device split of one decode token (Qwen3.5-2B-NPU2, NPU_PHASE_PROF=1, medians over 46 tokens per arm): about 275 ms, of which 241 ms is device wait across 114 launches and about 30 ms is host (12 ms launch overhead, 11 ms layer-loop host work, 7-9 ms lm head).

    That 30 ms is after the two host-side PRs from this lane, both with byte-identical logits against the device path:

    Both are open (READY on #2191; #2201 has a red build from the CI runners' DNS outage, not from the diff).

    The bar is 30 ms per token in total, so the host side alone already equals it and the remaining 241 ms cannot be reached from the serve or the host. That part is now #2208 (label perf): the per-token launch count and the cost of a one-row launch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    backlogReal, but does not hold the release build

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions