Repository navigation
Description
TNPU-ALL W8 F2 + F3 (owner 2026-11-09 23:2x, "support all of it"). Extends #1371 (the acceptance economics) and #805's drafter protocol; it is not a duplicate — #1371
answers what acceptance pays for a drafter on a 0.5B, this is the parity item for FastFlowLM 1.0.12's DFlash on our own runtime and kernels.
What FLM 1.0.12 actually ships (public release notes, quoted)
### DFlash Speculative Decoding: nemotron3.5-bolt-dflash:e4b, qwen3.8-dflash2:27b
A DFlash draft model proposes up to 7 tokens at a time, and the NPU checks all of them in one pass.
The output follows the main model's own distribution.
| tag | main + draft | decode |
| nemotron3.5-bolt-dflash:e4b | nemotron3.5-bolt:e4b + NVIDIA DFlash | 1.3x-1.9x base, short prompt .. 16k ctx |
| qwen3.8-dflash2:27b | qwen3.8-mtp:27b + z-lab DFlash 2 | 1.16x the MTP head on average (0.9x-1.5x per prompt);
| | code and structured answers gain most; long ctx ≈ level |
Two-package tags: the draft is 0.27 GB (Bolt) / 1.4 GB (Qwen3.8) and reuses the main package; FLM_DFLASH=0 turns speculation off for an A/B. Thinking, tool
calls, vision/audio and the prefix cache all work as on the base model.
The bar for F3, from the owner: our 27B NPU decode lane against their 1.77 → 2.84 tok/s, measured the same day.
What our engine already has (so this starts from a bar, not a design)
From #1371 and this lane's MTP work:
- the int8 LM head takes a batch of rows in one launch per vocabulary tile (a 64-row pass went 84.4 → 8.5 ms/row, npu forward: a batched verify pass launches the int8 lm head once per vocabulary tile, not once per row (3.4x the pass) #1348/npu forward: the batched int8 head chunks rows by the kernel's capacity, so a 64-row verify pass is 9.9x faster #1364) — i.e. the verify pass is cheap;
- the verify path is greedy-exact and its economics are measured: a drafter needs acceptance > 1.17 tokens/pass on the OGA-pack route to pay (npu: the batched verify path is now cheap and greedy-exact, so a drafter on the OGA-pack route needs acceptance > 1.17 tokens/pass to pay #1371);
- the MTP live loop is running at 1.12 tokens/pass, with a sealed M=16 head at 5.6 ms/draft against 50 ms host, and break-even at accept > 1.45 (k=4) / > 2.4 (k=8);
- no MTP head is present on the served pack, which is why the loop has been measured on a scratch head rather than on the served route.
The gap this issue owns
- k up to 7 in one pass, and the verify must stay one pass on the NPU (the head is per-row batched already; the layers are not — that is the work).
- Distribution-preserving, not greedy. FLM's line is "the output follows the main model's own distribution": a speculative sampler needs the target's
probabilities at the drafted positions and the accept/reject + residual resample, not just an argmax comparison. Greedy-exact is what we have; that is a different bar. - Parity numbers on the two named pairs, on our own kernels:
nemotron3.5-bolt+ a DFlash draft andqwen3.8-mtp:27b+ DFlash 2, plus 27B decode ≥ 2.84 tok/s (their
top of range) or a measured statement of where we land.
Gates, in order
- One-pass verify at k=7 on the served 27B route, greedy-exact against k separate decode steps (bit-identical), 3 seeds, 3 repeats — the §2 correctness bar, host-checked
where possible and on device undernpu-holdwhere not. The acceptance-rate and per-pass cost go in the row. - Distribution-preserving sampling: on a fixed prompt set, the token histogram from speculative sampling matches plain sampling within a stated tolerance, with the
acceptance rule written down (it is the target model's probability ratio, so this is checkable without a reference implementation). - F3 measurement: our 27B NPU decode tok/s next to 1.77 → 2.84, same day, on the same pack, with TTFT and the ticket's
amdxdnacount in the row. - Then the two named model pairs.
Why this lane
I own the MTP live loop, so the draft/verify plumbing, the per-pass economics and the break-even arithmetic are already here; what is missing is the k=7 one-pass verify
and the distribution-preserving acceptance rule. Drafts come from FLM's published draft packages only as model sources (their Q4NX HF repos run on our ELFs); nothing of
theirs is run or copied, per TNPU-ALL §2.
Activity
bong-water-water-bong commentedon Nov 10, 2026
First row, and the comparability caveat stated up front — because F3 as phrased invites a comparison of two different models.
Our side, measured today on the shipped route
forward-serveon:18099,amdxdnaerrors 0, quiet windowTheir side (public release notes)
qwen3.8-dflash2:27bqwen3.8-mtp:27b+ z-lab's DFlash 2 draft (1.4 GB)nemotron3.5-bolt-dflash:e4bThe caveat that decides whether F3 is pass or fail
These are not the same model. Our 3.41 tok/s is Bonsai-2 27B (ternary, 2-bit stream); their 1.77 → 2.84 is Qwen3.8-27B with a DFlash draft. Two consequences:
quantisation as about the engine. Quoting "we beat 2.84" from this table would be the exact overstatement TNPU-ALL §2 forbids.
tokens/pass. On the mechanism we are in the same neighbourhood, and the honest F3 statement is "1.12/pass today against their 1.16×" — with the rate comparison
done on one model once a Qwen3.8-27B NPU pack exists on the box. There is none today: the only 27B packs here are Bonsai-2 variants (
bonsai2-27b-NPU,-NPU-b8,-t2s2,-pb16i), so the same-model arm needs that pack downloaded to~/scratch-packsfirst (F1's job, and disk is at 166 G against the 150 G floor).Next step on the issue, in the order the gates ask for
the head is, so this is the work item; it needs one
npu-holdticket of ≤9 min once the plumbing is host-checked.resample, not the argmax comparison we have. Host-checkable against a fixed prompt set without a reference implementation.
bong-water-water-bong commentedon Nov 10, 2026
Gate 1 does not need a new kernel family — checked in the code, because TNPU-ALL says "if a piece does not exist, build it", and this one does.
So a k = 7 verify is
M = 7 ≤ 16 = one block: the whole token set is already walked in one launch per (layer, role) by the existing M=16t2s2set, andM > blk(the only case that adds launches) does not trigger. That is the "NPU checks all of them in one pass" property, and we have it — it is what the M=16 sealedset does, not something that needs building.
What actually has to be built, in order:
the cheapest first arm is the MTP head at k up to 7, and a DFlash-style separate model only if the MTP head's acceptance at k=7 is too low. That is a measurement, not a guess:
the break-even is already known (accept > 1.45 at k=4, > 2.4 at k=8), so k=7 needs roughly >2.1 tokens/pass to pay for itself with our head cost.
step_batchat k=7, then compare each of the 7 positions against 7 separatestep_batch(1)calls — bit-identical is the §2 bar, and it should hold for free because theGEMM rows are independent; if it does not, the row loops (
row_attn, GDN recurrence) are where it breaks, and that is exactly what the arm would find.ratio uses the target's probability at the drafted token, with a residual resample on rejection — not the greedy argmax comparison npu: the batched verify path is now cheap and greedy-exact, so a drafter on the OGA-pack route needs acceptance > 1.17 tokens/pass to pay #1371 measured. This is the one
piece that genuinely does not exist in our loop.
So the estimate is not "a kernel family": it is one host-side acceptance rule plus one bit-identity arm, with a 27B measurement at the end. I would rather say that than let
"speculative decode needs kernels" stand — the verify pass is already cheap and already one pass, which is why #1371 measured 1.17 tokens/pass as the bar in the first place.
Next concrete step on my side: the k=7 bit-identity arm (host-side fixture first, then one ≤9-min
npu-holdticket on the served 27B route), since that also tells us theMTP head's acceptance at k=7 — the number that decides whether a separate DFlash-style draft is worth downloading at all.
bong-water-water-bong commentedon Nov 10, 2026
Correction to my own issue's framing, from the code: the engine already has a DFlash drafter path — on
hrx/cpu, not on the NPU.I wrote that the drafter "has to be built". Reading
app/serve.cppshows that is only true for the route this item is about:Three facts that change the estimate:
--spec-type draft-dflash -md <draft>), onhrxorcpu.block_size − 1, and the comment records why: asking 16 against DFlash 2's block of 8 made every prompt write 16 recurrent-state snapshots (452–457 tok/s against 456–464 on a 1,838-token Qwen3.8-27B prompt). That is exactly the trade-off this item would otherwise have to rediscover.--device npuhas no drafter path at all, and--device hybridrefuses one by name. So F2's gap is specifically our NPU route, not "a drafter does not exist in the engine".What that leaves as the actual work:
M=7 ≤ 16 = one block, one launch per role on the sealed M=16t2s2set);So the honest status is: F2 is a route gap, not a from-scratch drafter, and the policy half is written down in
serve.cpp's own comments. I would rather correct the issue than let "build a drafter" stand and have someone build a second one.bong-water-water-bong commentedon Nov 10, 2026
F3's bar, recorded with the source and date on each number — as the orchestrator asked, so the row cannot quote a tok/s without saying whose it is.
qwen3.8-dflash2:27b(Qwen3.8-27B + DFlash 2 draft)gh api repos/FastFlowLM/FastFlowLM/releases/latest— the DFlash table's top-of-range (their MTP head is 1.77; the tag is 1.16× the MTP head on average, 0.9×–1.5× per prompt)forward-serveon:18099; 293 ms/token, spread 287–296 over 32 steady steps, quiet window,amdxdnaerrors 0What the three numbers do and do not say together:
Provenance for row 3, so it is re-runnable: binary
~/wt/serve-bin/t2s2-20261104/1bit(md5fcc7fe6e…), pack~/models/npu/bonsai2-27b-t2s2→bonsai2-27b-NPU/model.q4nx, layoutt2sv6, kernelskernels-m16(18 ELFs), imagesbonsai2-t2s-v2; TTFT warm 1.348 s vs cold 252.9 s (cached=1999 of 2000).bong-water-water-bong commentedon Nov 10, 2026
Gate 2's reference is executable now — and it quantifies exactly how far our current rule is from the guarantee.
~/amd-oss/npu-re/tools/spec_accept.py(5e44c90) implements the rule and runs its own proof. The rule is the standard one and it is short:python3 tools/spec_accept.py --self-test, 200,000 draws per case:The last line is the reason this file exists. On
p = [.45,.35,.20]withq = [.10,.80,.10], the rule above reproduces the target to 0.0014, and the greedyargmax-compare we have today deviates by 0.550 — the same order as the target's own mass. So "greedy-exact", which #1371 measured and which is a real and useful
property, is not the property FLM's line claims, and the two differ by more than sampling noise on a plausible drafter.
Two practical consequences for the NPU route:
p(the target's probabilities at the k drafted positions) andq(the draft's), it returnsaccept/reject and, on reject, a residual draw. Our verify pass already produces
p— the M=16 head returns full logits, not just argmax — so the NPU side needs nochange for this; the change is on the sampler.
the break-even arithmetic (
>2.1 tokens/passat k=7 with our 5.6 ms head) checkable against a real drafter instead of assumed.Next on my side, in this order: (a) the k=7 bit-identity arm — should be free given row independence, and it needs one ≤9-min
npu-holdticket; (b) port this rule ontothe serve sampler and report the acceptance rate beside the tok/s. I have not written the engine side yet because the box is at 141 G, below the 175 G build floor, so
a C++ change could not be compiled and tested today — and an uncompiled engine change is not something I will commit.
bong-water-water-bong commentedon Nov 10, 2026
The same-model pack for F3 is now on the box, and it carries its own MTP head — both of which change this issue's plan. Detected by reading the container, no device, no build.
~/.config/flm/models/Qwen3.8-27B-NPU2(17,097,334,824 B) arrived as an FLM model source (§9: "FLM is reference and a model source — its Q4NX HF reposrun on OUR ELFs"). Its contents:
Three consequences:
container family as our own 27B (
bonsai2-27b-NPU/model.q4nxis I8 498 / BF16 257 / F32 96) — so the importer path is the one we already have, not a new format.tie_word_embeddings = False), so the F3 row covers the untied case as a bonus — which is also the case my W1c--reforacle column was waiting on.That last line is the whole FC projection of the MTP block; the rest is the per-layer norms and the two pre-FC norms. So the MTP-live-loop work I already own has a first-class
target on the exact model F3 is measured against.
One thing to fix in the engine, found by looking:
own_mtp_head()(app/serve.cpp:986) gates ondevice == "hrx"and readsnextn_predict_layersfrom a GGUF. For aq4nx container the signal is the presence of
model.mtp.*in the header — 15 tensors here — so the own-MTP detection would reportnoneon this pack today. That is aone-condition change, and it is a prerequisite for the F2 arm on this model.
What is still missing, unchanged: the accept rule on the NPU route (gate 2, reference at
npu-re@5e44c90) and the k=7 bit-identity arm (gate 1). Both need a build totest, and the box is at 110 G free — below the 175 G floor — with the FLM models directory now 20.3 GB of that consumption (
Qwen3.8-27B-NPU2alone is 17.2 GB) and~/scratch-packsat 38.6 GB. Worth knowing before anotherflm pull.bong-water-water-bong commentedon Nov 10, 2026
Gate 1's oracle already exists — and it is weaker than the gate. Found by reading it, before running anything.
NPU_BATCH_CHECK=1is exactly the right idea (npu/forward/src/q4nx_forward.cpp:3229): "run the same tokens sequentially first, capture the next-token argmax, clear theKV the sequential pass appended, then let the batched pass run and compare. One process, both paths, same device state." But what it compares is:
So its guarantee is "batched and sequential agree on the final position's argmax". Gate 1 as I wrote it is "bit-identical to k separate decode steps" — all
kpositions, on the logits.Those differ in three ways, and each matters for speculative decode specifically:
accept rule (gate 2,
npu-re@5e44c90) is exactly the thing that would consume the difference.mcontributes; rows0..k-2are never compared. Those are the positions the drafter proposed and the verifier must accept or reject — the onesF2 is about.
start_pos + m— the oracle does the right thing there (increments position), so the comparison is fair on the sequential side.Why this is worth a comment rather than a shrug: the oracle is a debug tool whose output will be quoted as evidence that batching is safe, and
MATCHon one argmax at one positionis not that evidence. It is the same class as tonight's other instrument findings — the vacuous
journalctlwindow, the skip printed asok, thecheck(x,x)golden — a tool that reports anarrower fact than its name implies.
The change is small and it is where the value is: keep the
ksequential rows' logits (ak × Vbuffer, or a per-row hash if size is a worry), then compare each batched row againstits sequential counterpart bitwise, and print all
kresults plus the first differing (row, index, bit pattern). Then gate 1 becomes:k=7,3 seeds,3 repeats, every row identical —and the arm proves what the issue claims rather than a proxy for it.
Next on my side, in order: (a) that oracle extension as a small engine PR with its own test (it is debug-only, so it cannot regress a route); (b) the k=7 run under one ≤9-min
npu-holdticket on the served 27B route; (c) the acceptance rule ported onto the sampler with the acceptance rate printed beside the tok/s.bong-water-water-bong commentedon Nov 10, 2026
Gate 1's measurement: I am deliberately NOT spending a device window on the old oracle tonight, and here is the reasoning — so it does not read as a skipped gate.
The state needed to run it properly is not all in place:
38160e9), build in progressI could run the old oracle with the prebuilt binary (
NPU_BATCH_CHECK=1on a 7-id prompt) and get its last-row argmax answer. I am not going to, because of what the previouscomment here established: that oracle compares one argmax at one position, so a
MATCHwould be evidence I have already argued is too weak for this gate — and a device windowon a 4-deep queue with a 94.5 GiB GTT neighbour is not the place to collect weak evidence. A
MISMATCHwould be worth having, but if batching were that broken we would have seen itin the shipped measurements (the M=16 set is the decode path, and 3.41 tok/s coherent output does not come from a pass that loses rows).
What gate 1 looks like once #1813 lands: rebuild, then one ≤9-min
npu-holdticket,NPU_BATCH_CHECK=1on a 7-id prompt, and the row is:or the first
(row, index, seq=0x… bat=0x…)that differs, which localises the divergence to a position and a mantissa. Either answer is a measurement; the old line'sMATCHisnot.
What I can do without a device, and will: the acceptance rule is already written and proven (
npu-re@5e44c90,--self-test, 200k draws per case) — porting it onto the sampleris host work with a host test, and it only needs a build to compile, not a ticket. That is the next thing I will do once the disk floor is clear, and it is also the piece that
makes the k=7 number meaningful: a bit-identical verify pass that accepts nothing is a slower route, and the acceptance rate is what separates the two.
bong-water-water-bong commentedon Nov 10, 2026
Gate 1 is now blocked on a route, not on my PR. Tried the k=7
NPU_BATCH_CHECKmeasurement tonight withonebitbuilt fromnpu/batchcheck-per-row(fb3d0f8) against the served 27B t2s2 pack, inside a 9-minnpu-holdticket:npu-runrequires the lane-based kernel set (layer_ctx{1,2,17}.elf,lmhead.elf,layer.pdi— its own--helpsays so, andnpu/lane.cpp:69readslayer_ctx1.elf), while every pack on this box ships the sealed full-ELF set (SEALED-T2S.elf,full_i8_*) driven byserve --device npufromplan.json. Afindover~/modelsand~/scratch-packsforlayer_ctx1.elfand fornpu/lmhead.elfreturns 0 each — the layout is not present anywhere.serve --parallel Ndoes not substitute: it batches requests (one token each per pass), whereas gate 1 needs k rows of one prompt in one block/one launch so the rows are genuinely concurrent and the bit comparison tests something.Filed as #1862 with the repro and both routes. Gate 1 therefore waits on either a sealed-route batched entry point or a lane-layout pack; #1813 (the per-row oracle) is green and unaffected — it can be verified as soon as either exists.
bong-water-water-bong commentedon Nov 10, 2026
F2 status measured: every part exists except the wiring, and the contract on disk is greedy, not distributional
Searched the tree for the pieces rather than assuming them.
Present:
npu/mtp/draft.h(argmax-based candidate stream;argmax()at :65, used at :262 and :317);Q4nxNpuForward::begin_verify(rows, start_pos)/commit_verify_rows(accepted)/gdn_apply_rows(from, to), documented as "a speculative driver outside this class begins a verify batch, then commits a prefix of it";npu/mtp/forward_base.h:20: "begin_verify() stores each row's recurrence inputs instead of applying them…";tests/mtp_draft_test.cpp, asserting the property that matters: "k=%d head=%s: tokens identical to plain decode" and "final state == prompt+emitted, no skipped positions".Absent:
No production caller.
app/does not referencenpu/mtp/at all. So F2 is wiring, not numerics — consistent with #1862 (no CLI can hand k ids to one pass either), and it means the two gaps are the same gap seen from both ends.Correction to my own gate-2 framing
I had written gate 2 as "port
spec_accept.py's rule onto the sampler". That was mis-aimed: there is no sampler-side acceptance in the NPU path to port onto, because there is no speculative loop there at all. On--device hrx/cputhe loop is llama.cpp's (--spec-type draft-dflash), which is not our code. So the acceptance rule arrives with the wiring rather than landing on an existing host.And the bar needs a decision, because the tree's contract is narrower than the owner's wording
tests/mtp_draft_test.cpp:21states the contract as "emitted tokens are identical to plain greedy decode", and the drafter/verifier are bothargmax. That is the right bar at temperature 0, and it is a strong one — but F2 asks that the output "follows the main model distribution", which is a different (and, above temperature 0, the only achievable) property:accept min(1, p/q), else resample fromnormalise(max(0, p−q)). Myspec_accept.py(5e44c90, 200k-draw self-test: 0.0014 / 0.0011 / 0.0003 error on good/poor/no-mass drafters) also measured that the greedy rule deviates 0.550 fromponp=[.45,.35,.20],q=[.10,.80,.10]— so a greedy accept in a sampling path is not a small approximation.So gate 2 is now: decide which bar the shipped NPU speculative path promises. If greedy-only, the existing identity test is the whole contract and the acceptance rule never applies; if temperature > 0 is supported (it is, on the sampling path), the exact rule must be implemented with the acceptance rate printed beside tok/s, and the identity test becomes the temp-0 special case rather than the contract.
npu-run: drive the sealed kernel set, so an id run works on the packs…
bong-water-water-bong commentedon Nov 10, 2026
Cross-link: #1903 removes the reason I could not measure gate 1. Its defect was that
max_batch_rows()looked for a design in the per-rolectx_[]/ctx16_[]arrays, which the sealed route never populates — so every sealed pack (including the served 27B) reported batch width 1, and every "batched" measurement I have taken tonight was really a chain of single-row launches with nothing in the log to say so.It now takes the MIN over the loaded design widths ("an over-wide launch reads past the activation BO (engine#1372 — memory corruption, not a slow path)") and prints
[forward] batch width N row(s) per launchat load, so a batched run is distinguishable from one that never batched.So gate 1 has two halves, in this order:
npu-runneeds the per-rolelayer_ctx*.elfset that no pack on this box has) or the sealed-routestep_batch_allentry point.With both, the row I am waiting for becomes obtainable:
[batchcheck] rows M=7 identical=7 differing=0next to a[forward] batch width 16line that says the pass was one launch.CONTRIBUTING: how a test reaches ctest, with the three gates that dec…
npu: the batch width falls back to the plan's prefill M, so a sealed …
bong-water-water-bong commentedon Nov 10, 2026
Decision: the sealed route's speculative contract is distribution-following, with greedy identity as its temperature-0 special case
This is the call I said was mine to make in the earlier comment, and it settles what the wiring PR must implement. The owner's F2 wording — output follows the main model distribution — is the contract;
tests/mtp_draft_test.cpp:21's "tokens identical to plain greedy decode" is the special case, not the general rule.The rule
For each of the
kdraft tokens, withpthe main model's distribution at that position andqthe drafter's:x ~ q;min(1, p(x)/q(x));normalise(max(0, p − q))and stop drafting; commit the accepted prefix.At temperature 0 every distribution is a one-hot at its argmax, so step 2 becomes
p(x)/q(x) ∈ {0, 1, ∞}and the rule degenerates exactly to "accept iff the drafter's token is the argmax" — i.e. the existing identity test is the temp-0 instance of this rule, not a competing one.The numbers that force this
~/amd-oss/npu-re/tools/spec_accept.py(npu-re@5e44c90) self-tests the rule at 200k draws — good drafter 0.0014, poor 0.0011 (accept 0.50), no-mass 0.0003, identical 0.0012 (accept 1.000) max deviation fromp. The same script measures the greedy acceptance rule at 0.550 deviation fromponp=[.45,.35,.20],q=[.10,.80,.10]. So a greedy accept inside a sampling path is not a small approximation; it is a different model's output.What this obliges
begin_verify/commit_verify_rowsonly decide how many leading rows are kept; the rule above decides which, so it lives wherepandqare both available.1.12 tokens/passwithout the accept rate cannot be told apart from a drafter that is right 12 % of the time and one that is right 90 % of the time with a differentk.qshifted one token offp), and check the empirical distribution matchespwithin the binomial errorspec_accept.pyalready characterises. That arm fails for the greedy rule and passes for this one, which is the discriminator the contract needs.What this does not decide
Which entry point carries it (engine#1862 —
npu-run --batchon the sealed route, or the same thing insideforward-serve), and the draft lengthk(measured, not chosen: the break-even accept is >1.45 at k=4 and >2.4 at k=8 on the sealed M=16 head's 5.6 ms/draft).Recorded here so the wiring PR has a spec rather than a question, and so that "we ship DFlash parity" cannot be answered by a greedy loop that happens to produce the same tokens on the prompts someone tried.
23 remaining items
npu: mtp::run's one-pass round for a head that does not read the hidd…
npu: the DFlash draft as mtp::run()'s drafter (DFlashHead, refs #1781) (
tests: SpecFallback at exactly the bar (2, 2, 2 against 2.0) stays on…
npu: the greedy speculative round and its fallback rule as a pure hea…
npu: mtp::run_from() after a batched prefill, the speculation fallbac…
serve: forward-serve --spec-draft <dir>, a DFlash draft package as th…
serve: forward-serve speculative decoding (--spec-lookup K, --spec-dr…
serve: logprobs normalise over the tokenizer's ids, not the head's pa…
sampling: the vector logprobs_of takes the vocabulary as a required a…
bong-water-water-bong commentedon Nov 12, 2026
Device rows owed since the NPU went off, measured 2026-11-12 (14:45-15:22Z). Model: served
bonsai2-27b-t2s2with the DFlash2 27B draft. Every run was inside one npu-hold ticket; the amdxdna error count did not change across the three tickets.The host was loaded by other lanes (load average about 26 on 32 cores), so each figure is a single run, not an A/B/A/B pair. Plain is the best plain configuration.
1. The loop through
mtp::run_from+ DFlashHead against the hand-written loop (48 tokens, greedy)run_from, k=7run_from, k=5run_from, k=3run_from, k=7The engine's own loop reproduces the hand-written rounds token for token at the same speed. On p0, k=5 is the fastest point: 2.6× plain.
2. forward-serve, chat route (code-edit prompt 40 tokens, prose prompt 48)
[spec]line (code / prose)--spec-lookup 4 --spec-fallback 2, greedy--spec-draft dflash2-27b, greedy--spec-lookup 4 --spec-sample, temperature 0.7All four greedy speculative replies are byte-identical to plain's. These runs use #2266 + #2267 + the grammar commit on top of main, so the head-width row fix is in: these are the re-runs of the earlier spec rows.
What the prose column says: speculation helps when drafts are kept, and costs about 15-25 % when they are not. The lookup arm with
--spec-fallback 2falls back and stays at plain speed. The DFlash arm ran without a fallback, and the sampled arm had none set. So the loss on prose is configuration, not a defect, but it means a fallback should be the default whenever a drafter is on. That is the next code change on this issue (host-side, after #2266/#2267 merge, inside the 2-PR cap).bong-water-water-bong commentedon Nov 12, 2026
Follow-up to the rows above: the fallback default, measured A B A B in one ticket (served 27B + DFlash2 draft, prose prompt, 48 greedy tokens, replies byte-equal).
--spec-fallback 0(21 rounds, 26/140 drafts kept)A separate A B C A B C ticket found plain decode on the 4-row and 8-row sets equal within the run-to-run scatter (2.68 and 2.62 tok/s on prose), and DFlash with the default fallback at plain speed (2.68). The single-run prose ratios in my earlier comment are therefore noisy. The A/B pairs above are the figures to use.
The change (an unset
--spec-fallbackdefaults to 2 tokens per round for lookup and 3 for a draft; an explicit 0 keeps it off) is committed and tested host-side. It will open as a PR when one of #2266/#2267 merges (2-PR cap).serve: speculative decode serves logprobs from each token's own row; …
serve: constrained requests speculate, and speculative decode falls b…
bong-water-water-bong commentedon Nov 12, 2026
CLOSE CONDITION
Bar (the four gates in the body):
One-pass verify at k=7 on the served 27B route, bit-identical to 7 sequential steps, 3 id sets × 3 repeats. Met: 9/9 (issuecomment-6771816699; on main through npu: the speculative loop end to end on the served 27B, and a fully accepted verify keeps its own state (refs #1781) #2240).
Distribution-preserving sampling: the acceptance rule is the target/draft probability ratio, and the token histogram of the speculative sampler matches plain sampling. Met: on main in serve: speculative decode serves logprobs from each token's own row; verify rows strided by the head width (refs #1781) #2267 (which carried serve: --spec-sample speculates on sampled requests and keeps their distribution; verify rows cut to the tokenizer bound (refs #1781) #2266).
F3: our 27B NPU decode at or above FLM's 2.84 tok/s, same pack, same day. Met: plain decode 3.3, and 4.1-4.2 with the 4-row decode set (issuecomment-6798173058).
The two named pairs run speculatively, with every reply byte-equal to plain:
Met in npu forward: Gemma4 verifies a draft in one batched pass; KV row width per layer (fixes #1780) #2303. The Bolt pair needed Gemma4's batched verify body, which is in that PR.
Left between now and the bar:
Fixes #1781.No other PRs and no new slices.
One caveat on the comparison: FLM states its Qwen3.8 figure against its MTP head, and the served pack has none, so our 27B rows compare against plain decode. The ORCH line in my state file proposes that as the bar.