Skip to content

npu: speculative decode on the NPU -- a draft proposes up to 7 tokens and the NPU verifies them in one pass (FLM 1.0.12 DFlash parity, F2+F3) #1781

Closed

Description

@bong-water-water-bong

TNPU-ALL W8 F2 + F3 (owner 2026-11-09 23:2x, "support all of it"). Extends #1371 (the acceptance economics) and #805's drafter protocol; it is not a duplicate — #1371
answers what acceptance pays for a drafter on a 0.5B, this is the parity item for FastFlowLM 1.0.12's DFlash on our own runtime and kernels.

What FLM 1.0.12 actually ships (public release notes, quoted)

### DFlash Speculative Decoding: nemotron3.5-bolt-dflash:e4b, qwen3.8-dflash2:27b

A DFlash draft model proposes up to 7 tokens at a time, and the NPU checks all of them in one pass.
The output follows the main model's own distribution.

| tag                          | main + draft                       | decode                             |
| nemotron3.5-bolt-dflash:e4b  | nemotron3.5-bolt:e4b + NVIDIA DFlash | 1.3x-1.9x base, short prompt .. 16k ctx |
| qwen3.8-dflash2:27b          | qwen3.8-mtp:27b + z-lab DFlash 2     | 1.16x the MTP head on average (0.9x-1.5x per prompt);
                               |                                      | code and structured answers gain most; long ctx ≈ level |

Two-package tags: the draft is 0.27 GB (Bolt) / 1.4 GB (Qwen3.8) and reuses the main package; FLM_DFLASH=0 turns speculation off for an A/B. Thinking, tool
calls, vision/audio and the prefix cache all work as on the base model.

The bar for F3, from the owner: our 27B NPU decode lane against their 1.77 → 2.84 tok/s, measured the same day.

What our engine already has (so this starts from a bar, not a design)

From #1371 and this lane's MTP work:

The gap this issue owns

  1. k up to 7 in one pass, and the verify must stay one pass on the NPU (the head is per-row batched already; the layers are not — that is the work).
  2. Distribution-preserving, not greedy. FLM's line is "the output follows the main model's own distribution": a speculative sampler needs the target's
    probabilities at the drafted positions and the accept/reject + residual resample, not just an argmax comparison. Greedy-exact is what we have; that is a different bar.
  3. Parity numbers on the two named pairs, on our own kernels: nemotron3.5-bolt + a DFlash draft and qwen3.8-mtp:27b + DFlash 2, plus 27B decode ≥ 2.84 tok/s (their
    top of range) or a measured statement of where we land.

Gates, in order

  1. One-pass verify at k=7 on the served 27B route, greedy-exact against k separate decode steps (bit-identical), 3 seeds, 3 repeats — the §2 correctness bar, host-checked
    where possible and on device under npu-hold where not. The acceptance-rate and per-pass cost go in the row.
  2. Distribution-preserving sampling: on a fixed prompt set, the token histogram from speculative sampling matches plain sampling within a stated tolerance, with the
    acceptance rule written down (it is the target model's probability ratio, so this is checkable without a reference implementation).
  3. F3 measurement: our 27B NPU decode tok/s next to 1.77 → 2.84, same day, on the same pack, with TTFT and the ticket's amdxdna count in the row.
  4. Then the two named model pairs.

Why this lane

I own the MTP live loop, so the draft/verify plumbing, the per-pass economics and the break-even arithmetic are already here; what is missing is the k=7 one-pass verify
and the distribution-preserving acceptance rule. Drafts come from FLM's published draft packages only as model sources (their Q4NX HF repos run on our ELFs); nothing of
theirs is run or copied, per TNPU-ALL §2.

Activity

  1. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    First row, and the comparability caveat stated up front — because F3 as phrased invites a comparison of two different models.

    Our side, measured today on the shipped route

    what value conditions
    27B NPU decode 293 ms/token = 3.41 tok/s (spread 287–296, 32 steady steps) Ternary-Bonsai-2-27B, t2s 2-bit stream, M=16 sealed set, own forward-serve on :18099, amdxdna errors 0, quiet window
    same route, contended 2.22 tok/s (450 ms median) / 3.02–3.11 quiet on earlier runs the box was loaded
    MTP live loop 1.12 tokens/pass draft/verify plumbing in place, sealed M=16 head 5.6 ms/draft vs 50 ms host; break-even accept >1.45 (k=4) / >2.4 (k=8)

    Their side (public release notes)

    tag decode what it runs on
    qwen3.8-dflash2:27b 1.16× their MTP head on average (0.9×–1.5× per prompt; code and structured answers gain most; long ctx ≈ level) qwen3.8-mtp:27b + z-lab's DFlash 2 draft (1.4 GB)
    nemotron3.5-bolt-dflash:e4b 1.3×–1.9× base, short prompt to 16k ctx nemotron3.5-bolt:e4b + NVIDIA's DFlash (0.27 GB)
    the owner's bar for F3 1.77 → 2.84 tok/s their 27B

    The caveat that decides whether F3 is pass or fail

    These are not the same model. Our 3.41 tok/s is Bonsai-2 27B (ternary, 2-bit stream); their 1.77 → 2.84 is Qwen3.8-27B with a DFlash draft. Two consequences:

    1. Cross-model tok/s is not parity. Ours is higher, but a ternary 2-bit model streams ~0.47× the bytes of a 4-bit/8-bit one, so the comparison says as much about the
      quantisation as about the engine. Quoting "we beat 2.84" from this table would be the exact overstatement TNPU-ALL §2 forbids.
    2. The like-for-like number is the speculation gain, not the rate: their DFlash buys 1.16× over their own MTP head; our MTP loop currently delivers 1.12
      tokens/pass
      . On the mechanism we are in the same neighbourhood, and the honest F3 statement is "1.12/pass today against their 1.16×" — with the rate comparison
      done on one model once a Qwen3.8-27B NPU pack exists on the box. There is none today: the only 27B packs here are Bonsai-2 variants (bonsai2-27b-NPU,
      -NPU-b8, -t2s2, -pb16i), so the same-model arm needs that pack downloaded to ~/scratch-packs first (F1's job, and disk is at 166 G against the 150 G floor).

    Next step on the issue, in the order the gates ask for

    1. k=7 one-pass verify on the served 27B route, greedy-exact against k separate steps (bit-identical, 3 seeds, 3 repeats) — the layer path is not row-batched yet, only
      the head is, so this is the work item; it needs one npu-hold ticket of ≤9 min once the plumbing is host-checked.
    2. Distribution-preserving sampling — FLM's line is "the output follows the main model's own distribution", which is a target-probability accept/reject + residual
      resample, not the argmax comparison we have. Host-checkable against a fixed prompt set without a reference implementation.
    3. Same-model rate comparison once the Qwen3.8-27B pack is on disk.
  2. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Gate 1 does not need a new kernel family — checked in the code, because TNPU-ALL says "if a piece does not exist, build it", and this one does.

    npu/forward/include/q4nx_pack.h:70   #define Q4NX_DESIGN_M_PREFILL 128
    npu/forward/include/q4nx_pack.h:71   #define Q4NX_DESIGN_M_DECODE   16
    npu/forward/src/q4nx_forward.cpp:3703   return xd_ ? xd_->max_rows() : 1;        // 16 on the m16 target
    npu/forward/src/q4nx_forward.cpp:2989   const int blk = max_batch_rows();
    npu/forward/src/q4nx_forward.cpp:2987   // "Each row's result is independent of the others -- the row loops downstream
                                            //  (row_attn, the GDN recurrence) never needed the whole prompt in one launch,
                                            //  only the GEMMs did."
    

    So a k = 7 verify is M = 7 ≤ 16 = one block: the whole token set is already walked in one launch per (layer, role) by the existing M=16 t2s2 set, and
    M > blk (the only case that adds launches) does not trigger. That is the "NPU checks all of them in one pass" property, and we have it — it is what the M=16 sealed
    set does, not something that needs building.

    What actually has to be built, in order:

    1. The drafter. FLM's DFlash is a separate small model (0.27 GB Bolt / 1.4 GB Qwen3.8) proposing up to 7 tokens. We have an MTP head (sealed M=16, 5.6 ms/draft vs 50 ms host) — so
      the cheapest first arm is the MTP head at k up to 7, and a DFlash-style separate model only if the MTP head's acceptance at k=7 is too low. That is a measurement, not a guess:
      the break-even is already known (accept > 1.45 at k=4, > 2.4 at k=8), so k=7 needs roughly >2.1 tokens/pass to pay for itself with our head cost.
    2. step_batch at k=7, then compare each of the 7 positions against 7 separate step_batch(1) calls — bit-identical is the §2 bar, and it should hold for free because the
      GEMM rows are independent; if it does not, the row loops (row_attn, GDN recurrence) are where it breaks, and that is exactly what the arm would find.
    3. Distribution-preserving acceptance (gate 2), which is where FLM's line "the output follows the main model's own distribution" differs from what we have: the accept
      ratio uses the target's probability at the drafted token, with a residual resample on rejection — not the greedy argmax comparison npu: the batched verify path is now cheap and greedy-exact, so a drafter on the OGA-pack route needs acceptance > 1.17 tokens/pass to pay #1371 measured. This is the one
      piece that genuinely does not exist in our loop.

    So the estimate is not "a kernel family": it is one host-side acceptance rule plus one bit-identity arm, with a 27B measurement at the end. I would rather say that than let
    "speculative decode needs kernels" stand — the verify pass is already cheap and already one pass, which is why #1371 measured 1.17 tokens/pass as the bar in the first place.

    Next concrete step on my side: the k=7 bit-identity arm (host-side fixture first, then one ≤9-min npu-hold ticket on the served 27B route), since that also tells us the
    MTP head's acceptance at k=7 — the number that decides whether a separate DFlash-style draft is worth downloading at all.

  3. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correction to my own issue's framing, from the code: the engine already has a DFlash drafter path — on hrx/cpu, not on the NPU.

    I wrote that the drafter "has to be built". Reading app/serve.cpp shows that is only true for the route this item is about:

    // :54  --dflash <draft.gguf> on hrx or cpu -> a DFlash block-diffusion draft model
    // :1224  if (!o.mtp.empty() || !o.dflash.empty() || own_mtp) {
    // :1230      else argv.insert(argv.end(), {"--spec-type", "draft-dflash", "-md", o.dflash, "-ngld", "99"});
    // :1237      else if (!o.dflash.empty()) { const long long block = gguf_int(o.dflash, arch + ".block_size");
    // :1238          argv.insert(…, {"--spec-draft-n-max", block > 1 ? std::to_string(block - 1) : "16"});
    // :1082  --device hybrid  →  "takes no --mtp, --dflash, …"   (refused by name)
    // :986   own_mtp_head(...) applies only when device == "hrx"

    Three facts that change the estimate:

    1. A DFlash drafter already works, by driving the fork's own speculative machinery (--spec-type draft-dflash -md <draft>), on hrx or cpu.
    2. The draft-length policy is already worked out — the longest draft is the drafter's block_size − 1, and the comment records why: asking 16 against DFlash 2's block of 8 made every prompt write 16 recurrent-state snapshots (452–457 tok/s against 456–464 on a 1,838-token Qwen3.8-27B prompt). That is exactly the trade-off this item would otherwise have to rediscover.
    3. --device npu has no drafter path at all, and --device hybrid refuses one by name. So F2's gap is specifically our NPU route, not "a drafter does not exist in the engine".

    What that leaves as the actual work:

    • the verify pass on the NPU route — which, per my previous comment, already exists at the right width (M=7 ≤ 16 = one block, one launch per role on the sealed M=16 t2s2 set);
    • the acceptance rule on that route: the hrx/cpu path delegates accept/reject to llama-server, and our NPU route has no equivalent, so a distribution-preserving rule (target-probability accept + residual resample) is the piece that genuinely does not exist — unchanged from my last comment;
    • a draft model on the NPU, if the MTP head's acceptance at k=7 is too low to be worth the 0.27–1.4 GB (break-even >2.1 tokens/pass at k=7 with our 5.6 ms/draft head).

    So the honest status is: F2 is a route gap, not a from-scratch drafter, and the policy half is written down in serve.cpp's own comments. I would rather correct the issue than let "build a drafter" stand and have someone build a second one.

  4. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    F3's bar, recorded with the source and date on each number — as the orchestrator asked, so the row cannot quote a tok/s without saying whose it is.

    # number model hardware / route source date
    1 3 tok/s Qwen 3.8 27B Q4_0 Snapdragon X Elite NPU, fully on the NPU one user report relayed by the orchestrator (not a published benchmark; named as a report, not a release) 2026-11-09
    2 2.84 tok/s qwen3.8-dflash2:27b (Qwen3.8-27B + DFlash 2 draft) our chip (Ryzen AI Max+ 395), FastFlowLM 1.0.12 public release notes, gh api repos/FastFlowLM/FastFlowLM/releases/latest — the DFlash table's top-of-range (their MTP head is 1.77; the tag is 1.16× the MTP head on average, 0.9×–1.5× per prompt) 2026-11-09
    3 3.41 tok/s Ternary-Bonsai-2-27B (our 2-bit t2s stream) our chip, our engine, M=16 sealed set, own forward-serve on :18099; 293 ms/token, spread 287–296 over 32 steady steps, quiet window, amdxdna errors 0 this lane's measurement (TBONSAI-SPEED-STATE, and the #802 model-level row) 2026-11-09

    What the three numbers do and do not say together:

    • All three are 27B-class and none is the same model. Row 1 is Q4_0 on a different chip; row 2 is Qwen3.8-27B with a draft model on ours; row 3 is a ternary 2-bit 27B on ours with no speculation. A single "we beat 2.84" or "we beat Snapdragon" would be exactly the cross-model comparison TNPU-ALL §2 forbids.
    • The like-for-like axis is (model, quantisation, speculation), and we hold none of the three equal. Our 3.41 has no drafter; row 2 has one; row 1 is 4-bit-ish on another chip.
    • What can be concluded today: our unspeculated 27B NPU decode (3.41) already exceeds row 2's speculated figure on the same chip (2.84), and the honest reason is bandwidth — a ternary 2-bit stream is ~0.47× the bytes of a 4-bit one. That is a statement about our quantisation, and it is the one worth making.
    • What the same-day measurement must therefore do: run row 3 again with the drafter on (k=7), on the same pack, the same day, and report both our baseline and our speculated number next to rows 1 and 2 — plus their speculation gain (1.16× over their own MTP head) next to ours, which is the only quantity that isolates the engine from the quantisation.

    Provenance for row 3, so it is re-runnable: binary ~/wt/serve-bin/t2s2-20261104/1bit (md5 fcc7fe6e…), pack ~/models/npu/bonsai2-27b-t2s2 → bonsai2-27b-NPU/model.q4nx, layout t2s v6, kernels kernels-m16 (18 ELFs), images bonsai2-t2s-v2; TTFT warm 1.348 s vs cold 252.9 s (cached=1999 of 2000).

  5. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Gate 2's reference is executable now — and it quantifies exactly how far our current rule is from the guarantee.

    ~/amd-oss/npu-re/tools/spec_accept.py (5e44c90) implements the rule and runs its own proof. The rule is the standard one and it is short:

    draw x ~ q                       (the draft's proposal)
    accept x with prob  min(1, p(x)/q(x))
    on reject, resample from  normalise(max(0, p − q))        ← the residual, and the whole point
    

    python3 tools/spec_accept.py --self-test, 200,000 draws per case:

    ok  a good drafter (q close to p): histogram matches the TARGET (max |dev| 0.0014, accepted 0.97)
    ok  a poor drafter (q very different): matches the TARGET (max |dev| 0.0011, accepted 0.50)
    ok  a drafter with no mass where p has some: matches the TARGET (max |dev| 0.0003, accepted 0.50)
    ok  an identical drafter: matches the TARGET (max |dev| 0.0012, accepted 1.00)
    ok  an identical drafter is accepted essentially always (1.000)
    ok  and a poor one is not (0.498) — so acceptance is measuring the drafter
    ok  the GREEDY rule does not preserve the target here (max |dev| 0.550) — the two guarantees differ
    

    The last line is the reason this file exists. On p = [.45,.35,.20] with q = [.10,.80,.10], the rule above reproduces the target to 0.0014, and the greedy
    argmax-compare we have today deviates by 0.550 — the same order as the target's own mass. So "greedy-exact", which #1371 measured and which is a real and useful
    property, is not the property FLM's line claims, and the two differ by more than sampling noise on a plausible drafter.

    Two practical consequences for the NPU route:

    1. The acceptance rule is one function, not a subsystem: given p (the target's probabilities at the k drafted positions) and q (the draft's), it returns
      accept/reject and, on reject, a residual draw. Our verify pass already produces p — the M=16 head returns full logits, not just argmax — so the NPU side needs no
      change for this; the change is on the sampler.
    2. Acceptance rate becomes a reported number per run rather than a hidden behaviour: 1.000 for an identical drafter, 0.50 for the poor one above. That is what makes
      the break-even arithmetic (>2.1 tokens/pass at k=7 with our 5.6 ms head) checkable against a real drafter instead of assumed.

    Next on my side, in this order: (a) the k=7 bit-identity arm — should be free given row independence, and it needs one ≤9-min npu-hold ticket; (b) port this rule onto
    the serve sampler and report the acceptance rate beside the tok/s. I have not written the engine side yet because the box is at 141 G, below the 175 G build floor, so
    a C++ change could not be compiled and tested today — and an uncompiled engine change is not something I will commit.

  6. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The same-model pack for F3 is now on the box, and it carries its own MTP head — both of which change this issue's plan. Detected by reading the container, no device, no build.

    ~/.config/flm/models/Qwen3.8-27B-NPU2 (17,097,334,824 B) arrived as an FLM model source (§9: "FLM is reference and a model source — its Q4NX HF repos
    run on OUR ELFs"
    ). Its contents:

    model.q4nx          17,097,334,824 B   header 111,648 B, 962 tensors   dtypes: I8 506 / BF16 360 / F32 96
    vision_weight.q4nx  (a vision tower ships alongside)
    config.json         H 5120, NH 24, L 64, V 248,320, tie_word_embeddings = False
    .fconst files       NONE — this is the modern Q4NX form, not the OGA dd_metastate form
    

    Three consequences:

    1. F3's same-model arm is unblocked. My earlier row said the comparison had to stay cross-model because no Qwen3.8-27B NPU pack existed here. It does now, and it is the same
      container family
      as our own 27B (bonsai2-27b-NPU/model.q4nx is I8 498 / BF16 257 / F32 96) — so the importer path is the one we already have, not a new format.
    2. It is UNTIED (tie_word_embeddings = False), so the F3 row covers the untied case as a bonus — which is also the case my W1c --ref oracle column was waiting on.
    3. It carries its own MTP head — 15 tensors, so F2's drafter for this model is inside the pack, not a separate 1.4 GB download:
    model.mtp.layers.0.input_layernorm.weight            BF16 [5120]
    model.mtp.layers.0.post_attention_layernorm.weight   BF16 [5120]
    model.mtp.layers.0.self_attn.k_norm.weight           BF16 [256]
    model.mtp.layers.0.self_attn.q_norm.weight           BF16 [256]
    model.mtp.norm.weight                                BF16 [5120]
    model.mtp.pre_fc_norm_embedding.weight               BF16 [5120]
    model.mtp.pre_fc_norm_hidden.weight                  BF16 [5120]
    model.mtp.fc.weight                                  I8   [160, 40, 4736]
    

    That last line is the whole FC projection of the MTP block; the rest is the per-layer norms and the two pre-FC norms. So the MTP-live-loop work I already own has a first-class
    target on the exact model F3 is measured against
    .

    One thing to fix in the engine, found by looking: own_mtp_head() (app/serve.cpp:986) gates on device == "hrx" and reads nextn_predict_layers from a GGUF. For a
    q4nx container the signal is the presence of model.mtp.* in the header — 15 tensors here — so the own-MTP detection would report none on this pack today. That is a
    one-condition change, and it is a prerequisite for the F2 arm on this model.

    What is still missing, unchanged: the accept rule on the NPU route (gate 2, reference at npu-re@5e44c90) and the k=7 bit-identity arm (gate 1). Both need a build to
    test, and the box is at 110 G free — below the 175 G floor — with the FLM models directory now 20.3 GB of that consumption (Qwen3.8-27B-NPU2 alone is 17.2 GB) and
    ~/scratch-packs at 38.6 GB. Worth knowing before another flm pull.

  7. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Gate 1's oracle already exists — and it is weaker than the gate. Found by reading it, before running anything.

    NPU_BATCH_CHECK=1 is exactly the right idea (npu/forward/src/q4nx_forward.cpp:3229): "run the same tokens sequentially first, capture the next-token argmax, clear the
    KV the sequential pass appended, then let the batched pass run and compare. One process, both paths, same device state."
    But what it compares is:

    3229-3244:  for (int m = 0; m < M; m++) if (!step(tokens[m], start_pos + m, seq)) return false;   // seq overwritten each m
                … if (seq_argmax < 0 || seq[i] > seq[seq_argmax]) seq_argmax = i;                      // argmax of the LAST row only
    3681-3682:  fprintf(stderr, "[batchcheck] M=%d start=%d seq_argmax=%d batch_argmax=%d %s\n",
                        …, seq_argmax == ba ? "MATCH" : "MISMATCH");

    So its guarantee is "batched and sequential agree on the final position's argmax". Gate 1 as I wrote it is "bit-identical to k separate decode steps" — all k positions, on the logits.
    Those differ in three ways, and each matters for speculative decode specifically:

    1. Argmax vs logits. Speculative acceptance needs the target's probability at the drafted token, not its argmax. Two passes can agree on argmax and differ on the distribution — and the
      accept rule (gate 2, npu-re@5e44c90) is exactly the thing that would consume the difference.
    2. Last row vs all rows. Only the final m contributes; rows 0..k-2 are never compared. Those are the positions the drafter proposed and the verifier must accept or reject — the ones
      F2 is about.
    3. Sequential steps at start_pos + m — the oracle does the right thing there (increments position), so the comparison is fair on the sequential side.

    Why this is worth a comment rather than a shrug: the oracle is a debug tool whose output will be quoted as evidence that batching is safe, and MATCH on one argmax at one position
    is not that evidence. It is the same class as tonight's other instrument findings — the vacuous journalctl window, the skip printed as ok, the check(x,x) golden — a tool that reports a
    narrower fact than its name implies.

    The change is small and it is where the value is: keep the k sequential rows' logits (a k × V buffer, or a per-row hash if size is a worry), then compare each batched row against
    its sequential counterpart bitwise, and print all k results plus the first differing (row, index, bit pattern). Then gate 1 becomes: k=7, 3 seeds, 3 repeats, every row identical —
    and the arm proves what the issue claims rather than a proxy for it.

    Next on my side, in order: (a) that oracle extension as a small engine PR with its own test (it is debug-only, so it cannot regress a route); (b) the k=7 run under one ≤9-min
    npu-hold ticket on the served 27B route; (c) the acceptance rule ported onto the sampler with the acceptance rate printed beside the tok/s.

  8. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Gate 1's measurement: I am deliberately NOT spending a device window on the old oracle tonight, and here is the reasoning — so it does not read as a skipped gate.

    The state needed to run it properly is not all in place:

    what state
    the per-row, bitwise oracle in review — #1813 (38160e9), build in progress
    the binary that contains it not built (disk 162 G, below the 175 G build floor)
    the device 4 queued NPU jobs; GTT 94.5 GiB from another lane's ticket
    the 27B pack's cold load 252.9 s cold vs 1.348 s warm — a 9-minute ticket only works on a warm cache

    I could run the old oracle with the prebuilt binary (NPU_BATCH_CHECK=1 on a 7-id prompt) and get its last-row argmax answer. I am not going to, because of what the previous
    comment here established: that oracle compares one argmax at one position, so a MATCH would be evidence I have already argued is too weak for this gate — and a device window
    on a 4-deep queue with a 94.5 GiB GTT neighbour is not the place to collect weak evidence. A MISMATCH would be worth having, but if batching were that broken we would have seen it
    in the shipped measurements (the M=16 set is the decode path, and 3.41 tok/s coherent output does not come from a pass that loses rows).

    What gate 1 looks like once #1813 lands: rebuild, then one ≤9-min npu-hold ticket, NPU_BATCH_CHECK=1 on a 7-id prompt, and the row is:

    [batchcheck] rows M=7 identical=7 differing=0        ← bit-identical on all seven rows, 3 seeds, 3 repeats
    

    or the first (row, index, seq=0x… bat=0x…) that differs, which localises the divergence to a position and a mantissa. Either answer is a measurement; the old line's MATCH is
    not.

    What I can do without a device, and will: the acceptance rule is already written and proven (npu-re@5e44c90, --self-test, 200k draws per case) — porting it onto the sampler
    is host work with a host test, and it only needs a build to compile, not a ticket. That is the next thing I will do once the disk floor is clear, and it is also the piece that
    makes the k=7 number meaningful: a bit-identical verify pass that accepts nothing is a slower route, and the acceptance rate is what separates the two.

  9. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Gate 1 is now blocked on a route, not on my PR. Tried the k=7 NPU_BATCH_CHECK measurement tonight with onebit built from npu/batchcheck-per-row (fb3d0f8) against the served 27B t2s2 pack, inside a 9-min npu-hold ticket:

    1bit npu-run: cannot read …/bonsai2-27b-t2s2/kernels-m16/layer_ctx1.elf
    

    npu-run requires the lane-based kernel set (layer_ctx{1,2,17}.elf, lmhead.elf, layer.pdi — its own --help says so, and npu/lane.cpp:69 reads layer_ctx1.elf), while every pack on this box ships the sealed full-ELF set (SEALED-T2S.elf, full_i8_*) driven by serve --device npu from plan.json. A find over ~/models and ~/scratch-packs for layer_ctx1.elf and for npu/lmhead.elf returns 0 each — the layout is not present anywhere.

    serve --parallel N does not substitute: it batches requests (one token each per pass), whereas gate 1 needs k rows of one prompt in one block/one launch so the rows are genuinely concurrent and the bit comparison tests something.

    Filed as #1862 with the repro and both routes. Gate 1 therefore waits on either a sealed-route batched entry point or a lane-layout pack; #1813 (the per-row oracle) is green and unaffected — it can be verified as soon as either exists.

  10. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    F2 status measured: every part exists except the wiring, and the contract on disk is greedy, not distributional

    Searched the tree for the pieces rather than assuming them.

    Present:

    • drafter — npu/mtp/draft.h (argmax-based candidate stream; argmax() at :65, used at :262 and :317);
    • verify API — Q4nxNpuForward::begin_verify(rows, start_pos) / commit_verify_rows(accepted) / gdn_apply_rows(from, to), documented as "a speculative driver outside this class begins a verify batch, then commits a prefix of it";
    • the deferral contract — npu/mtp/forward_base.h:20: "begin_verify() stores each row's recurrence inputs instead of applying them…";
    • a host test — tests/mtp_draft_test.cpp, asserting the property that matters: "k=%d head=%s: tokens identical to plain decode" and "final state == prompt+emitted, no skipped positions".

    Absent:

    $ grep -rn 'begin_verify|commit_verify_rows|gdn_apply_rows' --include=*.cpp --include=*.h . | grep -v q4nx_forward
    tests/mtp_draft_test.cpp:120, :530, :742   # prose, in comments
    npu/mtp/draft.h:125                        # prose, in a comment
    $ grep -rln 'npu/mtp' --include=*.cpp app/
    (nothing)
    

    No production caller. app/ does not reference npu/mtp/ at all. So F2 is wiring, not numerics — consistent with #1862 (no CLI can hand k ids to one pass either), and it means the two gaps are the same gap seen from both ends.

    Correction to my own gate-2 framing

    I had written gate 2 as "port spec_accept.py's rule onto the sampler". That was mis-aimed: there is no sampler-side acceptance in the NPU path to port onto, because there is no speculative loop there at all. On --device hrx/cpu the loop is llama.cpp's (--spec-type draft-dflash), which is not our code. So the acceptance rule arrives with the wiring rather than landing on an existing host.

    And the bar needs a decision, because the tree's contract is narrower than the owner's wording

    tests/mtp_draft_test.cpp:21 states the contract as "emitted tokens are identical to plain greedy decode", and the drafter/verifier are both argmax. That is the right bar at temperature 0, and it is a strong one — but F2 asks that the output "follows the main model distribution", which is a different (and, above temperature 0, the only achievable) property:

    • at temp 0, "identical to plain greedy decode" ⇒ distribution-preserving trivially;
    • at temp > 0, token identity is impossible for any drafter that is not the model itself, so the exact rule is accept min(1, p/q), else resample from normalise(max(0, p−q)). My spec_accept.py (5e44c90, 200k-draw self-test: 0.0014 / 0.0011 / 0.0003 error on good/poor/no-mass drafters) also measured that the greedy rule deviates 0.550 from p on p=[.45,.35,.20], q=[.10,.80,.10] — so a greedy accept in a sampling path is not a small approximation.

    So gate 2 is now: decide which bar the shipped NPU speculative path promises. If greedy-only, the existing identity test is the whole contract and the acceptance rule never applies; if temperature > 0 is supported (it is, on the sampling path), the exact rule must be implemented with the acceptance rate printed beside tok/s, and the identity test becomes the temp-0 special case rather than the contract.

  11. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Cross-link: #1903 removes the reason I could not measure gate 1. Its defect was that max_batch_rows() looked for a design in the per-role ctx_[]/ctx16_[] arrays, which the sealed route never populates — so every sealed pack (including the served 27B) reported batch width 1, and every "batched" measurement I have taken tonight was really a chain of single-row launches with nothing in the log to say so.

    It now takes the MIN over the loaded design widths ("an over-wide launch reads past the activation BO (engine#1372 — memory corruption, not a slow path)") and prints [forward] batch width N row(s) per launch at load, so a batched run is distinguishable from one that never batched.

    So gate 1 has two halves, in this order:

    1. the width is real and stated — npu: the batch width is the minimum loaded design width, not the first (engine#1900 review) #1903;
    2. something can ask for k rows in one pass — npu-run cannot drive any current pack: it requires the per-role layer_ctx*.elf set, every pack ships the sealed full-ELF set #1862 (npu-run needs the per-role layer_ctx*.elf set that no pack on this box has) or the sealed-route step_batch_all entry point.

    With both, the row I am waiting for becomes obtainable: [batchcheck] rows M=7 identical=7 differing=0 next to a [forward] batch width 16 line that says the pass was one launch.

  12. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Decision: the sealed route's speculative contract is distribution-following, with greedy identity as its temperature-0 special case

    This is the call I said was mine to make in the earlier comment, and it settles what the wiring PR must implement. The owner's F2 wording — output follows the main model distribution — is the contract; tests/mtp_draft_test.cpp:21's "tokens identical to plain greedy decode" is the special case, not the general rule.

    The rule

    For each of the k draft tokens, with p the main model's distribution at that position and q the drafter's:

    1. draw x ~ q;
    2. accept with probability min(1, p(x)/q(x));
    3. on the first rejection, sample the residual normalise(max(0, p − q)) and stop drafting; commit the accepted prefix.

    At temperature 0 every distribution is a one-hot at its argmax, so step 2 becomes p(x)/q(x) ∈ {0, 1, ∞} and the rule degenerates exactly to "accept iff the drafter's token is the argmax" — i.e. the existing identity test is the temp-0 instance of this rule, not a competing one.

    The numbers that force this

    ~/amd-oss/npu-re/tools/spec_accept.py (npu-re@5e44c90) self-tests the rule at 200k draws — good drafter 0.0014, poor 0.0011 (accept 0.50), no-mass 0.0003, identical 0.0012 (accept 1.000) max deviation from p. The same script measures the greedy acceptance rule at 0.550 deviation from p on p=[.45,.35,.20], q=[.10,.80,.10]. So a greedy accept inside a sampling path is not a small approximation; it is a different model's output.

    What this obliges

    • The sampler, not the verifier, owns acceptance. begin_verify / commit_verify_rows only decide how many leading rows are kept; the rule above decides which, so it lives where p and q are both available.
    • Print the acceptance rate beside tok/s, always. 1.12 tokens/pass without the accept rate cannot be told apart from a drafter that is right 12 % of the time and one that is right 90 % of the time with a different k.
    • The temp-0 identity test stays and is the regression arm for the degenerate case: if the rule is implemented and the temp-0 path no longer reproduces plain decode exactly, the implementation is wrong.
    • A temp>0 distributional arm is the new obligation: sample N times with a drafter that is deliberately wrong (e.g. q shifted one token off p), and check the empirical distribution matches p within the binomial error spec_accept.py already characterises. That arm fails for the greedy rule and passes for this one, which is the discriminator the contract needs.

    What this does not decide

    Which entry point carries it (engine#1862 — npu-run --batch on the sealed route, or the same thing inside forward-serve), and the draft length k (measured, not chosen: the break-even accept is >1.45 at k=4 and >2.4 at k=8 on the sealed M=16 head's 5.6 ms/draft).

    Recorded here so the wiring PR has a spec rather than a question, and so that "we ship DFlash parity" cannot be answered by a greedy loop that happens to produce the same tokens on the prompts someone tried.

  13. 23 remaining items

  14. bong-water-water-bong commented on Nov 12, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Device rows owed since the NPU went off, measured 2026-11-12 (14:45-15:22Z). Model: served bonsai2-27b-t2s2 with the DFlash2 27B draft. Every run was inside one npu-hold ticket; the amdxdna error count did not change across the three tickets.

    The host was loaded by other lanes (load average about 26 on 32 cores), so each figure is a single run, not an A/B/A/B pair. Plain is the best plain configuration.

    1. The loop through mtp::run_from + DFlashHead against the hand-written loop (48 tokens, greedy)

    Prompt / arm tok/s Drafts accepted per round Base passes per token Equals plain greedy
    p0 plain 2.59 n/a 1 n/a
    p0 hand-written loop, k=7 5.11 6.00 n/a 48/48
    p0 run_from, k=7 5.21 5.86 0.146 48/48
    p0 run_from, k=5 6.78 4.33 0.188 48/48
    p0 run_from, k=3 4.43 2.69 0.271 48/48
    p1 plain 2.28 n/a 1 n/a
    p1 run_from, k=7 3.18 2.00 0.333 48/48

    The engine's own loop reproduces the hand-written rounds token for token at the same speed. On p0, k=5 is the fastest point: 2.6× plain.

    2. forward-serve, chat route (code-edit prompt 40 tokens, prose prompt 48)

    Arm Code Prose [spec] line (code / prose)
    plain, greedy 2.93 3.20 n/a
    --spec-lookup 4 --spec-fallback 2, greedy 7.43 (2.5×) 3.38 24/48 drafts kept / fell back after round 3
    --spec-draft dflash2-27b, greedy 9.84 (3.4×) 2.71 (0.85×) 35/35 kept / 26/140 kept, no fallback
    plain, temperature 0.7, seed 42 2.91 3.28 n/a
    --spec-lookup 4 --spec-sample, temperature 0.7 7.15 (2.5×) 2.50 (0.76×) 24/52 kept / 2/45 kept, 33 plain steps

    All four greedy speculative replies are byte-identical to plain's. These runs use #2266 + #2267 + the grammar commit on top of main, so the head-width row fix is in: these are the re-runs of the earlier spec rows.

    What the prose column says: speculation helps when drafts are kept, and costs about 15-25 % when they are not. The lookup arm with --spec-fallback 2 falls back and stays at plain speed. The DFlash arm ran without a fallback, and the sampled arm had none set. So the loss on prose is configuration, not a defect, but it means a fallback should be the default whenever a drafter is on. That is the next code change on this issue (host-side, after #2266/#2267 merge, inside the 2-PR cap).

  15. bong-water-water-bong commented on Nov 12, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Follow-up to the rows above: the fallback default, measured A B A B in one ticket (served 27B + DFlash2 draft, prose prompt, 48 greedy tokens, replies byte-equal).

    DFlash prose Run 1 Run 2 Mean
    --spec-fallback 0 (21 rounds, 26/140 drafts kept) 1.93 3.09 2.51 tok/s
    unset → default 3 (falls back after round 5) 3.89 3.86 3.88 tok/s

    A separate A B C A B C ticket found plain decode on the 4-row and 8-row sets equal within the run-to-run scatter (2.68 and 2.62 tok/s on prose), and DFlash with the default fallback at plain speed (2.68). The single-run prose ratios in my earlier comment are therefore noisy. The A/B pairs above are the figures to use.

    The change (an unset --spec-fallback defaults to 2 tokens per round for lookup and 3 for a draft; an explicit 0 keeps it off) is committed and tested host-side. It will open as a PR when one of #2266/#2267 merges (2-PR cap).

  16. bong-water-water-bong commented on Nov 12, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    CLOSE CONDITION

    Bar (the four gates in the body):

    1. One-pass verify at k=7 on the served 27B route, bit-identical to 7 sequential steps, 3 id sets × 3 repeats. Met: 9/9 (issuecomment-6771816699; on main through npu: the speculative loop end to end on the served 27B, and a fully accepted verify keeps its own state (refs #1781) #2240).

    2. Distribution-preserving sampling: the acceptance rule is the target/draft probability ratio, and the token histogram of the speculative sampler matches plain sampling. Met: on main in serve: speculative decode serves logprobs from each token's own row; verify rows strided by the head width (refs #1781) #2267 (which carried serve: --spec-sample speculates on sampled requests and keeps their distribution; verify rows cut to the tokenizer bound (refs #1781) #2266).

    3. F3: our 27B NPU decode at or above FLM's 2.84 tok/s, same pack, same day. Met: plain decode 3.3, and 4.1-4.2 with the 4-row decode set (issuecomment-6798173058).

    4. The two named pairs run speculatively, with every reply byte-equal to plain:

      • 27B + DFlash2: 2.93 → 9.84 tok/s on code, 3.20 → 3.88 on prose;
      • Bolt-E4B + DFlash: 0.557 → 3.40 tok/s on code, 0.562 → 0.830 on prose.

      Met in npu forward: Gemma4 verifies a draft in one batched pass; KV row width per layer (fixes #1780) #2303. The Bolt pair needed Gemma4's batched verify body, which is in that PR.

    Left between now and the bar:

    1. npu forward: Gemma4 verifies a draft in one batched pass; KV row width per layer (fixes #1780) #2303 merges (READY ×2 at 4d10370, CI green, mergeable). It carries Fixes #1781.

    No other PRs and no new slices.

    One caveat on the comparison: FLM states its Qwen3.8 figure against its MTP head, and the served pack has none, so our 27B rows compare against plain decode. The ORCH line in my state file proposes that as the bar.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    backlogReal, but does not hold the release build

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions