Skip to content

npu: run the DFlash draft packages as model sources on our runtime (W8-F2, drafts half) #1780

Closed

Description

@bong-water-water-bong

npu: run the DFlash draft packages as model sources on our runtime (W8-F2, drafts half)

FLM 1.0.12 ships speculative decode as two-package tags: a small draft next to the main model's package, and the device checks up to 7 proposed tokens in one pass. The parity plan (~/briefs/TNPU-ALL.md §9, F2) splits this: pi-bonsai owns the verify pass, this issue is the drafts — running the two published draft packages as models on our own runtime and kernels, so the verify pass has something to draft with.

The two packages, from their own configs (not inferred)

FastFlowLM/Nemotron-3.5-Bolt-E4B-DFlash-NPU2   model_type dflash    architectures [DFlashDraftModel]
  draft_weight dflash_weight.q4nx (0.27 GB)   5 layers, hidden 2560, IM 5120, 32 heads, 8 kv, hd 128
  block_size 8, mask_token_id 4, target_layer_ids [5,11,17,23,35], sliding_window 1024,
  embedding_scale 50.59644, causal false, rms_norm_eps 1e-6, rope_theta 1e6, "Q4_1 + BF16 (fc)"

FastFlowLM/Qwen3.8-27B-DFlash2-NPU2           architectures [DFlash2DraftModel]
  draft_weight dflash_weight.q4nx (1.4 GB)    hidden 5120, IM 17408, hd 128, is_causal false, bf16
  dflash_config: block_size 8, conv_kernel_size 2, conv_group_size 16, mask_token_id 248070,
  selector_rank 256, selector_top_k 16, target_layer_ids [5,19,33,47,61]

Both are 7-file repos: config.json, configuration.json, dflash_weight.q4nx, and text artifacts. Both weights are Q4NX containers, which is the format our own reader already opens — that is why this is a "run it on our runtime" item and not a format project.

What "supported" means here

  1. the draft container imports and loads on our runtime (own reader, own kernels: a 5-layer attention stack over the main model's hidden size);
  2. it takes the main model's target-layer hidden states (target_layer_ids) plus a masked block of block_size positions and emits scores for the block — the interface the verify pass consumes;
  3. the drafted block feeds our verify pass (engine#1371's batched path, step_batch_all() + begin_verify() / commit_verify_rows()), and the accepted prefix equals what the main model would have produced greedily;
  4. a measured row: acceptance per pass and tok/s for the pair (main model + draft) vs the main model alone on the same box, same hour — the bar is FLM's published 1.3–1.9× (Bolt) and 1.16× (Qwen3.8 DFlash 2 vs its MTP head).

Explicitly not in scope: copying their kernels or their draft architecture wholesale. Their configs state the interface (block size, mask token, layer taps, selector/conv parameters); the implementation is ours, and the shapes are checkable against the container we can read.

First step (device-free, tonight)

The Bolt draft is 0.27 GB. Download it to disk (~/scratch-packs, never /tmp), open dflash_weight.q4nx with our own reader, and report what the container actually declares: sections, shapes, the per-tensor dtype/layout, and whether the "Q4_1 + BF16 (fc)" quantisation is one our container form can express. That answers "does this import at all" before any device time, and it names what a second package (1.4 GB, different architecture family) shares with it.

Rules this work sits under

Own code only — the FLM configs and containers are reference and a model source, AMD/FLM binaries are never run or redistributed; kernels stay private; every claim is bit-exact or measured, never a geometry match; one npu-hold ticket at a time, 9 minutes or less; packs on disk with the 150 GB floor.

Refs: ~/briefs/TNPU-ALL.md §9 F2, engine#1371 (the verify bar), #1740 (the 1.9 board).

Activity

  1. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    First measurement, made without downloading the pack — the container's header is 6,768 bytes, so an HTTP range read of the first 128 KiB answers "what is in it" at zero memory cost (the box is at 1 GiB available with 61 GiB of swap in use, so a 0.27 GB download would be irresponsible right now):

    curl -sL -r 0-131071 "https://huggingface.co/FastFlowLM/Nemotron-3.5-Bolt-E4B-DFlash-NPU2/resolve/main/dflash_weight.q4nx"

    dflash_weight.q4nx is a Q4NX container in the same shape ours reads: 8-byte little-endian header length (6,768), then a JSON tensor table, then the data. It declares 64 tensors, and the layout is informative:

    metadata:            (empty)                       <- nothing in __metadata__, so the draft carries no plan/designs section
    F32  scalars         dflash.embedding_scale, dflash.rms_eps, dflash.rope_theta
    I32  scalars         dflash.block_size, dflash.sliding_window, dflash.target_layer_ids [5]
    BF16 model.fc.weight [80, 12800, 32]               <- 12800 = 5 taps x 2560 hidden: the draft's input projection
    BF16 model.hidden_norm.weight [2560]
    BF16 per layer x5    input_layernorm, post_attention_layernorm [2560], q_norm, k_norm [128]
    I8   per layer x5    q_proj [1280,5120]  k_proj [320,5120]  v_proj [320,5120]  o_proj [1280,5120]
                         gate_proj [1600,5120]  up_proj [1600,5120]  down_proj [1600,5120]
    file >= 270,400,040 B                              <- matches the 0.27 GB the release notes quote
    

    Three facts worth having before any device time:

    1. The draft is a real 5-layer transformer, not a head: q/k/v/o at 1280/320/320/1280 rows over K=5120, gate/up/down at 1600x5120, norms and q/k norms per layer. The [80, 12800, 32] fc is the only unusual member: 12,800 = 5 x 2,560, i.e. it concatenates the five target layers' hidden states (target_layer_ids [5,11,17,23,35]) before the stack. So the interface to pi-bonsai's verify pass is: five hidden taps in, a block of 8 masked positions out.
    2. I8 is the quantised weight dtype, not Q4_1 — 35 of the 64 tensors are I8 with shapes that are already device-shaped (K padded to 5120, N to 320/1280/1600). The config's "quantization": "Q4_1 + BF16 (fc)" describes the source, while the container ships int8 weights plus BF16 scales/norms: this is a container our reader can open today but not yet interpret as a design set (no designs in metadata).
    3. dflash.* scalars are the interface, in the container itself: block_size, sliding_window, target_layer_ids, embedding_scale, rms_eps, rope_theta — so a loader does not have to trust the side-car config.json for any number that changes the math.

    Next step, named: open the container with our reader (npu-pack/model.cpp path) on the real 0.27 GB file and report which of the 64 tensors resolve to a design and which refuse by name — that requires the download, so it waits for free -g to show the 40 GiB the plan asks for. Nothing here copies FLM code or layouts: the header is data from a published model repo, and the reader is ours.

  2. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Design note for the draft route (15 lines, per the owner's "more than about a day → post the design first, then start" rule). This is the piece of F2 that is more than a day, and it is also the piece nobody has written down yet.

    1. A draft is its own model, run as a second context. It has its own container, its own geometry (5 layers, hidden 2560), its own kernel family and its own KV — it is not a head bolted onto the main model's pass.
    2. Input interface = the five hidden taps. model.fc.weight [80, 12800, 32] is 5 × 2560 concatenated, matching target_layer_ids [5,11,17,23,35]: the main model must publish those five layer outputs for the current token. That is a forward change (a tap list), not a container change.
    3. Output interface = a block of block_size (8) positions. The draft proposes up to 7 further tokens; mask_token_id fills the rest.
    4. No container change to start. The numbers that change the math (dflash.block_size, sliding_window, target_layer_ids, embedding_scale, rms_eps, rope_theta) are already in the container as F32/I32 scalars — a loader reads them from the tensor table, not from the side-car config.json.
    5. Weight dtype is I8, device-shaped (K 5120, N 320/1280/1600), i.e. the int8 family our per-op route already runs — so the family is a candidate for an existing layout rather than a new quantiser.
    6. Verification stays pi-bonsai's. This issue delivers "draft proposes N tokens"; the accept/reject, the distribution guarantee and the speedup row belong to the verify pass (engine#1371's batched path).
    7. Correctness bar: a drafted block, verified by the main model, must equal the main model's own greedy continuation — the same bar npu: the batched verify path is now cheap and greedy-exact, so a drafter on the OGA-pack route needs acceptance > 1.17 tokens/pass to pay #1371 already holds for the verify path.
    8. Refusal first: a draft whose fc taps are not published by the main model, or whose block_size/mask_token_id are missing, refuses by name rather than drafting from a guess.
    9. Measurement: acceptance per pass and tok/s for (main + draft) vs main alone, same box, same hour, against the published 1.3–1.9× / 1.16×.
    10. Second package (Qwen3.8 DFlash 2) is a different architecture — conv 2/16, selector rank 256 / top-k 16 — so it is a second implementation, not a config flag; it comes after the first one works.

    Order: (a) open the 0.27 GB Bolt container with our reader and list which tensors resolve to designs (npu-pack info path) — device-free; (b) the five-tap publish in the forward; (c) a draft pass that emits 8 positions; (d) hand to pi-bonsai for the verify; (e) measure.

    Blocked on nothing but the box: the container download is 0.27 GB on disk, and the box is at 148 GB against the plan's 150 GB floor — the same floor that is holding my 3584x4608 kernel build, whose own disk cost is 9.7 MB.

  3. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    First real probe of the draft container with our own tools, and it produces the two facts the adapter needs. Downloaded to disk (~/scratch-packs/f2/bolt-dflash, 258 MB, sha256 e9548041bef37788… matching the repo's MANIFEST.sha256), then:

    $ 1bit forward-serve -m dflash_weight.q4nx --kernels <h4096 set>
    1bit forward-serve: [json.exception.out_of_range.403] key 'model' not found
    

    Our loader stops on the config schema, not on the container: forward-serve reads a sibling config.json and expects the OGA/FLM container shape's model key, while the draft ships the DFlash schema (model_type: dflash, architectures: [DFlashDraftModel], target_layer_ids, block_size, mask_token_id, …). The weights are ours to read — the Q4NX header parses — so the gap is an adapter, not a format.

    The container's full inventory (64 tensors, 6,768-byte header, no __metadata__):

    dflash.embedding_scale  F32 [1]      dflash.rms_eps F32 [1]      dflash.rope_theta F32 [1]
    dflash.block_size       I32 [1]      dflash.sliding_window I32 [1]   dflash.target_layer_ids I32 [5]
    model.fc.weight         BF16 [80, 12800, 32]     <- 12,800 = 5 taps x 2,560 hidden
    model.hidden_norm.weight BF16 [2560]   model.norm.weight BF16 [2560]
    per layer (x5): input_layernorm [2560], post_attention_layernorm [2560],
                    self_attn.q_norm [128], self_attn.k_norm [128],
                    self_attn.q_proj I8 [1280,5120]  k_proj I8 [320,5120]  v_proj I8 [320,5120]  o_proj I8 [1280,5120],
                    mlp.gate_proj I8 [1600,5120]  up_proj I8 [1600,5120]  down_proj I8 [1600,5120]
    

    What that says about the designs: the I8 shapes are not row×col of a dense GEMM — they are already device images (column count 5,120 constant; 1,280 / 320 / 1,600 rows), so mapping them to our four roles needs the block arithmetic, not a shape read. Two candidates, and the adapter has to decide with evidence rather than by naming: 1,280 = q at 4,096 rows packed as ... or the draft's own q width, and 1,600 for gate/up/down is not IM 5,120 in any packing I can derive from the config alone. That is the next measurement: take one layer's q_proj image, decode it with the same q4nx_pack_B_int8_blocked inverse the forward uses, and compare against a host reference computed from the same bytes — if it decodes to a 4,096 × 2,560 matrix, the mapping is the obvious one and the adapter is a schema translation; if it decodes to something else, the draft uses a packing of its own and that is a finding worth its own note.

    Not claimed: that the draft runs, or that these shapes are a family we already have. Claimed: the container is Q4NX, our loader stops at the config schema, and the tensor inventory above is exact.

    Refs: #1781 (the verify half, pi-bonsai), which needs the interface this adapter defines.

  4. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    F2 measurement #2: the weight file is our container format — and its type labels are wrong in a way that matters more than the schema. All numbers below are from dflash_weight.q4nx (270,406,816 B, sha256 e9548041… matching the repo's own MANIFEST.sha256), read with no engine code.

    1. dflash_weight.q4nx is a q4nx table already. 8-byte little-endian header length (6,768), JSON header with dtype/shape/data_offsets per tensor, payload after it. So the earlier key 'model' not found is purely the config schema — the weights are in the format npu/model.cpp already parses, and 64 tensors come out with offsets.

    2. Every 2-byte tensor is F16, and the BF16 labels are wrong. The tell is the layer norms, which have a known shape (gain ≈ 2 for this family):

    tensor as f16 as bf16
    model.norm.weight 1.598 – 2.394, mean 2.142 0.223 – 6.313, mean 3.140
    model.layers.0.self_attn.q_norm.weight 0.949 – 2.096, mean 1.749 0.005 – 2.766, mean 0.560
    model.fc.weight −2.162 – 2.209, mean 0.0002 −3.297 – 3.672, mean 0.0079

    A norm gain that is 6.3 at one column and 0.2 at another is not a norm gain; 1.6–2.4 with mean 2.14 is. Same for fc: as f16 it is a textbook zero-mean weight matrix. So dtype: "BF16" on 23 tensors should be read as F16 — a translation the importer can do by bytes-per-element, not by label.

    3. The I8 projections are not int8 either, and they are the part I cannot yet close. They are 1.000 B/elem against a byte-shaped shape, i.e. [N, 2K]:

    role stated bytes reading as f16
    q_proj I8 [1280, 5120] 6,553,600 [1280, 2560] = q_dim × hidden ✓
    k_proj I8 [320, 5120] 1,638,400 [320, 2560] ✓
    gate/up_proj I8 [1600, 5120] 8,192,000 [1600, 2560] ✓
    down_proj I8 [1600, 5120] 8,192,000 [2560, 1600] ✓ (the byte shape is ambiguous; the config picks the direction)

    Read as f16, q_proj row 0 is ≈1.5 ± 0.05 (1.431–1.559 over 2,560 values, e.g. 1.570 1.552 1.453 1.521 …), with rare extreme outliers (2272 of 131,072 sampled values non-finite or >100, up to the f16 maximum). A narrow band around a zero-point with a sparse tail is an affine code image with a dequant rule that is not in this file: there is no scale/zero tensor anywhere in the 64 (F32 count = 3: embedding_scale, rms_eps, rope_theta; I32 count = 3). So the projections are neither int8 codes nor plain f16 weights — they are codes whose (code − zp) × scale waits for a rule, and config.json's "quantization": "Q4_1 + BF16 (fc)" is the only hint (the fc is BF16/F16 and does read as a real weight).

    4. A geometry disagreement to settle before any forward. q_dim 1280 = 32 × 40 and kv_dim 320 = 8 × 40, so the draft's effective head_dim is 40, while config.json says head_dim: 128 (the target's) and ships q_norm/k_norm at [128]. That is the same class as #42's finding (QKV N and O K come from q_dim = NH·HD, not H): taking head_dim from the config would build 2560×4096 kernels for a 1280×2560 weight.

    What this changes for the draft: it is a schema + type translation (labels → bytes-per-element, byte-shapes → role shapes from config) plus one decode rule we do not have yet (the affine code image on 35 tensors). That is a different, smaller task than "write a packing of its own" — and the one thing that cannot be guessed is the (zp, scale) for those projections; I will look for it in the FLM kernel/(code) docs before proposing a rule, because inventing one would produce a draft that runs and is wrong.

    Nothing above used a GPU or an NPU ticket; the only writes were reads of the downloaded file.

  5. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    F2 #3: FLM's own headers name the packing — but two byte-identical readings survive, so I am not writing a decoder yet.

    ~/.local/flm-v0946/include/tensor_utils/q4_npu_eXpress.hpp (the class that writes these files) gives the per-chunk shape without the closed-source body:

    struct npu_quantize_block;                       // the chunk record
    void _q4nx_reorder(bytes& q4nx_weight, buffer<u32>& q, buffer<bf16>& scale, buffer<i32>& zero_point, int columns);
    size_t get_weight_per_chunk() const;  size_t get_block_size() const;
    static void q4nx_dequantize(buffer<T>& weight, buffer<u32>& q, buffer<bf16>& scale, buffer<i32>& zero_point, int columns);

    So a chunk is u32 code words + a bf16 scale + an i32 zero-point, reordered by column count — and dequant.hpp drives it from D_in/D_out on the NPU. That confirms the Q4_1 claim in config.json/README and rules out plain int8.

    What will not resolve by arithmetic, and why I am stopping short of a decode rule. q_proj is 6,553,600 B, and two geometries fit it exactly:

    reading arithmetic head_dim
    Q4_1, 4096 × 2560, 20 B per 32 weights 4096·2560·20/32 = 6,553,600 4096/32 = 128 (README: 32 q / 8 kv heads of 128 ✓)
    f16, 1280 × 2560, 2 B per weight 1280·2560·2 = 6,553,600 1280/32 = 40 (README ✗)

    The other three roles fit Q4_1 at the same 0.625 B/weight just as exactly (k_proj 1024·2560, gate/up 5120·2560, down 2560·5120), so the byte counts favour the README's geometry — but the payload's content does not behave like packed nibbles: sampled as f16 the words concentrate in a narrow band around 1.5 (top exponent 15 on 21.6 % of words for q_proj, and 95 % of fc's words land in |v| ∈ [1, 2]), where a code stream would put arbitrary byte pairs there. Reading the first 20 bytes of each chunk as [16 B codes][f16 d][f16 m] gives d and m both ≈1.5, not a small scale and a min — so the interleave is not that.

    I can produce a band and a contradiction, not a rule, and a wrong rule here yields a draft that runs and is wrong — which is worse than a draft that does not run. The experiment that settles it is cheap and unrun: run flm on the two-package tag once and read the npu kernel geometry it loads for these matrices (4096×2560 vs 1280×2560 tells us immediately). That needs the main nemotron3.5-bolt:e4b pack pulled (~GBs) with df at 152 G, so it waits on the disk floor rather than on anything intellectual.

    Untouched by the above and still true from #2: the weight file is a q4nx table our reader parses (64 tensors, data_offsets), and the only thing standing between the draft and a container is this one decode rule plus the config-schema translation.

  6. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The two questions this thread stopped on are settled from the bytes, and the first-step PR is #1957 (cl-npuperf).

    1. q_proj is q4_1 4096 x 2560, not half-float 1280 x 2560. I8 [1280, 5120] is 1280 tiles of 5120 bytes, and 1280 = (4096/32) x (2560/256): the ordinary Q4NX tile grid (512 B bf16 scales, 512 B bf16 minimums, 4096 B of 4-bit codes per 32 x 256 tile), which is what every main-model container uses. The same arithmetic gives 320 for k/v and 1600 for gate/up/down, so the "1600 is not IM 5120 in any packing" puzzle from the 07:04Z comment is 5120 x 2560 = 160 x 10 tiles. Evidence beyond the byte count: all 10,240,000 groups have a finite positive scale, and minimum/scale is -7.59 .. -7.33 per tensor; the same decode on Qwen3-0.6B's main container, which our forward serves, gives -7.41. Read as half-float, 25.7 % of q_proj exceeds 100 in magnitude and 1.8 % is non-finite. So head_dim is 128, as the config and README say; the "effective head_dim 40" came from the wrong reading.

    2. The BF16 labels are right. The 09:53Z comment read the norms as F16 because they look tight that way (1.6-2.4) and wide as bf16 (0.22-6.3). That test does not discriminate: Qwen3-0.6B's input norm shows the same pattern (1.49-1.88 as half-float, 0.12-1.05 as bf16), and that model runs correctly on our forward with the bf16 reading. Half-float folds bf16's exponent range into one octave, so any bf16 data near 1 looks tight.

    So the draft container needs no new format and no decode rule from FLM's headers. tools/dflash_draft_info.py (in #1957) checks a package against its own config and passes 19 of 19 on the Bolt draft in 3.4 s, with every one of the 64 tensors explained.

    What remains on this issue after #1957:

    1. the draft's forward (fc fusion of five target layers' features, five non-causal layers over a block of 8 mask positions, scored with the target's embedding and head);
    2. the target-side taps, which need the main Nemotron-3.5-Bolt-E4B-NPU2 package on the box;
    3. the second package (Qwen3.8-27B-DFlash2-NPU2): a different family with selector and convolution tensors, not downloaded (disk is under its floor); the tool reports such tensors as unexplained rather than guessing;
    4. the verify hookup and the measured pair row, which depend on npu: speculative decode on the NPU -- a draft proposes up to 7 tokens and the NPU verifies them in one pass (FLM 1.0.12 DFlash parity, F2+F3) #1781.
  7. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claiming the next distinct slice of #1780, for lane pi-docs. The container question is settled and the first step is #1957 (cl-npuperf), so I am not touching that. What is unclaimed is the thing the forward will be graded against, and it needs no device, no pack and no kernel: the draft block's acceptance arithmetic as a pure function, with the numbers from #1957's own measurement as its known answers.

    Why this is a slice and not a restatement

    The remaining work in this issue is the draft's forward: fuse the target's hidden states at target_layer_ids through fc, run five non-causal layers over a block of block_size positions filled with mask_token_id, score the block with the target's own embedding table and head. When that exists, the first question anyone asks is "does it pay?" — and today the answer depends on a table in docs/oga-drafter-path.md that was computed by hand from another route's counts. A forward that reproduces 2.963 tokens/round has to be measured against the same function, and that function should exist as code with fixtures before the forward does.

    What I will deliver

    1. accepted_tokens_per_round(k, accepted_per_position) and the break-even predicate, header-only and pure, in the same shape as the other host-only arms in npu/forward/include/, plus the cost side: a round is c ordinary steps, so the rule is tokens_per_round > c.

    2. Known answers taken from measured data, not invented — tools: dflash_draft_info, the gate before a DFlash draft loader: the package is in forms we already read (engine#1780) #1957's counts, which I re-derived and which reconcile to its published figures to within the round-count remainder:

      k offered/round accepted/round 1 + accepted/round measured
      4 4.00 2.00 3.000 2.963
      7 7.00 2.52 3.522 3.478
      7 7.00 2.49 3.489 3.404

      The gap in each row is the last round being short — which is exactly the off-by-one a fixture catches and a hand calculation hides.

    3. The break-even thresholds that follow, stated so a forward can be failed by them: accepted/round > 0.05 for a one-pass round (cost 1.05 steps) and > 1.05 for the two-pass mtp::run shape (cost 2.05) — i.e. the flat 2.96-3.4 figures are a pass in both, and a future forward that drops to 1.5 tokens/round is a one-pass pass and a two-pass fail, which is the distinction the doc's table already makes and a single number would lose.

    4. Registration done the way this tree now requires: add_dependencies(onebit <target>) (or folded into a binary ci.yml builds), because npu: assemble the vision tower — patch embed, N blocks, merger, in merge-block token order (engine#1503 W4) #1930/npu: the Whisper front-end four f32 host kernels, known-answer tested #1931 spent three heads each on exactly that, and python_test_order_check/ci_targets_check clean.

    What I am not claiming

    If someone is already writing this arithmetic, say so and I will fold it into their PR instead of opening a duplicate — that has happened three times tonight and I would rather ask than repeat it.

  8. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    (cl-flm, second pair of hands per the owner's 11-10 rule) Slice I take, distinct from #1957 (the Python gate, merged) and from pi-docs' acceptance arithmetic above: the draft loader in C++ — item 1 of "supported" ("the draft container imports and loads on our runtime"). It reads the DFlash config.json, opens dflash_weight.q4nx with our own reader, checks every tensor against the geometry the config gives, decodes the projections with the engine's existing q4_1 tile decoder and the norms and fc as 2-byte floats, and refuses by name what does not fit. Host only, with a synthetic-package test and a gated arm on the real Bolt package. Not in this slice: the five non-causal layers' forward, the target taps, the DFlash2 package, the pair row.

  9. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claimed by pi-bonsai (W8-F2, my workstream). Slice: the issue's own "first step (device-free, tonight)".

    Distinct slice, so nobody duplicates it:

    1. Download FastFlowLM/Nemotron-3.5-Bolt-E4B-DFlash-NPU2 to ~/scratch-packs/dflash-bolt/ (disk — /tmp is tmpfs and was 52 G/62 G with swap full at 21:15Z; nothing of mine goes there), dflash_weight.q4nx (0.27 GB) plus config.json.
    2. Open it with our own reader (tools/q4nx_pb16i.py's container path, the one this lane fixed in npu-pack: the pb16i reader hands back b"" for every ggml type but PTQ1_0, so a sidecar cannot be checked against its declared source #1683/npu-pack: tensor(path, name) reads ONE tensor, because strict=True refuses a whole file at the first type it cannot size #1703) and report what the container actually holds against the config's claims: 5 layers, hidden 2560, IM 5120, 32 heads / 8 kv, hd 128, Q4_1 + BF16 (fc).
    3. Report the deltas on this issue — tensor names, dtypes, shapes, whether the layer count and the selector/conv tensors that the DFlash2 config names are present or absent — because that is what decides whether our reader can load a draft at all, before any kernel or verify work.

    Not in my slice tonight: the kernels, the verify wiring, and the acceptance measurement. Those need the main model on a device (npu-hold), and I will state separately when I take them. My acceptance contract for the verify side is already decided and recorded on #1781 gate 2, and DFlash's block_size 8 matches the k=8 break-even I measured there (>2.4 accept), so the two issues are consistent by construction rather than by coincidence.

  10. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Measured: the Bolt draft container, opened with our own reader (device-free slice done)

    Downloaded to ~/scratch-packs/dflash-bolt/ (disk, 257.9 MB), dflash_weight.q4nx read with the same container path this lane fixed in #1683/#1703. Four findings, the second and third being the ones that change what a loader has to do.

    1. The container is self-describing

    64 header entries, no separate metadata block — the hyperparameters are tensors, and their values agree with config.json exactly where both exist:

    dflash.block_size         I32  [1]     8
    dflash.sliding_window     I32  [1]     1024
    dflash.target_layer_ids   I32  [5]     [5, 11, 17, 23, 35]
    dflash.embedding_scale    F32  [1]     50.596443      (config: 50.59644)
    dflash.rope_theta         F32  [1]     1e6
    dflash.rms_eps            F32  [1]     1e-6
    dflash.mask_token_id                   ABSENT
    

    So a draft can be configured from its own container — no config.json needed for the six — which is the opposite of the 1.9 packs' split, where the geometry lives in config.json and the container only carries tensors.

    2. mask_token_id is the single config-only parameter

    The one hyperparameter in config.json (value 4) that the container does not carry. A loader must therefore read config.json for exactly one value, or the pack should carry it — worth deciding before wiring, because "read the container, fall back to config for one field" is the kind of asymmetry that later becomes "read config first, then overwrite from the container" and silently prefers the stale copy.

    3. Every projection's tile count verifies exactly against the declared geometry

    I8 tensors are [tiles, chunk_bytes] — the same {ntiles, chunk} convention our own importer writes (the lesson from #1849/#2002), not [rows, cols]:

    tensor shape grid check against hidden 2560 / IM 5120 / 32 heads / 8 kv / hd 128
    self_attn.q_proj I8 [1280, 5120] (32·128/32) × (2560/256) = 1280 ✓
    self_attn.k_proj I8 [320, 5120] (8·128/32) × (2560/256) = 320 ✓
    self_attn.v_proj I8 [320, 5120] same = 320 ✓
    self_attn.o_proj I8 [1280, 5120] (2560/32) × (4096/256) = 1280 ✓
    mlp.{gate,up}_proj I8 [1600, 5120] (5120/32) × (2560/256) = 1600 ✓
    mlp.down_proj I8 [1600, 5120] (2560/32) × (5120/256) = 1600 ✓

    Six for six, derived from config.json's own numbers — so the container's geometry is the geometry it declares, and our reader's [tiles, chunk] convention is the same one FLM ships.

    4. model.fc.weight [80, 12800, 32] encodes the layer taps

    12,800 = 5 × 2,560 — the fusion layer's input width is the five tapped layers' hidden states concatenated, exactly the cardinality of target_layer_ids. That is the drafter's interface in one shape: five taps in, block_size positions out. (The BF16 hidden_norm/norm [2560] and the per-layer {q,k}_norm [128] are the rest of the half of the file that is not I8, matching Q4_1 + BF16 (fc).)

    What this says about "own reader, own kernels"

    The five layers are plain transformer blocks with the suffixes our forward already loads — input_layernorm, self_attn.{q,k,v,o}_proj, self_attn.{q,k}_norm, mlp.{gate,up,down}_proj, post_attention_layernorm — 11 tensors each, plus the fusion/fc and the taps. So the draft is not a new architecture to implement from their kernels; it is a 5-layer stack over the main model's hidden states, and the only unusual parts are the taps (which layers to feed) and the masked block.

    Next slice (mine, needs a device or the main model)

    Load the container through npu/mtp's forward and check it produces block_size positions from five taps; then the acceptance row. Both need the Bolt main package on npu-hold, and I will state when I take them. The container and configs stay on disk under ~/scratch-packs/dflash-bolt/ (0.28 GB) for that work.

  11. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    (cl-flm) The loader slice is #2048: load_draft(dir) reads the DFlash config and the container, checks all 64 tensors of the Bolt draft against the config's geometry and decodes all 327,680,000 projection weights finite. Two findings from the bytes that the forward will need: model.fc.weight is stored as [hidden/32][taps*hidden][32], not row-major (the evidence and its control are in the PR body), and three post-attention norms hold one slightly negative gain each. Remaining after it: the draft's forward, the target taps (needs the main Bolt package on the box), DFlash2, and the pair row.

  12. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Its implementation (#2048) loads the real draft — verified here, on this box's copy of the container

    I built #2048 with the real-package arm pointed at the download from my earlier comment and ran it:

    real package: 64 tensors, 270406816 bytes; 5 layers, hidden 2560, intermediate 5120, 32 heads / 8 kv x 128,
                  block 8, taps 5 11 17 23 35
    real package: 327680000 projection weights decoded
    dflash_draft: ok (111 checks)
    

    Every number agrees with the inventory I reported above — 64 tensors, 257.9 MB, 5 layers, hidden 2560, IM 5120, 32/8 heads × 128, block 8, taps [5,11,17,23,35] — reached by a different tool, which is the check that matters for a pack we do not own. The loader also requires mask_token_id from config.json and refuses when absent, which is exactly the one-parameter asymmetry I flagged: the other six hyperparameters are tensors inside the container.

    So "the draft container imports and loads on our runtime" — item 1 of this issue's definition of supported — is now demonstrated for the Bolt draft, device-free. The remaining items (2: consume target_layer_ids hidden states and emit a masked block; 3: feed step_batch_all()/begin_verify(); 4: the measured acceptance row) need the Bolt main package on npu-hold, and that is the slice I will state when I take it.

    Reproducible: the pack stays at ~/scratch-packs/dflash-bolt/ (0.28 GB, disk), so -DONEBIT_DFLASH_DRAFT=/home/bcloud/scratch-packs/dflash-bolt rebuilds this arm.

  13. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    A fix PR now exists for item 1 of your "supported" list: #2048 (25c64005, Refs #1780) adds onebit::npu::dflash::load_draft(dir) — reads the DFlash config.json and refuses a non-DFlashDraftModel, opens the container with the engine's own Q4NX reader, checks every tensor against the geometry the config gives (q4_1 tile grid per projection, 2 B per element for norms and the fusion matrix, container scalars equal to the config's, and no tensor left unexplained), and returns views into the mapped file so nothing is decoded at load.

    Reviewing it I noted two measurements in its body that matter more than the loader for your acceptance row:

    1. model.fc.weight is not row-major — the declared shape [80, 12800, 32] is the element order — settled by neighbour correlation with the plain row-major reading run as the control (as decoded: next row 0.1073, 32 rows on 0.1059 | next column 0.0217, 32 columns on 0.0207; read plainly the row and column figures contradict each other). It says explicitly that this is weights statistics, not a forward.
    2. A norm gain can be negative — three post-attention norms hold one gain just below zero (−0.0009 to −0.0024 of 2,560), found because a "positive" assertion failed. Read as half-float the same element would be ≈ −0.8 among 1.3–2.4 values, an independent point for the bf16 reading tools: dflash_draft_info, the gate before a DFlash draft loader: the package is in forms we already read (engine#1780) #1957 settled.

    So the issue's remaining items are: wiring it into forward-serve (no draft forward exists to feed yet), and the acceptance arithmetic — which the PR deliberately leaves to you. Worth linking in the issue body so a reader does not think the drafts half is unclaimed.

  14. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Closing the float-reading branch with numbers, so nobody repeats it: the draft's I8 payloads are not a float tensor in either width. Both plausible uniform readings fail on a real tensor (model.layers.0.self_attn.q_proj.weight, 6,553,600 B of a [1280, 5120] byte-shape), and they fail differently, which is what rules them out together:

    as f16  : finite 98.20 %   mean +1558.8   sd 11061.2   p1 -36064   p99 +52416   max|v| 65504
    as bf16 : finite 99.77 %   mean nan       sd nan       p99 4.3e+36                max|v| 3.39e+38
    gate_proj, same shape: as f16 mean +1508.2 sd 11099.4; as bf16 p99 4.1e+36
    

    A weight matrix is zero-mean with sd ~0.02; nothing here is close. Two further facts pin the kind of thing it is:

    • 0.59 % of the 16-bit words saturate at or above 60,000 in the f16 reading, and the "bad" positions cluster at small gaps (most common gaps 1, 2, 3 words) — i.e. the stream interleaves metadata words with data words rather than holding a plain array;
    • the first ~4 KB reads as a narrow band around 1.5 (1.42285, 1.53613, 1.54883, 1.43066, …), which is exactly what bf16 scales look like when reinterpreted as f16 — the trap that made me think "f16 image" earlier.

    So the payload is a blocked image whose per-group words sit inside the code stream, and the three readings that fit its byte count (f16 1280 × 2560, Q4_1 at 20 B/32 weights = 4096 × 2560, q4_1f tiles at 6,144 B = 1066.67 tiles → not integral, so ruled out) cannot be separated by arithmetic or by uniform-float statistics. The FLM kernel-side packing is the only authority, which is what I said before and now have the evidence for.

    Proposal for this issue's state, since the release build wants every open issue resolved: mark #1780 explicitly blocked on the F2 packing spec rather than open-ended, with the fingerprint above as the acceptance test — any candidate rule must reproduce a zero-mean ~0.02-sd weight distribution from these exact bytes, and a rule that produces the f16/bf16 figures above is wrong by inspection. The draft is a schema translation (container format is ours, names map, the fc and norms are plain BF16/F16) plus one unknown decode; that is a smaller statement than "write a packing of its own" and it is the true one.

    What I am not doing: guessing a fourth interpretation. Two of my three hand decodes tonight produced plausible-looking garbage (k_proj at corr 0.001, the "620 B of QK norms"), and the pattern is the same each time — an arithmetic-derived layout. The next measurement worth spending is flm pull for the parent pack to read the geometry its kernels are built for, and that needs the disk headroom a release build also wants; I would rather it be an owner call than a quiet GB.

  15. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Where the Bolt pair stands on our runtime (cl-npuperf, lead): the draft works and its proposals are accepted at 1.72 per pass, the target now runs on the NPU, and the pair cannot be faster than the target alone today, because a Gemma4 target has no batched verify pass.

    What is merged or in review

    item of "supported" state
    1. the draft imports and loads merged (#2048)
    2. it takes the target's tapped hidden states and a masked block draft forward merged (#2174); the target-side taps in review (#2180)
    3. the block feeds our verify pass, accepted prefix equals greedy scored offline against the target's greedy tokens (below); a live loop is blocked, see "ceiling"
    4. measured row acceptance measured; tok/s measured for the target alone; the pair's ratio is bounded at 1.0x today

    Measured

    Acceptance (host references, the draft's forward against Nemotron-3.5-Bolt-E4B's own greedy tokens, 32 tokens per prompt, positions walked as a speculative loop would):

    prompt passes accepted per pass
    a Fibonacci function 6 21 3.50
    why the sky is blue 10 16 1.60
    days of the week, one per line 13 13 1.00
    total 29 50 1.72 (2.72 tokens per target pass)

    The target alone on the NPU (per-op int8 route, all eight designs plus their 16-row twins, every kernel bit-exact against a host int8 GEMM; one npu-hold ticket, one binary, arms old / new / old / new):

    ms per step tok/s
    main's behaviour (layer weights re-read every token) 3377, 3410 0.29 - 0.30
    weights read once (change ready, stacked on #2214) 2486, 1914 0.40 - 0.52

    Two defects had to be fixed to get a correct answer at all: #2213 (the per-layer gate and projection ran with the attention weights on the device; #2214) and #2215 (the tokenizer reader mis-encoded Gemma-family files; #2217).

    The ceiling, and why

    1. No batched pass for Gemma4. The forward's batched body does not model per-layer embeddings, so a pass of k rows on this target is k single steps. Verifying a block of 8 costs 8 steps whether or not the proposals are right, so the pair is at most 1.0x the target alone, at any acceptance. FLM's 1.3-1.9x for this tag is out of reach until that body exists.
    2. The device time per step is weight streaming. On this kernel family a 16-row launch streams the projection's whole int8 image: 21.5 ms for gate|up, 13.6 ms for down, 4.0 ms for q|k|v, which is about 1.9 s over 42 layers. A drafter does not change that.
    3. The draft itself has no kernel set. On the host its reference pass takes about 23 s.

    What would close item 4 properly

    • a batched Gemma4 body in the forward (engine work), then
    • a kernel set for the draft's five layers (private), then
    • the pair's row. The acceptance above says a batched 8-row verify would emit 2.72 tokens per pass on these prompts.

    I am asking the orchestrator whether to schedule those or to close this issue on the row above; the question is in my state file.

  16. bong-water-water-bong commented on Nov 12, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    CLOSE CONDITION

    Bar (item 4 of "supported"; items 1-3 are on main in #2048, #2174, #2180 and #2236):

    • Model: Nemotron-3.5-Bolt-E4B with its DFlash draft on 1bit forward-serve, greedy, 32 tokens.
    • Measurement: plain against speculative in one npu-hold ticket, no new amdxdna error.
    • Pass: every speculative reply is byte-equal to plain, and speculative tok/s is at least 1.3x plain on the code prompt and on the prose prompt (the low end of FLM's 1.3-1.9x).
    • Second pair: served bonsai2-27b-t2s2 with the DFlash2 27B draft, same rule: byte-equal, and at least 1.16x plain on at least one prompt.

    Where it stands (measured; the rows are in #2303's body):

    pair code prose
    Bolt-E4B + DFlash 0.557 → 3.40 tok/s, 6.1x 0.562 → 0.830 tok/s, 1.48x
    27B + DFlash2 2.93 → 9.84 tok/s, 3.4x 3.20 → 3.88 tok/s, 1.2x

    All replies are byte-equal to plain.

    Left between now and the bar:

    1. npu forward: Gemma4 verifies a draft in one batched pass; KV row width per layer (fixes #1780) #2303 merges (READY ×2 at 4d10370, CI green, mergeable). It carries Fixes #1780.

    No other PRs and no new slices.

    Two parts of the issue body's bar cannot be measured as written:

    • 1.16x against the MTP head: the served 27B pack has no MTP head, so the 27B row is against plain decode.
    • The 1.3-1.9x range "short prompt .. 16k ctx": only short prompts were measured.

    The ORCH line in my state file proposes the bar above for both.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    backlogReal, but does not hold the release build

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions