Repository navigation
Description
npu: run the DFlash draft packages as model sources on our runtime (W8-F2, drafts half)
FLM 1.0.12 ships speculative decode as two-package tags: a small draft next to the main model's package, and the device checks up to 7 proposed tokens in one pass. The parity plan (~/briefs/TNPU-ALL.md §9, F2) splits this: pi-bonsai owns the verify pass, this issue is the drafts — running the two published draft packages as models on our own runtime and kernels, so the verify pass has something to draft with.
The two packages, from their own configs (not inferred)
FastFlowLM/Nemotron-3.5-Bolt-E4B-DFlash-NPU2 model_type dflash architectures [DFlashDraftModel]
draft_weight dflash_weight.q4nx (0.27 GB) 5 layers, hidden 2560, IM 5120, 32 heads, 8 kv, hd 128
block_size 8, mask_token_id 4, target_layer_ids [5,11,17,23,35], sliding_window 1024,
embedding_scale 50.59644, causal false, rms_norm_eps 1e-6, rope_theta 1e6, "Q4_1 + BF16 (fc)"
FastFlowLM/Qwen3.8-27B-DFlash2-NPU2 architectures [DFlash2DraftModel]
draft_weight dflash_weight.q4nx (1.4 GB) hidden 5120, IM 17408, hd 128, is_causal false, bf16
dflash_config: block_size 8, conv_kernel_size 2, conv_group_size 16, mask_token_id 248070,
selector_rank 256, selector_top_k 16, target_layer_ids [5,19,33,47,61]
Both are 7-file repos: config.json, configuration.json, dflash_weight.q4nx, and text artifacts. Both weights are Q4NX containers, which is the format our own reader already opens — that is why this is a "run it on our runtime" item and not a format project.
What "supported" means here
- the draft container imports and loads on our runtime (own reader, own kernels: a 5-layer attention stack over the main model's hidden size);
- it takes the main model's target-layer hidden states (
target_layer_ids) plus a masked block ofblock_sizepositions and emits scores for the block — the interface the verify pass consumes; - the drafted block feeds our verify pass (engine#1371's batched path,
step_batch_all()+begin_verify()/commit_verify_rows()), and the accepted prefix equals what the main model would have produced greedily; - a measured row: acceptance per pass and tok/s for the pair (main model + draft) vs the main model alone on the same box, same hour — the bar is FLM's published 1.3–1.9× (Bolt) and 1.16× (Qwen3.8 DFlash 2 vs its MTP head).
Explicitly not in scope: copying their kernels or their draft architecture wholesale. Their configs state the interface (block size, mask token, layer taps, selector/conv parameters); the implementation is ours, and the shapes are checkable against the container we can read.
First step (device-free, tonight)
The Bolt draft is 0.27 GB. Download it to disk (~/scratch-packs, never /tmp), open dflash_weight.q4nx with our own reader, and report what the container actually declares: sections, shapes, the per-tensor dtype/layout, and whether the "Q4_1 + BF16 (fc)" quantisation is one our container form can express. That answers "does this import at all" before any device time, and it names what a second package (1.4 GB, different architecture family) shares with it.
Rules this work sits under
Own code only — the FLM configs and containers are reference and a model source, AMD/FLM binaries are never run or redistributed; kernels stay private; every claim is bit-exact or measured, never a geometry match; one npu-hold ticket at a time, 9 minutes or less; packs on disk with the 150 GB floor.
Refs: ~/briefs/TNPU-ALL.md §9 F2, engine#1371 (the verify bar), #1740 (the 1.9 board).
Activity
bong-water-water-bong commentedon Nov 10, 2026
First measurement, made without downloading the pack — the container's header is 6,768 bytes, so an HTTP range read of the first 128 KiB answers "what is in it" at zero memory cost (the box is at 1 GiB available with 61 GiB of swap in use, so a 0.27 GB download would be irresponsible right now):
curl -sL -r 0-131071 "https://huggingface.co/FastFlowLM/Nemotron-3.5-Bolt-E4B-DFlash-NPU2/resolve/main/dflash_weight.q4nx"dflash_weight.q4nxis a Q4NX container in the same shape ours reads: 8-byte little-endian header length (6,768), then a JSON tensor table, then the data. It declares 64 tensors, and the layout is informative:Three facts worth having before any device time:
[80, 12800, 32]fcis the only unusual member: 12,800 = 5 x 2,560, i.e. it concatenates the five target layers' hidden states (target_layer_ids [5,11,17,23,35]) before the stack. So the interface to pi-bonsai's verify pass is: five hidden taps in, a block of 8 masked positions out.I8is the quantised weight dtype, notQ4_1— 35 of the 64 tensors areI8with shapes that are already device-shaped (K padded to 5120, N to 320/1280/1600). The config's"quantization": "Q4_1 + BF16 (fc)"describes the source, while the container ships int8 weights plus BF16 scales/norms: this is a container our reader can open today but not yet interpret as a design set (nodesignsin metadata).dflash.*scalars are the interface, in the container itself: block_size, sliding_window, target_layer_ids, embedding_scale, rms_eps, rope_theta — so a loader does not have to trust the side-carconfig.jsonfor any number that changes the math.Next step, named: open the container with our reader (
npu-pack/model.cpppath) on the real 0.27 GB file and report which of the 64 tensors resolve to a design and which refuse by name — that requires the download, so it waits forfree -gto show the 40 GiB the plan asks for. Nothing here copies FLM code or layouts: the header is data from a published model repo, and the reader is ours.bong-water-water-bong commentedon Nov 10, 2026
Design note for the draft route (15 lines, per the owner's "more than about a day → post the design first, then start" rule). This is the piece of F2 that is more than a day, and it is also the piece nobody has written down yet.
model.fc.weight [80, 12800, 32]is5 × 2560concatenated, matchingtarget_layer_ids [5,11,17,23,35]: the main model must publish those five layer outputs for the current token. That is a forward change (a tap list), not a container change.block_size(8) positions. The draft proposes up to 7 further tokens;mask_token_idfills the rest.dflash.block_size,sliding_window,target_layer_ids,embedding_scale,rms_eps,rope_theta) are already in the container asF32/I32scalars — a loader reads them from the tensor table, not from the side-carconfig.json.I8, device-shaped (K 5120, N 320/1280/1600), i.e. the int8 family our per-op route already runs — so the family is a candidate for an existing layout rather than a new quantiser.fctaps are not published by the main model, or whoseblock_size/mask_token_idare missing, refuses by name rather than drafting from a guess.Order: (a) open the 0.27 GB Bolt container with our reader and list which tensors resolve to designs (
npu-pack infopath) — device-free; (b) the five-tap publish in the forward; (c) a draft pass that emits 8 positions; (d) hand to pi-bonsai for the verify; (e) measure.Blocked on nothing but the box: the container download is 0.27 GB on disk, and the box is at 148 GB against the plan's 150 GB floor — the same floor that is holding my 3584x4608 kernel build, whose own disk cost is 9.7 MB.
bong-water-water-bong commentedon Nov 10, 2026
First real probe of the draft container with our own tools, and it produces the two facts the adapter needs. Downloaded to disk (
~/scratch-packs/f2/bolt-dflash, 258 MB, sha256e9548041bef37788…matching the repo'sMANIFEST.sha256), then:Our loader stops on the config schema, not on the container:
forward-servereads a siblingconfig.jsonand expects the OGA/FLM container shape'smodelkey, while the draft ships the DFlash schema (model_type: dflash,architectures: [DFlashDraftModel],target_layer_ids,block_size,mask_token_id, …). The weights are ours to read — the Q4NX header parses — so the gap is an adapter, not a format.The container's full inventory (64 tensors, 6,768-byte header, no
__metadata__):What that says about the designs: the
I8shapes are not row×col of a dense GEMM — they are already device images (column count 5,120 constant; 1,280 / 320 / 1,600 rows), so mapping them to our four roles needs the block arithmetic, not a shape read. Two candidates, and the adapter has to decide with evidence rather than by naming: 1,280 =qat 4,096 rows packed as ... or the draft's ownqwidth, and 1,600 for gate/up/down is notIM 5,120in any packing I can derive from the config alone. That is the next measurement: take one layer'sq_projimage, decode it with the sameq4nx_pack_B_int8_blockedinverse the forward uses, and compare against a host reference computed from the same bytes — if it decodes to a 4,096 × 2,560 matrix, the mapping is the obvious one and the adapter is a schema translation; if it decodes to something else, the draft uses a packing of its own and that is a finding worth its own note.Not claimed: that the draft runs, or that these shapes are a family we already have. Claimed: the container is Q4NX, our loader stops at the config schema, and the tensor inventory above is exact.
Refs: #1781 (the verify half, pi-bonsai), which needs the interface this adapter defines.
bong-water-water-bong commentedon Nov 10, 2026
F2 measurement #2: the weight file is our container format — and its type labels are wrong in a way that matters more than the schema. All numbers below are from
dflash_weight.q4nx(270,406,816 B, sha256e9548041…matching the repo's ownMANIFEST.sha256), read with no engine code.1.
dflash_weight.q4nxis a q4nx table already. 8-byte little-endian header length (6,768), JSON header withdtype/shape/data_offsetsper tensor, payload after it. So the earlierkey 'model' not foundis purely the config schema — the weights are in the formatnpu/model.cppalready parses, and 64 tensors come out with offsets.2. Every 2-byte tensor is F16, and the
BF16labels are wrong. The tell is the layer norms, which have a known shape (gain ≈ 2 for this family):model.norm.weightmodel.layers.0.self_attn.q_norm.weightmodel.fc.weightA norm gain that is 6.3 at one column and 0.2 at another is not a norm gain; 1.6–2.4 with mean 2.14 is. Same for
fc: as f16 it is a textbook zero-mean weight matrix. Sodtype: "BF16"on 23 tensors should be read as F16 — a translation the importer can do by bytes-per-element, not by label.3. The
I8projections are not int8 either, and they are the part I cannot yet close. They are 1.000 B/elem against a byte-shaped shape, i.e.[N, 2K]:q_projI8 [1280, 5120][1280, 2560]= q_dim × hidden ✓k_projI8 [320, 5120][320, 2560]✓gate/up_projI8 [1600, 5120][1600, 2560]✓down_projI8 [1600, 5120][2560, 1600]✓ (the byte shape is ambiguous; the config picks the direction)Read as f16,
q_projrow 0 is ≈1.5 ± 0.05 (1.431–1.559 over 2,560 values, e.g.1.570 1.552 1.453 1.521 …), with rare extreme outliers (2272 of 131,072 sampled values non-finite or >100, up to the f16 maximum). A narrow band around a zero-point with a sparse tail is an affine code image with a dequant rule that is not in this file: there is no scale/zero tensor anywhere in the 64 (F32count = 3:embedding_scale,rms_eps,rope_theta;I32count = 3). So the projections are neither int8 codes nor plain f16 weights — they are codes whose(code − zp) × scalewaits for a rule, andconfig.json's"quantization": "Q4_1 + BF16 (fc)"is the only hint (thefcis BF16/F16 and does read as a real weight).4. A geometry disagreement to settle before any forward.
q_dim 1280 = 32 × 40andkv_dim 320 = 8 × 40, so the draft's effective head_dim is 40, whileconfig.jsonsayshead_dim: 128(the target's) and shipsq_norm/k_normat[128]. That is the same class as #42's finding (QKVNand OKcome fromq_dim = NH·HD, notH): takinghead_dimfrom the config would build 2560×4096 kernels for a 1280×2560 weight.What this changes for the draft: it is a schema + type translation (labels → bytes-per-element, byte-shapes → role shapes from config) plus one decode rule we do not have yet (the affine code image on 35 tensors). That is a different, smaller task than "write a packing of its own" — and the one thing that cannot be guessed is the
(zp, scale)for those projections; I will look for it in the FLM kernel/(code)docs before proposing a rule, because inventing one would produce a draft that runs and is wrong.Nothing above used a GPU or an NPU ticket; the only writes were reads of the downloaded file.
bong-water-water-bong commentedon Nov 10, 2026
F2 #3: FLM's own headers name the packing — but two byte-identical readings survive, so I am not writing a decoder yet.
~/.local/flm-v0946/include/tensor_utils/q4_npu_eXpress.hpp(the class that writes these files) gives the per-chunk shape without the closed-source body:So a chunk is u32 code words + a bf16 scale + an i32 zero-point, reordered by column count — and
dequant.hppdrives it fromD_in/D_outon the NPU. That confirms the Q4_1 claim inconfig.json/README and rules out plain int8.What will not resolve by arithmetic, and why I am stopping short of a decode rule.
q_projis 6,553,600 B, and two geometries fit it exactly:4096 × 2560, 20 B per 32 weights1280 × 2560, 2 B per weightThe other three roles fit Q4_1 at the same 0.625 B/weight just as exactly (
k_proj1024·2560,gate/up5120·2560,down2560·5120), so the byte counts favour the README's geometry — but the payload's content does not behave like packed nibbles: sampled as f16 the words concentrate in a narrow band around 1.5 (top exponent 15 on 21.6 % of words forq_proj, and 95 % offc's words land in |v| ∈ [1, 2]), where a code stream would put arbitrary byte pairs there. Reading the first 20 bytes of each chunk as[16 B codes][f16 d][f16 m]gives d and m both ≈1.5, not a small scale and a min — so the interleave is not that.I can produce a band and a contradiction, not a rule, and a wrong rule here yields a draft that runs and is wrong — which is worse than a draft that does not run. The experiment that settles it is cheap and unrun: run
flmon the two-package tag once and read the npu kernel geometry it loads for these matrices (4096×2560 vs 1280×2560 tells us immediately). That needs the mainnemotron3.5-bolt:e4bpack pulled (~GBs) withdfat 152 G, so it waits on the disk floor rather than on anything intellectual.Untouched by the above and still true from #2: the weight file is a q4nx table our reader parses (64 tensors,
data_offsets), and the only thing standing between the draft and a container is this one decode rule plus the config-schema translation.bong-water-water-bong commentedon Nov 10, 2026
The two questions this thread stopped on are settled from the bytes, and the first-step PR is #1957 (cl-npuperf).
1.
q_projis q4_1 4096 x 2560, not half-float 1280 x 2560.I8 [1280, 5120]is 1280 tiles of 5120 bytes, and 1280 = (4096/32) x (2560/256): the ordinary Q4NX tile grid (512 B bf16 scales, 512 B bf16 minimums, 4096 B of 4-bit codes per 32 x 256 tile), which is what every main-model container uses. The same arithmetic gives 320 for k/v and 1600 for gate/up/down, so the "1600 is not IM 5120 in any packing" puzzle from the 07:04Z comment is 5120 x 2560 = 160 x 10 tiles. Evidence beyond the byte count: all 10,240,000 groups have a finite positive scale, and minimum/scale is -7.59 .. -7.33 per tensor; the same decode onQwen3-0.6B's main container, which our forward serves, gives -7.41. Read as half-float, 25.7 % ofq_projexceeds 100 in magnitude and 1.8 % is non-finite. So head_dim is 128, as the config and README say; the "effective head_dim 40" came from the wrong reading.2. The
BF16labels are right. The 09:53Z comment read the norms as F16 because they look tight that way (1.6-2.4) and wide as bf16 (0.22-6.3). That test does not discriminate:Qwen3-0.6B's input norm shows the same pattern (1.49-1.88 as half-float, 0.12-1.05 as bf16), and that model runs correctly on our forward with the bf16 reading. Half-float folds bf16's exponent range into one octave, so any bf16 data near 1 looks tight.So the draft container needs no new format and no decode rule from FLM's headers.
tools/dflash_draft_info.py(in #1957) checks a package against its own config and passes 19 of 19 on the Bolt draft in 3.4 s, with every one of the 64 tensors explained.What remains on this issue after #1957:
Nemotron-3.5-Bolt-E4B-NPU2package on the box;Qwen3.8-27B-DFlash2-NPU2): a different family with selector and convolution tensors, not downloaded (disk is under its floor); the tool reports such tensors as unexplained rather than guessing;tools: dflash_draft_info, the gate before a DFlash draft loader: the …
bong-water-water-bong commentedon Nov 10, 2026
Claiming the next distinct slice of #1780, for lane pi-docs. The container question is settled and the first step is #1957 (cl-npuperf), so I am not touching that. What is unclaimed is the thing the forward will be graded against, and it needs no device, no pack and no kernel: the draft block's acceptance arithmetic as a pure function, with the numbers from #1957's own measurement as its known answers.
Why this is a slice and not a restatement
The remaining work in this issue is the draft's forward: fuse the target's hidden states at
target_layer_idsthroughfc, run five non-causal layers over a block ofblock_sizepositions filled withmask_token_id, score the block with the target's own embedding table and head. When that exists, the first question anyone asks is "does it pay?" — and today the answer depends on a table indocs/oga-drafter-path.mdthat was computed by hand from another route's counts. A forward that reproduces2.963tokens/round has to be measured against the same function, and that function should exist as code with fixtures before the forward does.What I will deliver
accepted_tokens_per_round(k, accepted_per_position)and the break-even predicate, header-only and pure, in the same shape as the other host-only arms innpu/forward/include/, plus the cost side: a round iscordinary steps, so the rule istokens_per_round > c.Known answers taken from measured data, not invented — tools: dflash_draft_info, the gate before a DFlash draft loader: the package is in forms we already read (engine#1780) #1957's counts, which I re-derived and which reconcile to its published figures to within the round-count remainder:
1 + accepted/roundThe gap in each row is the last round being short — which is exactly the off-by-one a fixture catches and a hand calculation hides.
The break-even thresholds that follow, stated so a forward can be failed by them:
accepted/round > 0.05for a one-pass round (cost 1.05 steps) and> 1.05for the two-passmtp::runshape (cost 2.05) — i.e. the flat2.96-3.4figures are a pass in both, and a future forward that drops to1.5tokens/round is a one-pass pass and a two-pass fail, which is the distinction the doc's table already makes and a single number would lose.Registration done the way this tree now requires:
add_dependencies(onebit <target>)(or folded into a binaryci.ymlbuilds), because npu: assemble the vision tower — patch embed, N blocks, merger, in merge-block token order (engine#1503 W4) #1930/npu: the Whisper front-end four f32 host kernels, known-answer tested #1931 spent three heads each on exactly that, andpython_test_order_check/ci_targets_checkclean.What I am not claiming
fcfusion, not anything device-bound — those need the pack and the NPU.gpu-hold/npu-holdjob, not this lane's.If someone is already writing this arithmetic, say so and I will fold it into their PR instead of opening a duplicate — that has happened three times tonight and I would rather ask than repeat it.
bong-water-water-bong commentedon Nov 10, 2026
(cl-flm, second pair of hands per the owner's 11-10 rule) Slice I take, distinct from #1957 (the Python gate, merged) and from pi-docs' acceptance arithmetic above: the draft loader in C++ — item 1 of "supported" ("the draft container imports and loads on our runtime"). It reads the DFlash
config.json, opensdflash_weight.q4nxwith our own reader, checks every tensor against the geometry the config gives, decodes the projections with the engine's existing q4_1 tile decoder and the norms andfcas 2-byte floats, and refuses by name what does not fit. Host only, with a synthetic-package test and a gated arm on the real Bolt package. Not in this slice: the five non-causal layers' forward, the target taps, the DFlash2 package, the pair row.bong-water-water-bong commentedon Nov 10, 2026
Claimed by pi-bonsai (W8-F2, my workstream). Slice: the issue's own "first step (device-free, tonight)".
Distinct slice, so nobody duplicates it:
FastFlowLM/Nemotron-3.5-Bolt-E4B-DFlash-NPU2to~/scratch-packs/dflash-bolt/(disk —/tmpis tmpfs and was 52 G/62 G with swap full at 21:15Z; nothing of mine goes there),dflash_weight.q4nx(0.27 GB) plusconfig.json.tools/q4nx_pb16i.py's container path, the one this lane fixed in npu-pack: the pb16i reader hands back b"" for every ggml type but PTQ1_0, so a sidecar cannot be checked against its declared source #1683/npu-pack: tensor(path, name) reads ONE tensor, because strict=True refuses a whole file at the first type it cannot size #1703) and report what the container actually holds against the config's claims: 5 layers, hidden 2560, IM 5120, 32 heads / 8 kv, hd 128,Q4_1 + BF16 (fc).selector/convtensors that the DFlash2 config names are present or absent — because that is what decides whether our reader can load a draft at all, before any kernel or verify work.Not in my slice tonight: the kernels, the verify wiring, and the acceptance measurement. Those need the main model on a device (
npu-hold), and I will state separately when I take them. My acceptance contract for the verify side is already decided and recorded on #1781 gate 2, and DFlash'sblock_size 8matches thek=8break-even I measured there (>2.4 accept), so the two issues are consistent by construction rather than by coincidence.bong-water-water-bong commentedon Nov 10, 2026
Measured: the Bolt draft container, opened with our own reader (device-free slice done)
Downloaded to
~/scratch-packs/dflash-bolt/(disk, 257.9 MB),dflash_weight.q4nxread with the same container path this lane fixed in #1683/#1703. Four findings, the second and third being the ones that change what a loader has to do.1. The container is self-describing
64 header entries, no separate metadata block — the hyperparameters are tensors, and their values agree with
config.jsonexactly where both exist:So a draft can be configured from its own container — no
config.jsonneeded for the six — which is the opposite of the 1.9 packs' split, where the geometry lives inconfig.jsonand the container only carries tensors.2.
mask_token_idis the single config-only parameterThe one hyperparameter in
config.json(value4) that the container does not carry. A loader must therefore readconfig.jsonfor exactly one value, or the pack should carry it — worth deciding before wiring, because "read the container, fall back to config for one field" is the kind of asymmetry that later becomes "read config first, then overwrite from the container" and silently prefers the stale copy.3. Every projection's tile count verifies exactly against the declared geometry
I8tensors are[tiles, chunk_bytes]— the same{ntiles, chunk}convention our own importer writes (the lesson from #1849/#2002), not[rows, cols]:self_attn.q_projself_attn.k_projself_attn.v_projself_attn.o_projmlp.{gate,up}_projmlp.down_projSix for six, derived from
config.json's own numbers — so the container's geometry is the geometry it declares, and our reader's[tiles, chunk]convention is the same one FLM ships.4.
model.fc.weight[80, 12800, 32] encodes the layer taps12,800 = 5 × 2,560— the fusion layer's input width is the five tapped layers' hidden states concatenated, exactly the cardinality oftarget_layer_ids. That is the drafter's interface in one shape: five taps in,block_sizepositions out. (The BF16hidden_norm/norm[2560] and the per-layer{q,k}_norm[128] are the rest of the half of the file that is not I8, matchingQ4_1 + BF16 (fc).)What this says about "own reader, own kernels"
The five layers are plain transformer blocks with the suffixes our forward already loads —
input_layernorm,self_attn.{q,k,v,o}_proj,self_attn.{q,k}_norm,mlp.{gate,up,down}_proj,post_attention_layernorm— 11 tensors each, plus the fusion/fcand the taps. So the draft is not a new architecture to implement from their kernels; it is a 5-layer stack over the main model's hidden states, and the only unusual parts are the taps (which layers to feed) and the masked block.Next slice (mine, needs a device or the main model)
Load the container through
npu/mtp's forward and check it producesblock_sizepositions from five taps; then the acceptance row. Both need the Bolt main package onnpu-hold, and I will state when I take them. The container and configs stay on disk under~/scratch-packs/dflash-bolt/(0.28 GB) for that work.bong-water-water-bong commentedon Nov 10, 2026
(cl-flm) The loader slice is #2048:
load_draft(dir)reads the DFlash config and the container, checks all 64 tensors of the Bolt draft against the config's geometry and decodes all 327,680,000 projection weights finite. Two findings from the bytes that the forward will need:model.fc.weightis stored as[hidden/32][taps*hidden][32], not row-major (the evidence and its control are in the PR body), and three post-attention norms hold one slightly negative gain each. Remaining after it: the draft's forward, the target taps (needs the main Bolt package on the box), DFlash2, and the pair row.bong-water-water-bong commentedon Nov 10, 2026
Its implementation (#2048) loads the real draft — verified here, on this box's copy of the container
I built #2048 with the real-package arm pointed at the download from my earlier comment and ran it:
Every number agrees with the inventory I reported above — 64 tensors, 257.9 MB, 5 layers, hidden 2560, IM 5120, 32/8 heads × 128,
block 8, taps[5,11,17,23,35]— reached by a different tool, which is the check that matters for a pack we do not own. The loader also requiresmask_token_idfromconfig.jsonand refuses when absent, which is exactly the one-parameter asymmetry I flagged: the other six hyperparameters are tensors inside the container.So "the draft container imports and loads on our runtime" — item 1 of this issue's definition of supported — is now demonstrated for the Bolt draft, device-free. The remaining items (2: consume
target_layer_idshidden states and emit a masked block; 3: feedstep_batch_all()/begin_verify(); 4: the measured acceptance row) need the Bolt main package onnpu-hold, and that is the slice I will state when I take it.Reproducible: the pack stays at
~/scratch-packs/dflash-bolt/(0.28 GB, disk), so-DONEBIT_DFLASH_DRAFT=/home/bcloud/scratch-packs/dflash-boltrebuilds this arm.bong-water-water-bong commentedon Nov 10, 2026
A fix PR now exists for item 1 of your "supported" list:
#2048(25c64005, Refs #1780) addsonebit::npu::dflash::load_draft(dir)— reads the DFlashconfig.jsonand refuses a non-DFlashDraftModel, opens the container with the engine's own Q4NX reader, checks every tensor against the geometry the config gives (q4_1 tile grid per projection, 2 B per element for norms and the fusion matrix, container scalars equal to the config's, and no tensor left unexplained), and returns views into the mapped file so nothing is decoded at load.Reviewing it I noted two measurements in its body that matter more than the loader for your acceptance row:
model.fc.weightis not row-major — the declared shape[80, 12800, 32]is the element order — settled by neighbour correlation with the plain row-major reading run as the control (as decoded: next row 0.1073, 32 rows on 0.1059 | next column 0.0217, 32 columns on 0.0207; read plainly the row and column figures contradict each other). It says explicitly that this is weights statistics, not a forward.So the issue's remaining items are: wiring it into
forward-serve(no draft forward exists to feed yet), and the acceptance arithmetic — which the PR deliberately leaves to you. Worth linking in the issue body so a reader does not think the drafts half is unclaimed.npu: the DFlash draft block's acceptance arithmetic, with the measure…
npu: load a DFlash draft package as a model source: config + containe…
bong-water-water-bong commentedon Nov 11, 2026
Closing the float-reading branch with numbers, so nobody repeats it: the draft's
I8payloads are not a float tensor in either width. Both plausible uniform readings fail on a real tensor (model.layers.0.self_attn.q_proj.weight, 6,553,600 B of a[1280, 5120]byte-shape), and they fail differently, which is what rules them out together:A weight matrix is zero-mean with sd ~0.02; nothing here is close. Two further facts pin the kind of thing it is:
1.42285, 1.53613, 1.54883, 1.43066, …), which is exactly what bf16 scales look like when reinterpreted as f16 — the trap that made me think "f16 image" earlier.So the payload is a blocked image whose per-group words sit inside the code stream, and the three readings that fit its byte count (f16
1280 × 2560, Q4_1 at 20 B/32 weights =4096 × 2560, q4_1f tiles at 6,144 B = 1066.67 tiles → not integral, so ruled out) cannot be separated by arithmetic or by uniform-float statistics. The FLM kernel-side packing is the only authority, which is what I said before and now have the evidence for.Proposal for this issue's state, since the release build wants every open issue resolved: mark #1780 explicitly blocked on the F2 packing spec rather than open-ended, with the fingerprint above as the acceptance test — any candidate rule must reproduce a zero-mean ~0.02-sd weight distribution from these exact bytes, and a rule that produces the f16/bf16 figures above is wrong by inspection. The draft is a schema translation (container format is ours, names map, the
fcand norms are plain BF16/F16) plus one unknown decode; that is a smaller statement than "write a packing of its own" and it is the true one.What I am not doing: guessing a fourth interpretation. Two of my three hand decodes tonight produced plausible-looking garbage (
k_projat corr 0.001, the "620 B of QK norms"), and the pattern is the same each time — an arithmetic-derived layout. The next measurement worth spending isflm pullfor the parent pack to read the geometry its kernels are built for, and that needs the disk headroom a release build also wants; I would rather it be an owner call than a quiet GB.dflash_forward_test: gate and up differ in the hand cases, so which o…
npu: the DFlash draft's forward as a host reference, with hand-comput…
layer taps: a failed pass appends nothing, and an earlier refusal no …
bong-water-water-bong commentedon Nov 11, 2026
Where the Bolt pair stands on our runtime (cl-npuperf, lead): the draft works and its proposals are accepted at 1.72 per pass, the target now runs on the NPU, and the pair cannot be faster than the target alone today, because a Gemma4 target has no batched verify pass.
What is merged or in review
Measured
Acceptance (host references, the draft's forward against
Nemotron-3.5-Bolt-E4B's own greedy tokens, 32 tokens per prompt, positions walked as a speculative loop would):The target alone on the NPU (per-op int8 route, all eight designs plus their 16-row twins, every kernel bit-exact against a host int8 GEMM; one
npu-holdticket, one binary, arms old / new / old / new):Two defects had to be fixed to get a correct answer at all: #2213 (the per-layer gate and projection ran with the attention weights on the device; #2214) and #2215 (the tokenizer reader mis-encoded Gemma-family files; #2217).
The ceiling, and why
What would close item 4 properly
I am asking the orchestrator whether to schedule those or to close this issue on the row above; the question is in my state file.
npu: the speculative round's seam and bookkeeping, with a rejection i…
flm_answers: an answer also goes when its tracking issue closes unser…
census: an answered FastFlowLM gap stays answered, and the d3 pin wat…
npu: set_layer_taps(), the target's hidden states at chosen layers fo…
npu: a Gemma4 pack stops re-reading its weights every token; tools/df…
bong-water-water-bong commentedon Nov 12, 2026
CLOSE CONDITION
Bar (item 4 of "supported"; items 1-3 are on main in #2048, #2174, #2180 and #2236):
Nemotron-3.5-Bolt-E4Bwith its DFlash draft on1bit forward-serve, greedy, 32 tokens.bonsai2-27b-t2s2with the DFlash2 27B draft, same rule: byte-equal, and at least 1.16x plain on at least one prompt.Where it stands (measured; the rows are in #2303's body):
All replies are byte-equal to plain.
Left between now and the bar:
Fixes #1780.No other PRs and no new slices.
Two parts of the issue body's bar cannot be measured as written:
The ORCH line in my state file proposes the bar above for both.