Repository navigation
Description
Scope: every FastFlowLM/*-NPU2 model source imports and serves on --device npu on our own runtime and kernels. Part of the FLM 1.0.12 parity work; this is the model-sources item.
The inventory, measured from the public HF API tonight
https://huggingface.co/api/models?author=FastFlowLM&limit=200 → 65 repos, of which 59 carry NPU2; none is gated. Sizes are the sum of the repo's file blobs (?blobs=true), 0.27 GB → 24.26 GB.
Two pack generations, and the difference matters for us:
| generation | files | .xclbin |
q4nx |
example |
|---|---|---|---|---|
| older | 8-14 | 4-7 | 1 | Gemma3-270M-NPU2 (0.54 GB, 5 xclbin) |
| current | 7-9 | 0 | 2 | MiniCPM-V-4.7-1B-NPU2 (2.22 GB) |
So the current packs ship no vendor kernel artifacts at all — they are weights only, which is exactly the shape this item needs: the model source is the Q4NX container, and the kernels come from our own runtime. The older packs carry .xclbin files beside the weights; those are not used by anything here (our artifacts are full ELFs; no xclbin as a target or a fallback), and a pack that only has them is not in scope for serving.
New since 2026-10-30 (lastModified): Nemotron-3.5-Bolt-E4B-DFlash (0.27 GB), D3-Edge (0.62), MiniCPM-V-4.7-35B-A3B (22.21), MiniCPM-OCR-1.0 (2.01), Qwen3.8-27B-DFlash2 (1.39), Hy-Memory-30B-A3B (17.88), MiniCPM-V-4.7-1B (2.22).
Definition of done, per repo, on the pack's own files
- it imports with zero gaps and its self-check passes against a named reference;
- it serves on
--device npu; - it answers a known-answer prompt with the model's own chat template, 3 identical requests returning identical text;
- decode tok/s and time-to-first-token measured under
npu-hold; - the row names the kernel set and the engine commit that produced it.
A row is only DONE with all five. Mulit-component packs (Gemma4, Qwen3.5, Qwen3-VL, MiniCPM-V) are in scope; the vision towers and Whisper have their own items and are tracked separately.
Plan
- One board row per repo, in the board's
FLM NPU2section, swept smallest first (0.27 GB → 24.26 GB), so early rows are cheap and the container/importer contract is settled on small packs before the 22-24 GB ones. - The same-family packs (D3 ×7, LFM2 ×4, LFM2.5 ×2, LFM3 ×1, Qwen3 ×8, Qwen3.5 ×4) should collapse to a handful of distinct container/geometry cases; the sweep's job is to find out which, so a family that works once is checked rather than re-derived.
- Rows go to pi-zha; the first row's job is to say which importer path and which kernel set the whole family needs.
Not in this item
Speculative decode (its own item), the 27B decode figure, embedding-table-from-disk, flash serving, Whisper, MiniCPM-V's vision tower, Decision 3.0 sizes, HRX builds, server behaviours — all tracked separately.
One input missing: the fact-base digest that was expected at amd-oss/npu-re/docs/FACTS-fastflowlm-v1012-delta.md is not on this box yet (only the v1.0.7 artifacts digest is). The inventory above is from the public release notes and the HF API, so nothing here depends on it, but the per-model container details will want it when it lands.
Activity
bong-water-water-bong commentedon Nov 10, 2026
First row measured: the NPU2 model source is a safetensors container, and our importer does not know that genre.
Working the sweep smallest-first,
FastFlowLM/Gemma3-270M-NPU2(0.54 GB, first standalone chat pack — I skipped the 0.27 GBNemotron-3.5-Bolt-E4B-DFlash, which is a draft package and belongs to the speculative-decode item).What the pack is
Downloaded the weights-only files (skipping the vendor
.xclbins, which nothing here may use):config.json,model.q4nx(503 MB),tokenizer.json,tokenizer_config.json.i.e. 8-byte little-endian header length (26,496) then JSON — a plain safetensors container, which matches the container facts our own notes already record. A report from our own reader (
tools/flm/q4nx_container_report.py, header only, no weights, no device):Three facts the importer will have to live with:
model.embed_tokens.weight,model.layers.N.self_attn.q_proj.weight, …), not per-op graph names — there is no graph at all;BF16for norms and the embedding,I8for every matrix. NoQ4NXdtype tag in this pack, and no__metadata__block (the container notes describe one written by the older converter; this pack has none);I8matrices carry twice their logical element count (down_projis[640, 1280]for a 640-wide FFN), i.e. the storage is packed — so the reader must know the packing, and the shape is not the logical geometry.Where the gap is
Our pack importer's genre is ONNX + external data (
model.onnx, initializer walk,.fconsttail). An NPU2 pack has no graph and no ONNX file: it is a weights container plus aconfig.jsonwhose extra keys (flm_version,addr_qk/kk/kv) describe the vendor's own layout. So this is not a chunk-format problem — our engine already decodes the Q4NX chunk kind — it is a missing pack genre innpu-pack: read the container as the weights source, take the geometry fromconfig.json, and let our own kernels provide the compute.That is the work this row needs, and it is the same work for most of the 59 rows: ~40 of them are the same two-dtype safetensors shape (the 19 older repos that also ship xclbins are the ones with a different file set). Landing one container reader should therefore move the whole sweep, which is why the smallest pack was the right first target.
Row state: container identified and parsed; importer genre missing — not imported, not served, no measurement yet. Next step is the container reader behind
npu-pack, then import →--device npu→ the known-answer prompt → tok/s undernpu-hold.The report tool is committed (not in
/tmp) so the next row is a one-line repeat:tools/flm/q4nx_container_report.py <dir>.bong-water-water-bong commentedon Nov 10, 2026
Sweep classified: 58 of 59 containers read from their headers alone (no weights downloaded), and it corrects an estimate I made earlier in this thread.
Method: for each repo, resolve its largest
.q4nx, HTTP range-fetch the 8-byte header length, then range-fetch exactly that many bytes of JSON. 59 repos → header read for 58, ~2 requests each, a few hundred KB total instead of ~300 GB of payload.The correction first, because I stated the opposite an hour ago: I wrote that "~40 of them are the same two-dtype safetensors shape". The genre is uniform — every one is a safetensors container — but the contents are not:
So the importer's work list is 33 shapes, not one, and three dtypes appear beyond the two I measured on the first pack:
F32(in the mixed 24-layer and the layer-less packs) andU8(in a 24-layer pack). That changes the container-genre plan's job: a dtype-directed reader, not "BF16 and I8".One repo is not this genre at all:
FastFlowLM/smolvla-NPU2has no.q4nx— 3 files, a different shape entirely (a VLA model). It stays in the sweep but its row should say so rather than being counted as a container.What this buys the work: the importer can be driven by the shape census rather than by 59 individual packs. If the container-genre plan handles the top four shapes it covers 18 of 58; the remaining 15 shapes are one-offs and two-offs that each need a decision (does this pack's tensor naming match the same role map, and does its dtype mix appear anywhere else?). The next measurement is the per-shape tensor-name set, which the same range-fetch can produce — no weights, no device.
The first row's container (
Gemma3-270M-NPU2) islayers=18, {BF16: 110, I8: 127}and is one of the 33.bong-water-water-bong commentedon Nov 10, 2026
First row verified on this box: the container reader and the genre report build and pass.
The disk came back above the build threshold, so I configured and built
npu_pack_import_testin the worktree (llama.cpp submodule at5514ebea44) and ran it:And running it found two bugs — in my own test arms, not in the reader, which is worth recording because the reader was right both times:
data_offsetsspans is1280 + 819200. I had summed only the second span — i.e. the arm asserted a number I derived by hand instead of the number the offsets imply, which is precisely the kind of expectation that turns a test into a change-detector.std::ofstreamalive while the reader opened the same path, so the reader saw an empty file and reported "shorter than the 8-byte header length". The arm was testing its own missing flush. Scoping the writer fixed it.That is a small, concrete illustration of the night's recurring rule: a check that reports a number you cannot re-derive is a check you have to run to trust — and both failures came from the assertion side, not the code under test.
State of the first row now: the container is read (237 tensors, 0.503 GB,
{BF16: 110, I8: 127}), the genre is reported bynpu-pack infoinstead of refused at the first byte, and both are verified by a passing test on this box. What remains for the row is the part I said from the start: reading the container as a weights source — the compute mapping from HF tensor names to our kernels — and then import →--device npu→ known-answer prompt → tok/s undernpu-hold.Commits so far (all in
~/wt/zq-flm-npu2, branch1bit/f1-flm-npu2-container):848a0a107header report tool ·6ee38068econtainer reader + arms ·876177605plan/manifest wiring ·3924754ddgenre arm ·4aa5041afthe two arm fixes.bong-water-water-bong commentedon Nov 10, 2026
Reframing from the code, and it changes the size of this item: an NPU2 pack may need NO conversion at all.
I had assumed the row's work was "the compute mapping from HF tensor names to our kernels" (my own earlier comment says so). Then I read what our forward actually loads:
Three facts that together make the conversion unnecessary:
model.q4nxcontainer (Model::Modelmaps it; everyTensor::datapoints into that mapping — engine#1784/npu: read the token embedding table in place, one row per token, instead of materialising it (engine#1784) #1795);model.layers.N.pre_feedforward_layernorm.weight,model.layers.0.mlp.gate_exps_proj.weight, … exactly the spellings the NPU2 containers use;q4nx_dequant_plain_i8, "Gemma4") — which is the dtype mix the sweep found in 127 of the first pack's 237 tensors.So the pack is already in our runtime's shape, and what our
npu-packimporter converts is AMD's ONNX packs — a different genre that genuinely needs a graph walk. For an NPU2 pack the missing step may be nothing at all:1bit serve -m <pack dir> --device npu, with the geometry and rope semantics read from the pack's ownconfig.json(which #1794 is currently teaching to read the nestedrope_parameters— that work is on this path).What that means for the row: the first measurement is not "write a mapping" but "does it load" — and I tried it tonight:
The probe queued behind other lanes' NPU tickets and my wait expired — at that moment three lanes were holding the device (an F4 27B RSS probe, an F7 d3 pack run, and one more). Nothing failed; the ticket simply never came up. The probe is written and lives at
~/scratch-packs/f1-serve-probe.sh(server killed by a trap, port 18110, health wait, then a greedy/completion), so the next free window answers it in one load of a 0.54 GB pack.If it loads, the 59-row sweep becomes a serving sweep rather than a conversion project — and the only per-row work is the semantics in each pack's
config.json. If it does not, the failure's first line tells us which genre assumption was wrong, which is exactly the thing worth knowing before writing a mapping nobody needed.bong-water-water-bong commentedon Nov 10, 2026
Measured: the pack supplies the model, and what it lacks is our
npu/kernel set — exactly the split this item assumes. The row is now blocked on a kernel set, not on a container reader.The probe got its ticket (202 s queued behind other lanes, then 2 s) and
1bit serveanswered immediately with a named reason:That message is precise, and the check behind it (
app/unified.cpp:347) says exactly what the two halves are:The model half is already satisfied by the FLM pack — it ships
model.q4nx,config.jsonandtokenizer.json, all three, and our forward reads them by HF names with plain-int8 support. The missing half is ours: a kernel set for this pack's geometry, innpu/.So the earlier reframing holds and is now proven end to end: no conversion, no compute mapping — weights from the vendor's pack, kernels from us. That is precisely "FLM is a model source; our runtime and kernels do the work".
The geometry, so the kernel set is a specification rather than a request
config.jsonon the pack:Gemma3ForCausalLMsliding_window_pattern)which implies the projections a family set must carry:
(plus the M=16 twins, per the naming in
~/models/npu/d3-lite-elfs:full_i8_QKV_K640_N1536.elf,…_m16.elf, and so on.)What this hands to pi-zha
One kernel set for a hidden-640 / 18-layer gemma3 family, and then this row is measurable: the pack goes in place with
npu/beside it, and the run isnpu-hold -t 9 -- 1bit serve -m <pack> --device npu→ known-answer prompt → tok/s.Two things the sweep suggests for that set:
sliding_window 512with a pattern is in the pack's config (so the sliding/full split matters as it does on flint), and the int8 container carries a per-32-column scale, which ourq4nx_dequant_plain_i8already reads — so the kernel's input format is the same one the Gemma4 rows use.Also for the record, the probe's own behaviour was clean: ticket acquired and released in the same second as the exit,
gpu-hold released NPU-TOKENin the log, no server left behind.bong-water-water-bong commentedon Nov 10, 2026
One more measured fact for the hand-off, and it changes which route the rows should be measured on.
serve --device npu's refusal namednpu/, so I looked at what fills that directory and at what a working pack looks like on this box:and the files
has_lane_kernelswants are read by the lane runtime, not produced by anything here:So there are two routes, and the done-criterion's one is currently unprovisioned on this box:
1bit serve -m <pack> --device npu(the criterion)npu/lane kernel set:layer_ctx1/2/17.elf,lmhead.elf,layer.pdi1bit forward-serve -m <pack>/model.q4nx --kernels <dir>full_i8_*family set: e.g.~/models/npu/d3-lite-elfsThat matters for how this item is worked, and it is a decision worth making explicitly rather than discovering per row:
forward-serve --kernelsfirst (it is the route that exists, and it separates "the pack is good" from "the lane kernel set exists"), then treatserve --device npuas the packaging step once a lane set is built for a family;npu/lane set for one family and settle the criterion on that one before sweeping.Either way the per-row work is the same kernel geometry; the difference is whether the first measurement is blocked on a build that does not exist yet.
What I am not doing, deliberately: running a mismatched
full_i8_*set against the gemma3 pack to read the refusal. A geometry mismatch is exactly the case that can end in a TDR rather than a refusal, and the rule on this box is that the first TDR stops the lane — so the missing-projections list is better obtained from the geometry arithmetic (already given:QKV K640 N1536,O K1024 N640,GU K640 N4096,D K2048 N640) than by provoking the device.bong-water-water-bong commentedon Nov 10, 2026
Which rows can be measured TODAY: per-projection coverage against the ELF family sets on this box. Four of 57, and my first pass wrongly said nine.
Method: read every pack's
config.json(57 of 59 parsed; 2 have no top-level ortext_configgeometry), compute its four projection shapes, and check them against thefull_i8_<role>_K<n>_N<m>.elfnames in each set under~/models/npu/:A pack counts as covered only if all four projections are present, because a family set is a geometry, not a hidden size.
And the correction, because my first pass was wrong in the optimistic direction: matching on (hidden, intermediate) gave nine candidates — 4 at hidden 4096/IM 12288 for
d3-flash-elfs, 5 at hidden 1024/IM 3584 ford3-lite-elfs— and all nine fail the per-projection check, because QKV'sNdepends on heads/kv/head_dim, not on hidden and IM.d3-flash-elfshasQKV_K4096_N10240andN8192; Qwen3-8B needsN6144. A hidden+IM match is a hypothesis; the ELF'sNper role is the test.So the sweep's shape is now: 4 rows measurable today with an existing set, and 53 that need an ELF family set for their geometry — of which the biggest clusters are hidden 2560 with IM 10240 / 9728 / 9216 / 10752 / 10240 (10 packs), hidden 2048 with IM 8192 / 6144 / 11264 / 10752 / 7680 / 6912 / 512 (13 packs), and hidden 3072 (2 packs).
Next step, and it is the row's actual measurement:
Qwen3-0.6B-NPU2is the smallest of the four (0.70 GB), andqwen3-coder-kernelscovers it — so that row can go all the way to a tok/s figure with an existing kernel set. I will do that one and report before asking anyone for a new family.bong-water-water-bong commentedon Nov 10, 2026
First row answered: an FLM NPU2 model source serves on OUR runtime and OUR kernels, and it answers correctly
Qwen3-0.6B-NPU2— the smallest of the four packs that an existing ELF family set covers — runs end to end. No conversion, no import step: the vendor'smodel.q4nx+config.json+ tokenizer, and ourqwen3-coder-kernelsset.Criterion 3 — the known-answer prompt on the model's own template:
Paris,finish_reason: stop, and three streamed requests returned identical content. (It is a thinking model: without/no_thinkthe answer lands inreasoning_contentand gets cut off bymax_tokens— the template matters, which is why criterion 3 says "the model's own template".)Criterion 4 — TTFT and decode, measured:
The decode figure is head-dominated and that is worth saying rather than leaving as a number. This model is 0.6B with
hidden 1024 / vocab 151936, solm_headis 156M parameters — 26 % of the model — and it is read once per token. That is the shape of a slow rate on a small model, and the next measurement that would separate it is a longer run with the head cost isolated (or a model in the same family with a smaller vocabulary).Criterion 5 — the set and the engine:
qwen3-coder-kernels(81 ELFs,full_i8_*family, M=128 decode shape), deployed build1bit 0.0.1from~/wt/payload-c25/…/1bit/1bit(2026-11-04 23:31).What this settles, and the one criterion it does not
It settles the item's central question: the FLM NPU2 packs are model sources that need no conversion. The pack is the container our runtime reads, with the tensor names our forward addresses; the only missing half is the kernel geometry, and
qwen3-coder-kernelshappens to cover this one.The criterion it does not settle is the route. The done definition says
1bit serve --device npu, whosenpu/gate wants the lane kernel set (layer_ctx1/2/17.elf,lmhead.elf,layer.pdi) — which does not exist on this box for any family. This row ran throughforward-serve --kernels, the route the F7 lane uses for d3 packs. So one of two things needs deciding, and it is an owner-level call rather than mine:--device npu(either route)", which is what the engine can actually do today and what this row demonstrates; orserve --device npu'snpu/layout as the packaging target and build one family's lane set, which makes this row's measurement aforward-serveprerequisite rather than its completion.And it makes the sweep actionable: 3 more rows (
Gemma4-E2B-IT,Gemma4-E4B-IT,Nemotron-3.5-Bolt-E4B) are covered by the same existing set and can be measured the same way; the other 53 need a family set, clustered at hidden 2560 (10 packs), hidden 2048 (13), hidden 3072 (2).Probe scripts are committed in the record:
~/scratch-packs/f1-fserve-probe.sh(serve + answer),f1-fserve-measure.sh(TTFT + 3 identical),f1-decode-rate.sh(decode rate). All three takenpu-hold, kill the server in a trap, and released the token cleanly on every run tonight.bong-water-water-bong commentedon Nov 10, 2026
Second row: got to the last missing kernel and names it exactly — and it corrects my own coverage predicate.
Gemma4-E2B-IT-NPU2(5.99 GB, a multi-component pack:model.q4nx4.67 GB text tower +vision_weight.q4nx337 MB +audio_weight.q4nx952 MB) loaded and initialised kernels one after another, then refused:That is the best possible failure: it names the one file to build, by its geometry, after initialising everything it did have. The row's blocker is now a single ELF, not an investigation.
And it corrects the coverage predicate I published an hour ago. I checked four roles (
QKV,O,GU,D) and concludedqwen3-coder-kernelscovers this pack; the model also needs:PLEGATEK1536_N256K1536_N512QKV2/O2(a second attention variant)QKV_K1536_N5120✓ /O_K4096_N1536✓PLEGATEaboveSo the predicate needs at least six roles, and my "4 of 57 fully covered" is an upper bound, not a count:
Qwen3-0.6Bgenuinely ran (it has no per-layer gate), butGemma4-E2B-ITand by extension the other gemma4 rows are one ELF short. The corrected statement is 1 row measured, 3 rows one-or-more ELFs short, 53 rows needing a family set — and the way to know for any of them is to run it, because the loader's refusal names the gap exactly.What this hands to pi-zha, concretely:
full_i8_PLEGATE_K1536_N512.elf(+ its_m16twin, per the naming convention) for the gemma4-e2b family — and, for the sweep, the corrected role list so a coverage check does not repeat my mistake:QKV,O,GU,D,PLEGATE, and the2-suffixed variants where a pack carries two attention shapes.Also worth recording about the pack itself: it is genuinely multi-component (vision and audio towers beside the text tower), and 709 tensors with structures like
inp_gate.weight [8,1536,32]andinp_gate.weight_prefill [4,6,16384]— i.e. the per-layer embedding table that F4 describes. The text tower loads frommodel.q4nxalone, so F1's scope (text serving) is not blocked by the other two towers; F6 owns the vision side.bong-water-water-bong commentedon Nov 10, 2026
The coverage question is now answered causally, and it completes the correction: the other three packs on my "fully covered" list all carry a per-layer gate, which is exactly the ELF
gemma4-e2brefused for.Measured by range-fetching each container's header (no weights) and looking for
inp_gate/plegate/per_layertensors:Those three are exactly the other three entries on my earlier "4 of 57 fully covered" list. So the sentence I published an hour ago — that four packs were servable with an existing ELF set — was wrong in a way that is now fully explained:
Qwen3-0.6B-NPU2Gemma4-E2B-IT-NPU2missing full_i8_PLEGATE_K1536_N512.elfGemma4-E4B-IT-NPU2PLEGATEgeometryNemotron-3.5-Bolt-E4B-NPU2So the honest count is 1 row measured, 3 rows blocked on a
PLEGATEELF, 55 rows needing a family set — and the discriminator is not hidden size or intermediate size but whether the container carries per-layer gate tensors at all. That is a property of the container, readable from the header in two HTTP range requests, which makes it a cheap pre-check for any future coverage claim: my four-role predicate missed a fifth role that only 3 of 58 packs need.Hand-off to pi-zha, now complete for this family:
full_i8_PLEGATE_K1536_N512.elf(+_m16) — unlocksGemma4-E2B-IT-NPU2(and the sweep's 1536-wide row);Gemma4-E4B-ITandNemotron-3.5-Bolt-E4B(theirPLEGATEN follows their per-layer table width —qwen3-coder-kernelsalready has aPLEGATE_K1536_N256, so the role name and machinery exist; only the geometry is missing).And one methodological note for the family's docs, since this is the second time tonight a coverage claim of mine was too generous: a pack is covered when every role its container names has an ELF, and the way to know is to run it or to read its header — not to compute four projections from
config.jsonand assume that is the whole list. The loader's refusal is the authority, and it names the gap exactly.bong-water-water-bong commentedon Nov 10, 2026
The sweep is not one format with 59 geometries — it is 22 architectures, and that changes what "support all of it" costs.
Measured by range-fetching each container's header (no weights) and normalising tensor names (
.N.→.*.), then grouping by the resulting name set:No single signature covers more than 9 packs, so this is not a handful of families with long tails — it is 22 shapes of roughly comparable size. And the markers inside those signatures say which of them are architecture differences rather than naming ones:
linear_attn/ssm_conv.(conv.conv.weight,conv.in_proj.weight)model.layer.(singular)model.layers., so these miss by spelling aloneinp_gate(per-layer gate)PLEGATEELF class found earliermodel.visual.dflash.Three consequences, and the first is the one that changes the plan:
model.layer.vsmodel.layers.). The honest split is: kernels for some, forward support for others, a name normalisation for 3, and one is not a model at all.model.layer.variant is a normalisation in the tensor lookup, anddflash.should be excluded from the sweep rather than attempted (it is a draft package, F2's item). Both are single-line judgements that would otherwise be discovered one probe at a time.What I would hand to the owner with this: the sweep's shape is now measurable and the remaining work is scoped by architecture, not by pack count. If the intent was "make the 59 packs run", the measurement says that means 22 forward/kernel configurations, of which one (Qwen3-0.6B's signature) is proven end to end tonight, three are one ELF away, and eleven of the 22 groups contain a layer type our forward may not have.
Next measurement from here, and it needs no device: for each of the 22 signatures, list the distinct layer-type markers so the ones needing forward work are separated from the ones needing only kernels. That is the same range-fetch, and it produces the actual work breakdown.
bong-water-water-bong commentedon Nov 10, 2026
Claiming this (pi-zha, sweep lead) — starting with the current-generation, weights-only packs
Claiming on the board per the standing loop. Why this lane: everything it asks for is the loop I have been running all night on the AMD packs — import with
zero gaps →
serve --device npuundernpu-hold→ a known-answer prompt in the pack's own chat template, 3 identical repeats → a measured tok/s and TTFT,then a row that names the kernel set and the engine commit.
Starting point, from the issue's own inventory: the current generation (no
.xclbin, 2 ×q4nx, 7-9 files) is the in-scope shape, and the smallest isNemotron-3.5-Bolt-E4B-DFlashat 0.27 GB. That is a cheap first probe of the genre — if a weights-only FLM pack imports and serves, the whole family is asweep of downloads rather than a new reader; if it does not, the refusal will say which genre assumption breaks, and it costs 0.27 GB to learn.
Order I intend: smallest current-generation pack first (0.27 GB), then the next two by size, then whatever the first three teach decides whether the 22-24 GB
packs are worth a lane's time before the disk floor is comfortable again.
What I will not do: touch the older
.xclbinpacks. Our artifacts are full ELFs and the plan forbids xclbin as a target or a fallback, so those are out ofscope by rule rather than by difficulty.
Will report each pack as a row with its numbers, and will say plainly if the genre needs reader work that belongs to another lane rather than pretending a sweep
can carry it.
bong-water-water-bong commentedon Nov 10, 2026
First probe (1.0 GB,
LFM2.5-1.2B-Thinking-NPU2): the pack is safetensors with a JSON table, our serve reads its geometry correctly and then cannot read its weights — so #1786 is a format route, not a sweepDownloaded the smallest model pack of the current generation (skipping the 0.27 GB one, which is a DFlash draft and belongs to #1780):
FastFlowLM/LFM2.5-1.2B-Thinking-NPU2, 959 MB, 7 files, onemodel.q4nx, no.xclbin— exactly the in-scope shape the inventory describes.What works:
1bit serve -m <dir> --device npuopens it as a model-generic container, reads the geometry fromconfig.json(
H=2048 NH=32 NKV=8 HD=64 IM=8192, rope theta 1e6), derives all four designs, loads the ELFs from thenpu/dir I supplied, and comes up:decode family: 4 role(s) at M=16,ready on … (full ELF).What does not: every request fails with
And the reason is the container format, which is not ours.
model.q4nxis a safetensors-style file —<u64 header_len><JSON table><payload>— withThree things a reader has to deal with, all measured:
grep 'scale|zero'over the 149 names → 0) while every I8 weight is exactly 1.0000 byte/element — so the quantisation envelope is not stated in the file the way ours is;shortconv.conv.weighttensors — LFM2's conv blocks are part of the model, and the forward has no shortconv operator;model.layers.N.mlp.down_proj.weight), not our container's sections.So the honest reading of this item's first data point: the current-generation FLM sources are a distinct container genre (safetensors table + a packed int8 whose scale convention is not in the table) plus a new operator (
shortconv), and neither is a sweep. Our side already does the parts that are generic — geometry, design derivation, ELF loading, serve — which is why the failure is a clean, named one at the first weight read.Next, and it is a decision for the owner/plan rather than for a sweep: reading this table is much easier than the AMD
.fconstwork (a JSON dict with explicit offsets — own code, no heuristics), but the I8 envelope needs to be established from the payload (the 2.5 B/weight ratio I first computed was wrong: at 1 B/element the extra is elsewhere, and the shape's5120already shows the packs pad K). I will write that up as a 15-line design on this issue before touching code, per the plan's rule for anything over a day, and in the meantime report the remaining 33 current-gen packs as format-blocked, not sweep-blocked so nobody queues them as a re-run task.bong-water-water-bong commentedon Nov 10, 2026
Convergence measured against the kernel lane's two device-proven families — and it corrects something I said on that PR an hour ago.
npu-kernels#53 proved two b8 families bit-exact on the device (maxabs = 0): 1536x2048 ("Qwen 1.5B class, 13 packs") and 2304x4096 ("gemma-2-2b class, 4 packs"). I wrote on it that "those two classes' geometries match groups in my sweep's uncovered 55, so naming the mapping would move rows." I have now done the mapping against the 57 packs whoseconfig.jsonI can read, and it moves almost nothing:Gemma4-E2B-IT-NPU2So the kernel lane's proven geometries serve the AMD 1.9 packs (its "13 packs" and "4 packs" are rai packs, not these), and the one overlap is the row I already probed — which still needs its
PLEGATEELF, theper_layer_inputscarry (#1825) and the.fconstblock layout (W1c). I over-called that in the review and am correcting it here; the correct statement is that the W3b work unblocks the AMD side, and F1's side needs its own geometries.What the sweep's hidden-size distribution says the families should be, instead:
Two families cover 32 of 58 packs: hidden 2048 (17) and 2560 (15). That is where a b8 generator would pay, and neither is the 2304 geometry that was just proven. The next four (4096, 1024, 5120, 3072) cover another 18; the remaining 8 are one-offs.
Two things worth deciding with this number, and they are the kernel lane's call, not mine:
full_i8_*naming convention.Either way the sweep's remaining work is now sized: 2 families → 32 rows, 4 more → 18 rows, 12 one-offs → 8 rows, plus the 23 hybrid/convolutional packs that need forward work before any family helps them.
bong-water-water-bong commentedon Nov 10, 2026
Design note (15 lines, posted before any code): the FLM container is already readable — the open question is what its int8 weights mean
The first probe's "distinct container genre" reading was too pessimistic, and the existing tooling proves it.
model.q4nxis plain safetensors, andtools/safetensors_to_q4nx.py packreads the whole file — 148 of 149 tensors pack cleanly, including the 30shortconv.conv.weighttensors (as BF16) and anembedding tiled to
I8 [16384, 5120] q4_1— and then stops on exactly one thing:So the plan is not "write a container reader". It is:
I8tensor means in these packs. In the table the 93 I8 tensors are exactly 1.0000 byte per element (down_proj [2048, 5120]=10,485,760 B) and there is no scale or zero tensor anywhere in the 149 names, nor any quantisation field in
config.json(40 keys, checked forquant|scale|bits|int8|group|zero→ nothing). Three candidates, in the order I would test them: (a) the payload interleaves per-group scales in bytes the tableaccounts for under the padded second dimension (
5120against a 2048-wide row is 2.5 B per 4 weights, exactly the room a 4-bit-group scale table needs);(b) a per-tensor scale carried in the file outside the tensor table; (c) the codes are unscaled and the runtime is expected to normalise per row.
model.token_embd.weight-style neighbours — the packs ship BF16 for every norm, and a wrong envelope shows up as a row-norm distribution off by orders ofmagnitude, which is the same check that caught the tail-head envelope story.
I8(2-D → q4_1 re-quantisation is the natural step, since our kernels already take_b8tiles), or add an int8 containerpath — the second only if step 1 shows the codes already carry their scales and re-quantising would lose precision.
shortconvis a separate, smaller item: LFM2's 30 conv blocks are a host-side depthwise conv overconv_L_cache = 3positions; it is an operator, not aformat, and the converter already carries the weight through as BF16.
every other row, and
Hy-MT2-1.8B-NPU2(1.5 GB) is the second pack to confirm it generalises.Nothing here needs the owner's decision to start: step 1 is measurement on a pack I already have. What it does need is the same thing the AMD
.fconstworkneeded — a reference to check a decode against — and these packs ship BF16 norms plus an upstream GGUF, so both are available.
If another lane is already on the FLM container (this is #1786's model-sources half and #1780's draft half), say so here and I will hand the step-1 measurement to
them rather than run it twice.
62 remaining items
npu: the per-op forward runs GPT-OSS-20B-NPU2 on the host: D2560/MXFP…
bong-water-water-bong commentedon Nov 12, 2026
The third key for gemma4 is in its config, and it is an exact layer boundary:
use_double_wide_mlpwithnum_kv_shared_layers.#2270 (now on
main) gives the writer a (layer-elided name, layer type) key, which resolves MiniCPM-V, bolt and gemma3 to 0 spans and leaves gemma4 with six:gate_proj,up_projanddown_projeach in two shapes under both layer types. The remaining question was what the third key is. It is not a mystery:A contiguous boundary, the last
num_kv_shared_layers(20) of 35 layers, and the two widths are 1:2 — so the MLP is doubled on that upper block. It is not correlated withlayer_typesat all (both attention kinds occur on both sides), which is precisely why a (role, layer type) key produced spans instead of rows.So the writer's key is (layer-elided name, layer type) plus "is this layer in the double-wide block", and the third component is declared, not inferred. No guessing and no per-layer table:
use_double_wide_mlpsays it happens,num_kv_shared_layerssays how many layers at the top, and the measurement above confirms both (20 of 35, contiguous, exactly the wider half).State of the key across the family, measured: MiniCPM-V 35B / 1B, bolt-e4b and gemma3-270m are two-key (0 spans); gemma4-e2b is three-key with the third component now identified. That is the whole role→destination mapping problem for the container genre, and none of it required a device.
What is still open for 3c (unchanged by this): the placement code itself — turn each (stem, key) into the destination field's slot in the layout — and the
gemma_local/container_extratensors that this engine's forward does not read at all, which a full-family writer must consciously skip rather than copy.docs: the [N][K] expert-slice rule also fires on MiniCPM-V-4.7-35B-A3…
npu: Hy-Memory-30B-A3B (hy_v3) runs on the existing MoE path, read th…
forward-serve: GPT-OSS replies are split by harmony markup, not qwen3…
forward: read the hub Qwen3.5 packs' Q8 code order from the codes (refs
forward: every un-taken tie measurement says why and what it keeps (r…
forward: an unstated tie_word_embeddings is decided from the two tabl…
bong-water-water-bong commentedon Nov 12, 2026
CLOSE CONDITION
The bar. This item is done when every in-scope
FastFlowLM/*-NPU2repo has a row meeting the body's five items. Each row is measured on the hub's own files at enginemainwith1bit serve -m <pack dir> --device npu, insidenpu-hold:npu-pack importstep);/v1/embeddingswithin cos 0.999 of transformers' own model on the same ids;The final table is posted on this issue.
In scope: 46 of the 59 NPU2 repos. Out per the body, or owned by another issue:
smolvla-NPU2is a robot-policy tarball with nomodel.q4nx, config or tokenizer, so there is nothing for the chat route to serve. I propose it is out (owner's call).Where it stands (board, 11-12 23:55Z): 30 rows DONE on the hub files. That includes the last day's fixes, all merged: harmony replies (#2298), the Q8 code order of the 0.9.45 Qwen3.5 packs (#2301), and an unstated
tie_word_embeddings(#2309).Left: PRs still to write or land (no others):
Embedding-Gemma-300M-NPU2on/v1/embeddings. Open, ready at3985fd9; host cos 0.99999, NPU 0.9991 against transformers.Qwen3.6-35B-A3B-NPU2: the pack's small BF16 matrices stored[in, out](ssm_alpha/beta) and itsshared_expert_gatename. The host forward reasons with both fixes and was noise without them; the device row is running. Stacked on npu: Embedding-Gemma-300M-NPU2 serves /v1/embeddings on the per-op forward (refs #1786) #2314 (it extends that PR's BF16 reader).Qwen3.8-27B-NPU2: the device row. This becomes a PR only if the row needs a code change; it has not been measured on currentmain.Left, device rows only (no PR):
Nanbeige4.1-3B-NPU2(host-checked, device row owed),medgemma-4b-it-NPU2(ttft), and a pass over the 11 rows measured on 11-11 (LFM2/LFM2.5 ×3, Llama-3.2 ×2, Hy-MT2, Qwen3-1.7B, Qwen3-4B ×3, Phi4-mini) to fill whichever of items 4–5 their rows lack.Reachable this week: yes, unless the 27B needs more than one PR. Perf follow-ups stay listed under this issue, not as new slices: device MXFP4 experts for GPT-OSS, and the per-token prefill of gemma4, gpt_oss and the MoE packs.
tests: embed_pooled_test writes its vector through onebit_fopen_priva…
npu: Embedding-Gemma-300M-NPU2 serves /v1/embeddings on the per-op fo…
forward: Qwen3.6-35B-A3B-NPU2 answers -- [in, out] BF16 matrices and …
forward: Qwen3.6-35B-A3B-NPU2 answers -- [in, out] BF16 matrices and …
bong-water-water-bong commentedon Nov 13, 2026
The full table: every FastFlowLM NPU2 source
The bar is the one in the CLOSE CONDITION above. Each row is measured on the hub's own files with
1bit serve -m <pack dir> --device npu, insidenpu-hold, with amdxdna clean before and after. A row passes when the pack's own template returns the known answer, 3 identical replies, with decode tok/s and ttft measured. A pack that always reasons is checked on one request long enough to reach the answer, with the reasoning split out. The kernel sets were built locally for each geometry; they are named on the board and are not shipped.59 repos:
/v1/embeddings.smolvla-NPU2, is not a chat source: a robot-policy tarball with nomodel.q4nx, config or tokenizer. Out, per the plan.Rows that first ran on
~/.config/flm's rewritten copies or on the fused route were re-run on the hub files and the per-op route tonight. Those are 6, 9, 11, 12, 14, 17, 19, 26, 28, 29, 30, 31, 33, 36 and 38.Nemotron-3.5-Bolt-E4B-DFlash-NPU2Gemma3-270M-NPU2D3-Edge-NPU2Embedding-Gemma-300M-NPU2/v1/embeddingsWhisper-V3-Turbo-NPU2Qwen3-0.6B-NPU2smolvla-NPU2D3-Lite-NPU2LFM2.5-1.2B-Thinking-NPU2LFM2-1.2B-Tool-NPU2LFM2.5-1.2B-NPU2LFM2-1.2B-NPU2Gemma3-1B-NPU2Llama-3.2-1B-NPU2Qwen3.5-0.8B-NPU2Qwen3.8-27B-DFlash2-NPU2Hy-MT2-1.8B-NPU2LFM3-1.7B-NPU2Qwen3-1.7B-NPU2LFM2-2.6B-Transcript-NPU2LFM2-2.6B-NPU2MiniCPM-OCR-1.0-NPU2D3-Nano-NPU2MiniCPM-V-4.7-1B-NPU2Qwen2.5-3B-Instruct-NPU2Llama-3.2-3B-NPU2Nanbeige4.1-3B-NPU2Qwen3.5-2B-NPU2Qwen3-4B-Instruct-2507-NPU2Qwen3-4B-Thinking-2507-NPU2Qwen3-4B-NPU2D3-Mini-NPU2Phi4-mini-Instruct-NPU2Gemma3-4B-Text-NPU2Qwen2.5-VL-3B-Instruct-NPU2Qwen3-VL-4B-Instruct-NPU2Translategemma-4B-Instruct-NPU2medgemma-4b-it-NPU2medgemma-1.5-4b-it-NPU2Gemma3-4B-NPU2Qwen3.5-4B-NPU2Llama-3.1-8B-NPU2Deepseek-R1-Distill-Llama-8B-NPU2Qwen3-8B-NPU2Gemma4-E2B-IT-NPU2D3-Flash-NPU2Qwen3.5-9B-NPU2Gemma4-E4B-IT-NPU2Nemotron-3.5-Bolt-E4B-NPU2Gemma4-12B-IT-NPU2DeepSeek-R1-0528-Qwen3-8B-NPU2GPT-OSS-Safeguard-20b-NPU2GPT-OSS-20B-NPU2D3-Turbo-NPU2D3-Ultra-NPU2Hy-Memory-30B-A3B-NPU2Qwen3.8-27B-NPU2MiniCPM-V-4.7-35B-A3B-NPU2Qwen3.6-35B-A3B-NPU2What it took (all merged): #2250, #2254, #2259, #2261, #2265, #2274, #2279 (families and formats); #2298 (harmony replies); #2301 (the 0.9.45 Qwen3.5 packs' Q8 code order); #2309 (an unstated
tie_word_embeddings); #2314 (the bidirectional encoder on/v1/embeddings); #2318 ([in, out]BF16 matrices and theshared_expert_gatename).Perf follow-ups (not correctness; kept as this item's notes, no new slices):
Closing on this table.