Skip to content

npu: every FastFlowLM/*-NPU2 model source imports and serves on --device npu (FLM 1.0.12 model sources, 59 repos) #1786

Closed

Description

@bong-water-water-bong

Scope: every FastFlowLM/*-NPU2 model source imports and serves on --device npu on our own runtime and kernels. Part of the FLM 1.0.12 parity work; this is the model-sources item.

The inventory, measured from the public HF API tonight

https://huggingface.co/api/models?author=FastFlowLM&limit=200 → 65 repos, of which 59 carry NPU2; none is gated. Sizes are the sum of the repo's file blobs (?blobs=true), 0.27 GB → 24.26 GB.

Two pack generations, and the difference matters for us:

generation files .xclbin q4nx example
older 8-14 4-7 1 Gemma3-270M-NPU2 (0.54 GB, 5 xclbin)
current 7-9 0 2 MiniCPM-V-4.7-1B-NPU2 (2.22 GB)

So the current packs ship no vendor kernel artifacts at all — they are weights only, which is exactly the shape this item needs: the model source is the Q4NX container, and the kernels come from our own runtime. The older packs carry .xclbin files beside the weights; those are not used by anything here (our artifacts are full ELFs; no xclbin as a target or a fallback), and a pack that only has them is not in scope for serving.

New since 2026-10-30 (lastModified): Nemotron-3.5-Bolt-E4B-DFlash (0.27 GB), D3-Edge (0.62), MiniCPM-V-4.7-35B-A3B (22.21), MiniCPM-OCR-1.0 (2.01), Qwen3.8-27B-DFlash2 (1.39), Hy-Memory-30B-A3B (17.88), MiniCPM-V-4.7-1B (2.22).

Definition of done, per repo, on the pack's own files

  1. it imports with zero gaps and its self-check passes against a named reference;
  2. it serves on --device npu;
  3. it answers a known-answer prompt with the model's own chat template, 3 identical requests returning identical text;
  4. decode tok/s and time-to-first-token measured under npu-hold;
  5. the row names the kernel set and the engine commit that produced it.

A row is only DONE with all five. Mulit-component packs (Gemma4, Qwen3.5, Qwen3-VL, MiniCPM-V) are in scope; the vision towers and Whisper have their own items and are tracked separately.

Plan

  • One board row per repo, in the board's FLM NPU2 section, swept smallest first (0.27 GB → 24.26 GB), so early rows are cheap and the container/importer contract is settled on small packs before the 22-24 GB ones.
  • The same-family packs (D3 ×7, LFM2 ×4, LFM2.5 ×2, LFM3 ×1, Qwen3 ×8, Qwen3.5 ×4) should collapse to a handful of distinct container/geometry cases; the sweep's job is to find out which, so a family that works once is checked rather than re-derived.
  • Rows go to pi-zha; the first row's job is to say which importer path and which kernel set the whole family needs.

Not in this item

Speculative decode (its own item), the 27B decode figure, embedding-table-from-disk, flash serving, Whisper, MiniCPM-V's vision tower, Decision 3.0 sizes, HRX builds, server behaviours — all tracked separately.

One input missing: the fact-base digest that was expected at amd-oss/npu-re/docs/FACTS-fastflowlm-v1012-delta.md is not on this box yet (only the v1.0.7 artifacts digest is). The inventory above is from the public release notes and the HF API, so nothing here depends on it, but the per-model container details will want it when it lands.

Activity

  1. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    First row measured: the NPU2 model source is a safetensors container, and our importer does not know that genre.

    Working the sweep smallest-first, FastFlowLM/Gemma3-270M-NPU2 (0.54 GB, first standalone chat pack — I skipped the 0.27 GB Nemotron-3.5-Bolt-E4B-DFlash, which is a draft package and belongs to the speculative-decode item).

    What the pack is

    Downloaded the weights-only files (skipping the vendor .xclbins, which nothing here may use): config.json, model.q4nx (503 MB), tokenizer.json, tokenizer_config.json.

    $ xxd -l 32 model.q4nx
    00000000: 8067 0000 0000 0000 7b22 6d6f 6465 6c2e  .g......{"model.
    00000010: 656d 6265 645f 746f 6b65 6e73 2e77 6569  embed_tokens.wei
    

    i.e. 8-byte little-endian header length (26,496) then JSON — a plain safetensors container, which matches the container facts our own notes already record. A report from our own reader (tools/flm/q4nx_container_report.py, header only, no weights, no device):

    model.q4nx: header 26496 B, 237 tensors, payload 0.503 GB
      dtypes {'BF16': 110, 'I8': 127}
      layers seen: 18   metadata keys: []
        model.layers.0.input_layernorm.weight          BF16 [640]
        model.layers.0.self_attn.q_proj.weight         I8   [320, 1280]
        model.layers.0.mlp.down_proj.weight            I8   [640, 1280]
        …
    

    Three facts the importer will have to live with:

    1. the tensor names are the model's own HF names (model.embed_tokens.weight, model.layers.N.self_attn.q_proj.weight, …), not per-op graph names — there is no graph at all;
    2. two dtypes only: BF16 for norms and the embedding, I8 for every matrix. No Q4NX dtype tag in this pack, and no __metadata__ block (the container notes describe one written by the older converter; this pack has none);
    3. the I8 matrices carry twice their logical element count (down_proj is [640, 1280] for a 640-wide FFN), i.e. the storage is packed — so the reader must know the packing, and the shape is not the logical geometry.

    Where the gap is

    $ 1bit npu-pack info ~/scratch-packs/gemma3-270m-npu2
    1bit npu-pack: cannot open …/model.onnx
    

    Our pack importer's genre is ONNX + external data (model.onnx, initializer walk, .fconst tail). An NPU2 pack has no graph and no ONNX file: it is a weights container plus a config.json whose extra keys (flm_version, addr_qk/kk/kv) describe the vendor's own layout. So this is not a chunk-format problem — our engine already decodes the Q4NX chunk kind — it is a missing pack genre in npu-pack: read the container as the weights source, take the geometry from config.json, and let our own kernels provide the compute.

    That is the work this row needs, and it is the same work for most of the 59 rows: ~40 of them are the same two-dtype safetensors shape (the 19 older repos that also ship xclbins are the ones with a different file set). Landing one container reader should therefore move the whole sweep, which is why the smallest pack was the right first target.

    Row state: container identified and parsed; importer genre missing — not imported, not served, no measurement yet. Next step is the container reader behind npu-pack, then import → --device npu → the known-answer prompt → tok/s under npu-hold.

    The report tool is committed (not in /tmp) so the next row is a one-line repeat: tools/flm/q4nx_container_report.py <dir>.

  2. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Sweep classified: 58 of 59 containers read from their headers alone (no weights downloaded), and it corrects an estimate I made earlier in this thread.

    Method: for each repo, resolve its largest .q4nx, HTTP range-fetch the 8-byte header length, then range-fetch exactly that many bytes of JSON. 59 repos → header read for 58, ~2 requests each, a few hundred KB total instead of ~300 GB of payload.

    The correction first, because I stated the opposite an hour ago: I wrote that "~40 of them are the same two-dtype safetensors shape". The genre is uniform — every one is a safetensors container — but the contents are not:

    distinct (dtypes, layers) shapes: 33
       6 repos: layers=36  dtypes={BF16: 146, I8: 253}
       5 repos: layers=34  dtypes={BF16: 206, I8: 239}
       4 repos: layers=16  dtypes={BF16: 56,  I8: 93}
       3 repos: layers=24  dtypes={BF16: 134, F32: 36, I8: 187}
       2 repos: layers=?   dtypes={BF16: 302, F32: 60, I8: 371}
       2 repos: layers=32  dtypes={BF16: 67,  I8: 225}
       2 repos: layers=28  dtypes={BF16: 114, I8: 197}
       2 repos: layers=24  dtypes={BF16: 314, I8: 97, U8: 24}
    

    So the importer's work list is 33 shapes, not one, and three dtypes appear beyond the two I measured on the first pack: F32 (in the mixed 24-layer and the layer-less packs) and U8 (in a 24-layer pack). That changes the container-genre plan's job: a dtype-directed reader, not "BF16 and I8".

    One repo is not this genre at all: FastFlowLM/smolvla-NPU2 has no .q4nx — 3 files, a different shape entirely (a VLA model). It stays in the sweep but its row should say so rather than being counted as a container.

    What this buys the work: the importer can be driven by the shape census rather than by 59 individual packs. If the container-genre plan handles the top four shapes it covers 18 of 58; the remaining 15 shapes are one-offs and two-offs that each need a decision (does this pack's tensor naming match the same role map, and does its dtype mix appear anywhere else?). The next measurement is the per-shape tensor-name set, which the same range-fetch can produce — no weights, no device.

    The first row's container (Gemma3-270M-NPU2) is layers=18, {BF16: 110, I8: 127} and is one of the 33.

  3. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    First row verified on this box: the container reader and the genre report build and pass.

    The disk came back above the build threshold, so I configured and built npu_pack_import_test in the worktree (llama.cpp submodule at 5514ebea44) and ran it:

    $ cmake --build build --target npu_pack_import_test -j 8      → ok
    $ ./build/npu_pack_import_test tests/golden
    all npu-pack checks passed -- pack leg SKIPPED, no pack directory given
    

    And running it found two bugs — in my own test arms, not in the reader, which is worth recording because the reader was right both times:

    1. the payload arm expected 819200 and the reader said 820480. The reader was correct: the sum of the data_offsets spans is 1280 + 819200. I had summed only the second span — i.e. the arm asserted a number I derived by hand instead of the number the offsets imply, which is precisely the kind of expectation that turns a test into a change-detector.
    2. the implausible-length arm left its std::ofstream alive while the reader opened the same path, so the reader saw an empty file and reported "shorter than the 8-byte header length". The arm was testing its own missing flush. Scoping the writer fixed it.

    That is a small, concrete illustration of the night's recurring rule: a check that reports a number you cannot re-derive is a check you have to run to trust — and both failures came from the assertion side, not the code under test.

    State of the first row now: the container is read (237 tensors, 0.503 GB, {BF16: 110, I8: 127}), the genre is reported by npu-pack info instead of refused at the first byte, and both are verified by a passing test on this box. What remains for the row is the part I said from the start: reading the container as a weights source — the compute mapping from HF tensor names to our kernels — and then import → --device npu → known-answer prompt → tok/s under npu-hold.

    Commits so far (all in ~/wt/zq-flm-npu2, branch 1bit/f1-flm-npu2-container): 848a0a107 header report tool · 6ee38068e container reader + arms · 876177605 plan/manifest wiring · 3924754dd genre arm · 4aa5041af the two arm fixes.

  4. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Reframing from the code, and it changes the size of this item: an NPU2 pack may need NO conversion at all.

    I had assumed the row's work was "the compute mapping from HF tensor names to our kernels" (my own earlier comment says so). Then I read what our forward actually loads:

    npu/forward/src/q4nx_forward.cpp:417   // ---- config.json FIRST, then model_load ----
    npu/forward/src/q4nx_forward.cpp:629   model_tensor_by_name(mw_, "model.layers.0.mlp.gate_exps_proj.weight") != nullptr,
    npu/forward/src/q4nx_forward.cpp:1590  if (!rd("model.layers.%d.pre_feedforward_layernorm.weight", …
    npu/forward/src/q4nx_forward.cpp:693   /* Plain int8 with a per-32-column scale (Gemma4). */
    npu/forward/src/q4nx_forward.cpp:700   rc = q4nx_dequant_plain_i8((const uint8_t*)e, sc, NV, H, embed_f_.data());
    

    Three facts that together make the conversion unnecessary:

    1. our runtime already consumes a model.q4nx container (Model::Model maps it; every Tensor::data points into that mapping — engine#1784/npu: read the token embedding table in place, one row per token, instead of materialising it (engine#1784) #1795);
    2. it addresses the tensors by the model's own HF names — model.layers.N.pre_feedforward_layernorm.weight, model.layers.0.mlp.gate_exps_proj.weight, … exactly the spellings the NPU2 containers use;
    3. it already decodes plain int8 with a per-32-column scale (q4nx_dequant_plain_i8, "Gemma4") — which is the dtype mix the sweep found in 127 of the first pack's 237 tensors.

    So the pack is already in our runtime's shape, and what our npu-pack importer converts is AMD's ONNX packs — a different genre that genuinely needs a graph walk. For an NPU2 pack the missing step may be nothing at all: 1bit serve -m <pack dir> --device npu, with the geometry and rope semantics read from the pack's own config.json (which #1794 is currently teaching to read the nested rope_parameters — that work is on this path).

    What that means for the row: the first measurement is not "write a mapping" but "does it load" — and I tried it tonight:

    $ npu-hold -t 6 -l "f1 gemma3-270m" -- f1-serve-probe.sh
    gpu-hold[2697531] queued 'f1 gemma3-270m' as 01794289588315224508.f1_gemma3-270m
    Terminated  sleep "$POLL"
    

    The probe queued behind other lanes' NPU tickets and my wait expired — at that moment three lanes were holding the device (an F4 27B RSS probe, an F7 d3 pack run, and one more). Nothing failed; the ticket simply never came up. The probe is written and lives at ~/scratch-packs/f1-serve-probe.sh (server killed by a trap, port 18110, health wait, then a greedy /completion), so the next free window answers it in one load of a 0.54 GB pack.

    If it loads, the 59-row sweep becomes a serving sweep rather than a conversion project — and the only per-row work is the semantics in each pack's config.json. If it does not, the failure's first line tells us which genre assumption was wrong, which is exactly the thing worth knowing before writing a mapping nobody needed.

  5. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Measured: the pack supplies the model, and what it lacks is our npu/ kernel set — exactly the split this item assumes. The row is now blocked on a kernel set, not on a container reader.

    The probe got its ticket (202 s queued behind other lanes, then 2 s) and 1bit serve answered immediately with a named reason:

    $ npu-hold -t 6 -l "f1 gemma3-270m" -- f1-serve-probe.sh
    gpu-hold took NPU-TOKEN after 202s (die 69C)
    1bit serve: …/gemma3-270m-npu2 is not an NPU model directory (the fast lane needs npu/)
    

    That message is precise, and the check behind it (app/unified.cpp:347) says exactly what the two halves are:

    bool has_lane_kernels(dir)   // npu/layer_ctx1.elf, layer_ctx2.elf, layer_ctx17.elf, npu/lmhead.elf, npu/layer.pdi
    bool has_model_files(dir)    // model.q4nx, config.json, tokenizer.json

    The model half is already satisfied by the FLM pack — it ships model.q4nx, config.json and tokenizer.json, all three, and our forward reads them by HF names with plain-int8 support. The missing half is ours: a kernel set for this pack's geometry, in npu/.

    So the earlier reframing holds and is now proven end to end: no conversion, no compute mapping — weights from the vendor's pack, kernels from us. That is precisely "FLM is a model source; our runtime and kernels do the work".

    The geometry, so the kernel set is a specification rather than a request

    config.json on the pack:

    architecture Gemma3ForCausalLM
    hidden / heads / kv / head_dim 640 / 4 / 1 / 256
    layers 18
    intermediate 2048
    vocab 262144
    sliding_window 512 (and a sliding_window_pattern)

    which implies the projections a family set must carry:

    QKV   K=640   N=1024+256+256 = 1536      O   K=1024  N=640
    GU    K=640   N=2*2048 = 4096            D   K=2048  N=640
    

    (plus the M=16 twins, per the naming in ~/models/npu/d3-lite-elfs: full_i8_QKV_K640_N1536.elf, …_m16.elf, and so on.)

    What this hands to pi-zha

    One kernel set for a hidden-640 / 18-layer gemma3 family, and then this row is measurable: the pack goes in place with npu/ beside it, and the run is npu-hold -t 9 -- 1bit serve -m <pack> --device npu → known-answer prompt → tok/s.

    Two things the sweep suggests for that set: sliding_window 512 with a pattern is in the pack's config (so the sliding/full split matters as it does on flint), and the int8 container carries a per-32-column scale, which our q4nx_dequant_plain_i8 already reads — so the kernel's input format is the same one the Gemma4 rows use.

    Also for the record, the probe's own behaviour was clean: ticket acquired and released in the same second as the exit, gpu-hold released NPU-TOKEN in the log, no server left behind.

  6. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    One more measured fact for the hand-off, and it changes which route the rows should be measured on.

    serve --device npu's refusal named npu/, so I looked at what fills that directory and at what a working pack looks like on this box:

    $ ls ~/models/npu/Qwen3-0.6B ~/models/npu/ZR1-1.5B-npu-bf16
    config.json  model.q4nx  plan.json  tokenizer.json  tokenizer_config.json      # <- no npu/
    $ find ~/models ~/scratch-packs -name "layer_ctx1.elf"            # -> nothing
    $ find ~/models -maxdepth 4 -name "layer_ctx*.elf"                # -> nothing
    

    and the files has_lane_kernels wants are read by the lane runtime, not produced by anything here:

    npu/lane.cpp:69    ctx1 = read_file(dir + "/layer_ctx1.elf");
    npu/lane.cpp:70    pdi  = read_file(dir + "/layer.pdi");
    npu/lane.cpp:71    map  = ContextMap::derive(ctx1, read_file(dir + "/layer_ctx2.elf"), read_file(dir + "/layer_ctx17.elf"));
    app/serve.cpp:3460 if (!lane.empty() && fs::exists(fs::path(lane) / "layer_ctx1.elf")) { … }
    

    So there are two routes, and the done-criterion's one is currently unprovisioned on this box:

    route what it needs state here
    1bit serve -m <pack> --device npu (the criterion) an npu/ lane kernel set: layer_ctx1/2/17.elf, lmhead.elf, layer.pdi none exists on this box, and nothing in the tree emits those names
    1bit forward-serve -m <pack>/model.q4nx --kernels <dir> a full_i8_* family set: e.g. ~/models/npu/d3-lite-elfs proven — the F7 lane is running d3 packs this way right now

    That matters for how this item is worked, and it is a decision worth making explicitly rather than discovering per row:

    1. Measure rows on forward-serve --kernels first (it is the route that exists, and it separates "the pack is good" from "the lane kernel set exists"), then treat serve --device npu as the packaging step once a lane set is built for a family;
    2. or build the npu/ lane set for one family and settle the criterion on that one before sweeping.

    Either way the per-row work is the same kernel geometry; the difference is whether the first measurement is blocked on a build that does not exist yet.

    What I am not doing, deliberately: running a mismatched full_i8_* set against the gemma3 pack to read the refusal. A geometry mismatch is exactly the case that can end in a TDR rather than a refusal, and the rule on this box is that the first TDR stops the lane — so the missing-projections list is better obtained from the geometry arithmetic (already given: QKV K640 N1536, O K1024 N640, GU K640 N4096, D K2048 N640) than by provoking the device.

  7. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Which rows can be measured TODAY: per-projection coverage against the ELF family sets on this box. Four of 57, and my first pass wrongly said nine.

    Method: read every pack's config.json (57 of 59 parsed; 2 have no top-level or text_config geometry), compute its four projection shapes, and check them against the full_i8_<role>_K<n>_N<m>.elf names in each set under ~/models/npu/:

    QKV  K=H        N=q_out + 2*kv_out        O   K=q_out   N=H
    GU   K=H        N=2*IM                    D   K=IM      N=H
    

    A pack counts as covered only if all four projections are present, because a family set is a geometry, not a hidden size.

    packs fully covered by an existing ELF family set: 4
      Gemma4-E2B-IT-NPU2          <- qwen3-coder-kernels (H=1536, IM=6144)
      Gemma4-E4B-IT-NPU2          <- qwen3-coder-kernels (H=2560, IM=10240)
      Nemotron-3.5-Bolt-E4B-NPU2  <- qwen3-coder-kernels (H=2560, IM=10240)
      Qwen3-0.6B-NPU2             <- qwen3-coder-kernels (H=1024, IM=3072)
    

    And the correction, because my first pass was wrong in the optimistic direction: matching on (hidden, intermediate) gave nine candidates — 4 at hidden 4096/IM 12288 for d3-flash-elfs, 5 at hidden 1024/IM 3584 for d3-lite-elfs — and all nine fail the per-projection check, because QKV's N depends on heads/kv/head_dim, not on hidden and IM. d3-flash-elfs has QKV_K4096_N10240 and N8192; Qwen3-8B needs N6144. A hidden+IM match is a hypothesis; the ELF's N per role is the test.

    So the sweep's shape is now: 4 rows measurable today with an existing set, and 53 that need an ELF family set for their geometry — of which the biggest clusters are hidden 2560 with IM 10240 / 9728 / 9216 / 10752 / 10240 (10 packs), hidden 2048 with IM 8192 / 6144 / 11264 / 10752 / 7680 / 6912 / 512 (13 packs), and hidden 3072 (2 packs).

    Next step, and it is the row's actual measurement: Qwen3-0.6B-NPU2 is the smallest of the four (0.70 GB), and qwen3-coder-kernels covers it — so that row can go all the way to a tok/s figure with an existing kernel set. I will do that one and report before asking anyone for a new family.

  8. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    First row answered: an FLM NPU2 model source serves on OUR runtime and OUR kernels, and it answers correctly

    Qwen3-0.6B-NPU2 — the smallest of the four packs that an existing ELF family set covers — runs end to end. No conversion, no import step: the vendor's model.q4nx + config.json + tokenizer, and our qwen3-coder-kernels set.

    $ npu-hold -t 8 -l "f1 qwen3-0.6b" -- f1-fserve-probe.sh
    gpu-hold took NPU-TOKEN after 0s (die 72C)
    1bit forward-serve: qwen3-0.6b-npu2 ready on http://127.0.0.1:18111 (full ELF)
    == chat 1: HTTP 200 in 16294 ms     == chat 2: HTTP 200 → identical text
    log: I8Ctx::init_elf …/qwen3-coder-kernels/full_i8_QKV_K1024_N4096.elf M=128 K=1024 N=4096 nl=1
         full ELF ok: uuid=a5d9f45de811f5e6a992edd6e4c82f3d
         [forward] decode family: 0 role(s) at M=16, 4 on M=128
         lm head: 25 packed weight tile(s) resident (H=1024 T=6144)
    

    Criterion 3 — the known-answer prompt on the model's own template:

    $ curl … /v1/chat/completions  {"messages":[{"role":"user","content":"/no_think In one word, the capital of France is"}],"max_tokens":8}
    {"message":{"content":"\n\nParis","reasoning_content":"\n\n"},"finish_reason":"stop",
     "usage":{"completion_tokens":5}}
    

    Paris, finish_reason: stop, and three streamed requests returned identical content. (It is a thinking model: without /no_think the answer lands in reasoning_content and gets cut off by max_tokens — the template matters, which is why criterion 3 says "the model's own template".)

    Criterion 4 — TTFT and decode, measured:

    TTFT, warm (requests 2 and 3) 4.01 s / 4.09 s
    TTFT, first request after load 12.2 s (the head tiles pack then)
    decode ≈ 1.77 tok/s — 100 content events in 56.35 s

    The decode figure is head-dominated and that is worth saying rather than leaving as a number. This model is 0.6B with hidden 1024 / vocab 151936, so lm_head is 156M parameters — 26 % of the model — and it is read once per token. That is the shape of a slow rate on a small model, and the next measurement that would separate it is a longer run with the head cost isolated (or a model in the same family with a smaller vocabulary).

    Criterion 5 — the set and the engine: qwen3-coder-kernels (81 ELFs, full_i8_* family, M=128 decode shape), deployed build 1bit 0.0.1 from ~/wt/payload-c25/…/1bit/1bit (2026-11-04 23:31).

    What this settles, and the one criterion it does not

    It settles the item's central question: the FLM NPU2 packs are model sources that need no conversion. The pack is the container our runtime reads, with the tensor names our forward addresses; the only missing half is the kernel geometry, and qwen3-coder-kernels happens to cover this one.

    The criterion it does not settle is the route. The done definition says 1bit serve --device npu, whose npu/ gate wants the lane kernel set (layer_ctx1/2/17.elf, lmhead.elf, layer.pdi) — which does not exist on this box for any family. This row ran through forward-serve --kernels, the route the F7 lane uses for d3 packs. So one of two things needs deciding, and it is an owner-level call rather than mine:

    1. amend the criterion to "serves on --device npu (either route)", which is what the engine can actually do today and what this row demonstrates; or
    2. treat serve --device npu's npu/ layout as the packaging target and build one family's lane set, which makes this row's measurement a forward-serve prerequisite rather than its completion.

    And it makes the sweep actionable: 3 more rows (Gemma4-E2B-IT, Gemma4-E4B-IT, Nemotron-3.5-Bolt-E4B) are covered by the same existing set and can be measured the same way; the other 53 need a family set, clustered at hidden 2560 (10 packs), hidden 2048 (13), hidden 3072 (2).

    Probe scripts are committed in the record: ~/scratch-packs/f1-fserve-probe.sh (serve + answer), f1-fserve-measure.sh (TTFT + 3 identical), f1-decode-rate.sh (decode rate). All three take npu-hold, kill the server in a trap, and released the token cleanly on every run tonight.

  9. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Second row: got to the last missing kernel and names it exactly — and it corrects my own coverage predicate.

    Gemma4-E2B-IT-NPU2 (5.99 GB, a multi-component pack: model.q4nx 4.67 GB text tower + vision_weight.q4nx 337 MB + audio_weight.q4nx 952 MB) loaded and initialised kernels one after another, then refused:

    [forward] D   M=128 K=6144  N=1536   …/full_i8_D_K6144_N1536.elf          ok
    [forward] QKV2 M=128 K=1536 N=5120   …/full_i8_QKV_K1536_N5120.elf        ok
    [forward] O2   M=128 K=4096 N=1536   …/full_i8_O_K4096_N1536.elf           ok
    1bit forward-serve: forward init failed: missing full ELF …/full_i8_PLEGATE_K1536_N512.elf
    

    That is the best possible failure: it names the one file to build, by its geometry, after initialising everything it did have. The row's blocker is now a single ELF, not an investigation.

    And it corrects the coverage predicate I published an hour ago. I checked four roles (QKV, O, GU, D) and concluded qwen3-coder-kernels covers this pack; the model also needs:

    role the set has this pack needs
    PLEGATE K1536_N256 K1536_N512
    QKV2 / O2 (a second attention variant) QKV_K1536_N5120 ✓ / O_K4096_N1536 ✓ present
    the per-layer gate — PLEGATE above

    So the predicate needs at least six roles, and my "4 of 57 fully covered" is an upper bound, not a count: Qwen3-0.6B genuinely ran (it has no per-layer gate), but Gemma4-E2B-IT and by extension the other gemma4 rows are one ELF short. The corrected statement is 1 row measured, 3 rows one-or-more ELFs short, 53 rows needing a family set — and the way to know for any of them is to run it, because the loader's refusal names the gap exactly.

    What this hands to pi-zha, concretely: full_i8_PLEGATE_K1536_N512.elf (+ its _m16 twin, per the naming convention) for the gemma4-e2b family — and, for the sweep, the corrected role list so a coverage check does not repeat my mistake: QKV, O, GU, D, PLEGATE, and the 2-suffixed variants where a pack carries two attention shapes.

    Also worth recording about the pack itself: it is genuinely multi-component (vision and audio towers beside the text tower), and 709 tensors with structures like inp_gate.weight [8,1536,32] and inp_gate.weight_prefill [4,6,16384] — i.e. the per-layer embedding table that F4 describes. The text tower loads from model.q4nx alone, so F1's scope (text serving) is not blocked by the other two towers; F6 owns the vision side.

  10. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The coverage question is now answered causally, and it completes the correction: the other three packs on my "fully covered" list all carry a per-layer gate, which is exactly the ELF gemma4-e2b refused for.

    Measured by range-fetching each container's header (no weights) and looking for inp_gate / plegate / per_layer tensors:

    packs carrying a PER-LAYER GATE:  3
      Gemma4-E2B-IT-NPU2            179 such tensors of 709
      Gemma4-E4B-IT-NPU2            214 of 849
      Nemotron-3.5-Bolt-E4B-NPU2    214 of 849
    
    packs with NO per-layer gate:    55
    

    Those three are exactly the other three entries on my earlier "4 of 57 fully covered" list. So the sentence I published an hour ago — that four packs were servable with an existing ELF set — was wrong in a way that is now fully explained:

    pack per-layer gate state
    Qwen3-0.6B-NPU2 no served and measured (TTFT 4.01 s, decode ≈1.77 tok/s, answered "Paris")
    Gemma4-E2B-IT-NPU2 179 tensors refuses by name: missing full_i8_PLEGATE_K1536_N512.elf
    Gemma4-E4B-IT-NPU2 214 tensors same class — will refuse for its own PLEGATE geometry
    Nemotron-3.5-Bolt-E4B-NPU2 214 tensors same class

    So the honest count is 1 row measured, 3 rows blocked on a PLEGATE ELF, 55 rows needing a family set — and the discriminator is not hidden size or intermediate size but whether the container carries per-layer gate tensors at all. That is a property of the container, readable from the header in two HTTP range requests, which makes it a cheap pre-check for any future coverage claim: my four-role predicate missed a fifth role that only 3 of 58 packs need.

    Hand-off to pi-zha, now complete for this family:

    • full_i8_PLEGATE_K1536_N512.elf (+ _m16) — unlocks Gemma4-E2B-IT-NPU2 (and the sweep's 1536-wide row);
    • the same role at the 2560-wide geometry for Gemma4-E4B-IT and Nemotron-3.5-Bolt-E4B (their PLEGATE N follows their per-layer table width — qwen3-coder-kernels already has a PLEGATE_K1536_N256, so the role name and machinery exist; only the geometry is missing).

    And one methodological note for the family's docs, since this is the second time tonight a coverage claim of mine was too generous: a pack is covered when every role its container names has an ELF, and the way to know is to run it or to read its header — not to compute four projections from config.json and assume that is the whole list. The loader's refusal is the authority, and it names the gap exactly.

  11. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The sweep is not one format with 59 geometries — it is 22 architectures, and that changes what "support all of it" costs.

    Measured by range-fetching each container's header (no weights) and normalising tensor names (.N. → .*.), then grouping by the resulting name set:

    containers read: 58      DISTINCT architecture signatures: 22
    group sizes: [9, 7, 7, 6, 5, 4, 2, 2, 2, 2, 1, 1, …]
    largest group: 9 of 58
    

    No single signature covers more than 9 packs, so this is not a handful of families with long tails — it is 22 shapes of roughly comparable size. And the markers inside those signatures say which of them are architecture differences rather than naming ones:

    marker packs what it means for us
    linear_attn / ssm_ 15 hybrid linear-attention / SSM layers (Mamba-style), beside attention
    conv. (conv.conv.weight, conv.in_proj.weight) 8 convolutional layers (LFM3 and relatives)
    model.layer. (singular) 3 a naming variant — our forward addresses model.layers., so these miss by spelling alone
    inp_gate (per-layer gate) 3 the PLEGATE ELF class found earlier
    model.visual. 1 vision tensors in the same container (MiniCPM-V)
    dflash. 1 a draft package, not a servable model (its "tensors" are config keys)

    Three consequences, and the first is the one that changes the plan:

    1. "53 rows need a family set" was too optimistic in the same way my earlier count was. Kitting an ELF family only helps a row whose architecture the forward already implements. At least 23 of the 58 (15 SSM-hybrid + 8 conv) involve layer types that are a forward question, not a kernel-set question — and 3 more would load under the wrong name (model.layer. vs model.layers.). The honest split is: kernels for some, forward support for others, a name normalisation for 3, and one is not a model at all.
    2. The two cheapest wins are not kernel work. The model.layer. variant is a normalisation in the tensor lookup, and dflash. should be excluded from the sweep rather than attempted (it is a draft package, F2's item). Both are single-line judgements that would otherwise be discovered one probe at a time.
    3. Per-architecture is the right unit of work, not per-row: the 9-pack largest group is the best first target after the row already measured, because one architecture's support moves nine rows at once.

    What I would hand to the owner with this: the sweep's shape is now measurable and the remaining work is scoped by architecture, not by pack count. If the intent was "make the 59 packs run", the measurement says that means 22 forward/kernel configurations, of which one (Qwen3-0.6B's signature) is proven end to end tonight, three are one ELF away, and eleven of the 22 groups contain a layer type our forward may not have.

    Next measurement from here, and it needs no device: for each of the 22 signatures, list the distinct layer-type markers so the ones needing forward work are separated from the ones needing only kernels. That is the same range-fetch, and it produces the actual work breakdown.

  12. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claiming this (pi-zha, sweep lead) — starting with the current-generation, weights-only packs

    Claiming on the board per the standing loop. Why this lane: everything it asks for is the loop I have been running all night on the AMD packs — import with
    zero gaps → serve --device npu under npu-hold → a known-answer prompt in the pack's own chat template, 3 identical repeats → a measured tok/s and TTFT,
    then a row that names the kernel set and the engine commit.

    Starting point, from the issue's own inventory: the current generation (no .xclbin, 2 × q4nx, 7-9 files) is the in-scope shape, and the smallest is
    Nemotron-3.5-Bolt-E4B-DFlash at 0.27 GB. That is a cheap first probe of the genre — if a weights-only FLM pack imports and serves, the whole family is a
    sweep of downloads rather than a new reader; if it does not, the refusal will say which genre assumption breaks, and it costs 0.27 GB to learn.

    Order I intend: smallest current-generation pack first (0.27 GB), then the next two by size, then whatever the first three teach decides whether the 22-24 GB
    packs are worth a lane's time before the disk floor is comfortable again.

    What I will not do: touch the older .xclbin packs. Our artifacts are full ELFs and the plan forbids xclbin as a target or a fallback, so those are out of
    scope by rule rather than by difficulty.

    Will report each pack as a row with its numbers, and will say plainly if the genre needs reader work that belongs to another lane rather than pretending a sweep
    can carry it.

  13. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    First probe (1.0 GB, LFM2.5-1.2B-Thinking-NPU2): the pack is safetensors with a JSON table, our serve reads its geometry correctly and then cannot read its weights — so #1786 is a format route, not a sweep

    Downloaded the smallest model pack of the current generation (skipping the 0.27 GB one, which is a DFlash draft and belongs to #1780):
    FastFlowLM/LFM2.5-1.2B-Thinking-NPU2, 959 MB, 7 files, one model.q4nx, no .xclbin — exactly the in-scope shape the inventory describes.

    What works: 1bit serve -m <dir> --device npu opens it as a model-generic container, reads the geometry from config.json
    (H=2048 NH=32 NKV=8 HD=64 IM=8192, rope theta 1e6), derives all four designs, loads the ELFs from the npu/ dir I supplied, and comes up:
    decode family: 4 role(s) at M=16, ready on … (full ELF).

    What does not: every request fails with

    forward step failed: MLP dequant : data_size 0 at geometry 2048x2048 = 512 tiles
    (chunk 0 B is not a kind, and no padded grid fits): the container and config.json disagree
    

    And the reason is the container format, which is not ours. model.q4nx is a safetensors-style file — <u64 header_len><JSON table><payload> — with

    header_len = 16,208 B; 149 tensors
      model.layers.0.input_layernorm.weight: {"dtype":"BF16","shape":[2048],"data_offsets":[0,4096]}
      model.layers.0.mlp.down_proj.weight:   {"dtype":"I8","shape":[2048,5120],"data_offsets":[…]}   bytes/numel = 1.0000
      dtype census: BF16 56 · I8 93
    

    Three things a reader has to deal with, all measured:

    1. no scale/zero tensors exist anywhere in the table (grep 'scale|zero' over the 149 names → 0) while every I8 weight is exactly 1.0000 byte/element — so the quantisation envelope is not stated in the file the way ours is;
    2. 30 shortconv.conv.weight tensors — LFM2's conv blocks are part of the model, and the forward has no shortconv operator;
    3. the table names are HuggingFace-style (model.layers.N.mlp.down_proj.weight), not our container's sections.

    So the honest reading of this item's first data point: the current-generation FLM sources are a distinct container genre (safetensors table + a packed int8 whose scale convention is not in the table) plus a new operator (shortconv), and neither is a sweep. Our side already does the parts that are generic — geometry, design derivation, ELF loading, serve — which is why the failure is a clean, named one at the first weight read.

    Next, and it is a decision for the owner/plan rather than for a sweep: reading this table is much easier than the AMD .fconst work (a JSON dict with explicit offsets — own code, no heuristics), but the I8 envelope needs to be established from the payload (the 2.5 B/weight ratio I first computed was wrong: at 1 B/element the extra is elsewhere, and the shape's 5120 already shows the packs pad K). I will write that up as a 15-line design on this issue before touching code, per the plan's rule for anything over a day, and in the meantime report the remaining 33 current-gen packs as format-blocked, not sweep-blocked so nobody queues them as a re-run task.

  14. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Convergence measured against the kernel lane's two device-proven families — and it corrects something I said on that PR an hour ago.

    npu-kernels #53 proved two b8 families bit-exact on the device (maxabs = 0): 1536x2048 ("Qwen 1.5B class, 13 packs") and 2304x4096 ("gemma-2-2b class, 4 packs"). I wrote on it that "those two classes' geometries match groups in my sweep's uncovered 55, so naming the mapping would move rows." I have now done the mapping against the 57 packs whose config.json I can read, and it moves almost nothing:

    family packs in the FLM NPU2 sweep
    1536x2048 (hidden 1536, q_out 2048) 1 — Gemma4-E2B-IT-NPU2
    2304x4096 (hidden 2304, q_out 4096) 0

    So the kernel lane's proven geometries serve the AMD 1.9 packs (its "13 packs" and "4 packs" are rai packs, not these), and the one overlap is the row I already probed — which still needs its PLEGATE ELF, the per_layer_inputs carry (#1825) and the .fconst block layout (W1c). I over-called that in the review and am correcting it here; the correct statement is that the W3b work unblocks the AMD side, and F1's side needs its own geometries.

    What the sweep's hidden-size distribution says the families should be, instead:

    hidden 2048: 17 packs     hidden 2560: 15 packs     hidden 4096: 6
    hidden 1024:  6           hidden 5120:  4           hidden 3072: 2
    hidden 2880:  2           then 1152, 640, 768, 1536, 3840: 1 each
    

    Two families cover 32 of 58 packs: hidden 2048 (17) and 2560 (15). That is where a b8 generator would pay, and neither is the 2304 geometry that was just proven. The next four (4096, 1024, 5120, 3072) cover another 18; the remaining 8 are one-offs.

    Two things worth deciding with this number, and they are the kernel lane's call, not mine:

    1. whether F1's rows should wait for their own geometries or be served by a generic path — a family per hidden size is 12 more families for the one-off tail, which is a lot of engineering for 8 rows;
    2. whether the 2048 and 2560 families are worth proving first, since 32 of 58 packs share them and both already have the QKV/O/GU/D role machinery in the full_i8_* naming convention.

    Either way the sweep's remaining work is now sized: 2 families → 32 rows, 4 more → 18 rows, 12 one-offs → 8 rows, plus the 23 hybrid/convolutional packs that need forward work before any family helps them.

  15. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Design note (15 lines, posted before any code): the FLM container is already readable — the open question is what its int8 weights mean

    The first probe's "distinct container genre" reading was too pessimistic, and the existing tooling proves it. model.q4nx is plain safetensors, and
    tools/safetensors_to_q4nx.py pack reads the whole file — 148 of 149 tensors pack cleanly, including the 30 shortconv.conv.weight tensors (as BF16) and an
    embedding tiled to I8 [16384, 5120] q4_1 — and then stops on exactly one thing:

    /tmp/lfm-src/model.safetensors: lm_head.weight has dtype I8, which this converter does not read (F32, F16 and BF16 only)
    

    So the plan is not "write a container reader". It is:

    1. Decide what an I8 tensor means in these packs. In the table the 93 I8 tensors are exactly 1.0000 byte per element (down_proj [2048, 5120] =
      10,485,760 B) and there is no scale or zero tensor anywhere in the 149 names, nor any quantisation field in config.json (40 keys, checked for
      quant|scale|bits|int8|group|zero → nothing). Three candidates, in the order I would test them: (a) the payload interleaves per-group scales in bytes the table
      accounts for under the padded second dimension (5120 against a 2048-wide row is 2.5 B per 4 weights, exactly the room a 4-bit-group scale table needs);
      (b) a per-tensor scale carried in the file outside the tensor table; (c) the codes are unscaled and the runtime is expected to normalise per row.
    2. Settle it by measurement, not by guessing: dequantise one I8 matrix under each candidate and check the result against the model's own BF16
      model.token_embd.weight-style neighbours — the packs ship BF16 for every norm, and a wrong envelope shows up as a row-norm distribution off by orders of
      magnitude, which is the same check that caught the tail-head envelope story.
    3. Then either teach the converter I8 (2-D → q4_1 re-quantisation is the natural step, since our kernels already take _b8 tiles), or add an int8 container
      path
      — the second only if step 1 shows the codes already carry their scales and re-quantising would lose precision.
    4. shortconv is a separate, smaller item: LFM2's 30 conv blocks are a host-side depthwise conv over conv_L_cache = 3 positions; it is an operator, not a
      format, and the converter already carries the weight through as BF16.
    5. Verification for any of it: one pack (this one, 959 MB) imports, serves, answers a known-answer prompt 3× identically, and reports tok/s — the same five points as
      every other row, and Hy-MT2-1.8B-NPU2 (1.5 GB) is the second pack to confirm it generalises.

    Nothing here needs the owner's decision to start: step 1 is measurement on a pack I already have. What it does need is the same thing the AMD .fconst work
    needed — a reference to check a decode against — and these packs ship BF16 norms plus an upstream GGUF, so both are available.

    If another lane is already on the FLM container (this is #1786's model-sources half and #1780's draft half), say so here and I will hand the step-1 measurement to
    them rather than run it twice.

  16. 62 remaining items

  17. bong-water-water-bong commented on Nov 12, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The third key for gemma4 is in its config, and it is an exact layer boundary: use_double_wide_mlp with num_kv_shared_layers.

    #2270 (now on main) gives the writer a (layer-elided name, layer type) key, which resolves MiniCPM-V, bolt and gemma3 to 0 spans and leaves gemma4 with six: gate_proj, up_proj and down_proj each in two shapes under both layer types. The remaining question was what the third key is. It is not a mystery:

    config.json (text_config):  use_double_wide_mlp: true      num_kv_shared_layers: 20      num_hidden_layers: 35
    
    gate_proj width 1152  -> layers [0 … 14]         (15 layers)
    gate_proj width 2304  -> layers [15 … 34]        (20 layers)   <- exactly the double-wide half
    

    A contiguous boundary, the last num_kv_shared_layers (20) of 35 layers, and the two widths are 1:2 — so the MLP is doubled on that upper block. It is not correlated with layer_types at all (both attention kinds occur on both sides), which is precisely why a (role, layer type) key produced spans instead of rows.

    So the writer's key is (layer-elided name, layer type) plus "is this layer in the double-wide block", and the third component is declared, not inferred. No guessing and no per-layer table: use_double_wide_mlp says it happens, num_kv_shared_layers says how many layers at the top, and the measurement above confirms both (20 of 35, contiguous, exactly the wider half).

    State of the key across the family, measured: MiniCPM-V 35B / 1B, bolt-e4b and gemma3-270m are two-key (0 spans); gemma4-e2b is three-key with the third component now identified. That is the whole role→destination mapping problem for the container genre, and none of it required a device.

    What is still open for 3c (unchanged by this): the placement code itself — turn each (stem, key) into the destination field's slot in the layout — and the gemma_local/container_extra tensors that this engine's forward does not read at all, which a full-family writer must consciously skip rather than copy.

  18. bong-water-water-bong commented on Nov 12, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    CLOSE CONDITION

    The bar. This item is done when every in-scope FastFlowLM/*-NPU2 repo has a row meeting the body's five items. Each row is measured on the hub's own files at engine main with 1bit serve -m <pack dir> --device npu, inside npu-hold:

    1. the per-op forward loads every tensor of the published container (item 1 is read this way: these packs are served as published, with no npu-pack import step);
    2. it serves;
    3. "What is the capital of France?" through the pack's own template, 3 identical replies, answer Paris. An encoder instead answers /v1/embeddings within cos 0.999 of transformers' own model on the same ids;
    4. decode tok/s and ttft;
    5. the kernel set and engine commit are named.

    The final table is posted on this issue.

    In scope: 46 of the 59 NPU2 repos. Out per the body, or owned by another issue:

    smolvla-NPU2 is a robot-policy tarball with no model.q4nx, config or tokenizer, so there is nothing for the chat route to serve. I propose it is out (owner's call).

    Where it stands (board, 11-12 23:55Z): 30 rows DONE on the hub files. That includes the last day's fixes, all merged: harmony replies (#2298), the Q8 code order of the 0.9.45 Qwen3.5 packs (#2301), and an unstated tie_word_embeddings (#2309).

    Left: PRs still to write or land (no others):

    1. npu: Embedding-Gemma-300M-NPU2 serves /v1/embeddings on the per-op forward (refs #1786) #2314: Embedding-Gemma-300M-NPU2 on /v1/embeddings. Open, ready at 3985fd9; host cos 0.99999, NPU 0.9991 against transformers.
    2. Qwen3.6-35B-A3B-NPU2: the pack's small BF16 matrices stored [in, out] (ssm_alpha/beta) and its shared_expert_gate name. The host forward reasons with both fixes and was noise without them; the device row is running. Stacked on npu: Embedding-Gemma-300M-NPU2 serves /v1/embeddings on the per-op forward (refs #1786) #2314 (it extends that PR's BF16 reader).
    3. Qwen3.8-27B-NPU2: the device row. This becomes a PR only if the row needs a code change; it has not been measured on current main.

    Left, device rows only (no PR): Nanbeige4.1-3B-NPU2 (host-checked, device row owed), medgemma-4b-it-NPU2 (ttft), and a pass over the 11 rows measured on 11-11 (LFM2/LFM2.5 ×3, Llama-3.2 ×2, Hy-MT2, Qwen3-1.7B, Qwen3-4B ×3, Phi4-mini) to fill whichever of items 4–5 their rows lack.

    Reachable this week: yes, unless the 27B needs more than one PR. Perf follow-ups stay listed under this issue, not as new slices: device MXFP4 experts for GPT-OSS, and the per-token prefill of gemma4, gpt_oss and the MoE packs.

  19. bong-water-water-bong commented on Nov 13, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The full table: every FastFlowLM NPU2 source

    The bar is the one in the CLOSE CONDITION above. Each row is measured on the hub's own files with 1bit serve -m <pack dir> --device npu, inside npu-hold, with amdxdna clean before and after. A row passes when the pack's own template returns the known answer, 3 identical replies, with decode tok/s and ttft measured. A pack that always reasons is checked on one request long enough to reach the answer, with the reasoning split out. The kernel sets were built locally for each geometry; they are named on the board and are not shipped.

    59 repos:

    Rows that first ran on ~/.config/flm's rewritten copies or on the fused route were re-run on the hub files and the per-op route tonight. Those are 6, 9, 11, 12, 14, 17, 19, 26, 28, 29, 30, 31, 33, 36 and 38.

    # pack GB state known answer decode ttft PRs
    1 Nemotron-3.5-Bolt-E4B-DFlash-NPU2 0.27 out of scope: DFlash drafter, tracked in #1780 (cl-npuperf)
    2 Gemma3-270M-NPU2 0.54 serves Paris ×3 identical 7.8-8.0 tok/s 0.35 s
    3 D3-Edge-NPU2 0.62 out of scope: Decision 3.0 sizes are tracked separately (#1786 "Not in this item")
    4 Embedding-Gemma-300M-NPU2 0.65 serves /v1/embeddings cos 0.9991 (npu) vs transformers 5 inputs in 0.88 s 2314
    5 Whisper-V3-Turbo-NPU2 0.65 out of scope: Whisper, tracked in #1783 (pi-demo)
    6 Qwen3-0.6B-NPU2 0.70 serves Paris ×3 identical 2.91-2.95 tok/s 0.98 s
    7 smolvla-NPU2 0.71 not a chat source (robot-policy tarball); proposed out, owner decides
    8 D3-Lite-NPU2 0.78 out of scope: Decision 3.0 sizes are tracked separately (#1786 "Not in this item")
    9 LFM2.5-1.2B-Thinking-NPU2 1.00 serves Paris (reasoning split) 1.89 tok/s 3.3 s
    10 LFM2-1.2B-Tool-NPU2 1.01 serves Paris ×3 identical 1.66-1.99 tok/s 1.4-1.7 s
    11 LFM2.5-1.2B-NPU2 1.01 serves Paris ×3 identical 1.74-1.94 tok/s 1.5-1.7 s 2318
    12 LFM2-1.2B-NPU2 1.01 serves Paris ×3 identical 1.95-1.96 tok/s 1.5 s
    13 Gemma3-1B-NPU2 1.26 serves Paris ×3 identical 1.67-1.71 tok/s 1.67 s
    14 Llama-3.2-1B-NPU2 1.31 serves Paris ×3 identical 1.68-1.82 tok/s 2.1-2.2 s
    15 Qwen3.5-0.8B-NPU2 1.32 serves Paris ×3 identical 2.59 tok/s 1.05 s 2301
    16 Qwen3.8-27B-DFlash2-NPU2 1.39 out of scope: DFlash drafter, tracked in #1780 (cl-npuperf)
    17 Hy-MT2-1.8B-NPU2 1.52 serves Paris ×3 identical 1.22-1.30 tok/s 1.6 s 2218
    18 LFM3-1.7B-NPU2 1.62 serves Paris ×3 identical 1.45 tok/s 2.05 s 2254
    19 Qwen3-1.7B-NPU2 1.71 serves Paris ×3 identical 1.27-1.28 tok/s 2.2-2.3 s
    20 LFM2-2.6B-Transcript-NPU2 1.88 serves Paris ×3 identical 0.79-0.84 tok/s 3.2-3.9 s
    21 LFM2-2.6B-NPU2 1.88 serves Paris ×3 identical 0.84-0.96 tok/s 3.1-5.5 s
    22 MiniCPM-OCR-1.0-NPU2 2.01 out of scope: tracked in #1788 MiniCPM-V on the NPU (pi-verify)
    23 D3-Nano-NPU2 2.02 out of scope: Decision 3.0 sizes are tracked separately (#1786 "Not in this item")
    24 MiniCPM-V-4.7-1B-NPU2 2.22 out of scope: tracked in #1788 MiniCPM-V on the NPU (pi-verify)
    25 Qwen2.5-3B-Instruct-NPU2 2.59 serves Paris ×3 identical 0.66-0.72 tok/s 5.4 s 2259
    26 Llama-3.2-3B-NPU2 2.81 serves Paris ×3 identical 0.66-0.70 tok/s 5.4-5.7 s
    27 Nanbeige4.1-3B-NPU2 3.10 serves Paris (reasoning split) 0.60 tok/s 10.8 s
    28 Qwen3.5-2B-NPU2 3.13 serves Paris ×3 identical 1.42-1.47 tok/s 2.0 s 2301
    29 Qwen3-4B-Instruct-2507-NPU2 3.31 serves Paris ×3 identical 0.51-0.57 tok/s 5.2-8.5 s
    30 Qwen3-4B-Thinking-2507-NPU2 3.31 serves Paris ×3 identical 0.50-0.58 tok/s 5.2-5.3 s
    31 Qwen3-4B-NPU2 3.31 serves Paris ×3 identical 0.57 tok/s 5.2 s
    32 D3-Mini-NPU2 3.43 out of scope: Decision 3.0 sizes are tracked separately (#1786 "Not in this item")
    33 Phi4-mini-Instruct-NPU2 3.64 serves Paris ×3 identical 0.62-0.63 tok/s 3.2-3.3 s
    34 Gemma3-4B-Text-NPU2 3.80 serves Paris ×3 identical 0.60 tok/s 5.0 s
    35 Qwen2.5-VL-3B-Instruct-NPU2 4.02 serves Paris ×3 identical 0.67-0.68 tok/s 4.1-4.3 s 2259, 2261
    36 Qwen3-VL-4B-Instruct-NPU2 4.14 serves Paris ×3 identical 0.59 tok/s 5.0 s 2324
    37 Translategemma-4B-Instruct-NPU2 4.78 serves Paris ×3 identical 0.56-0.59 tok/s 4.9 s
    38 medgemma-4b-it-NPU2 4.78 serves Paris ×3 identical 0.68 tok/s 4.4 s 2242, 2248
    39 medgemma-1.5-4b-it-NPU2 4.78 serves Paris ×3 identical 0.58-0.62 tok/s 4.5-4.6 s
    40 Gemma3-4B-NPU2 4.78 serves Paris ×3 identical 0.56-0.61 tok/s 5.3-5.5 s
    41 Qwen3.5-4B-NPU2 4.98 serves Paris ×3 identical 0.61-0.62 tok/s 4.8 s 2301
    42 Llama-3.1-8B-NPU2 5.75 serves Paris ×3 identical 0.28-0.29 tok/s 14.1-14.3 s
    43 Deepseek-R1-Distill-Llama-8B-NPU2 5.76 serves Paris 0.32 tok/s 23.6 s
    44 Qwen3-8B-NPU2 5.99 serves Paris ×3 identical 0.28-0.30 tok/s 9.1-10.6 s
    45 Gemma4-E2B-IT-NPU2 5.99 serves Paris ×3 identical 0.27-0.28 tok/s 79-80 s
    46 D3-Flash-NPU2 6.14 out of scope: Decision 3.0 sizes are tracked separately (#1786 "Not in this item")
    47 Qwen3.5-9B-NPU2 8.67 serves Paris ×3 identical 0.29-0.30 tok/s 9.0-10.4 s 2301, 2309
    48 Gemma4-E4B-IT-NPU2 9.09 serves Paris ×3 identical 0.44-0.47 tok/s 40-44 s
    49 Nemotron-3.5-Bolt-E4B-NPU2 9.09 serves Paris ×3 identical 0.49 tok/s 38-44 s
    50 Gemma4-12B-IT-NPU2 9.35 serves Paris ×3 identical 0.21 tok/s 108-114 s 2265
    51 DeepSeek-R1-0528-Qwen3-8B-NPU2 11.96 serves Paris (reasoning split) 0.33 tok/s 2298
    52 GPT-OSS-Safeguard-20b-NPU2 14.48 serves (experts on host) Paris (reasoning split) 0.21 tok/s 2298
    53 GPT-OSS-20B-NPU2 14.48 serves (experts on host) Paris 0.16 tok/s 2274, 2298
    54 D3-Turbo-NPU2 16.56 out of scope: Decision 3.0 sizes are tracked separately (#1786 "Not in this item")
    55 D3-Ultra-NPU2 16.56 out of scope: Decision 3.0 sizes are tracked separately (#1786 "Not in this item")
    56 Hy-Memory-30B-A3B-NPU2 17.88 serves Paris ×3 identical 0.10-0.20 tok/s 131-242 s 2279
    57 Qwen3.8-27B-NPU2 18.07 serves Paris (reasoning split) 0.125 tok/s
    58 MiniCPM-V-4.7-35B-A3B-NPU2 22.21 out of scope: tracked in #1788 MiniCPM-V on the NPU (pi-verify)
    59 Qwen3.6-35B-A3B-NPU2 24.26 serves Paris 0.27 tok/s 71 s 2301, 2309, 2314, 2318

    What it took (all merged): #2250, #2254, #2259, #2261, #2265, #2274, #2279 (families and formats); #2298 (harmony replies); #2301 (the 0.9.45 Qwen3.5 packs' Q8 code order); #2309 (an unstated tie_word_embeddings); #2314 (the bidirectional encoder on /v1/embeddings); #2318 ([in, out] BF16 matrices and the shared_expert_gate name).

    Perf follow-ups (not correctness; kept as this item's notes, no new slices):

    • device MXFP4 experts for GPT-OSS, which run on the host today;
    • the per-token prefill of the gemma4 E-models (40–80 s ttft), gpt_oss and the MoE packs;
    • decode bound by the M=16 kernels on the 4B–27B rows.

    Closing on this table.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    backlogReal, but does not hold the release build

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions