Skip to content

F9: FLM 1.0.12's HRX-targeted build — side-by-side against ours on the same models, and a row for every family we do not serve #1787

Closed

Description

@bong-water-water-bong

TNPU-ALL §9 item F9 (owner 2026-11-09 23:2x, "support all of it"). Measurement item: nothing to install, nothing left serving.

What

FastFlowLM 1.0.12 ships two builds: its usual one, and an HRX-targeted build of the same product. The second is directly comparable to
us, because HRX is the route our engine already serves on — so the same model can be run by their front-end and by ours on the same device, in
the same hour
, and the two numbers belong on the same board row.

This issue is that measurement, plus the row-generation it exposes.

Definition of done

  1. Side-by-side, one gpu-hold ticket, same models, interleaved (A B B A with their build and ours): decode tok/s and time to first token for
    each, on the models both can run. The numbers are quoted only if the ticket's own spread supports them — the scoreboard's existing drift and
    treatment checks apply unchanged, including the positive control that proves the two arms were different conditions.
  2. Every family their build serves that we do not becomes a row under §1's five conditions (imports with zero gaps and self-check; serves on
    the device; known-answer prompt with the model's own template, three identical answers; measured decode tok/s and TTFT under a ticket; row
    names the kernel set and engine commit). Their build's per-family libraries give the checklist; the row is what settles each family.
  3. The result is written where the scoreboard is recomputed from, not in a comment.

What is already established (host-side, no device)

  • The artifact is present on the box for the side-by-side; it is not installed system-wide and nothing of it is left running.
  • The family list their build carries is enumerated from its own files, and the family list the public release notes add is: MiniCPM-V 4.7 35B-A3B,
    d3:edge, d3:flash, qwen3.5-flash:2b, plus DFlash tags for Nemotron 3.5 Bolt and Qwen3.8-27B. Those are the families to check against our
    own coverage first, since a family we do not serve at all is a row by definition.
  • The measurement cannot start until the box has memory to load a model: MemAvailable was 2 GiB at 23:5x, against the ≥40 GiB the plan
    requires before a pack. It starts on the first ticket with headroom, and the preconditions are re-checked at that moment, not assumed from
    now.

Not in this issue

Installing their build, serving from it, or shipping anything derived from it. It is a reference measurement: run under a ticket, recorded, and
stopped.

Activity

  1. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    First F9 measurement — same model, same device, one probe: ours decodes 5.3× faster, and the TTFT gap is prefix reuse, not a cold-start win

    Ticket gpu-hold -l f9-flm-hrx (212 s, MemoryMax=48G, oom_score_adj=800), artifact raw/f9-flm-hrx-20261110-001736. Both engines measured by
    f9-probe.py: streaming /v1/chat/completions, temperature 0, 64 tokens, 3 reps, TTFT = first chunk, decode = chunks after the first. Neither
    engine's self-report is used.

    FLM 1.0.12 (HRX build) ours (llama-server/HRX) ratio
    decode 2.03 tok/s (2.00/2.03/2.05, spread 1.027×) 10.79 tok/s (10.10/10.79/10.90, spread 1.079×) ours 5.3×
    TTFT (warm) 3.546 s (3.622/3.364/3.546) 0.349 s (0.349/0.229 after rep 1) ours 10×
    TTFT (first request after load) — all reps ≈3.5 s 3.387 s ≈ equal

    The structure is the finding. Their TTFT does not improve across requests; ours falls from 3.387 s to 0.349/0.229 s. So the 10× is prefix
    reuse
    , and on a cold prompt the engines are indistinguishable. Decode has no such structure — 5.3× at both arms' spreads.

    Bounded by the container difference, stated rather than implied: theirs is its own Q4NX pack (qwen3.8-mtp:27b, 17 GB), ours is
    Qwen3.8-27B-UD-Q4_K_XL.gguf (17.6 GB). Same model family and size, not bit-identical weights. Binaries recorded per run (flm bf4b6cd8…,
    our server 4251a167…).

    Also from their side, separately: Model loaded in 32.29 s for the 27B on HRX (measured in the earlier ticket). Not compared to ours: the
    arms loaded in different tickets, and load time is not what this row claims.

    Still open on this issue: the row-generation half — their catalogue lists ~28 tags across d3, deepseek-r1, nemotron, phi4, qwen2.5/3/3.5/3.6/3.8,
    qwen3vl, translategemma and whisper; only two packs are on disk here (27b, whisper), so each family we do not serve needs a pack pull under the
    plan's memory rules before it can become a row.

  2. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The row-generation half: their catalogue is 71 tags, 4 of which we have rows for

    Generated from the model_list.json their build ships (the same file their CLI reads) against our own SUMMARY.md sections — table in
    F9-COVERAGE.md at commit 0e5954d. Summary:

    • 71 tags in the catalogue, 4 have a row of ours: qwen3:0.6b → Qwen3-0.6B, qwen3:1.7b → qwen3-1.7b, qwen3.6-moe:35b-a3b →
      qwen36-35b-a3b, qwen3.8-mtp:27b → qwen38-27b (the one F9 measured).
    • 67 tags have no row. That is the backlog, and it is much larger than the ~28 a truncated flm list suggested: the catalogue also carries
      the -core dev04 variants and the gemma3 / medgemma / lfm2 / lfm3 / nanbeige / hy-mt2 lines.
    • Only a handful are reachable without a download — the last column marks where a GGUF of that model is already on disk. Everything else
      needs a pack pull, under the plan's rules (one pack over 8 GB at a time, ≥40 GiB free before starting, packs on disk not /tmp).

    Families to prioritise, by what the plan already scopes: qwen3:4b (a GGUF is on disk, so a row is a ticket away), qwen3.5:{0.8b,2b,4b,9b}
    (F5's qwen3.5-flash:2b target lives here), d3:{edge,flash} (F7), minicpm-v-4.7-moe:35b-a3b (F6), whisper-v3:turbo (F8, pack already on
    disk), and qwen3.8-dflash2:27b (F2's draft, GGUFs on disk).

    Not claiming anything about their speed on those tags — the only measured comparison is the 27B above. Each row would be measured as that one
    was: one probe for both engines, one ticket, both arms' spreads quoted, container difference stated.

  3. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correction to the coverage table I just posted — it matched the family and ignored the size

    My first table (commit 0e5954d) put Qwen3.5-4B-Q4_K_M.gguf in the "reachable now" column for qwen3.5:0.8b, :2b, :4b and :9b — a 4B file covering four sizes. That is the same error as matching qwen3 as a substring of everything else: the check found the family and never looked at the size.

    Fixed at 85c872d: the column now requires both the family and the size token in the filename. The reachable-without-a-pull set is smaller and correct:

    their tag local file of that family and size
    qwen3:0.6b Qwen3-0.6B-Q4_K_M.gguf
    qwen3:4b Qwen3-4B-Q4_K_M.gguf
    qwen3:8b amd-rai/pard2/pard2-qwen3-8b-Q4_K_M.gguf (a PARD-converted 8B, not a plain Qwen3-8B — a row from it would be a different model and must say so)
    qwen3.5:4b Qwen3.5-4B-Q4_K_M.gguf

    So the honest shape of the row half: 67 tags without a row, and only about four are reachable without a download — one of them (the 8B) not on plain weights. Everything else needs a pack pull under plan §2.

    The count of 71 tags and 4 covered rows is unaffected; the error was confined to the "reachable" column, and it is exactly the kind of thing this table exists to prevent someone else from doing by eye.

  4. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    A model it runs that we do not, measured — and the first number my own probe got wrong by 36×

    lfm2:1.2b is a family we serve nowhere: no row on our board and no local GGUF, so under F9 it is a row candidate by definition. Pulled from their own catalogue (959 MB, FastFlowLM/LFM2-1.2B-NPU2) and run on their HRX build under a gpu-hold ticket.

    The instrument failed first, loudly, and that is the useful part. My streaming probe reported

    lfm2 rep 1: ttft 0.100s  chunks 65  decode 906.51 tok/s
    lfm2 rep 2: ttft 0.082s  chunks 65  decode 1201.39 tok/s
    RESULT lfm2: ttft_median 0.082s decode_median 1201.39 tok/s
    

    1201 tok/s on a 1.2B model over an iGPU is not a fast result, it is an impossible one — 65 chunks in about 0.05 s. Their server bursts: it delivers the whole answer in one flush when the request completes, so the "interval between chunks" is not the decode time. No engine decodes like that, and the only reason it did not become a row is that the number was absurd.

    Non-streaming, using the server's own usage.completion_tokens:

    rep total completion_tokens decode
    1 2.020 s 64 31.69 tok/s
    2 1.902 s 64 33.65 tok/s
    3 1.701 s 64 37.61 tok/s

    ≈33.7 tok/s median (spread 1.19×), TTFT not separable from the non-streaming total — so that cell is decode only, and says so.

    The probe is fixed (de4f695): it now prefers usage.completion_tokens over chunk counting, and marks a run BURST when the chunk interval is a small fraction of the whole request, so the next model that flushes in one go reports a number from the server's own token count instead of a fake rate. The 27B comparison is unaffected — 3.5 s TTFT and ~31 s of streaming for 64 tokens is smooth delivery, which is why its 2.03 tok/s was plausible.

    The row for F9, as it stands: lfm2:1.2b runs on their HRX build at ≈33.7 tok/s decode; our engine has no support for the LFM family (no board row, no local weights), so this is a gap to close rather than a comparison — §1's five conditions cannot even start without a GGUF or a QNX import path for it.

  5. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Second same-model pair: Qwen3-0.6B — same shape as the 27B, at a different size

    Ticket gpu-hold -l f9-pair06 (12 s, MemoryMax=48G), artifact raw/f9-pair-06b-20261110-035705, both engines through f9-probe.py.

    FLM 1.0.12 (HRX build) ours (llama-server/HRX) ratio
    decode 96.33 tok/s (spread 1.026×) 345.02 tok/s (spread 1.038×) ours 3.6×
    TTFT warm 0.509 s 0.011 s (rep 1 = 0.265 s cold) ours 46×
    TTFT first request ≈0.50 s on every rep 0.265 s ours ≈1.9×

    Both pairs now, one table:

    model their decode our decode our advantage their TTFT our TTFT
    Qwen3.8-27B 2.03 tok/s 10.79 tok/s 5.3× 3.546 s 0.349 s
    Qwen3-0.6B 96.33 tok/s 345.02 tok/s 3.6× 0.509 s 0.011 s

    What is now established, and what is not: decode is 3.6–5.3× in our favour on two models and two sizes with both arms' spreads under 4 %, so that part is no longer a single measurement. The TTFT gap is prefix reuse, not cold-start speed: their server shows no improvement across requests while ours falls from 0.265 s to 0.011 s, and on a cold prompt the difference is ~2×. A row that quoted 46× without that sentence would be misleading, which is why the table has both columns.

    Instrument note, because it changed the numbers twice tonight: f9-probe.py now prefers usage.completion_tokens over chunk counting and marks a run BURST when tokens arrive in one late flush. The lfm2:1.2b run forced that fix after reporting 1201 tok/s (impossible); this pair streams smoothly, so chunk timing and usage agree.

  6. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    cl-perf, second pair of hands on this (orchestrator 21:40Z). pi-supreme keeps the issue, the scoreboard files and the row-generation half. My slice, chosen so nothing is measured twice:

    1. The interleave this issue's definition of done asks for. Both existing pairs ran one arm then the other (theirs, then ours). I will run the Qwen3-0.6B pair as A B B A in one gpu-hold ticket (each engine loaded twice), so drift across the ticket shows up as a spread inside each arm and not as a difference between arms.
    2. A third same-model pair at a middle size, qwen3:4b (our Qwen3-4B-Q4_K_M.gguf is on disk; their Qwen3-4B-NPU2 pack needs one pull), also A B B A.

    Same instrument (f9-probe.py at de4f695+, unmodified), same server binaries as the two existing pairs so the rows stay comparable. Artifacts go under ~/lb/aval/clperf-f9-*; I will post the numbers here and hand the artifact paths to pi-supreme for F9-RESULT.md, since that repo is theirs to write. Not taking: the 27B pair (already measured), qwen3.5:4b, and any of the 67 no-row tags.

  7. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    cl-perf: before adding pairs I checked what device their build drives, and it changes how the two existing rows should be read. FLM 1.0.12's "HRX" build runs on the NPU, not the iGPU. "HRX" there names the runtime library that replaced XRT underneath their NPU engine; the device is still the XDNA array.

    Measured, one 14 s npu-hold ticket (~/lb/aval/clperf-f9-device-20261110T214258Z), their server on qwen3:0.6b:

    device nodes held open by flm-real, after load and again during the probe:
          1 /dev/accel/accel0
          1 /dev/null
    (no /dev/kfd, no /dev/dri/renderD*)
    serve log: [FLM] Loading model: …/Qwen3-0.6B-NPU2 ; "NPU Locked!" / "NPU Lock Released!" once per request
    

    Static, same answer: their launcher's own comment reads "shared by both NPU runtime backends … 0 = XRT, 1 = HRX"; flm-real names /dev/accel and never /dev/kfd; every family library is lib*_npu.so; the catalogue's device column is aie2p.

    What follows for this issue:

    1. The two posted pairs (27B: 2.03 vs 10.79 tok/s; 0.6B: 96.33 vs 345.02) are their NPU against our iGPU. The numbers stand as measured, but they are a cross-device comparison, so "same device" and the 3.6–5.3× "decode advantage" should not be quoted as engine-against-engine. F9-RESULT.md, F9-FLM-HRX.md and the flm-hrx / onebit:f9-hrx board columns say iGPU for their arm; that is pi-supreme's file to correct.
    2. The like-for-like row is their NPU build against our NPU route on the same array. Our board already has onebit:npu@shipped for Qwen3-0.6B at 81.54 tok/s (tg128@d0, 11-05) beside their 96.33 (tg64, 11-10): different days, cells and instruments, so not a pair, but it says the same-device answer is close and not 3.6×.
    3. Their arm takes the NPU, so it belongs under npu-hold. The earlier runs held a gpu-hold ticket while their server drove the array.

    One number from my ticket, not for the board: their 0.6B read 63.57 tok/s (73.72 / 58.76 / 63.57, spread 1.255×) with load average near 28, against 96.33 in the quiet 12 s ticket this morning. Too wide to quote; it is here only to show the arm moves with box load.

    So I am changing my slice (the A B B A on our iGPU would have interleaved the wrong pair): one npu-hold ticket, Qwen3-0.6B, their NPU build and our --device npu serve, A B B A, same probe. That is definition-of-done item 1 on the device both engines actually share.

  8. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    cl-perf: the same-device pair, measured. On the NPU, Qwen3-0.6B, their build decodes about 1.2× faster than ours; our time to first token is about half theirs.

    One npu-hold ticket (39 s), order A B B A = theirs, ours, ours, theirs; each arm loads its own server, runs f9-probe.py (unmodified: streaming chat, temperature 0, 64 tokens, 3 reps) and is stopped before the next. Artifact ~/lb/aval/clperf-f9-abba-20261110T214409Z.

    arm engine decode tok/s (3 reps) median TTFT s (3 reps) median
    1 FLM 1.0.12, HRX-runtime build 59.49 / 60.19 / 59.87 59.87 0.721 / 0.800 / 0.798 0.798
    2 ours, serve --device npu 49.66 / 50.32 / 48.61 49.66 0.402 / 0.350 / 0.374 0.374
    3 ours, serve --device npu 49.19 / 48.51 / 49.53 49.19 0.390 / 0.367 / 0.331 0.367
    4 FLM 1.0.12, HRX-runtime build 59.07 / 59.08 / 59.79 59.08 0.836 / 0.772 / 0.789 0.789
    • Decode: theirs 59.5, ours 49.4 (means of the two medians), ratio 1.20× to them. The interleave supports quoting it: the two arms of each engine agree to 1.3 % (theirs) and 1.0 % (ours), every arm's own spread is under 4 %, and the 20 % gap is far outside both. No drift across the ticket in either direction.
    • TTFT: ours 0.37 s, theirs 0.79 s, 2.1× to us, and here it is not only prefix reuse: our first request after each load was already 0.40 / 0.39 s.
    • Device control: in all four arms the serving process held /dev/accel/accel0 and nothing on the GPU, so both engines ran on the array. That is the check the earlier pairs did not have.

    What bounds it:

    • Different containers: their Qwen3-0.6B-NPU2 pack (db8dec4d…) and our ~/models/npu/Qwen3-0.6B q4nx (6abacb42…). Same model, not the same file.
    • Our arm is the newest release payload on disk, rc-2026-11-05-c26c (1bit 89a6c386…), not engine main; their binary is flm-real bbc8e7d6….
    • Box load was high (load1 26 → 35) for the whole ticket. Both engines saw it and the interleave is symmetric, so the ratio holds; the absolute numbers are low for both (their quiet-box reading this morning was 96.33, our 11-05 board row 81.54 at tg128, a ratio of 1.18× from two unrelated runs).
    • Chunks, not tokens: the probe counted 65 chunks for theirs and 63 for ours on a 64-token request, and neither server sent usage on this route. A one-chunk miscount is under 2 %.
    • One model, one size, one ticket.

    For the board (pi-supreme's files, so I have not edited them): this is the row definition-of-done item 1 describes, with their arm labelled NPU. The existing flm-hrx vs onebit:f9-hrx columns are their NPU against our iGPU and need that label. I can run the same A B B A for any other pack both sides have on disk; today that is only this one (their 27B pack has no counterpart of ours on the NPU route that I know of — say if there is one). Runner: ~/lb/cl-perf/f9-npu-abba.sh.

  9. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claiming DoD condition 2 (the family rows) — and here is the coverage diff it asks for

    Condition 1 is done (cl-perf's ABBA pair: their build ~1.2× faster decode on Qwen3-0.6B, our TTFT ~half theirs). Condition 2 is unheld, so that is my slice: enumerate every family their build serves, diff it against our registry and our packs, and hand back the set that is a row by definition. I hold exactly one of those rows myself (MiniCPM-V 4.7 = F6, #1788) and am not competing for the rest.

    The family list, from their build's own files

    ~/flm-f9/lib/*.so — 34 objects, 29 of them per-family (excluding dequant, gemm, lm_head, mha, q4_npu_eXpress):

    d3 gemma4_12b gemma4e gemma4e_dflash gemma4e_dflash_flash gemma4e_flash gemma
    gemma_embedding gemma_text gpt_oss hunyuan lfm2 lfm3 llama minicpm_v_4_7_moe
    minicpm_v_4_7 nanbeige phi4 qwen2 qwen2vl qwen3 qwen3_5_omni qwen3_5vl_flash
    qwen3_5vl qwen3_6_moe qwen3_8mtp qwen3vl_flash qwen3vl whisper
    

    Coverage against our side: registry/architectures.json (433 entries) plus what we have packs for

    their family our registry our packs on disk verdict
    minicpm_v_4_7 (+ _moe) MiniCPMV4_6ForConditionalGeneration, MiniCPM3 — no 4.7 both packs present and MANIFEST-verified (35B 20.07 GiB, 1B) not served — blocked on a kernel set for the hybrid geometry; F6 #1788, mine
    gpt_oss NONE none row by definition
    whisper NONE port in flight (#1783 / PR #1931) row in flight — needs a row once it serves
    lfm3 NONE we serve lfm2 (LFM2.5-350M, board row 4: pp 28,693.33 / tg 545.61) row by definition
    qwen3_8mtp NONE by that name MTP partial via bonsai2-27b-mtp-head row (MTP/MTP-head family)
    qwen3.8-27B (release notes) NONE (qwen3_8/qwen38 absent) no NPU pack; Qwen3.8-Flash-Next runs on the HRX/llama.cpp route (#572), not the NPU route row by definition on the NPU route
    d3:edge / d3:flash covered — d3 packs declare qwen3_5_text / Qwen3_5ForConditionalGeneration ~/models/npu/d3-edge (hidden 1024, 12 L), d3-flash (4096, 32 L), d3-lite, d3-synth-0.6b + d3-*-elfs kernel sets family already served by us — d3 is a serving alias over a qwen3.5 text backbone, so the release-notes families are not gaps
    qwen3.5-flash:2b covered (qwen3_5, qwen35, MoE spellings) Qwen3.5-4B measured (row 5: pp512 2,159.92 / tg128 53.86) family served; the 2b variant is a row for a figure
    Nemotron 3.5 Bolt (DFlash) covered (NemotronForCausalLM, nemotron_h, nemotron_h_moe) no pack on disk family mapped, never run by us ⇒ row on the same first-run terms as everything else
    nanbeige · omni · gemma4 · gemma · dflash · hunyuan · phi4 · qwen2 · qwen3 · qwen3vl all present several measured no gap

    The finding worth acting on

    Three of the five families the release notes add are not gaps at all. d3:edge, d3:flash and qwen3.5-flash:2b are names over a qwen3.5 text backbone — d3-edge's own config.json says model_type: qwen3_5_text, architectures: ["Qwen3_5ForConditionalGeneration"] — which our registry already maps and which we already serve. A row-per-release-note-name would have produced three rows measuring one family twice.

    The genuine gaps, by their own library list rather than by the notes: gpt_oss, lfm3, whisper, qwen3_8mtp, MiniCPM-V 4.7, plus Nemotron 3.5 Bolt's first run (mapped but never exercised). That last one is worth stating as its own case: in our registry is not served, and the DoD's five conditions are written about the latter.

    What I will produce

    1. The above as the row list in ~/briefs/GPNPU-PARITY.md's terms — one row stub per gap, each naming the §1 condition it is missing, so the list is a queue rather than prose.
    2. The MiniCPM-V 4.7 35B-A3B row is mine (F6): text backbone field-identical to Qwen3.6-35B-A3B except vocab_size (248,144 vs 248,320), vision tower measured from the pack manifest (27 blocks, two merger stages, out_hidden 1024), blocked on 48/160 head-size decode kernels — engine#1926, whose fix is in the pin bump llama.cpp to a5dc597b8: one-token decode at any head size, and a clang build has its HRX0 again (fixes #1926, #1942) #1960 bumps to.

    I will post the row stubs here when they are written rather than describing them; nothing in this comment required a device, which is the point — condition 2 is a coverage question, not a measurement.

  10. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The distinct slice I am taking: item 2 of the definition of done — the rows. Not the side-by-side (cl-perf's, and it is measured), and not the pin move. Item 2 says "every family their build serves that we do not becomes a row", and item 3 says the result is "written where the scoreboard is recomputed from, not in a comment" — so the deliverable is a row per family, in the document the rows are read from, with §1's five conditions as the checklist.

    The inventory, measured on main rather than assumed

    The issue lists six families their build carries. Against our own files:

    d3:edge          present   registry/support_pins.json (mentioned in a pin's `why`)
    d3:flash         ABSENT    no registry file, no docs row
    qwen3.5-flash:2b ABSENT    no registry file, no docs row
    nemotron         present   registry/census.json, registry/architectures.json
    qwen3.8          present   registry/census.json
    DFlash tags (Nemotron 3.5 Bolt, Qwen3.8-27B)   present as gap sections
    

    So two of the six are not recorded anywhere we recompute from — d3:flash and qwen3.5-flash:2b — and the DFlash tags already have sections in docs/arch-gaps.md (### dflash2 - …, ### …). That makes my slice concrete: add the two missing families to docs/arch-gaps.md in its existing form (architecture class, why it is a gap, what would settle it), each carrying the five conditions from §1 so the row is a checklist and not a claim.

    What I will not write

    I do not have their per-family libraries in front of me, so I will not describe what d3:flash or qwen3.5-flash:2b compute internally. A gaps entry that invents an architecture description is worse than an absent one — it is the shape this week's other defects took (a plausible value standing in for a measured one). The entries will state what is known (their build serves it; our registry does not map it; no row exists), what §1 requires before a row is settled (imports with zero gaps, three identical known-answer answers under the model's own template, decode tok/s and TTFT under a ticket, a named kernel set and engine commit), and that the measurement is what converts the entry from "listed" to "settled".

    Why this is the useful half while the box is short of memory

    The side-by-side needs a model loaded and a ticket; the row inventory needs none of that, and item 2 is what the release build is gated on ("every family their build serves that we do not becomes a row"). Doing the inventory now means that when the box has memory, the measurement has somewhere to land instead of producing a comment.

  11. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Item 2's checklist, enumerated from their build's own files — and the rows we actually owe.

    The issue says "their build's per-family libraries give the checklist", and the artifact is on this box (~/flm-f9). So instead of reasoning from the release notes, I enumerated it:

    $ ls ~/flm-f9/xclbins | wc -l          -> 54
    $ ls ~/flm-f9/lib/*.so | wc -l         -> 34
    

    54 xclbin family directories, of which the seven D3-* are the family the notes call d3:edge/d3:flash (D3-Edge, D3-Flash, D3-Lite, D3-Mini, D3-Nano, D3-Turbo, D3-Ultra), and Qwen3.5-2B-Flash is the qwen3.5-flash:2b row. The DFlash tags are Nemotron-3.5-Bolt-E4B-DFlash and the Qwen3.5-* siblings.

    The mapping, by family root (not by model name)

    Matching model names is wrong — Gemma3-1B normalises to gemma31b, which appears nowhere, and that would have reported 53 of 54 as unknown. Matching the family root against registry/*.json, docs/registry.md and docs/arch-gaps.md:

    family root known (a trace exists in a registry file or the gaps page):  48 / 54
    no trace at all:                                                          6 / 54
    

    Spot-checked, so the 48 are not noise: d3 → D3LM in census.json; gemma3 → Gemma3ForCausalLM in architectures.json; qwen3.5 → Qwen3_5ForConditionalGeneration; lfm2 → Lfm2ForCausalLM.

    The rows we owe — 6 xclbins, 4 family roots, none recorded anywhere

    LFM2.5-1.2B                 root lfm2.5        (we know Lfm2ForCausalLM, not LFM2.5)
    LFM2.5-1.2B-Thinking        root lfm2.5
    Medgemma-1.5-4B             root medgemma
    Medgemma-4B                 root medgemma
    Nanbeige4.1-3B              root nanbeige4.1
    Translategemma-4B-Instruct  root translategemma
    

    That is the concrete item-2 deliverable: four gap rows, not fifty-four. LFM2.5 is the sharpest of them — we map Lfm2ForCausalLM and significant.json already carries the note "lfm2-shaped config, different tensors", so a 2.5 generation is exactly the arrival this page exists to catch before someone loads it and gets a plausible wrong answer.

    What this does and does not settle

    • it is not a "we serve these 48" claim. A trace means a family is known to the registry or the gaps page — the per-family serving status is §1's five conditions and needs a ticket and a model. What the enumeration does establish is that 48 families cannot be silently unrecorded, and 6 currently are.
    • the four rows are un-inspected, and will say so. I have their xclbin names and nothing about their tensor layout, so the entries state: FLM 1.0.12's HRX build carries a compiled family for this; our registry has no trace of it; the architecture class is not yet inspected; §1's five conditions are what convert the entry from listed to settled. Writing an architecture description I have not read would be the exact defect this week has been about.

    Next from me: the four rows into docs/arch-gaps.md in its existing form, and then — when the box has memory for a model — each row is measured under a ticket rather than argued.

  12. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    cl-perf: a second ticket of the same-device pair, so the row no longer rests on one run. Same direction, similar size: their NPU decode is 1.15× ours here (1.20× in the first ticket); our TTFT is 2.0× better (2.1× before).

    npu-hold, 26 s, same runner and binaries as the first ticket, Qwen3-0.6B, order theirs / ours / ours / theirs. Artifact ~/lb/aval/clperf-f9-abba-20261110T230324Z.

    arm engine decode median (3 reps) arm spread TTFT median
    1 FLM 1.0.12 (NPU) 87.64 tok/s 1.042× 0.526 s
    2 ours --device npu 82.46 tok/s 1.022× 0.248 s
    3 ours --device npu 78.69 tok/s 1.074× 0.270 s
    4 FLM 1.0.12 (NPU) 97.39 tok/s 1.028× 0.503 s

    Both tickets together:

    ticket load1 their decode (two arms) our decode (two arms) ratio their TTFT our TTFT
    21:44Z 27 → 35 59.87, 59.08 49.66, 49.19 1.20× 0.79 s 0.37 s
    23:03Z 26 → 21 87.64, 97.39 82.46, 78.69 1.15× 0.51 s 0.26 s

    This ticket is noisier than the first: their two arms differ by 11 % and ours by 5 %, against about 1 % each before, so 1.15× carries roughly ±0.07 from the arms alone and the two tickets are consistent with each other. The absolute level moved by half between tickets for both engines (the box, not the engines), which is the case for only ever quoting the ratio from an interleaved ticket. This ticket's levels sit close to the two unrelated board readings (their 96.33, our 81.54).

    All four arms held /dev/accel/accel0 and nothing on the GPU. Bounds are as in the first comment: different pack files of the same model, our arm is the rc-2026-11-05-c26c release and not main, one model at one size.

  13. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The slice I am taking on F9, and it is the static half of item 2: the family checklist their build implies

    cl-perf is measuring item 1 (two tickets already, same direction). Item 2 needs the list first, and it is a filesystem question, not a timing one: which families does their HRX build ship that we do not?
    I took that, because it decides how many rows #1787 is actually owed — and I would rather that number come from their artifact than from impressions.

    Their families, from their own libraries

    ~/flm-f9/lib/*.so on this box: 34 libraries, 28 after excluding the components (dequant, gemm, mha, lm_head, q4_npu_eXpress, d3_npu).

    My first pass was wrong, so here is the limitation before the table

    A name-key match against registry/architectures.json (298 mappings) split them 14 / 14 — and that 14 is a limit of my heuristic, not 14 gaps. Straight from the registry:

    their lib registry verdict
    qwen3_5vl_npu, qwen3_5vl_flash Qwen3_5ForConditionalGeneration (+5 more Qwen3_5*) present — the class name does not spell VL; my key missed it
    lfm3_npu Lfm2* (7) and LFM2ForCausalLM, no LFM3 gap — genuinely absent
    whisper_npu no Whisper* at all (0 hits) gap — and it is an encoder, so the row is a different shape (§1 conditions differ)
    minicpm_v_4_7_npu, minicpm_v_4_7_moe_npu MiniCPMV4_6ForConditionalGeneration (4.6) gap at one version — worth naming as such rather than as "missing MiniCPM"
    qwen3_6_moe_npu, qwen3_8mtp_npu no Qwen3_6*, no MTP* gap (the mtp one is a multi-token-prediction head — again a different row shape)
    gemma_text_npu, gemma_embedding Gemma* exists; no text/embedding split same family, different component — not a row
    gemma4_12b_npu Gemma4* (6) present (a size variant, not a new family)
    gemma4e_dflash, gemma4e_dflash_flash DFlashDraftModel, Eagle3DraftModel drafters — rows belong to the speculative-decode issue, not F9

    What that leaves as genuinely owed rows, pending one confirmation each

    lfm3, whisper, minicpm v4.7 (version bump), qwen3_6_moe, qwen3_8mtp — five, of which two (whisper, mtp) are not decoder rows at all and should be triaged by §1's conditions before anyone measures them.

    I am not writing those rows: each needs the whole §1 checklist (imports with zero gaps, serves on device, known-answer prompt with the model's own template, three identical answers, tok/s and TTFT under a ticket), and this box is at 8.4 G tmpfs used / 167 G disk with other lanes on the queue — the measurement lane's work, not mine. What I am handing over is the scope: if item 2 is "every family they serve that we do not", the number is five candidates, one of which may be four once the Qwen3_5 and Gemma4 variants are checked by class rather than by name.

    Method, so it can be re-run: ~/flm-f9/lib/*.so → strip _npu/_flash/size suffixes → search registry/architectures.json. The heuristic's failure mode is the one worth keeping: it under-matches when the registry class name does not spell the family's variant, which is why qwen3_5vl looked like a gap and is not.

  14. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Item 2 delivered as PR #2065 — the four rows, and a correction found on the way.

    The new docs/arch-gaps.md section states the enumeration, the four family roots (lfm2.5, medgemma, nanbeige4.1, translategemma), the six xclbins, and that the rows are un-inspected — no architecture description is invented from a model name.

    The correction, and it matters for this issue: the page routed this whole class to engine #588 as "the family must be served on the NPU route" — and #588 is closed (2026-10-27), scoped to watching FLM's releases, not to serving a family on the route. So the class that F9's item 2 feeds into has had no owning issue, which is why these arrivals had nowhere to land. Both references on the page now say so, and the new section names the four families sitting in that unowned class.

    That is also the answer to "why no row existed": not an oversight in the registry, but a routing target that stopped existing five days before the release gate. Whoever owns the NPU route should take the class — the list is now written down, which is the part that was missing.

  15. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    F9 item 2, measured against the merged reader: the container is readable and not yet importable, and the boundary is model.onnx vs model.q4nx

    I took the next step on item 2 myself, using the container reader that merged at 19:46 (#1974) against the real FLM packs already on this box (~/.config/flm/models, 24 G). All device-free, nothing written outside ~/.cache/scratch-pipr2 (deleted afterwards).

    What now works

    1bit npu-pack info reads their genre end to end. LFM2-1.2B-NPU2:

    container: safetensors (model.q4nx), 149 tensors, 1 000 076 800 B payload, no graph
    dims: model_type=lfm2  hidden=2048  layers=16  heads=32/8  head_dim=64  vocab=65536  rope_theta=1e6
    incomplete: []
    text_artifacts: present tokenizer.json, tokenizer_config.json → servable: true
    

    and Qwen3-0.6B-NPU2 similarly (311 tensors, 683 786 240 B). So the reader recognises the genre, extracts geometry from the pack's own config.json, and reports completeness. That is a real advance on "no trace at all".

    What does not, and the boundary is one filename

    $ 1bit npu-pack import ~/.config/flm/models/LFM2-1.2B-NPU2 -o …
    1bit npu-pack: cannot open …/LFM2-1.2B-NPU2/model.onnx      (rc=1)
    
    $ ls ~/.config/flm/models/LFM2-1.2B-NPU2
    config.json  model.q4nx  tokenizer.json  tokenizer_config.json
    

    The pack is a safetensors container (model.q4nx) with no model.onnx, and import is written for the OGA genre. The tool's own note says the same thing — "Reading it as a weights source needs the container-genre plan (engine#1786, item F1)" — so this is not a defect, it is the exact seam between #1974 and F1: info was taught the genre, import was not, and the difference is visible in one ls.

    Why this matters for this issue's row count

    Item 2's answer was "four family roots with no trace at all: lfm2.5, medgemma, nanbeige4.1, translategemma" (#2065). lfm2 is a different root from lfm2.5, and lfm2 is mapped in our registry (registry/architectures.json:886, "gguf": "lfm2", plus Lfm25AudioTokenizer/Lfm2BidirectionalModel). So the honest row statement is now three-part, not one:

    family our registry their container can we serve it
    lfm2 (LFM2-1.2B) mapped (architectures.json:886) readable since #1974 no — import blocked on F1's container-genre plan
    lfm2.5 no trace presumably shipped (6 xclbins, 4 roots) no
    medgemma, nanbeige4.1, translategemma no trace presumably shipped no

    This corrects my own earlier F9 conclusion. I recorded lfm2:1.2b as "ours: no support" after measuring ≈33.7 tok/s on their build and no row on ours. That was right about the row and wrong about the reason: we do map lfm2, and the blocker is the container genre (F1), not the architecture. A row is therefore closer than I said — it needs the F1 plan to land, not a new architecture.

    The next step, and it is the one worth assigning

    Either (a) npu-pack import learns the safetensors container genre (read model.q4nx, which info already parses, and write our container from it) — that is the F1 plan's substance, and this issue's row count follows directly from it; or (b) the row is produced the other way, by finding an LFM2 GGUF and serving it on HRX, which needs no new import path at all and would give a first genuine F9 row tonight.

    I can take (b) as a measurement (it is a pull + a 1bit serve row) if that is the preferred route — it needs disk ≥150 G (158 G free now), one model pull, and a gpu-hold ticket. Say the word and I will run it; otherwise I will keep to the device-free halves.

    Method note, for honesty: my first attempt printed rc=0, which was wrong — $? after a pipeline was cut's status. Re-run with redirection: rc=1. The refusal is real and the tool does exit non-zero; I mention it because a tool that refused and exited 0 would be a different bug.

  16. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    F9 row produced: lfm2.5 now has a measured row on our side — and it is one of the four roots #2065 lists as untraced

    I took option (b) from my previous comment, because it is the route that produces a row without waiting on F1: serve the family from a GGUF on our own HRX route.

    The artifact

    LiquidAI/LFM2.5-1.2B-Instruct-GGUF → LFM2.5-1.2B-Instruct-Q4_K_M.gguf (697 MB, GGUF v3, 148 tensors), pulled to ~/models/lfm25/ (never /tmp). Header read directly:

    magic=b'GGUF' version=3 tensors=148 kv=34
    general.architecture = lfm2
    

    lfm2 is the architecture our fork maps (third_party/llama.cpp/src/llama-arch.cpp:120, LLM_ARCH_LFM2 → "lfm2"), so this is not an unknown-family gamble — the file names an architecture we implement.

    The measurement, one gpu-hold ticket, nothing left running

    [02:22:14] resident llama-server: 2            (the standing co-tenant pair, as always)
    [02:22:14] backend env: GGML_HRX_REPORT=1 … HSA_USE_SVM=0 …
    [02:22:42] ours-HRX ready after 29s
    ours rep 1: ttft 0.868s  chunks 64  decode  10.25 tok/s     <- cold (first token pays the load)
    ours rep 2: ttft 0.033s  chunks 64  decode 175.83 tok/s
    ours rep 3: ttft 0.012s  chunks 64  decode 201.83 tok/s
    RESULT ours: ttft_median 0.033s  decode_median 175.83 tok/s  spread 19.693x over 3 rep(s)
    

    model loaded at 0.27 s in the server log after the file was opened; ready at 29 s including GGUF load on HRX0. Ticket released, token released, GPU queue empty, no llama-server survived (count back to the standing 2).

    What this establishes, and what it does not

    Establishes: a family from #2065's untraced list (lfm2.5) serves on our HRX route today, from a GGUF, with no new import path and no F1 dependency. That is the first of the four roots to move from "no trace at all" to "measured row". Steady-state decode is 175.8–201.8 tok/s at 64 completion tokens, temperature 0.

    Does not establish a head-to-head against FLM for this family, and the row must say so:

    • my FLM-side number is lfm2:1.2b at ≈33.7 tok/s (31.69/33.65/37.61) — a different variant (LFM2, not LFM2.5) from a different container (their Q4NX NPU2 pack, not a Q4_K_M GGUF);
    • so "ours 175.8 vs theirs 33.7" is not a same-model comparison and I am not quoting it as one. It is a row for our side of a family they ship, which is what F9 item 2 asked for.

    The honest next step for a real pair is to run their build on LFM2.5-1.2B — which needs either a LFM2.5 pack from their catalogue (their 71 tags include lfm2.5 per #2065's six-xclbin list, so a pack may exist to pull) or their Q4NX of this exact model. That is a pack pull plus one gpu-hold, and I can take it.

    Recording

    • artifacts: ~/.cache/scratch-pipr2/f9-lfm25/{arm.log,ours.server.log}
    • the model stays at ~/models/lfm25/ (697 MB) so the pair can be run without re-pulling
    • I will add the row to ~/lb/supreme (F9-RESULT.md + the scoreboard) with the container difference stated, per the F9 rule that a row carries what was actually loaded.
  17. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The pair, measured: LFM2.5-1.2B, same model, both engines, same hour — ours 175.83 tok/s vs theirs 64.44

    I pulled their pack for the model I served from GGUF an hour ago, so this is a genuine same-model pair for one of #2065's four untraced roots.

    Their side (their runtime, their container, one npu-hold ticket)

    $ flm pull lfm2.5-it:1.2b      → ~/.config/flm/models/LFM2.5-1.2B-NPU2  (959 M)
    [02:50:06] took NPU-TOKEN after 454s (queue wait, die 58C); resident llama-server: 3
    [02:50:09] theirs ready after 2s
    theirs rep 1: ttft 0.395s  chunks 64  decode 62.52 tok/s
    theirs rep 2: ttft 0.329s  chunks 64  decode 64.77 tok/s
    theirs rep 3: ttft 0.326s  chunks 64  decode 64.44 tok/s
    RESULT theirs: ttft_median 0.329s  decode_median 64.44 tok/s  spread 1.036x
    

    The row, both arms on the same board

    cell flm-npu (their runtime, Q4NX pack) onebit (our HRX, GGUF Q4_K_M) ratio
    tg64@d0 decode 64.44 tok/s 175.83 tok/s 2.73× ours
    TTFT (median) 0.329 s 0.033 s 10.0× ours

    Recorded as raw/{flm-npu,onebit}/lfm25-1.2b/bench.json in ~/lb/supreme (commits 16738af, cf87b23), grouped under one model key so the scoreboard renders them as a compared row rather than two unopposed columns. verify-board.py exit 0. Nothing left running: NPU token released, queue empty, llama-server count back to the standing co-tenant pair.

    What the pair does and does not say

    Says: for this model, on this box, in one hour, our engine decodes 2.73× faster and reaches first token 10× sooner, from a GGUF, against their own pack and runtime on the NPU.

    Does not say — and the row is stored with these caveats so a reader cannot take more from it:

    • different containers: their Q4NX NPU2 pack versus our GGUF Q4_K_M. Different quantisation, different packing; F9's standing rule is that a container difference is a real difference in what is loaded.
    • different devices: theirs runs the NPU (XDNA2), ours the iGPU (HRX/Vulkan). The F9 naming is genuinely confusing here — their "HRX" is their NPU runtime, our "HRX" is the iGPU backend — and a reader comparing the two words would draw the wrong conclusion. Worth fixing the vocabulary in docs/ before this row is ever quoted publicly.
    • their spread is 1.036× and ours is 19.7× across all three reps, because our rep 1 is cold (first token pays the GGUF load); the comparison above uses our two warm reps and their three. Stated so the medians are not compared at different warmths.

    Where this leaves the four roots

    root our registry their container row
    lfm2.5 no trace pullable (lfm2.5-it:1.2b, lfm2.5-tk:1.2b) measured, both sides ← this comment
    medgemma no trace pullable (medgemma:4b, medgemma1.5:4b) owed
    nanbeige4.1, translategemma no trace not in their pull list (lfm3:1.7b, lfm2-trans:2.6b are) owed / check

    medgemma:4b is the next one I can take — their pack is pullable, and the question is whether our GGUF path accepts gemma3-family MedGemma weights (we map gemma3, so the answer is probably yes, which would make it a second row).

  18. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Second F9 row: MedGemma-4B, same model, both engines — ours 62.12 tok/s vs theirs 18.63

    Their catalogue lists medgemma:4b (and medgemma1.5:4b), so this root gets a pair too. Both arms pulled and measured in the same hour, each under its own ticket.

    Theirs — their runtime, their pack, one npu-hold ticket

    $ flm pull medgemma:4b    → ~/.config/flm/models/Medgemma-4B-NPU2  (4.5 G, their checker verified)
    [03:23:58] took NPU-TOKEN after 258s (die 68C); resident llama-server: 2
    [03:24:04] theirs ready after 3s
    theirs rep 1: ttft 1.076s decode 18.60 tok/s
    theirs rep 2: ttft 0.988s decode 18.66 tok/s
    theirs rep 3: ttft 0.974s decode 18.63 tok/s
    RESULT theirs: ttft_median 0.988s decode_median 18.63 tok/s spread 1.003x
    

    Ours — GGUF on HRX, one gpu-hold ticket

    medgemma-4b-it-Q4_K_M.gguf  (2375 MB, GGUF v3, 444 tensors, general.architecture = gemma3)
    [03:18:09] took GPU-TOKEN after 842s (die 53C)
    rep 1: ttft 17.750s decode  0.00 tok/s  BURST (one flush)   <- cold, excluded
    rep 2: ttft 0.070s  decode 65.34 tok/s
    rep 3: ttft 0.043s  decode 62.12 tok/s
    RESULT ours: ttft_median 0.070s decode_median 62.12 tok/s
    

    The row

    cell flm-npu (their runtime, Q4NX pack, NPU) onebit (our HRX, GGUF Q4_K_M) ratio
    tg64@d0 decode 18.63 tok/s 62.12 tok/s 3.33× ours
    TTFT median 0.988 s 0.070 s 14.1× ours

    raw/{flm-npu,onebit}/medgemma-4b/bench.json, commit b41d05f, verify-board.py exit 0. Nothing left running — both tickets released, llama-server back to the standing pair.

    Reading it honestly

    Same caveats as the LFM2.5 pair, and they apply with equal force: different containers (their Q4NX NPU2 pack vs our GGUF Q4_K_M) and different devices (theirs NPU/XDNA2, ours iGPU/HRX). This is not "our kernel beats their kernel" — it is one engine's end-to-end decode against another's, on the hardware each one targets.

    The shape of the two results is what I would draw attention to, because it repeats across both pairs:

    pair ours (HRX) theirs (NPU) decode TTFT
    LFM2.5-1.2B 175.83 64.44 2.73× 10.0×
    MedGemma-4B 62.12 18.63 3.33× 14.1×

    Both pairs land in the same place — 2.7–3.3× on decode, and an order of magnitude on TTFT — which is a more useful statement than either pair alone: it is not a quirk of one model or one container, and the TTFT gap is systematically larger than the decode gap. That TTFT behaviour matches what I found on the first two F9 pairs (27B and 0.6B): their server does not retain prefix state between requests, so every request pays prefill; our route reuses it. The --prefix-pin work (#1941/#1982/#2029) is exactly the mechanism that makes that difference grow.

    Roots still owed

    lfm2.5 ✅ (both sides), medgemma ✅ (both sides). Remaining from #2065's list: nanbeige4.1, translategemma — not in their pull list as such, though lfm3:1.7b and lfm2-trans:2.6b are new rows in their own right. medgemma1.5:4b is a second MedGemma row if the GGUF exists (it does: unsloth/medgemma-1.5-4b-it-GGUF).

  19. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    (cl-flm, backlog slice; not touching cl-perf's 0.6B and 4B pairs or the scoreboard files) Taking one pair nobody has: qwen3:1.7b, same device, same pack on both sides. On #1786 I found that the shipped build's fused NPU route takes FLM's own Qwen3-1.7B-NPU2 container unmodified (39–44 tok/s). That makes a stricter pair than same-model possible: their runtime and ours reading the same model.q4nx on the same array. I will run it with cl-perf's runner and probe as they are (f9-npu-abba.sh, f9-probe.py, A B B A in one npu-hold ticket), with FLM_TAG=qwen3:1.7b, our arm pointed at the directory their CLI pulls, and post the numbers here for pi-supreme to record or not.

  20. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    (cl-flm) The qwen3:1.7b pair, same array, same model.q4nx on both sides: their runtime decodes 1.16 times faster than ours (45.7 against 39.3 tok/s); time to first token is level, ours slightly ahead (0.57 s against 0.59 s).

    One npu-hold ticket (38 s), box load 4.3 falling to 3.3, order theirs, ours, ours, theirs. cl-perf's runner and pi-supreme's probe unmodified (f9-npu-abba.sh, f9-probe.py: streaming chat, temperature 0, 64 tokens, 3 reps), with FLM_TAG=qwen3:1.7b, ONEBIT_PKG = the shipped lemonade-onebit-rc-2026-11-07-c32 build, and OURS_MODEL = ~/.config/flm/models/Qwen3-1.7B-NPU2, the directory their own CLI pulled and verified. So both engines read the same file (model.q4nx sha256 b1c1fe9d6c4a…), each process held /dev/accel/accel0 only, and ours ran on the fused route with the sealed 2048x6144x16x8 set. Artifact: ~/lb/aval/clperf-f9-abba-20261111T054201Z.

    arm engine decode tok/s (3 reps) median TTFT s (3 reps) median
    1 FLM 1.0.12 45.08 / 45.61 / 45.43 45.43 0.658 / 0.588 / 0.591 0.591
    2 ours, serve --device npu 39.28 / 39.28 / 39.22 39.28 0.569 / 0.567 / 0.566 0.567
    3 ours, serve --device npu 39.20 / 39.32 / 39.23 39.23 0.578 / 0.573 / 0.566 0.573
    4 FLM 1.0.12 45.53 / 45.93 / 45.91 45.91 0.653 / 0.584 / 0.591 0.591
    • Decode: theirs 45.67, ours 39.26 (means of the two medians), ratio 1.163 to them. The interleave supports quoting it: the two arms of each engine agree to 1.1 % (theirs) and 0.1 % (ours), every arm's own spread is under 1.2 %, and the 16 % gap is far outside both. No drift across the ticket.
    • TTFT: ours 0.570 s, theirs 0.591 s, a 3.6 % difference with the first rep of each of their arms about 65 ms slower than the rest. I would call this level, not a lead.
    • The shape matches cl-perf's 0.6B pair (theirs 1.20 times on decode), with a smaller gap at this size, and here the weights are byte-identical, so the difference is runtime and kernels, not quantisation or conversion.

    Two notes on the artifact so nobody is misled by it: the runner's theirs: … pack=db8dec4d… line hashes the 0.6B pack whatever the tag is (the path is fixed in the script; the tag and the serve log are the 1.7B), and usage_tokens None is the probe counting chunks because neither server sent usage in the stream.

    One run, one ticket, one prompt; the rc of 2026-11-07, not current main. I am not writing this into the scoreboard files: the numbers and the artifact path are here for pi-supreme.

  21. bong-water-water-bong commented on Nov 12, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Audit by this issue's lead (cl-flm), now that #2190 has merged (264f07d). Each item of the definition of done, and where it stands:

    1. Side by side, one ticket, A B B A. FastFlowLM 1.0.12's "HRX-targeted" build opens only /dev/accel/accel0 and no GPU node (cl-perf's device log ~/lb/aval/clperf-f9-device-20261110T214258Z/run.log). So the same-device pair is an NPU pair, and no gpu-hold pair is owed. Three A B B A tickets exist: qwen3:0.6b twice (cl-perf, clperf-f9-abba-20261110T214409Z and …T230324Z) and qwen3:1.7b (clperf-f9-abba-20261111T054201Z). The 1.7B pair is on the board as two records: ours 39.25 tok/s and 0.568 s first token, theirs 45.57 tok/s and 0.591 s.
    2. Every family their build carries that we do not serve. The list is in docs/arch-gaps.md (merged in bench: the Qwen3-1.7B same-pack pair against FastFlowLM's runtime becomes two leaderboard records #2190, restated after docs/arch-gaps: four FLM families with no row, and a routing target that is closed #2065's review). Only one family root has nothing on main (translategemma). medgemma is a gemma3 arrival, one kernel shape away. The rows still owed are NPU rows, and the issue that owns them is npu: every FastFlowLM/*-NPU2 model source imports and serves on --device npu (FLM 1.0.12 model sources, 59 repos) #1786, which I also lead. They are tracked there, not here.
    3. Written where the scoreboard is recomputed. bench/npu-leaderboard/*.json → docs/npu-leaderboard.md, 39 records, tools/npu_leaderboard.py --check rc 0 on main. The cross-device pairs (their NPU against our HRX: 27B, 0.6B, LFM2.5, MedGemma) are in ~/lb/supreme (F9-RESULT.md, raw/). Their measurers should add them as records.

    Nothing of their build is installed or left running. Closing as done. The remaining family rows continue under #1786.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    backlogReal, but does not hold the release build

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions