Skip to content

npu: Whisper encoder on our runtime — measure against FLM 1.0.12's 0.6 s/30 s window and close the gap (W8/F8) #1783

Open

Description

@bong-water-water-bong

W8/F8: Whisper on the NPU — measure our encoder against FLM 1.0.12's claim (~3× faster: 0.6 s per 30 s window, was 1.7-2.0 s) and close the gap.

Owner 2026-11-09 23:2x, plan ~/briefs/TNPU-ALL.md §9, item F8 (after the gemma-4 rows). FastFlowLM is reference and
a model source only
— nothing of theirs is run or copied; our engine does the same thing on its own runtime and kernels.

The claim to beat (public release notes, FastFlowLM/FastFlowLM v1.0.12, 2026-11-05)

The Whisper encoder is about 3× faster: 0.6 s per 30-second window instead of 1.7-2.0 s. A 5-minute recording now
takes 25 s instead of 45 s.
⚠️ A Whisper-only server now uses about 2.3 GB of memory instead of 1.0 GB. Set WHISPER_CACHE_DEQUANT=0 to get back to
1.0 GB.

Their benchmark page: fastflowlm.com/docs/benchmarks/whisper_results/ (per-model tables; to be quoted when the digest lands —
see the note at the end).

Where we are (measured, 2026-11-08, ~/briefs/TDEMO-AMDCOL.md)

Seven amd/whisper-*-onnx-npu packs are in scope and all seven are measured, every transcript exact — but on ONNX
Runtime's CPU EP
, because our engine has no Whisper route at all (no whisper/audio code under npu/, app/ or
docs/; the NPU-Whisper packs are listed in the board as "SD/Whisper on our NPU runtime" scope, not as rows):

pack size enc dec (steps) transcript
whisper-large-v3-onnx-npu 6.95 GB 7.33 s (enc only, load_avg 112) 170.20 s (26) [CONTENDED] exact
whisper-small-onnx-npu 1.09 GB 0.39-0.91 s 7.2 s (26) exact
whisper-small-en-onnx-npu 1.09 GB 0.39 s 10.0 s (27) exact
whisper-base-onnx-npu 0.32 GB 0.43 s 4.1 s (26) exact
whisper-tiny-onnx-npu 0.17 GB 0.12-0.45 s 1.5-3.0 s (24) exact

So large-v3's encoder is ~12× their 0.6 s on the same 30 s window, and the small/base ones are inside their old band —
the gap is real for the model people actually use.

Plan (own code, no vendor artifacts)

  1. Baseline harness — one audio fixture, decoded into 30 s windows, our current ORT-CPU encoder timed per window and per
    recording, transcripts recorded. That is the number every later claim is compared to (the plan's rule: measured, never
    inferred). Also measure peak RSS, since FLM's own gain cost them 1.3 GB.
  2. Encoder route on our runtime — take the pack's own ONNX encoder graph (encoder_model.onnx and its .data), map it
    onto our NPU path the same way the LLM packs are mapped (importer → container → forward), writing whatever is missing
    from scratch: mel front-end, conv stack, attention blocks. Kernel families as needed go to the private npu-kernels
    repo, claimed in NPU-KERNEL-FAMILIES.md first; full ELF, Loom AIE2P or Peano.
  3. Correctness bar — bit-exact transcripts against the CPU baseline on ≥3 fixtures plus an edge case (silence, short
    clip), 3 repeats returning identical text. Speed is claimed only after that.
  4. Measure and close the gap — per-window encoder time on the device under npu-hold (≤9 min tickets, host-side proof
    first), then the same 5-minute recording FLM quotes (25 s vs 45 s), and the memory number.
  5. Row — the five DONE points (import zero gaps + self-check, serve on --device npu, known-answer 3× identical,
    measured tok/s or RTF, row names kernel set + engine commit).

Rules this issue operates under

Own code only (FLM and AMD artifacts are reference: observe formats and behaviour, write our own); kernels and RE detail
stay private; full ELF only, Loom AIE2P or Peano, never xchesscc/aietools/Vitis; AMD packs run as published and are never
redistributed (licence recorded per row); device work only under npu-hold; packs and build trees on disk
(~/scratch-packs), never /tmp; commit the same hour; no publishing.

Note for pi-verify: ~/amd-oss/npu-re/docs/FACTS-fastflowlm-v1012-delta.md (named as the fact base in §9) is not on
disk yet
— I am working from the public release notes above and will fold the digest's per-model tables in when it lands.

Tracking: this issue. Row lands on ~/briefs/AMD-RAI-BOARD.md via ~/lb/amd-rai-row.py when the five points hold.

Activity

  1. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claimed (pi-verify). Same worktree as the FLM v1.0.12 digest I finished tonight, so the "quote the benchmark page when the digest lands" note is now dischargeable — and it changes one number in the issue's framing.

    The claim, with its conditions (public, fastflowlm.com/docs/benchmarks/whisper_results/)

    before (#941 not in) now (FLM v1.0.12)
    encoder, per 30 s window 1,889 ms 606 ms
    decoder, per token 20.3 ms 20.0 ms
    5-min recording (308 s, 15 windows) 44.7 s 25.3 s
    6.4 s clip 2.51 s 1.16 s

    Conditions the page states, all of which matter to us: AMD Ryzen AI MAX+ PRO 395 (Strix Halo) — our own box class — FLM's default NPU Performance power mode, same runner and build in both columns, only the Whisper engine and kernels differ; Whisper runs 30 s windows with both encoder and decoder on the NPU. RTF table (FLEURS, before → now): en 0.259→0.130 (20 clips), en 0.140→0.082 (252 s), en 0.146→0.082 (308 s), de 0.203→0.109, es 0.230→0.123, fr 0.269→0.144, ja 0.215→0.121, zh 0.275→0.148, ar 0.292→0.171.

    The memory trade is now stated rather than implied: 1.0–1.1 GB → 2.3–2.4 GB for a Whisper-only server, because the encoder's dequantised weights are cached; WHISPER_CACHE_DEQUANT=0 restores 1.0 GB and costs about 0.8 s per 30 s window. So their 606 ms encoder is only reachable with the larger footprint — our own bar needs both numbers or it is not comparable.

    One correction to the issue's comparison

    The issue compares our 7.33 s (large-v3 encoder, ORT CPU EP, load_avg 112 contended) against their 0.6 s, and calls it ~12×. That mixes execution units: their 0.6 s is an NPU number and their CPU-free path is not on offer (on Windows their encoder's CPU steps run single-threaded, ~0.7 s/window). The honest bar for us is therefore: their NPU 0.606 s vs our CPU 7.33 s is not the gap — the gap is that we have no NPU encoder route at all, and their number says what the route must reach once it exists. Their old 1.889 s is the more useful intermediate target, because it is what the same NPU reached before the kernel work.

    Feasibility of step 1 on this box (checked, not assumed)

    whisper-large-v3-onnx-npu and the other six packs are not under ~/models here, and python3 -c "import onnxruntime" raises ModuleNotFoundError. The numbers in the issue's table come from the AMD Developer Cloud run recorded in ~/briefs/TDEMO-AMDCOL.md, not from strixhalo. So step 1 needs one of:

    1. the cloud box (where ORT and the packs already exist) — reproduce the table there, which is also the machine the current numbers came from, or
    2. an ORT venv here plus the packs, which is a download of several GB (large-v3 alone is 6.95 GB) and an install — a decision I am not taking unilaterally while the box is at 188 G free and the owner is snapshotting.

    I will write the harness against whichever of those is chosen, and I am not claiming the device side (steps 2–4) here: no Whisper route exists in our engine, so that is a kernel-family project for the npu-kernels lane, not a slice this lane can land.

  2. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Baseline measured on FLM's own unit — seconds per 30-second window — for the two models that matter. Our encoder is a CPU path today, and the number to beat is 0.6 s.

    Harness: ~/wt/amdcol-tools/whisper_window_bench.py (in this repo's spirit: --repeats, mel timed separately, peak RSS beside the latency because FLM's 3× cost them 1.3 GB). onnxruntime 1.31.1, jfk.wav (11 s → 1 window), both packs' plain ONNX encoders — the honest Linux path, since the .rai is inert outside AMD's Windows-only runtime.

    pack ORT threads encoder s / 30 s window peak RSS
    whisper-small (80 mel bins) default / 4 / 16 1.60 / 0.87 / 0.72 ~1120 MiB
    whisper-large-v3 (128 mel bins, 1280-wide) 16 1.62 (1.56–1.68) 3140 MiB

    Against their number: FLM 1.0.12 says "0.6 s per 30-second window instead of 1.7–2.0 s" on the NPU, at 2.3 GB with WHISPER_CACHE_DEQUANT on. So our CPU large-v3 baseline (1.62 s) sits at their pre-optimisation level, and their NPU is 2.7× faster than our CPU — which is the gap F8 is actually about, and it is a gap in route (we have no NPU encoder at all), not in tuning a kernel.

    The variable that swamps everything else, measured: encoder latency moves >2× with ORT thread count alone (1.60 s default → 0.72 s at 16 threads, same pack, same audio, same box). Any comparison — theirs or ours — that does not state the thread/queue configuration is measuring the box. Both numbers above are at 16 threads with the box at load ~15; the earlier 7.33 s I recorded for large-v3 was the same pack at load_avg 112, i.e. the same lesson.

    Two harness facts worth keeping: the mel bin count must come from the model (80 for small/base, 128 for large-v3 — hard-coding 80 fails large-v3 with index: 1 Got: 80 Expected: 128), and the mel front-end belongs in the report separately (0.03 s for the 128-bin path, 0.13 s for 80) so it can be matched against whatever the other side counts as "the encoder".

    Where this leaves F8: the baseline half is done and reproducible; the gap is a route we do not have. The next step is the encoder on our own NPU path (conv stack + attention blocks from the pack's ONNX, kernels claimed in NPU-KERNEL-FAMILIES.md), gated on bit-exact transcripts against this CPU baseline before any speed claim — and the row's number is s/window at a stated thread/queue configuration, with RSS beside it.

  3. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Design (15 lines): the Whisper encoder on our NPU path, per the owner's 22:4x from-scratch go-ahead. Starting here, not reporting blocked.

    1. Shape of the job, measured (large_v3_encoder.onnx, 1031 nodes, 492 initializers): x[1,128,3000] → layer_norm_64[1,1500,1280]; ops = MatMul 256, Add 225, Reshape 192, Transpose 161, LayerNormalization 65, Mul 64, Gelu 34, Softmax 32, Conv 2. So it is a transformer, not a convolution stack: two stride-2 convs with GELU at the front, then 32 blocks of LayerNorm → MHA → GELU MLP at hidden 1280, 20 heads (1280/64).
    2. It is one graph with no NPU-specific ops, which is the good case: our importer already maps MatMul+Add families onto the NPU for the LLM line, and the rest (LayerNorm, Gelu, Softmax, Transpose/Reshape) are the same elementwise/shape ops the LLM path already handles or falls back on host.
    3. First slice (host, no device): an ONNX→our-container path for this graph. The encoder is 3.5 GiB of f32 weights; quantising to the container's int8/Q4 form and running it on the host GEMM path gives a working reference implementation in our own code, which is the thing the NPU version will be compared against.
    4. Second slice: the kernels. The MHA and MLP GEMMs are the heavy ops; their shapes are known per block (K = N = 1280, and 1280×5120 for the MLP) and go into NPU-KERNEL-FAMILIES.md as a claim before any build, per the rules — full ELF, Loom AIE2P or Peano, grp=32-style quantisation as the LLM line uses.
    5. Third slice: the conv front-end. Two stride-2 convs are the one op our NPU path does not have; they are small (K = 128→1280 over 3000→1500 frames) and can stay on the host in slice 2 without weakening the measurement, because the timing says where the time is (see 7).
    6. Correctness bar: bit-exact transcripts against the CPU baseline already measured (large-v3: 1.62 s/window, 3140 MiB; small: 0.72 s at 16 threads) on ≥3 fixtures plus an edge case (silence, sub-window clip), 3 repeats returning identical text — the plan's §2 rule, and for speech the transcript is the only honest end-to-end check.
    7. Where the time actually goes, to be measured before any kernel work: the CPU baseline at 1.62 s/window is 32 blocks × (4 GEMMs of 1500×1280×1280 plus MLP) — if the GEMMs are ~90 % of it, the NPU route is the answer; if the front-end or the softmax is, the kernels are the wrong first move. This measurement comes before slice 2, and it decides whether the 0.6 s target is a GEMM problem at all.
    8. Target: 0.6 s/window at a stated thread/queue configuration, with RSS beside it — the number FLM 1.0.12 quotes, and the reason to use their unit rather than tok/s.
    9. No vendor artifact is used: the pack's own ONNX graph and weights are the input, our container format and our kernels are the output; the .rai in the pack is inert outside AMD's runtime and is not touched.
    10. Own code only: the mel front-end is ours (whisper_window_bench.py already computes it with numpy), the graph reader is ours (npu/pack_import.cpp), the kernels are ours (private npu-kernels).
    11. Measurement discipline: every number states ORT thread count / kernel set / engine commit; encoder latency moves >2× with threads alone on this box, which is larger than the 2.7× gap being closed.
    12. Refusal stance: if a block's GEMM does not fit an available kernel family, the encoder refuses by name at plan time rather than falling back silently to host and reporting a "mixed" number as if it were an NPU one.
    13. Milestones: (a) host implementation + transcript parity, (b) time attribution, (c) GEMM kernels + NPU path for the blocks that dominate, (d) the conv front-end if it matters, (e) 0.6 s/window measured and a row.
    14. Risk: Whisper's attention is bidirectional and full-sequence (1500×1500 scores), unlike the LLM line's causal decode; if the fused attention op needs a new kernel, that is the largest single piece and it is why (b) comes before (c).
    15. Tracking: engine#1783 (this issue), kernels claimed in ~/briefs/NPU-KERNEL-FAMILIES.md, packs on disk at ~/scratch-packs/pi-demo/whisper-{small,lv3}, and the CPU baseline recorded in ~/briefs/TDEMO-SHOWCASE-STATE.md §I.1.
  4. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Step 7 of the design is answered by measurement: the GEMMs are ~78 % of the encoder, the conv front-end is 1 %, so kernels are the right first move and the conv stack can stay host-side without distorting the number.

    ~/wt/amdcol-tools/whisper_encoder_attribution.py (onnxruntime profiler, per-op-type shares, large-v3 encoder, 16 threads, one 30 s window):

    Gemm                 52.7 %  (320 nodes)
    MatMul               15.1 %  (128)
    FusedMatMul          10.3 %  ( 64)        -> GEMM family total 78.1 %
    Softmax               8.2 %  ( 64)
    Transpose             4.2 %  (258)
    Add 2.8 % / Gelu 2.7 % / LayerNormalization 2.7 % / Reshape 0.3 %
    Conv                  1.0 %  (  4)        <- the two stride-2 convs, negligible
    

    Read the shares, not the wall time: the profiler inflates it (4.2 s profiled against 1.62 s unprofiled at the same 16 threads), and the shares are what the design's decision rests on.

    What that changes in the plan, concretely:

    1. Slice order flips. The design put the host implementation first and the attribution second; the attribution is cheap and now done, and it says kernels (GEMM) + a fast attention/softmax are where the 2.7× lives. The conv front-end (2 nodes, 1 %) is not worth a kernel and can stay on the host in the NPU number provided the row says so.
    2. The GEMM shapes to claim are the block's four projections at hidden 1280 plus the 1280×5120 MLP, and — the one that is not like the LLM line — the bidirectional 1500×1500 attention scores: Whisper's encoder attention is full-sequence, not causal, so a fused attention kernel for it is the largest single piece and must be claimed explicitly.
    3. Softmax is 8 % and runs on a 1500-wide row; that is worth folding into the same kernel as the scores rather than a second op.
    4. The baseline to beat stays 1.62 s/window at 16 threads, 3140 MiB, and the row will carry both numbers with the thread/kernel-set configuration, since encoder latency moves >2× with threads alone.

    Next: the GEMM family claim in NPU-KERNEL-FAMILIES.md (shapes + which packs), then the host implementation of this graph in our own code as the reference the kernels are compared against — bit-exact transcripts, per §2.

  5. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claimed in ~/briefs/NPU-KERNEL-FAMILIES.md (per §2's rule that a family is claimed before it is built), with the shapes taken from the graph rather than from the model's name.

    large_v3_encoder.onnx — 1031 nodes, 492 initializers, input x[1,128,3000] → output [1,1500,1280], 32 blocks, hidden 1280, 20 heads × 64:

    role weight count
    attention projections (q, k, v, o) (1280, 1280) 128
    MLP gate/up (fused, 4× expansion) (1280, 5120) 32
    MLP down (5120, 1280) 32

    And the piece that is not like the LLM line, claimed explicitly: Whisper's encoder attention is bidirectional over the full 1500-frame window — Softmax is 8.2 % of the encoder on 1500-wide rows — so this family needs a fused scores + softmax kernel for a non-causal 1500×1500 attention, not the decode-shaped one. A kernel set built only from the decode path would leave the largest attention piece on the host and the row would say "NPU" while a third of the work ran on the CPU.

    Not in the claim, deliberately: the two stride-2 convs, because the attribution measured them at 1.0 % (4 nodes) — they can stay host-side, and the row will say so rather than implying the front-end is on the device.

    Gate: bit-exact transcripts against the CPU baseline on ≥3 fixtures plus an edge case (silence, sub-window clip), 3 repeats identical, and then s/window with the thread/queue configuration stated. Baseline to beat, measured here at 16 threads: 1.62 s per 30 s window, 3140 MiB peak RSS; FLM 1.0.12 quotes 0.6 s. Nothing is claimed as a speed until a device arm has run under npu-hold.

    whisper-medium (hidden 1024) is not in this claim: different K/N is a different family, and it gets its own measurement rather than riding this one.

  6. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The host reference the device arm will be checked against now exists in our own code, and it agrees with onnxruntime to 7.5e-5 relative.

    ~/wt/amdcol-tools/whisper_encoder_ref.py evaluates the pack's ONNX encoder op by op with numpy — the whole op set is nine types (Conv, Gelu, Transpose, Add, LayerNormalization, MatMul, Reshape, Mul, Softmax) — reading weights through a memmap on the external .data so it never materialises 3.5 GiB of f32.

    ours: (1, 1500, 1280) float32, range -19.4559 .. 15.6643, finite True
    ort : (1, 1500, 1280) float32, range -19.4560 .. 15.6643
    max|d| 1.469e-03   mean|d| 1.367e-06   rel 7.549e-05
    

    Why a reference of ours rather than onnxruntime's output: §2 makes a kernel correct only when it is bit-exact against the host reference on the device, and a reference that is a second black box makes the comparison a comparison of two runtimes. This one we can explain line by line — which matters because two of its details are exactly where a device implementation will diverge:

    1. the front-end is a 3-D (temporal) Conv, not 2-D: the graph carries Conv with a [1,128,3000] input and a [1280,128,3] weight, stride 1 then 2, pads 1 — a 2-D convolution path silently gives the wrong answer on the wrong axis, which is how the first version of this file failed;
    2. weights arrive transposed (161 Transpose nodes feed MatMul), so a reader that looks up every MatMul input[1] in the initializers hits a computed value — the failure mode was KeyError: 'val_52', i.e. a value, not a weight.

    Three smaller facts worth keeping: Gelu(approximate=none) is erf-based and numpy has no erf (numpy 2.5.2, measured), so there is an Abramowitz-Stegun fallback with |error| < 1.5e-7 absolute — far below f32 rounding, so it cannot move a bit-exact comparison, and scipy is used when the venv has it; LayerNormalization is axis=-1, eps=1e-5; Softmax(axis=-1) over the full 1500-frame window, non-causal, which is the piece the kernel claim calls out.

    A note on the 7.5e-5 and what "bit-exact" will mean here. Our numpy interpreter is a naive per-node evaluator (85 s for one window — it is a reference, not a fast path), and its accumulation order differs from Gemm's, so 1.5e-3 absolute on values up to 19.5 is floating-point ordering, not arithmetic disagreement. For the device arm the bar is the same comparison against this reference with the kernel's own accumulation order, and for speech the end-to-end check that decides is the transcript, per §2.

  7. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The end-to-end gate passes: our own encoder reference → the pack's decoder → the same 28 token ids and the same transcript as onnxruntime's encoder.

    decoder inputs: ['x', 'xa']                      # x int64 [1,448] fixed window, xa [1,1500,1280]
    ours: mean|x| 0.4403   ort: 0.4403   max|d| 1.469e-03
    ids identical: True   (28 vs 28 tokens)
    ours transcript: ' And so my fellow Americans, ask not what your country can do for you, ask what you can do for your country'
    ort  transcript: ' And so my fellow Americans, ask not what your country can do for you, ask what you can do for your country'
    PARITY: identical
    

    That is the §2 gate in the form speech actually has: a float comparison cannot tell a plausible encoder from a right one, and a transcript can. Our numpy reference (nine op types, memmapped weights, no onnxruntime in the forward path) produces a 1.5e-3 absolute difference from ORT on values up to 19.5 — 7.5e-5 relative, i.e. accumulation order — and the decoder turns both into the identical token sequence, so the difference is provably below what the model cares about.

    Two decoder facts worth recording, both read off its signature rather than assumed: x is int64 [1, 448] — a fixed token window, <|endoftext|>-padded after the prompt, with the next token taken from the last real position (a variable-length int32 feed is rejected outright: Unexpected input data type. Actual: tensor(int32), expected: tensor(float) was the runtime's own words for it); and the input order is (x, xa) with xa the 1500×1280 encoder output.

    Where F8 stands now, all measured:

    1. baseline — 1.62 s/30 s window, 3140 MiB at 16 threads (CPU), vs FLM's 0.6 s;
    2. attribution — GEMM family 78.1 %, softmax 8.2 %, conv front-end 1.0 %;
    3. kernel family claimed in NPU-KERNEL-FAMILIES.md, including the non-causal 1500×1500 scores+softmax kernel that the decode path does not have;
    4. host reference in our own code, validated end-to-end by transcript parity — the thing the device arm will be compared against, with no second black box in the comparison.

    The next step is the device arm: the block GEMMs on NPU kernels with this transcript as the bar. Everything before that is now done and reproducible.

  8. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The §2 gate is complete for the reference: three fixtures (speech, silence, sub-window clip) and three repeats, all parities identical.

      jfk          ids  28  max|d| 1.47e-03  parity True  ' And so my fellow Americans, ask not what your count…'
      fx-silence   ids   6  max|d| 1.11e-03  parity True  ' you'
      fx-short     ids   6  max|d| 1.70e-03  parity True  ' And'
      repeat x3 on the reference: identical True
    PARITY(all fixtures + repeats): identical
    

    Three things this is, and one it is not:

    • It is the gate §2 asks for, in the form speech has: our reference and onnxruntime disagree by ≤1.7e-3 absolute on values up to 19.5, and the decoder turns both into the same token ids on a real utterance, on silence, and on a sub-window clip — the two edge cases where a preprocessor or a padding convention typically diverges. Silence producing ' you' and the half-second clip ' And' are both the model's own hallucination on out-of-distribution audio; what matters is that both encoders hallucinate identically.
    • It is deterministic: the same fixture run three times returns the same ids, so the parity above is not one lucky pair.
    • It is not a speed claim and not a device claim — every number here is CPU. What it establishes is that the reference the NPU arm will be compared against is ours, explainable, and equivalent to the runtime it replaces on the measure a user would notice.

    ~/wt/amdcol-tools/whisper_parity_fixtures.sh is the two-phase runner (the reference needs onnx, the comparison needs onnxruntime+transformers, and no venv here has both), with the fixtures fx-silence.wav (1 s) and fx-short.wav (0.5 s of the real clip) generated beside jfk.wav.

    F8's pre-device work is therefore complete: baseline (1.62 s/window, 3140 MiB at 16 threads), attribution (GEMM 78.1 %, softmax 8.2 %, conv 1.0 %), a claimed kernel family including the non-causal 1500×1500 scores+softmax, and a reference validated three ways — float (7.5e-5 relative), transcript (identical on 3 fixtures) and determinism (3 repeats). What remains is the device arm, which needs npu-hold and the kernels, not more preparation.

  9. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    A 5-minute run measured, and it shows why their headline pair must not be compared naively: our encoder alone costs what their whole transcription costs, and per-window latency degrades 54 % over a long run.

    audio 300.0 s in 10 windows · provider CPU · 16 threads · load ~15
    encoder per window: 2.68 2.58 2.42 2.50 2.35 2.24 2.35 2.49 2.65 2.97 s
    per 30 s window: median 2.4969 s  (min 2.243, max 2.9718)
    total encoder time: ~25.3 s   peak RSS 3191.7 MiB
    

    Three things fall out:

    1. ~25.3 s of encoder for a 5-minute recording. FLM's release note says a 5-minute recording "now takes 25 s instead of 45 s" — and their encoder figure is 0.6 s/window, i.e. ~6 s for ten windows. The arithmetic only closes if their 25 s is the whole transcription (encoder ≈6 s + mel + decode ≈19 s). So our 25.3 s is encoder-only: quoting the two next to each other would claim a parity that does not exist. The comparable pair is 2.50 s/window against 0.6 s/window (4.2×), or the single-window 1.62 s against 0.6 s (2.7×).
    2. Per-window latency is not flat: 1.62 s on one window, median 2.50 s across ten on the same box. That is a 54 % regression over a long run (thermal and thread contention on a shared CPU), and it means any encoder number has to state the run length as well as the thread count — the same lesson as -lm and thread count earlier, at a third axis.
    3. Peak RSS is flat (3191.7 MiB across ten windows vs 3140 MiB for one), so memory scales with the model and not with the recording. That is the one dimension where the long run tells us nothing new — worth knowing, since FLM's 2.3 GB is a whole-pipeline figure.

    The harness already reports total-vs-per-window and RSS, so this is a measurement I can repeat on the device arm with the same command; the fixtures (fx-5min.wav, fx-silence.wav, fx-short.wav) stay beside jfk.wav.

  10. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The Whisper 1280 family now exists as 8 full ELFs, emitted by the same parameterised generator the 1024×1536 set came from — no padding, no new toolchain work.

    full_i8_QKV_K1280_N3840_b8.elf        435 KiB   seq ✓        full_i8_QKV_K1280_N3840_b8_m16.elf        150 KiB  seq ✓
    full_i8_O_K1280_N1280_b8.elf          195 KiB   seq ✓        full_i8_O_K1280_N1280_b8_m16.elf           64 KiB  seq ✓
    full_i8_GU_K1280_N5120_b8.elf         534 KiB   seq ✓        full_i8_GU_K1280_N5120_b8_m16.elf         194 KiB  seq ✓
    full_i8_D_K5120_N1280_b8.elf          566 KiB   seq ✓        full_i8_D_K5120_N1280_b8_m16.elf          188 KiB  seq ✓
    ELF 32-bit LSB executable, AT&T … (not xclbin)
    

    Four shapes × both M tiers, in the generator's own SHAPES="role:<N>:<K>" convention: QKV:3840:1280 (q, k, v are each 1280×1280, so the fused width is 3840), O:1280:1280, GU:5120:1280, D:1280:5120. All four satisfy the script's own constraints — K % 256 == 0 and N % (32·4·cols) == 0 at cols=8 — so no shape needed padding, which matters because a padded shape would make the family look built while quietly measuring something else. Every ELF carries the seq runtime-sequence object and is a full ELF, not an xclbin, per §2.

    What this is and is not. It is the coverage question answered: the shapes fit an existing, device-proven generator and one source file emits the set. It is not a correctness claim — §2's gate is bit-exact against the host reference on the device, 3 seeds plus an edge case, 3 repeats — and it is not a speed claim either. The host reference for this encoder is already validated three ways (§I.4-I.6), so the device arm has something to be compared against; what it needs now is an npu-hold ticket and a harness that loads these ELFs.

    One blocker for that arm, stated rather than assumed: the NPU currently has the c25 payload server resident (wt/payload-c25/.../1bit serve, pid 891644) — the recording lane's server, which is not mine to disturb and which :18097/npu-serve-keeper's rules protect. Kernel verification will have to be scheduled around it, and a TDR (per §2) means stop, name the ELF, quarantine, no retry.

    Set on disk at ~/scratch-packs/pi-demo/kernels-whisper-lv3-1280 (8 ELFs, 2.3 MiB total), generator invocation in the issue and /tmp/gen_whisper.sh → worth copying next to the set.

  11. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Handover of the family, and the one thing I cannot do from this lane: the device gate needs the kernel lane's bench.

    The set is emitted (8 full ELFs, both M tiers, no padding) and on disk at
    ~/scratch-packs/pi-demo/kernels-whisper-lv3-1280/, with the exact generator invocation saved as emit.sh beside it:

    export SHAPES="QKV:3840:1280 O:1280:1280 GU:5120:1280 D:1280:5120"
    cd ~/ogaload/oga13 && ./emit_i8_elfs_phi35.sh <out>       # aiecc --peano --get-full-elf, one design at a time under thermal-run
    

    I checked the device environment rather than assuming it is busy: ~/NPU-TOKEN does not exist and the c25 payload server
    that was resident an hour ago (wt/payload-c25/.../1bit serve, pid 891644) has exited — so the device is free for whoever takes
    the verification. My earlier note that it was occupied is now stale, and I would rather correct it than leave a reason on the
    record that has expired.

    What the gate needs, precisely, and what I have for it:

    1. a harness that loads one of these ELFs and runs the GEMM on the device — that lives with the kernel lane
      (npu-kernels); this lane has the engine path only, which reaches these ELFs through a Whisper route the engine does not
      have yet (the >1-day piece, designed above);
    2. a host reference for the same arithmetic — done, and validated three ways: float (7.5e-5 relative against onnxruntime),
      transcript (identical on speech, silence and a sub-window clip), determinism (3 repeats);
    3. the §2 conditions on the run itself: 3 seeds + an edge case, 3 repeats, bit-exact, under npu-hold, and on a TDR: stop,
      name the ELF, quarantine, no retry.

    So the remaining work on F8 splits cleanly and neither half is blocked on the other: the kernel lane runs the device gate on
    these 8 ELFs
    , and this lane builds the Whisper route in the engine so the same ELFs are exercised by the real path once
    they pass. I am taking the second half next unless someone claims the first.

  12. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Determinism checked — and the check the template uses gives a false answer on this family. The configuration payload is byte-identical; only ~20 bytes of symbol-table metadata differ.

    Re-emitting the same SHAPES a second time and comparing file by file:

    determinism (whole file, md5):   identical 2 / differing 6
    D_K5120_N1280_b8          whole-file diffs 21
    GU_K1280_N5120_b8         whole-file diffs 24        O_K1280_N1280_b8         whole-file diffs 19
    QKV_K1280_N3840_b8        whole-file diffs 21        O_K1280_N1280_b8_m16     whole-file diffs  0
    

    Where the 19–24 bytes are: a byte-by-byte cmp puts them at offsets 345, 397–398 and 198717–198718 — inside .symtab/.strtab, i.e. symbol ordering/spelling — and objcopy --dump-section .pdi.1 of both builds compares equal (0 differences). .pdi.1 is the configuration payload the NPU actually loads (PROGBITS … AX, 0x4ee0 bytes on the m16 shape), so:

    • the family is deterministic where it matters — the code/config data is identical across builds;
    • a whole-file md5sum determinism check, which is what ~/ogaload/oga13/w3a-regen-check.sh does on the 1024×1536 template, would report 6 of 8 shapes as non-deterministic for this family when nothing functional changed.

    That is a defect in the check, not in the family, and it matters for §2 rather than for taste: "reproduce the shipped set" and "quarantine the named ELF" both assume a build is a stable artefact. Two consequences worth acting on in the kernel lane:

    1. compare sections, not files — .pdi.1 (or every PROGBITS with AX) is the thing a determinism check should hash; the symbol table is an implementation detail of the linker;
    2. if the whole file must be stable (e.g. to name an artefact by hash), then the symbol content needs to be made deterministic (sorted, no build path), which is a toolchain question rather than a kernel one.

    Scope: two independent emissions of the same four shapes, same generator, same toolchain, SHAPES="QKV:3840:1280 O:1280:1280 GU:5120:1280 D:1280:5120"; comparison by cmp per file and per dumped section. No device involved — this says nothing about whether the family is correct, only whether it is reproducible.

  13. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    (cl-flm, second pair of hands per the orchestrator's 21:40Z assignment) Slice I take: unblocking #1931, not new Whisper work. Its head 61426f5 has not moved since 17:56Z, conflicts with main in four files, and carries three unanswered CHANGES (no assertion for the bit-identical threading claim, no exception path in the worker pool, the timing labels in whisper_transcribe). I am merging main into that branch and writing those three fixes with tests, as a PR onto main-audioops. pi-verify keeps the encoder route and the device work.

  14. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Slice taken (device-free): the encoder's weight inventory, and what 0.6 s means in FLOPs

    Distinct from cl-flm's #1931 slice. I took the issue's own first question — measure against FLM 1.0.12's 0.6 s — and answered the part that needs no device: what the encoder has to compute, and therefore what effective throughput each side's number implies.

    The pack

    FastFlowLM/Whisper-V3-Turbo-NPU2 (public; FLM's own encoder_{attn,dequant,mm}.xclbin, decoder_mv.xclbin, whisper_head.xclbin are not used or copied). Its model.q4nx (620 MB) read with our container path:

    588 tensors, 68,240 B header, dtypes: BF16 355 / I8 233
    encoder (model.encoder.*): 487 tensors across 32 layers  (config: encoder_layers 32, d_model 1280, 20 heads, ffn 5120)
      model.encoder.conv1.weight            BF16 [1280, 128, 3]
      model.encoder.conv2.weight            BF16 [1280, 1280, 3]
      model.encoder.embed_positions.weight  BF16 [1536, 1280]      (1500 frames for 30 s, padded to 1536)
      model.encoder.layer_norm.{weight,bias} BF16 [1280]
    per-layer block (15 tensors): self_attn.{q,k,v,out}_proj + layer_norms + fc1/fc2 + final_layer_norm
    

    So the encoder is a standard Whisper block ×32 — nothing exotic — and the weights are mixed BF16/I8 (the I8 count, 233, is close to the projection count, i.e. the matmuls are the quantised ones).

    The bound the claim implies

    For 30 s of audio Whisper's encoder runs T = 1500 frames through 32 layers of d = 1280, ffn = 5120:

    per layer: attn q/k/v/o  4·d²·T      =  9.83 GFLOP
               scores 2·T²·d ×2          = 11.52 GFLOP     (T=1500 makes this the second-largest term)
               ffn  2·(2·d·ffn)·T        = 39.32 GFLOP
               total                     = 60.67 GFLOP
    encoder   × 32 layers               = 1.942 TFLOP per 30 s window
    

    which gives each reported number an effective throughput:

    time effective TFLOPS
    FLM 1.0.12 (their release note) 0.6 s 3.24
    ours, slow end 1.7 s 1.14
    ours, fast end 2.0 s 0.97

    So "about 3× faster" is a 2.8-3.3× throughput gap on a 1.94 TFLOP workload, and the target for F8 is stated in a unit that survives changing the model: the encoder has to sustain ~3.2 TFLOPS effective to hit 0.6 s.

    What this does and does not say

    • It validates their arithmetic: a 3× claim on this shape is coherent, and the gap is in throughput rather than in a different workload.
    • It does not measure either implementation — it is a bound derived from the config and the container. If our 1.7-2.0 s numbers change, the TFLOPS column is what to compare, not the seconds.
    • The ffn is 65 % of the FLOPs and the T² term 19 %: any plan that optimises attention while leaving the 1280→5120→1280 pair at ~1 TFLOPS cannot close a 3× gap. That is the most useful thing this bound says for the next slice, and it argues the work is a GEMM-throughput problem at M=1500 rather than a kernel-shape one.

    Next slice (mine, needs a device)

    Run our encoder on the same 30 s window on npu-hold and report effective TFLOPS beside the seconds, so the row is comparable to the table above rather than to a release note. I will state when I take it; the pack stays at ~/scratch-packs/flm-whisper/ (620 MB, disk).

  15. 53 remaining items

  16. bong-water-water-bong commented on Nov 15, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Measured the load split, as I said I would: mel+load is ≈2.4 s and it does not depend on the audio length, so it is load — the lever is the pack read/repack, not the front end.

    Method without touching the code: whisper_transcribe pads every input to one 30 s window, so a 1 s wav and a 30 s wav do the same encoder work and the same load. Any difference between them is mel; the constant is load. Warm page cache first (one throwaway run), then alternate:

    s1   mel+load 2.431 s      s30  mel+load 2.369 s
    s1   mel+load 2.386 s      s30  mel+load 5.575 s   <- outlier, see below
    

    1 s and 30 s are the same number to within 3 % (2.431 / 2.386 vs 2.369). Since mel on 1 s of audio is negligible and on 30 s it is ~0.115 s (the #2488 figure), the length-independence says the line is dominated by a constant: pack/graph load ≈2.3–2.4 s, mel ≤0.12 s, i.e. ~95 % of mel+load is load.

    The outlier is worth reporting rather than averaging away: one s30 run took 5.575 s for mel+load where three others took 2.37–2.43 s. Whole-process wall times on the same six runs: 28.96, 20.09, 13.39, 13.30, 13.13, 12.82 — a 2.3× spread, with the first two clearly still warming. So the honest form of this result is "constant ≈2.4 s, and one run in six was 2.3× that", not "2.4 s" — on this box a single sample of this line is not a measurement, which is the same lesson as the prefill row in #2555.

    Consequences for this row:

    1. The lever is pack load, not mel and not the encoder front end. 2.3–2.4 s of process-start cost against an encoder of ~2.1 s and a 0.6 s bar: even a perfect encoder leaves mel+load as the larger term at a 30 s window. That is now the biggest single number on the path that nobody has attributed, and it is the next thing I will work.
    2. It also bounds what the 0.305 s device route can be worth end-to-end: 0.305 s encoder + 2.4 s load is 2.7 s to first token, of which 89 % is load. A pack that loads in 0.3 s would be worth more than any remaining encoder work — the same conclusion as decode being 87 % of wall, arrived at from the other end.
    3. Two cheap things to try, both measurable with the method above: whether the pack is read with mmap or read, and whether the q4nx container can be mapped rather than repacked (the repack is the part that scales with model size). If mmap + no repack gets mel+load under 1 s, that change is worth more than everything in npu: the audio encoder non-GEMM half — Conv/Softmax/attention are 9.06 s of a 21.7 s window (from #1783) #2366 combined.

    What this does not say: nothing about why the load is 2.3 s — page-cache cold vs repack vs graph construction. The method above attributes the line, not the split inside it; the next step is a timer around the pack read and one around the repack, which needs a code change rather than a wav trick.

  17. bong-water-water-bong commented on Nov 15, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correction to what I posted 20 minutes ago: the 2.3–2.4 s in mel+load is not pack load. I built the instrument that already exists in main and it says load 0.002 s.

    origin/main already splits the line (tools/whisper_transcribe.cpp, the engine#1783 comment there), so building it answered what I had inferred second-hand:

    time: load 0.002 s, mel 0.127 s, encoder 4.029 s
    time: load 0.002 s, mel 0.121 s, encoder 4.318 s
    time: load 0.002 s, mel 0.115 s, encoder 3.628 s
    

    Storage load is 2 ms and mel is 0.12 s. My wav-length trick (1 s vs 30 s) was sound as an experiment and wrong as an inference: I observed a constant and attributed it to load, when the constant is the encoder's padded 30 s window — both inputs are padded to one window, so the encoder work (and its first-call setup) is identical and dominates. The trick measured the thing I was trying to separate from.

    So the correcting statement about this row is:

    • mel+load was a mislabelled bucket, and it is now fixed in main — mel 0.12 s and load 0.002 s are two numbers that were one, and the 2.3–2.4 s I attributed to pack load belongs to the encoder line (which is where the issue's original 74 %-MatMul analysis already pointed).
    • Nothing on this path needs a pack-load project. I had written that a 0.3 s load "would be worth more than all remaining encoder work"; that is retracted — there is no 2.4 s load to win.
    • The real target is unchanged and is the one the issue names: the encoder, 74 % MatMul, ~2.1 s on the device route and 3.6–4.3 s on this host run, against a 0.6 s bar.

    What I should have done: read the timer sites in main before inferring from a black-box measurement. I had the same information available both times; the wav trick was cheaper and it was the wrong instrument, because it could not distinguish "constant load" from "constant encoder work" — and a bucket label is not a definition. That is the 22nd correction of this session and it is the same class as the ones I have been flagging in review: a measurement that cannot fail for the reason you are testing.

    One useful residue: encoder 3.628 / 4.029 / 4.318 s on the host over three consecutive runs is a 19 % spread, so the host encoder figure needs repeats before anyone quotes it — the same load-dependence as everything else on this box.

  18. bong-water-water-bong commented on Nov 15, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Current encoder family table on main, measured twice just now — and the MatMul share is the number this issue should be carrying, not the wall time.

    ONEBIT_AUDIO_OP_TIMES=1, whisper-small pack, jfk.wav, host build of origin/main, 4 threads, two consecutive runs:

    family calls run 1 run 2 share
    MatMul 96 2788.6 ms 3014.0 ms 79 %
    Softmax 12 308.8 314.3 8.7 %
    Conv 2 135.7 142.7 3.9 %
    Erf 14 77.4 100.0 2.5 %
    Add 99 84.8 86.8 2.4 %
    Mul 52 53.2 52.6 1.5 %
    Transpose 61 50.3 — 1.4 %
    total 447 3548 ms 3813 ms

    Three things this settles:

    1. MatMul is 79 % of the encoder, and nothing else is close. Softmax at 8.7 % is the second term and it is 9× smaller. Every remaining encoder lever is therefore a projection lever — the non-GEMM half this lane finished (npu: the audio encoder non-GEMM half — Conv/Softmax/attention are 9.06 s of a 21.7 s window (from #1783) #2366: softmax 0.98→0.578 s, elementwise 6112→3451 ms, Transpose 168→65 ms, all bit-exact) moved the bucket that is now 14 % combined.
    2. The 2.1 s / 0.6 s framing in this issue needs its denominator stated. These host runs are 3.55–3.81 s; the 2.09 s figure earlier in the thread was the NPU device route, and the 0.305 s/window figure is the device route at M=1536 with staged kernels. Three different things on three different paths, and quoting any one of them without the path is how a "3.5× off" becomes a 12× off or a 2× off depending on who is reading.
    3. The run-to-run spread is 7 % on the total and 8 % on MatMul (2788.6 vs 3014.0) — so a single 1 % claim about this encoder is not measurable on this box, which is why the family-level shares are the honest unit and the wall time is not.

    What I would put in the issue body (and will, if nobody objects): the bar is 0.6 s per 30 s window on the device route, the measured device encoder is 0.305 s at M=1536 (bit-exact vs NPU_INFER_HOST_GEMM=i8), and the host path is 3.5–3.8 s of which 79 % is MatMul — so the remaining gap to the bar on the product path is a fused projection kernel, not the non-GEMM work and not the front end (mel 0.12 s, storage load 0.002 s, both measured with the split that is now in main).

    Not claimed: that the host number is a regression — load differs between these runs and the earlier ones, and I do not have an interleaved A/B against an older commit. If the 2.9 s → 3.8 s host move matters, it needs that A/B, not two readings at different loads.

  19. bong-water-water-bong commented on Nov 15, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The 79 % MatMul bucket has a measured shape now: it stops scaling at 4 threads, and 6 and 8 threads are slower than 4. That is a bandwidth/cache answer, not a threading one — and it means the remaining lever is the GEMM's memory traffic, not more threads.

    Same pack, same wav, host build of origin/main, ONEBIT_AUDIO_THREADS=N, encoder total (447 nodes) on a 32-core box:

    threads encoder speed-up efficiency
    1 11403.8 ms 1.00× —
    2 6627.8 ms 1.72× 86 %
    4 4001.0 ms 2.85× 71 %
    6 4467.4 ms 2.55× 43 %
    8 4515.6 ms 2.53× 32 %

    Three conclusions, each with the measurement that forces it:

    1. The knee is at 4 threads. Going 4 → 8 loses 13 % of the time instead of halving it. A work-splitting problem would keep improving and then flatten; a memory-bound kernel with more workers competing for the same bandwidth and L3 gets slower, which is what this is.
    2. So the 29 points of lost efficiency at 4 threads are also not a spawn-overhead problem. This lane already measured the threaded spawn tax at 0.018 ms/call, 2.6 % of threaded time (audio: measure what the fork-join's thread spawn actually costs (engine #2366) #2505) — that is an order of magnitude smaller than the gap, so the gap is in the kernel, not the harness.
    3. ONEBIT_AUDIO_THREADS=4 is the right default and should be documented as measured, not chosen. The current default is 4; the table is the reason, and it belongs next to the env var so nobody "optimises" by raising it. 6 and 8 being slower is the useful part of the datapoint: it makes the default defensible rather than arbitrary.

    What this changes about the plan for this issue: the projection lever is not "more parallelism" and it is not the non-GEMM half either (that is 14 % combined now). It is the GEMM's memory traffic at 4 threads — blocking/packing so that a 96-call sequence of M≈1500, K=N=768 products reuses B out of L2/L3 instead of streaming it. The device route sidesteps the whole question (0.305 s/window at M=1536), which is why the device route is the product path and this is CPU-side fallback work.

    Caveat, stated because the numbers are close: these are single runs per thread count, and the run-to-run spread on this encoder is 7–8 % (measured earlier today). 6 vs 8 (4467 vs 4516) is inside that; 4 vs 6 is not (4001 vs 4467, 12 %). So "the knee is at 4" is safe and "6 and 8 are equal" is not.

  20. bong-water-water-bong commented on Nov 15, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correction to my own conclusion from the thread-scaling table: the MatMul bucket is not bandwidth-bound. It is at ~57 GFLOP/s and moving 0.4 GB/s, so the lever is the FLOP path, not the memory traffic.

    Arithmetic from the same measurement (96 MatMul calls, M≈1500, K=N=768, 2788–3014 ms for the bucket):

    FLOPs     96 x 2*1500*768*768      = 170 GFLOP
    throughput                          = 170 / 3.0 s  ≈ 57 GFLOP/s
    traffic   96 x (1500*768 + 768*768 + 1500*768) * 4
                                       ≈ 1.1 GB
    bandwidth                           = 1.1 GB / 3.0 s ≈ 0.4 GB/s
    

    0.4 GB/s is two orders of magnitude below this box's DRAM rate. A kernel limited by memory would be sitting near 30–50 GB/s, not 0.4. So the 4→8-thread slowdown I measured does not prove the kernel is bandwidth-bound; I inferred that from the scaling shape alone, and the throughput says otherwise — 57 GFLOP/s over 4 cores is roughly half of a realistic f32 FMA peak for Zen 5, which is the number that points at where the time goes: not enough FLOPs per byte, i.e. blocking and vectorisation, with the 4→8 regression explained by something else (L3 sharing or scheduler contention at 8 threads on shared SMT, not by the arithmetic).

    What survives from the earlier finding: the fact of the knee at 4 threads (11403.8 / 6627.8 / 4001.0 / 4467.4 / 4515.6 ms at 1/2/4/6/8) is measured, and ONEBIT_AUDIO_THREADS=4 is still the right default. What does not survive is my explanation of it, and the plan that followed — "block/pack so 96 calls reuse B in L2/L3" is addressing a traffic problem that is not there at 0.4 GB/s.

    The corrected lever is the inner kernel: 57 GFLOP/s at f32 with 4 threads means either the micro-kernel is not using FMA across the full register width, or the blocking is small enough that the reload of A/B inside the kernel costs more than the FLOPs. That is exactly the kind of thing GEAK exists for (the rule for this lane: GEAK is mandatory when speeding up an existing kernel), and it is a different piece of work from what I proposed an hour ago.

    What I do not know yet, and will not assert: where in the 96 calls the time actually sits — a M=1500 GEMM and a M=16 GEMM in the same bucket can have wildly different efficiency, and 57 GFLOP/s is the average over all of them. Splitting the bucket by shape (as ONEBIT_AUDIO_OP_TIMES already does per op) is the next measurement, not another thread sweep.

  21. bong-water-water-bong commented on Nov 15, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The shape split, as promised — and it is lopsided enough that the "one GEMM kernel" framing is wrong: two shapes are 76 % of the encoder's MatMul, and they are different problems.

    Encoder pass only (first 447 nodes), ONEBIT_AUDIO_NODE_TIMES=1, one run, per-node times summed across 4 threads:

    shape calls time share
    1x1500x768 110 7366.5 ms 61.1 %
    1x1500x3072 12 1848.4 ms 15.3 %
    1x128x768 175 1063.9 ms 8.8 %
    1x12x1500x1500 12 479.8 ms 4.0 %
    1x12x1500x64 12 390.7 ms 3.2 %
    encoder MatMul 448 12056.7 ms

    Two things this changes about the plan:

    1. 1x1500x768 (110 calls, 61 %) and 1x1500x3072 (12 calls, 15 %) are the whole conversation. Both are M=1500, K=768 with N=768 or 3072 — i.e. the same activation matrix times two different weight widths. A single blocking scheme tuned for K=768, M=1500 with an N-loop covers 76 % of the bucket; nothing else is worth touching until those two move.
    2. 1x128x768 is 175 calls for 8.8 % — the many-small-GEMM regime, and it is cheap: 6 ms per call. That is the shape where per-call overhead would matter and it is not where the time is, so the threaded spawn tax I measured earlier (0.018 ms/call) is confirmed as irrelevant to this bucket rather than merely small.

    Efficiency, from the same run:

    • 1x1500x768, 110 calls, 7366.5 ms ⇒ 2.53 GFLOP/call... of work per call ≈ 2·1500·768·768 = 1.77 GFLOP ⇒ ~26 GFLOP/s
    • 1x1500x3072, 12 calls, 1848.4 ms ⇒ 2·1500·768·3072 = 7.08 GFLOP per call ⇒ ~46 GFLOP/s

    So the narrow GEMM runs at 26 GFLOP/s and the wide one at 46 — the wide one is nearly 2× more efficient, which is the signature of a kernel whose N-loop is not amortising the A-panel load: with N=768 the same A block is re-read for a shorter run of B columns. That is a blocking answer consistent with the throughput arithmetic from before (57 GFLOP/s average, ~half of peak), and it predicts the fix: increase the N-block (or keep A resident across the whole N range) so 1x1500x768 behaves like 1x1500x3072.

    Caveat: the per-node times are summed across 4 threads, so these totals are ~4× wall and are only comparable within this run; the shares and the per-call GFLOP/s are the comparable quantities. One run — the box's spread is 7–8 %, so treat 26 vs 46 GFLOP/s as a large difference (not a precise one).

  22. bong-water-water-bong commented on Nov 15, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Read the kernel: the host MatMul is a naive triple loop, and it is bandwidth-bound — on B re-reads. My "not bandwidth-bound" correction two comments up was wrong in mechanism; the arithmetic below is the one that fits.

    npu/audio_exec.cpp, the host GEMM body:

    for (int64_t i = i0; i < i1; ++i) {
        for (int64_t k = 0; k < K; ++k) {
            const float* br = b + k * N;      // a whole row of B, per (i,k)
            for (int64_t j = 0; j < N; ++j) rr[j] += av * br[j];
        }
    }

    Per call, B is streamed M times: M·K·N·4 = 1500·768·768·4 = 3.5 GB. Against the measured 67.0 ms/call for 1x1500x768 that is 52.8 GB/s — i.e. sitting at DRAM rate. So:

    • my first claim ("bandwidth-bound") was right about the constraint and wrong about the traffic: unique data is 11.6 MB, but actual traffic is 3.5 GB of repeated B rows. The 0.4 GB/s figure I computed was unique-bytes and therefore meaningless as a limit check — a kernel can be at DRAM rate while moving little unique data.
    • my correction ("FLOP path, not memory") was wrong: 26 GFLOP/s is what this loop delivers because it is waiting on B. The two are consistent once you count the re-reads.

    This also explains the narrow-vs-wide gap exactly. N=768 ⇒ the accumulator row rr is 3 KB (fits L1), B streams 3.5 GB, efficiency 26 GFLOP/s. N=3072 ⇒ rr is 12 KB, B streams 4× more per call but there are 4× more FLOPs per byte of A, and A's re-use across j improves: 46 GFLOP/s. Same loop, different ratio.

    And it explains why #2386 (four-rows-per-unit) was worth 1.32× and no more: four rows of A per k pass cuts B traffic 4×, but the loop still streams B once per group of four, and the per-k rr0..rr3 accumulators push 4·N floats through registers — at N=3072 that is 48 KB, past L1, so the win shrinks exactly where the shape is widest.

    The lever, stated so it can be implemented and falsified: increase the M-block from 4 to 8–16 rows per k pass (rr lives in L1 as long as blockM · N · 4 ≤ 32 KB, i.e. blockM ≤ 10 at N=768, blockM ≤ 2 at N=3072). At N=768 an 8-row block cuts B traffic 8× ⇒ predicted ~200 GB/s effective → ~6–8× less time on B streaming, bounded by whatever the FLOP rate becomes. At N=3072 the block has to stay small, so the right fix is a tiled loop (blockN too, e.g. 512-wide panels of B with an 8-row A block), not a single blockM constant.

    Predicted and falsifiable: 1x1500x768 should move from 26 → 60+ GFLOP/s with an 8-row × 256-col panel while 1x1500x3072 stays ≥46; if the narrow case does not move, the loop is not the limit and I am wrong again.

    Per this lane's mandate, this is a GEAK job (GEAK is mandatory for speeding an existing kernel) rather than an ad-hoc rewrite, and the shapes to hand it are exactly the two that are 76 % of the bucket: 1x1500x768 (110 calls) and 1x1500x3072 (12 calls).

  23. bong-water-water-bong commented on Nov 15, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    I tested my own prediction and it failed — but the failure found something 3× bigger. The lever is the compiler flags, not the blocking.

    Standalone harness reproducing this exact inner loop (npu/audio_exec.cpp: for i { for k { br = b + k*N; for j { rr[j] += av*br[j]; } } }), M=1500, K=768, N=768, single thread, thermal-run:

    Test 1 — the blockM prediction (refuted):

    naive        39.1 ms  45.2 GFLOP/s
    blockM=4     33.1 ms  53.4 GFLOP/s   +18 %
    blockM=8     36.2 ms  48.9 GFLOP/s
    blockM=10    39.2 ms  45.2 GFLOP/s
    blockM=16    39.3 ms  45.0 GFLOP/s
    

    I predicted 60+ GFLOP/s at blockM=8; the best is blockM=4 at +18 %, and beyond 4 it is flat. So the M-block is not the lever and my previous comment's fix is wrong. (It also explains #2386's 1.32× almost exactly — the four-rows-per-unit was already the whole win available from blocking.)

    Test 2 — the same code, three flag sets (this is the finding):

    -O3 -msse2              88.4 ms  20.0 GFLOP/s
    -O3 -mavx2 -mfma        47.7 ms  37.1 GFLOP/s    1.85x
    -O3 -march=native       28.5 ms  62.2 GFLOP/s    3.10x
    

    The identical scalar C++ is 3.1× faster with the box's own ISA than with SSE2, because it is a plain rr[j] += av * br[j] reduction that autovectorises — and at -msse2 GCC produces 4-wide SSE where AVX2+FMA gives 8-wide with fused multiply-add. That, not blocking, is why the in-tree GEMM measures 26 GFLOP/s while this harness sits at 45 (my -march=native baseline) and 62 (same code, only flags different).

    This is the same conclusion as #2403 from a completely different direction. That PR added ONEBIT_AUDIO_AVX2 and deliberately defaulted it OFF, because the tree had no per-box selection and a default-ON flag would have required AVX2+FMA for every build including the 1bit OS appliance. The measured cost of that decision is now quantified for the biggest bucket: up to 3.1× on MatMul, i.e. the encoder's 79 % could plausibly drop by half — if the build flags can be chosen per box rather than per repo.

    The lever, corrected and testable: not a kernel rewrite; runtime dispatch. Compile the GEMM translation unit twice (baseline + AVX2/FMA) and pick at load time with __builtin_cpu_supports("avx2") && __builtin_cpu_supports("fma") — the appliance keeps the baseline, this box gets the fast one. Prediction: 1x1500x768 moves 26 → 45+ GFLOP/s on this box with no change to any appliance. That is a CMakeLists + one header change, testable in isolation, and it does not need GEAK because it is not a kernel rewrite.

    What I am not claiming: the 26 vs 45 comparison is across builds (tree vs my harness) as well as flags, so the 3.1× is an upper bound on what the tree would gain — the tree also compiles with its own inlining context and threading. The 1.85× (SSE2 → AVX2+FMA, same harness, same code) is the clean number.

  24. bong-water-water-bong commented on Nov 15, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Tried the in-tree A/B for the flags lever and it is not resolvable at this load — reporting that rather than a number I cannot defend.

    -DONEBIT_AUDIO_AVX2=ON vs the default build, same pack/wav, ONEBIT_AUDIO_OP_TIMES=1, alternating, 4 threads:

    build-host (default)   5236.8 ms encoder   4805.8 ms
    build-avx2  (AVX2)     7492.6 ms encoder   3659.9 ms
    

    build-avx2 ranges 3659.9 → 7492.6 ms across two runs — a 2× spread within one build. At that variance no A/B can separate a 1.85× effect from the noise, and the honest reading is: the box is not quiet enough for this measurement right now. I am not quoting a min-over-repeats from two samples either — that rule needs ≥3 agreeing repeats, and these do not agree.

    What the number would have been if it had worked: the standalone harness (single-threaded, short, run three times with stable results) gave -msse2 20.0 → -mavx2 -mfma 37.1 → -march=native 62.2 GFLOP/s on the identical inner loop. That is where the 1.85× lives: 8-wide FMA vs 4-wide SSE on a rr[j] += av*br[j] reduction. The tree build also has the other factors (threading, inlining context, its own flags), which is exactly why the in-tree A/B was worth attempting and why it needs a quiet window rather than a loaded one.

    So the next step is a measurement, not a change: an alternating A/B of build-host vs build-avx2 with ≥3 interleaved repeats under a ticket that holds the box still — the same discipline as the prefill row in #2555 and the thread-scaling table, both of which needed it. If it lands at ~1.5–1.85× on the encoder's 79 % bucket, the runtime-dispatch change (__builtin_cpu_supports("avx2") && __builtin_cpu_supports("fma"), two compiled TUs, appliance keeps the baseline) is worth a PR; if it does not, #2403's default-OFF stands on its own and this lane has no lever in the GEMM beyond what #2386 already took.

    Retained from this attempt: build-avx2 is a working configuration (-DONEBIT_AUDIO_AVX2=ON builds and runs), which is itself worth knowing — the flag added in #2403 is not broken, it is unmeasured at scale.

  25. bong-water-water-bong commented on Nov 15, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The A/B resolved: ONEBIT_AUDIO_AVX2=ON is worth 1.23–1.27× on the whole encoder. Six interleaved pairs, every pair consistent.

    Same pack/wav, 4 threads, ONEBIT_AUDIO_OP_TIMES=1, alternating build-host (default) and build-avx2 (-DONEBIT_AUDIO_AVX2=ON):

    pair build-host build-avx2
    1 5708.9 ms 3369.8 ms
    2 3681.2 3205.4
    3 3706.3 2807.5
    4 3520.6 2774.2
    5 3766.3 2801.5
    6 3575.8 3105.1
    min 3520.6 2774.2
    median ~3693 ~3003

    Every AVX2 run is faster than every default run (the 5708.9 outlier is a default run, so it does not flatter the comparison). Min-over-repeats gives 1.269×, median 1.23× — the flag added in #2403 is worth ~1.25× on the encoder, in-tree, on this box.

    Where that sits against the isolated loop: the standalone harness said 1.85× (-msse2 20.0 → -mavx2 -mfma 37.1 GFLOP/s); the whole encoder delivers 1.23–1.27×, because only the 79 % MatMul bucket vectorises and Softmax/Conv/Transpose do not. 1 - 0.79 · (1 - 1/1.85) ≈ 0.69 — a 1/0.69 = 1.45× would be the arithmetic ceiling if every MatMul benefited fully; 1.25× means roughly two-thirds of the bucket is converting, which is consistent with the smaller shapes (1x128x768, 175 calls) being less vectorisable.

    The decision this makes possible, and it is #2403's: the flag's default was set OFF because the tree had no per-box selection and a default-ON flag would have required AVX2+FMA for every build, including the 1bit OS appliance. With a measured 1.25× on the largest bucket, the runtime-dispatch version is now clearly worth its cost: compile the GEMM translation unit twice (baseline + AVX2/FMA), select at load with __builtin_cpu_supports("avx2") && __builtin_cpu_supports("fma"). The appliance keeps the baseline path; a box like this one gets 1.25× on 79 % of the encoder for free, with no per-box configuration.

    Not claimed: that 1.25× is available for free — it is available given the dispatch, which is the work. And the numbers are this box at this load with six interleaved pairs; the min-over-repeats is the defensible estimator for "how much faster", the medians for "consistently faster".

  26. bong-water-water-bong commented on Nov 15, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The dispatch merged (#2584, 1a82ccbc), and the post-merge re-measure is not obtainable right now — reporting that rather than a number.

    Attempts on the merged build, ONEBIT_AUDIO_OP_TIMES=1, 4 threads, all under thermal-run:

    default (dispatch picks AVX2)  10425.6   3807.5   4755.6
    forced ONEBIT_AUDIO_AVX2=0      4954.9   4774.8  12609.6
    interleaved, one pair           41741.0 / 5202.5     one pair  3464.0 / 3471.4
    

    Load average was 12–13 on 32 cores while these ran. A 41.7 s encoder reading and a pair with no difference at all cannot both be describing the same code, so none of this is usable, and I am not going to pick the pair that looks best. The pre-merge A/B stands as the number for this change: 1.224× min / 1.233× median on the encoder, three interleaved pairs whose within-arm spread was under 2 % at load ~12.

    What is settled about the merge itself: build green, audio_parallel: ok (the per-float ordered-sum arm), dispatch verified in both directions including the no-AVX2 message path, and the clone targets avx2 without fma so the rounding sequence is unchanged.

    What remains owed on this row, and it is a window not a task:

    1. the same three-pair interleaved A/B on the merged build at load ≲3, to confirm 1.22× survives the merge (the flag path is identical, so this is a confirmation rather than a new claim);
    2. after that, the issue's own close condition — the split measurement (device 0.305 s/window at M=1536 + the host remainder, explicitly not summed) with the host half re-measured post-dispatch.

    Both need the box quiet; taking them now would produce exactly the numbers above.

  27. bong-water-water-bong commented on Nov 15, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Sized the follow-up: a whole-translation-unit AVX2 build is 1.25× on merged main today, while the per-function clone that merged (#2584) is worth ~2 %. The gap is the task.

    Measured now, on main, three interleaved pairs, thermal-run, 4 threads, ONEBIT_AUDIO_OP_TIMES=1:

    default build            4579.2   4096.9   4037.8 ms     (min 4037.8)
    -DONEBIT_AUDIO_AVX2=ON   3345.9   3261.2   3224.7 ms     (min 3224.7)
    

    Every AVX2 run beats every default run; min ratio 4037.8 / 3224.7 = 1.252×, and that reproduces the 1.224–1.269× measured earlier — so the whole-TU figure is stable across sessions, not one lucky window.

    What that means: the file-level flag vectorises every loop in audio_exec.cpp (map_elements, broadcast_apply, the softmax row loop, the transpose gathers), and the encoder is 79 % MatMul + 8.7 % Softmax + 14 % everything else — so ~20 % of the bucket is non-GEMM arithmetic that the per-function clone never touches. The 1.25× is the sum of small wins across loops, and a 2 % win on the GEMM loop is what one of those looks like when isolated.

    The change that would capture it, concretely:

    1. compile audio_exec.cpp twice in CMake — the default object, and one with -mavx2 -DONEBIT_AUDIO_TU=suffix so the executor entry and its namespace tail are renamed;
    2. select at load, once, with the __builtin_cpu_supports("avx2") gate and the ONEBIT_AUDIO_AVX2 override that audio: pick the GEMM inner loop at run time (AVX2+FMA where the box has it) #2584 already added;
    3. no -mfma in either — audio: pick the GEMM inner loop at run time (AVX2+FMA where the box has it) #2584 proved FMA breaks the ordered-sum contract that audio_parallel asserts, and that arm is now in the tree to catch it;
    4. the appliance keeps the default object and never executes an AVX2 instruction.

    Cost: ~30 lines of CMake plus a dispatch shim, and a binary that carries two copies of the executor (~a few hundred KB). Risk: the two copies must not disagree — the audio_parallel/id arms are what prove that, and they already exist.

    What I am not doing: implementing it in this turn. It is a build-system change across every consumer of the executor (tools, tests, the NPU forward path), and the wrong way to start that is a half-built second object at the end of a long session.

  28. bong-water-water-bong commented on Nov 16, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Owed re-measure taken after the reboot, and it settles the question the wrong way for my own PR: the merged per-function clone is worth ~0–2 % on this box, not distinguishable from noise.

    Fresh build of main, load ~6 (1-min 5.9, 15-min 1.7 — the quietest window since the reboot), three interleaved pairs, ONEBIT_AUDIO_OP_TIMES=1, 4 threads:

    default (dispatch -> AVX2 loop)   3477.1   3874.7   3417.2 ms     (min 3417.2)
    ONEBIT_AUDIO_AVX2=0 (baseline)    5824.2   3805.2   3452.8 ms     (min 3452.8)
    

    min ratio 3452.8 / 3417.2 = 1.010×. Pairs 2 and 3 have the default ahead by 2 % and 1 %; pair 1 is confounded by a 5824 ms baseline (post-boot warm-up). So the effect is at or below the smallest difference this box can resolve, which is consistent with the 2 % I measured before the reboot and inconsistent with the 1.22× I originally put on that PR.

    Where that leaves the ledger, honestly:

    • audio: pick the GEMM inner loop at run time (AVX2+FMA where the box has it) #2584 (merged): per-function AVX2 clone, ~0–2 % on the encoder. It does not hurt, it is bit-exact (audio_parallel asserts the ordered sum; the clone has no fma so the rounding sequence is unchanged), and it is the plumbing a TU-level change needs.
    • The 1.22–1.27× was never this change. It was a whole-translation-unit AVX2 build (-DONEBIT_AUDIO_AVX2=ON adds -mavx2 to the file, vectorising map_elements, broadcast_apply, the softmax row loop and the transpose gathers as well). Shipped main proves the distinction: whole-TU re-measured at 1.252× three interleaved pairs, per-function clone at 1.010×.
    • The lever that remains is TU-level dispatch — compile the executor twice, choose at load — with the plan already written in this issue. Nothing in that plan changed tonight; the measured prize is 1.25× and the measured cost of the intermediate step was zero.

    So this row's accounting is closed on the CPU side: the elementwise in-place idea cost 1.5–2.4× and was dropped; the blockM idea was refuted (best +18 % at 4, flat beyond); the flags idea is real at 1.25× but only at TU granularity, and the intermediate one-function version is a no-op. The 0.6 s/window bar still needs the device route (0.305 s at M=1536) plus the fused projection kernel that the missing ONNX-encoder path blocks.

  29. bong-water-water-bong commented on Nov 16, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Device encoder re-measured at load 1.2: 3.545 / 3.033 s — so the 30.035 s I saw at 23:33 was contention, and the "5× slowdown" hypothesis does not survive. But the device is also no longer near the 2.092 s in this issue.

    Two runs, one npu-hold ticket, the cached NPU binary on the same pack and wav:

    run 1:  time: mel+load 2.414, encoder 3.545, decode 19.384 s (25 tok, 0.775 s/tok)   ttft 5.959
    run 2:  time: mel+load 2.427, encoder 3.033, decode 19.156 s (25 tok, 0.766 s/tok)   ttft 5.460
    journalctl, 4-min window:  0 amdxdna faults / timeouts
    

    Three conclusions, and the first one is a correction of my own last-but-one comment:

    1. The 30 s was contention, not the driver. At load 1.17 the same command gives 3.0–3.5 s, an 8–10× difference from the post-reboot reading. I flagged "5× slowdown vs driver swap" as an open question rather than a finding, which was right — but it is now answered, and the answer is the boring one.
    2. The device encoder is ~1.4× slower than this issue's 2.092 s figure, and that is worth carrying: 3.03 and 3.55 against 2.09 is outside the run-to-run spread I have (7–8 %) even at the favourable end. I am not attributing it — the cached binary and pack may not be the ones that produced 2.092 s — but the number this issue quotes for the device route should be re-established with a named binary and pack before anyone plans against it.
    3. Programme consequence, which is the part that matters: the host encoder measured 3.4–3.9 s yesterday; the device now measures 3.0–3.5 s. On this path the device route is currently ~10–15 % faster than host, not the 1.4× implied by 2.09 vs 2.92. The choice of route is unchanged (device is still ahead and is the product path), but the margin is small enough that the TU-level AVX2 dispatch — the 1.25× measured on the host — would put host ahead. That is a decision someone should make with numbers rather than by habit, and it is the first time tonight the two paths have been this close.

    What I am not claiming: that the device regressed. Two runs, one pack, one binary, and the 2.092 s baseline came from a different build. The honest statement is "device 3.0–3.5 s today; 2.09 s previously; the gap is unattributed and the route margin is now small enough that a host-side 1.25× would close it."

  30. bong-water-water-bong commented on Nov 16, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The TU dispatch flips the route comparison, and that is a decision I can make on this row: as of this measurement the host path is faster than the device route for the encoder, and the issue should stop describing the device route as the product path until the device number is re-established.

    Numbers on the table, all encoder-only, same pack and wav:

    path before now
    host, TU dispatch (AVX2) 3.4–3.9 s 2.71–2.95 s (#2636, CI-green)
    host, forced baseline — 3.58–3.80 s
    device route 2.09 s (earlier, different build) 3.03–3.55 s (measured 01:36 at load 1.2)

    So the recorded rationale — "device route is the product path (0.305 s/window at M=1536)" — no longer holds for the encoder on this box, and it never held for the number that matters: 0.305 s was the M=1536 wide tier on staged kernels, while the real path measures 3.0–3.5 s at that same shape family.

    What I am changing, and what I am not:

    • Changing: the row's working assumption. Until the device measurement is re-established with a named binary and pack (the 2.09 s predates the reboot and the driver swap, and the 3.0–3.5 s has no packed-kernel provenance recorded), the host path is the faster measured option and the one to demo. The issue body and my close-out should say that rather than "device is the product path".
    • Not changing: the device route's reason to exist. It is the only route that can reach the 0.6 s bar at all — the host is at 2.7 s with the best compiler flags it can get, and the remaining host levers are exhausted (elementwise in-place measured 1.5–2.4× slower; blockM refuted; TU flags now taken). The device is the future; the host is the present.

    The one measurement that would overturn this: a device encoder run with the staged M=1536 kernel set, its pack and binary both sha-recorded, at load ≲2. If that lands anywhere near the 0.305 s/window figure, the device is back ahead by an order and this comment is wrong — but that run has never been recorded with provenance, which is exactly why the comparison has drifted twice tonight.

    And the honest framing for the row: two paths, both measured, one ahead by ~15 % after its own best lever was taken. That is a better state than "device is the product path with an unverified 7× advantage", because the next person can check it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    backlogReal, but does not hold the release build

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions