Repository navigation
Description
npu: the audio encoder's non-GEMM half — Conv, Softmax and the attention products are 9.06 s of a 21.7 s window
Split out of #1783 as its close condition says ("the sub-second half is a separate row"), with the measurements that make the scope of it exact. Nothing here is required for #1783's host chain, which is merged; this is the work that stands between the projections being fast and the encoder being fast.
What is measured (whisper-small, ONEBIT_AUDIO_TIMING=1, encoder 21.74 s, 447 nodes)
| family | nodes | shape | time | rate |
|---|---|---|---|---|
Conv |
2 | [768, 80, 3] stride 1, [768, 768, 3] stride 2 |
3.74 s | 6.4 GFLOP → 1.7 GFLOP/s |
Softmax |
12 | 1500 x 1500, axis -1 |
2.56 s | 27 M elements → ~10 M/s |
attention MatMul |
2 per layer | 1500x64 (scores·V) and 1500x1500 (Q·Kᵀ) |
2.76 s | — |
elementwise (Add/Mul/Erf/Div) |
175 | — | ~2.8 s | — |
The projections (O/GU/D) are 0.305 s/window on the device at M=1536 and are not part of this issue. For FLM's 0.6 s per 30 s window, these four families would have to fit in ≤0.29 s — they are 9.06 s, so ~30× too slow today.
The Conv number is the most interesting one: 1.7 GFLOP/s. The same array does 182 GFLOP/s on a D projection, and a plain im2col + GEMM on the host would beat 1.7 GFLOP/s by a wide margin without any device work at all. That suggests the cheapest first win here is host (im2col + the existing GEMM, and a blocked softmax), not a kernel family — worth measuring before anyone writes kernels.
What "done" looks like
Whichever route gets there, the measurable bar is the one #1783 uses:
npu-kernel-bitx.sh <container> <kernels> --repeats 3 # if kernels are involved: bit-exact vs the integer host arm
whisper_transcribe <pack> <wav> tests/data/whisper-jfk-ids.json
-> "transcript ids match the reference decode" and encoder <= 0.6 s per 30 s window at stated load
with ONEBIT_AUDIO_TIMING=1 naming which family got faster, so the before/after is per family and not a single encoder number.
Constraints that bound the design
- The widest launch on this part is M=128 (six array rows; the emitter's
rows <= colsrule and the placer'srow index must be < 6both bite above it) — npu: Whisper encoder on our runtime — measure against FLM 1.0.12's 0.6 s/30 s window and close the gap (W8/F8) #1783 measured this, and a wider tier needs-DQ4NX_DESIGN_M_PREFILL, which changes the engine's prefill M for every model. Convis not a GEMM shape the current generator can emit; im2col + theD/GUfamily is the shortest path to the device, and it changes the memory traffic story (an im2col of[768,80,3]over 3000 frames is a 5.5 MB matrix — it fits nowhere near the kernels' assumptions without blocking).- Everything here is own code (TNPU-ALL §2): a kernel goes to the private
npu-kernelsrepo and is claimed inNPU-KERNEL-FAMILIES.mdbefore it is built.
Why it is worth doing at all
The projections are now measured and fast (0.305 s/window at M=1536, from 2.05 s at M=128), and the host chain is merged. The encoder's remainder is what decides whether the row's 0.6 s is reachable, and it is dominated by two convolutions and a softmax — not by anything the kernel families cover. If the im2col+GEMM route lands at even 20% of the device's GEMM rate, Conv alone drops by ~20× and this issue becomes about Softmax.
Activity
bong-water-water-bong commentedon Nov 13, 2026
Correction to this issue's table — the numbers were pre-merge and pre-quiet, and the shape of the problem changes
I filed this with per-family numbers from a branch build measured at load ~40. Re-measured just now on
origin/main(which has the merged parallel Conv) with the instrument that actually ships,ONEBIT_AUDIO_OP_TIMES=1:Corrected table, replacing the one in the body:
ConvSoftmaxMatMulAdd/Mul/Erf/Div)What that changes about the issue: the remainder I am asking someone to attack is ≈4.1 s, not 9.06 s — Conv 0.75 + Softmax 0.98 + attention ≈1.7 + elementwise ≈1.4 — and the projections are the other ≈5.2 s of
MatMul, which the device route replaces at 0.305 s. So a full route lands the encoder near 3.9 s/window, and the 0.6 s bar needs that remainder cut by ~13×, not ~30×. Still a large factor, but a different problem: it is spread across four families rather than dominated by Conv.And the method point, which is mine to own: the instrument named in the body (
ONEBIT_AUDIO_TIMING=1, per node) never existed in the tree — it was a local, uncommitted diagnostic, so nobody could reproduce those numbers frommain.ONEBIT_AUDIO_OP_TIMES=1does ship, prints the block above, and is the command to quote from here on. A measurement whose instrument is not in the repo is not reproducible, and I filed one — this comment replaces it with one that is.bong-water-water-bong commentedon Nov 13, 2026
First measurement on this row, and it refutes the plan I filed: the attention products are not the lever, and k-blocking makes them worse.
The two attention
MatMuls, timed on their own with the executor's exact loop (k ascending, j inner) against a k-blocked-by-4 variant:Two things follow.
1. The executor's loop is already the right shape for these two. 16–18 GFLOP/s single-threaded for a K=64 and a K=1500 case, and the "obvious" improvement (four accumulator rows so the inner loop has ILP) is 3–4× slower, because it turns one streaming row into four and blows the cache for exactly these small-K/large-N and large-K/small-N shapes. That is worth knowing before anyone spends a day on blocking: the generic loop is not the problem here.
2. My own ≈1.7 s attribution was share arithmetic, and it is probably too high. 24 attention launches × ~16 ms ≈ 0.39 s single-threaded, and the executor runs four threads — so the real figure is somewhere at or below 0.4 s, not 1.7 s. I derived 1.7 s from "24 of the 96
MatMuls are attention" applied to the 6.87 s total, which assumes projections and attention run at the same rate. They do not: the projections are 768- and 3072-wide GEMMs (much better per-FLOP) and the attention products are the small-K shapes above.So the corrected remainder is smaller than I said in the correction, and the next step is not an optimisation. What this issue needs first is a per-node instrument (the one I wrongly claimed existed):
[audio-op]aggregates by op type, which cannot separate the 72 projections from the 24 attention products, and shape-share arithmetic is how I got 1.7 s for something that measures 0.4 s. Until that exists, every optimisation here is aimed at a number nobody has measured directly.Concretely, in order:
npu/audio_exec.cppunder an env var (the TODO I filed on this issue, and the thing whose absence produced the wrong attribution);Softmax(0.98 s measured, 12 launches of 1500×1500, ~82 ms each for 3 passes over 2.25 M elements ≈ memory-bound) or the elementwise set (1.42 s over 179 calls, ~8 ms each), not the attention MatMuls.bong-water-water-bong commentedon Nov 13, 2026
The instrument this issue needed exists now (#2377), and its first reading replaces my share arithmetic with per-shape numbers.
ONEBIT_AUDIO_NODE_TIMES=1prints one line per node with its output shape, so the singleMatMulrow splits:1x1500x7681x1500x30721x12x1500x15001x12x1500x64projections 16.38 s (53 %), attention 5.16 s (17 %), everything else 9.26 s (30 %) of a 30.80 s encoder at this box's load. The absolute numbers are load-dependent (the same encoder measured 10.36 s quiet); the ratio attention/projections = 0.31 is the stable part, and it is consistent with the 1.7 s / 5.2 s split I had guessed — so the guess was right in proportion and wrong in absolute size, which is what share arithmetic does.
Consequences for this issue:
audio_exec: read a stride-0 broadcast operand once per row (refs #2366)
bong-water-water-bong commentedon Nov 13, 2026
Measured negative result, so nobody spends a PR on it:
std::erffis not faster thanstd::erfhere.Erf is 14 calls / 365 ms of the encoder and the loop calls
std::erf(the double entry point) on float data, which looks like an obvious 2-3x for free. It is not:(1x1500x3072, 10 reps, min over reps so load cannot flatter either arm;
max |diff| 0because glibc rounds both to the same float.) So the Erf family's 365 ms is the polynomial's real cost, not a type promotion, and the only route left is an approximation — which changes results and would need the transcript-ids check to earn its place. Files that attempt it: none yet.Where the non-GEMM remainder stands after #2379 (merged) and #2383: elementwise ~3.4 s at this load (was ~6.0 s), softmax 0.58 s (was 0.98 s), Conv 0.75 s untouched and now the largest single non-GEMM node pair (the mel stem and the second conv, each 1x768x3000), Transpose 0.21 s (parallelised by #2382, keeps its per-element index dot product).
bong-water-water-bong commentedon Nov 13, 2026
Measured the host GEMM core, because Conv turned out to be GEMM-bound and so is everything left. The loop (
npu/audio_exec.cpp:339) is one output row at a time, i-k-j, so everyb[k]row streams once per output row — M passes over the whole weight matrix.Sharing the streamed row across four output rows (same operands, same order, so bit-identical):
(1x1500x768 projection — one whisper q/k/v/o, M=1500 K=768 N=768; best of 3, one process so load cancels.)
Two things this settles:
Not claiming this as a win until it is in situ under alternating A/B, which is the mistake I made twice tonight with loop-level numbers.
audio_exec: split the elementwise ops and Transpose over the executor…
audio_exec: split the elementwise ops and Transpose over the executor…
audio_exec: split the elementwise ops and Transpose over the executor…
pre-pr-checks: a stale branch read as a revert — the reviewer's side …
bong-water-water-bong commentedon Nov 14, 2026
A non-English fixture, because the bug that hid from me once is hidden exactly here.
When I fixed the special-token ids in #2183 I recorded the reason the defect survived testing: whisper-small hid it because
translateandtranscribeare the same operation on English. Everything I have measured since — softmax order, elementwise flat paths, GEMM row grouping, Transpose gather — has been certified against one English clip (jfk.wav, ids md591c656106060a). So I ran the same binary on a German clip, twice:13 tokens, identical across runs, on a path that English does not exercise. That is a second regression fixture and the first evidence that the four host changes hold outside the one clip that certified them.
The two edge fixtures behave sanely rather than well, and I am reporting them as observed rather than as correct:
A one-token output for a sub-second clip may be right (nothing to transcribe before EOT) or may be the decoder window being dominated by padding; I have no reference to compare against, so I am not calling it either way. What I can say is that both are deterministic and neither faults.
Ask for whoever owns the fixture set: these three belong in the repo's regression list with expected ids recorded, because a single English clip cannot distinguish "the maths is preserved" from "the maths is preserved on the one input where two different paths coincide" — which is precisely the shape of the defect that got past me before.
branch_stale_check: name the files a behind branch would write back o…
branch_stale_check: name the files a behind branch would write back o…
bong-water-water-bong commentedon Nov 14, 2026
Taking this row, and hitting a blocker before writing any code: the code these measurements come from is not on
main.So there is no
ONEBIT_AUDIO_TIMINGonmainand no audio encoder directory — only a Python baseline and the licence copy. The 21.74 s / 447-node window in the issue was produced by something else: an unmerged branch, a lane's worktree, or a tree that has since been reorganised.Could you point me at the branch or commit that produced that table? Concretely I need to know where these live:
Convnodes ([768, 80, 3]stride 1 and[768, 768, 3]stride 2) — 3.74 s at 1.7 GFLOP/s,Softmaxpasses over1500 x 1500— 2.56 s, ~10 M elements/s,ONEBIT_AUDIO_TIMING=1instrumentation that produced the per-family split.Why I am not just guessing at it: the worst rate in your table is the Softmax — 27 M elements in 2.56 s is ~10 M/s, which is roughly two orders of magnitude below a straightforward vectorised exp-and-accumulate pass, so it is almost certainly a naive per-element loop with a transcendental in it. That is a ~30-line change with a microbenchmark before and after, and it is the cheapest real win in the 9.06 s. But if I write it against a reimplementation on
mainrather than against the encoder that produced your numbers, I will have optimised a copy — which is the second-copy-of-a-rule defect in code form, and this row is too well measured to deserve that.Releasing my claim on the board until I can see the tree: anyone with the branch should take it, and I will pick it back up the moment the location is known. If the instrumentation is simply not landed, that is worth an issue of its own — a per-family timing split is the thing that made this row actionable in the first place.
bong-water-water-bong commentedon Nov 14, 2026
Closed a claim I made without evidence: "Conv is GEMM-bound, so it follows the projection row." Now measured. Mel-stem conv shape (C=768, T=3000, taps=3 → im2col 2304×3000, out 768×3000), single-threaded,
-O3 -mavx2 -mfma:So the claim holds, and by a wider margin than I would have guessed: the im2col copy is under 2 % of the conv. There is nothing to win from fusing or from a smarter im2col layout at this shape — the whole cost is 10.6 GFLOP through the GEMM, and that is the same core the projections use.
Two things this also settles:
Recording it because the honest version of my #2366 status is: of the six families, the four I optimised were chosen by measurement, Erf was measured to a dead end, and Conv was measured to have no separate lever — which leaves the projections' GEMM as the only remaining target, and that one is already covered.
29 remaining items
bong-water-water-bong commentedon Nov 16, 2026
Reversing my own correction an hour later: the LM head IS the decode block's entire MatMul cost. My "12 %" was reading an aggregate as a per-step figure.
Every MatMul in the 786-node decode block is the LM head. There are no "other 168 shapes" —
[audio-op] MatMul 169 callsis the tally across decode steps, not within one step. So:The mistake, named
[audio-op] MatMul 169 calls 520.8 msis printed per decode block in that table, and I divided 520 ms by 169 to get "2.7 ms each", then noticed 24 × 64 ms did not fit and concluded the LM head was 12 %. The 169 was blocks × 26. I used a number from one scope and a count from another, which is the same error as comparing a cold run to a warm one — the units matched and the scope did not.Where this leaves the row, with every number earned
ConvMatMulMatMulSo the honest target order is: (1) the encoder's 96
MatMulat 2.11 s — that is 71 % of the 2.96 s encoder and the only family over a second; (2) the 786-node per-step overhead in decode, which is where the 17 s actually goes; (3) the LM head last, at ~9 % of decode and already 2–4 % ahead of the host.I am not writing a fourth proposal. Three of the last four in this thread were built on a number that measured something else, and each correction was cheap only because the measurement existed. The encoder
MatMulshapes at M=1536 are the next thing to look at, andONEBIT_AUDIO_NODE_TIMES=1already prints them.bong-water-water-bong commentedon Nov 16, 2026
The encoder's projections are already at device peak: 177 GFLOP/s. There is no lever there, and I have to withdraw the target I named one comment ago.
Per-node shapes from a warm run (node index is per-block, so this is all blocks):
1x1500x768is M=1500, K=768, N=768 — 1500 · 768 · 768 = 885 M MAC = 1.77 GFLOP. Timing it:~145–205 GFLOP/s, against the 182 GFLOP/s the issue measured on a
Dprojection. The encoder'sMatMulis not slow; it is at peak, and its 2.11 s is simply the work: ~96–424 real GEMMs of 1.77 GFLOP each.So my "target 1" was wrong
One comment ago I wrote "the encoder's 96
MatMulat 2.11 s — that is 71 % of the 2.96 s encoder and the only family over a second" and implied it was the thing to attack. It is 71 % because it is 71 % of the arithmetic, and it is running at the hardware's own rate. Reducing it means doing fewer FLOPs (a different algorithm, a smaller model, or fusion that removes a projection), not a kernel.The arithmetic I should have done first: 2.11 s of
MatMul= 96 × 1.77 GFLOP ÷ 2.11 s ≈ 81 GFLOP/s aggregate, which is unremarkable but not pathological once the per-call spread is included — and it is bounded by the FLOP count, not by a 25× inefficiency like the Conv was.Where each family now stands, with the rate rather than the time
ConvMatMulSoftmaxGatherNothing in that table has a 25× gap left in it. The one that did —
Conv— was fixed before this thread started, which is why the original 21.74 s table no longer describes this box.What that means for engine#2366
The row's premise ("four families are 9.06 s of a 21.7 s window and ~30× too slow") is no longer the state of the code, and no remaining family is 30× off. If the FLM target of 0.6 s per 30 s window still matters, the lever is fewer FLOPs or fewer passes, not a faster kernel — and that is a design question for whoever owns the row, not a measurement I can settle with
ONEBIT_AUDIO_NODE_TIMES.I have now corrected this thread six times, and every correction came from a measurement rather than from re-reading my own prose. The last one is the useful kind: the answer is "no remaining inefficiency", which is a result, not a failure to find one.
bong-water-water-bong commentedon Nov 16, 2026
Completed the A/B on the device build: the
1x128x51865GEMV node is 71.7 / 59.8 ms there, against 81.8 ms on the host build — so the device is nominally ahead on this op, and both are round-trip-bound at ~350–440 MB/s.Same pack, same wav, one
npu-holdticket (load 3.6, die 52C, token free, no faults):Revised reading of the switch's first result. My earlier comment said the host looked competitive; with the device number in hand the honest statement is:
FORCE_HOSTis doing exactly what it was built for: it let a claim ("the device is better here") be tested instead of assumed, and the answer is "marginally, and it does not matter, because the read is the cost."What this makes actionable, and it is the fourth time tonight the same shape has appeared: a 1×128×51865 GEMV reads its whole 26.6 MB weight matrix per token for 512 B of activation. At M=1 the arithmetic is free, so the only lever is not re-reading the matrix per token — batch the LM head across the window (the encoder does 24 of these), or keep the head resident across calls. That is a caller-level change, not a kernel change, and it belongs on #2366 rather than here.
Not claimed: a repeatable ratio. The host figure is one quiet pair (the other pair read 239 ms under load) and the device figure is two runs on one packed kernel set; 1.1–1.4× is the range the data supports, not a number to plan against.
bong-water-water-bong commentedon Nov 16, 2026
The sibling path already solved this, and it is worth copying rather than reinventing:
npu/forward/src/q4nx_forward.cpp:7154-7166has the int8 head on the host by default (engine#1785), with anNPU_HEAD_HOST=altmode that alternates host and device per call so the A/B happens inside one process on identical inputs — plus a thread split for the host side (NPU_HEAD_HOST_…), and it is documented as "measured on two models with byte-identical results".That is three ideas the audio path does not yet have, and each maps onto what this issue needs:
altas the measurement mode.ONEBIT_AUDIO_FORCE_HOST(merged in audio: ONEBIT_AUDIO_FORCE_HOST forces the host path and names itself (engine#2366) #2647) is the coarse version: one process, one choice.altis strictly better for A/B — same process, same inputs, same cache state, alternating — which is the technique that produced the trustworthy numbers in this lane all evening (interleaved pairs at load ≲3).map-style loop over 51 865 rows; the sibling path splits the same work over a few threads. At 81.8 ms host vs 59.8 ms device, spreading 26.6 MB across 4 threads is the kind of change that could put host ahead of device on this node without touching a kernel.What I would do next on this row, in one line each: add
NPU_HEAD_HOST-style host-default +altto the audio LM head (mirroring #1785), measure with interleaved pairs, and only then decide whether the bigger lever — batching the encoder's 24 head calls — is needed. The batching fix is real but it is the second step; the first is to stop paying device round-trip costs on a shape the device cannot win.Not claimed: that host wins. 81.8 ms host vs 59.8–71.7 ms device is the measured range today, so host loses on the current single-thread loop — but the sibling path shows what closing that gap looks like, and it is a caller-level change this lane has not tried.
bong-water-water-bong commentedon Nov 16, 2026
Measured the head-shaped node with a per-call switch (mirroring
NPU_HEAD_HOST=alt), and the result says the divert target is wrong, not the idea. Reverted; nothing pushed.The switch worked — 26 head-shaped nodes detected on the encoder,
altalternating — and on a host build:Read carefully, this is host-vs-host, not host-vs-device. The binary is
-DONEBIT_NPU_HOST=ON, soHEAD_HOST=0already runs on the host — throughoffer(), which on a CPU-only build takes its own host fallback. What the three numbers compare is therefore two different host implementations, and the answer is that the executor's own host GEMM (197 ms) is 1.36× slower than whateveroffer()does on the host (145 ms).So the useful finding is a redirect, not a switch: diverting a head node to the audio executor's host loop makes it slower, and the reason to prefer
offer()'s path is presumably its threading (the head is onemap-style loop over 51 865 rows;NPU_HEAD_HOSTinq4nx_forwardsplits the same work over several threads — that is the piece this lane has not ported). Andaltlanding between the two is exactly what an alternating 50/50 mix should produce: (145+197)/2 ≈ 171 against the measured 132… which does not match, so the alt number deserves a second look before anyone quotes it (the counter is per call, so 13 of 26 diverted; the mismatch is either load drift between the three runs or the divert's cost not being what a mean suggests).What I would do next, one line each: port the thread split into the host head path first (that is the actual lever the sibling path has), then re-run this three-way comparison; a per-call switch is worth shipping only once the host arm is the good host arm, because a switch whose host branch is slower is a way to make the device look better than it is.
Not claiming: that the device is faster. The device numbers from the earlier A/B stand on their own (59.8–71.7 ms on the device build), and this run says nothing about them.
bong-water-water-bong commentedon Nov 16, 2026
Retracting the plan I posted 20 minutes ago: the head node is ALREADY threaded, 4.3× from 1→4 threads. So "port the thread split" is not the lever — it exists.
Measured on the current
mainbuild, same pack and wav,ONEBIT_AUDIO_NODE_TIMES=1, first1x128x51865node:Two things follow, and the second matters more:
parallel_units(batches * groups)would be ~1). The scaling says otherwise —batchesfor this node is not 1 (the encoder's head is issued over the window, so there are many batch indices), and the existing split covers it. The proposed change would have added a second parallelisation on top of an existing one.altvalue not matching the half-and-half prediction (132 vs the expected ~171) was the first sign of that. I quoted a 1.36× difference between two host paths that may differ by nothing at all.What survives: the device numbers stand (59.8–71.7 ms on the device build, 3033–3545 ms for the whole encoder), and the observation that the head is ~26 × 300 ms ≈ 7.8 s of summed per-node time on a ~3.5 s wall clock with 4 threads — i.e. it is a large fraction of the encoder and it is already parallel, so the remaining lever is not more threads.
Where I would look instead, in one line each: the node does one full pass over 26 MB per call with
M=1, andbatchesindices share the same weight matrix — so the lever is reusing the loaded weights across the 26 calls (keep the head resident, or batch the calls into one M=26 launch), which is the second item on my earlier list and the one I had ranked below threading. Ranking corrected.Not claiming: any number for the host-vs-device head comparison at this point — my own three-way measurement is contaminated by thread contention and load drift, and I am withdrawing the 1.36× rather than defending it.
bong-water-water-bong commentedon Nov 16, 2026
Checked whether the head's weights are re-decoded per call or already resident — they are resident — so the lever is neither threads nor decoding: it is the 26 separate passes over the same 26 MB.
npu/audio_exec.cpp:345-356:The first use materialises the head into
tand every later use returns that same tensor, so the 26 calls share one resident weight buffer. That closes the two cheaper explanations and leaves exactly one:So the only remaining lever is to make the graph issue ONE call instead of 26 — a batched head of
M = 26(or the window length) against the sameB. That is a graph-shape change, not a kernel change: it needs the 26 nodes to be recognised as the same projection over different rows, which is a property of how the exporter emits the head, not of this executor. It also happens to cut the read traffic 26× rather than 26→(26/threads).What I would do with this, concretely: the executor's
offer()already receives(A, M, K, B, N)withMas the row count — so a pre-pass that folds consecutiveMatMuls with an identicalBand equalK/Ninto a single call is expressible in the same code that builds the plan. That is the change I would attempt next, withaudio_paralleland the ids as the bit-exactness arms, and the note thatM=26must not change accumulation order per row (it does not — each row reads all ofB).Not claiming: that the fold is safe for every graph. A generic
MatMulfold changes the schedule, so it needs to be limited to nodes whoseAis not reused afterwards — thetake_if_lastmachinery already answers that question for the copy-elision work, and it is the right predicate to reuse here.bong-water-water-bong commentedon Nov 16, 2026
Measured the prize before writing the graph change: folding the 26 head calls into one M=26 GEMM is worth 4.45× at 4 threads on this shape, and only 1.20× at 1 thread.
Standalone harness — same
B(128 × 51 865 = 26.6 MB, does not fit L2), same work, 4 threads, one pass each:Why the thread count changes the gain so much, which is the part I did not expect: with one thread both arms are bandwidth-bound and the read is the same 692 MB either way — folding only saves the per-call loop overhead, hence 1.20×. With four threads each
M=1call still streams the whole 26.6 MB per thread-stride pattern, so the 26 single-row calls multiply contention; the folded call splits the 26 rows across threads and each row still reads all ofB, but the reads are far better interleaved. So the fold's value is not "read B once instead of 26 times" — it is still read 26 times, once per row — it is that 26 rows give the scheduler 26-way parallelism instead of 4-way over column blocks, and the memory system sees a steadier stream.Two consequences, and the second changes the plan:
What I would do next, in one line: implement the fold with the row-parallel framing above (a pre-pass that groups consecutive
MatMuls sharingB/K/N, guarded bytake_if_lastso a reusedAprevents it), and expect ~2–4× on the head rather than any large reduction in bytes — the[audio-node]line should then show one1x26x51865instead of 261x1x51865.Not claimed: that the executor currently issues
M=1per call. The node label says1x128x51865, i.e.(1, 128)activation against(128, 51865)weights — one row each — and the harness reproduces that shape exactly. If the real graph'sbatchesare already > 1, the measured 4.45× is an upper bound rather than a prediction.bong-water-water-bong commentedon Nov 16, 2026
Correction to what I posted 15 minutes ago: I measured the wrong shape. At the real shape the fold is worth 1.17×, not 4.45× — and my two earlier measurements only looked compatible because I had mis-read the node label.
The label.
[audio-node] 786 MatMul 1x128x51865isr.dims = bp.dims + [M, N](audio_exec.cpp:470-472), i.e. batch 1, M = 128, N = 51865. I read it asM = 1and ran the harness that way:And the mis-read is what made my earlier claim look wrong too. I had written "4.34× from 1→4 threads, therefore the node is parallel" and then "M=1 means one work unit, therefore it cannot be parallel" — an outright contradiction I noticed and did not resolve. With
M=128,parallel_units(batches × groups)=ceil(128/4)= 32 units, so the 4.34× scaling is exactly what the code predicts. One label read correctly would have settled both.What stands and what does not:
Bover 128 rows, so there is nothing like 4.45× available from folding — 1.17×, and that is the upper bound of a shape-only change.M=128run at ~300 ms when the harness's equivalent 26 × M=128 takes 1683 ms total (≈65 ms each). A 4.6× gap between the harness and the executor at the same shape — that is a much more interesting question than the fold, and it is the one I would look at next: the executor's per-node time includes the operand materialisation and anytake_if_lastcopies, and the harness has neither.The discipline note, for whoever reads this thread: three of my last four claims on this row were wrong in the same way — derived from a label or an inference rather than from the value. The two that survived were the ones I ran a harness for. The harness was right each time; the shape I fed it was the error.
bong-water-water-bong commentedon Nov 16, 2026
Tested the L2 hypothesis for the executor's 4.6× gap and it is wrong — 4-row blocking is faster here, 4-row 22.6 ms vs 1-row 38.0 ms at N=51865. So the gap is elsewhere, and I would stop guessing at it from outside.
Same shape as the head node (
M=128, K=128, N=51865, 4 threads,-march=native), varying rows per accumulator block:So the block being larger than L2 costs nothing measurable here — 8 rows per group is the fastest of the three — and my "830 KB accumulator thrashes L2" story is refuted by the simplest possible test. That is the fourth hypothesis of mine on this row to die to a harness, which is the correct base rate for guesses about a hot loop.
Where the 4.6× actually lives is now a list of things I cannot see from outside, and each needs an instrument rather than an argument:
-march=nativevs the TU's-mavx2without FMA — worth maybe 1.3× by this lane's own earlier measurement, not 4.6×.op_t0is taken before operand materialisation (init_f32,take_if_lastcopies) and the 786-node position means several such nodes run per step, some of the 300 ms may not be GEMM at all. The[audio-op] MatMul 96 calls … mstotal vs the sum of node times would settle it: if the two disagree, the node timer is measuring more than the GEMM.offer()being taken on some nodes and not others — on a host builddev == nullptrfor the audio path in the host configuration, butoffer()still exists; a node that goes throughoffer()and one that goes through the executor's own loop have different code.The cheapest next measurement, and the one I would do first: compare
[audio-op] MatMultotal against the sum of[audio-node]MatMul times in the same run. Agreement ⇒ the 300 ms is arithmetic and the gap is the kernel; disagreement ⇒ the node timer includes work the op timer attributes elsewhere, and the "4.6×" is a measurement artefact — which after tonight is the outcome I would bet on.Not claimed: any explanation for the gap. Three of mine have been refuted (threading missing, weights re-decoded, L2 thrash); the honest state is "the executor is 4.6× slower than an equivalent standalone kernel at this shape, cause unidentified", and that is a perfectly good bug report for whoever has profiler time.
bong-water-water-bong commentedon Nov 16, 2026
Found it — and it is not the fold, not the threads, and not the kernel. The head is preceded by a
Transposeof its own weight, and that copy costs 101 ms per call:101 + 193 ≈ 294 ms, which is the "~300 ms per head node" I have been quoting and mis-attributing to the MatMul since this morning. The MatMul is 193 ms; a third of the cost is a 159 MB weight copy immediately before it (
768 × 51865 × 4 B), repeated for every one of the 26 calls.That reframes everything I have written on this row tonight:
-mavx2-without-FMA budget far better than 300 ms did;Bin the transposed layout so the copy never happens. At 101 ms × 26 that is ~2.6 s of summed time on a 3.5 s wall clock — the largest single item I have found in this encoder, and it is pure redundant copying.And the L2/threading/scheduling questions I chased all afternoon were beside the point, because the node I should have looked at was the one before the MatMul. The
[audio-node]instrument has been printing the answer every run: a 101 msTransposesitting immediately above a 193 msMatMulon the same 51865-wide tensor.What I would do next, in order: (1) confirm the transpose's operand is the head initializer and that it is the same initializer on every call —
tcaches by name, so a hoisted result would be reused; (2) hoist it (atake_if_last-style cached transposed copy, computed once), measure withONEBIT_AUDIO_OP_TIMES; (3) only then reconsider batching. The bit-exactness arms are unchanged:audio_parallelplus the ids.Not claimed: that the transpose is removable — a
MatMulwhoseBis stored[N,K]needs either the copy or a kernel that reads it that way, and which of those applies depends on the ONNX export. But the repetition is certainly removable, and that is 25 of the 26 copies.bong-water-water-bong commentedon Nov 16, 2026
Tried the hoist and it does not help as written: the transpose happens once per graph execution, not 26 times inside one. Reverted. The real change has to cache across executions, which is a different (larger) one.
What I built: cache the transposed initializer keyed by
(input name, perm)insideexecute_graph, which is wheretalready caches initializers by name. Result:Why it was never going to fire: the 26 head calls are 26 separate graph executions —
whisper_transcriberuns the decoder 25 times after the encoder — and my cache lives for the duration of oneexecute_graphcall. Within one execution eachTransposenode appears once, so there is nothing to hit. The 101 ms I measured is per execution, i.e. ~101 ms × 26 passes ≈ 2.6 s, exactly as reported, and removing it needs a cache that outlives a single pass.So the next change, stated precisely: the cache belongs on whatever object holds the parsed model — the structure that already caches decoded initializers across calls (
init_f32'stis per-execution; the per-call cost of decoding is avoided becausewhisper_transcribere-enters with the same graph info, but the transpose result has no such home). Concretely: amutable std::map<std::string, Tensor>onOnnxGraphInfoor on the audio pack handle, keyed the same way, consulted before the transpose loop. That is ~10 lines in the same place plus a lifetime decision, and it is worth ~2.6 s of a 3.5 s wall clock.What this attempt did establish, cheaply:
audio_parallelok, ids identical), which is the same statement the eventual hoist will need;Transposebranch, withinits.findsaying whether the operand is an initializer.Not claiming the numbers beyond that: a cross-execution cache changes memory (159 MB held for the model's lifetime), which is a real trade against a 3.5 s encoder — worth stating in the PR that does it, and worth checking against the pack's own budget on a 4 GB NPU device before it lands.
bong-water-water-bong commentedon Nov 16, 2026
Caching the transposed weight across executions does not help either — and the reason is decisive: every node output is copied out of the cache anyway, so the 159 MB copy is the cost, not the computation. Reverted.
Built it properly this time (cache on
OnnxGraphInfo, keyed by initializer name + perm, so it survives the 25 decoder passes):Two things went wrong, and the first is the important one:
r.data = cv->secondcopies the cached vector — the executor's node contract is that each node produces its ownr, so a cache saves only the index arithmetic, never the 159 MB copy. That copy is 101 ms; the cache therefore saves nothing and adds a second 159 MB resident.So the conclusion is sharper than "cache it": the transpose is expensive because the tensor has to be materialised, not because computing it is expensive. The only fixes that can work are the ones that avoid materialising it:
Bin[K,N]while the export gives[N,K], so either the reader or the GEMM must change), orBtransposed (indexb[k + j*K]rather thanb[k*N + j]), which removes the copy and keeps the layout — the standard answer, and the change that actually addresses 101 ms × 26.Third dead end on this row tonight, all honest: threading (already there), folding (1.17× because the real cost was elsewhere), caching the transpose (saves the arithmetic, not the copy). The measurement that matters stands — 101 ms of pure copy per pass, ~2.6 s per transcription — and the remaining fix is a GEMM index change, which is a kernel change and so falls under GEAK/TraceLens rather than this loop.
Not claiming any of tonight's three hypotheses as likely; the base rate on my guesses about this loop is now 0 for 5.
bong-water-water-bong commentedon Nov 16, 2026
Correcting my own last suggestion before anyone acts on it: "teach the GEMM to read
Btransposed" is the wrong fix, and the reason is why the transpose exists at all.The head's weight comes from the export as
[N, K]=[51865, 768]. TheTransposemakes it[K, N]=[768, 51865], which is what the GEMM's inner loop wants:Reading the untransposed
[N,K]instead would make that inner loop read stride K (b[j*K + k]), i.e. one float per 3 KB — a cache line per element. The 101 ms copy buys a great deal more than it costs, which is why the transpose is in the export in the first place. My earlier suggestion would have replaced a 101 ms copy with a several-hundred-ms strided GEMM.So the only two fixes that can work are the ones that materialise the
[K,N]layout once for the model and then read it without copying:Transposenode at all (either the loader rewrites the node away, or the export is asked for the pre-transposed form). No per-call copy, no second resident copy, and nothing to keep in sync.Tensorholdsstd::vector<float>, so this needs ashared_ptr-backed buffer in the executor. That is the same lifetime changeOnnxModeljust got in npu-pack: OnnxModel owns the graph mapping, so OnnxTensor::raw outlives the parse (fixes #2668) #2670 for the graph mapping, applied to tensors; it is the general fix and it is bigger.And a third that needs no memory at all: if the 26 passes share the same
[K,N]result, then a cache plus a shared buffer is (1) and (2) together — one 159 MB copy for the model, referenced 26 times. The trade is 159 MB of host RAM against ~2.6 s per transcription, which on this box is obviously worth it and on a 4 GB device is irrelevant because this is host-side.What I am not doing: writing it now. Options (1) and (2) both change a lifetime or a loader contract, and tonight's five dead ends on this loop are the argument for stopping at the decision rather than starting a sixth attempt at the end of a session. The measured facts are in the thread — 101 ms per pass, 26 passes,
audio_paralleland the ids as the arms — and the next attempt should start from option (1), which needs no allocator change.Base rate, kept honest: five hypotheses on this loop, zero correct — but the measurement that closed it (
Transpose101 ms besideMatMul193 ms) came from a log line already in hand, and it is worth more than all five guesses.bong-water-water-bong commentedon Nov 16, 2026
And option (1) as I framed it also saves nothing — worked through, it collapses to "the packer must store the weight pre-transposed". That is the decision I am recording, because it puts the fix in
npu-pack, not in this executor.The chain, one step at a time:
tat the start ofexecute_graph) still does the transpose 26 times, once per pass — no gain;Tensorowns astd::vector<float>, sot[name] = cachedcopies 159 MB per pass. That is what my two reverted attempts measured;Tensor(ashared_ptrview over the cached buffer) removes the copy — but it changes tensor ownership across the whole executor, which is a project, not a patch, and it makes everyTensora potential alias of a cached weight (exactly the hazardtake_if_lastexists to police).Whereas if the pack stores the head weight already
[K, N], there is noTransposenode in the graph at all: no copy, no cache, no lifetime question, nothing to keep in sync. 26 × 101 ms → 0, and the exporter's own layout is the thing that changed.So the row's decision: this is a
npu-packtask (emit/convert the head initializer with the GEMM's layout, and let the graph drop the transposing node), not anaudio_execoptimisation. The executor's[audio-node]line is how the packer's author verifies it: node 785 should simply disappear.What this costs the next owner, honestly: the packer change has to know which
Transposenodes are weight-layout transposes (operand is an initializer, perm is a plain rank-2 reversal) and rewrite both the tensor and the consumer's expectation — plus a pack-format compatibility question for packs already shipped with the node. That is a real piece of work with a real compatibility decision, which is why it does not belong at the end of this session.Everything measured stands:
Transpose 768x51865costs 101 ms beside a 193 msMatMul, once per graph execution, 26 executions per transcription, ≈2.6 s of a ≈3.5 s wall clock. Arms for whoever moves it:audio_parallel(ordered-sum bit-exactness) and the transcript ids (91c6561060…).And the honest scoreboard for this row tonight: six hypotheses, zero correct — the two that produced real information were both measurements I ran, and the one that closed it was a log line that had been printing the answer all along.