You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
npu: the host audio GEMM core runs at 12.8 GFLOP/s (~25 % of single-pass peak) — packed/blocked micro-kernel #2399
Scope note first: engine#2366 is the encoder's non-GEMM half (Conv/Softmax/attention/elementwise). The projections are the other 61 % of the encoder, so the lever measured while closing #2366 has no home — filing it here rather than leaving it in a comment.
Where it stands, measured
The host GEMM core (npu/audio_exec.cpp, the MatMul branch) accumulates with one output row per parallel unit in i-k-j order, no packing. After engine#2386 (four rows per unit) it runs at:
one row 189.8 ms (9.3 GFLOP/s) four rows 138.7 ms (12.8 GFLOP/s) 1.35x, max|diff| 0
and in situ on the whisper-small encoder, alternating the two binaries under one load: MatMul 3367/3283 ms → 2518/2530 ms (1.32x), ids md5 unchanged.
12.8 GFLOP/s is roughly 25 % of what four AVX-512 cores do in a single pass. The measured reason the cheap fix stopped there: sharing the streamed weight row across four output rows removes the traffic, but then the four live accumulators become the limit — a memory-traffic argument predicted 3x and the hardware gave 1.35x. Going further needs packing/blocking: a packed micro-kernel (B panel packed once per block, k-blocked, register tiles) rather than the current rank-1 updates over a streamed row.
What "done" would look like
Bit-exactness: transcript ids md5 91c6561060 on jfk.wav and the German fixture unchanged; audio_parallel_test green at ONEBIT_AUDIO_THREADS=1,2,4 (it compares each thread count against an ordered serial sum, and test_row_group_edges covers M<4 and the all-zero group).
Speed: ≥2x over the current core, stated as an alternating A/B under one load. Non-negotiable: four consecutive runs of the same binary on this box have spanned 2.8–7.6 s as load moved, so a cross-window comparison can "prove" either direction.
Instrumentation already exists: ONEBIT_AUDIO_NODE_TIMES=1 (engine#2377) prints one line per node with its output shape, so the per-shape split (96 MatMuls, 1x1500x768 ×60 for q/k/v/o+fc2, 1x1500x3072 ×12 for fc1) needs no new tooling.
Why it is worth a row at all
The device route replaces the projections at 0.305 s/window (engine#1783), so the host GEMM is not the only path to closing that row — but the device route needs the ONNX-encoder route that does not exist, while this is a contained host change with an existing bit-exactness harness. It also benefits Conv, which is GEMM-bound (2 calls, 343 ms of the encoder, and it goes through this same core).
Claiming this lane (pi-i679). I reviewed both halves that led here -- #2386's four-rows-per-unit and #2389's row-at-a-time Transpose -- so the measurement chain is familiar, and this issue names the lever both of them stopped short of.
What I will try. The issue's own arithmetic puts the ceiling at four AVX-512 cores in a single pass, so the target is a packed/blocked B rather than another loop-shape tweak: pack B into k-blocks so a block of b[k] rows is read once and used by every output row in the unit, with the packed layout chosen so the inner loop is contiguous on both sides. The two prior changes fixed the streaming and left the access pattern alone; this one is about the access pattern.
The arms I will hold myself to, because the last two changes in this file set the bar and the issue already states them:
bit-exactness: after audio_exec: four output rows per GEMM unit, 1.32x on the projection family (engine#2366) #2386 the outputs are byte-identical by construction ("same reads, same writes, same order"). A packed B changes the order of accumulation unless the k-blocking is done carefully, so I will treat identical ids md5 91c6561060 as the goal and prove it rather than assume it. If a packed path cannot be bit-identical, that is a finding to report, not a rounding error to wave through.
No device needed to develop: this is npu/audio_exec.cpp's host branch, so the work is CPU-side under thermal-run; the only device question is whether the whisper ids still match, which wants one window at the end rather than a ticket to develop in.
This issue's premise is wrong, and the measurement that corrects it is worth more than the micro-kernel it asked for. Two findings, both reproduced twice, same source, one process, load could only hurt.
1. The same kernel runs at 9.8 or 69 GFLOP/s depending only on compiler flags
-O2 (what the tree uses) cur4 180.0 ms 9.8 GFLOP/s
-O2 -mavx2 -mfma cur4 135.9 ms 13.0 GFLOP/s
-O3 cur4 47.2 ms 37.5 GFLOP/s
-O3 -mavx2 -mfma cur4 27.5 ms 64.3 GFLOP/s
-O3 -march=native cur4 25.5 ms 69.3 GFLOP/s
7× from the flag line alone, on the exact loop I proposed to replace. The 12.8 GFLOP/s figure in the issue was the in-situ -O2-built binary, so it was never measuring the algorithm — it was measuring that the vectoriser was not asked to do FMA. A packed/blocked rewrite of a core that is being compiled without FMA would have been optimising the wrong thing, and I would have spent a project finding that out.
2. The packed prototype I built first is worse, not better
cur4 25.1 ms (70.5 GFLOP/s) packed 180.6 ms (9.8 GFLOP/s) 0.14x max|diff| 0
The naive panel copy (per column block, per k block) costs more than the register tiling saves at this size — 1500×768×768, where B entirely fits in L2 anyway. So at this shape the win is not available from packing; it may be at much larger K/N, but that is not this workload.
What this means for the row
Cheapest real lever, one line, no algorithm change: build the audio TUs with -O3 and FMA. -O3 alone is 3.8x over -O2 and needs no -march contract; -O3 -mavx2 -mfma is 6.6x and states the ISA explicitly (fleet is AVX2/FMA everywhere since Zen; -march=native adds 7 % and a distributability problem, so I would take -mavx2 -mfma).
Correctness cost: none expected — same source, same order in k, and FMA changes rounding. So the bar is the one already written into this issue: ids md5 91c6561060 on jfk and the German fixture, audio_parallel_test green at 1/2/4 threads. FMA fusing is exactly the kind of change that can move a last bit, so it must be checked, not assumed.
A packed micro-kernel is NOT justified at this shape by the above, and this issue should say so rather than carry an unexplored plan.
I will bring the flag change as a PR with the alternating-A/B in-situ number (4 threads, ONEBIT_AUDIO_NODE_TIMES=1 to keep the per-shape split honest) and the ids check, unless the build flags are deliberately -O2 for a reason I have not found — in which case that reason belongs in this thread before I touch CMakeLists.txt.
First measurements on #2399: a baseline, one dead hypothesis, and an instrument that turned out to be inert.
The instrument works and is cheap: whisper_transcribe on whisper-small + jfk.wav, host-only (ONEBIT_HRX=OFF, NPU runtime from a lane cache), no device and no window. It reproduces the reference ids exactly (400 370 452 7177 6280 ... 1941 13), so it is the same path the issue measured.
Baseline, 4 threads, three reps:
encoder 3.989 / 4.023 / 3.994 s median 3.994 s
Node split from ONEBIT_AUDIO_NODE_TIMES=1 over 20,883 nodes (whole run: 4.4 s encoder + 27 s decode):
node
share
MatMul
78.6 % (24.4 s of 31.05 s, n=4490)
Transpose
8.8 %
Softmax
3.8 %
Add
2.0 %
So the encoder's ~3.1 s of MatMul agrees with the issue's 3.3 s, and MatMul is the whole story.
__restrict was my first hypothesis and it is noise.ap, bpp and r0 are separate tensors, so the promise is true -- and A/B/A with three reps each gives baseline 3.994 s against 4.047 s, i.e. 1.3 % slower. Reverted. Two things worth recording from that: the effect I first saw (4.438 -> 4.080) was a cold first rep, 11 % above warm -- the same effect that turned item 3a's "2.0x" into 7.5 %, now on the host side; and one run is not a measurement even of a four-second loop.
Where the headroom is not. After #2386 the inner loop is already contiguous on both sides (br = bpp + k*N, and four contiguous r0[...] streams), B is 2.25 MB streamed once per k-block, and the four output streams are 12 KB and fit L1. So it is not cache and not access order. perf stat on the run gives IPC 4.63 (426 G cycles / 2,052 G instructions), which is well-vectorised -- so the 25 % is not a codegen failure either.
And the FLOP counter I wanted is inert on this box.fp_ret_sse_avx_ops.packed_single_flops reads 0 for the whisper run, and the control -- numpy 1024x1024 float32 matmul, certainly packed SIMD -- also reads 0. So that counter cannot be used here, and the GFLOP/s figures have to stay derived from shapes and wall time, as the issue does it.
Next, in order: check the generated code for packed FMA in the MatMul inner loop (-fopt-info-vec on the file, or the disassembly) rather than guessing from IPC -- IPC 4.63 is consistent with scalar loads and integer work as well as with vector FP. If the loop is not vectorised, the fix is a j-blocked variant with an explicit vector width (the packed B the issue proposes is one way, but if the access pattern is already contiguous, an explicit 16-wide FMA loop over j is the smaller change and the same 4-stream shape). If it is vectorised, then the 25 % is a bandwidth limit on four read-modify-write streams and packing will not help -- which would make this issue's title the wrong lever, and that is worth knowing before anyone spends a day on it.
The lever in the title is the wrong one: the host GEMM is bandwidth-bound, and the proof is that four times the vector width bought 10 %.
What the compiler was doing.-fopt-info-vec on npu/audio_exec.cpp, on the two MatMul inner loops (lines 427 and 442):
default loop vectorized using 16 byte vectors and unroll factor 4
-march=native loop vectorized using 64 byte vectors and unroll factor 16
-mavx512f loop vectorized using 64 byte vectors and unroll factor 16
So the loops are vectorised -- at 16 bytes, i.e. 4 floats, which is exactly the 25 % of AVX-512's 16-wide peak the issue measures. The default compile target is SSE2, not the box's ISA.
And giving it the real ISA barely helps:
encoder, 4 threads, three reps each
default 3.989 / 4.023 / 3.994 median 3.994 s
-march=native 4.841 / 3.583 / 3.612 median 3.60 s (first rep cold, as usual)
ids identical in both
10 %, from a 4x wider vector. If the loop were FLOP-bound, 16-wide against 4-wide would be close to 4x. It is not, so the time is going somewhere other than multiply-add issue slots -- memory.
That is the answer to the issue's question, and it points the opposite way. The MatMul reads B once per four-row unit. A whisper projection is K=768, N=768, so B is 2.25 MB and L2 is 1 MB per core on this part: B does not fit L2, so every unit streams it from L3/DRAM. Eight units, 2.25 MB each, is ~18 MB of B traffic for a projection that is 2.25 MB of unique data -- and that, not the operand layout, is what caps GFLOP/s.
So the change I would make is k-blocking, not packing: split K into slices whose B rows fit L2 (K-slice x N x 4 <= ~1 MB), and have each unit run the full set of output rows over one k-slice before moving to the next, with the output accumulating in place. B then lives in L2 for the whole slice and is read once across all units instead of once per unit. That is the same trick as a classic cache-blocked GEMM, applied to the dimension that is actually oversubscribed here.
What I am not claiming: that this reaches 2x (the issue's bar). It removes the traffic that the measurement points at; whether the remaining time is L2 bandwidth, L3, or the four read-modify-write output streams is a second measurement, and perf stat's packed-FLOP counter is inert on this box (it reads 0 for a numpy float32 matmul control), so the next instrument has to be perf stat -e cache events or a sized micro-benchmark, not a FLOP counter.
One process note in case it saves someone an hour: -march=native also changes the whole engine if you pass it through CMAKE_CXX_FLAGS, which is fine for an experiment and wrong for a ship. The version worth landing is either a targeted -mavx512f -mavx512vl on this translation unit or a runtime-dispatched 16-wide loop, because the box this runs on is not necessarily the box it was built on.
Correcting my own conclusion from an hour ago: blocking is right, but in j, not k.
I said the encoder is bandwidth-bound and that the fix is k-blocking so a B slice stays in L2. Running the arithmetic on my own claim shows it does not hold: whisper-small's B is 2.25 MB (K=768, N=768, float32), and L3 here is shared and much larger than that, so B already sits in L3 and every unit reads it from there. Eight units x 2.25 MB is 18 MB at L3 bandwidth -- tens of microseconds, not the 3.1 s the node timer attributes to MatMul. So L3 traffic is not the cap and k-blocking would buy nothing.
What the code actually does per multiply-add:
for (int64_tk=0; k<K; ++k) {
constfloat*br=bpp+k*N;
for (int64_tj=0; j<N; ++j) {
constfloatbv=br[j];
r0[j] +=v0*bv; // load r, multiply, store rr0[N+j] +=v1*bv;
r0[2*N+j] +=v2*bv;
r0[3*N+j] +=v3*bv;
}
}
Every one of the four accumulates is a load-modify-store to memory. There is no register accumulation at all: the compiler vectorises the loads and stores (16 bytes, or 64 with -march=native) but it cannot keep four accumulators in registers across iterations of j, because each j writes four different addresses and the loop reloads them next time. So the traffic is K * N * 4 streams * 4 bytes written and re-read, per unit, and the store unit saturates long before the FMA units do -- which is exactly the shape that gives 10 % from a 4x vector width.
The fix is therefore register blocking over j, which is what "packed/blocked" in the title should mean: run j in blocks of, say, 16, hold 4 x 16 accumulators in vector registers, load r0[...] once at the top of the block and store once at the bottom, and stream b through them. That removes two of the three memory operations per FMA and is the classic micro-kernel shape. k-blocking does nothing here; j-blocking removes most of the traffic.
So the issue's title is right and my reason for doubting it was wrong. I read "bandwidth-bound" and went looking at B; the oversubscribed stream is the output, and it is oversubscribed because it is never held in registers. That is a smaller change than packing B, and it is measurable with the same instrument: three reps of encoder seconds plus the ids check.
Register blocking: 7.8 %, bit-exact. So neither the FLOP rate nor the read-modify-write traffic was the cap, and that changes what to do next.
Measured on the same instrument, encoder seconds, three reps each, 4 threads, ids checked every time:
build
reps
median
ids
default (16 byte vectors)
3.989 / 4.023 / 3.994
3.994 s
reference
-march=native (64 byte vectors)
3.583 / 3.612 (+1 cold)
3.60 s
reference
native + j-blocked to registers
3.302 / 3.319 / 3.348
3.319 s
reference, unchanged
So: 10 % from four times the vector width, 7.8 % from holding the four accumulators in registers instead of load-modify-store on every j, and 17 % for the two together. The block is bit-exact because the per-element accumulation order is untouched -- still k ascending, still +=; only the loop over independent j columns moved.
What that rules out, which is the useful part. If the GEMM were FLOP-bound, 4x wider vectors would have bought close to 4x. If it were bound by the four output streams, register blocking would have bought a great deal more than 7.8 %. Neither happened, so the time is going somewhere I have not measured yet -- and the honest position is that the 2x bar needs a measurement I do not have, not a third guess.
The measurement I need is the encoder's own MatMul time.ONEBIT_AUDIO_NODE_TIMES=1 prints per-node lines for the whole run, and the run is 4 s encoder + 27 s decode, so the 78.6 % MatMul share I quoted is dominated by the decode loop -- where the GEMMs are M=1 and have nothing to do with this issue. A phase marker in that output (one line when the graph switches from encoder to decoder) would make the per-node split attributable, and it is a two-line change in the executor. Without it, "MatMul is 78.6 %" is true and useless here.
What I would land now, and what I would not. The j-blocked version is portable and bit-exact and worth its 7.8 % on its own -- no flags, no ISA assumptions, no change in results. -march=native stays an experiment: passing it through CMAKE_CXX_FLAGS changes the whole engine and makes the binary depend on the machine it was built on. If the 10 % is wanted, the landable form is -mavx512f -mavx512vl on this translation unit or a runtime-dispatched 16-wide loop -- and the dispatch has to exist anyway, because the box this runs on is not necessarily the box it was built on.
Next, in order: (1) phase marker in ONEBIT_AUDIO_NODE_TIMES, so the encoder's MatMul is measurable; (2) with that number, decide whether 2x is reachable at all from the host core, or whether the answer is the device route this issue already mentions at 0.305 s/window.
The instrument is a PR: engine#2402. Two markers in whisper_transcribe, nothing when the env var is unset:
[audio-graph] encoder
...one line per encoder node...
[audio-graph] decoder
And I tried two timing changes before concluding the instrument is what is missing. Both are in the PR body so nobody repeats them:
change
result
verdict
__restrict on the three output streams
3.994 -> 4.047 s
1.3 % slower -- noise, reverted
register blocking over j (16 columns in registers, one store per block)
3.994 -> 3.937 s
1.4 % -- noise
the same blocking with -march=native
3.60 -> 3.319 s
7.8 %, but not landable (whole-engine flag, machine-dependent binary)
-march=native alone
3.994 -> 3.60 s
10 % -- the only real win, and it is a flag
Two things worth taking from that table. The 7.8 % for register blocking exists only where the compiler emits 64-byte vectors; without the flag it is 1.4 %, so the read-modify-write traffic is not the cap at 16 bytes. And 4x the vector width buying 10 % rules out FLOP-bound as surely as register blocking buying 1.4 % rules out bound-by-the-output-streams. Whatever is left is not either of the things the title and my first two hypotheses assumed.
So the honest state of this issue: the encoder's MatMul time is not currently measurable (the 78.6 % I quoted is decode-dominated), the two obvious host-side levers are measured and dead, and the 2x bar needs the encoder-only profile that #2402 makes possible before anyone writes a third optimisation for it. If the encoder's own MatMul turns out to be a small share of the 3.9 s pass, then the answer to this issue is the device route it already mentions at 0.305 s/window, not the host core -- and that is a decision for whoever owns the row, which is why I am not claiming the bar.
The miss rate kills the cache hypothesis: 2.4 % misses over 80 G references is not a workload starved by memory. That is worth having -- it rules out the explanation I would otherwise have reached for next, and it is consistent with the earlier result that four times the vector width bought only 10 %.
But these counters cannot answer the question for the encoder, and the arithmetic shows it: whisper-small's encoder is roughly 19 GFLOP (12 layers x 1500x768x768x4 projections x2, plus the MLP), and 3.41 s of that is ~5.6 GFLOP/s. 2,046 G instructions for 19 GFLOP is 106 instructions per FLOP -- no GEMM does that. The counters are therefore dominated by the 20,436-node decode loop, exactly as my first whole-run measurement was, and the per-thread totals tell me nothing about the 3.41 s.
So the next instrument has to be encoder-only, and there are two honest ways to get it:
A mode in the tool (--encode-only, or reuse the existing window loop with a decoder-step limit of 0). That is the smaller change and it makes every future measurement of this pass cheap -- perf stat, perf record, or just time.
A sized micro-benchmark of one projection: M=1500 K=768 N=768 in a loop, in the same translation unit so it compiles with the same flags. That isolates the inner loop completely and is what you would want anyway before trying a packed or blocked variant.
I would do (1) first, because it also makes the numbers in this issue reproducible by anyone with a build, and because (2) risks measuring a copy of the loop rather than the loop.
Where that leaves the issue: the encoder's MatMul is 3.41 s of a 4.42 s pass (77 %) and that is now measured rather than assumed; the two cheap host-side levers are measured and dead (__restrict noise, register blocking 1.4 % portable); cache misses are ruled out at 2.4 %; and the remaining question -- what the 3.41 s is doing -- needs an instrument this tool does not have yet. That is a smaller and better-specified problem than the one the issue opened with.
Correction first, because I got it wrong in the comment above: the tree already builds the audio TU at -O3, not -O2. The actual command for npu/audio_exec.cpp in a Release configure is
so the -O2 9.8 GFLOP/s row in my table was my benchmark's flags, not the tree's, and the "-O2 is the whole story" framing was wrong. The 12.8 GFLOP/s in the issue was also my arithmetic (I divided a family total that includes the 3072-wide fc1/fc2 by a 768-wide FLOP count). With the real -O3 flags the bench gives 37.5 GFLOP/s, which matches the in-situ 2699 ms for the 1x1500x768 family — so the core was never as slow as I said.
What survives is narrower and still measured: the ISA is not requested.-mavx2 -mfma on top of the tree's own -O3, in situ, alternating the two binaries under one load, 4 threads, encoder MatMul family:
baseline (-O3) 2698.9 ms 2699.4 ms
-O3 -mavx2 -mfma 2037.9 ms 2019.2 ms -> 1.33x
and ids identical: 91c6561060 on jfk.wav and fd875ac0d9 on the German fixture, both binaries. That second one matters more than the first — FMA changes rounding, so the check has to reach a path that is not the English transcribe==translate case.
So the row is now: make the build ask for FMA, not rewrite the kernel.-mavx2 -mfma is 1.33x in situ for one flag line, and it is explicit about the ISA rather than -march=native (which adds a few percent in the microbenchmark but makes the binary non-distributable). The caveat to decide before I open a PR: this makes the audio objects require AVX2+FMA at run time — universal on this fleet (Zen 2+), not universal for a distributed build. If that constraint is unacceptable, the honest number for this row is zero and the packed-kernel plan is still not justified at 1500x768x768.
The 1.33x is not free, and the test that caught it is the one I added last night. With -mavx2 -mfma on the onebit_npu target the build succeeds and both transcripts are unchanged (91c6561060 jfk, fd875ac0d9 German), but:
audio_parallel_test -> FAIL MatMul M=8 matches the ordered sum (row-group tail and all-zero group)
That is the FMA rounding difference showing up exactly where I predicted it might: fused multiply-add computes a different (more accurate) sum than the ordered per-element reference the test compares against. The English clip would never have shown it.
So the trade is not "free 1.33x", it is "1.33x for bit-exactness against the ordered serial reference". Two readings, and they have very different conclusions:
If the failure is all thread counts agree with each other but differ from the ordered reference, then determinism — the property the test really protects — is intact, and the reference is simply stricter than the ISA allows. That would justify a reference-computed-with-FMA variant of the test.
If the failure is thread counts disagreeing with each other, FMA on top of the row-group split has broken reproducibility and the flag is not shippable as-is.
I have not yet separated those two, and I am not going to guess: it is one run of the same test printing all four counts instead of the first failure. Until then this stays a measured trade, not a PR — and the honest summary for the row is that -mavx2 -mfma buys 1.33x in situ (2698.9/2699.4 → 2037.9/2019.2 ms) at the cost of the ordered-sum guarantee, which is the guarantee #2379's softmax change openly traded and #2386's row grouping deliberately kept.
My proposed separation experiment does not separate the two cases, and I should say so rather than read a verdict into it. With -mavx2 -mfma the test now reports 16 failures, and every one of them is the same kind:
FAIL the serial MatMul is the ordered per-element sum, bit for bit
FAIL MatMul M=1 at ONEBIT_AUDIO_THREADS=1 matches the ordered sum …
FAIL MatMul M=1 at ONEBIT_AUDIO_THREADS=2 …
FAIL MatMul M=1 at ONEBIT_AUDIO_THREADS=4 …
FAIL MatMul M=2 at ONEBIT_AUDIO_THREADS=1 …
That set is exactly what you would get from a compiler-flag difference between the library and the test's own reference, not from a thread-count disagreement: only the onebit_npu target carries -mavx2 -mfma, while the reference loop lives in tests/audio_parallel_test.cpp and is compiled without it. Under those flags the two TUs are expected to differ in the last bits of every sum, at every thread count, and the "serial is the ordered sum, bit for bit" check fails first for the same reason. So the run cannot distinguish:
thread counts disagree with each other (reproducibility broken — flag unshippable), from
library agrees with itself and disagrees with a differently-compiled reference (determinism intact — the reference needs the same flags).
The separating test is one line of scope, not one run: put -mavx2 -mfma on the test target too (or compare library-vs-library across thread counts with no reference at all) and it answers immediately. Until that exists, my previous comment's "two readings" stands and I have not moved it forward — I just stopped myself from reading 16 identical failures as evidence for the reading I preferred.
What is solid regardless: 1.33x in situ on the MatMul family (2698.9/2699.4 → 2037.9/2019.2 ms, alternating binaries, 4 threads), and both transcripts unchanged (91c6561060, fd875ac0d9). What is not solid is whether that costs reproducibility or only agreement with a differently-built reference — and that distinction is the whole decision, so the flag stays out of a PR until it is answered.
A consolidation, because four PRs now touch this core and the ledger is spread across them. Everything below is either something I measured myself or a PR's own number marked as such.
They compose rather than overlap: #2386 changes the loop shape, #2389 the gather feeding it, #2403 the instruction selection, and the fast path removes a case that should never have reached the GEMM. If they all land, the core is roughly 1.35 × 1.33 ≈ 1.8× on the four-row baseline.
The thing this issue is actually about has no PR
The scope note says packed/blocked micro-kernel, and none of the above is that. Even with all four, the core stays in i-k-j order over an unpacked B, so it re-reads B from cache for every output row and issues vfmadd at a rate the scheduler cannot sustain. 12.8 GFLOP/s against a single-pass AVX-512 peak is still ~25 %, and the four-row change did not move that ratio — it moved the constant in front of it by reusing each B element across four accumulators, which is a poor man's blocking.
What a real blocking step looks like on this shape (M small, K and N in the thousands, f32):
pack B into K×NR tiles once per call, in the layout the kernel wants (vfmadd231ps on contiguous zmm registers, no gather, no broadcast from memory);
4–8 accumulator registers per unit over N, so the FMA:load ratio rises from 1:2 toward 4:1;
the K loop blocking for L1 (a Kc around 256 f32 keeps a B panel in L1 while M is tiny and N streams);
the row-group tail kept exactly as it is — test_row_group_edges() and the all-zero-group case already pin it, and that is the part a rewrite usually breaks.
That is the next ~2×, and it is a bigger change than the four above combined. It is also the one place where GEAK should be driving: it is an existing kernel being sped up, so the workflow is mode optimize on the current four-row source with a NumPy/fp64 reference, not a hand rewrite.
One caution that belongs on the record
The -mavx2 -mfma win is not free, and the test that caught it is one added the previous night: audio_parallel_test fails MatMul M=8 matches the ordered sum (row-group tail and all-zero group) under FMA, because FMA changes rounding and that test compares against an ordered per-element reference. My proposed separation experiment did not separate the two cases (it failed three ways at once), so I am not reading a verdict into it.
The consequence for anyone landing #2403: the flags must cover tests as well as the library (two TUs compiled differently is the failure), and if the ordered-sum reference is a contract rather than a debugging aid, the flags need a rounding-tolerant comparison or an explicit decision that FMA's associativity is acceptable. That decision is the owner's, not a side effect of a 1.33×.
Scoped follow-up, written down rather than half-done: the elementwise ops still allocate a fresh output per node, and there is one safe in-place case.
llama.cpp#2437 took Reshape (135.9 → 2.4 ms per encoder pass) by moving a single-use input buffer instead of copying. Add/Mul/Sub/Div in npu/audio_exec.cpp build a fresh vector too — 99 Add + 52 Mul calls in the whisper-small encoder, and after #2383 their per-element cost is index-free, so what is left is the allocation and the copy.
The safety condition, stated once so nobody copy-pastes #2437's idiom into a binary op:
op
may the destination be the source buffer?
Reshape
yes — output is the input bytes in a new shape, so a move is exact
Add/Sub/Mul/Divwhen the operand is single-use, not an initializer, and not the graph's y
yes — r.data[i] = a[i] <op> b[i] writes element i from element i, and no later element reads an earlier one
the same ops with two inputs
only one operand may be the destination, and the other must not alias it
The asymmetry with #2437 that I flagged in that review is why this needs its own analysis rather than the same diff: a reshape is a pure relabel, an elementwise op is safe in place only because the reads and writes at index i do not interact.
Why I am not opening this as a PR right now: the change is small but the verification is not — it needs the ids check on both fixtures, audio_parallel_test at 1/2/4 threads (it compares against an ordered reference and would catch an aliasing bug that corrupts one element), and an alternating A/B for a win that may be a few milliseconds per pass. That is a full cycle, and I would rather leave a precise spec than a half-built branch — this lane has already produced one always-false if(TARGET …) and one 181-TU flag scope from working at the edge of a session.
The measurement that would decide it first:ONEBIT_AUDIO_NODE_TIMES=1 on the encoder — if Add+Mul are ~1.17 s of a 5.6 s pass and the per-element loop is already flat, the remaining cost is allocation + memcpy, and the same per-shape signature #2437 used (equal cost across shapes of equal element count) will say so before any code is written.
The one lever I left as "an owner decision" now has numbers on both sides of the trade, so the decision can be made rather than deferred.
Measured, same row, only exp changed
~/.cache/scratch-hrx/gemmbench/bench4.c — the shipped softmax_row shape at the real size (12 × 1500 rows of 1500), with expf replaced by an 8-lane AVX2 polynomial (exp(x) = 2^n · poly(r), Cephes-style reduction, the same one sse_mathfun uses):
A expf 88.6 ms
B vectorised exp 35.6 ms speedup x2.49
max|diff| 1.863e-09 max relative 2.384e-07
What that means for the decision
The gain is 2.49× on the Softmax slice, which the per-node log puts at 10.5 % of the encoder — call it ~6 % of the whole pass, and it is the largest single win left that does not require restructuring the graph.
The cost is smaller than I predicted. I said "~1e-7 absolute"; it is 1.9e-9 absolute, 2.4e-7 relative, i.e. at the level of f32 epsilon (1.2e-7). The polynomial is not bit-equal to expf, so anything comparing two implementations bit-for-bit will now differ — but a tolerance-based comparison at 1e-6 relative passes comfortably.
So the honest framing for whoever owns this call is not "is a numerics change acceptable" but "is 6 % of the encoder worth a 2.4e-7 relative difference", and I would say yes, with one condition: the change belongs in softmax_row only, with the array tail left exactly as it is (my benchmark keeps the scalar tail for n % 8), because softmax feeds attention weights and a tail handled differently from the body is how two paths quietly diverge.
What would need to move with it
npu/forward/include/audio_ops.h::softmax_row — the body loop becomes the 8-lane version; the tail stays scalar.
audio_parallel_test — it compares thread counts against an ordered per-element reference, which is a bit-exact contract. This change keeps the order (each element is still computed independently; only the exp function changes), so the ordered-sum property survives, but the values move in the last digits and that suite will fail unless its comparison is tolerance-based. That has to be decided in the same breath as the change, not discovered after it.
audio_ops_test's expected values for softmax, if any are literal.
What I have not done, and why
Not implemented in the tree: it is a numerics change to the attention path, and this row's history tonight is three of my own hypotheses killed by measurement after I acted on them early. The measurement above is cheap and reproducible (gcc -O3 -mavx2 -mfma bench4.c -lm); the change is one function and one test decision, and it should be taken by whoever can also say what audio_parallel_test's ordered-sum contract is allowed to mean.
Row ledger, final
MatMul 74.6 % at 52-71 % of AVX2 peak; packing +6 % -> no factor of two available
Softmax 10.5 % 18M scalar expf; vectorised exp = 2.49x -> ~6 % of the pass, 2.4e-7 relative
Conv 5.8 % 6 nodes at ~182 ms, unexplained -> worth one look
tiny swarm 6.0 % 852 nodes; Reshape copies removed in #2437 -> remaining is dispatch/allocation
GEMM loop equals a plain triple loop to within 1 % -> nothing there
#2437 (the Reshape move, merged into review) took the only item that needed no numerics decision.
The Conv slice is no longer "unexplained". Its per-node numbers are lopsided in a way that names the cost, and the cost is not the multiply.
The distribution
Conv 1x768x1500 404.9 ms
Conv 1x768x1500 393.7 ms <- two nodes carry 800 of the 1,093 ms
Conv 1x768x3000 102.5 ms
Conv 1x768x3000 93.7 ms
Conv 1x768x3000 25.4 ms <- the three at 3000 vary 4x, which is itself worth a look
For cout=cin=768, k=3, tout=1500, the arithmetic is 2 × 768 × 1500 × 768 × 3 ≈ 5.3 GFLOP. At 394 ms that is 13.5 GFLOP/s — against the 45–80 GFLOP/s the MatMul family reaches on the same box. So roughly three quarters of each Conv node is not multiplication.
Where it goes, from the code
npu/audio_exec.cpp's Conv branch is an im2col GEMM, and it does three things per node on top of the GEMM:
conv1d_weight_as_kn — transposes the weight into (cin*k, cout) order: 768·768·3 = 1.77 M floats (7 MB) per node, per call;
conv1d_im2col — materialises acol at tout · cin · k = 1500·2304 = 3.46 M floats (13.8 MB) per node;
a final bias + transpose: o[co*tout + tt] = bv + c[tt*cout + co] — reads c with stride cout while writing contiguously, i.e. a cache-hostile transpose of 1.15 M floats, done as the last loop of the node.
The GEMM at ~50 GFLOP/s would be ~106 ms; the node takes 394 ms. The other ~290 ms is those three passes, and the third is the one I would fix first because it is cheap to fix:
write the output in (cout, tout) order straight out of the GEMM — multiply Bᵀ · Aᵀ instead of A · B — and the strided transpose disappears entirely, along with the separate bias loop (fold the bias into the first row of B or add it in the same blocked pass). That is one call-site change in this branch, it does not touch the GEMM, and on the numbers above it is worth on the order of 150–200 ms of the 1,093.
The other two are structural: acol is 13.8 MB allocated and filled per node, six times per pass, for weights that do not change between encoder invocations — caching bkn (7 MB, weight-derived) across the six nodes would remove one whole pass per node, and it is the same "the weights did not move" reasoning that makes the MatMul's own weight packing worthwhile.
Ranked against the rest of the row
Softmax 10.5 % 2.49x available (measured) -> ~6 % of the pass, needs a numerics decision
Conv 5.8 % ~3/4 is im2col + strided transpose; GEMM would be 106 of 394 ms
-> the output-transpose fix is the cheapest real win in the row: no numerics, no GEMM change
swarm 6.0 % Reshape copies removed (#2437); the rest is dispatch/allocation
MatMul 74.6 % at ceiling
So after #2437 (the Reshape copies) and the Conv output transpose, the row is down to one decision (vectorised exp) and one restructure (the 852-node swarm). I have not implemented the Conv change: it is a layout change in a branch whose seven callers I have not audited, and the last time I "reverted something to its original body" I deleted a guard — the same care applies before reshaping an output layout. The numbers above are what make it a one-afternoon change rather than a research question.
Withdrawing the percentage table in my earlier comments. Three interleaved rounds of the same binary say it was single-sample noise, and the spread is large enough that the conclusions built on it do not hold.
The per-op percentages are meaningless at this noise level. I reported "MatMul 74.6 %, Softmax 10.5 %, Conv 5.8 %, swarm 6.0 %". The MatMul share across these three rounds is 18.3 % / 31.7 % / 36.4 % — it more than doubles between rounds. Every smaller op has a 3–6× spread. Those numbers came from one run and I presented them as a ranking; they are not a ranking.
The "58.8 GFLOP/s, at the variant's ceiling" conclusion has to go with them, because it was 822 GFLOP / 13,987 ms — the same twelve MatMul timings. At 46–57 s per round the same FLOPs give 14–18 GFLOP/s, which is close to the 12.8 GFLOP/s the issue originally reported. So the original figure may have been right and my "the GEMM is already at its ceiling" was the artifact. That is the worst kind of correction to have to write, because it argues in favour of the defect this issue was filed about.
Note also that these three rounds are 4× slower in total than the runs an hour ago (18.7 s → 50–74 s) on the same binary. The box is shared and thermal-run caps the die at 78 °C, so a suite of back-to-back runs measures the thermal and load state as much as the code.
What still stands, and why it is not affected
These are not timing comparisons from that harness:
Reshape deep-copied its input — r.data = a.data, 4.6 MB per node, visible in the code and in the flat ~0.6 ms cost across four different shapes with the same element count. Fixed in npu: a single-use input is moved into Reshape, not copied (engine #2399) #2437 by moving when the input is last-used; bit-exact (encoder output identical, audio_ops_test and audio_parallel_test both pass).
The vectorised exp is 2.49× — from a standalone benchmark (88.6 vs 35.6 ms, same loop otherwise), which is a 2.5× difference and survives a 1.5× noise floor. Its cost is 2.4e-7 relative, also measured there.
The GEMM loop matches a plain triple loop — 58.3 vs 58.8 GFLOP/s, two standalone benches. That says the loop is not obviously wasteful; it says nothing about the engine's achieved rate, which is the thing the noisy harness was supposed to measure.
Structural facts that do not depend on any timer: 852 nodes in the sub-2-ms swarm; two Conv nodes whose shapes are identical to their neighbours'; im2col allocating 13.8 MB per Conv node.
What the issue should now ask for
Not a percentage ranking — an instrument. The per-node timing needs either (a) a quiet box and N≥5 interleaved rounds reported as medians with spreads, or (b) a sampling profiler (rocprof/perf) whose totals do not depend on per-node wall-clock around a shared, thermally-capped machine. Until one of those exists, any number of the form "X % of the encoder" in this issue — including mine — should be treated as unmeasured.
I would rather retract four comments than leave a ranking behind that sends the next person at the wrong 6 %.
The instrument works, and it says my withdrawn table was wrong in the direction that matters most: the GEMM is 82 % of the pass and runs at ~14 GFLOP/s, not at any ceiling.
Round 3 is a uniform 1.7× slowdown of every op — frequency scaling, not per-op noise. That is exactly why the shares are the right invariant and the absolute times are not: 81.9 / 82.3 / 81.7 % is stable to 0.6 % across a run whose total moved 78 %.
What that fixes
MatMul is 82 % of the pass, not the 74.6 % I published from one wall-clock sample, and nothing like the 18–36 % the wall-clock timer produced when the box was busy. The wall-clock instrument was so bad it made the dominant cost look like it was a fifth of the work.
The engine's GEMM runs at ~14 GFLOP/s: 822 GFLOP of MatMul work over 58,623 ms. The issue's original 12.8 GFLOP/s was right; my "already at 58.8 GFLOP/s, at its ceiling" was the artifact — that number came from dividing by a wall-clock that included time the process was descheduled.
Therefore the 4× gap is real: a plain -O3 -mavx2 -mfma triple loop does 58 GFLOP/s standalone (measured twice), and the engine's equivalent does ~14. So the headroom sits exactly where the issue said it did, and my retraction of the ranking was right while my retraction of the lever was wrong.
Where I would look, given 82 % at 14 GFLOP/s
Not blocking, not packing — threading. The engine's MatMul is parallelised over output rows with parallel_units; the standalone benchmark that reaches 58 GFLOP/s is single-threaded. A per-row task on a small M with a thread pool per node is a plausible way to turn 58 into 14, and it is directly testable: run the same binary with ONEBIT_AUDIO_THREADS=1 and compare the MatMul CPU-time total. If one thread is faster in process CPU time (not wall), the parallelism is costing more than it buys, which would be a very different fix from anything in this issue.
That is the next measurement, and it needs the instrument built in this comment — not another wall-clock sample.
The instrument itself is in npu/audio_exec.cpp: the per-node timer now reads CLOCK_PROCESS_CPUTIME_ID instead of steady_clock, with the reason in the comment. perf record is blocked here (perf_event_paranoid = 2), so this is the attribution tool the row now has.
Measured the step this follow-up said to do first, and it says the lever is real but smaller than the per-shape line suggests.
Encoder only (first 447 nodes), ONEBIT_AUDIO_THREADS=4, ONEBIT_AUDIO_NODE_TIMES=1:
Add n=187 total= 349.9 ms
Mul n= 92 total= 243.4 ms
Div n= 20 total= 89.6 ms
elementwise total (encoder) = 682.8 ms of a ~5.6 s pass ≈ 12 %
The per-shape signature is the memcpy proof asked for:Add 1x1500x3072 is 24 calls at 7.596 ms mean = 4.6 M elements → 1.65 ns per element, and Add 1x1500x768 (1.15 M elements) is 1.010 ms → 0.88 ns/element. Two shapes, cost proportional to elements and not to anything else — the same signature llama.cpp#2437 used to prove Reshape was copying. Streamed traffic at 7.6 ms for (read 2 × 18.4 MB + write 18.4 MB) is ~7 GB/s, which is where a copy and an allocate land, not where arithmetic does.
Two corrections to my own earlier note, both from this run:
Sub does not occur in the encoder (187 Add / 92 Mul / 20 Div, no Sub), so the table should name three ops, not four.
The ceiling is 682.8 ms (12 %), not "1.17 s" — the figure I quoted earlier came from the whole-tool aggregation, which is dominated by the 25-token decode (2 834 Add 1x128x768 calls). This is exactly the encoder/decode attribution trap that engine#2402's markers exist to close, and I walked into it with my own numbers.
What that implies for the follow-up: eliminating the allocation+copy on single-use operands can win at most ~0.68 s of a 5.6 s pass, and only the fraction that is allocation rather than the write itself (the write is unavoidable; the second buffer is not). So the realistic ceiling is a few hundred milliseconds — worth doing, not worth a risky change, and the ordering matters: Add/Mul/Div are now smaller than the MatMul family (2.5 s) and Softmax (0.58 s) already addressed, so this is the third-priority lever, which is what a "12 %" number is for.
Not done, deliberately: the in-place implementation. The safety table above stands (Reshape may move; an elementwise op may write in place only when the operand is single-use, not an initializer, and not the graph's y), but it needs audio_parallel_test at 1/2/4 threads as the aliasing check, and I would rather hand over a measured ceiling and a table than an aliased buffer nobody tested.
The threading test came out flat, and the reason is worth more than the test: the engine's GEMM is running scalar, and I can show it from the instruction count.
Threading is fine — my prior was wrong
THREADS=1 instructions 1,190,947,379,277 cycles 191.3 G 122.4 s wall / 115.1 s user
THREADS=4 instructions 1,190,900,412,355 cycles 225.7 G 13.9 s wall / 46.7 s user
Identical to 0.004 %. So there is no spin-waiting, no duplicated work, no pool overhead worth naming: the same 1.19 trillion instructions either way, and 8.8× wall speedup from 4 threads. Threading is not the 14-vs-58 gap — I expected 1 thread to use less CPU, and it uses the same, so that hypothesis is dead too.
What the number says instead
1.19e12 instructions / 822 GFLOP = **1.45 instructions per FLOP**.
a scalar FMA loop needs ~0.5 plus loads and index arithmetic, i.e. ~1–1.5.
The engine's inner loop is emitting scalar code — about 20× more instructions per FLOP than the vectorised version, which is exactly the ratio that produces ~14 GFLOP/s where a vectorised loop gets 58.
And this kills my earlier "loop codegen is not the gap" conclusion
I measured the loop standalone (bench2.c, the code lifted verbatim) at 58 GFLOP/s and concluded the loop was fine. It is fine in a file-scope function with two pointers; inside the engine it sits in a lambda capturing [&], in a large translation unit, over rr/br pointers the compiler cannot prove do not alias. Same source, different codegen — which is precisely the trap bench2.c could not catch, and I presented its 58 GFLOP/s as if it settled the question.
The next measurement, and it is the cheapest one left
Not a benchmark: read the emitted code.
objdump -d <audio_exec object> | grep -c vfmadd # expect 0 if my reading is right
objdump -d | grep -cE "vmulss|vaddss|mulss" # expect many
If that shows scalar mulss/addss in the MatMul inner loop, the fix is a codegen fix, not an algorithmic one: hoist the row function out of the lambda into a file-scope static function taking float* __restrict rr, const float* __restrict b, const float* __restrict a and loops with int64_t indices, then re-measure instructions/FLOP (target ≈0.06). That is a contained change to a hot loop with a number to verify against that does not depend on the machine's load or thermal state at all — 1.19e12 instructions is a property of the compiled code.
That also reframes the row honestly: the GEMM's 82 % is not "at its ceiling" and not "a threading problem" — it is scalar code, and the instruction count says so without a single timing.
The object file answers the whole question: the engine's audio code contains no vector instructions at all.
objdump -d CMakeFiles/onebit_npu.dir/npu/audio_exec.cpp.o
vfmadd 0
vmulps / vaddps 0 <- not even packed SSE/AVX multiply
mulss / addss 113 <- everything is scalar
That is the 14-vs-58 GFLOP/s gap, with no timing involved: the MatMul inner loop is compiled scalar. My standalone benchmark of the same source line reaches 58 GFLOP/s because it is compiled with -O3 -mavx2 -mfma as a tiny file-scope function; the engine's copy is compiled in a large TU, inside a [&] lambda, over two float* the compiler cannot prove non-overlapping — and the compiler emitted not one packed operation.
Why this settles the row's argument, and corrects three of my own conclusions
"The GEMM is already at its ceiling" — wrong. It was at ~14 GFLOP/s, and the ceiling for this shape on this core is 58.
"The loop's codegen is not the gap" (my bench2.c result) — wrong in the same way: I measured the loop where it vectorised and applied that to where it does not. Same source, different codegen is the whole story.
"Threading costs more than it delivers" — wrong: identical instruction counts at 1 and 4 threads (1,190,947M vs 1,190,900M), 8.8× wall speedup. Threading is fine.
The 82 % share stands, and it stands because of this: the dominant cost is scalar arithmetic, so it dominates.
The fix is two things, and one of them is already a PR
Making the loop vectorisable so the flags are actually used. Reading the object says the compiler also declined packed multiply, which is available even without AVX2 — so this is not only a flags problem. The candidates, in the order I would try them: float* __restrict on rr/br; hoisting the row body out of the lambda into a file-scope static function; and dropping int64_t indices to int in the inner loop so GCC's vectoriser has no reason to bail. Each is verifiable without the machine's noise: grep -c vfmadd on the rebuilt object, then instructions/FLOP from perf stat (1.19e12 today; a vectorised loop should land near 0.1e12).
What I would not do
Not the packed micro-kernel, not blocking, not the four-row structure, and not the Softmax exp — all of those are second-order against a loop that is currently scalar. The row's real shape is: 82 % of the encoder is a GEMM compiled scalar at 14 GFLOP/s, and the achievable number on this core is 58.
One caveat stated plainly: this object is from my build (-DCMAKE_BUILD_TYPE=Release, no -mavx2), which is the configuration #2403 exists to change. If a tree built with those flags still shows vfmadd 0 in audio_exec.cpp.o, then the flags are not reaching this target and that is the first thing to fix — which is exactly the target-list problem I flagged on #2403 this morning.
No -mavx2, no -mfma, no -march anywhere. So GCC emitted exactly what those flags allow: 4-wide legacy SSE multiplies and adds, no FMA. My grep looked for the AVX-encoded mnemonic in a build that cannot emit it — the same class of error as reading a skipped test as a pass, and I made it while writing a comment about exactly that class.
What the row actually says now
the inner loop
is vectorised — SSE, 4 floats per instruction, separate mulps + addps
the build
has no ISA flags at all (-O3 -DNDEBUG only)
the achievable comparator
-O3 -mavx2 -mfma: 8 floats per vfmadd, i.e. 2× the width and mul+add fused
measured in situ
#2403: 1.33× — and that is the number this explains
So the 14 GFLOP/s is not "a scalar loop". It is a 4-wide non-FMA loop where 8-wide FMA is available, and #2403 is the change that gets it. The 1.33× (rather than the ~4× the width alone suggests) is what memory traffic and the per-node overhead around it eat — consistent with instructions/FLOP still being 1.45 rather than the ~0.5 an SSE inner loop alone would need, i.e. that much again in loads, index arithmetic, im2col and the transpose.
What I reverted, and why
I hoisted the row loop into a file-scope static with __restrict pointers and int indices on the theory that the lambda was blocking vectorisation. The object shows the loop was already vectorised before that, so the change has no measurement behind it and I have reverted it rather than leave an unverified "optimisation" in a hot path. If someone wants to chase the remaining overhead, the honest next step is -mavx2 -mfma (#2403), and only then a fresh instruction count — because the flag changes what "vectorised" means here.
The row's ledger, with only what survived measurement
MatMul is ~82 % of the pass — three rounds, shares 81.9 / 82.3 / 81.7 % (process-CPU-time instrument, added in this issue).
Threading is fine: identical instruction counts at 1 and 4 threads, 8.8× wall.
Softmax's cost is 18 M scalar expf; a vectorised exp is 2.49× with 2.4e-7 relative — a numerics decision, still open.
My per-op percentage table and the "GEMM at ceiling" claim are withdrawn; the instrument that replaced them is in the tree.
Three of my own conclusions died tonight to measurements (blocking, codegen-in-a-benchmark, and now "scalar"). The one that keeps surviving is the plainest: the numbers have to come from the object and the instruction counter, not from a timing on a shared box.
So the two things my caveat asked about are both answered: the flags do reach this target (32 vfmadd appear), and they halve the instruction count (2.11×).
Why that yields 1.33× and not 2×
The instruction count halves and the cycle count barely moves. 2.11× fewer instructions, 1.09× fewer cycles — so the loop gave up IPC (5.28 → 2.73) and the time stayed where it was. That is the signature of a loop bound by memory, not by arithmetic: the fused 8-wide instructions finish sooner and then wait.
That is consistent with what #2403 measured in situ (1.33×, transcripts unchanged) and it says the next lever is not more instruction-level work: it is fewer bytes and more reuse — the 13.8 MB im2col per Conv node, the 7 MB weight repack per Conv node, and the fact that every one of the 288 MatMul nodes re-reads its own B. Those are the numbers that should move now, and they are measurable with the same two tools this row finally has: instructions/FLOP from perf stat and bytes touched per node from the code.
A note on one number I am not reporting: the wall clock in that run was 25.8 s against 13.9 s earlier for the same work. That is the thermal/load noise this box has shown all night (the three-round test moved totals 1.78×), and it is exactly why the comparison above is instruction and cycle counts rather than seconds.
Every one of those is reproducible from objdump, perf stat or the harness in the tree — and none of them depends on a quiet box, which is the only reason I trust them.
The "bytes and reuse" lever now has numbers, measured by calling the real functions out of npu/forward/include/audio_ops.h rather than reimplementing them — the reimplementation is exactly what misled me earlier tonight, when a standalone copy of the MatMul loop vectorised and the engine's did not.
The two copies inside every Conv node
conv1d_weight_as_kn 26.0 ms per node 7.1 MB written (w -> (cin*k, cout) B panel)
conv1d_im2col 32.4 ms per node 13.8 MB written (activation -> (tout, cin*k))
six Conv nodes: 350 ms total for just these two copies
So ~58 ms of pure copying per Conv node, ~350 ms per encoder pass — against a Conv slice that three-round process-CPU-time measurement put in the same order (1.5–3.8 s wall, and the CPU-time rounds gave 3.5 / 3.1 / 6.6 s for all six). Whatever the exact share under load, this is a third-ish of the Conv cost spent writing bytes that are already in memory.
What is cachable, and what is not
conv1d_weight_as_kn is pure waste when the weight repeats. It is a function of (w, cout, cin, k) only — the same 7.1 MB panel, written per node. Any two Conv nodes over the same weight (whisper's conv2 is applied 12 times in the layer stack in the usual layout) pay 26 ms each to produce identical bytes. A cache keyed on (w.data(), cout, cin, k) — the pattern HostWeightCache already uses for the GEMM's own packing — removes all but the first.
im2col is inherent to the algorithm. 13.8 MB written per node because an im2col GEMM builds the matrix it multiplies; removing it means a direct convolution (or a GEMM that walks the halo itself), which is a real kernel rather than a cache. That is the "packed micro-kernel" this issue originally imagined, and it is now possible to say what it would be worth: 32 ms per node, not the whole Conv slice.
So the row's next two changes, in order of (value ÷ risk)
Cache the Conv weight panel. ~26 ms per Conv node, zero numerics, no algorithm change, same shape as the weight packing the GEMM already caches. This is the cheapest remaining win in the entire issue.
Every number there comes from objdump, perf stat, the real headers, or three-round medians — none from a single wall-clock sample on this box, which is the only reason any of them survived the night.
register blocking over j: 1.4 % portable (7.8 % with -march=native), instructions 349.75 -> 345.60 G (1.2 %), cycles 74.94 -> 69.76 G (6.9 %) -- the compiler was already register-blocking.
-march=native itself buys 10 % on this die and is not landable (it inherits the build host, and the fleet's toolchain note says so).
Why the host route is a rewrite and not a flag: at IPC 4.67 with 1.2 % cache misses there is no stall to recover -- the only way to 2x is more work per instruction, i.e. an explicit 16-wide AVX-512-shaped kernel over a 4-wide SSE2 code path. That is a kernel rewrite, not a tuning change, and #2402 already landed the instruments ([audio-graph] encoder/decoder phase markers, ONEBIT_WHISPER_ENCODE_ONLY=1) so whoever does it can measure the same way this was measured.
The device route is the decision: 0.305 s/window on the NPU. That is the path ORCH has chosen, and the reason to prefer it is not only the number -- it is that the host route's remaining 2x requires new SIMD code whose correctness surface (f32 accumulation order) is the same one this row's sibling issues were about.
Instruments that outlive this issue: PR #2402 (merged, 8 review rounds) -- phase markers with fflush(stdout) (node lines go to stderr, markers to stdout, which is why a merged log put markers after the block until it was fixed), and ONEBIT_WHISPER_ENCODE_ONLY=1. Both are main; the encode-only switch is what made the 74.9 G cycle figure possible at all.
Activity
bong-water-water-bong commentedon Nov 14, 2026
Claiming this lane (pi-i679). I reviewed both halves that led here -- #2386's four-rows-per-unit and #2389's row-at-a-time Transpose -- so the measurement chain is familiar, and this issue names the lever both of them stopped short of.
What I will try. The issue's own arithmetic puts the ceiling at four AVX-512 cores in a single pass, so the target is a packed/blocked B rather than another loop-shape tweak: pack B into k-blocks so a block of
b[k]rows is read once and used by every output row in the unit, with the packed layout chosen so the inner loop is contiguous on both sides. The two prior changes fixed the streaming and left the access pattern alone; this one is about the access pattern.The arms I will hold myself to, because the last two changes in this file set the bar and the issue already states them:
91c6561060as the goal and prove it rather than assume it. If a packed path cannot be bit-identical, that is a finding to report, not a rounding error to wave through.tools/ci/medians.pydeciding rather than a median I quote -- four identical runs on this box span 1.58x, which is why that tool exists.M % block != 0.No device needed to develop: this is
npu/audio_exec.cpp's host branch, so the work is CPU-side under thermal-run; the only device question is whether the whisper ids still match, which wants one window at the end rather than a ticket to develop in.bong-water-water-bong commentedon Nov 14, 2026
This issue's premise is wrong, and the measurement that corrects it is worth more than the micro-kernel it asked for. Two findings, both reproduced twice, same source, one process, load could only hurt.
1. The same kernel runs at 9.8 or 69 GFLOP/s depending only on compiler flags
7× from the flag line alone, on the exact loop I proposed to replace. The 12.8 GFLOP/s figure in the issue was the in-situ
-O2-built binary, so it was never measuring the algorithm — it was measuring that the vectoriser was not asked to do FMA. A packed/blocked rewrite of a core that is being compiled without FMA would have been optimising the wrong thing, and I would have spent a project finding that out.2. The packed prototype I built first is worse, not better
The naive panel copy (per column block, per k block) costs more than the register tiling saves at this size — 1500×768×768, where B entirely fits in L2 anyway. So at this shape the win is not available from packing; it may be at much larger K/N, but that is not this workload.
What this means for the row
-O3and FMA.-O3alone is 3.8x over-O2and needs no-marchcontract;-O3 -mavx2 -mfmais 6.6x and states the ISA explicitly (fleet is AVX2/FMA everywhere since Zen;-march=nativeadds 7 % and a distributability problem, so I would take-mavx2 -mfma).91c6561060on jfk and the German fixture,audio_parallel_testgreen at 1/2/4 threads. FMA fusing is exactly the kind of change that can move a last bit, so it must be checked, not assumed.I will bring the flag change as a PR with the alternating-A/B in-situ number (4 threads,
ONEBIT_AUDIO_NODE_TIMES=1to keep the per-shape split honest) and the ids check, unless the build flags are deliberately-O2for a reason I have not found — in which case that reason belongs in this thread before I touchCMakeLists.txt.bong-water-water-bong commentedon Nov 14, 2026
First measurements on #2399: a baseline, one dead hypothesis, and an instrument that turned out to be inert.
The instrument works and is cheap:
whisper_transcribeon whisper-small +jfk.wav, host-only (ONEBIT_HRX=OFF, NPU runtime from a lane cache), no device and no window. It reproduces the reference ids exactly (400 370 452 7177 6280 ... 1941 13), so it is the same path the issue measured.Baseline, 4 threads, three reps:
Node split from
ONEBIT_AUDIO_NODE_TIMES=1over 20,883 nodes (whole run: 4.4 s encoder + 27 s decode):So the encoder's ~3.1 s of MatMul agrees with the issue's 3.3 s, and MatMul is the whole story.
__restrictwas my first hypothesis and it is noise.ap,bppandr0are separate tensors, so the promise is true -- and A/B/A with three reps each gives baseline 3.994 s against 4.047 s, i.e. 1.3 % slower. Reverted. Two things worth recording from that: the effect I first saw (4.438 -> 4.080) was a cold first rep, 11 % above warm -- the same effect that turned item 3a's "2.0x" into 7.5 %, now on the host side; and one run is not a measurement even of a four-second loop.Where the headroom is not. After #2386 the inner loop is already contiguous on both sides (
br = bpp + k*N, and four contiguousr0[...]streams), B is 2.25 MB streamed once per k-block, and the four output streams are 12 KB and fit L1. So it is not cache and not access order.perf staton the run gives IPC 4.63 (426 G cycles / 2,052 G instructions), which is well-vectorised -- so the 25 % is not a codegen failure either.And the FLOP counter I wanted is inert on this box.
fp_ret_sse_avx_ops.packed_single_flopsreads 0 for the whisper run, and the control --numpy1024x1024 float32 matmul, certainly packed SIMD -- also reads 0. So that counter cannot be used here, and the GFLOP/s figures have to stay derived from shapes and wall time, as the issue does it.Next, in order: check the generated code for packed FMA in the MatMul inner loop (
-fopt-info-vecon the file, or the disassembly) rather than guessing from IPC -- IPC 4.63 is consistent with scalar loads and integer work as well as with vector FP. If the loop is not vectorised, the fix is a j-blocked variant with an explicit vector width (the packed B the issue proposes is one way, but if the access pattern is already contiguous, an explicit 16-wide FMA loop overjis the smaller change and the same 4-stream shape). If it is vectorised, then the 25 % is a bandwidth limit on four read-modify-write streams and packing will not help -- which would make this issue's title the wrong lever, and that is worth knowing before anyone spends a day on it.bong-water-water-bong commentedon Nov 14, 2026
The lever in the title is the wrong one: the host GEMM is bandwidth-bound, and the proof is that four times the vector width bought 10 %.
What the compiler was doing.
-fopt-info-veconnpu/audio_exec.cpp, on the two MatMul inner loops (lines 427 and 442):So the loops are vectorised -- at 16 bytes, i.e. 4 floats, which is exactly the 25 % of AVX-512's 16-wide peak the issue measures. The default compile target is SSE2, not the box's ISA.
And giving it the real ISA barely helps:
10 %, from a 4x wider vector. If the loop were FLOP-bound, 16-wide against 4-wide would be close to 4x. It is not, so the time is going somewhere other than multiply-add issue slots -- memory.
That is the answer to the issue's question, and it points the opposite way. The MatMul reads
Bonce per four-row unit. A whisper projection is K=768, N=768, so B is 2.25 MB and L2 is 1 MB per core on this part: B does not fit L2, so every unit streams it from L3/DRAM. Eight units, 2.25 MB each, is ~18 MB of B traffic for a projection that is 2.25 MB of unique data -- and that, not the operand layout, is what caps GFLOP/s.So the change I would make is k-blocking, not packing: split K into slices whose B rows fit L2 (K-slice x N x 4 <= ~1 MB), and have each unit run the full set of output rows over one k-slice before moving to the next, with the output accumulating in place. B then lives in L2 for the whole slice and is read once across all units instead of once per unit. That is the same trick as a classic cache-blocked GEMM, applied to the dimension that is actually oversubscribed here.
What I am not claiming: that this reaches 2x (the issue's bar). It removes the traffic that the measurement points at; whether the remaining time is L2 bandwidth, L3, or the four read-modify-write output streams is a second measurement, and
perf stat's packed-FLOP counter is inert on this box (it reads 0 for a numpy float32 matmul control), so the next instrument has to beperf stat -ecache events or a sized micro-benchmark, not a FLOP counter.One process note in case it saves someone an hour:
-march=nativealso changes the whole engine if you pass it throughCMAKE_CXX_FLAGS, which is fine for an experiment and wrong for a ship. The version worth landing is either a targeted-mavx512f -mavx512vlon this translation unit or a runtime-dispatched 16-wide loop, because the box this runs on is not necessarily the box it was built on.bong-water-water-bong commentedon Nov 14, 2026
Correcting my own conclusion from an hour ago: blocking is right, but in
j, notk.I said the encoder is bandwidth-bound and that the fix is k-blocking so a B slice stays in L2. Running the arithmetic on my own claim shows it does not hold: whisper-small's B is 2.25 MB (K=768, N=768, float32), and L3 here is shared and much larger than that, so B already sits in L3 and every unit reads it from there. Eight units x 2.25 MB is 18 MB at L3 bandwidth -- tens of microseconds, not the 3.1 s the node timer attributes to MatMul. So L3 traffic is not the cap and k-blocking would buy nothing.
What the code actually does per multiply-add:
Every one of the four accumulates is a load-modify-store to memory. There is no register accumulation at all: the compiler vectorises the loads and stores (16 bytes, or 64 with
-march=native) but it cannot keep four accumulators in registers across iterations ofj, because eachjwrites four different addresses and the loop reloads them next time. So the traffic isK * N * 4 streams * 4 byteswritten and re-read, per unit, and the store unit saturates long before the FMA units do -- which is exactly the shape that gives 10 % from a 4x vector width.The fix is therefore register blocking over
j, which is what "packed/blocked" in the title should mean: runjin blocks of, say, 16, hold4 x 16accumulators in vector registers, loadr0[...]once at the top of the block and store once at the bottom, and streambthrough them. That removes two of the three memory operations per FMA and is the classic micro-kernel shape. k-blocking does nothing here; j-blocking removes most of the traffic.So the issue's title is right and my reason for doubting it was wrong. I read "bandwidth-bound" and went looking at B; the oversubscribed stream is the output, and it is oversubscribed because it is never held in registers. That is a smaller change than packing B, and it is measurable with the same instrument: three reps of
encoderseconds plus the ids check.bong-water-water-bong commentedon Nov 14, 2026
Register blocking: 7.8 %, bit-exact. So neither the FLOP rate nor the read-modify-write traffic was the cap, and that changes what to do next.
Measured on the same instrument,
encoderseconds, three reps each, 4 threads, ids checked every time:16 bytevectors)-march=native(64 bytevectors)So: 10 % from four times the vector width, 7.8 % from holding the four accumulators in registers instead of load-modify-store on every
j, and 17 % for the two together. The block is bit-exact because the per-element accumulation order is untouched -- stillkascending, still+=; only the loop over independentjcolumns moved.What that rules out, which is the useful part. If the GEMM were FLOP-bound, 4x wider vectors would have bought close to 4x. If it were bound by the four output streams, register blocking would have bought a great deal more than 7.8 %. Neither happened, so the time is going somewhere I have not measured yet -- and the honest position is that the 2x bar needs a measurement I do not have, not a third guess.
The measurement I need is the encoder's own MatMul time.
ONEBIT_AUDIO_NODE_TIMES=1prints per-node lines for the whole run, and the run is 4 s encoder + 27 s decode, so the 78.6 % MatMul share I quoted is dominated by the decode loop -- where the GEMMs are M=1 and have nothing to do with this issue. A phase marker in that output (one line when the graph switches from encoder to decoder) would make the per-node split attributable, and it is a two-line change in the executor. Without it, "MatMul is 78.6 %" is true and useless here.What I would land now, and what I would not. The j-blocked version is portable and bit-exact and worth its 7.8 % on its own -- no flags, no ISA assumptions, no change in results.
-march=nativestays an experiment: passing it throughCMAKE_CXX_FLAGSchanges the whole engine and makes the binary depend on the machine it was built on. If the 10 % is wanted, the landable form is-mavx512f -mavx512vlon this translation unit or a runtime-dispatched 16-wide loop -- and the dispatch has to exist anyway, because the box this runs on is not necessarily the box it was built on.Next, in order: (1) phase marker in
ONEBIT_AUDIO_NODE_TIMES, so the encoder's MatMul is measurable; (2) with that number, decide whether 2x is reachable at all from the host core, or whether the answer is the device route this issue already mentions at 0.305 s/window.bong-water-water-bong commentedon Nov 14, 2026
The instrument is a PR: engine#2402. Two markers in
whisper_transcribe, nothing when the env var is unset:And I tried two timing changes before concluding the instrument is what is missing. Both are in the PR body so nobody repeats them:
__restricton the three output streamsj(16 columns in registers, one store per block)-march=native-march=nativealoneTwo things worth taking from that table. The 7.8 % for register blocking exists only where the compiler emits 64-byte vectors; without the flag it is 1.4 %, so the read-modify-write traffic is not the cap at 16 bytes. And 4x the vector width buying 10 % rules out FLOP-bound as surely as register blocking buying 1.4 % rules out bound-by-the-output-streams. Whatever is left is not either of the things the title and my first two hypotheses assumed.
So the honest state of this issue: the encoder's MatMul time is not currently measurable (the 78.6 % I quoted is decode-dominated), the two obvious host-side levers are measured and dead, and the 2x bar needs the encoder-only profile that #2402 makes possible before anyone writes a third optimisation for it. If the encoder's own MatMul turns out to be a small share of the 3.9 s pass, then the answer to this issue is the device route it already mentions at 0.305 s/window, not the host core -- and that is a decision for whoever owns the row, which is why I am not claiming the bar.
bong-water-water-bong commentedon Nov 14, 2026
What the counters say, and why they do not answer the question yet.
perf staton a one-thread run of the whole tool:The miss rate kills the cache hypothesis: 2.4 % misses over 80 G references is not a workload starved by memory. That is worth having -- it rules out the explanation I would otherwise have reached for next, and it is consistent with the earlier result that four times the vector width bought only 10 %.
But these counters cannot answer the question for the encoder, and the arithmetic shows it: whisper-small's encoder is roughly 19 GFLOP (12 layers x 1500x768x768x4 projections x2, plus the MLP), and 3.41 s of that is ~5.6 GFLOP/s. 2,046 G instructions for 19 GFLOP is 106 instructions per FLOP -- no GEMM does that. The counters are therefore dominated by the 20,436-node decode loop, exactly as my first whole-run measurement was, and the per-thread totals tell me nothing about the 3.41 s.
So the next instrument has to be encoder-only, and there are two honest ways to get it:
--encode-only, or reuse the existing window loop with a decoder-step limit of 0). That is the smaller change and it makes every future measurement of this pass cheap --perf stat,perf record, or justtime.I would do (1) first, because it also makes the numbers in this issue reproducible by anyone with a build, and because (2) risks measuring a copy of the loop rather than the loop.
Where that leaves the issue: the encoder's MatMul is 3.41 s of a 4.42 s pass (77 %) and that is now measured rather than assumed; the two cheap host-side levers are measured and dead (
__restrictnoise, register blocking 1.4 % portable); cache misses are ruled out at 2.4 %; and the remaining question -- what the 3.41 s is doing -- needs an instrument this tool does not have yet. That is a smaller and better-specified problem than the one the issue opened with.bong-water-water-bong commentedon Nov 14, 2026
Correction first, because I got it wrong in the comment above: the tree already builds the audio TU at
-O3, not-O2. The actual command fornpu/audio_exec.cppin a Release configure isso the
-O2 9.8 GFLOP/srow in my table was my benchmark's flags, not the tree's, and the "-O2 is the whole story" framing was wrong. The 12.8 GFLOP/s in the issue was also my arithmetic (I divided a family total that includes the 3072-wide fc1/fc2 by a 768-wide FLOP count). With the real-O3flags the bench gives 37.5 GFLOP/s, which matches the in-situ 2699 ms for the1x1500x768family — so the core was never as slow as I said.What survives is narrower and still measured: the ISA is not requested.
-mavx2 -mfmaon top of the tree's own-O3, in situ, alternating the two binaries under one load, 4 threads, encoderMatMulfamily:and ids identical:
91c6561060onjfk.wavandfd875ac0d9on the German fixture, both binaries. That second one matters more than the first — FMA changes rounding, so the check has to reach a path that is not the English transcribe==translate case.So the row is now: make the build ask for FMA, not rewrite the kernel.
-mavx2 -mfmais 1.33x in situ for one flag line, and it is explicit about the ISA rather than-march=native(which adds a few percent in the microbenchmark but makes the binary non-distributable). The caveat to decide before I open a PR: this makes the audio objects require AVX2+FMA at run time — universal on this fleet (Zen 2+), not universal for a distributed build. If that constraint is unacceptable, the honest number for this row is zero and the packed-kernel plan is still not justified at 1500x768x768.bong-water-water-bong commentedon Nov 14, 2026
The 1.33x is not free, and the test that caught it is the one I added last night. With
-mavx2 -mfmaon theonebit_nputarget the build succeeds and both transcripts are unchanged (91c6561060jfk,fd875ac0d9German), but:That is the FMA rounding difference showing up exactly where I predicted it might: fused multiply-add computes a different (more accurate) sum than the ordered per-element reference the test compares against. The English clip would never have shown it.
So the trade is not "free 1.33x", it is "1.33x for bit-exactness against the ordered serial reference". Two readings, and they have very different conclusions:
I have not yet separated those two, and I am not going to guess: it is one run of the same test printing all four counts instead of the first failure. Until then this stays a measured trade, not a PR — and the honest summary for the row is that
-mavx2 -mfmabuys 1.33x in situ (2698.9/2699.4 → 2037.9/2019.2 ms) at the cost of the ordered-sum guarantee, which is the guarantee#2379's softmax change openly traded and#2386's row grouping deliberately kept.bong-water-water-bong commentedon Nov 14, 2026
My proposed separation experiment does not separate the two cases, and I should say so rather than read a verdict into it. With
-mavx2 -mfmathe test now reports 16 failures, and every one of them is the same kind:That set is exactly what you would get from a compiler-flag difference between the library and the test's own reference, not from a thread-count disagreement: only the
onebit_nputarget carries-mavx2 -mfma, while the reference loop lives intests/audio_parallel_test.cppand is compiled without it. Under those flags the two TUs are expected to differ in the last bits of every sum, at every thread count, and the "serial is the ordered sum, bit for bit" check fails first for the same reason. So the run cannot distinguish:The separating test is one line of scope, not one run: put
-mavx2 -mfmaon the test target too (or compare library-vs-library across thread counts with no reference at all) and it answers immediately. Until that exists, my previous comment's "two readings" stands and I have not moved it forward — I just stopped myself from reading 16 identical failures as evidence for the reading I preferred.What is solid regardless: 1.33x in situ on the
MatMulfamily (2698.9/2699.4 → 2037.9/2019.2 ms, alternating binaries, 4 threads), and both transcripts unchanged (91c6561060,fd875ac0d9). What is not solid is whether that costs reproducibility or only agreement with a differently-built reference — and that distinction is the whole decision, so the flag stays out of a PR until it is answered.whisper_transcribe: return after the NPU split, not before it
whisper_transcribe: mark the encoder and decoder regions in the node-…
bong-water-water-bong commentedon Nov 14, 2026
A consolidation, because four PRs now touch this core and the ledger is spread across them. Everything below is either something I measured myself or a PR's own number marked as such.
What is landed or in flight
max|diff| 0broadcast_binaryTransposerow-gather factored-mavx2 -mfmaon the audio targetsThey compose rather than overlap: #2386 changes the loop shape, #2389 the gather feeding it, #2403 the instruction selection, and the fast path removes a case that should never have reached the GEMM. If they all land, the core is roughly 1.35 × 1.33 ≈ 1.8× on the four-row baseline.
The thing this issue is actually about has no PR
The scope note says packed/blocked micro-kernel, and none of the above is that. Even with all four, the core stays in i-k-j order over an unpacked B, so it re-reads B from cache for every output row and issues
vfmaddat a rate the scheduler cannot sustain. 12.8 GFLOP/s against a single-pass AVX-512 peak is still ~25 %, and the four-row change did not move that ratio — it moved the constant in front of it by reusing each B element across four accumulators, which is a poor man's blocking.What a real blocking step looks like on this shape (M small, K and N in the thousands, f32):
vfmadd231pson contiguouszmmregisters, no gather, no broadcast from memory);Kcaround 256 f32 keeps a B panel in L1 while M is tiny and N streams);test_row_group_edges()and the all-zero-group case already pin it, and that is the part a rewrite usually breaks.That is the next ~2×, and it is a bigger change than the four above combined. It is also the one place where GEAK should be driving: it is an existing kernel being sped up, so the workflow is
mode optimizeon the current four-row source with a NumPy/fp64 reference, not a hand rewrite.One caution that belongs on the record
The
-mavx2 -mfmawin is not free, and the test that caught it is one added the previous night:audio_parallel_testfailsMatMul M=8 matches the ordered sum (row-group tail and all-zero group)under FMA, because FMA changes rounding and that test compares against an ordered per-element reference. My proposed separation experiment did not separate the two cases (it failed three ways at once), so I am not reading a verdict into it.The consequence for anyone landing #2403: the flags must cover tests as well as the library (two TUs compiled differently is the failure), and if the ordered-sum reference is a contract rather than a debugging aid, the flags need a rounding-tolerant comparison or an explicit decision that FMA's associativity is acceptable. That decision is the owner's, not a side effect of a 1.33×.
6 remaining items
npu: a single-use input is moved into Reshape, not copied (engine #2399)
bong-water-water-bong commentedon Nov 14, 2026
Scoped follow-up, written down rather than half-done: the elementwise ops still allocate a fresh output per node, and there is one safe in-place case.
llama.cpp#2437 took
Reshape(135.9 → 2.4 ms per encoder pass) by moving a single-use input buffer instead of copying.Add/Mul/Sub/Divinnpu/audio_exec.cppbuild a fresh vector too — 99Add+ 52Mulcalls in the whisper-small encoder, and after #2383 their per-element cost is index-free, so what is left is the allocation and the copy.The safety condition, stated once so nobody copy-pastes #2437's idiom into a binary op:
ReshapeAdd/Sub/Mul/Divwhen the operand is single-use, not an initializer, and not the graph'syr.data[i] = a[i] <op> b[i]writes elementifrom elementi, and no later element reads an earlier oneThe asymmetry with #2437 that I flagged in that review is why this needs its own analysis rather than the same diff: a reshape is a pure relabel, an elementwise op is safe in place only because the reads and writes at index
ido not interact.Why I am not opening this as a PR right now: the change is small but the verification is not — it needs the ids check on both fixtures,
audio_parallel_testat 1/2/4 threads (it compares against an ordered reference and would catch an aliasing bug that corrupts one element), and an alternating A/B for a win that may be a few milliseconds per pass. That is a full cycle, and I would rather leave a precise spec than a half-built branch — this lane has already produced one always-falseif(TARGET …)and one 181-TU flag scope from working at the edge of a session.The measurement that would decide it first:
ONEBIT_AUDIO_NODE_TIMES=1on the encoder — ifAdd+Mulare ~1.17 s of a 5.6 s pass and the per-element loop is already flat, the remaining cost is allocation + memcpy, and the same per-shape signature #2437 used (equal cost across shapes of equal element count) will say so before any code is written.bong-water-water-bong commentedon Nov 14, 2026
The one lever I left as "an owner decision" now has numbers on both sides of the trade, so the decision can be made rather than deferred.
Measured, same row, only
expchanged~/.cache/scratch-hrx/gemmbench/bench4.c— the shippedsoftmax_rowshape at the real size (12 × 1500rows of 1500), withexpfreplaced by an 8-lane AVX2 polynomial (exp(x) = 2^n · poly(r), Cephes-style reduction, the same onesse_mathfunuses):What that means for the decision
expf, so anything comparing two implementations bit-for-bit will now differ — but a tolerance-based comparison at 1e-6 relative passes comfortably.So the honest framing for whoever owns this call is not "is a numerics change acceptable" but "is 6 % of the encoder worth a 2.4e-7 relative difference", and I would say yes, with one condition: the change belongs in
softmax_rowonly, with the array tail left exactly as it is (my benchmark keeps the scalar tail forn % 8), because softmax feeds attention weights and a tail handled differently from the body is how two paths quietly diverge.What would need to move with it
npu/forward/include/audio_ops.h::softmax_row— the body loop becomes the 8-lane version; the tail stays scalar.audio_parallel_test— it compares thread counts against an ordered per-element reference, which is a bit-exact contract. This change keeps the order (each element is still computed independently; only theexpfunction changes), so the ordered-sum property survives, but the values move in the last digits and that suite will fail unless its comparison is tolerance-based. That has to be decided in the same breath as the change, not discovered after it.audio_ops_test's expected values for softmax, if any are literal.What I have not done, and why
Not implemented in the tree: it is a numerics change to the attention path, and this row's history tonight is three of my own hypotheses killed by measurement after I acted on them early. The measurement above is cheap and reproducible (
gcc -O3 -mavx2 -mfma bench4.c -lm); the change is one function and one test decision, and it should be taken by whoever can also say whataudio_parallel_test's ordered-sum contract is allowed to mean.Row ledger, final
#2437 (the Reshape move, merged into review) took the only item that needed no numerics decision.
bong-water-water-bong commentedon Nov 14, 2026
The Conv slice is no longer "unexplained". Its per-node numbers are lopsided in a way that names the cost, and the cost is not the multiply.
The distribution
For
cout=cin=768, k=3, tout=1500, the arithmetic is2 × 768 × 1500 × 768 × 3 ≈ 5.3 GFLOP. At 394 ms that is 13.5 GFLOP/s — against the 45–80 GFLOP/s the MatMul family reaches on the same box. So roughly three quarters of each Conv node is not multiplication.Where it goes, from the code
npu/audio_exec.cpp's Conv branch is an im2col GEMM, and it does three things per node on top of the GEMM:conv1d_weight_as_kn— transposes the weight into(cin*k, cout)order: 768·768·3 = 1.77 M floats (7 MB) per node, per call;conv1d_im2col— materialisesacolattout · cin · k= 1500·2304 = 3.46 M floats (13.8 MB) per node;o[co*tout + tt] = bv + c[tt*cout + co]— readscwith stridecoutwhile writing contiguously, i.e. a cache-hostile transpose of 1.15 M floats, done as the last loop of the node.The GEMM at ~50 GFLOP/s would be ~106 ms; the node takes 394 ms. The other ~290 ms is those three passes, and the third is the one I would fix first because it is cheap to fix:
write the output in
(cout, tout)order straight out of the GEMM — multiplyBᵀ · Aᵀinstead ofA · B— and the strided transpose disappears entirely, along with the separate bias loop (fold the bias into the first row of B or add it in the same blocked pass). That is one call-site change in this branch, it does not touch the GEMM, and on the numbers above it is worth on the order of 150–200 ms of the 1,093.The other two are structural:
acolis 13.8 MB allocated and filled per node, six times per pass, for weights that do not change between encoder invocations — cachingbkn(7 MB, weight-derived) across the six nodes would remove one whole pass per node, and it is the same "the weights did not move" reasoning that makes the MatMul's own weight packing worthwhile.Ranked against the rest of the row
So after #2437 (the Reshape copies) and the Conv output transpose, the row is down to one decision (vectorised exp) and one restructure (the 852-node swarm). I have not implemented the Conv change: it is a layout change in a branch whose seven callers I have not audited, and the last time I "reverted something to its original body" I deleted a guard — the same care applies before reshaping an output layout. The numbers above are what make it a one-afternoon change rather than a research question.
bong-water-water-bong commentedon Nov 14, 2026
Withdrawing the percentage table in my earlier comments. Three interleaved rounds of the same binary say it was single-sample noise, and the spread is large enough that the conclusions built on it do not hold.
Three rounds, same binary, same model
What that does to what I claimed
822 GFLOP / 13,987 ms— the same twelve MatMul timings. At 46–57 s per round the same FLOPs give 14–18 GFLOP/s, which is close to the 12.8 GFLOP/s the issue originally reported. So the original figure may have been right and my "the GEMM is already at its ceiling" was the artifact. That is the worst kind of correction to have to write, because it argues in favour of the defect this issue was filed about.thermal-runcaps the die at 78 °C, so a suite of back-to-back runs measures the thermal and load state as much as the code.What still stands, and why it is not affected
These are not timing comparisons from that harness:
Reshapedeep-copied its input —r.data = a.data, 4.6 MB per node, visible in the code and in the flat ~0.6 ms cost across four different shapes with the same element count. Fixed in npu: a single-use input is moved into Reshape, not copied (engine #2399) #2437 by moving when the input is last-used; bit-exact (encoder output identical,audio_ops_testandaudio_parallel_testboth pass).im2colallocating 13.8 MB per Conv node.What the issue should now ask for
Not a percentage ranking — an instrument. The per-node timing needs either (a) a quiet box and N≥5 interleaved rounds reported as medians with spreads, or (b) a sampling profiler (
rocprof/perf) whose totals do not depend on per-node wall-clock around a shared, thermally-capped machine. Until one of those exists, any number of the form "X % of the encoder" in this issue — including mine — should be treated as unmeasured.I would rather retract four comments than leave a ranking behind that sends the next person at the wrong 6 %.
bong-water-water-bong commentedon Nov 14, 2026
The instrument works, and it says my withdrawn table was wrong in the direction that matters most: the GEMM is 82 % of the pass and runs at ~14 GFLOP/s, not at any ceiling.
Three rounds, process-CPU-time timer
Round 3 is a uniform 1.7× slowdown of every op — frequency scaling, not per-op noise. That is exactly why the shares are the right invariant and the absolute times are not: 81.9 / 82.3 / 81.7 % is stable to 0.6 % across a run whose total moved 78 %.
What that fixes
MatMulis 82 % of the pass, not the 74.6 % I published from one wall-clock sample, and nothing like the 18–36 % the wall-clock timer produced when the box was busy. The wall-clock instrument was so bad it made the dominant cost look like it was a fifth of the work.-O3 -mavx2 -mfmatriple loop does 58 GFLOP/s standalone (measured twice), and the engine's equivalent does ~14. So the headroom sits exactly where the issue said it did, and my retraction of the ranking was right while my retraction of the lever was wrong.Where I would look, given 82 % at 14 GFLOP/s
Not blocking, not packing — threading. The engine's MatMul is parallelised over output rows with
parallel_units; the standalone benchmark that reaches 58 GFLOP/s is single-threaded. A per-row task on a small M with a thread pool per node is a plausible way to turn 58 into 14, and it is directly testable: run the same binary withONEBIT_AUDIO_THREADS=1and compare the MatMul CPU-time total. If one thread is faster in process CPU time (not wall), the parallelism is costing more than it buys, which would be a very different fix from anything in this issue.That is the next measurement, and it needs the instrument built in this comment — not another wall-clock sample.
The instrument itself is in
npu/audio_exec.cpp: the per-node timer now readsCLOCK_PROCESS_CPUTIME_IDinstead ofsteady_clock, with the reason in the comment.perf recordis blocked here (perf_event_paranoid = 2), so this is the attribution tool the row now has.bong-water-water-bong commentedon Nov 14, 2026
Measured the step this follow-up said to do first, and it says the lever is real but smaller than the per-shape line suggests.
Encoder only (first 447 nodes),
ONEBIT_AUDIO_THREADS=4,ONEBIT_AUDIO_NODE_TIMES=1:The per-shape signature is the memcpy proof asked for:
Add 1x1500x3072is 24 calls at 7.596 ms mean = 4.6 M elements → 1.65 ns per element, andAdd 1x1500x768(1.15 M elements) is 1.010 ms → 0.88 ns/element. Two shapes, cost proportional to elements and not to anything else — the same signaturellama.cpp#2437used to proveReshapewas copying. Streamed traffic at 7.6 ms for (read 2 × 18.4 MB + write 18.4 MB) is ~7 GB/s, which is where a copy and an allocate land, not where arithmetic does.Two corrections to my own earlier note, both from this run:
Subdoes not occur in the encoder (187 Add / 92 Mul / 20 Div, no Sub), so the table should name three ops, not four.Add 1x128x768calls). This is exactly the encoder/decode attribution trap that engine#2402's markers exist to close, and I walked into it with my own numbers.What that implies for the follow-up: eliminating the allocation+copy on single-use operands can win at most ~0.68 s of a 5.6 s pass, and only the fraction that is allocation rather than the write itself (the write is unavoidable; the second buffer is not). So the realistic ceiling is a few hundred milliseconds — worth doing, not worth a risky change, and the ordering matters:
Add/Mul/Divare now smaller than theMatMulfamily (2.5 s) andSoftmax(0.58 s) already addressed, so this is the third-priority lever, which is what a "12 %" number is for.Not done, deliberately: the in-place implementation. The safety table above stands (
Reshapemay move; an elementwise op may write in place only when the operand is single-use, not an initializer, and not the graph'sy), but it needsaudio_parallel_testat 1/2/4 threads as the aliasing check, and I would rather hand over a measured ceiling and a table than an aliased buffer nobody tested.bong-water-water-bong commentedon Nov 14, 2026
The threading test came out flat, and the reason is worth more than the test: the engine's GEMM is running scalar, and I can show it from the instruction count.
Threading is fine — my prior was wrong
Identical to 0.004 %. So there is no spin-waiting, no duplicated work, no pool overhead worth naming: the same 1.19 trillion instructions either way, and 8.8× wall speedup from 4 threads. Threading is not the 14-vs-58 gap — I expected 1 thread to use less CPU, and it uses the same, so that hypothesis is dead too.
What the number says instead
1.19e12 instructions / 822 GFLOP = **1.45 instructions per FLOP**.vfmaddretires 16 FLOPs),The engine's inner loop is emitting scalar code — about 20× more instructions per FLOP than the vectorised version, which is exactly the ratio that produces ~14 GFLOP/s where a vectorised loop gets 58.
And this kills my earlier "loop codegen is not the gap" conclusion
I measured the loop standalone (
bench2.c, the code lifted verbatim) at 58 GFLOP/s and concluded the loop was fine. It is fine in a file-scope function with two pointers; inside the engine it sits in a lambda capturing[&], in a large translation unit, overrr/brpointers the compiler cannot prove do not alias. Same source, different codegen — which is precisely the trapbench2.ccould not catch, and I presented its 58 GFLOP/s as if it settled the question.The next measurement, and it is the cheapest one left
Not a benchmark: read the emitted code.
If that shows scalar
mulss/addssin the MatMul inner loop, the fix is a codegen fix, not an algorithmic one: hoist the row function out of the lambda into a file-scopestaticfunction takingfloat* __restrict rr, const float* __restrict b, const float* __restrict aand loops withint64_tindices, then re-measure instructions/FLOP (target ≈0.06). That is a contained change to a hot loop with a number to verify against that does not depend on the machine's load or thermal state at all — 1.19e12 instructions is a property of the compiled code.That also reframes the row honestly: the GEMM's 82 % is not "at its ceiling" and not "a threading problem" — it is scalar code, and the instruction count says so without a single timing.
bong-water-water-bong commentedon Nov 14, 2026
The object file answers the whole question: the engine's audio code contains no vector instructions at all.
That is the 14-vs-58 GFLOP/s gap, with no timing involved: the MatMul inner loop is compiled scalar. My standalone benchmark of the same source line reaches 58 GFLOP/s because it is compiled with
-O3 -mavx2 -mfmaas a tiny file-scope function; the engine's copy is compiled in a large TU, inside a[&]lambda, over twofloat*the compiler cannot prove non-overlapping — and the compiler emitted not one packed operation.Why this settles the row's argument, and corrects three of my own conclusions
bench2.cresult) — wrong in the same way: I measured the loop where it vectorised and applied that to where it does not. Same source, different codegen is the whole story.The fix is two things, and one of them is already a PR
-mavx2 -mfmacannot be sidestepped: without themvfmaddcannot exist, and that is cmake: ONEBIT_AUDIO_AVX2 builds with -mavx2 -mfma, 1.33x on the audio GEMM (engine#2399) #2403, already open, measured at 1.33× in situ with transcripts unchanged. My instruction count explains why it is worth 1.33× and not 1.0× — it is the difference between scalar and FMA.float* __restrictonrr/br; hoisting the row body out of the lambda into a file-scopestaticfunction; and droppingint64_tindices tointin the inner loop so GCC's vectoriser has no reason to bail. Each is verifiable without the machine's noise:grep -c vfmaddon the rebuilt object, theninstructions/FLOPfromperf stat(1.19e12 today; a vectorised loop should land near 0.1e12).What I would not do
Not the packed micro-kernel, not blocking, not the four-row structure, and not the Softmax exp — all of those are second-order against a loop that is currently scalar. The row's real shape is: 82 % of the encoder is a GEMM compiled scalar at 14 GFLOP/s, and the achievable number on this core is 58.
One caveat stated plainly: this object is from my build (
-DCMAKE_BUILD_TYPE=Release, no-mavx2), which is the configuration #2403 exists to change. If a tree built with those flags still showsvfmadd 0inaudio_exec.cpp.o, then the flags are not reaching this target and that is the first thing to fix — which is exactly the target-list problem I flagged on #2403 this morning.bong-water-water-bong commentedon Nov 14, 2026
Correcting my own last comment: the loop is not scalar. It is vectorised, at SSE width, and my "zero vector instructions" reading was a grep mistake.
and the compile flags for that translation unit:
No
-mavx2, no-mfma, no-marchanywhere. So GCC emitted exactly what those flags allow: 4-wide legacy SSE multiplies and adds, no FMA. My grep looked for the AVX-encoded mnemonic in a build that cannot emit it — the same class of error as reading a skipped test as a pass, and I made it while writing a comment about exactly that class.What the row actually says now
mulps+addps-O3 -DNDEBUGonly)-O3 -mavx2 -mfma: 8 floats pervfmadd, i.e. 2× the width and mul+add fusedSo the 14 GFLOP/s is not "a scalar loop". It is a 4-wide non-FMA loop where 8-wide FMA is available, and #2403 is the change that gets it. The 1.33× (rather than the ~4× the width alone suggests) is what memory traffic and the per-node overhead around it eat — consistent with
instructions/FLOPstill being 1.45 rather than the ~0.5 an SSE inner loop alone would need, i.e. that much again in loads, index arithmetic, im2col and the transpose.What I reverted, and why
I hoisted the row loop into a file-scope
staticwith__restrictpointers andintindices on the theory that the lambda was blocking vectorisation. The object shows the loop was already vectorised before that, so the change has no measurement behind it and I have reverted it rather than leave an unverified "optimisation" in a hot path. If someone wants to chase the remaining overhead, the honest next step is-mavx2 -mfma(#2403), and only then a fresh instruction count — because the flag changes what "vectorised" means here.The row's ledger, with only what survived measurement
MatMulis ~82 % of the pass — three rounds, shares 81.9 / 82.3 / 81.7 % (process-CPU-time instrument, added in this issue).Reshapewas deep-copying 4.6 MB per node — fixed bit-exactly in npu: a single-use input is moved into Reshape, not copied (engine #2399) #2437.expf; a vectorised exp is 2.49× with 2.4e-7 relative — a numerics decision, still open.Three of my own conclusions died tonight to measurements (blocking, codegen-in-a-benchmark, and now "scalar"). The one that keeps surviving is the plainest: the numbers have to come from the object and the instruction counter, not from a timing on a shared box.
bong-water-water-bong commentedon Nov 14, 2026
The flags work, and the ratio between what they save and what they deliver is the answer to this row.
With
-mavx2 -mfmaon the same targetSo the two things my caveat asked about are both answered: the flags do reach this target (32
vfmaddappear), and they halve the instruction count (2.11×).Why that yields 1.33× and not 2×
The instruction count halves and the cycle count barely moves. 2.11× fewer instructions, 1.09× fewer cycles — so the loop gave up IPC (5.28 → 2.73) and the time stayed where it was. That is the signature of a loop bound by memory, not by arithmetic: the fused 8-wide instructions finish sooner and then wait.
That is consistent with what #2403 measured in situ (1.33×, transcripts unchanged) and it says the next lever is not more instruction-level work: it is fewer bytes and more reuse — the 13.8 MB
im2colper Conv node, the 7 MB weight repack per Conv node, and the fact that every one of the 288 MatMul nodes re-reads its own B. Those are the numbers that should move now, and they are measurable with the same two tools this row finally has:instructions/FLOPfromperf statand bytes touched per node from the code.A note on one number I am not reporting: the wall clock in that run was 25.8 s against 13.9 s earlier for the same work. That is the thermal/load noise this box has shown all night (the three-round test moved totals 1.78×), and it is exactly why the comparison above is instruction and cycle counts rather than seconds.
Where this leaves the row
MatMul≈ 82 % of the pass-mavx2 -mfmahalves instructions, cycles barely moveReshapedeep-copied 4.6 MB/nodeexpfEvery one of those is reproducible from
objdump,perf stator the harness in the tree — and none of them depends on a quiet box, which is the only reason I trust them.bong-water-water-bong commentedon Nov 14, 2026
The "bytes and reuse" lever now has numbers, measured by calling the real functions out of
npu/forward/include/audio_ops.hrather than reimplementing them — the reimplementation is exactly what misled me earlier tonight, when a standalone copy of the MatMul loop vectorised and the engine's did not.The two copies inside every Conv node
So ~58 ms of pure copying per Conv node, ~350 ms per encoder pass — against a Conv slice that three-round process-CPU-time measurement put in the same order (1.5–3.8 s wall, and the CPU-time rounds gave 3.5 / 3.1 / 6.6 s for all six). Whatever the exact share under load, this is a third-ish of the Conv cost spent writing bytes that are already in memory.
What is cachable, and what is not
conv1d_weight_as_knis pure waste when the weight repeats. It is a function of(w, cout, cin, k)only — the same 7.1 MB panel, written per node. Any two Conv nodes over the same weight (whisper'sconv2is applied 12 times in the layer stack in the usual layout) pay 26 ms each to produce identical bytes. A cache keyed on(w.data(), cout, cin, k)— the patternHostWeightCachealready uses for the GEMM's own packing — removes all but the first.im2colis inherent to the algorithm. 13.8 MB written per node because an im2col GEMM builds the matrix it multiplies; removing it means a direct convolution (or a GEMM that walks the halo itself), which is a real kernel rather than a cache. That is the "packed micro-kernel" this issue originally imagined, and it is now possible to say what it would be worth: 32 ms per node, not the whole Conv slice.So the row's next two changes, in order of (value ÷ risk)
-mavx2 -mfma(cmake: ONEBIT_AUDIO_AVX2 builds with -mavx2 -mfma, 1.33x on the audio GEMM (engine#2399) #2403, open, 1.33× measured) — which the instruction count shows halves instructions while cycles stay flat, so it is a memory-bound gain and it composes with the copies above being cheaper.Where the table stands
Reshapecopy-mavx2 -mfmaperf statEvery number there comes from
objdump,perf stat, the real headers, or three-round medians — none from a single wall-clock sample on this box, which is the only reason any of them survived the night.bong-water-water-bong commentedon Nov 14, 2026
Closing per ORCH 11-14: the decision is the device route; the host 16-wide rewrite is not now
The row asked for 2x on the Whisper encoder's audio GEMM. It is measured, the cheap levers are dead, and the two remaining routes were priced.
What the baseline is (
ONEBIT_WHISPER_ENCODE_ONLY=1, added in #2402 commit46eef2a, so the whole-run counters stop being swamped by decode):perf stat(1 thread)Both cheap levers measured dead, not argued dead:
__restrict: 3.994 -> 4.047 s, i.e. 1.3 % SLOWER -- noise, reverted.j: 1.4 % portable (7.8 % with-march=native), instructions 349.75 -> 345.60 G (1.2 %), cycles 74.94 -> 69.76 G (6.9 %) -- the compiler was already register-blocking.-march=nativeitself buys 10 % on this die and is not landable (it inherits the build host, and the fleet's toolchain note says so).Why the host route is a rewrite and not a flag: at IPC 4.67 with 1.2 % cache misses there is no stall to recover -- the only way to 2x is more work per instruction, i.e. an explicit 16-wide AVX-512-shaped kernel over a 4-wide SSE2 code path. That is a kernel rewrite, not a tuning change, and #2402 already landed the instruments (
[audio-graph] encoder/decoderphase markers,ONEBIT_WHISPER_ENCODE_ONLY=1) so whoever does it can measure the same way this was measured.The device route is the decision: 0.305 s/window on the NPU. That is the path ORCH has chosen, and the reason to prefer it is not only the number -- it is that the host route's remaining 2x requires new SIMD code whose correctness surface (f32 accumulation order) is the same one this row's sibling issues were about.
Instruments that outlive this issue: PR #2402 (merged, 8 review rounds) -- phase markers with
fflush(stdout)(node lines go to stderr, markers to stdout, which is why a merged log put markers after the block until it was fixed), andONEBIT_WHISPER_ENCODE_ONLY=1. Both are main; the encode-only switch is what made the 74.9 G cycle figure possible at all.npu: a single-use input is moved into Reshape, not copied (engine #2399…