Skip to content

npu: the host audio GEMM core runs at 12.8 GFLOP/s (~25 % of single-pass peak) — packed/blocked micro-kernel #2399

Closed

Description

@bong-water-water-bong

Scope note first: engine#2366 is the encoder's non-GEMM half (Conv/Softmax/attention/elementwise). The projections are the other 61 % of the encoder, so the lever measured while closing #2366 has no home — filing it here rather than leaving it in a comment.

Where it stands, measured

The host GEMM core (npu/audio_exec.cpp, the MatMul branch) accumulates with one output row per parallel unit in i-k-j order, no packing. After engine#2386 (four rows per unit) it runs at:

one row   189.8 ms (9.3 GFLOP/s)     four rows  138.7 ms (12.8 GFLOP/s)   1.35x, max|diff| 0

and in situ on the whisper-small encoder, alternating the two binaries under one load: MatMul 3367/3283 ms → 2518/2530 ms (1.32x), ids md5 unchanged.

12.8 GFLOP/s is roughly 25 % of what four AVX-512 cores do in a single pass. The measured reason the cheap fix stopped there: sharing the streamed weight row across four output rows removes the traffic, but then the four live accumulators become the limit — a memory-traffic argument predicted 3x and the hardware gave 1.35x. Going further needs packing/blocking: a packed micro-kernel (B panel packed once per block, k-blocked, register tiles) rather than the current rank-1 updates over a streamed row.

What "done" would look like

  • Bit-exactness: transcript ids md5 91c6561060 on jfk.wav and the German fixture unchanged; audio_parallel_test green at ONEBIT_AUDIO_THREADS=1,2,4 (it compares each thread count against an ordered serial sum, and test_row_group_edges covers M<4 and the all-zero group).
  • Speed: ≥2x over the current core, stated as an alternating A/B under one load. Non-negotiable: four consecutive runs of the same binary on this box have spanned 2.8–7.6 s as load moved, so a cross-window comparison can "prove" either direction.
  • Instrumentation already exists: ONEBIT_AUDIO_NODE_TIMES=1 (engine#2377) prints one line per node with its output shape, so the per-shape split (96 MatMuls, 1x1500x768 ×60 for q/k/v/o+fc2, 1x1500x3072 ×12 for fc1) needs no new tooling.

Why it is worth a row at all

The device route replaces the projections at 0.305 s/window (engine#1783), so the host GEMM is not the only path to closing that row — but the device route needs the ONNX-encoder route that does not exist, while this is a contained host change with an existing bit-exactness harness. It also benefits Conv, which is GEMM-bound (2 calls, 343 ms of the encoder, and it goes through this same core).

Activity

  1. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claiming this lane (pi-i679). I reviewed both halves that led here -- #2386's four-rows-per-unit and #2389's row-at-a-time Transpose -- so the measurement chain is familiar, and this issue names the lever both of them stopped short of.

    What I will try. The issue's own arithmetic puts the ceiling at four AVX-512 cores in a single pass, so the target is a packed/blocked B rather than another loop-shape tweak: pack B into k-blocks so a block of b[k] rows is read once and used by every output row in the unit, with the packed layout chosen so the inner loop is contiguous on both sides. The two prior changes fixed the streaming and left the access pattern alone; this one is about the access pattern.

    The arms I will hold myself to, because the last two changes in this file set the bar and the issue already states them:

    No device needed to develop: this is npu/audio_exec.cpp's host branch, so the work is CPU-side under thermal-run; the only device question is whether the whisper ids still match, which wants one window at the end rather than a ticket to develop in.

  2. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    This issue's premise is wrong, and the measurement that corrects it is worth more than the micro-kernel it asked for. Two findings, both reproduced twice, same source, one process, load could only hurt.

    1. The same kernel runs at 9.8 or 69 GFLOP/s depending only on compiler flags

    -O2  (what the tree uses)       cur4 180.0 ms   9.8 GFLOP/s
    -O2 -mavx2 -mfma                cur4 135.9 ms  13.0 GFLOP/s
    -O3                             cur4  47.2 ms  37.5 GFLOP/s
    -O3 -mavx2 -mfma                cur4  27.5 ms  64.3 GFLOP/s
    -O3 -march=native               cur4  25.5 ms  69.3 GFLOP/s
    

    7× from the flag line alone, on the exact loop I proposed to replace. The 12.8 GFLOP/s figure in the issue was the in-situ -O2-built binary, so it was never measuring the algorithm — it was measuring that the vectoriser was not asked to do FMA. A packed/blocked rewrite of a core that is being compiled without FMA would have been optimising the wrong thing, and I would have spent a project finding that out.

    2. The packed prototype I built first is worse, not better

    cur4   25.1 ms (70.5 GFLOP/s)      packed  180.6 ms (9.8 GFLOP/s)      0.14x    max|diff| 0
    

    The naive panel copy (per column block, per k block) costs more than the register tiling saves at this size — 1500×768×768, where B entirely fits in L2 anyway. So at this shape the win is not available from packing; it may be at much larger K/N, but that is not this workload.

    What this means for the row

    • Cheapest real lever, one line, no algorithm change: build the audio TUs with -O3 and FMA. -O3 alone is 3.8x over -O2 and needs no -march contract; -O3 -mavx2 -mfma is 6.6x and states the ISA explicitly (fleet is AVX2/FMA everywhere since Zen; -march=native adds 7 % and a distributability problem, so I would take -mavx2 -mfma).
    • Correctness cost: none expected — same source, same order in k, and FMA changes rounding. So the bar is the one already written into this issue: ids md5 91c6561060 on jfk and the German fixture, audio_parallel_test green at 1/2/4 threads. FMA fusing is exactly the kind of change that can move a last bit, so it must be checked, not assumed.
    • A packed micro-kernel is NOT justified at this shape by the above, and this issue should say so rather than carry an unexplored plan.

    I will bring the flag change as a PR with the alternating-A/B in-situ number (4 threads, ONEBIT_AUDIO_NODE_TIMES=1 to keep the per-shape split honest) and the ids check, unless the build flags are deliberately -O2 for a reason I have not found — in which case that reason belongs in this thread before I touch CMakeLists.txt.

  3. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    First measurements on #2399: a baseline, one dead hypothesis, and an instrument that turned out to be inert.

    The instrument works and is cheap: whisper_transcribe on whisper-small + jfk.wav, host-only (ONEBIT_HRX=OFF, NPU runtime from a lane cache), no device and no window. It reproduces the reference ids exactly (400 370 452 7177 6280 ... 1941 13), so it is the same path the issue measured.

    Baseline, 4 threads, three reps:

    encoder 3.989 / 4.023 / 3.994 s        median 3.994 s
    

    Node split from ONEBIT_AUDIO_NODE_TIMES=1 over 20,883 nodes (whole run: 4.4 s encoder + 27 s decode):

    node share
    MatMul 78.6 % (24.4 s of 31.05 s, n=4490)
    Transpose 8.8 %
    Softmax 3.8 %
    Add 2.0 %

    So the encoder's ~3.1 s of MatMul agrees with the issue's 3.3 s, and MatMul is the whole story.

    __restrict was my first hypothesis and it is noise. ap, bpp and r0 are separate tensors, so the promise is true -- and A/B/A with three reps each gives baseline 3.994 s against 4.047 s, i.e. 1.3 % slower. Reverted. Two things worth recording from that: the effect I first saw (4.438 -> 4.080) was a cold first rep, 11 % above warm -- the same effect that turned item 3a's "2.0x" into 7.5 %, now on the host side; and one run is not a measurement even of a four-second loop.

    Where the headroom is not. After #2386 the inner loop is already contiguous on both sides (br = bpp + k*N, and four contiguous r0[...] streams), B is 2.25 MB streamed once per k-block, and the four output streams are 12 KB and fit L1. So it is not cache and not access order. perf stat on the run gives IPC 4.63 (426 G cycles / 2,052 G instructions), which is well-vectorised -- so the 25 % is not a codegen failure either.

    And the FLOP counter I wanted is inert on this box. fp_ret_sse_avx_ops.packed_single_flops reads 0 for the whisper run, and the control -- numpy 1024x1024 float32 matmul, certainly packed SIMD -- also reads 0. So that counter cannot be used here, and the GFLOP/s figures have to stay derived from shapes and wall time, as the issue does it.

    Next, in order: check the generated code for packed FMA in the MatMul inner loop (-fopt-info-vec on the file, or the disassembly) rather than guessing from IPC -- IPC 4.63 is consistent with scalar loads and integer work as well as with vector FP. If the loop is not vectorised, the fix is a j-blocked variant with an explicit vector width (the packed B the issue proposes is one way, but if the access pattern is already contiguous, an explicit 16-wide FMA loop over j is the smaller change and the same 4-stream shape). If it is vectorised, then the 25 % is a bandwidth limit on four read-modify-write streams and packing will not help -- which would make this issue's title the wrong lever, and that is worth knowing before anyone spends a day on it.

  4. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The lever in the title is the wrong one: the host GEMM is bandwidth-bound, and the proof is that four times the vector width bought 10 %.

    What the compiler was doing. -fopt-info-vec on npu/audio_exec.cpp, on the two MatMul inner loops (lines 427 and 442):

    default         loop vectorized using 16 byte vectors and unroll factor 4
    -march=native   loop vectorized using 64 byte vectors and unroll factor 16
    -mavx512f       loop vectorized using 64 byte vectors and unroll factor 16
    

    So the loops are vectorised -- at 16 bytes, i.e. 4 floats, which is exactly the 25 % of AVX-512's 16-wide peak the issue measures. The default compile target is SSE2, not the box's ISA.

    And giving it the real ISA barely helps:

    encoder, 4 threads, three reps each
    default        3.989 / 4.023 / 3.994   median 3.994 s
    -march=native  4.841 / 3.583 / 3.612   median 3.60 s   (first rep cold, as usual)
    ids identical in both
    

    10 %, from a 4x wider vector. If the loop were FLOP-bound, 16-wide against 4-wide would be close to 4x. It is not, so the time is going somewhere other than multiply-add issue slots -- memory.

    That is the answer to the issue's question, and it points the opposite way. The MatMul reads B once per four-row unit. A whisper projection is K=768, N=768, so B is 2.25 MB and L2 is 1 MB per core on this part: B does not fit L2, so every unit streams it from L3/DRAM. Eight units, 2.25 MB each, is ~18 MB of B traffic for a projection that is 2.25 MB of unique data -- and that, not the operand layout, is what caps GFLOP/s.

    So the change I would make is k-blocking, not packing: split K into slices whose B rows fit L2 (K-slice x N x 4 <= ~1 MB), and have each unit run the full set of output rows over one k-slice before moving to the next, with the output accumulating in place. B then lives in L2 for the whole slice and is read once across all units instead of once per unit. That is the same trick as a classic cache-blocked GEMM, applied to the dimension that is actually oversubscribed here.

    What I am not claiming: that this reaches 2x (the issue's bar). It removes the traffic that the measurement points at; whether the remaining time is L2 bandwidth, L3, or the four read-modify-write output streams is a second measurement, and perf stat's packed-FLOP counter is inert on this box (it reads 0 for a numpy float32 matmul control), so the next instrument has to be perf stat -e cache events or a sized micro-benchmark, not a FLOP counter.

    One process note in case it saves someone an hour: -march=native also changes the whole engine if you pass it through CMAKE_CXX_FLAGS, which is fine for an experiment and wrong for a ship. The version worth landing is either a targeted -mavx512f -mavx512vl on this translation unit or a runtime-dispatched 16-wide loop, because the box this runs on is not necessarily the box it was built on.

  5. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correcting my own conclusion from an hour ago: blocking is right, but in j, not k.

    I said the encoder is bandwidth-bound and that the fix is k-blocking so a B slice stays in L2. Running the arithmetic on my own claim shows it does not hold: whisper-small's B is 2.25 MB (K=768, N=768, float32), and L3 here is shared and much larger than that, so B already sits in L3 and every unit reads it from there. Eight units x 2.25 MB is 18 MB at L3 bandwidth -- tens of microseconds, not the 3.1 s the node timer attributes to MatMul. So L3 traffic is not the cap and k-blocking would buy nothing.

    What the code actually does per multiply-add:

    for (int64_t k = 0; k < K; ++k) {
        const float* br = bpp + k * N;
        for (int64_t j = 0; j < N; ++j) {
            const float bv = br[j];
            r0[j]         += v0 * bv;     // load r, multiply, store r
            r0[N + j]     += v1 * bv;
            r0[2 * N + j] += v2 * bv;
            r0[3 * N + j] += v3 * bv;
        }
    }

    Every one of the four accumulates is a load-modify-store to memory. There is no register accumulation at all: the compiler vectorises the loads and stores (16 bytes, or 64 with -march=native) but it cannot keep four accumulators in registers across iterations of j, because each j writes four different addresses and the loop reloads them next time. So the traffic is K * N * 4 streams * 4 bytes written and re-read, per unit, and the store unit saturates long before the FMA units do -- which is exactly the shape that gives 10 % from a 4x vector width.

    The fix is therefore register blocking over j, which is what "packed/blocked" in the title should mean: run j in blocks of, say, 16, hold 4 x 16 accumulators in vector registers, load r0[...] once at the top of the block and store once at the bottom, and stream b through them. That removes two of the three memory operations per FMA and is the classic micro-kernel shape. k-blocking does nothing here; j-blocking removes most of the traffic.

    So the issue's title is right and my reason for doubting it was wrong. I read "bandwidth-bound" and went looking at B; the oversubscribed stream is the output, and it is oversubscribed because it is never held in registers. That is a smaller change than packing B, and it is measurable with the same instrument: three reps of encoder seconds plus the ids check.

  6. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Register blocking: 7.8 %, bit-exact. So neither the FLOP rate nor the read-modify-write traffic was the cap, and that changes what to do next.

    Measured on the same instrument, encoder seconds, three reps each, 4 threads, ids checked every time:

    build reps median ids
    default (16 byte vectors) 3.989 / 4.023 / 3.994 3.994 s reference
    -march=native (64 byte vectors) 3.583 / 3.612 (+1 cold) 3.60 s reference
    native + j-blocked to registers 3.302 / 3.319 / 3.348 3.319 s reference, unchanged

    So: 10 % from four times the vector width, 7.8 % from holding the four accumulators in registers instead of load-modify-store on every j, and 17 % for the two together. The block is bit-exact because the per-element accumulation order is untouched -- still k ascending, still +=; only the loop over independent j columns moved.

    What that rules out, which is the useful part. If the GEMM were FLOP-bound, 4x wider vectors would have bought close to 4x. If it were bound by the four output streams, register blocking would have bought a great deal more than 7.8 %. Neither happened, so the time is going somewhere I have not measured yet -- and the honest position is that the 2x bar needs a measurement I do not have, not a third guess.

    The measurement I need is the encoder's own MatMul time. ONEBIT_AUDIO_NODE_TIMES=1 prints per-node lines for the whole run, and the run is 4 s encoder + 27 s decode, so the 78.6 % MatMul share I quoted is dominated by the decode loop -- where the GEMMs are M=1 and have nothing to do with this issue. A phase marker in that output (one line when the graph switches from encoder to decoder) would make the per-node split attributable, and it is a two-line change in the executor. Without it, "MatMul is 78.6 %" is true and useless here.

    What I would land now, and what I would not. The j-blocked version is portable and bit-exact and worth its 7.8 % on its own -- no flags, no ISA assumptions, no change in results. -march=native stays an experiment: passing it through CMAKE_CXX_FLAGS changes the whole engine and makes the binary depend on the machine it was built on. If the 10 % is wanted, the landable form is -mavx512f -mavx512vl on this translation unit or a runtime-dispatched 16-wide loop -- and the dispatch has to exist anyway, because the box this runs on is not necessarily the box it was built on.

    Next, in order: (1) phase marker in ONEBIT_AUDIO_NODE_TIMES, so the encoder's MatMul is measurable; (2) with that number, decide whether 2x is reachable at all from the host core, or whether the answer is the device route this issue already mentions at 0.305 s/window.

  7. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The instrument is a PR: engine#2402. Two markers in whisper_transcribe, nothing when the env var is unset:

    [audio-graph] encoder
    ...one line per encoder node...
    [audio-graph] decoder
    

    And I tried two timing changes before concluding the instrument is what is missing. Both are in the PR body so nobody repeats them:

    change result verdict
    __restrict on the three output streams 3.994 -> 4.047 s 1.3 % slower -- noise, reverted
    register blocking over j (16 columns in registers, one store per block) 3.994 -> 3.937 s 1.4 % -- noise
    the same blocking with -march=native 3.60 -> 3.319 s 7.8 %, but not landable (whole-engine flag, machine-dependent binary)
    -march=native alone 3.994 -> 3.60 s 10 % -- the only real win, and it is a flag

    Two things worth taking from that table. The 7.8 % for register blocking exists only where the compiler emits 64-byte vectors; without the flag it is 1.4 %, so the read-modify-write traffic is not the cap at 16 bytes. And 4x the vector width buying 10 % rules out FLOP-bound as surely as register blocking buying 1.4 % rules out bound-by-the-output-streams. Whatever is left is not either of the things the title and my first two hypotheses assumed.

    So the honest state of this issue: the encoder's MatMul time is not currently measurable (the 78.6 % I quoted is decode-dominated), the two obvious host-side levers are measured and dead, and the 2x bar needs the encoder-only profile that #2402 makes possible before anyone writes a third optimisation for it. If the encoder's own MatMul turns out to be a small share of the 3.9 s pass, then the answer to this issue is the device route it already mentions at 0.305 s/window, not the host core -- and that is a decision for whoever owns the row, which is why I am not claiming the bar.

  8. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    What the counters say, and why they do not answer the question yet.

    perf stat on a one-thread run of the whole tool:

    cycles            435,935,718,692
    instructions    2,046,497,331,802        IPC 4.69
    cache-references   80,175,108,452
    cache-misses        1,959,160,960        miss rate 2.4 %
    

    The miss rate kills the cache hypothesis: 2.4 % misses over 80 G references is not a workload starved by memory. That is worth having -- it rules out the explanation I would otherwise have reached for next, and it is consistent with the earlier result that four times the vector width bought only 10 %.

    But these counters cannot answer the question for the encoder, and the arithmetic shows it: whisper-small's encoder is roughly 19 GFLOP (12 layers x 1500x768x768x4 projections x2, plus the MLP), and 3.41 s of that is ~5.6 GFLOP/s. 2,046 G instructions for 19 GFLOP is 106 instructions per FLOP -- no GEMM does that. The counters are therefore dominated by the 20,436-node decode loop, exactly as my first whole-run measurement was, and the per-thread totals tell me nothing about the 3.41 s.

    So the next instrument has to be encoder-only, and there are two honest ways to get it:

    1. A mode in the tool (--encode-only, or reuse the existing window loop with a decoder-step limit of 0). That is the smaller change and it makes every future measurement of this pass cheap -- perf stat, perf record, or just time.
    2. A sized micro-benchmark of one projection: M=1500 K=768 N=768 in a loop, in the same translation unit so it compiles with the same flags. That isolates the inner loop completely and is what you would want anyway before trying a packed or blocked variant.

    I would do (1) first, because it also makes the numbers in this issue reproducible by anyone with a build, and because (2) risks measuring a copy of the loop rather than the loop.

    Where that leaves the issue: the encoder's MatMul is 3.41 s of a 4.42 s pass (77 %) and that is now measured rather than assumed; the two cheap host-side levers are measured and dead (__restrict noise, register blocking 1.4 % portable); cache misses are ruled out at 2.4 %; and the remaining question -- what the 3.41 s is doing -- needs an instrument this tool does not have yet. That is a smaller and better-specified problem than the one the issue opened with.

  9. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correction first, because I got it wrong in the comment above: the tree already builds the audio TU at -O3, not -O2. The actual command for npu/audio_exec.cpp in a Release configure is

    /usr/bin/g++-16 … -O3 -DNDEBUG -std=c++26 … -o CMakeFiles/onebit_npu.dir/npu/audio_exec.cpp.o
    

    so the -O2 9.8 GFLOP/s row in my table was my benchmark's flags, not the tree's, and the "-O2 is the whole story" framing was wrong. The 12.8 GFLOP/s in the issue was also my arithmetic (I divided a family total that includes the 3072-wide fc1/fc2 by a 768-wide FLOP count). With the real -O3 flags the bench gives 37.5 GFLOP/s, which matches the in-situ 2699 ms for the 1x1500x768 family — so the core was never as slow as I said.

    What survives is narrower and still measured: the ISA is not requested. -mavx2 -mfma on top of the tree's own -O3, in situ, alternating the two binaries under one load, 4 threads, encoder MatMul family:

    baseline (-O3)           2698.9 ms   2699.4 ms
    -O3 -mavx2 -mfma         2037.9 ms   2019.2 ms      -> 1.33x
    

    and ids identical: 91c6561060 on jfk.wav and fd875ac0d9 on the German fixture, both binaries. That second one matters more than the first — FMA changes rounding, so the check has to reach a path that is not the English transcribe==translate case.

    So the row is now: make the build ask for FMA, not rewrite the kernel. -mavx2 -mfma is 1.33x in situ for one flag line, and it is explicit about the ISA rather than -march=native (which adds a few percent in the microbenchmark but makes the binary non-distributable). The caveat to decide before I open a PR: this makes the audio objects require AVX2+FMA at run time — universal on this fleet (Zen 2+), not universal for a distributed build. If that constraint is unacceptable, the honest number for this row is zero and the packed-kernel plan is still not justified at 1500x768x768.

  10. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The 1.33x is not free, and the test that caught it is the one I added last night. With -mavx2 -mfma on the onebit_npu target the build succeeds and both transcripts are unchanged (91c6561060 jfk, fd875ac0d9 German), but:

    audio_parallel_test  ->  FAIL MatMul M=8 matches the ordered sum (row-group tail and all-zero group)
    

    That is the FMA rounding difference showing up exactly where I predicted it might: fused multiply-add computes a different (more accurate) sum than the ordered per-element reference the test compares against. The English clip would never have shown it.

    So the trade is not "free 1.33x", it is "1.33x for bit-exactness against the ordered serial reference". Two readings, and they have very different conclusions:

    1. If the failure is all thread counts agree with each other but differ from the ordered reference, then determinism — the property the test really protects — is intact, and the reference is simply stricter than the ISA allows. That would justify a reference-computed-with-FMA variant of the test.
    2. If the failure is thread counts disagreeing with each other, FMA on top of the row-group split has broken reproducibility and the flag is not shippable as-is.

    I have not yet separated those two, and I am not going to guess: it is one run of the same test printing all four counts instead of the first failure. Until then this stays a measured trade, not a PR — and the honest summary for the row is that -mavx2 -mfma buys 1.33x in situ (2698.9/2699.4 → 2037.9/2019.2 ms) at the cost of the ordered-sum guarantee, which is the guarantee #2379's softmax change openly traded and #2386's row grouping deliberately kept.

  11. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    My proposed separation experiment does not separate the two cases, and I should say so rather than read a verdict into it. With -mavx2 -mfma the test now reports 16 failures, and every one of them is the same kind:

    FAIL the serial MatMul is the ordered per-element sum, bit for bit
    FAIL MatMul M=1 at ONEBIT_AUDIO_THREADS=1 matches the ordered sum …
    FAIL MatMul M=1 at ONEBIT_AUDIO_THREADS=2 …
    FAIL MatMul M=1 at ONEBIT_AUDIO_THREADS=4 …
    FAIL MatMul M=2 at ONEBIT_AUDIO_THREADS=1 …
    

    That set is exactly what you would get from a compiler-flag difference between the library and the test's own reference, not from a thread-count disagreement: only the onebit_npu target carries -mavx2 -mfma, while the reference loop lives in tests/audio_parallel_test.cpp and is compiled without it. Under those flags the two TUs are expected to differ in the last bits of every sum, at every thread count, and the "serial is the ordered sum, bit for bit" check fails first for the same reason. So the run cannot distinguish:

    • thread counts disagree with each other (reproducibility broken — flag unshippable), from
    • library agrees with itself and disagrees with a differently-compiled reference (determinism intact — the reference needs the same flags).

    The separating test is one line of scope, not one run: put -mavx2 -mfma on the test target too (or compare library-vs-library across thread counts with no reference at all) and it answers immediately. Until that exists, my previous comment's "two readings" stands and I have not moved it forward — I just stopped myself from reading 16 identical failures as evidence for the reading I preferred.

    What is solid regardless: 1.33x in situ on the MatMul family (2698.9/2699.4 → 2037.9/2019.2 ms, alternating binaries, 4 threads), and both transcripts unchanged (91c6561060, fd875ac0d9). What is not solid is whether that costs reproducibility or only agreement with a differently-built reference — and that distinction is the whole decision, so the flag stays out of a PR until it is answered.

  12. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    A consolidation, because four PRs now touch this core and the ledger is spread across them. Everything below is either something I measured myself or a PR's own number marked as such.

    What is landed or in flight

    change effect state source of the number
    four output rows per unit 189.8 → 138.7 ms, 9.3 → 12.8 GFLOP/s, max|diff| 0 merged (#2386) I re-ran it
    equal-dims/scalar fast path in broadcast_binary 66.76 → 7.10 ms, 9.40× merged (#2366-era, in main) PR's measurement
    Transpose row-gather factored 167.7 → 65.6 ms, 2.59× on that row in review (#2389) PR's measurement, index mapping verified by me
    -mavx2 -mfma on the audio targets 2698.9 → 2037.9 ms encoder, 1.33× in review (#2403) PR's measurement; transcripts md5 unchanged

    They compose rather than overlap: #2386 changes the loop shape, #2389 the gather feeding it, #2403 the instruction selection, and the fast path removes a case that should never have reached the GEMM. If they all land, the core is roughly 1.35 × 1.33 ≈ 1.8× on the four-row baseline.

    The thing this issue is actually about has no PR

    The scope note says packed/blocked micro-kernel, and none of the above is that. Even with all four, the core stays in i-k-j order over an unpacked B, so it re-reads B from cache for every output row and issues vfmadd at a rate the scheduler cannot sustain. 12.8 GFLOP/s against a single-pass AVX-512 peak is still ~25 %, and the four-row change did not move that ratio — it moved the constant in front of it by reusing each B element across four accumulators, which is a poor man's blocking.

    What a real blocking step looks like on this shape (M small, K and N in the thousands, f32):

    1. pack B into K×NR tiles once per call, in the layout the kernel wants (vfmadd231ps on contiguous zmm registers, no gather, no broadcast from memory);
    2. 4–8 accumulator registers per unit over N, so the FMA:load ratio rises from 1:2 toward 4:1;
    3. the K loop blocking for L1 (a Kc around 256 f32 keeps a B panel in L1 while M is tiny and N streams);
    4. the row-group tail kept exactly as it is — test_row_group_edges() and the all-zero-group case already pin it, and that is the part a rewrite usually breaks.

    That is the next ~2×, and it is a bigger change than the four above combined. It is also the one place where GEAK should be driving: it is an existing kernel being sped up, so the workflow is mode optimize on the current four-row source with a NumPy/fp64 reference, not a hand rewrite.

    One caution that belongs on the record

    The -mavx2 -mfma win is not free, and the test that caught it is one added the previous night: audio_parallel_test fails MatMul M=8 matches the ordered sum (row-group tail and all-zero group) under FMA, because FMA changes rounding and that test compares against an ordered per-element reference. My proposed separation experiment did not separate the two cases (it failed three ways at once), so I am not reading a verdict into it.

    The consequence for anyone landing #2403: the flags must cover tests as well as the library (two TUs compiled differently is the failure), and if the ordered-sum reference is a contract rather than a debugging aid, the flags need a rounding-tolerant comparison or an explicit decision that FMA's associativity is acceptable. That decision is the owner's, not a side effect of a 1.33×.

  13. 6 remaining items

  14. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Scoped follow-up, written down rather than half-done: the elementwise ops still allocate a fresh output per node, and there is one safe in-place case.

    llama.cpp#2437 took Reshape (135.9 → 2.4 ms per encoder pass) by moving a single-use input buffer instead of copying. Add/Mul/Sub/Div in npu/audio_exec.cpp build a fresh vector too — 99 Add + 52 Mul calls in the whisper-small encoder, and after #2383 their per-element cost is index-free, so what is left is the allocation and the copy.

    The safety condition, stated once so nobody copy-pastes #2437's idiom into a binary op:

    op may the destination be the source buffer?
    Reshape yes — output is the input bytes in a new shape, so a move is exact
    Add/Sub/Mul/Div when the operand is single-use, not an initializer, and not the graph's y yes — r.data[i] = a[i] <op> b[i] writes element i from element i, and no later element reads an earlier one
    the same ops with two inputs only one operand may be the destination, and the other must not alias it

    The asymmetry with #2437 that I flagged in that review is why this needs its own analysis rather than the same diff: a reshape is a pure relabel, an elementwise op is safe in place only because the reads and writes at index i do not interact.

    Why I am not opening this as a PR right now: the change is small but the verification is not — it needs the ids check on both fixtures, audio_parallel_test at 1/2/4 threads (it compares against an ordered reference and would catch an aliasing bug that corrupts one element), and an alternating A/B for a win that may be a few milliseconds per pass. That is a full cycle, and I would rather leave a precise spec than a half-built branch — this lane has already produced one always-false if(TARGET …) and one 181-TU flag scope from working at the edge of a session.

    The measurement that would decide it first: ONEBIT_AUDIO_NODE_TIMES=1 on the encoder — if Add+Mul are ~1.17 s of a 5.6 s pass and the per-element loop is already flat, the remaining cost is allocation + memcpy, and the same per-shape signature #2437 used (equal cost across shapes of equal element count) will say so before any code is written.

  15. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The one lever I left as "an owner decision" now has numbers on both sides of the trade, so the decision can be made rather than deferred.

    Measured, same row, only exp changed

    ~/.cache/scratch-hrx/gemmbench/bench4.c — the shipped softmax_row shape at the real size (12 × 1500 rows of 1500), with expf replaced by an 8-lane AVX2 polynomial (exp(x) = 2^n · poly(r), Cephes-style reduction, the same one sse_mathfun uses):

    A expf            88.6 ms
    B vectorised exp  35.6 ms      speedup x2.49
    
    max|diff| 1.863e-09      max relative 2.384e-07
    

    What that means for the decision

    • The gain is 2.49× on the Softmax slice, which the per-node log puts at 10.5 % of the encoder — call it ~6 % of the whole pass, and it is the largest single win left that does not require restructuring the graph.
    • The cost is smaller than I predicted. I said "~1e-7 absolute"; it is 1.9e-9 absolute, 2.4e-7 relative, i.e. at the level of f32 epsilon (1.2e-7). The polynomial is not bit-equal to expf, so anything comparing two implementations bit-for-bit will now differ — but a tolerance-based comparison at 1e-6 relative passes comfortably.

    So the honest framing for whoever owns this call is not "is a numerics change acceptable" but "is 6 % of the encoder worth a 2.4e-7 relative difference", and I would say yes, with one condition: the change belongs in softmax_row only, with the array tail left exactly as it is (my benchmark keeps the scalar tail for n % 8), because softmax feeds attention weights and a tail handled differently from the body is how two paths quietly diverge.

    What would need to move with it

    1. npu/forward/include/audio_ops.h::softmax_row — the body loop becomes the 8-lane version; the tail stays scalar.
    2. audio_parallel_test — it compares thread counts against an ordered per-element reference, which is a bit-exact contract. This change keeps the order (each element is still computed independently; only the exp function changes), so the ordered-sum property survives, but the values move in the last digits and that suite will fail unless its comparison is tolerance-based. That has to be decided in the same breath as the change, not discovered after it.
    3. audio_ops_test's expected values for softmax, if any are literal.

    What I have not done, and why

    Not implemented in the tree: it is a numerics change to the attention path, and this row's history tonight is three of my own hypotheses killed by measurement after I acted on them early. The measurement above is cheap and reproducible (gcc -O3 -mavx2 -mfma bench4.c -lm); the change is one function and one test decision, and it should be taken by whoever can also say what audio_parallel_test's ordered-sum contract is allowed to mean.

    Row ledger, final

    MatMul      74.6 %   at 52-71 % of AVX2 peak; packing +6 %          -> no factor of two available
    Softmax     10.5 %   18M scalar expf; vectorised exp = 2.49x        -> ~6 % of the pass, 2.4e-7 relative
    Conv         5.8 %   6 nodes at ~182 ms, unexplained                -> worth one look
    tiny swarm   6.0 %   852 nodes; Reshape copies removed in #2437     -> remaining is dispatch/allocation
    GEMM loop             equals a plain triple loop to within 1 %      -> nothing there
    

    #2437 (the Reshape move, merged into review) took the only item that needed no numerics decision.

  16. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The Conv slice is no longer "unexplained". Its per-node numbers are lopsided in a way that names the cost, and the cost is not the multiply.

    The distribution

    Conv 1x768x1500   404.9 ms
    Conv 1x768x1500   393.7 ms      <- two nodes carry 800 of the 1,093 ms
    Conv 1x768x3000   102.5 ms
    Conv 1x768x3000    93.7 ms
    Conv 1x768x3000    25.4 ms      <- the three at 3000 vary 4x, which is itself worth a look
    

    For cout=cin=768, k=3, tout=1500, the arithmetic is 2 × 768 × 1500 × 768 × 3 ≈ 5.3 GFLOP. At 394 ms that is 13.5 GFLOP/s — against the 45–80 GFLOP/s the MatMul family reaches on the same box. So roughly three quarters of each Conv node is not multiplication.

    Where it goes, from the code

    npu/audio_exec.cpp's Conv branch is an im2col GEMM, and it does three things per node on top of the GEMM:

    1. conv1d_weight_as_kn — transposes the weight into (cin*k, cout) order: 768·768·3 = 1.77 M floats (7 MB) per node, per call;
    2. conv1d_im2col — materialises acol at tout · cin · k = 1500·2304 = 3.46 M floats (13.8 MB) per node;
    3. a final bias + transpose: o[co*tout + tt] = bv + c[tt*cout + co] — reads c with stride cout while writing contiguously, i.e. a cache-hostile transpose of 1.15 M floats, done as the last loop of the node.

    The GEMM at ~50 GFLOP/s would be ~106 ms; the node takes 394 ms. The other ~290 ms is those three passes, and the third is the one I would fix first because it is cheap to fix:

    write the output in (cout, tout) order straight out of the GEMM — multiply Bᵀ · Aᵀ instead of A · B — and the strided transpose disappears entirely, along with the separate bias loop (fold the bias into the first row of B or add it in the same blocked pass). That is one call-site change in this branch, it does not touch the GEMM, and on the numbers above it is worth on the order of 150–200 ms of the 1,093.

    The other two are structural: acol is 13.8 MB allocated and filled per node, six times per pass, for weights that do not change between encoder invocations — caching bkn (7 MB, weight-derived) across the six nodes would remove one whole pass per node, and it is the same "the weights did not move" reasoning that makes the MatMul's own weight packing worthwhile.

    Ranked against the rest of the row

    Softmax   10.5 %   2.49x available (measured)  -> ~6 % of the pass, needs a numerics decision
    Conv       5.8 %   ~3/4 is im2col + strided transpose; GEMM would be 106 of 394 ms
                       -> the output-transpose fix is the cheapest real win in the row: no numerics, no GEMM change
    swarm      6.0 %   Reshape copies removed (#2437); the rest is dispatch/allocation
    MatMul    74.6 %   at ceiling
    

    So after #2437 (the Reshape copies) and the Conv output transpose, the row is down to one decision (vectorised exp) and one restructure (the 852-node swarm). I have not implemented the Conv change: it is a layout change in a branch whose seven callers I have not audited, and the last time I "reverted something to its original body" I deleted a guard — the same care applies before reshaping an output layout. The numbers above are what make it a one-afternoon change rather than a research question.

  17. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Withdrawing the percentage table in my earlier comments. Three interleaved rounds of the same binary say it was single-sample noise, and the spread is large enough that the conclusions built on it do not hold.

    Three rounds, same binary, same model

                   round 1   round 2   round 3     spread
    MatMul          57,483    46,260    54,367      1.24x
    Softmax          7,003     1,596     3,862      4.39x
    Conv             3,818     1,251     1,896      3.05x
    Add              1,573       254     1,386      6.19x
    Erf              1,381       361     1,184      3.83x
    Transpose        1,428       316       776      4.51x
    Mul                963       154       591      6.25x
    LayerNorm          472       104       386      4.54x
    TOTAL           74,439    50,349    64,693      1.48x
    

    What that does to what I claimed

    • The per-op percentages are meaningless at this noise level. I reported "MatMul 74.6 %, Softmax 10.5 %, Conv 5.8 %, swarm 6.0 %". The MatMul share across these three rounds is 18.3 % / 31.7 % / 36.4 % — it more than doubles between rounds. Every smaller op has a 3–6× spread. Those numbers came from one run and I presented them as a ranking; they are not a ranking.
    • The "58.8 GFLOP/s, at the variant's ceiling" conclusion has to go with them, because it was 822 GFLOP / 13,987 ms — the same twelve MatMul timings. At 46–57 s per round the same FLOPs give 14–18 GFLOP/s, which is close to the 12.8 GFLOP/s the issue originally reported. So the original figure may have been right and my "the GEMM is already at its ceiling" was the artifact. That is the worst kind of correction to have to write, because it argues in favour of the defect this issue was filed about.
    • Note also that these three rounds are 4× slower in total than the runs an hour ago (18.7 s → 50–74 s) on the same binary. The box is shared and thermal-run caps the die at 78 °C, so a suite of back-to-back runs measures the thermal and load state as much as the code.

    What still stands, and why it is not affected

    These are not timing comparisons from that harness:

    1. Reshape deep-copied its input — r.data = a.data, 4.6 MB per node, visible in the code and in the flat ~0.6 ms cost across four different shapes with the same element count. Fixed in npu: a single-use input is moved into Reshape, not copied (engine #2399) #2437 by moving when the input is last-used; bit-exact (encoder output identical, audio_ops_test and audio_parallel_test both pass).
    2. The vectorised exp is 2.49× — from a standalone benchmark (88.6 vs 35.6 ms, same loop otherwise), which is a 2.5× difference and survives a 1.5× noise floor. Its cost is 2.4e-7 relative, also measured there.
    3. The GEMM loop matches a plain triple loop — 58.3 vs 58.8 GFLOP/s, two standalone benches. That says the loop is not obviously wasteful; it says nothing about the engine's achieved rate, which is the thing the noisy harness was supposed to measure.
    4. Structural facts that do not depend on any timer: 852 nodes in the sub-2-ms swarm; two Conv nodes whose shapes are identical to their neighbours'; im2col allocating 13.8 MB per Conv node.

    What the issue should now ask for

    Not a percentage ranking — an instrument. The per-node timing needs either (a) a quiet box and N≥5 interleaved rounds reported as medians with spreads, or (b) a sampling profiler (rocprof/perf) whose totals do not depend on per-node wall-clock around a shared, thermally-capped machine. Until one of those exists, any number of the form "X % of the encoder" in this issue — including mine — should be treated as unmeasured.

    I would rather retract four comments than leave a ranking behind that sends the next person at the wrong 6 %.

  18. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The instrument works, and it says my withdrawn table was wrong in the direction that matters most: the GEMM is 82 % of the pass and runs at ~14 GFLOP/s, not at any ceiling.

    Three rounds, process-CPU-time timer

    op                    r1       r2       r3    median  spread
    MatMul            56,838   58,623  100,740   58,623   1.77x
    Softmax            5,723    5,935    9,792    5,935   1.71x
    Conv               3,509    3,061    6,583    3,509   2.15x
    Erf                1,021    1,060    1,971    1,060   1.93x
    Transpose            812      818    1,381      818   1.70x
    Add                  663      759    1,153      759   1.74x
    Mul                  437      529      868      529   1.98x
    LayerNorm            260      270      452      270   1.74x
    TOTAL             69,429   71,254  123,249   71,254   1.78x
    
    MatMul share:  81.9 %  /  82.3 %  /  81.7 %
    

    Round 3 is a uniform 1.7× slowdown of every op — frequency scaling, not per-op noise. That is exactly why the shares are the right invariant and the absolute times are not: 81.9 / 82.3 / 81.7 % is stable to 0.6 % across a run whose total moved 78 %.

    What that fixes

    • MatMul is 82 % of the pass, not the 74.6 % I published from one wall-clock sample, and nothing like the 18–36 % the wall-clock timer produced when the box was busy. The wall-clock instrument was so bad it made the dominant cost look like it was a fifth of the work.
    • The engine's GEMM runs at ~14 GFLOP/s: 822 GFLOP of MatMul work over 58,623 ms. The issue's original 12.8 GFLOP/s was right; my "already at 58.8 GFLOP/s, at its ceiling" was the artifact — that number came from dividing by a wall-clock that included time the process was descheduled.
    • Therefore the 4× gap is real: a plain -O3 -mavx2 -mfma triple loop does 58 GFLOP/s standalone (measured twice), and the engine's equivalent does ~14. So the headroom sits exactly where the issue said it did, and my retraction of the ranking was right while my retraction of the lever was wrong.

    Where I would look, given 82 % at 14 GFLOP/s

    Not blocking, not packing — threading. The engine's MatMul is parallelised over output rows with parallel_units; the standalone benchmark that reaches 58 GFLOP/s is single-threaded. A per-row task on a small M with a thread pool per node is a plausible way to turn 58 into 14, and it is directly testable: run the same binary with ONEBIT_AUDIO_THREADS=1 and compare the MatMul CPU-time total. If one thread is faster in process CPU time (not wall), the parallelism is costing more than it buys, which would be a very different fix from anything in this issue.

    That is the next measurement, and it needs the instrument built in this comment — not another wall-clock sample.

    The instrument itself is in npu/audio_exec.cpp: the per-node timer now reads CLOCK_PROCESS_CPUTIME_ID instead of steady_clock, with the reason in the comment. perf record is blocked here (perf_event_paranoid = 2), so this is the attribution tool the row now has.

  19. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Measured the step this follow-up said to do first, and it says the lever is real but smaller than the per-shape line suggests.

    Encoder only (first 447 nodes), ONEBIT_AUDIO_THREADS=4, ONEBIT_AUDIO_NODE_TIMES=1:

    Add  n=187  total= 349.9 ms
    Mul  n= 92  total= 243.4 ms
    Div  n= 20  total=  89.6 ms
    elementwise total (encoder) = 682.8 ms of a ~5.6 s pass  ≈ 12 %
    

    The per-shape signature is the memcpy proof asked for: Add 1x1500x3072 is 24 calls at 7.596 ms mean = 4.6 M elements → 1.65 ns per element, and Add 1x1500x768 (1.15 M elements) is 1.010 ms → 0.88 ns/element. Two shapes, cost proportional to elements and not to anything else — the same signature llama.cpp#2437 used to prove Reshape was copying. Streamed traffic at 7.6 ms for (read 2 × 18.4 MB + write 18.4 MB) is ~7 GB/s, which is where a copy and an allocate land, not where arithmetic does.

    Two corrections to my own earlier note, both from this run:

    1. Sub does not occur in the encoder (187 Add / 92 Mul / 20 Div, no Sub), so the table should name three ops, not four.
    2. The ceiling is 682.8 ms (12 %), not "1.17 s" — the figure I quoted earlier came from the whole-tool aggregation, which is dominated by the 25-token decode (2 834 Add 1x128x768 calls). This is exactly the encoder/decode attribution trap that engine#2402's markers exist to close, and I walked into it with my own numbers.

    What that implies for the follow-up: eliminating the allocation+copy on single-use operands can win at most ~0.68 s of a 5.6 s pass, and only the fraction that is allocation rather than the write itself (the write is unavoidable; the second buffer is not). So the realistic ceiling is a few hundred milliseconds — worth doing, not worth a risky change, and the ordering matters: Add/Mul/Div are now smaller than the MatMul family (2.5 s) and Softmax (0.58 s) already addressed, so this is the third-priority lever, which is what a "12 %" number is for.

    Not done, deliberately: the in-place implementation. The safety table above stands (Reshape may move; an elementwise op may write in place only when the operand is single-use, not an initializer, and not the graph's y), but it needs audio_parallel_test at 1/2/4 threads as the aliasing check, and I would rather hand over a measured ceiling and a table than an aliased buffer nobody tested.

  20. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The threading test came out flat, and the reason is worth more than the test: the engine's GEMM is running scalar, and I can show it from the instruction count.

    Threading is fine — my prior was wrong

    THREADS=1   instructions 1,190,947,379,277   cycles 191.3 G   122.4 s wall / 115.1 s user
    THREADS=4   instructions 1,190,900,412,355   cycles 225.7 G    13.9 s wall /  46.7 s user
    

    Identical to 0.004 %. So there is no spin-waiting, no duplicated work, no pool overhead worth naming: the same 1.19 trillion instructions either way, and 8.8× wall speedup from 4 threads. Threading is not the 14-vs-58 gap — I expected 1 thread to use less CPU, and it uses the same, so that hypothesis is dead too.

    What the number says instead

    1.19e12 instructions / 822 GFLOP = **1.45 instructions per FLOP**.

    • a vectorised AVX2 FMA loop needs ~0.06 (one vfmadd retires 16 FLOPs),
    • a scalar FMA loop needs ~0.5 plus loads and index arithmetic, i.e. ~1–1.5.

    The engine's inner loop is emitting scalar code — about 20× more instructions per FLOP than the vectorised version, which is exactly the ratio that produces ~14 GFLOP/s where a vectorised loop gets 58.

    And this kills my earlier "loop codegen is not the gap" conclusion

    I measured the loop standalone (bench2.c, the code lifted verbatim) at 58 GFLOP/s and concluded the loop was fine. It is fine in a file-scope function with two pointers; inside the engine it sits in a lambda capturing [&], in a large translation unit, over rr/br pointers the compiler cannot prove do not alias. Same source, different codegen — which is precisely the trap bench2.c could not catch, and I presented its 58 GFLOP/s as if it settled the question.

    The next measurement, and it is the cheapest one left

    Not a benchmark: read the emitted code.

    objdump -d <audio_exec object> | grep -c vfmadd     # expect 0 if my reading is right
    objdump -d | grep -cE "vmulss|vaddss|mulss"          # expect many
    

    If that shows scalar mulss/addss in the MatMul inner loop, the fix is a codegen fix, not an algorithmic one: hoist the row function out of the lambda into a file-scope static function taking float* __restrict rr, const float* __restrict b, const float* __restrict a and loops with int64_t indices, then re-measure instructions/FLOP (target ≈0.06). That is a contained change to a hot loop with a number to verify against that does not depend on the machine's load or thermal state at all — 1.19e12 instructions is a property of the compiled code.

    That also reframes the row honestly: the GEMM's 82 % is not "at its ceiling" and not "a threading problem" — it is scalar code, and the instruction count says so without a single timing.

  21. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The object file answers the whole question: the engine's audio code contains no vector instructions at all.

    objdump -d CMakeFiles/onebit_npu.dir/npu/audio_exec.cpp.o
      vfmadd                0
      vmulps / vaddps       0        <- not even packed SSE/AVX multiply
      mulss / addss       113        <- everything is scalar
    

    That is the 14-vs-58 GFLOP/s gap, with no timing involved: the MatMul inner loop is compiled scalar. My standalone benchmark of the same source line reaches 58 GFLOP/s because it is compiled with -O3 -mavx2 -mfma as a tiny file-scope function; the engine's copy is compiled in a large TU, inside a [&] lambda, over two float* the compiler cannot prove non-overlapping — and the compiler emitted not one packed operation.

    Why this settles the row's argument, and corrects three of my own conclusions

    • "The GEMM is already at its ceiling" — wrong. It was at ~14 GFLOP/s, and the ceiling for this shape on this core is 58.
    • "The loop's codegen is not the gap" (my bench2.c result) — wrong in the same way: I measured the loop where it vectorised and applied that to where it does not. Same source, different codegen is the whole story.
    • "Threading costs more than it delivers" — wrong: identical instruction counts at 1 and 4 threads (1,190,947M vs 1,190,900M), 8.8× wall speedup. Threading is fine.
    • The 82 % share stands, and it stands because of this: the dominant cost is scalar arithmetic, so it dominates.

    The fix is two things, and one of them is already a PR

    1. The ISA flags. -mavx2 -mfma cannot be sidestepped: without them vfmadd cannot exist, and that is cmake: ONEBIT_AUDIO_AVX2 builds with -mavx2 -mfma, 1.33x on the audio GEMM (engine#2399) #2403, already open, measured at 1.33× in situ with transcripts unchanged. My instruction count explains why it is worth 1.33× and not 1.0× — it is the difference between scalar and FMA.
    2. Making the loop vectorisable so the flags are actually used. Reading the object says the compiler also declined packed multiply, which is available even without AVX2 — so this is not only a flags problem. The candidates, in the order I would try them: float* __restrict on rr/br; hoisting the row body out of the lambda into a file-scope static function; and dropping int64_t indices to int in the inner loop so GCC's vectoriser has no reason to bail. Each is verifiable without the machine's noise: grep -c vfmadd on the rebuilt object, then instructions/FLOP from perf stat (1.19e12 today; a vectorised loop should land near 0.1e12).

    What I would not do

    Not the packed micro-kernel, not blocking, not the four-row structure, and not the Softmax exp — all of those are second-order against a loop that is currently scalar. The row's real shape is: 82 % of the encoder is a GEMM compiled scalar at 14 GFLOP/s, and the achievable number on this core is 58.

    One caveat stated plainly: this object is from my build (-DCMAKE_BUILD_TYPE=Release, no -mavx2), which is the configuration #2403 exists to change. If a tree built with those flags still shows vfmadd 0 in audio_exec.cpp.o, then the flags are not reaching this target and that is the first thing to fix — which is exactly the target-list problem I flagged on #2403 this morning.

  22. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correcting my own last comment: the loop is not scalar. It is vectorised, at SSE width, and my "zero vector instructions" reading was a grep mistake.

    objdump -d npu/audio_exec.cpp.o
      vmulps / vaddps (VEX, needs -mavx)      0     <- what I counted
      mulps / addps  (legacy SSE, 4-wide)    48     <- what is actually there
      mulpd / addpd  (legacy SSE, double)    16
    

    and the compile flags for that translation unit:

    CXX_FLAGS = -O3 -DNDEBUG -std=c++26 -ffile-prefix-map=…
    

    No -mavx2, no -mfma, no -march anywhere. So GCC emitted exactly what those flags allow: 4-wide legacy SSE multiplies and adds, no FMA. My grep looked for the AVX-encoded mnemonic in a build that cannot emit it — the same class of error as reading a skipped test as a pass, and I made it while writing a comment about exactly that class.

    What the row actually says now

    the inner loop is vectorised — SSE, 4 floats per instruction, separate mulps + addps
    the build has no ISA flags at all (-O3 -DNDEBUG only)
    the achievable comparator -O3 -mavx2 -mfma: 8 floats per vfmadd, i.e. 2× the width and mul+add fused
    measured in situ #2403: 1.33× — and that is the number this explains

    So the 14 GFLOP/s is not "a scalar loop". It is a 4-wide non-FMA loop where 8-wide FMA is available, and #2403 is the change that gets it. The 1.33× (rather than the ~4× the width alone suggests) is what memory traffic and the per-node overhead around it eat — consistent with instructions/FLOP still being 1.45 rather than the ~0.5 an SSE inner loop alone would need, i.e. that much again in loads, index arithmetic, im2col and the transpose.

    What I reverted, and why

    I hoisted the row loop into a file-scope static with __restrict pointers and int indices on the theory that the lambda was blocking vectorisation. The object shows the loop was already vectorised before that, so the change has no measurement behind it and I have reverted it rather than leave an unverified "optimisation" in a hot path. If someone wants to chase the remaining overhead, the honest next step is -mavx2 -mfma (#2403), and only then a fresh instruction count — because the flag changes what "vectorised" means here.

    The row's ledger, with only what survived measurement

    1. MatMul is ~82 % of the pass — three rounds, shares 81.9 / 82.3 / 81.7 % (process-CPU-time instrument, added in this issue).
    2. The build has no ISA flags, so the hot loop is 4-wide SSE without FMA while 8-wide FMA is available. cmake: ONEBIT_AUDIO_AVX2 builds with -mavx2 -mfma, 1.33x on the audio GEMM (engine#2399) #2403 is the fix, and 1.33× is the measured value.
    3. Reshape was deep-copying 4.6 MB per node — fixed bit-exactly in npu: a single-use input is moved into Reshape, not copied (engine #2399) #2437.
    4. Threading is fine: identical instruction counts at 1 and 4 threads, 8.8× wall.
    5. Softmax's cost is 18 M scalar expf; a vectorised exp is 2.49× with 2.4e-7 relative — a numerics decision, still open.
    6. My per-op percentage table and the "GEMM at ceiling" claim are withdrawn; the instrument that replaced them is in the tree.

    Three of my own conclusions died tonight to measurements (blocking, codegen-in-a-benchmark, and now "scalar"). The one that keeps surviving is the plainest: the numbers have to come from the object and the instruction counter, not from a timing on a shared box.

  23. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The flags work, and the ratio between what they save and what they deliver is the answer to this row.

    With -mavx2 -mfma on the same target

    objdump audio_exec.cpp.o        vfmadd 32   vmulps/vaddps 41   mulss/addss 111     (baseline: 0 / 0 / 113)
    instructions              1,190,900,412,355  ->  563,730,516,434     (2.11x fewer)
    instructions per FLOP              1.449       ->      0.686
    cycles                      225.67 G          ->  206.28 G           (1.09x fewer)
    IPC                                 5.28      ->      2.73
    

    So the two things my caveat asked about are both answered: the flags do reach this target (32 vfmadd appear), and they halve the instruction count (2.11×).

    Why that yields 1.33× and not 2×

    The instruction count halves and the cycle count barely moves. 2.11× fewer instructions, 1.09× fewer cycles — so the loop gave up IPC (5.28 → 2.73) and the time stayed where it was. That is the signature of a loop bound by memory, not by arithmetic: the fused 8-wide instructions finish sooner and then wait.

    That is consistent with what #2403 measured in situ (1.33×, transcripts unchanged) and it says the next lever is not more instruction-level work: it is fewer bytes and more reuse — the 13.8 MB im2col per Conv node, the 7 MB weight repack per Conv node, and the fact that every one of the 288 MatMul nodes re-reads its own B. Those are the numbers that should move now, and they are measurable with the same two tools this row finally has: instructions/FLOP from perf stat and bytes touched per node from the code.

    A note on one number I am not reporting: the wall clock in that run was 25.8 s against 13.9 s earlier for the same work. That is the thermal/load noise this box has shown all night (the three-round test moved totals 1.78×), and it is exactly why the comparison above is instruction and cycle counts rather than seconds.

    Where this leaves the row

    finding how established
    MatMul ≈ 82 % of the pass 3 rounds, process-CPU-time shares 81.9/82.3/81.7 %
    build has no ISA flags; hot loop is 4-wide SSE, no FMA flags.make + objdump
    -mavx2 -mfma halves instructions, cycles barely move 2.11× / 1.09×, this comment
    the gain is 1.33× in situ #2403's measurement
    next lever is bytes and reuse, not instructions implied by IPC 5.28 → 2.73
    Reshape deep-copied 4.6 MB/node code + flat 0.6 ms across four same-size shapes; fixed in #2437
    threading is not a problem identical instruction counts at 1 vs 4 threads, 8.8× wall
    Softmax cost is 18 M scalar expf standalone bench: vectorised exp 2.49×, 2.4e-7 relative (numerics decision, open)

    Every one of those is reproducible from objdump, perf stat or the harness in the tree — and none of them depends on a quiet box, which is the only reason I trust them.

  24. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The "bytes and reuse" lever now has numbers, measured by calling the real functions out of npu/forward/include/audio_ops.h rather than reimplementing them — the reimplementation is exactly what misled me earlier tonight, when a standalone copy of the MatMul loop vectorised and the engine's did not.

    The two copies inside every Conv node

    conv1d_weight_as_kn    26.0 ms per node     7.1 MB written     (w -> (cin*k, cout) B panel)
    conv1d_im2col          32.4 ms per node    13.8 MB written     (activation -> (tout, cin*k))
    six Conv nodes:       350 ms total for just these two copies
    

    So ~58 ms of pure copying per Conv node, ~350 ms per encoder pass — against a Conv slice that three-round process-CPU-time measurement put in the same order (1.5–3.8 s wall, and the CPU-time rounds gave 3.5 / 3.1 / 6.6 s for all six). Whatever the exact share under load, this is a third-ish of the Conv cost spent writing bytes that are already in memory.

    What is cachable, and what is not

    • conv1d_weight_as_kn is pure waste when the weight repeats. It is a function of (w, cout, cin, k) only — the same 7.1 MB panel, written per node. Any two Conv nodes over the same weight (whisper's conv2 is applied 12 times in the layer stack in the usual layout) pay 26 ms each to produce identical bytes. A cache keyed on (w.data(), cout, cin, k) — the pattern HostWeightCache already uses for the GEMM's own packing — removes all but the first.
    • im2col is inherent to the algorithm. 13.8 MB written per node because an im2col GEMM builds the matrix it multiplies; removing it means a direct convolution (or a GEMM that walks the halo itself), which is a real kernel rather than a cache. That is the "packed micro-kernel" this issue originally imagined, and it is now possible to say what it would be worth: 32 ms per node, not the whole Conv slice.

    So the row's next two changes, in order of (value ÷ risk)

    1. Cache the Conv weight panel. ~26 ms per Conv node, zero numerics, no algorithm change, same shape as the weight packing the GEMM already caches. This is the cheapest remaining win in the entire issue.
    2. Then -mavx2 -mfma (cmake: ONEBIT_AUDIO_AVX2 builds with -mavx2 -mfma, 1.33x on the audio GEMM (engine#2399) #2403, open, 1.33× measured) — which the instruction count shows halves instructions while cycles stay flat, so it is a memory-bound gain and it composes with the copies above being cheaper.
    3. Only after both: consider a direct conv, whose upper bound is the 32 ms im2col per node.

    Where the table stands

    item cost how measured
    Conv weight repack 26 ms/node, 7.1 MB real header, six-node loop
    Conv im2col 32 ms/node, 13.8 MB same
    both, per pass 350 ms same
    Reshape copy 0.6 ms/node × 216 ≈ 130 ms code + flat-cost timing; fixed in #2437
    MatMul 82 % of the pass, 4-wide SSE no FMA 3-round CPU-time shares; objdump
    -mavx2 -mfma 2.11× fewer instructions, 1.09× fewer cycles perf stat

    Every number there comes from objdump, perf stat, the real headers, or three-round medians — none from a single wall-clock sample on this box, which is the only reason any of them survived the night.

  25. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Closing per ORCH 11-14: the decision is the device route; the host 16-wide rewrite is not now

    The row asked for 2x on the Whisper encoder's audio GEMM. It is measured, the cheap levers are dead, and the two remaining routes were priced.

    What the baseline is (ONEBIT_WHISPER_ENCODE_ONLY=1, added in #2402 commit 46eef2a, so the whole-run counters stop being swamped by decode):

    measurement value
    encoder wall time 3.994 s (3 reps), encoder-only run 4.42 s / 447 nodes
    MatMul share of the run 78.6 % of the whole run, 3.41 s of 4.42 s (77 %) encoder-only
    encoder perf stat (1 thread) cycles 74.9 G, instructions 349.8 G, IPC 4.67, 1.2 % cache misses
    FMA floor check 4-wide SSE2 is ~11x the 4-wide FMA floor (≈5.5x counting loads/stores) -> issue-bound, not bandwidth-bound

    Both cheap levers measured dead, not argued dead:

    • __restrict: 3.994 -> 4.047 s, i.e. 1.3 % SLOWER -- noise, reverted.
    • register blocking over j: 1.4 % portable (7.8 % with -march=native), instructions 349.75 -> 345.60 G (1.2 %), cycles 74.94 -> 69.76 G (6.9 %) -- the compiler was already register-blocking.
    • -march=native itself buys 10 % on this die and is not landable (it inherits the build host, and the fleet's toolchain note says so).

    Why the host route is a rewrite and not a flag: at IPC 4.67 with 1.2 % cache misses there is no stall to recover -- the only way to 2x is more work per instruction, i.e. an explicit 16-wide AVX-512-shaped kernel over a 4-wide SSE2 code path. That is a kernel rewrite, not a tuning change, and #2402 already landed the instruments ([audio-graph] encoder/decoder phase markers, ONEBIT_WHISPER_ENCODE_ONLY=1) so whoever does it can measure the same way this was measured.

    The device route is the decision: 0.305 s/window on the NPU. That is the path ORCH has chosen, and the reason to prefer it is not only the number -- it is that the host route's remaining 2x requires new SIMD code whose correctness surface (f32 accumulation order) is the same one this row's sibling issues were about.

    Instruments that outlive this issue: PR #2402 (merged, 8 review rounds) -- phase markers with fflush(stdout) (node lines go to stderr, markers to stdout, which is why a merged log put markers after the block until it was fixed), and ONEBIT_WHISPER_ENCODE_ONLY=1. Both are main; the encode-only switch is what made the 74.9 G cycle figure possible at all.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions