Skip to content

npu: the audio encoder non-GEMM half — Conv/Softmax/attention are 9.06 s of a 21.7 s window (from #1783) #2366

Open

Description

@bong-water-water-bong

npu: the audio encoder's non-GEMM half — Conv, Softmax and the attention products are 9.06 s of a 21.7 s window

Split out of #1783 as its close condition says ("the sub-second half is a separate row"), with the measurements that make the scope of it exact. Nothing here is required for #1783's host chain, which is merged; this is the work that stands between the projections being fast and the encoder being fast.

What is measured (whisper-small, ONEBIT_AUDIO_TIMING=1, encoder 21.74 s, 447 nodes)

family nodes shape time rate
Conv 2 [768, 80, 3] stride 1, [768, 768, 3] stride 2 3.74 s 6.4 GFLOP → 1.7 GFLOP/s
Softmax 12 1500 x 1500, axis -1 2.56 s 27 M elements → ~10 M/s
attention MatMul 2 per layer 1500x64 (scores·V) and 1500x1500 (Q·Kᵀ) 2.76 s —
elementwise (Add/Mul/Erf/Div) 175 — ~2.8 s —

The projections (O/GU/D) are 0.305 s/window on the device at M=1536 and are not part of this issue. For FLM's 0.6 s per 30 s window, these four families would have to fit in ≤0.29 s — they are 9.06 s, so ~30× too slow today.

The Conv number is the most interesting one: 1.7 GFLOP/s. The same array does 182 GFLOP/s on a D projection, and a plain im2col + GEMM on the host would beat 1.7 GFLOP/s by a wide margin without any device work at all. That suggests the cheapest first win here is host (im2col + the existing GEMM, and a blocked softmax), not a kernel family — worth measuring before anyone writes kernels.

What "done" looks like

Whichever route gets there, the measurable bar is the one #1783 uses:

npu-kernel-bitx.sh <container> <kernels> --repeats 3   # if kernels are involved: bit-exact vs the integer host arm
whisper_transcribe <pack> <wav> tests/data/whisper-jfk-ids.json
   -> "transcript ids match the reference decode"   and   encoder <= 0.6 s per 30 s window at stated load

with ONEBIT_AUDIO_TIMING=1 naming which family got faster, so the before/after is per family and not a single encoder number.

Constraints that bound the design

  • The widest launch on this part is M=128 (six array rows; the emitter's rows <= cols rule and the placer's row index must be < 6 both bite above it) — npu: Whisper encoder on our runtime — measure against FLM 1.0.12's 0.6 s/30 s window and close the gap (W8/F8) #1783 measured this, and a wider tier needs -DQ4NX_DESIGN_M_PREFILL, which changes the engine's prefill M for every model.
  • Conv is not a GEMM shape the current generator can emit; im2col + the D/GU family is the shortest path to the device, and it changes the memory traffic story (an im2col of [768,80,3] over 3000 frames is a 5.5 MB matrix — it fits nowhere near the kernels' assumptions without blocking).
  • Everything here is own code (TNPU-ALL §2): a kernel goes to the private npu-kernels repo and is claimed in NPU-KERNEL-FAMILIES.md before it is built.

Why it is worth doing at all

The projections are now measured and fast (0.305 s/window at M=1536, from 2.05 s at M=128), and the host chain is merged. The encoder's remainder is what decides whether the row's 0.6 s is reachable, and it is dominated by two convolutions and a softmax — not by anything the kernel families cover. If the im2col+GEMM route lands at even 20% of the device's GEMM rate, Conv alone drops by ~20× and this issue becomes about Softmax.

Activity

  1. bong-water-water-bong commented on Nov 13, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correction to this issue's table — the numbers were pre-merge and pre-quiet, and the shape of the problem changes

    I filed this with per-family numbers from a branch build measured at load ~40. Re-measured just now on origin/main (which has the merged parallel Conv) with the instrument that actually ships, ONEBIT_AUDIO_OP_TIMES=1:

    [audio-op] 447 nodes, 10363.8 ms, 4 thread(s)
    [audio-op] MatMul             96 calls    6872.5 ms
    [audio-op] Softmax            12 calls     979.3 ms
    [audio-op] Conv                2 calls     750.9 ms
    [audio-op] Add                99 calls     559.9 ms
    [audio-op] Mul                52 calls     463.4 ms
    [audio-op] Erf                14 calls     244.6 ms
    [audio-op] Transpose          61 calls     207.5 ms
    [audio-op] Div                14 calls     154.1 ms
    [audio-op] LayerNormalization 25 calls      70.4 ms
    

    Corrected table, replacing the one in the body:

    family was filed as measured on main note
    Conv 3.74 s 0.75 s the merged per-channel parallel loop already took it 5×; the "im2col may beat it" idea is much less attractive than I wrote
    Softmax 2.56 s 0.98 s same code, quieter box
    attention MatMul 2.76 s ≈1.7 s computed from the projection/attention share of the 96 MatMuls (72 projections, 24 attention products)
    elementwise (Add/Mul/Erf/Div) ~2.8 s ≈1.4 s 179 calls, 1.42 s
    encoder total 21.74 s 10.36 s

    What that changes about the issue: the remainder I am asking someone to attack is ≈4.1 s, not 9.06 s — Conv 0.75 + Softmax 0.98 + attention ≈1.7 + elementwise ≈1.4 — and the projections are the other ≈5.2 s of MatMul, which the device route replaces at 0.305 s. So a full route lands the encoder near 3.9 s/window, and the 0.6 s bar needs that remainder cut by ~13×, not ~30×. Still a large factor, but a different problem: it is spread across four families rather than dominated by Conv.

    And the method point, which is mine to own: the instrument named in the body (ONEBIT_AUDIO_TIMING=1, per node) never existed in the tree — it was a local, uncommitted diagnostic, so nobody could reproduce those numbers from main. ONEBIT_AUDIO_OP_TIMES=1 does ship, prints the block above, and is the command to quote from here on. A measurement whose instrument is not in the repo is not reproducible, and I filed one — this comment replaces it with one that is.

  2. bong-water-water-bong commented on Nov 13, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    First measurement on this row, and it refutes the plan I filed: the attention products are not the lever, and k-blocking makes them worse.

    The two attention MatMuls, timed on their own with the executor's exact loop (k ascending, j inner) against a k-blocked-by-4 variant:

    Q.Kt    (M1500 K64   N1500): executor 17.3 ms (16.6 GFLOP/s)   k-blocked-by-4 69.0 ms   -> 4.0x SLOWER
    scores.V(M1500 K1500 N64  ): executor 15.6 ms (18.4 GFLOP/s)   k-blocked-by-4 50.9 ms   -> 3.3x SLOWER
    

    Two things follow.

    1. The executor's loop is already the right shape for these two. 16–18 GFLOP/s single-threaded for a K=64 and a K=1500 case, and the "obvious" improvement (four accumulator rows so the inner loop has ILP) is 3–4× slower, because it turns one streaming row into four and blows the cache for exactly these small-K/large-N and large-K/small-N shapes. That is worth knowing before anyone spends a day on blocking: the generic loop is not the problem here.

    2. My own ≈1.7 s attribution was share arithmetic, and it is probably too high. 24 attention launches × ~16 ms ≈ 0.39 s single-threaded, and the executor runs four threads — so the real figure is somewhere at or below 0.4 s, not 1.7 s. I derived 1.7 s from "24 of the 96 MatMuls are attention" applied to the 6.87 s total, which assumes projections and attention run at the same rate. They do not: the projections are 768- and 3072-wide GEMMs (much better per-FLOP) and the attention products are the small-K shapes above.

    So the corrected remainder is smaller than I said in the correction, and the next step is not an optimisation. What this issue needs first is a per-node instrument (the one I wrongly claimed existed): [audio-op] aggregates by op type, which cannot separate the 72 projections from the 24 attention products, and shape-share arithmetic is how I got 1.7 s for something that measures 0.4 s. Until that exists, every optimisation here is aimed at a number nobody has measured directly.

    Concretely, in order:

    1. add per-node timing to npu/audio_exec.cpp under an env var (the TODO I filed on this issue, and the thing whose absence produced the wrong attribution);
    2. re-derive the four families' shares from that, not from op totals;
    3. only then pick a target — and per the numbers above it will probably be Softmax (0.98 s measured, 12 launches of 1500×1500, ~82 ms each for 3 passes over 2.25 M elements ≈ memory-bound) or the elementwise set (1.42 s over 179 calls, ~8 ms each), not the attention MatMuls.
  3. bong-water-water-bong commented on Nov 13, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The instrument this issue needed exists now (#2377), and its first reading replaces my share arithmetic with per-shape numbers.

    ONEBIT_AUDIO_NODE_TIMES=1 prints one line per node with its output shape, so the single MatMul row splits:

    shape nodes time what it is
    1x1500x768 60 10.887 s q/k/v/o (48 launches) + fc2 (12)
    1x1500x3072 12 5.494 s fc1
    1x12x1500x1500 12 2.621 s Q·Kᵀ
    1x12x1500x64 12 2.541 s scores·V

    projections 16.38 s (53 %), attention 5.16 s (17 %), everything else 9.26 s (30 %) of a 30.80 s encoder at this box's load. The absolute numbers are load-dependent (the same encoder measured 10.36 s quiet); the ratio attention/projections = 0.31 is the stable part, and it is consistent with the 1.7 s / 5.2 s split I had guessed — so the guess was right in proportion and wrong in absolute size, which is what share arithmetic does.

    Consequences for this issue:

    1. The three host families here are 30 % of the encoder, so the remainder should be re-derived from a quiet run before a target is picked.
    2. Attention is 17 % — 5.16 s loaded, ~1.7 s quiet — a real target, but not the first.
    3. The projections are 53 %, and they are what the device route replaces (0.305 s/window at M=1536). With this instrument, any family's before/after is one command per run.
  4. bong-water-water-bong commented on Nov 13, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Measured negative result, so nobody spends a PR on it: std::erff is not faster than std::erf here.

    Erf is 14 calls / 365 ms of the encoder and the loop calls std::erf (the double entry point) on float data, which looks like an obvious 2-3x for free. It is not:

    erf(double) 12.15 ms   erff(float) 12.14 ms   (1.00x)   max |diff| 0
    erf(double) 12.45 ms   erff(float) 12.32 ms   (1.01x)   max |diff| 0
    

    (1x1500x3072, 10 reps, min over reps so load cannot flatter either arm; max |diff| 0 because glibc rounds both to the same float.) So the Erf family's 365 ms is the polynomial's real cost, not a type promotion, and the only route left is an approximation — which changes results and would need the transcript-ids check to earn its place. Files that attempt it: none yet.

    Where the non-GEMM remainder stands after #2379 (merged) and #2383: elementwise ~3.4 s at this load (was ~6.0 s), softmax 0.58 s (was 0.98 s), Conv 0.75 s untouched and now the largest single non-GEMM node pair (the mel stem and the second conv, each 1x768x3000), Transpose 0.21 s (parallelised by #2382, keeps its per-element index dot product).

  5. bong-water-water-bong commented on Nov 13, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Measured the host GEMM core, because Conv turned out to be GEMM-bound and so is everything left. The loop (npu/audio_exec.cpp:339) is one output row at a time, i-k-j, so every b[k] row streams once per output row — M passes over the whole weight matrix.

    Sharing the streamed row across four output rows (same operands, same order, so bit-identical):

    one row   189.8 ms (9.3 GFLOP/s)    four rows  138.7 ms (12.8 GFLOP/s)   1.37x   max|diff| 0
    one row   185.8 ms (9.5 GFLOP/s)    four rows  138.0 ms (12.8 GFLOP/s)   1.35x   max|diff| 0
    

    (1x1500x768 projection — one whisper q/k/v/o, M=1500 K=768 N=768; best of 3, one process so load cancels.)

    Two things this settles:

    1. The projections run at 9.5 GFLOP/s. At 16.38 s for the encoder's projection family that is ~15 % of what four AVX-512 cores can do single-pass — so the host remainder in npu: Whisper encoder on our runtime — measure against FLM 1.0.12's 0.6 s/30 s window and close the gap (W8/F8) #1783 is not near its floor, and the device route's 0.305 s is not the only way to close the gap.
    2. The cheap fix is only 1.35x. Reusing the streamed weight row across four output rows is bit-identical and free to land, but it does not reach 3-4x: the limit is the four output-row accumulators, so the next step after this is a packed/blocked kernel, which is a much larger change. Reporting 1.35x as measured, not as the 3x the "M passes over the weights" argument predicts — the argument was about traffic, and the core is not purely traffic-bound.

    Not claiming this as a win until it is in situ under alternating A/B, which is the mistake I made twice tonight with loop-level numbers.

  6. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    A non-English fixture, because the bug that hid from me once is hidden exactly here.

    When I fixed the special-token ids in #2183 I recorded the reason the defect survived testing: whisper-small hid it because translate and transcribe are the same operation on English. Everything I have measured since — softmax order, elementwise flat paths, GEMM row grouping, Transpose gather — has been certified against one English clip (jfk.wav, ids md5 91c656106060a). So I ran the same binary on a German clip, twice:

    test_speech_de.wav, main 9ddf61c4 (+ #2379 #2383 #2386 #2389)
    run 1: ttft 6.527 s  ids 42833 35570 0 3229 5185 716 47906 9801 11 1987 27853 12699 13
    run 2: ttft 6.497 s  ids 42833 35570 0 3229 5185 716 47906 9801 11 1987 27853 12699 13
    

    13 tokens, identical across runs, on a path that English does not exercise. That is a second regression fixture and the first evidence that the four host changes hold outside the one clip that certified them.

    The two edge fixtures behave sanely rather than well, and I am reporting them as observed rather than as correct:

    fx-silence.wav  ttft 6.927 s  ids 291                 (one token, no crash)
    fx-short.wav    ttft 6.477 s  ids 400                 (one token — the same first token as jfk)
    

    A one-token output for a sub-second clip may be right (nothing to transcribe before EOT) or may be the decoder window being dominated by padding; I have no reference to compare against, so I am not calling it either way. What I can say is that both are deterministic and neither faults.

    Ask for whoever owns the fixture set: these three belong in the repo's regression list with expected ids recorded, because a single English clip cannot distinguish "the maths is preserved" from "the maths is preserved on the one input where two different paths coincide" — which is precisely the shape of the defect that got past me before.

  7. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Taking this row, and hitting a blocker before writing any code: the code these measurements come from is not on main.

    $ git grep -l "AUDIO_TIMING\|audio_timing" origin/main        # (no output)
    $ ls npu/ | grep -i audio                                     # (no output)
    $ find . -name "*audio*" -o -name "*whisper*" | grep -v third_party
    ./os/licenses/whispercpp
    ./tools/whisper_encoder_baseline.py
    ./tools/amd-rai-sweep/run_whisper_onnx.py
    

    So there is no ONEBIT_AUDIO_TIMING on main and no audio encoder directory — only a Python baseline and the licence copy. The 21.74 s / 447-node window in the issue was produced by something else: an unmerged branch, a lane's worktree, or a tree that has since been reorganised.

    Could you point me at the branch or commit that produced that table? Concretely I need to know where these live:

    • the Conv nodes ([768, 80, 3] stride 1 and [768, 768, 3] stride 2) — 3.74 s at 1.7 GFLOP/s,
    • the 12 Softmax passes over 1500 x 1500 — 2.56 s, ~10 M elements/s,
    • the ONEBIT_AUDIO_TIMING=1 instrumentation that produced the per-family split.

    Why I am not just guessing at it: the worst rate in your table is the Softmax — 27 M elements in 2.56 s is ~10 M/s, which is roughly two orders of magnitude below a straightforward vectorised exp-and-accumulate pass, so it is almost certainly a naive per-element loop with a transcendental in it. That is a ~30-line change with a microbenchmark before and after, and it is the cheapest real win in the 9.06 s. But if I write it against a reimplementation on main rather than against the encoder that produced your numbers, I will have optimised a copy — which is the second-copy-of-a-rule defect in code form, and this row is too well measured to deserve that.

    Releasing my claim on the board until I can see the tree: anyone with the branch should take it, and I will pick it back up the moment the location is known. If the instrumentation is simply not landed, that is worth an issue of its own — a per-family timing split is the thing that made this row actionable in the first place.

  8. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Closed a claim I made without evidence: "Conv is GEMM-bound, so it follows the projection row." Now measured. Mel-stem conv shape (C=768, T=3000, taps=3 → im2col 2304×3000, out 768×3000), single-threaded, -O3 -mavx2 -mfma:

    im2col  6.7 / 8.2 ms
    gemm  490.0 / 477.0 ms  (21.7 / 22.3 GFLOP/s)
    im2col share: 1.3 / 1.7 %
    

    So the claim holds, and by a wider margin than I would have guessed: the im2col copy is under 2 % of the conv. There is nothing to win from fusing or from a smarter im2col layout at this shape — the whole cost is 10.6 GFLOP through the GEMM, and that is the same core the projections use.

    Two things this also settles:

    Recording it because the honest version of my #2366 status is: of the six families, the four I optimised were chosen by measurement, Erf was measured to a dead end, and Conv was measured to have no separate lever — which leaves the projections' GEMM as the only remaining target, and that one is already covered.

  9. 29 remaining items

  10. bong-water-water-bong commented on Nov 16, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Reversing my own correction an hour later: the LM head IS the decode block's entire MatMul cost. My "12 %" was reading an aggregate as a per-step figure.

    $ … | grep " 786 " | grep -c MatMul
    26
    $ … | grep " 786 " | grep MatMul | head -3
    [audio-node]  786 MatMul  1x128x51865   60.676 ms
    [audio-node]  786 MatMul  1x128x51865   55.721 ms
    [audio-node]  786 MatMul  1x128x51865   60.390 ms
    

    Every MatMul in the 786-node decode block is the LM head. There are no "other 168 shapes" — [audio-op] MatMul 169 calls is the tally across decode steps, not within one step. So:

    • 100 % of the decode block's MatMul time is the vocabulary projection (~60 ms × 26 calls ≈ 1.6 s).
    • Against the decode wall (17 s for 25 tokens) the LM head is ~9 %, because the block is 786 nodes of many small ops, not one GEMM.
    • My "12 %, the other 168 calls at 2.7 ms" was wrong twice: there are no other MatMuls, and I derived a per-call figure by dividing an aggregate by a count from a different scope.

    The mistake, named

    [audio-op] MatMul 169 calls 520.8 ms is printed per decode block in that table, and I divided 520 ms by 169 to get "2.7 ms each", then noticed 24 × 64 ms did not fit and concluded the LM head was 12 %. The 169 was blocks × 26. I used a number from one scope and a count from another, which is the same error as comparing a cold run to a warm one — the units matched and the scope did not.

    Where this leaves the row, with every number earned

    quantity value note
    encoder, warm 2.19–2.29 s device 2–4 % faster than forced host (#2647 A/B/A/B)
    encoder Conv 210 ms was 3.74 s in the opening table — im2col+GEMM works
    encoder MatMul 2.11 s of 2.96 s the projections at M=1536
    decode, warm ~0.63 s of ops per step 17 s wall for 25 tokens
    decode MatMul 100 % LM head, ~60 ms/step 26 calls, one per step
    LM head vs decode wall ~9 % the node count, not the GEMM, is the wall

    So the honest target order is: (1) the encoder's 96 MatMul at 2.11 s — that is 71 % of the 2.96 s encoder and the only family over a second; (2) the 786-node per-step overhead in decode, which is where the 17 s actually goes; (3) the LM head last, at ~9 % of decode and already 2–4 % ahead of the host.

    I am not writing a fourth proposal. Three of the last four in this thread were built on a number that measured something else, and each correction was cheap only because the measurement existed. The encoder MatMul shapes at M=1536 are the next thing to look at, and ONEBIT_AUDIO_NODE_TIMES=1 already prints them.

  11. bong-water-water-bong commented on Nov 16, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The encoder's projections are already at device peak: 177 GFLOP/s. There is no lever there, and I have to withdraw the target I named one comment ago.

    Per-node shapes from a warm run (node index is per-block, so this is all blocks):

    1248 × 1x128x768      728 × 1x12x128      424 × 1x1500x768
     156 × 1x128x3072      24 × 1x12x1500      12 × 1x1500x3072
    

    1x1500x768 is M=1500, K=768, N=768 — 1500 · 768 · 768 = 885 M MAC = 1.77 GFLOP. Timing it:

    8.728 ms → 203 GFLOP/s
    12.269 ms → 144 GFLOP/s
    9.206 ms →  192 GFLOP/s
    

    ~145–205 GFLOP/s, against the 182 GFLOP/s the issue measured on a D projection. The encoder's MatMul is not slow; it is at peak, and its 2.11 s is simply the work: ~96–424 real GEMMs of 1.77 GFLOP each.

    So my "target 1" was wrong

    One comment ago I wrote "the encoder's 96 MatMul at 2.11 s — that is 71 % of the 2.96 s encoder and the only family over a second" and implied it was the thing to attack. It is 71 % because it is 71 % of the arithmetic, and it is running at the hardware's own rate. Reducing it means doing fewer FLOPs (a different algorithm, a smaller model, or fusion that removes a projection), not a kernel.

    The arithmetic I should have done first: 2.11 s of MatMul = 96 × 1.77 GFLOP ÷ 2.11 s ≈ 81 GFLOP/s aggregate, which is unremarkable but not pathological once the per-call spread is included — and it is bounded by the FLOP count, not by a 25× inefficiency like the Conv was.

    Where each family now stands, with the rate rather than the time

    family time rate verdict
    Conv 210 ms ~37 GFLOP/s was 1.7 — fixed by im2col+GEMM
    encoder MatMul 2.11 s 145–205 GFLOP/s at peak; only fewer FLOPs helps
    Softmax 279 ms 27 M elements memory-bound, small
    LM head (decode) ~60 ms/step 415 MB/s device 2–4 % faster than host (#2647 A/B)
    Gather 13.5–16.1 ms memcpy not a target

    Nothing in that table has a 25× gap left in it. The one that did — Conv — was fixed before this thread started, which is why the original 21.74 s table no longer describes this box.

    What that means for engine#2366

    The row's premise ("four families are 9.06 s of a 21.7 s window and ~30× too slow") is no longer the state of the code, and no remaining family is 30× off. If the FLM target of 0.6 s per 30 s window still matters, the lever is fewer FLOPs or fewer passes, not a faster kernel — and that is a design question for whoever owns the row, not a measurement I can settle with ONEBIT_AUDIO_NODE_TIMES.

    I have now corrected this thread six times, and every correction came from a measurement rather than from re-reading my own prose. The last one is the useful kind: the answer is "no remaining inefficiency", which is a result, not a failure to find one.

  12. bong-water-water-bong commented on Nov 16, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Completed the A/B on the device build: the 1x128x51865 GEMV node is 71.7 / 59.8 ms there, against 81.8 ms on the host build — so the device is nominally ahead on this op, and both are round-trip-bound at ~350–440 MB/s.

    Same pack, same wav, one npu-hold ticket (load 3.6, die 52C, token free, no faults):

    device build:  [audio-node] 786 MatMul 1x128x51865   71.730 ms / 59.799 ms
    host build, FORCE_HOST=1:                              81.777 ms (one quiet pair; a loaded pair read 239 ms)
    

    Revised reading of the switch's first result. My earlier comment said the host looked competitive; with the device number in hand the honest statement is:

    • the device is 1.1–1.4× faster on this node, not slower — and that is a small win for a 26.6 MB read, which is the tell;
    • both engines land at ~350–440 MB/s for the same 26.6 MB, so neither is doing anything clever — the op is round-trip-bound on both sides, and the ×24 in the encoder (~1.4–1.7 s) is inherent to the shape;
    • so FORCE_HOST is doing exactly what it was built for: it let a claim ("the device is better here") be tested instead of assumed, and the answer is "marginally, and it does not matter, because the read is the cost."

    What this makes actionable, and it is the fourth time tonight the same shape has appeared: a 1×128×51865 GEMV reads its whole 26.6 MB weight matrix per token for 512 B of activation. At M=1 the arithmetic is free, so the only lever is not re-reading the matrix per token — batch the LM head across the window (the encoder does 24 of these), or keep the head resident across calls. That is a caller-level change, not a kernel change, and it belongs on #2366 rather than here.

    Not claimed: a repeatable ratio. The host figure is one quiet pair (the other pair read 239 ms under load) and the device figure is two runs on one packed kernel set; 1.1–1.4× is the range the data supports, not a number to plan against.

  13. bong-water-water-bong commented on Nov 16, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The sibling path already solved this, and it is worth copying rather than reinventing: npu/forward/src/q4nx_forward.cpp:7154-7166 has the int8 head on the host by default (engine#1785), with an NPU_HEAD_HOST=alt mode that alternates host and device per call so the A/B happens inside one process on identical inputs — plus a thread split for the host side (NPU_HEAD_HOST_…), and it is documented as "measured on two models with byte-identical results".

    That is three ideas the audio path does not yet have, and each maps onto what this issue needs:

    1. Host-by-default for a bandwidth-bound head. The 51865 GEMV measures 26.6 MB of weights per token for 512 B of activation, at ~350–440 MB/s on both engines. Where a head is bandwidth-bound and the host is within 1.1–1.4×, the default should be host — that is what npu: F5 — a single-turn flash serving mode (fixed 1k ctx, clean state per request, system prompt reused), measured against 0.49 s TTFT / 33 tok/s #1785 decided for the LLM head after the same kind of measurement, and the audio path still defaults to offering it to the device.
    2. alt as the measurement mode. ONEBIT_AUDIO_FORCE_HOST (merged in audio: ONEBIT_AUDIO_FORCE_HOST forces the host path and names itself (engine#2366) #2647) is the coarse version: one process, one choice. alt is strictly better for A/B — same process, same inputs, same cache state, alternating — which is the technique that produced the trustworthy numbers in this lane all evening (interleaved pairs at load ≲3).
    3. A thread split on the host side. The host GEMV here is one map-style loop over 51 865 rows; the sibling path splits the same work over a few threads. At 81.8 ms host vs 59.8 ms device, spreading 26.6 MB across 4 threads is the kind of change that could put host ahead of device on this node without touching a kernel.

    What I would do next on this row, in one line each: add NPU_HEAD_HOST-style host-default + alt to the audio LM head (mirroring #1785), measure with interleaved pairs, and only then decide whether the bigger lever — batching the encoder's 24 head calls — is needed. The batching fix is real but it is the second step; the first is to stop paying device round-trip costs on a shape the device cannot win.

    Not claimed: that host wins. 81.8 ms host vs 59.8–71.7 ms device is the measured range today, so host loses on the current single-thread loop — but the sibling path shows what closing that gap looks like, and it is a caller-level change this lane has not tried.

  14. bong-water-water-bong commented on Nov 16, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Measured the head-shaped node with a per-call switch (mirroring NPU_HEAD_HOST=alt), and the result says the divert target is wrong, not the idea. Reverted; nothing pushed.

    The switch worked — 26 head-shaped nodes detected on the encoder, alt alternating — and on a host build:

    ONEBIT_AUDIO_HEAD_HOST=0    (through offer(), i.e. the existing path)   26 nodes, mean 145.42 ms
    ONEBIT_AUDIO_HEAD_HOST=alt  (half diverted to the executor's host GEMM) 26 nodes, mean 132.20 ms
    ONEBIT_AUDIO_HEAD_HOST=1    (all diverted)                              26 nodes, mean 197.44 ms
    

    Read carefully, this is host-vs-host, not host-vs-device. The binary is -DONEBIT_NPU_HOST=ON, so HEAD_HOST=0 already runs on the host — through offer(), which on a CPU-only build takes its own host fallback. What the three numbers compare is therefore two different host implementations, and the answer is that the executor's own host GEMM (197 ms) is 1.36× slower than whatever offer() does on the host (145 ms).

    So the useful finding is a redirect, not a switch: diverting a head node to the audio executor's host loop makes it slower, and the reason to prefer offer()'s path is presumably its threading (the head is one map-style loop over 51 865 rows; NPU_HEAD_HOST in q4nx_forward splits the same work over several threads — that is the piece this lane has not ported). And alt landing between the two is exactly what an alternating 50/50 mix should produce: (145+197)/2 ≈ 171 against the measured 132… which does not match, so the alt number deserves a second look before anyone quotes it (the counter is per call, so 13 of 26 diverted; the mismatch is either load drift between the three runs or the divert's cost not being what a mean suggests).

    What I would do next, one line each: port the thread split into the host head path first (that is the actual lever the sibling path has), then re-run this three-way comparison; a per-call switch is worth shipping only once the host arm is the good host arm, because a switch whose host branch is slower is a way to make the device look better than it is.

    Not claiming: that the device is faster. The device numbers from the earlier A/B stand on their own (59.8–71.7 ms on the device build), and this run says nothing about them.

  15. bong-water-water-bong commented on Nov 16, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Retracting the plan I posted 20 minutes ago: the head node is ALREADY threaded, 4.3× from 1→4 threads. So "port the thread split" is not the lever — it exists.

    Measured on the current main build, same pack and wav, ONEBIT_AUDIO_NODE_TIMES=1, first 1x128x51865 node:

    ONEBIT_AUDIO_THREADS=1   1352.3 ms
    ONEBIT_AUDIO_THREADS=4    311.7 ms      -> 4.34x
    

    Two things follow, and the second matters more:

    1. My hypothesis was that an M=1 GEMV gets one work unit and therefore one thread (the four-row grouping has nothing to group, so parallel_units(batches * groups) would be ~1). The scaling says otherwise — batches for this node is not 1 (the encoder's head is issued over the window, so there are many batch indices), and the existing split covers it. The proposed change would have added a second parallelisation on top of an existing one.
    2. Which means the earlier three-way comparison has a different explanation than "the executor's host GEMM is worse": with 26 head nodes and heavy thread contention, the mean over one run is not separable from load drift, and the alt value not matching the half-and-half prediction (132 vs the expected ~171) was the first sign of that. I quoted a 1.36× difference between two host paths that may differ by nothing at all.

    What survives: the device numbers stand (59.8–71.7 ms on the device build, 3033–3545 ms for the whole encoder), and the observation that the head is ~26 × 300 ms ≈ 7.8 s of summed per-node time on a ~3.5 s wall clock with 4 threads — i.e. it is a large fraction of the encoder and it is already parallel, so the remaining lever is not more threads.

    Where I would look instead, in one line each: the node does one full pass over 26 MB per call with M=1, and batches indices share the same weight matrix — so the lever is reusing the loaded weights across the 26 calls (keep the head resident, or batch the calls into one M=26 launch), which is the second item on my earlier list and the one I had ranked below threading. Ranking corrected.

    Not claiming: any number for the host-vs-device head comparison at this point — my own three-way measurement is contaminated by thread contention and load drift, and I am withdrawing the 1.36× rather than defending it.

  16. bong-water-water-bong commented on Nov 16, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Checked whether the head's weights are re-decoded per call or already resident — they are resident — so the lever is neither threads nor decoding: it is the 26 separate passes over the same 26 MB.

    npu/audio_exec.cpp:345-356:

    auto init_f32 = [&](const std::string& name) -> const Tensor& {
        auto it = t.find(name);
        if (it != t.end()) return it->second;          // already materialised: returned as-is
        ... load_init_f32(g, *ii->second, &v);
        return t.emplace(name, std::move(v)).first->second;
    };

    The first use materialises the head into t and every later use returns that same tensor, so the 26 calls share one resident weight buffer. That closes the two cheaper explanations and leaves exactly one:

    • decoding per call: no, it is cached by name;
    • insufficient threading: no, 4.34× from 1→4 threads (previous comment);
    • therefore: 26 × 26.6 MB = ~692 MB of reads per encoder pass, and at 4 threads' bandwidth that is the 26 × ~300 ms ≈ 7.8 s of summed node time that dominates the encoder wall clock. Nothing in the per-node loop can avoid it, because each call genuinely needs every weight element for its single row.

    So the only remaining lever is to make the graph issue ONE call instead of 26 — a batched head of M = 26 (or the window length) against the same B. That is a graph-shape change, not a kernel change: it needs the 26 nodes to be recognised as the same projection over different rows, which is a property of how the exporter emits the head, not of this executor. It also happens to cut the read traffic 26× rather than 26→(26/threads).

    What I would do with this, concretely: the executor's offer() already receives (A, M, K, B, N) with M as the row count — so a pre-pass that folds consecutive MatMuls with an identical B and equal K/N into a single call is expressible in the same code that builds the plan. That is the change I would attempt next, with audio_parallel and the ids as the bit-exactness arms, and the note that M=26 must not change accumulation order per row (it does not — each row reads all of B).

    Not claiming: that the fold is safe for every graph. A generic MatMul fold changes the schedule, so it needs to be limited to nodes whose A is not reused afterwards — the take_if_last machinery already answers that question for the copy-elision work, and it is the right predicate to reuse here.

  17. bong-water-water-bong commented on Nov 16, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Measured the prize before writing the graph change: folding the 26 head calls into one M=26 GEMM is worth 4.45× at 4 threads on this shape, and only 1.20× at 1 thread.

    Standalone harness — same B (128 × 51 865 = 26.6 MB, does not fit L2), same work, 4 threads, one pass each:

    T=1   26 x M=1 vs 1 x M=26   1.20x
    T=4   26 x M=1 vs 1 x M=26   4.45x      (23.6 ms -> 9.8 ms with -march=native)
    

    Why the thread count changes the gain so much, which is the part I did not expect: with one thread both arms are bandwidth-bound and the read is the same 692 MB either way — folding only saves the per-call loop overhead, hence 1.20×. With four threads each M=1 call still streams the whole 26.6 MB per thread-stride pattern, so the 26 single-row calls multiply contention; the folded call splits the 26 rows across threads and each row still reads all of B, but the reads are far better interleaved. So the fold's value is not "read B once instead of 26 times" — it is still read 26 times, once per row — it is that 26 rows give the scheduler 26-way parallelism instead of 4-way over column blocks, and the memory system sees a steadier stream.

    Two consequences, and the second changes the plan:

    1. The lever is real and worth ~4× on the head, which is ~7.8 s of summed node time ⇒ roughly 1.9 s of wall clock at 4 threads, i.e. over half of the encoder's 3.5 s. That is larger than every aggregate I have quoted from this issue tonight.
    2. But it is not the read reduction I predicted in my last comment. I wrote "it cuts traffic 26×" — the harness says traffic is unchanged (each row still needs every weight element); what changes is parallelism. So the right framing for the change is "give the head 26-way row parallelism instead of 4-way column blocking", and the reason it matters is contention, not bandwidth.

    What I would do next, in one line: implement the fold with the row-parallel framing above (a pre-pass that groups consecutive MatMuls sharing B/K/N, guarded by take_if_last so a reused A prevents it), and expect ~2–4× on the head rather than any large reduction in bytes — the [audio-node] line should then show one 1x26x51865 instead of 26 1x1x51865.

    Not claimed: that the executor currently issues M=1 per call. The node label says 1x128x51865, i.e. (1, 128) activation against (128, 51865) weights — one row each — and the harness reproduces that shape exactly. If the real graph's batches are already > 1, the measured 4.45× is an upper bound rather than a prediction.

  18. bong-water-water-bong commented on Nov 16, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correction to what I posted 15 minutes ago: I measured the wrong shape. At the real shape the fold is worth 1.17×, not 4.45× — and my two earlier measurements only looked compatible because I had mis-read the node label.

    The label. [audio-node] 786 MatMul 1x128x51865 is r.dims = bp.dims + [M, N] (audio_exec.cpp:470-472), i.e. batch 1, M = 128, N = 51865. I read it as M = 1 and ran the harness that way:

    shape 26 × M=1 vs 1 × M=26 4.45×
    shape 26 × M=128 vs 1 × M=3328 1.17× (1683.1 ms → 1436.5 ms, 4 threads)

    And the mis-read is what made my earlier claim look wrong too. I had written "4.34× from 1→4 threads, therefore the node is parallel" and then "M=1 means one work unit, therefore it cannot be parallel" — an outright contradiction I noticed and did not resolve. With M=128, parallel_units(batches × groups) = ceil(128/4) = 32 units, so the 4.34× scaling is exactly what the code predicts. One label read correctly would have settled both.

    What stands and what does not:

    • Stands: the head is parallel; the weights are resident; the node is ~26 × ~300 ms of summed time on a 3.5 s wall clock.
    • Withdrawn: the 4.45× prize, and the "row-parallelism instead of column blocking" mechanism I built on it. At the real shape each call already amortises B over 128 rows, so there is nothing like 4.45× available from folding — 1.17×, and that is the upper bound of a shape-only change.
    • Still open: why 26 calls of M=128 run at ~300 ms when the harness's equivalent 26 × M=128 takes 1683 ms total (≈65 ms each). A 4.6× gap between the harness and the executor at the same shape — that is a much more interesting question than the fold, and it is the one I would look at next: the executor's per-node time includes the operand materialisation and any take_if_last copies, and the harness has neither.

    The discipline note, for whoever reads this thread: three of my last four claims on this row were wrong in the same way — derived from a label or an inference rather than from the value. The two that survived were the ones I ran a harness for. The harness was right each time; the shape I fed it was the error.

  19. bong-water-water-bong commented on Nov 16, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Tested the L2 hypothesis for the executor's 4.6× gap and it is wrong — 4-row blocking is faster here, 4-row 22.6 ms vs 1-row 38.0 ms at N=51865. So the gap is elsewhere, and I would stop guessing at it from outside.

    Same shape as the head node (M=128, K=128, N=51865, 4 threads, -march=native), varying rows per accumulator block:

    rows/group=1   38.0 ms   acc block  202 KB
    rows/group=4   22.6 ms   acc block  810 KB     <- what the executor does (kRowGroup=4)
    rows/group=8   19.9 ms   acc block 1620 KB
    

    So the block being larger than L2 costs nothing measurable here — 8 rows per group is the fastest of the three — and my "830 KB accumulator thrashes L2" story is refuted by the simplest possible test. That is the fourth hypothesis of mine on this row to die to a harness, which is the correct base rate for guesses about a hot loop.

    Where the 4.6× actually lives is now a list of things I cannot see from outside, and each needs an instrument rather than an argument:

    1. -march=native vs the TU's -mavx2 without FMA — worth maybe 1.3× by this lane's own earlier measurement, not 4.6×.
    2. The node timer's boundaries — if op_t0 is taken before operand materialisation (init_f32, take_if_last copies) and the 786-node position means several such nodes run per step, some of the 300 ms may not be GEMM at all. The [audio-op] MatMul 96 calls … ms total vs the sum of node times would settle it: if the two disagree, the node timer is measuring more than the GEMM.
    3. offer() being taken on some nodes and not others — on a host build dev == nullptr for the audio path in the host configuration, but offer() still exists; a node that goes through offer() and one that goes through the executor's own loop have different code.

    The cheapest next measurement, and the one I would do first: compare [audio-op] MatMul total against the sum of [audio-node] MatMul times in the same run. Agreement ⇒ the 300 ms is arithmetic and the gap is the kernel; disagreement ⇒ the node timer includes work the op timer attributes elsewhere, and the "4.6×" is a measurement artefact — which after tonight is the outcome I would bet on.

    Not claimed: any explanation for the gap. Three of mine have been refuted (threading missing, weights re-decoded, L2 thrash); the honest state is "the executor is 4.6× slower than an equivalent standalone kernel at this shape, cause unidentified", and that is a perfectly good bug report for whoever has profiler time.

  20. bong-water-water-bong commented on Nov 16, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Found it — and it is not the fold, not the threads, and not the kernel. The head is preceded by a Transpose of its own weight, and that copy costs 101 ms per call:

    [audio-node]  785 Transpose   768x51865    101.258 ms
    [audio-node]  786 MatMul      1x128x51865  193.072 ms
    

    101 + 193 ≈ 294 ms, which is the "~300 ms per head node" I have been quoting and mis-attributing to the MatMul since this morning. The MatMul is 193 ms; a third of the cost is a 159 MB weight copy immediately before it (768 × 51865 × 4 B), repeated for every one of the 26 calls.

    That reframes everything I have written on this row tonight:

    • the "4.6× gap vs the harness" was this transpose, not the GEMM: 193 ms matches the harness's ~65 ms plus the executor's -mavx2-without-FMA budget far better than 300 ms did;
    • the fold (1.17×) is the wrong lever — even a perfect fold still transposes the weight 26 times unless it also hoists the transpose;
    • the right lever is to do the transpose once, either by hoisting it out of the call sequence (it is loop-invariant: same initializer, same shape) or by teaching the GEMM to read B in the transposed layout so the copy never happens. At 101 ms × 26 that is ~2.6 s of summed time on a 3.5 s wall clock — the largest single item I have found in this encoder, and it is pure redundant copying.

    And the L2/threading/scheduling questions I chased all afternoon were beside the point, because the node I should have looked at was the one before the MatMul. The [audio-node] instrument has been printing the answer every run: a 101 ms Transpose sitting immediately above a 193 ms MatMul on the same 51865-wide tensor.

    What I would do next, in order: (1) confirm the transpose's operand is the head initializer and that it is the same initializer on every call — t caches by name, so a hoisted result would be reused; (2) hoist it (a take_if_last-style cached transposed copy, computed once), measure with ONEBIT_AUDIO_OP_TIMES; (3) only then reconsider batching. The bit-exactness arms are unchanged: audio_parallel plus the ids.

    Not claimed: that the transpose is removable — a MatMul whose B is stored [N,K] needs either the copy or a kernel that reads it that way, and which of those applies depends on the ONNX export. But the repetition is certainly removable, and that is 25 of the 26 copies.

  21. bong-water-water-bong commented on Nov 16, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Tried the hoist and it does not help as written: the transpose happens once per graph execution, not 26 times inside one. Reverted. The real change has to cache across executions, which is a different (larger) one.

    What I built: cache the transposed initializer keyed by (input name, perm) inside execute_graph, which is where t already caches initializers by name. Result:

    audio_parallel: ok                      (bit-exact arm passes)
    [audio-node] 785 Transpose 768x51865   94.941 ms / 89.931 ms      (unchanged)
    ids md5 91c6561060…                     (unchanged)
    

    Why it was never going to fire: the 26 head calls are 26 separate graph executions — whisper_transcribe runs the decoder 25 times after the encoder — and my cache lives for the duration of one execute_graph call. Within one execution each Transpose node appears once, so there is nothing to hit. The 101 ms I measured is per execution, i.e. ~101 ms × 26 passes ≈ 2.6 s, exactly as reported, and removing it needs a cache that outlives a single pass.

    So the next change, stated precisely: the cache belongs on whatever object holds the parsed model — the structure that already caches decoded initializers across calls (init_f32's t is per-execution; the per-call cost of decoding is avoided because whisper_transcribe re-enters with the same graph info, but the transpose result has no such home). Concretely: a mutable std::map<std::string, Tensor> on OnnxGraphInfo or on the audio pack handle, keyed the same way, consulted before the transpose loop. That is ~10 lines in the same place plus a lifetime decision, and it is worth ~2.6 s of a 3.5 s wall clock.

    What this attempt did establish, cheaply:

    • the change is numerically inert (audio_parallel ok, ids identical), which is the same statement the eventual hoist will need;
    • the transpose cost is stable across runs (94.9 / 89.9 / 101.3 ms across three measurements) — a large, repeatable, loop-invariant copy;
    • and the hook point is exactly where I put it: the Transpose branch, with inits.find saying whether the operand is an initializer.

    Not claiming the numbers beyond that: a cross-execution cache changes memory (159 MB held for the model's lifetime), which is a real trade against a 3.5 s encoder — worth stating in the PR that does it, and worth checking against the pack's own budget on a 4 GB NPU device before it lands.

  22. bong-water-water-bong commented on Nov 16, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Caching the transposed weight across executions does not help either — and the reason is decisive: every node output is copied out of the cache anyway, so the 159 MB copy is the cost, not the computation. Reverted.

    Built it properly this time (cache on OnnxGraphInfo, keyed by initializer name + perm, so it survives the 25 decoder passes):

    audio_parallel: ok                    ids md5 91c6561060…   (numerically inert, as predicted)
    [audio-node] 785 Transpose 768x51865   168.874 / 159.497 ms     (was 94.9 / 89.9)
    time: load 0.002 s, mel 0.293 s, encoder 10.362 s
    

    Two things went wrong, and the first is the important one:

    1. r.data = cv->second copies the cached vector — the executor's node contract is that each node produces its own r, so a cache saves only the index arithmetic, never the 159 MB copy. That copy is 101 ms; the cache therefore saves nothing and adds a second 159 MB resident.
    2. The 10.4 s encoder is a regression signal (against ~3.5 s), consistent with holding 159 MB extra and copying it twice on the first pass — though I did not re-measure load, so I am not claiming the number, only that it is not an improvement.

    So the conclusion is sharper than "cache it": the transpose is expensive because the tensor has to be materialised, not because computing it is expensive. The only fixes that can work are the ones that avoid materialising it:

    • store the weight pre-transposed at load (once per model, no per-call copy — but the MatMul then needs B in [K,N] while the export gives [N,K], so either the reader or the GEMM must change), or
    • teach the GEMM to read B transposed (index b[k + j*K] rather than b[k*N + j]), which removes the copy and keeps the layout — the standard answer, and the change that actually addresses 101 ms × 26.

    Third dead end on this row tonight, all honest: threading (already there), folding (1.17× because the real cost was elsewhere), caching the transpose (saves the arithmetic, not the copy). The measurement that matters stands — 101 ms of pure copy per pass, ~2.6 s per transcription — and the remaining fix is a GEMM index change, which is a kernel change and so falls under GEAK/TraceLens rather than this loop.

    Not claiming any of tonight's three hypotheses as likely; the base rate on my guesses about this loop is now 0 for 5.

  23. bong-water-water-bong commented on Nov 16, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correcting my own last suggestion before anyone acts on it: "teach the GEMM to read B transposed" is the wrong fix, and the reason is why the transpose exists at all.

    The head's weight comes from the export as [N, K] = [51865, 768]. The Transpose makes it [K, N] = [768, 51865], which is what the GEMM's inner loop wants:

    for (k) { const float* br = b + k*N; for (j) rr[j] += a * br[j]; }      // contiguous along j

    Reading the untransposed [N,K] instead would make that inner loop read stride K (b[j*K + k]), i.e. one float per 3 KB — a cache line per element. The 101 ms copy buys a great deal more than it costs, which is why the transpose is in the export in the first place. My earlier suggestion would have replaced a 101 ms copy with a several-hundred-ms strided GEMM.

    So the only two fixes that can work are the ones that materialise the [K,N] layout once for the model and then read it without copying:

    1. Do the transpose at load, into the pack's own cache — the same place the device weight images live. The graph then has no Transpose node at all (either the loader rewrites the node away, or the export is asked for the pre-transposed form). No per-call copy, no second resident copy, and nothing to keep in sync.
    2. Make the node output share the cached buffer instead of copying it — Tensor holds std::vector<float>, so this needs a shared_ptr-backed buffer in the executor. That is the same lifetime change OnnxModel just got in npu-pack: OnnxModel owns the graph mapping, so OnnxTensor::raw outlives the parse (fixes #2668) #2670 for the graph mapping, applied to tensors; it is the general fix and it is bigger.

    And a third that needs no memory at all: if the 26 passes share the same [K,N] result, then a cache plus a shared buffer is (1) and (2) together — one 159 MB copy for the model, referenced 26 times. The trade is 159 MB of host RAM against ~2.6 s per transcription, which on this box is obviously worth it and on a 4 GB device is irrelevant because this is host-side.

    What I am not doing: writing it now. Options (1) and (2) both change a lifetime or a loader contract, and tonight's five dead ends on this loop are the argument for stopping at the decision rather than starting a sixth attempt at the end of a session. The measured facts are in the thread — 101 ms per pass, 26 passes, audio_parallel and the ids as the arms — and the next attempt should start from option (1), which needs no allocator change.

    Base rate, kept honest: five hypotheses on this loop, zero correct — but the measurement that closed it (Transpose 101 ms beside MatMul 193 ms) came from a log line already in hand, and it is worth more than all five guesses.

  24. bong-water-water-bong commented on Nov 16, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    And option (1) as I framed it also saves nothing — worked through, it collapses to "the packer must store the weight pre-transposed". That is the decision I am recording, because it puts the fix in npu-pack, not in this executor.

    The chain, one step at a time:

    • a per-execution pre-pass (compute the transposes into t at the start of execute_graph) still does the transpose 26 times, once per pass — no gain;
    • a cross-execution cache only helps if the consumer reads it without copying, and Tensor owns a std::vector<float>, so t[name] = cached copies 159 MB per pass. That is what my two reverted attempts measured;
    • a non-owning Tensor (a shared_ptr view over the cached buffer) removes the copy — but it changes tensor ownership across the whole executor, which is a project, not a patch, and it makes every Tensor a potential alias of a cached weight (exactly the hazard take_if_last exists to police).

    Whereas if the pack stores the head weight already [K, N], there is no Transpose node in the graph at all: no copy, no cache, no lifetime question, nothing to keep in sync. 26 × 101 ms → 0, and the exporter's own layout is the thing that changed.

    So the row's decision: this is a npu-pack task (emit/convert the head initializer with the GEMM's layout, and let the graph drop the transposing node), not an audio_exec optimisation. The executor's [audio-node] line is how the packer's author verifies it: node 785 should simply disappear.

    What this costs the next owner, honestly: the packer change has to know which Transpose nodes are weight-layout transposes (operand is an initializer, perm is a plain rank-2 reversal) and rewrite both the tensor and the consumer's expectation — plus a pack-format compatibility question for packs already shipped with the node. That is a real piece of work with a real compatibility decision, which is why it does not belong at the end of this session.

    Everything measured stands: Transpose 768x51865 costs 101 ms beside a 193 ms MatMul, once per graph execution, 26 executions per transcription, ≈2.6 s of a ≈3.5 s wall clock. Arms for whoever moves it: audio_parallel (ordered-sum bit-exactness) and the transcript ids (91c6561060…).

    And the honest scoreboard for this row tonight: six hypotheses, zero correct — the two that produced real information were both measurements I ran, and the one that closed it was a log line that had been printing the answer all along.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions