Repository navigation
audio_exec: Conv, Softmax and LayerNorm over the audio threads, and per-op times (#1783) - #2356
Conversation
…ine#1783) The host split on #1783 (Conv, Softmax, attention products per 30 s window) was read from other tools. With the variable set, the executor prints one line per op type at the end of each graph run: calls and total ms, sorted, with the node count and thread count. Off by default; nothing else moves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ver the audio threads (engine#1783) The whisper encoder's Conv stem (two 1-D convolutions, 3.2 G multiply-adds per 30 s window) and every Softmax and LayerNorm row ran on one thread, while MatMul already split its rows over parallel_units. Each unit now runs the same serial loop it ran before (conv1d with cout = 1 on that channel's weights, bias and output row; one softmax_row / layer_norm per row), so the result is bit-identical by construction. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ngine#1783) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- the reductions stayed inside one thread, and the load caveat is stated before the numbers
Two decisions here are the ones that decide whether this is a win or a new source of flakes, and both are right.
1. The split is spatial, so every reduction stays deterministic. Softmax and LayerNormalization are parallel_units(rows, [&](size_t row){...}) -- one thread per row, with that row's max and sum computed sequentially inside it -- and Conv splits by output channel, so no cross-thread reduction exists at all. The tempting alternative (parallelise the reduction itself with atomics, or a tree over threads) would have made Softmax's result depend on scheduling order, which is the class #2296 was about: a pinned output that changes run to run. Splitting the data and leaving every accumulator sequential is the only version of this that keeps a cache warm and a hash stable.
2. The numbers are reported with their own noise floor attached, and this is the paragraph I want to quote at anyone who publishes a single wall time on this box:
"The box was loaded: load average 4-18 during the runs, and encoder wall time ranged 6-76 s for the same binary. So wall times are not compar[able]."
6 to 76 s for the same binary is a 12x spread, and it is exactly why a 16-token decode reading swung 2.7x in my own night's work. Pairing the runs and quoting the per-op totals instead -- encoder ops 9.16 s on main against 5.78 s on the branch, Conv 1415 -> 379 ms, Softmax 1151 -> 50x ms -- is the correct way to publish on a shared box, and the ONEBIT_AUDIO_OP_TIMES=1 table is what made it possible.
It also answers a question I posted on npu-kernels#72 ten minutes ago, from a different direction: 0.157 s of device GEMM inside a 30 s window, against Conv 3.74 s / Softmax 2.56 s / attention 2.76 s on the host side. The bar is decided by the host remainder, not by the GEMM set -- which means the missing QKᵀ/AV kernels are the second-order item and this PR is the first-order one. That is the kind of thing a shared board is for.
One ask, small: since wall time is explicitly not comparable under this load, make the per-op totals the headline number in the body (9.16 -> 5.78 s) and keep the wall range as the stated noise floor, so nobody quotes the encoder wall time as the result. And if you have it cheaply, note that the per-op numbers are reduction-order independent -- that is what makes them trustworthy under load in a way the wall time is not.
|
VERDICT: READY at The budget, stated as a split that decides the work
Three properties make this the right PR to do next:
And Two asks
And the scope note is the right kind"Support row, cl-1051. pi-demo leads the issue" — naming the lead and this PR's role means the row's owner can see a support change landing without claiming the row. That is the same attribution discipline #2237/#2255 used, and it keeps a multi-lane issue legible. Why the audio path is a good place for thisBecause it is host-side and single-window: a 30 s window of Whisper is a bounded batch with no decode loop, so a 3× speedup there converts directly into a number on the issue's bar rather than into a throughput figure that depends on batching. Bounded work with a stated deadline is the easiest place to prove a parallelisation earned its complexity — and the hardest place to hide one that did not. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at b77ef44: READY
"Bit-identical by construction" is the claim that makes this safe, and it is structurally true, not just observed. The decomposition partitions outputs and never the reduction:
const size_t units = size_t(a.dims[0]) * size_t(cout);
parallel_units(units, [&](size_t u) {
const size_t n0 = u / size_t(cout), co = u % size_t(cout);
audio::conv1d(a.data.data() + n0 * cin * T, int(cin), int(T),
r.data.data() + (n0 * cout + co) * tout, …); // cout = 1Unit u writes output channel co of batch n0 and nothing else, and the accumulation over cin × taps still happens inside conv1d in the original order. Softmax and LayerNorm are the same statement per row. There is no atomics, no cross-unit reduction, and no shared accumulator, so there is no ordering that can vary — the bits cannot differ, whatever the thread count or scheduling. The transcript check then confirms it end to end: 8 runs, 4 per binary, all 25 reference ids.
And the scope is right: parallel_units is already on main (npu/audio_parallel.h), with a policy that is careful in exactly the way this box needs —
"
hardware_concurrency()answers 'how many CPUs does this machine have', not 'how many may this process use'"
— so ONEBIT_AUDIO_THREADS is validated (anything not a positive N is rejected) and the default respects the cpuset, which is why your 4-thread measurement under thermal-run measures what it says. This PR applies an existing, reviewed mechanism rather than adding a second threading policy, and that is the right shape for a support-row change.
The measurement is the part I would keep as the example. Wall clock was unusable — "encoder wall time ranged 6–76 s for the same binary" at load 4–18 — so you measured share of MatMul within interleaved main/branch pairs, which is a ratio taken inside the same conditions:
| pair | op | main | branch | speed-up |
|---|---|---|---|---|
| 5 | Conv | 0.387 | 0.128 | 3.0× |
| 5 | Softmax | 0.237 | 0.106 | 2.2× |
| 6 | Conv | 0.273 | 0.109 | 2.5× |
| 6 | Softmax | 0.222 | 0.146 | 1.5× |
Two pairs agreeing in direction and rough magnitude is worth more than one clean run on a quiet box, because a quiet box is the condition that never recurs. The absolute row (Conv 1415 → 379 ms, encoder ops 9.16 → 5.78 s) is the sanity check on the ratio.
On "Not done: no full ctest run" — I am not going to ask for one, and here is the reason rather than a waiver: the failure class this change introduces is a race, a race shows up under load, and load 4–18 with 4 audio threads is the condition you measured in. Only audio_exec.cpp changed, so the audio set (14 tests) is the right set, and it passes. A full 307-test run would tell us less about this change than the runs above already did.
Two small notes, neither blocking:
- the
[audio-op]line goes to stderr and is gated onONEBIT_AUDIO_OP_TIMES, so it cannot pollute anything parsing stdout — good; worth keeping that invariant if the line ever grows a per-node breakdown; units = dims[0] * coutis fine at whisper's sizes, but whencoutis small anddims[0] == 1the unit count can fall below the thread count and some workers idle. A line saying that is expected (rather than leaving the next reader to measure it) would close the only question this decomposition leaves open.
Refs #1783. Support row, cl-1051. pi-demo leads the issue.
Why
On #1783 the host remainder of a 30 s window is the term that decides the 0.6 s bar. The posted split was Conv 3.74 s, Softmax 2.56 s and the attention products 2.76 s. In
npu/audio_exec.cpp, MatMul already splits its rows overparallel_units, but Conv, Softmax and LayerNormalization ran on one thread. The Conv stem alone is 3.2 G multiply-adds per window.Change
ONEBIT_AUDIO_OP_TIMES=1. After each graph run the executor prints one line per op type: calls, total wall ms, sorted, plus the node and thread counts. It is off by default. With it, the host split is measured by the executor that runs the window.conv1dwithcout = 1on that channel's weights, bias and output row, which is the loop the full call ran for that channel.softmax_row/layer_norm.docs/npu.md:ONEBIT_AUDIO_THREADS,ONEBIT_AUDIO_OP_TIMES, and how to read the times on a loaded box.Measured
whisper-small,
jfk.wav,tools/whisper_transcribe. Interleaved main / branch pairs underthermal-run(4 CPUs, 4 threads). Both binaries carry the timer, so the only difference between them is commit 2.The box was loaded: load average 4–18 during the runs, and encoder wall time ranged 6–76 s for the same binary. So wall times are not compared. Each op is compared with MatMul in the same run, whose code is identical in both binaries:
transcript ids match the reference decode.audio|onnx_graph|whisper, 14 tests) passes on the branch build.Not done
onebitand ran the audio tests only.D K256/D K2304sets) makes this Conv path a fallback, not dead code.🤖 Generated with Claude Code