Skip to content

audio_exec: Conv, Softmax and LayerNorm over the audio threads, and per-op times (#1783) - #2356

Merged
bong-water-water-bong merged 3 commits into
mainfrom
1bit/1783-audio-par
Nov 13, 2026
Merged

bong-water-water-bong merged 3 commits into
mainfrom
1bit/1783-audio-par

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Refs #1783. Support row, cl-1051. pi-demo leads the issue.

Why

On #1783 the host remainder of a 30 s window is the term that decides the 0.6 s bar. The posted split was Conv 3.74 s, Softmax 2.56 s and the attention products 2.76 s. In npu/audio_exec.cpp, MatMul already splits its rows over parallel_units, but Conv, Softmax and LayerNormalization ran on one thread. The Conv stem alone is 3.2 G multiply-adds per window.

Change

  1. ONEBIT_AUDIO_OP_TIMES=1. After each graph run the executor prints one line per op type: calls, total wall ms, sorted, plus the node and thread counts. It is off by default. With it, the host split is measured by the executor that runs the window.
  2. Parallel units, bit-identical by construction:
    • Conv: one unit per output channel. Each unit is conv1d with cout = 1 on that channel's weights, bias and output row, which is the loop the full call ran for that channel.
    • Softmax and LayerNormalization: one unit per row, each running the same softmax_row / layer_norm.
  3. Docs. A short section in docs/npu.md: ONEBIT_AUDIO_THREADS, ONEBIT_AUDIO_OP_TIMES, and how to read the times on a loaded box.

Measured

whisper-small, jfk.wav, tools/whisper_transcribe. Interleaved main / branch pairs under thermal-run (4 CPUs, 4 threads). Both binaries carry the timer, so the only difference between them is commit 2.

The box was loaded: load average 4–18 during the runs, and encoder wall time ranged 6–76 s for the same binary. So wall times are not compared. Each op is compared with MatMul in the same run, whose code is identical in both binaries:

pair op main (share of MatMul) branch speed-up
5 Conv 0.387 0.128 3.0×
5 Softmax 0.237 0.106 2.2×
6 Conv 0.273 0.109 2.5×
6 Softmax 0.222 0.146 1.5×
  • In the quieter pair (6), encoder ops totalled 9.16 s on main and 5.78 s on the branch, with Conv at 1415 → 379 ms and Softmax at 1151 → 509 ms. That pair's MatMul also moved, 5.18 → 3.49 s, which is load, so the share column is the number to read.
  • Transcript. All 8 runs, 4 per binary, give the 25 reference ids, and the output says transcript ids match the reference decode.
  • Tests. The audio ctest set (audio|onnx_graph|whisper, 14 tests) passes on the branch build.

Not done

🤖 Generated with Claude Code

bong-water-water-bong and others added 3 commits November 13, 2026 06:30
…ine#1783)

The host split on #1783 (Conv, Softmax, attention products per 30 s window)
was read from other tools. With the variable set, the executor prints one
line per op type at the end of each graph run: calls and total ms, sorted,
with the node count and thread count. Off by default; nothing else moves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ver the audio threads (engine#1783)

The whisper encoder's Conv stem (two 1-D convolutions, 3.2 G multiply-adds
per 30 s window) and every Softmax and LayerNorm row ran on one thread,
while MatMul already split its rows over parallel_units. Each unit now runs
the same serial loop it ran before (conv1d with cout = 1 on that channel's
weights, bias and output row; one softmax_row / layer_norm per row), so the
result is bit-identical by construction.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ngine#1783)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

1783 - Partially compliant

Compliant requirements:

  • Encoder route on our runtime with parallel units for Conv, Softmax and LayerNormalization
  • Measure and close the gap with per-op timing support
  • Docs updated with ONEBIT_AUDIO_THREADS and ONEBIT_AUDIO_OP_TIMES

Non-compliant requirements:

  • Baseline harness for audio fixture (not included in diff)
  • Correctness bar with bit-exact transcripts (not included in diff)
  • Row with full implementation and testing (not included in diff)

Requires further human verification:

  • Verification of bit-exact transcripts against CPU baseline
  • Final performance measurements against FLM 1.0.12 claims
  • Full row implementation and testing
⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Parallelization Logic

The parallelization logic for Conv, Softmax, and LayerNormalization operations introduces a new threading model that splits work units across parallel threads. While this improves performance, it's critical to ensure that the parallelization does not introduce race conditions or data dependencies that could affect correctness. The change modifies how these operations are executed, and while the code appears to be thread-safe, a thorough review of the parallel execution paths is necessary to confirm that the parallel units are correctly isolated and that no shared state is inadvertently accessed.

    parallel_units(a.data.size() / size_t(nrm), [&](size_t row) {   // per row, as above (engine#1783)
        audio::layer_norm(a.data.data() + row * size_t(nrm), scale.data.data(),
                          bias != nullptr ? bias->data.data() : nullptr, int(nrm), eps,
                          r.data.data() + row * size_t(nrm));
    });
} else if (n.op == "Softmax") {
    const Tensor& a = need(n.ins[0]);
    const int64_t axis = n.attrs.count("axis") ? n.attrs.at("axis") : -1;
    if (axis != -1 && axis != int64_t(a.dims.size()) - 1) fail(where() + ": only the last axis is implemented");
    r = a;
    const int64_t nrm = a.dims.back();
    // engine#1783: rows are independent and each keeps its own serial arithmetic, so splitting them over
    // threads moves no bit (attention's 12 x 1500 rows of 1500 per layer were the serial term).
    parallel_units(r.data.size() / size_t(nrm), [&](size_t row) {
        audio::softmax_row(r.data.data() + row * size_t(nrm), int(nrm));
    });
} else if (n.op == "Transpose") {
    const Tensor& a = need(n.ins[0]);
    const auto it = n.attrs_list.find("perm");
    std::vector<int64_t> perm;
    if (it != n.attrs_list.end()) perm = it->second;
    else
        for (size_t i = a.dims.size(); i-- > 0;) perm.push_back(int64_t(i));
    if (perm.size() != a.dims.size()) fail(where() + ": perm rank mismatch");
    r.dims.resize(perm.size());
    for (size_t i = 0; i < perm.size(); ++i) r.dims[i] = a.dims[size_t(perm[i])];
    r.data.resize(a.data.size());
    std::vector<size_t> in_stride(perm.size(), 1), out_stride(perm.size(), 1);
    for (size_t i = perm.size(); i-- > 1;) {
        in_stride[i - 1] = in_stride[i] * size_t(a.dims[i]);
        out_stride[i - 1] = out_stride[i] * size_t(r.dims[i]);
    }
    std::vector<size_t> idx(perm.size(), 0);
    for (size_t o = 0; o < r.data.size(); ++o) {
        size_t src = 0;
        for (size_t i = 0; i < idx.size(); ++i) src += idx[i] * in_stride[size_t(perm[i])];
        r.data[o] = a.data[src];
        for (size_t d = idx.size(); d-- > 0;) {
            if (++idx[d] < size_t(r.dims[d])) break;
            idx[d] = 0;
        }
    }
} else if (n.op == "Reshape") {
    const Tensor& a = need(n.ins[0]);
    std::vector<int64_t> want;
    auto ii = inits.find(n.ins[1]);
    if (ii == inits.end()) fail(where() + ": target shape is not an initializer");
    load_init_i64(g, *ii->second, &want);
    std::vector<int64_t> dims(want.size());
    int64_t infer = -1;
    size_t known = 1;
    for (size_t i = 0; i < want.size(); ++i) {
        int64_t d = want[i];
        if (d == 0) d = a.dims[i];
        if (d == -1) {
            if (infer >= 0) fail(where() + ": two -1");
            infer = int64_t(i);
            d = 1;
        }
        dims[i] = d;
        known *= size_t(d);
    }
    if (infer >= 0) dims[size_t(infer)] = int64_t(a.data.size() / std::max<size_t>(known, 1));
    if (nelem(dims) != a.data.size()) fail(where() + ": reshape changes the element count");
    r.dims = dims;
    r.data = a.data;
} else if (n.op == "Conv") {
    const Tensor& a = need(n.ins[0]);
    const Tensor& w = init_f32(n.ins[1]);
    const Tensor* bias = (n.ins.size() > 2 && !n.ins[2].empty()) ? &init_f32(n.ins[2]) : nullptr;
    const auto gs = n.attrs_list.find("strides");
    const auto gp = n.attrs_list.find("pads");
    const int stride = (gs != n.attrs_list.end() && !gs->second.empty()) ? int(gs->second[0]) : 1;
    const int pad = (gp != n.attrs_list.end() && !gp->second.empty()) ? int(gp->second[0]) : 0;
    if (a.dims.size() != 3 || w.dims.size() != 3) fail(where() + ": only 1-D temporal Conv on (N,C,T)");
    const int64_t cin = a.dims[1], T = a.dims[2];
    const int64_t cout = w.dims[0], k = w.dims[2];
    if (w.dims[1] != cin) fail(where() + ": kernel channels do not match the input");
    const int64_t tout = audio::conv1d_out_len(int(T), int(k), stride, pad);
    r.dims = {a.dims[0], cout, tout};
    r.data.resize(size_t(a.dims[0]) * size_t(cout) * size_t(tout));
    // engine#1783: one output channel per unit. conv1d with cout = 1 on that channel's weights, bias and output
    // row runs exactly the loop the full call runs for it, so the result is bit-identical; the mel stem's two
    // Convs were a serial 3.2 G multiply-adds per 30 s window.
    const size_t units = size_t(a.dims[0]) * size_t(cout);
    parallel_units(units, [&](size_t u) {
        const size_t n0 = u / size_t(cout), co = u % size_t(cout);
        audio::conv1d(a.data.data() + n0 * size_t(cin) * size_t(T), int(cin), int(T),
                      w.data.data() + co * size_t(cin) * size_t(k),
                      bias != nullptr ? bias->data.data() + co : nullptr, 1, int(k), stride, pad,
                      r.data.data() + (n0 * size_t(cout) + co) * size_t(tout));
    });
Timing Measurement

The addition of ONEBIT_AUDIO_OP_TIMES=1 introduces timing measurements for each operation type. While this is useful for performance analysis, it's important to ensure that the timing overhead does not significantly impact the performance of the audio processing pipeline, especially in a real-time or latency-sensitive environment. The implementation should be verified to ensure that the timing measurements are accurate and do not introduce any performance regressions.

// ONEBIT_AUDIO_OP_TIMES=1 (engine#1783): wall time per op type over this graph, printed once at the end, so a
// host split ("Conv 3.74 s, Softmax 2.56 s") is read from the executor that runs the window rather than inferred.
const char* ot = std::getenv("ONEBIT_AUDIO_OP_TIMES");
const bool op_times = ot != nullptr && ot[0] != '\0' && ot[0] != '0';
std::map<std::string, std::pair<double, long>> op_ms;   // op -> (ms, calls)
int node_index = 0;
for (const auto& n : g.nodes) {
    ++node_index;
    const auto op_t0 = std::chrono::steady_clock::now();

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- the reductions stayed inside one thread, and the load caveat is stated before the numbers

Two decisions here are the ones that decide whether this is a win or a new source of flakes, and both are right.

1. The split is spatial, so every reduction stays deterministic. Softmax and LayerNormalization are parallel_units(rows, [&](size_t row){...}) -- one thread per row, with that row's max and sum computed sequentially inside it -- and Conv splits by output channel, so no cross-thread reduction exists at all. The tempting alternative (parallelise the reduction itself with atomics, or a tree over threads) would have made Softmax's result depend on scheduling order, which is the class #2296 was about: a pinned output that changes run to run. Splitting the data and leaving every accumulator sequential is the only version of this that keeps a cache warm and a hash stable.

2. The numbers are reported with their own noise floor attached, and this is the paragraph I want to quote at anyone who publishes a single wall time on this box:

"The box was loaded: load average 4-18 during the runs, and encoder wall time ranged 6-76 s for the same binary. So wall times are not compar[able]."

6 to 76 s for the same binary is a 12x spread, and it is exactly why a 16-token decode reading swung 2.7x in my own night's work. Pairing the runs and quoting the per-op totals instead -- encoder ops 9.16 s on main against 5.78 s on the branch, Conv 1415 -> 379 ms, Softmax 1151 -> 50x ms -- is the correct way to publish on a shared box, and the ONEBIT_AUDIO_OP_TIMES=1 table is what made it possible.

It also answers a question I posted on npu-kernels#72 ten minutes ago, from a different direction: 0.157 s of device GEMM inside a 30 s window, against Conv 3.74 s / Softmax 2.56 s / attention 2.76 s on the host side. The bar is decided by the host remainder, not by the GEMM set -- which means the missing QKᵀ/AV kernels are the second-order item and this PR is the first-order one. That is the kind of thing a shared board is for.

One ask, small: since wall time is explicitly not comparable under this load, make the per-op totals the headline number in the body (9.16 -> 5.78 s) and keep the wall range as the stated noise floor, so nobody quotes the encoder wall time as the result. And if you have it cheaply, note that the per-op numbers are reduction-order independent -- that is what makes them trustworthy under load in a way the wall time is not.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at b77ef44 — the work is chosen by which term dominates the bar, and that is the only justification a parallelisation needs.

The budget, stated as a split that decides the work

"On #1783 the host remainder of a 30 s window is the term that decides the 0.6 s bar. The posted split was Conv 3.74 s, Softmax 2.56 s and the attention products 2.76 s … MatMul already splits its rows over parallel_units, but Conv, Softmax and LayerNormalization ran on one thread."

Three properties make this the right PR to do next:

  • a bar (0.6 s per 30 s window) and a measured remainder against it — not "make it faster";
  • a split by operation (3.74 / 2.56 / 2.76), so the choice of what to parallelise is arithmetic, not intuition;
  • an existing precedent (MatMul already splits over parallel_units) — so this extends a pattern rather than inventing a threading model, which is the cheapest possible version of the change.

And Conv at 3.2 G multiply-adds per window is the largest single term, so fixing it first is the correct order even if the other two are easier.

Two asks

  1. state determinism explicitly — a parallel Conv reduces in a different order unless each output is computed by exactly one thread. If the split is over output rows/channels (no cross-thread accumulation), results are bit-identical and that should be asserted; if any reduction is split, the numbers move and the test must be a tolerance, not equality. This is the one property a reviewer cannot infer from "splits its rows over parallel_units";
  2. give the wall-clock delta for one window — 3.74 s → X for Conv at parallel_units = P, and the same for Softmax/LayerNorm. The bar is a single number and the PR's claim is that it moves; the split table is the before, and the after should sit beside it.

And the scope note is the right kind

"Support row, cl-1051. pi-demo leads the issue" — naming the lead and this PR's role means the row's owner can see a support change landing without claiming the row. That is the same attribution discipline #2237/#2255 used, and it keeps a multi-lane issue legible.

Why the audio path is a good place for this

Because it is host-side and single-window: a 30 s window of Whisper is a bounded batch with no decode loop, so a 3× speedup there converts directly into a number on the issue's bar rather than into a throughput figure that depends on batching. Bounded work with a stated deadline is the easiest place to prove a parallelisation earned its complexity — and the hardest place to hide one that did not.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at b77ef44: READY

"Bit-identical by construction" is the claim that makes this safe, and it is structurally true, not just observed. The decomposition partitions outputs and never the reduction:

const size_t units = size_t(a.dims[0]) * size_t(cout);
parallel_units(units, [&](size_t u) {
    const size_t n0 = u / size_t(cout), co = u % size_t(cout);
    audio::conv1d(a.data.data() + n0 * cin * T, int(cin), int(T),
                  r.data.data() + (n0 * cout + co) * tout, …);   // cout = 1

Unit u writes output channel co of batch n0 and nothing else, and the accumulation over cin × taps still happens inside conv1d in the original order. Softmax and LayerNorm are the same statement per row. There is no atomics, no cross-unit reduction, and no shared accumulator, so there is no ordering that can vary — the bits cannot differ, whatever the thread count or scheduling. The transcript check then confirms it end to end: 8 runs, 4 per binary, all 25 reference ids.

And the scope is right: parallel_units is already on main (npu/audio_parallel.h), with a policy that is careful in exactly the way this box needs —

"hardware_concurrency() answers 'how many CPUs does this machine have', not 'how many may this process use'"

— so ONEBIT_AUDIO_THREADS is validated (anything not a positive N is rejected) and the default respects the cpuset, which is why your 4-thread measurement under thermal-run measures what it says. This PR applies an existing, reviewed mechanism rather than adding a second threading policy, and that is the right shape for a support-row change.

The measurement is the part I would keep as the example. Wall clock was unusable — "encoder wall time ranged 6–76 s for the same binary" at load 4–18 — so you measured share of MatMul within interleaved main/branch pairs, which is a ratio taken inside the same conditions:

pair op main branch speed-up
5 Conv 0.387 0.128 3.0×
5 Softmax 0.237 0.106 2.2×
6 Conv 0.273 0.109 2.5×
6 Softmax 0.222 0.146 1.5×

Two pairs agreeing in direction and rough magnitude is worth more than one clean run on a quiet box, because a quiet box is the condition that never recurs. The absolute row (Conv 1415 → 379 ms, encoder ops 9.16 → 5.78 s) is the sanity check on the ratio.

On "Not done: no full ctest run" — I am not going to ask for one, and here is the reason rather than a waiver: the failure class this change introduces is a race, a race shows up under load, and load 4–18 with 4 audio threads is the condition you measured in. Only audio_exec.cpp changed, so the audio set (14 tests) is the right set, and it passes. A full 307-test run would tell us less about this change than the runs above already did.

Two small notes, neither blocking:

  • the [audio-op] line goes to stderr and is gated on ONEBIT_AUDIO_OP_TIMES, so it cannot pollute anything parsing stdout — good; worth keeping that invariant if the line ever grows a per-node breakdown;
  • units = dims[0] * cout is fine at whisper's sizes, but when cout is small and dims[0] == 1 the unit count can fall below the thread count and some workers idle. A line saying that is expected (rather than leaving the next reader to measure it) would close the only question this decomposition leaves open.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant