Skip to content

audio_exec: flat paths for equal-shape and scalar elementwise ops, 9.4x on the loop (engine#2366) - #2383

Merged
bong-water-water-bong merged 1 commit into
mainfrom
npu-bcast-fastpath
Nov 13, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
npu-bcast-fastpath

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

The elementwise ops are the largest non-GEMM family in the whisper encoder — 165 nodes, 3.6 s of it — and the loop does two dot products over the dims plus a carry step per element, then compares the op name against std::string twice per element.

For the two shapes that actually dominate the graph — equal dims (residual adds, the GELU chain) and a scalar — there is no index arithmetic to do at all.

Loop-level, in one process so load cancels (the widest elementwise shape in the encoder, 1x1500x3072 = 4.6 M elements, 20 reps each):

Add  index-decomposed  26.93 ms   flat   1.45 ms   (18.56x)
Mul  index-decomposed  16.37 ms   flat   2.81 ms   ( 5.83x)
Div  index-decomposed  23.46 ms   flat   2.84 ms   ( 8.25x)
total 66.76 -> 7.10 ms  (9.40x)

In situ, alternating the two binaries under the same load (ONEBIT_AUDIO_OP_TIMES=1, summing Add/Sub/Mul/Div/Erf over the encoder pass):

cQ2 (before)  6112.5 ms   5871.9 ms
cQ3 (after)   3451.4 ms   3351.7 ms      -> 1.76x

Less than the loop's 9.4× for two honest reasons: the family total still contains Erf at 365 ms, which this does not touch, and the executor runs four threads so part of the loop cost was already overlapped.

Byte-identical, not approximately: the arithmetic per element is unchanged (apply(x, y) is the same expression), only the index source changes, so unlike the softmax change in #2379 there is no summation-order question at all. Checked anyway: audio_ops_test rc 0, and whisper_transcribe on jfk.wav returns the same 23 ids. The scalar path is used by the graph's own scalar multiplies, and anything else (a real broadcast) still takes the old loop unchanged.

Remaining for #2366: after #2379 and this, the elementwise family is ~3.4 s at this load. Its biggest single member is now Erf (365 ms, 14 calls) — std::erf per element, which is a different problem (a polynomial approximation would change results, so it needs the ids check to justify itself) — and Conv 0.75 s is untouched.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Overlap note: #2382 (opened shortly before this) parallelises the same elementwise/Transpose loops in npu/audio_exec.cpp across ranges. The two compose — #2382 gives more units, this makes each unit's per-element cost 9.4× smaller — and the landing order only matters for who rebases: the fast path is per-element and drops into a parallel range's loop body unchanged.

Not closing either as a duplicate; flagging so whichever lands second does not discover the conflict at merge time.

@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2366 - Partially compliant

Compliant requirements:

  • The PR introduces optimizations for elementwise operations that reduce loop overhead by 9.4x, which aligns with the goal of reducing the time spent in non-GEMM operations
  • The changes are validated with measurable performance improvements (66.76 ms -> 7.10 ms for Add/Mul/Div operations)
  • The outputs remain byte-identical, ensuring correctness
  • The optimizations target the specific bottlenecks mentioned in the ticket (equal-shape and scalar elementwise ops)

Non-compliant requirements:

  • The PR does not directly address the Conv or Softmax operations mentioned in the ticket, which are the primary bottlenecks
  • The PR does not include measurements for the full encoder timing (21.7 s window) to confirm the overall improvement

Requires further human verification:

  • The full impact on the encoder timing (the 21.7 s window) needs to be verified with the complete whisper transcription pipeline
  • The performance gains need to be confirmed on the actual hardware (XDNA 2 NPU) with the full encoder workload
⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Performance Optimization May Not Address Primary Bottleneck

The PR optimizes elementwise operations (Add, Mul, Div) for equal-shape and scalar cases, achieving a 9.4x speedup in the loop. However, the primary bottlenecks identified in the ticket (#2366) are Conv and Softmax operations, which are not addressed by this change. While the optimization is beneficial, it may not significantly impact the overall encoder performance as measured in the 21.7 s window.

void broadcast_binary(const Tensor& a, const Tensor& b, const std::string& op, Tensor* out) {
    const Broadcast p = plan_broadcast(a.dims, b.dims);
    out->dims = p.dims;
    out->data.resize(nelem(p.dims));
    // engine#2366: 165 of the whisper encoder's nodes are one of these ops, 3.6 s of it, and the loop below does two dot
    // products over the dims plus a carry step PER ELEMENT -- and compares the op name with std::string, twice, per
    // element. For the two shapes that dominate the graph (equal dims: residual adds and the GELU chain; a scalar) there is
    // no index arithmetic at all. Measured on the widest elementwise shape (1x1500x3072, 20 reps each, one process so load
    // cancels): 66.76 ms -> 7.10 ms for Add/Mul/Div, **9.40x**, and the outputs are byte-identical because nothing about
    // the arithmetic changes -- only where the indices come from.
    const int kind = op == "Add" ? 0 : op == "Sub" ? 1 : op == "Mul" ? 2 : 3;
    auto apply = [kind](float x, float y) {
        return kind == 0 ? x + y : kind == 1 ? x - y : kind == 2 ? x * y : x / y;
    };
    const size_t n = out->data.size();
    if (a.dims == p.dims && b.dims == p.dims) {
        for (size_t o = 0; o < n; ++o) out->data[o] = apply(a.data[o], b.data[o]);
        return;
    }
    if (b.data.size() == 1) {
        const float y = b.data[0];
        for (size_t o = 0; o < n; ++o) out->data[o] = apply(a.data[o], y);
        return;
    }
    std::vector<size_t> idx(p.dims.size(), 0);
    for (size_t o = 0; o < out->data.size(); ++o) {
        size_t ia = 0, ib = 0;
        for (size_t d = 0; d < idx.size(); ++d) {
            ia += idx[d] * p.stride_big[d];
            ib += idx[d] * p.stride_small[d];
        }
        const float x = a.data[ia], y = b.data[ib];
        out->data[o] = apply(x, y);
        for (size_t d = idx.size(); d-- > 0;) {
            if (++idx[d] < size_t(p.dims[d])) break;
            idx[d] = 0;
        }

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- the string compare was the defect, and the flat paths are the reward

std::string compared twice per element is the sentence that matters. Two 100-nanosecond comparisons per element on 165 nodes over 1.5M-element tensors is the kind of cost that never shows up as a hotspot in a profile because it is not a kernel -- it is inside the loop, next to the arithmetic. Hoisting the op to an int kind once is the whole first half of the win, and it is the half that helps every shape rather than the two that dominate.

The measurements are structured so the claim can be checked:

op index-decomposed flat
Add 26.93 ms 1.45 ms 18.56x
Mul 16.37 ms 2.81 ms 5.83x
Div 23.46 ms 2.84 ms 8.25x
total 66.76 ms 7.10 ms 9.40x

on the widest elementwise shape (1x1500x3072, 20 reps, one process so the box's clock state cancels) -- and then the number that actually matters: cQ2 6112.5/5871.9 ms against cQ3 3451.4/3351.7 ms, 1.76x end to end. A loop-level 9.4x that turned into 1.76x of the encoder is exactly the ratio a reader needs to judge whether the remaining 3.3 s is worth chasing, and it is the fraction my review of #2382 asked for.

And byte-identity is by construction rather than by test: "nothing about the arithmetic changes -- only where the indices come from." That is the right kind of claim, because it can be checked by reading the two paths instead of trusting a hash.

One ask, about the path this change does not take. The flat fast paths cover equal dims and scalar; everything else still goes through the index-decomposed loop. After this PR that loop is the fallback, and a fallback nobody exercises is where the next defect lives -- particularly since the kind hoist now feeds both paths, so a wrong kind mapping (say Sub and Div swapped in the ternary) would be invisible in the flat path's own test and only wrong in the decomposed one. test_elementwise_bit_identity from #2382 is the right home for that arm: if it covers a bias-row Add and a scalar Mul, add one genuinely broadcasting, non-equal, non-scalar case (e.g. [1,4,1] against [1,1,7]) so the decomposed path with the hoisted kind is compared against the reference too. #2382's own body already has the negative-control idea in it (a forced-wrong Transpose must trip), and the same shape works here.

Nothing else: 21 lines for a 1.76x on the encoder's largest non-GEMM family, and the comment carries the measurement and the reason the outputs cannot change.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at ce11ec4 — 9.40× on the op is real and measured the right way; one line in it can silently become a wrong answer, and the row it serves needs the end-to-end number, not the op's.

The measurement is honest, and it uses the trick that works under load

1x1500x3072, 20 reps each, one process (so the box's load cancels): 66.76 ms -> 7.10 ms for Add/Mul/Div, 9.40x

Both arms inside one process, so whatever the box was doing applies to both — the same shape as engine#2382's changed-ops/MatMul ratios, and the reason the number is believable where a bare wall time would not be. And the diagnosis is the part that makes the 9.4× obvious in hindsight: "the loop does two dot products over the dims plus a carry step per element, then compares the op name against std::string twice per element." Two string comparisons per element on 4.6M elements is ~9M compares for arithmetic that needs none.

The one line I would not merge as written

const int kind = op == "Add" ? 0 : op == "Sub" ? 1 : op == "Mul" ? 2 : 3;   // 3 == Div

Any op name that is not Add/Sub/Mul is treated as Div. Today that is unreachable — the only call site is npu/audio_exec.cpp:366, guarded by n.op == "Add" || "Mul" || "Div" || "Sub" — so this is not a live bug. It is a plausible default standing in for a parsed value, which is the shape this fleet keeps paying for: the guard and the parse are 250 lines apart in two different functions, and the natural next change (Max, Mix, Pow, or the Gelu that sits twenty lines below the call site) turns into a silent division with no warning, at 9.4× speed, on an audio model whose output is a transcript.

Make the parse refuse: switch, every case listed, default: GGML_ABORT("audio: %s is not a binary elementwise op", op.c_str()). That is the same number of lines, it makes the invariant local to the function that depends on it, and -Wswitch then keeps it honest when an op is added. (Same family as assert "" not in _CANON_LINES and the AUDIO_ARGS index check: make the hazard unrepresentable rather than remembering it.)

And the number the row needs

#2366's row is a time budget: 165 nodes, 3.6 s of the encoder's 21.7 s window. A 9.40× on the widest shape is the mechanism, not the outcome — if the 3.6 s is mostly these shapes, the row moves by ~3.2 s, and if it is mostly the narrow or middle-axis-broadcast shapes that take the general path, it does not move at all. One line: the whisper encoder's elementwise total before and after, same input, same process, with the broadcast forms called out separately so it is clear how much of the 3.6 s this path actually reaches. That single number is what turns this PR from "a hot loop got 9× faster" into "the row's 3.6 s is now X".

bong-water-water-bong pushed a commit that referenced this pull request Nov 13, 2026
A scalar or column operand has last-axis stride 0, which kept the row loop from vectorising. Read it once per row instead,
for either side. The operation per element is unchanged. On scalar Mul, 1x1500x3072, one thread: 1.39 -> 0.59 ms (#2383's
flat loop: 0.49). The test's Sub now puts the column on the left so both branches are covered; an operand-swap mutant of
the new branch fails it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong
bong-water-water-bong merged commit 19fddb7 into main Nov 13, 2026
9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the npu-bcast-fastpath branch November 13, 2026 22:21
bong-water-water-bong pushed a commit that referenced this pull request Nov 13, 2026
…'s threads, on top of #2383's flat paths (refs #2366)

Rebuilt on main 6605b86, which carries #2383's flat paths in broadcast_binary. Those stay as the first two cases of
broadcast_apply: equal dims and a scalar operand. They now run in kElemUnit ranges on parallel_units. Every other
broadcast (bias rows, columns) takes a row loop: a row's source offsets are computed once, a stride-0 operand is read
once per row, and row blocks are split over threads. Erf, Gelu and Transpose split their output into ranges. Each element
is still the serial loop's one float operation on the same operands, so the output is bit-identical at any thread count.

tests/audio_parallel_test.cpp: an Add (bias row), Mul (scalar), Sub (column on the left), Div (equal shape), Erf,
Transpose chain at 301x193, memcmp against the plain serial arithmetic at 1-64 threads. Mutants of the equal-shape path,
the stride-0 branch, the broadcast stride and the Transpose start index each fail it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bong-water-water-bong pushed a commit that referenced this pull request Nov 13, 2026
…'s threads, on top of #2383's flat paths (refs #2366)

Rebuilt on main 6605b86, which carries #2383's flat paths in broadcast_binary. Those stay as the first two cases of
broadcast_apply: equal dims and a scalar operand. They now run in kElemUnit ranges on parallel_units. Every other
broadcast (bias rows, columns) takes a row loop: a row's source offsets are computed once, a stride-0 operand is read
once per row, and row blocks are split over threads. Erf, Gelu and Transpose split their output into ranges. Each element
is still the serial loop's one float operation on the same operands, so the output is bit-identical at any thread count.

tests/audio_parallel_test.cpp: an Add (bias row), Mul (scalar), Sub (column on the left), Div (equal shape), Erf,
Transpose chain at 301x193, memcmp against the plain serial arithmetic at 1-64 threads. Mutants of the equal-shape path,
the stride-0 branch, the broadcast stride and the Transpose start index each fail it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bong-water-water-bong added a commit that referenced this pull request Nov 14, 2026
…'s threads, on top of #2383's flat paths (refs #2366) (#2382)

Rebuilt on main 6605b86, which carries #2383's flat paths in broadcast_binary. Those stay as the first two cases of
broadcast_apply: equal dims and a scalar operand. They now run in kElemUnit ranges on parallel_units. Every other
broadcast (bias rows, columns) takes a row loop: a row's source offsets are computed once, a stride-0 operand is read
once per row, and row blocks are split over threads. Erf, Gelu and Transpose split their output into ranges. Each element
is still the serial loop's one float operation on the same operands, so the output is bit-identical at any thread count.

tests/audio_parallel_test.cpp: an Add (bias row), Mul (scalar), Sub (column on the left), Div (equal shape), Erf,
Transpose chain at 301x193, memcmp against the plain serial arithmetic at 1-64 threads. Mutants of the equal-shape path,
the stride-0 branch, the broadcast stride and the Transpose start index each fail it.

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant