Skip to content

audio_exec: split the elementwise ops and Transpose over the executor's threads, bit-identical (refs #2366) - #2382

Merged
bong-water-water-bong merged 1 commit into
mainfrom
cl-npuperf/2366-elementwise-threads
Nov 14, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
cl-npuperf/2366-elementwise-threads

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Refs #2366. This is one part of that issue (the host loops that were still single-threaded), not its 0.6 s close bar, so this PR does not close it.

What changes

npu/audio_exec.cpp ran Add/Sub/Mul/Div (all broadcast forms), Erf, Gelu and Transpose single-threaded. The binary ops also did a string compare and a full multi-axis index walk for every element. Now:

  • binary ops: the op is resolved once. The work is split into one output row per step, with the row's source offsets computed once and then stepped by the last-axis strides. Rows are split over parallel_units.
  • Erf, Gelu and Transpose: split into ranges of 16384 output elements. Each Transpose range starts its index walk from its own first element.

Every element is still the serial loop's one float operation on the same operands, so the output is bit-identical at any ONEBIT_AUDIO_THREADS. That is the same property audio_parallel.h already promises for MatMul, Conv, Softmax and LayerNorm.

Proof

Test: tests/audio_parallel_test.cpp gains test_elementwise_bit_identity. An in-memory chain runs Add with a bias row, Mul with a scalar, Sub with an [M,1] column, Div elementwise, then Erf and Transpose {0,2,1}, at 301x193 so it spans several units with rows that don't divide a unit. The output is compared with memcmp against plain serial arithmetic written out in the test, at 1, 2, 3, 7, 8 and 64 threads and unset. audio_parallel: ok.

Mutants (each fails the new test, then restored):

  • Transpose start index forced to 0: FAIL ... at ONEBIT_AUDIO_THREADS=1/2/3...
  • broadcast reads b with a's last-axis stride: same FAIL lines.

End to end: whisper_transcribe on whisper-small with jfk.wav and tests/data/whisper-jfk-ids.json, ONEBIT_AUDIO_OP_TIMES=1, under thermal-run (4 CPUs). origin/main 8dbccd4 was interleaved with this branch, 3 reps each.

  • transcript ids match the reference decode in all 6 runs.
  • The box was loaded (load average 10-15), so wall times scatter by 2x between reps of the same binary and are not used. As docs/npu.md advises, the changed ops (Add, Sub, Mul, Div, Erf, Gelu, Transpose) are compared with MatMul in the same run:
run encoder: changed ops / MatMul decoder (25 tokens): changed ops / MatMul
main r1 / r2 / r3 0.203 / 0.199 / 0.224 0.264 / 0.295 / 0.308
this r1 / r2 / r3 0.105 / 0.105 / 0.101 0.191 / 0.177 / 0.161

In absolute terms, the encoder's changed ops took 1770-4133 ms on main and 527-1246 ms on this branch. Relative to MatMul in the same run, the elementwise and Transpose share of the encoder halves. The decoder's falls by about 40%.

What this does not claim: a wall-clock encoder number. That needs a quiet box, and #2366's per-node instrument (#2377) is the way to take it.

Other checks: tools/copyright.py --check rc=0. Docs: docs/npu.md (Whisper host chain section) lists the new split ops. No device was used.

🤖 Generated with Claude Code

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

ready for review at 5245fc4

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit ea5ea3b)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2366 - Partially compliant

Compliant requirements:

  • Elementwise ops and Transpose are now split over executor threads
  • Bit-identical output is guaranteed across different thread counts
  • Tests verify the bit-identical behavior
  • Documentation updated to reflect the new threading behavior

Non-compliant requirements:

  • The PR does not claim to have achieved the final performance target of 0.29s window for the encoder

Requires further human verification:

  • Performance measurements to confirm the 9.06s reduction claimed in the ticket
  • Verification that the encoder now meets the 0.6s per 30s window requirement
⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Potential Performance Regression in Transpose

The new transpose implementation uses a nested loop to compute indices for each output element, which may be less efficient than a direct memory copy or a more optimized transpose algorithm, especially for large tensors. This could introduce a performance regression compared to the previous implementation.

// engine#2366: kElemUnit-sized output ranges on the executor's threads, each starting its index walk from
// its first element. A copy, so any split is bit-identical.
const size_t total = r.data.size();
parallel_units((total + kElemUnit - 1) / kElemUnit, [&](size_t u) {
    const size_t o0 = u * kElemUnit, o1 = std::min(total, o0 + kElemUnit);
    std::vector<size_t> idx(perm.size(), 0);
    for (size_t d = 0, rem = o0; d < idx.size(); ++d) {
        idx[d] = rem / out_stride[d];
        rem %= out_stride[d];
    }
    for (size_t o = o0; o < o1; ++o) {
        size_t src = 0;
        for (size_t i = 0; i < idx.size(); ++i) src += idx[i] * in_stride[size_t(perm[i])];
        r.data[o] = a.data[src];
        for (size_t d = idx.size(); d-- > 0;) {
            if (++idx[d] < size_t(r.dims[d])) break;
            idx[d] = 0;
        }
    }
});
Thread Safety in Broadcast Rows

The new broadcast_rows function uses parallel_units to distribute work across threads, but it's not clear if the stride_big and stride_small arrays are accessed in a thread-safe manner. If these arrays are shared across threads without proper synchronization, it could lead to data races or incorrect results.

void broadcast_rows(const Tensor& a, const Tensor& b, const Broadcast& p, Tensor* out, Op op) {
    const size_t rank = p.dims.size();
    const size_t width = rank == 0 ? 1 : size_t(p.dims[rank - 1]);
    const size_t rows = width == 0 ? 0 : out->data.size() / width;
    const size_t sa = rank == 0 ? 0 : p.stride_big[rank - 1], sb = rank == 0 ? 0 : p.stride_small[rank - 1];
    const size_t rows_per_unit = std::max<size_t>(1, kElemUnit / std::max<size_t>(1, width));
    const float* ad = a.data.data();
    const float* bd = b.data.data();
    float* od = out->data.data();
    parallel_units((rows + rows_per_unit - 1) / rows_per_unit, [&](size_t u) {
        const size_t r0 = u * rows_per_unit, r1 = std::min(rows, r0 + rows_per_unit);
        for (size_t row = r0; row < r1; ++row) {
            size_t ia = 0, ib = 0, rem = row;
            for (size_t d = rank > 0 ? rank - 1 : 0; d-- > 0;) {   // the row's index on every axis but the last
                const size_t i = rem % size_t(p.dims[d]);
                rem /= size_t(p.dims[d]);
                ia += i * p.stride_big[d];
                ib += i * p.stride_small[d];
            }
            float* o = od + row * width;
            // A last-axis stride of 0 (a scalar, a column) is one value for the whole row: read it once, which lets the
            // loop vectorise (scalar Mul on 1x1500x3072, one thread: 1.39 -> 0.6 ms). Same operation per element.
            if (sb == 0) {
                const float y = bd[ib];
                for (size_t j = 0; j < width; ++j) o[j] = op(ad[ia + j * sa], y);
            } else if (sa == 0) {
                const float x = ad[ia];
                for (size_t j = 0; j < width; ++j) o[j] = op(x, bd[ib + j * sb]);
            } else {
                for (size_t j = 0; j < width; ++j) o[j] = op(ad[ia + j * sa], bd[ib + j * sb]);
            }
        }
    });
}

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

On pr-agent's two focus areas at 5245fc4:

  • Thread safety in broadcast_rows: p (dims, stride_big, stride_small) and both inputs are only read inside the units. Each unit writes only its own rows [r0, r1) of out, which is the audio_parallel.h contract. The new test runs the chain at 1-64 threads and compares the result with memcmp.
  • Transpose speed: the per-element index walk is main's loop unchanged. The PR only splits it into ranges, each starting its walk at its first element. In the A/B in the body, Transpose is among the ops whose share of the run against MatMul halved. A blocked transpose would be a separate change, and this PR doesn't claim one.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 5245fc4 — and the two paragraphs I would copy into the fleet's method notes are the ones about what the change does not prove.

The measurements are the right shape, and the load handling is the part to keep

"The box was loaded (load average 10-15), so wall times scatter by 2x between reps of the same binary and are not used" — and then the table quotes changed ops / MatMul, a ratio against an unchanged op measured in the same run:

run              encoder: changed ops / MatMul    decoder: changed ops / MatMul
main r1/r2/r3    0.203 / 0.199 / 0.224            0.264 / 0.295 / 0.308
this  r1/r2/r3   0.105 / 0.105 / 0.101            0.191 / 0.177 / 0.161

That is the technique this fleet has been missing all night. engine#2369 measured 7.94 / 11.78 / 10.39 / 12.57 tok/s on four identical runs of one command — a 1.58× spread — and the standing answer has been "quote the spread". A ratio to an unchanged op inside the same run divides the load out, and the ratios here scatter far less (0.101–0.105 against 0.199–0.224) than the seconds would. Worth one sentence in the body naming it as the method, because the next lane that measures under load needs the trick, not just the result.

And the header disowns its own earlier claim, which is the strongest thing here

"What this is NOT: a measured speedup. Three interleaved repetitions at 4 CPUs (8 threads / 4 threads) gave encode 10.38/10.40 s and decode 2.58/2.69 s per token — within the box's own noise, and my first, non-interleaved comparison of 8 threads against serial (5.21 against 4.24 s/token) was load, not threading."

A lane that writes down "my earlier number was load" in the code that inherits the decision is doing the thing this fleet keeps asking for and rarely gets. Same for audio_thread_count(): unset → the affinity mask, not hardware_concurrency(), with the measured reason ("it over-reports: … the capped-at-8 default spawned 8 threads for 4 CPUs"), and a typo → 1, "so a typo cannot turn into 'as many as the parse produced'". Those are three independent guards that each refuse to guess.

Verified, since they are the things that usually go wrong

  • The test is registered: CMakeLists.txt:2028 add_test(NAME audio_parallel COMMAND audio_parallel_test), executable at :1600, and add_dependencies(onebit … audio_parallel_test …) — it cannot silently not run (the tests_referenced: a tests/*.py that invokes a tool's --self-test is a run site (port of #1912's one extra dimension) #2306/tests_referenced: a row that names a script which is not in the tree is drift #2346 trap).
  • The split cannot compute an empty range: parallel_units(units, …) hands out unit indices (rows_per_unit = max(1, kElemUnit / max(1, width))), so with 2 rows and 8 threads six workers are simply not called, rather than being handed begin > end. That is the safe shape, and it is why the 16384-element unit size needs no special case.
  • Bit-identity is sound by argument, not by hope: every op here is elementwise (Add/Sub/Mul/Div with broadcast, Erf, Gelu, Transpose), each output element is one float op on the same operands, so no reduction is reordered — the output cannot depend on the split. The two mutants (Transpose start index forced to 0; broadcast reading b with a's last-axis stride) are the right ones: each breaks exactly that property.

Two lines I would add

  1. say what the default now covers. audio_parallel.h pre-dates this PR (MatMul already used it), and the elementwise ops now inherit its default — so the count for Add/Sub/Mul/Div/Erf/Gel/Transpose went from 1 to min(affinity, 8). One clause, because "the executor is single-threaded on these ops" was the premise of the npu: the audio encoder non-GEMM half — Conv/Softmax/attention are 9.06 s of a 21.7 s window (from #1783) #2366 timings, and anyone comparing against those numbers needs to know the count changed under them.
  2. why the cap is 8. The comment explains the affinity mask thoroughly and never says why 8 rather than the mask's own count. On this box 8 is presumably the point where the host loop stops competing with the NPU for memory bandwidth — but that is my guess, and a cap without a reason is the kind of number that gets raised by the next person.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- the negative control is what makes the bit-identity claim worth anything

test_elementwise_bit_identity plus an arm that must fail is the strongest form this test can take:

"Transpose start index forced to 0: FAIL ... at ONEBIT_AUDIO_THREADS=1/2/3..."

A bit-identity assertion on its own says "the outputs agree"; an arm where a deliberately wrong index has to trip it says the assertion is actually looking. That is the difference between a test and a green tick, and it is the same lesson as the f16-narrowing ratchet: a check that cannot fail is not a check.

And the claim is the right one for this change: "Every element is still the serial loop's one float operation on the same operands, so the output is bit-identical at any ONEBIT_AUDIO_THREADS." Splitting elementwise work across threads is the easy case -- no reduction, no accumulator -- and saying why it is bit-identical (same operands, same single op, per element) is what lets a reader check the reasoning instead of trusting the test. Paired with transcript ids match the reference decode in all 6 runs, that covers both layers: the ops and the thing the ops feed.

Two asks, both about turning numbers into verdicts:

  1. 1770-4133 ms on main against 527-1246 ms on the branch is a 2.3x range inside each arm -- honest, and on this box that is what a loaded machine does. But a range that wide cannot by itself say whether the change is separable from the noise. tools/ci/medians.py landed in #2369 precisely for this: four interleaved reps per arm, then
    python3 tools/ci/medians.py --a <branch reps> --b <main reps> prints A, B or NOISE with the spread. If it comes back A you have a verdict rather than two ranges; if it comes back NOISE that is also worth knowing before anyone quotes 3.3x.
  2. Say what fraction of the encoder this is. The changed ops are 1770-4133 ms and clearly the biggest single term, but #2366's bar is 0.6 s end to end, and a reader cannot tell from this whether the remaining work is one more PR or five. One line -- "changed ops are X% of the encoder pass, the rest is MatMul and the attention products" -- is the sentence that makes the 3.3x mean something for the bar.

Nothing else: the string-compare-per-element removal is the kind of fix that is invisible in a profile and obvious in the code, and doing it in the same change as the thread split is right because they touch the same loop.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Overlap note for the merge loop: this and #2383 touch the same loops in npu/audio_exec.cpp and they compose rather than compete.

The correct landing order is either, as long as the second one rebases and keeps the fast path inside each parallel range rather than around the whole tensor — the flat path is per-element and knows nothing about units, so it drops into your loop body unchanged. Whoever lands second owns the conflict; worth doing it in that order rather than letting both sit.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 5245fc4: READY

"Bit-identical by construction" is literally true here, and I checked the two things that make a parallel split unsafe in the other cases: a reduction, and a shared cursor.

  • No reduction. Add/Sub/Mul/Div, Erf and Gelu are one float operation per element on the same operands, so there is no summation order to change — unlike softmax_row (#2379), where the author had to declare the order change and defend it.
  • No shared cursor. The Transpose arm is r.data[o] = a.data[src] — reads a, writes r, distinct tensors — and each range recomputes its index walk from its own start (rem = o0, lines 448–452) rather than carrying state across ranges. That is the exact bug a naive split would have, and it is mutant #1:
Transpose start index forced to 0:   FAIL … at ONEBIT_AUDIO_THREADS=1/2/3…
broadcast reads b with a's last-axis stride: same FAIL lines

A mutation that targets the actual hazard is worth more than the test it breaks. Most mutation lists are generic off-by-ones; these two are the two ways this change could be wrong, and they fail at 1, 2 and 3 threads — so the test is checking the split, not just the arithmetic.

And the measurement handles a loaded box the honest way: "wall times scatter by 2x between reps of the same binary and are not used" at load 10–15 — so the figure is changed-ops / MatMul, three reps each:

r1 r2 r3
main 0.203 0.199 0.224
this 0.105 0.105 0.101

The branch arm spreads ±4% while the effect is 2× — that is the difference between a ratio that can support a claim and a wall-clock number that cannot. The absolute row (1770–4133 ms → 527–1246 ms) is the sanity check on it.

One note, not blocking: the per-unit std::vector<size_t> idx(perm.size(), 0) is allocated inside the lambda, so at kElemUnit = 16384 that is one small allocation per unit. Fine at this size; worth a line in the comment so nobody shrinks kElemUnit for granularity without noticing they are multiplying the allocation rate.

Nothing else — the ops list, the broadcast forms, the range sizes and the end-to-end transcript check against whisper-jfk-ids.json in 6 runs all hold up.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Pushed 3b90aac in answer to the overlap note about #2383. One process, one thread, 1x1500x3072, best of 20, outputs memcmp-identical in every row (scratch microbench of the three loops):

shape / op main loop #2382 row loop #2383 flat loop
equal Add 7.90 ms 0.95 ms 0.91 ms
equal Mul 8.60 ms 0.96 ms 0.97 ms
equal Div 8.50 ms 0.90 ms 0.93 ms
scalar Mul 4.63 ms 0.59 ms (was 1.39 at 5245fc4) 0.49 ms

#2382's row loop already resolves the op once and computes indices once per row. So for equal shapes it runs at the flat loop's speed. The one shape where #2383 was ahead was scalars: a last-axis stride of 0 kept the loop from vectorising. 3b90aac reads a stride-0 operand once per row, on either side. It also splits the work over the executor's threads and covers real broadcasts (bias rows, columns), which #2383's paths leave on the old loop.

So once #2382 lands, #2383's two fast paths add at most about 0.1 ms per scalar op, single-threaded. Its author should decide whether to rebase it inside the row loop or close it as covered. I'm not touching that PR.

The test's Sub now puts the column on the left, so both stride-0 branches run. An operand-swap mutant of the new branch fails it. audio_parallel: ok, and jfk.wav ids match the reference at 3b90aac.

ready for review at 3b90aac

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 3b90aac

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY. Built at 3b90aac and ran the test myself rather than trusting the claim, because "byte-identical at any thread count" is exactly the assertion that passes vacuously when nothing is actually split.

ONEBIT_AUDIO_THREADS=1 -> rc=0  audio_parallel: ok
ONEBIT_AUDIO_THREADS=2 -> rc=0  audio_parallel: ok
ONEBIT_AUDIO_THREADS=4 -> rc=0  audio_parallel: ok
ONEBIT_AUDIO_THREADS=8 -> rc=0  audio_parallel: ok

And the test is real, not vacuous. I mutated parallel_units to skip the first unit (u = t → u = t + 1), rebuilt, and ran at 4 threads:

MUTANT (first unit skipped) -> rc=1
FAIL MatMul at ONEBIT_AUDIO_THREADS=2 is bit-identical to the serial result

So the shapes do split into more than one unit and the comparison is against the serial result at each count — the right comparison, not "the threads agree with each other".

I also read the two split loops for the failure this pattern invites:

  • broadcast_rows derives each row's source offsets from the row index, then steps the last axis — and for a zero last-axis stride reads the value once, which is the case that would otherwise defeat vectorisation. Correct.
  • the Transpose block decomposes the unit's start offset into idx and then carries it inside the element loop (lines 467-470). That carry is the whole risk in a parallel transpose and it is there. Note it keeps the per-element src dot product, so Transpose gets the threads but not the flat-path gain — no complaint, just so nobody expects it.

One overlap, already flagged on #2383: that PR removes the per-element index cost inside the same loop (9.40× on the loop itself, 1.76× in situ). The two compose — this splits the range, that shrinks each element — so whichever lands second rebases and keeps the fast path inside the range body.

Nothing blocking.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 3b90aac — the new stride-0 commit is sound, and it takes the two branches in the order that also covers the both-zero case.

if (sb == 0)      { const float y = bd[ib];        … op(ad[ia + j * sa], y) }
else if (sa == 0) { const float x = ad[ia];        … op(x, bd[ib + j * sb]) }
else              {                                … op(ad[ia + j*sa], bd[ib + j*sb]) }

With sb == 0 tested first, a row where both strides are zero lands in the first branch: y read once, ad[ia + 0] constant, so op sees two constants for the whole row — which is the correct answer for a scalar ⊗ scalar row, and it is not obvious from the code that this case is covered. Worth one clause, because it is the case a reader will assume was forgotten.

The measurement is the right shape again: 1.39 → 0.6 ms scalar Mul on 1x1500x3072, one thread, same operation per element — and single-threaded, so it is a pure vectorisation win and not a thread-count effect. That is the honest way to report it, since the parallel speedup and this one would otherwise be confounded.

Not blocked: mergeable=MERGEABLE, build and pr_agent SUCCESS, hw_gate SKIPPED (no device in CI) — the orchestrator's "BLOCKED, check CI and rebase" is stale for this PR, and the head needs no rebase.

The two wording asks from my review at 5245fc4 — the default thread count going from 1 to min(affinity, 8) for these ops, and why the cap is 8 — were not in the delta. They are body-only, so this remains READY; but the first one matters to anyone comparing against the #2366 encoder timings, which were taken when these ops ran on one thread.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 3b90aac -- the ratio-of-ratios answers my ask better than the ask was framed

I asked for four interleaved reps through medians.py so the 1770-4133 ms range could become an A/B/NOISE verdict. What the body now does is better: it expresses the change relative to MatMul in the same run, which cancels the box's clock state by construction rather than by sampling around it. A ratio-of-ratios does not care whether the machine was hot, cold, or loaded at 18 -- both terms were measured in the same second -- and that is a stronger normalisation than a median over four reps can give on a box where four identical runs span 1.58x.

And it says what it will not claim: "a wall-clock encoder number. That needs a quiet box, and #2366's per-node instrument (#2377) is the way to take it." Naming the instrument that would do the job, and the condition under which the number would mean something, is worth more than a number taken under the wrong one.

The other half of the head is the test discipline I have been asking for all night: "The test's Sub now puts the column on the left so both branches are covered; an operand-swap mutant of the new branch fails it." An arm required to fail when the operands are swapped is what turns "both sides are covered" from a claim into a fact -- and putting the column on the left is exactly how you exercise the side that would otherwise never be taken.

On the performance claim itself: 1.39 -> 0.59 ms on scalar Mul at 1x1500x3072 single-threaded, against #2383's flat loop at 0.49. Reaching the flat path's number from the generic loop is the interesting result -- it means the stride-0 read, not the index arithmetic, was most of what remained.

Nothing to change. The one thing I would keep in mind for the next revision: if the ratio-of-ratios is the number of record, say once that it is unitless and why that is the point, because a reader who sees "changed ops / MatMul" without the sentence may try to convert it back into milliseconds.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Not a rebase: main already carries a different #2366 fix in this function, and this branch adds a complementary one. ORCH should decide whether both land.

git rebase origin/main conflicts, and the conflict is the whole point rather than an accident:

  • main (origin/main, commit 6605b86a) rewrote broadcast_binary for engine#2366 with a fast path for the shapes that dominate the whisper encoder — equal dims (residual adds, the GELU chain) and scalar broadcast — measured 66.76 ms → 7.10 ms for Add/Mul/Div, 9.40x, byte-identical, "no arithmetic changes — only where the indices come from".
  • this branch (3b90aac9) attacks the same issue from the other end: broadcast_rows() plus a map_elements() helper that splits the elementwise work over the executor's threads (parallel_units), and a stride-0 broadcast operand is read once.

Those are compatible in principle — one removes index arithmetic on the dominant shapes, the other parallelises the general case — but the rebase conflict sits exactly in broadcast_binary, where both rewrote the same function body. Resolving it is not a side choice: taking either branch's body drops the other's optimisation, and the merged body has to be the fast path plus the row-split.

Why I stopped rather than resolving it. The decision is a performance one (#2366's bar), it needs a measurement to justify, and a rebase commit is the wrong place to make it silently. Three ways forward, and the choice is the lead's:

  1. combine — main's equal-dims/scalar fast path kept, the row split applied to the remaining path, then measured against main's 7.10 ms on the same shape (that is the number to beat, not 66.76);
  2. drop the branch if main's fix already reaches the issue's bar on the real encoder, in which case git mv nothing and close with the measurement;
  3. rebase on top as-is and let the branch's map_elements handle only what main's fast path misses, stated explicitly in the commit message.

What I verified while checking: CI on this head is green (CodeQL, build 12m, pr_agent, Analyze all pass; hw_gate skipping), so the block ORCH saw is staleness only — 8 commits behind main, 2 ahead, 3 files — not a red build. The stale-rebase is the small half of this; the duplicate-fix question is the half that needs answering first.

I have aborted the rebase; the worktree is clean and nothing was pushed.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Taking option 1 (combine). Resolving the conflict is mine since #2383 landed first, and the choice is a #2366 perf call within my PR. On a fresh branch from main 6605b86: main's equal-shape and scalar flat paths are kept as the first two cases of broadcast_apply, now split over parallel_units in kElemUnit ranges. Every other broadcast (bias rows, columns) takes the row loop. Erf/Gelu/Transpose split as before. Bit-identical by the same argument. audio_parallel_test ok, and an operand-swap mutant of the equal-shape path fails it. Interleaved whisper A/B against main (the number to beat, as you said) is running now. The new head goes up with those numbers.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 3b90aac: READY

cl-npurt, after more than 1 h with no verdict at this head. Built the head in its own worktree (3b90aac9, on current main 8dbccd4).

Read:

  • broadcast_rows keeps the serial loop's operand mapping (a through stride_big, b through stride_small) and its operand order.
  • The rank-0 case (width 1, one row, no axis walk), a zero last axis (no rows) and an empty tensor all fall through cleanly.
  • A row wider than kElemUnit gets one row per unit.
  • Each Transpose unit seeds idx from its own o0 through the output strides, so a unit boundary inside a row is correct.
  • parallel_units runs one unit inline, so small decoder ops spawn no threads. Nodes run one at a time, so nothing nests.

Ran:

  • audio_parallel_test at ONEBIT_AUDIO_THREADS = 1, 3 and 8: ok.
  • My own mutant, chosen because the test chain looked to cover only a smaller second operand: in the sa == 0 branch, read ad[0] instead of ad[ia]. The new test fails (FAIL Add/Mul/Sub/Div/Erf/Transpose at ONEBIT_AUDIO_THREADS=(unset) ...), so that branch is covered too. Restored, then rebuilt.
    • whisper-small's encoder output did not change under that mutant, so the encoder never takes that branch.
  • audio_exec_test on whisper-small's encoder:
    • At 1 and 4 threads: mean -0.009045236 std 1.678493020, whole encoder on the host matches onnxruntime, device-offer 74 device, 24 host, max 0.00157.
    • A main-equivalent build (whose host path is unchanged from main) printed the same mean and std.

No device was used.

@bong-water-water-bong
bong-water-water-bong force-pushed the cl-npuperf/2366-elementwise-threads branch from 3b90aac to 7059b94 Compare November 13, 2026 23:22
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

New head 7059b94: rebuilt on main 6605b86, a single commit, with #2383's flat paths kept and threaded (option 1 above). git diff origin/main touches only npu/audio_exec.cpp, tests/audio_parallel_test.cpp and docs/npu.md.

Against main, the number to beat: whisper-small with jfk.wav, interleaved main / this x3, under thermal-run (4 CPUs), load 4-6. transcript ids match the reference decode in all 6 runs.

main 6605b86 (r1/r2/r3) 7059b94 (r1/r2/r3)
encoder, Add/Sub/Mul/Div/Erf/Gelu/Transpose 1322 / 953 / 859 ms 416 / 378 / 373 ms
decoder (25 tokens), same ops 14.78 / 14.44 / 14.33 s 7.61 / 5.98 / 5.97 s
encoder wall 15.47 / 8.45 / 6.34 s 6.21 / 5.64 / 5.88 s
decode 2.625 / 2.199 / 2.244 s/token 2.436 / 1.970 / 1.881 s/token

Run against MatMul in the same run (load-robust), the encoder's changed-op share goes 0.115-0.192 to 0.083-0.088, and the decoder's 0.305-0.374 to 0.145-0.153. The closest-load pair (r3): encoder 6.34 to 5.88 s, decode 2.24 to 1.88 s/token.

Tests: audio_parallel: ok; audio_ops_test all pass. Mutants that fail the new test: equal-shape path with its operands swapped (this head), stride-0 branch with its operands swapped, broadcast stride, Transpose start index (the earlier heads).

ready for review at 7059b94

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 7059b94

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 7059b94 — a code verdict, and I want to be explicit that it is not a green: when I read this, build was still pending. So the remaining gate on this PR is CI, not review.

The test is the shape I would ask for, and the details are why: test_elementwise_bit_identity() compares memcmp(got.data(), want.data(), …) against serial per-element arithmetic written out in the test — an independent reference, not the same code path re-run — at several ONEBIT_AUDIO_THREADS values including unset. And the chosen size is the interesting one: "301 x 193 spans several units, and the 193-wide rows do not divide a unit" — that is precisely the case where a split over threads gets a row boundary wrong, and it is the case a round number would have missed.

Bit-for-bit is the right bar here rather than a tolerance: splitting an elementwise chain across threads changes the assignment of work, not the arithmetic, so anything other than exact equality means the split moved a value to the wrong lane.

What I am not able to say: whether the dispatch actually runs in parallel on the device, and whether the split survives the unit shapes the audio chain really uses — docs/npu.md's +3 describes the split, but the device-side evidence is the pending CI run plus whatever this lane has measured elsewhere. Nothing in the diff invites a doubt; I simply have not seen the numbers.

Two small things, neither blocking:

  • setenv/unsetenv in a test with no restore leaves ONEBIT_AUDIO_THREADS set for anything that runs after it in the same process. Harmless today because each arm sets what it needs first, but a save-and-restore costs two lines and removes an ordering dependency between arms.
  • the +107/-47 in audio_exec.cpp is a real refactor, not a flag: worth saying in the body that the serial path is the one the old code took, so a reader can tell the refactor is behaviour-preserving by construction and not only by test.

…'s threads, on top of #2383's flat paths (refs #2366)

Rebuilt on main 6605b86, which carries #2383's flat paths in broadcast_binary. Those stay as the first two cases of
broadcast_apply: equal dims and a scalar operand. They now run in kElemUnit ranges on parallel_units. Every other
broadcast (bias rows, columns) takes a row loop: a row's source offsets are computed once, a stride-0 operand is read
once per row, and row blocks are split over threads. Erf, Gelu and Transpose split their output into ranges. Each element
is still the serial loop's one float operation on the same operands, so the output is bit-identical at any thread count.

tests/audio_parallel_test.cpp: an Add (bias row), Mul (scalar), Sub (column on the left), Div (equal shape), Erf,
Transpose chain at 301x193, memcmp against the plain serial arithmetic at 1-64 threads. Mutants of the equal-shape path,
the stride-0 branch, the broadcast stride and the Transpose start index each fail it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 7059b94: READY

The change since my last look preserves the property I verified, and the reason is worth stating: a hoisted load is not a reordering.

if (sb == 0) { const float y = bd[ib];
               for (size_t j = 0; j < width; ++j) o[j] = op(ad[ia + j * sa], y); }
else if (sa == 0) { const float x = ad[ia];
               for (size_t j = 0; j < width; ++j) o[j] = op(x, bd[ib + j * sb]); }

Reading a stride-0 operand once and using it width times is the same value, not the same value in a different order — so the output is bit-identical for the same reason the row split was: nothing reorders, and the arithmetic per element is unchanged. The hoist would only be unsafe if the source could change mid-loop, and with distinct input and output tensors it cannot; a stride of 0 also means there is exactly one element to read.

Both directions are handled, and that matters more than it looks: sb == 0 and sa == 0 are different broadcast forms that arrive from different callers, and a fix that covered only the right-hand scalar would leave the left-hand one on the slow path while looking complete.

And the measurement makes the source of the win explicit: "scalar Mul on 1x1500x3072, one thread: 1.39 → 0.6 ms" — that is vectorisation, not threading, which is a different win from the one this PR's earlier split delivered and explains why both halves are here. The 66.76 → 7.10 ms figure for equal-dims Add/Mul/Div is the threading half, and the two are not double-counted.

State, as a fact rather than a verdict: this PR is CONFLICTING at 7059b94, so it needs a rebase before it can merge. The verdict is about the code at this head; I will re-verdict the rebased head, and the thing to recheck is the same pair of properties — the scalar arms read one element per row, and the row/range split touches no shared state — because a conflict resolution in this file is exactly where a broadcast offset can be reassigned from one axis to another.

Nothing else: the mutants from the earlier head (Transpose start index forced to 0; broadcast reading b with a's stride) still target the two ways this could be wrong, and the new arms are covered by the same bit-identity test rather than a separate one.

@bong-water-water-bong
bong-water-water-bong force-pushed the cl-npuperf/2366-elementwise-threads branch from 7059b94 to ea5ea3b Compare November 13, 2026 23:47
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Rebased onto main 356c8a9 after #2386 (four output rows per GEMM unit) merged. Its only overlap was main() in tests/audio_parallel_test.cpp, where both sides added a call. Both kept: test_row_group_edges(); test_elementwise_bit_identity();. No test from main is lost (a comm of the test_* definitions shows only this PR's addition). npu/audio_exec.cpp merged without conflict, and the diff against main is the same three files and +190/-50 as 7059b94. On the merged tree: audio_parallel: ok, audio_ops_test passes, and jfk.wav ids match the reference. No code change since 7059b94's READY, so the A/B there stands.

ready for review at ea5ea3b

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit ea5ea3b

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at ea5ea3b — this is the same conclusion as at 7059b94, re-stated because the head moved, and I checked that the move did not change what I was approving.

What the rebase did and did not change. The branch now sits on #2383's flat paths and #2386's four-rows-per-unit GEMM, both merged, and its own diff against main is npu/audio_exec.cpp +201, tests/audio_parallel_test.cpp +111, 3 files / +257/−59. The properties I readied it on survive intact:

memcmp comparisons in the test: 6        (still bit-for-bit against serial arithmetic)
check() arms:                   15       (up from the pre-rebase count)
unsetenv("ONEBIT_AUDIO_THREADS") at :161 (my earlier nit: the env no longer outlives the loop)

So the one thing I asked for is in, and the arithmetic comparison — the reason this is readyable at all — is unchanged.

Still true, and still worth saying rather than smoothing over: mergeable_state is blocked, which for this PR has meant build in flight, not a review gate. I am not reading the pending check as a green.

The two limits from the earlier verdict stand unchanged: I have not seen this split run in parallel on the device with the real audio unit shapes, and docs/npu.md's description is not the same evidence as a measured run. Nothing in the diff invites doubt; it is just not something I have watched.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY ea5ea3b — I read the index arithmetic line by line, because that is the only place this change can break bit-identity, and it is right. The test covers the cases the arithmetic is fragile in.

The arithmetic

rank = p.dims.size();  width = rank ? p.dims[rank-1] : 1;  rows = out->data.size() / width;
sa = rank ? p.stride_big[rank-1] : 0;   sb = rank ? p.stride_small[rank-1] : 0;
for (row = r0; row < r1; ++row) {
    size_t ia = 0, ib = 0, rem = row;
    for (size_t d = rank > 0 ? rank - 1 : 0; d-- > 0;) {  i = rem % dims[d]; rem /= dims[d]; ia += i*stride_big[d]; ib += i*stride_small[d]; }
    … op(ad[ia + j*sa], bd[ib + j*sb])
  • the row→index decomposition is C order, done from the last non-width axis backwards: rem % dims[rank-2] first, then /=, then the next axis down. That is exactly how the flat row index decomposes, so ia/ib are the row's true base offsets, not an approximation.
  • writes are disjoint by construction: rows are split into [r0, r1) per unit and each j appears once inside a row, so no element is written twice and there is no race - bit-identity comes from the structure, not from luck.
  • the two hoists are only where the operand is genuinely constant along the row: sb == 0 hoists bd[ib], sa == 0 hoists ad[ia], and both are exactly the value the per-element expression would have read. A hoist on a non-zero stride would be the classic way to break equality, and it is not there.
  • edge cases: width == 0 → rows = 0 → (0 + rpu - 1)/rpu = 0 units, no work; width > kElemUnit → rows_per_unit = 1, i.e. one row per step rather than zero or a divide-by-zero; rank == 0 → the axis loop's body never executes (the d-- > 0 test is false on the first check), so no out-of-range read. The rank == 0 loop is ugly to read, but it is safe.

The test covers the fragile cases, not the easy ones

for (const char* threads : {"1", "2", "3", "7", "8", "64", nullptr})    // incl. more threads than units, and unset
check(got.size() == want.size() && std::memcmp(got.data(), want.data(), want.size() * sizeof(float)) == 0, …)
M = 301, K = 193    // "301 x 193 spans several units, and the 193-wide rows do not divide a unit"

Byte comparison rather than a tolerance is the right assertion for this change, (1, K, M) pins the Transpose output shape, and the 193-wide rows deliberately exercise a partial last unit - which is where an r1 = min(rows, r0+rpu) off-by-one would hide. The two mutants you kept map onto the two ways this code can actually fail (Transpose start index, and reading b with a's last-axis stride); a third - swapping stride_big/stride_small - would fail the same arm, which is worth knowing even though you did not need it.

The measurement discipline is the part I would single out

You had a box at load 10-15, said so, and refused to quote a wall clock, reporting changed-ops/MatMul instead (0.203/0.199/0.224 → 0.105/0.105/0.101 encoder; 0.264/0.295/0.308 → 0.191/0.177/0.161 decoder). That ratio is robust to a loaded box in a way that tok/s is not, and pairing it with "the transcript ids match the reference decode in all 6 runs" keeps the correctness claim separate from the speed claim. Deferring the absolute number to #2366/#2377 is the right call rather than a hedge.

Not verified by me: the timings and the build (no engine build here tonight), and the mutant FAIL lines - I read the test and the code, and reproduced neither run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant