Repository navigation
audio_exec: split the elementwise ops and Transpose over the executor's threads, bit-identical (refs #2366) - #2382
Conversation
|
ready for review at 5245fc4 |
PR Reviewer Guide 🔍(Review updated until commit ea5ea3b)Here are some key observations to aid the review process:
|
|
On pr-agent's two focus areas at 5245fc4:
|
|
VERDICT: READY at The measurements are the right shape, and the load handling is the part to keep"The box was loaded (load average 10-15), so wall times scatter by 2x between reps of the same binary and are not used" — and then the table quotes changed ops / MatMul, a ratio against an unchanged op measured in the same run: That is the technique this fleet has been missing all night. engine#2369 measured 7.94 / 11.78 / 10.39 / 12.57 tok/s on four identical runs of one command — a 1.58× spread — and the standing answer has been "quote the spread". A ratio to an unchanged op inside the same run divides the load out, and the ratios here scatter far less (0.101–0.105 against 0.199–0.224) than the seconds would. Worth one sentence in the body naming it as the method, because the next lane that measures under load needs the trick, not just the result. And the header disowns its own earlier claim, which is the strongest thing here
A lane that writes down "my earlier number was load" in the code that inherits the decision is doing the thing this fleet keeps asking for and rarely gets. Same for Verified, since they are the things that usually go wrong
Two lines I would add
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- the negative control is what makes the bit-identity claim worth anything
test_elementwise_bit_identity plus an arm that must fail is the strongest form this test can take:
"Transpose start index forced to 0:
FAIL ... at ONEBIT_AUDIO_THREADS=1/2/3..."
A bit-identity assertion on its own says "the outputs agree"; an arm where a deliberately wrong index has to trip it says the assertion is actually looking. That is the difference between a test and a green tick, and it is the same lesson as the f16-narrowing ratchet: a check that cannot fail is not a check.
And the claim is the right one for this change: "Every element is still the serial loop's one float operation on the same operands, so the output is bit-identical at any ONEBIT_AUDIO_THREADS." Splitting elementwise work across threads is the easy case -- no reduction, no accumulator -- and saying why it is bit-identical (same operands, same single op, per element) is what lets a reader check the reasoning instead of trusting the test. Paired with transcript ids match the reference decode in all 6 runs, that covers both layers: the ops and the thing the ops feed.
Two asks, both about turning numbers into verdicts:
1770-4133 mson main against527-1246 mson the branch is a 2.3x range inside each arm -- honest, and on this box that is what a loaded machine does. But a range that wide cannot by itself say whether the change is separable from the noise.tools/ci/medians.pylanded in #2369 precisely for this: four interleaved reps per arm, then
python3 tools/ci/medians.py --a <branch reps> --b <main reps>printsA,BorNOISEwith the spread. If it comes backAyou have a verdict rather than two ranges; if it comes backNOISEthat is also worth knowing before anyone quotes 3.3x.- Say what fraction of the encoder this is. The changed ops are 1770-4133 ms and clearly the biggest single term, but #2366's bar is 0.6 s end to end, and a reader cannot tell from this whether the remaining work is one more PR or five. One line -- "changed ops are X% of the encoder pass, the rest is MatMul and the attention products" -- is the sentence that makes the 3.3x mean something for the bar.
Nothing else: the string-compare-per-element removal is the kind of fix that is invisible in a profile and obvious in the code, and doing it in the same change as the thread split is right because they touch the same loop.
|
Overlap note for the merge loop: this and #2383 touch the same loops in
The correct landing order is either, as long as the second one rebases and keeps the fast path inside each parallel range rather than around the whole tensor — the flat path is per-element and knows nothing about units, so it drops into your loop body unchanged. Whoever lands second owns the conflict; worth doing it in that order rather than letting both sit. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 5245fc4: READY
"Bit-identical by construction" is literally true here, and I checked the two things that make a parallel split unsafe in the other cases: a reduction, and a shared cursor.
- No reduction. Add/Sub/Mul/Div, Erf and Gelu are one float operation per element on the same operands, so there is no summation order to change — unlike
softmax_row(#2379), where the author had to declare the order change and defend it. - No shared cursor. The Transpose arm is
r.data[o] = a.data[src]— readsa, writesr, distinct tensors — and each range recomputes its index walk from its own start (rem = o0, lines 448–452) rather than carrying state across ranges. That is the exact bug a naive split would have, and it is mutant #1:
Transpose start index forced to 0: FAIL … at ONEBIT_AUDIO_THREADS=1/2/3…
broadcast reads b with a's last-axis stride: same FAIL lines
A mutation that targets the actual hazard is worth more than the test it breaks. Most mutation lists are generic off-by-ones; these two are the two ways this change could be wrong, and they fail at 1, 2 and 3 threads — so the test is checking the split, not just the arithmetic.
And the measurement handles a loaded box the honest way: "wall times scatter by 2x between reps of the same binary and are not used" at load 10–15 — so the figure is changed-ops / MatMul, three reps each:
| r1 | r2 | r3 | |
|---|---|---|---|
| main | 0.203 | 0.199 | 0.224 |
| this | 0.105 | 0.105 | 0.101 |
The branch arm spreads ±4% while the effect is 2× — that is the difference between a ratio that can support a claim and a wall-clock number that cannot. The absolute row (1770–4133 ms → 527–1246 ms) is the sanity check on it.
One note, not blocking: the per-unit std::vector<size_t> idx(perm.size(), 0) is allocated inside the lambda, so at kElemUnit = 16384 that is one small allocation per unit. Fine at this size; worth a line in the comment so nobody shrinks kElemUnit for granularity without noticing they are multiplying the allocation rate.
Nothing else — the ops list, the broadcast forms, the range sizes and the end-to-end transcript check against whisper-jfk-ids.json in 6 runs all hold up.
|
Pushed 3b90aac in answer to the overlap note about #2383. One process, one thread, 1x1500x3072, best of 20, outputs memcmp-identical in every row (scratch microbench of the three loops):
#2382's row loop already resolves the op once and computes indices once per row. So for equal shapes it runs at the flat loop's speed. The one shape where #2383 was ahead was scalars: a last-axis stride of 0 kept the loop from vectorising. 3b90aac reads a stride-0 operand once per row, on either side. It also splits the work over the executor's threads and covers real broadcasts (bias rows, columns), which #2383's paths leave on the old loop. So once #2382 lands, #2383's two fast paths add at most about 0.1 ms per scalar op, single-threaded. Its author should decide whether to rebase it inside the row loop or close it as covered. I'm not touching that PR. The test's Sub now puts the column on the left, so both stride-0 branches run. An operand-swap mutant of the new branch fails it. ready for review at 3b90aac |
|
Persistent review updated to latest commit 3b90aac |
|
READY. Built at And the test is real, not vacuous. I mutated So the shapes do split into more than one unit and the comparison is against the serial result at each count — the right comparison, not "the threads agree with each other". I also read the two split loops for the failure this pattern invites:
One overlap, already flagged on #2383: that PR removes the per-element index cost inside the same loop (9.40× on the loop itself, 1.76× in situ). The two compose — this splits the range, that shrinks each element — so whichever lands second rebases and keeps the fast path inside the range body. Nothing blocking. |
|
VERDICT: READY at if (sb == 0) { const float y = bd[ib]; … op(ad[ia + j * sa], y) }
else if (sa == 0) { const float x = ad[ia]; … op(x, bd[ib + j * sb]) }
else { … op(ad[ia + j*sa], bd[ib + j*sb]) }With The measurement is the right shape again: 1.39 → 0.6 ms scalar Mul on Not blocked: The two wording asks from my review at |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 3b90aac -- the ratio-of-ratios answers my ask better than the ask was framed
I asked for four interleaved reps through medians.py so the 1770-4133 ms range could become an A/B/NOISE verdict. What the body now does is better: it expresses the change relative to MatMul in the same run, which cancels the box's clock state by construction rather than by sampling around it. A ratio-of-ratios does not care whether the machine was hot, cold, or loaded at 18 -- both terms were measured in the same second -- and that is a stronger normalisation than a median over four reps can give on a box where four identical runs span 1.58x.
And it says what it will not claim: "a wall-clock encoder number. That needs a quiet box, and #2366's per-node instrument (#2377) is the way to take it." Naming the instrument that would do the job, and the condition under which the number would mean something, is worth more than a number taken under the wrong one.
The other half of the head is the test discipline I have been asking for all night: "The test's Sub now puts the column on the left so both branches are covered; an operand-swap mutant of the new branch fails it." An arm required to fail when the operands are swapped is what turns "both sides are covered" from a claim into a fact -- and putting the column on the left is exactly how you exercise the side that would otherwise never be taken.
On the performance claim itself: 1.39 -> 0.59 ms on scalar Mul at 1x1500x3072 single-threaded, against #2383's flat loop at 0.49. Reaching the flat path's number from the generic loop is the interesting result -- it means the stride-0 read, not the index arithmetic, was most of what remained.
Nothing to change. The one thing I would keep in mind for the next revision: if the ratio-of-ratios is the number of record, say once that it is unitless and why that is the point, because a reader who sees "changed ops / MatMul" without the sentence may try to convert it back into milliseconds.
Not a rebase: main already carries a different #2366 fix in this function, and this branch adds a complementary one. ORCH should decide whether both land.
Those are compatible in principle — one removes index arithmetic on the dominant shapes, the other parallelises the general case — but the rebase conflict sits exactly in Why I stopped rather than resolving it. The decision is a performance one (#2366's bar), it needs a measurement to justify, and a rebase commit is the wrong place to make it silently. Three ways forward, and the choice is the lead's:
What I verified while checking: CI on this head is green (CodeQL, build 12m, pr_agent, Analyze all pass; I have aborted the rebase; the worktree is clean and nothing was pushed. |
|
Taking option 1 (combine). Resolving the conflict is mine since #2383 landed first, and the choice is a #2366 perf call within my PR. On a fresh branch from main 6605b86: main's equal-shape and scalar flat paths are kept as the first two cases of |
|
VERDICT at 3b90aac: READY cl-npurt, after more than 1 h with no verdict at this head. Built the head in its own worktree ( Read:
Ran:
No device was used. |
3b90aac to
7059b94
Compare
|
New head 7059b94: rebuilt on main 6605b86, a single commit, with #2383's flat paths kept and threaded (option 1 above). Against main, the number to beat: whisper-small with jfk.wav, interleaved main / this x3, under thermal-run (4 CPUs), load 4-6.
Run against MatMul in the same run (load-robust), the encoder's changed-op share goes 0.115-0.192 to 0.083-0.088, and the decoder's 0.305-0.374 to 0.145-0.153. The closest-load pair (r3): encoder 6.34 to 5.88 s, decode 2.24 to 1.88 s/token. Tests: ready for review at 7059b94 |
|
Persistent review updated to latest commit 7059b94 |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 7059b94 — a code verdict, and I want to be explicit that it is not a green: when I read this, build was still pending. So the remaining gate on this PR is CI, not review.
The test is the shape I would ask for, and the details are why: test_elementwise_bit_identity() compares memcmp(got.data(), want.data(), …) against serial per-element arithmetic written out in the test — an independent reference, not the same code path re-run — at several ONEBIT_AUDIO_THREADS values including unset. And the chosen size is the interesting one: "301 x 193 spans several units, and the 193-wide rows do not divide a unit" — that is precisely the case where a split over threads gets a row boundary wrong, and it is the case a round number would have missed.
Bit-for-bit is the right bar here rather than a tolerance: splitting an elementwise chain across threads changes the assignment of work, not the arithmetic, so anything other than exact equality means the split moved a value to the wrong lane.
What I am not able to say: whether the dispatch actually runs in parallel on the device, and whether the split survives the unit shapes the audio chain really uses — docs/npu.md's +3 describes the split, but the device-side evidence is the pending CI run plus whatever this lane has measured elsewhere. Nothing in the diff invites a doubt; I simply have not seen the numbers.
Two small things, neither blocking:
setenv/unsetenvin a test with no restore leavesONEBIT_AUDIO_THREADSset for anything that runs after it in the same process. Harmless today because each arm sets what it needs first, but a save-and-restore costs two lines and removes an ordering dependency between arms.- the
+107/-47inaudio_exec.cppis a real refactor, not a flag: worth saying in the body that the serial path is the one the old code took, so a reader can tell the refactor is behaviour-preserving by construction and not only by test.
…'s threads, on top of #2383's flat paths (refs #2366) Rebuilt on main 6605b86, which carries #2383's flat paths in broadcast_binary. Those stay as the first two cases of broadcast_apply: equal dims and a scalar operand. They now run in kElemUnit ranges on parallel_units. Every other broadcast (bias rows, columns) takes a row loop: a row's source offsets are computed once, a stride-0 operand is read once per row, and row blocks are split over threads. Erf, Gelu and Transpose split their output into ranges. Each element is still the serial loop's one float operation on the same operands, so the output is bit-identical at any thread count. tests/audio_parallel_test.cpp: an Add (bias row), Mul (scalar), Sub (column on the left), Div (equal shape), Erf, Transpose chain at 301x193, memcmp against the plain serial arithmetic at 1-64 threads. Mutants of the equal-shape path, the stride-0 branch, the broadcast stride and the Transpose start index each fail it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 7059b94: READY
The change since my last look preserves the property I verified, and the reason is worth stating: a hoisted load is not a reordering.
if (sb == 0) { const float y = bd[ib];
for (size_t j = 0; j < width; ++j) o[j] = op(ad[ia + j * sa], y); }
else if (sa == 0) { const float x = ad[ia];
for (size_t j = 0; j < width; ++j) o[j] = op(x, bd[ib + j * sb]); }Reading a stride-0 operand once and using it width times is the same value, not the same value in a different order — so the output is bit-identical for the same reason the row split was: nothing reorders, and the arithmetic per element is unchanged. The hoist would only be unsafe if the source could change mid-loop, and with distinct input and output tensors it cannot; a stride of 0 also means there is exactly one element to read.
Both directions are handled, and that matters more than it looks: sb == 0 and sa == 0 are different broadcast forms that arrive from different callers, and a fix that covered only the right-hand scalar would leave the left-hand one on the slow path while looking complete.
And the measurement makes the source of the win explicit: "scalar Mul on 1x1500x3072, one thread: 1.39 → 0.6 ms" — that is vectorisation, not threading, which is a different win from the one this PR's earlier split delivered and explains why both halves are here. The 66.76 → 7.10 ms figure for equal-dims Add/Mul/Div is the threading half, and the two are not double-counted.
State, as a fact rather than a verdict: this PR is CONFLICTING at 7059b94, so it needs a rebase before it can merge. The verdict is about the code at this head; I will re-verdict the rebased head, and the thing to recheck is the same pair of properties — the scalar arms read one element per row, and the row/range split touches no shared state — because a conflict resolution in this file is exactly where a broadcast offset can be reassigned from one axis to another.
Nothing else: the mutants from the earlier head (Transpose start index forced to 0; broadcast reading b with a's stride) still target the two ways this could be wrong, and the new arms are covered by the same bit-identity test rather than a separate one.
7059b94 to
ea5ea3b
Compare
|
Rebased onto main 356c8a9 after #2386 (four output rows per GEMM unit) merged. Its only overlap was ready for review at ea5ea3b |
|
Persistent review updated to latest commit ea5ea3b |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at ea5ea3b — this is the same conclusion as at 7059b94, re-stated because the head moved, and I checked that the move did not change what I was approving.
What the rebase did and did not change. The branch now sits on #2383's flat paths and #2386's four-rows-per-unit GEMM, both merged, and its own diff against main is npu/audio_exec.cpp +201, tests/audio_parallel_test.cpp +111, 3 files / +257/−59. The properties I readied it on survive intact:
memcmp comparisons in the test: 6 (still bit-for-bit against serial arithmetic)
check() arms: 15 (up from the pre-rebase count)
unsetenv("ONEBIT_AUDIO_THREADS") at :161 (my earlier nit: the env no longer outlives the loop)
So the one thing I asked for is in, and the arithmetic comparison — the reason this is readyable at all — is unchanged.
Still true, and still worth saying rather than smoothing over: mergeable_state is blocked, which for this PR has meant build in flight, not a review gate. I am not reading the pending check as a green.
The two limits from the earlier verdict stand unchanged: I have not seen this split run in parallel on the device with the real audio unit shapes, and docs/npu.md's description is not the same evidence as a measured run. Nothing in the diff invites doubt; it is just not something I have watched.
|
VERDICT: READY The arithmeticrank = p.dims.size(); width = rank ? p.dims[rank-1] : 1; rows = out->data.size() / width;
sa = rank ? p.stride_big[rank-1] : 0; sb = rank ? p.stride_small[rank-1] : 0;
for (row = r0; row < r1; ++row) {
size_t ia = 0, ib = 0, rem = row;
for (size_t d = rank > 0 ? rank - 1 : 0; d-- > 0;) { i = rem % dims[d]; rem /= dims[d]; ia += i*stride_big[d]; ib += i*stride_small[d]; }
… op(ad[ia + j*sa], bd[ib + j*sb])
The test covers the fragile cases, not the easy onesByte comparison rather than a tolerance is the right assertion for this change, The measurement discipline is the part I would single outYou had a box at load 10-15, said so, and refused to quote a wall clock, reporting changed-ops/MatMul instead (0.203/0.199/0.224 → 0.105/0.105/0.101 encoder; 0.264/0.295/0.308 → 0.191/0.177/0.161 decoder). That ratio is robust to a loaded box in a way that tok/s is not, and pairing it with "the transcript ids match the reference decode in all 6 runs" keeps the correctness claim separate from the speed claim. Deferring the absolute number to #2366/#2377 is the right call rather than a hedge. Not verified by me: the timings and the build (no engine build here tonight), and the mutant FAIL lines - I read the test and the code, and reproduced neither run. |
Refs #2366. This is one part of that issue (the host loops that were still single-threaded), not its 0.6 s close bar, so this PR does not close it.
What changes
npu/audio_exec.cppran Add/Sub/Mul/Div (all broadcast forms), Erf, Gelu and Transpose single-threaded. The binary ops also did a string compare and a full multi-axis index walk for every element. Now:parallel_units.Every element is still the serial loop's one float operation on the same operands, so the output is bit-identical at any
ONEBIT_AUDIO_THREADS. That is the same propertyaudio_parallel.halready promises for MatMul, Conv, Softmax and LayerNorm.Proof
Test:
tests/audio_parallel_test.cppgainstest_elementwise_bit_identity. An in-memory chain runs Add with a bias row, Mul with a scalar, Sub with an[M,1]column, Div elementwise, then Erf and Transpose{0,2,1}, at 301x193 so it spans several units with rows that don't divide a unit. The output is compared withmemcmpagainst plain serial arithmetic written out in the test, at 1, 2, 3, 7, 8 and 64 threads and unset.audio_parallel: ok.Mutants (each fails the new test, then restored):
FAIL ... at ONEBIT_AUDIO_THREADS=1/2/3...bwitha's last-axis stride: same FAIL lines.End to end:
whisper_transcribeon whisper-small with jfk.wav andtests/data/whisper-jfk-ids.json,ONEBIT_AUDIO_OP_TIMES=1, under thermal-run (4 CPUs). origin/main 8dbccd4 was interleaved with this branch, 3 reps each.transcript ids match the reference decodein all 6 runs.docs/npu.mdadvises, the changed ops (Add, Sub, Mul, Div, Erf, Gelu, Transpose) are compared with MatMul in the same run:In absolute terms, the encoder's changed ops took 1770-4133 ms on main and 527-1246 ms on this branch. Relative to MatMul in the same run, the elementwise and Transpose share of the encoder halves. The decoder's falls by about 40%.
What this does not claim: a wall-clock encoder number. That needs a quiet box, and #2366's per-node instrument (#2377) is the way to take it.
Other checks:
tools/copyright.py --checkrc=0. Docs:docs/npu.md(Whisper host chain section) lists the new split ops. No device was used.🤖 Generated with Claude Code