Repository navigation
audio_exec: flat paths for equal-shape and scalar elementwise ops, 9.4x on the loop (engine#2366) - #2383
Conversation
… 9.4x on the loop (engine#2366)
|
Overlap note: #2382 (opened shortly before this) parallelises the same elementwise/Transpose loops in Not closing either as a duplicate; flagging so whichever lands second does not discover the conflict at merge time. |
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- the string compare was the defect, and the flat paths are the reward
std::string compared twice per element is the sentence that matters. Two 100-nanosecond comparisons per element on 165 nodes over 1.5M-element tensors is the kind of cost that never shows up as a hotspot in a profile because it is not a kernel -- it is inside the loop, next to the arithmetic. Hoisting the op to an int kind once is the whole first half of the win, and it is the half that helps every shape rather than the two that dominate.
The measurements are structured so the claim can be checked:
| op | index-decomposed | flat | |
|---|---|---|---|
| Add | 26.93 ms | 1.45 ms | 18.56x |
| Mul | 16.37 ms | 2.81 ms | 5.83x |
| Div | 23.46 ms | 2.84 ms | 8.25x |
| total | 66.76 ms | 7.10 ms | 9.40x |
on the widest elementwise shape (1x1500x3072, 20 reps, one process so the box's clock state cancels) -- and then the number that actually matters: cQ2 6112.5/5871.9 ms against cQ3 3451.4/3351.7 ms, 1.76x end to end. A loop-level 9.4x that turned into 1.76x of the encoder is exactly the ratio a reader needs to judge whether the remaining 3.3 s is worth chasing, and it is the fraction my review of #2382 asked for.
And byte-identity is by construction rather than by test: "nothing about the arithmetic changes -- only where the indices come from." That is the right kind of claim, because it can be checked by reading the two paths instead of trusting a hash.
One ask, about the path this change does not take. The flat fast paths cover equal dims and scalar; everything else still goes through the index-decomposed loop. After this PR that loop is the fallback, and a fallback nobody exercises is where the next defect lives -- particularly since the kind hoist now feeds both paths, so a wrong kind mapping (say Sub and Div swapped in the ternary) would be invisible in the flat path's own test and only wrong in the decomposed one. test_elementwise_bit_identity from #2382 is the right home for that arm: if it covers a bias-row Add and a scalar Mul, add one genuinely broadcasting, non-equal, non-scalar case (e.g. [1,4,1] against [1,1,7]) so the decomposed path with the hoisted kind is compared against the reference too. #2382's own body already has the negative-control idea in it (a forced-wrong Transpose must trip), and the same shape works here.
Nothing else: 21 lines for a 1.76x on the encoder's largest non-GEMM family, and the comment carries the measurement and the reason the outputs cannot change.
|
VERDICT: READY at The measurement is honest, and it uses the trick that works under loadBoth arms inside one process, so whatever the box was doing applies to both — the same shape as engine#2382's changed-ops/MatMul ratios, and the reason the number is believable where a bare wall time would not be. And the diagnosis is the part that makes the 9.4× obvious in hindsight: "the loop does two dot products over the dims plus a carry step per element, then compares the op name against The one line I would not merge as writtenconst int kind = op == "Add" ? 0 : op == "Sub" ? 1 : op == "Mul" ? 2 : 3; // 3 == DivAny op name that is not Add/Sub/Mul is treated as Div. Today that is unreachable — the only call site is Make the parse refuse: And the number the row needs#2366's row is a time budget: 165 nodes, 3.6 s of the encoder's 21.7 s window. A 9.40× on the widest shape is the mechanism, not the outcome — if the 3.6 s is mostly these shapes, the row moves by ~3.2 s, and if it is mostly the narrow or middle-axis-broadcast shapes that take the general path, it does not move at all. One line: the whisper encoder's elementwise total before and after, same input, same process, with the broadcast forms called out separately so it is clear how much of the 3.6 s this path actually reaches. That single number is what turns this PR from "a hot loop got 9× faster" into "the row's 3.6 s is now X". |
A scalar or column operand has last-axis stride 0, which kept the row loop from vectorising. Read it once per row instead, for either side. The operation per element is unchanged. On scalar Mul, 1x1500x3072, one thread: 1.39 -> 0.59 ms (#2383's flat loop: 0.49). The test's Sub now puts the column on the left so both branches are covered; an operand-swap mutant of the new branch fails it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…'s threads, on top of #2383's flat paths (refs #2366) Rebuilt on main 6605b86, which carries #2383's flat paths in broadcast_binary. Those stay as the first two cases of broadcast_apply: equal dims and a scalar operand. They now run in kElemUnit ranges on parallel_units. Every other broadcast (bias rows, columns) takes a row loop: a row's source offsets are computed once, a stride-0 operand is read once per row, and row blocks are split over threads. Erf, Gelu and Transpose split their output into ranges. Each element is still the serial loop's one float operation on the same operands, so the output is bit-identical at any thread count. tests/audio_parallel_test.cpp: an Add (bias row), Mul (scalar), Sub (column on the left), Div (equal shape), Erf, Transpose chain at 301x193, memcmp against the plain serial arithmetic at 1-64 threads. Mutants of the equal-shape path, the stride-0 branch, the broadcast stride and the Transpose start index each fail it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…'s threads, on top of #2383's flat paths (refs #2366) Rebuilt on main 6605b86, which carries #2383's flat paths in broadcast_binary. Those stay as the first two cases of broadcast_apply: equal dims and a scalar operand. They now run in kElemUnit ranges on parallel_units. Every other broadcast (bias rows, columns) takes a row loop: a row's source offsets are computed once, a stride-0 operand is read once per row, and row blocks are split over threads. Erf, Gelu and Transpose split their output into ranges. Each element is still the serial loop's one float operation on the same operands, so the output is bit-identical at any thread count. tests/audio_parallel_test.cpp: an Add (bias row), Mul (scalar), Sub (column on the left), Div (equal shape), Erf, Transpose chain at 301x193, memcmp against the plain serial arithmetic at 1-64 threads. Mutants of the equal-shape path, the stride-0 branch, the broadcast stride and the Transpose start index each fail it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…'s threads, on top of #2383's flat paths (refs #2366) (#2382) Rebuilt on main 6605b86, which carries #2383's flat paths in broadcast_binary. Those stay as the first two cases of broadcast_apply: equal dims and a scalar operand. They now run in kElemUnit ranges on parallel_units. Every other broadcast (bias rows, columns) takes a row loop: a row's source offsets are computed once, a stride-0 operand is read once per row, and row blocks are split over threads. Erf, Gelu and Transpose split their output into ranges. Each element is still the serial loop's one float operation on the same operands, so the output is bit-identical at any thread count. tests/audio_parallel_test.cpp: an Add (bias row), Mul (scalar), Sub (column on the left), Div (equal shape), Erf, Transpose chain at 301x193, memcmp against the plain serial arithmetic at 1-64 threads. Mutants of the equal-shape path, the stride-0 branch, the broadcast stride and the Transpose start index each fail it. Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The elementwise ops are the largest non-GEMM family in the whisper encoder — 165 nodes, 3.6 s of it — and the loop does two dot products over the dims plus a carry step per element, then compares the op name against
std::stringtwice per element.For the two shapes that actually dominate the graph — equal dims (residual adds, the GELU chain) and a scalar — there is no index arithmetic to do at all.
Loop-level, in one process so load cancels (the widest elementwise shape in the encoder,
1x1500x3072= 4.6 M elements, 20 reps each):In situ, alternating the two binaries under the same load (
ONEBIT_AUDIO_OP_TIMES=1, summing Add/Sub/Mul/Div/Erf over the encoder pass):Less than the loop's 9.4× for two honest reasons: the family total still contains Erf at 365 ms, which this does not touch, and the executor runs four threads so part of the loop cost was already overlapped.
Byte-identical, not approximately: the arithmetic per element is unchanged (
apply(x, y)is the same expression), only the index source changes, so unlike the softmax change in #2379 there is no summation-order question at all. Checked anyway:audio_ops_testrc 0, andwhisper_transcribeonjfk.wavreturns the same 23 ids. The scalar path is used by the graph's own scalar multiplies, and anything else (a real broadcast) still takes the old loop unchanged.Remaining for #2366: after #2379 and this, the elementwise family is ~3.4 s at this load. Its biggest single member is now Erf (365 ms, 14 calls) —
std::erfper element, which is a different problem (a polynomial approximation would change results, so it needs the ids check to justify itself) — and Conv 0.75 s is untouched.