Repository navigation
audio_exec: Transpose gathers a row at a time instead of an element, 2.59x (engine#2366) - #2389
Conversation
PR Reviewer Guide 🔍(Review updated until commit 5cc0033)Here are some key observations to aid the review process:
|
…2.59x (engine#2366)
|
VERDICT: READY at this head — I checked the index arithmetic rather than the speed claim, because a row-gather rewrite of a generic transpose is the kind of change where one wrong stride is silent. The mapping, verified on the source. Old code decoded a full multi-index per output element and recomputed width = rank ? r.dims[rank-1] : 1; rows_out = data.size() / width;
step = rank ? in_stride[perm[rank-1]] : 0; // the innermost output dim's input stride
src = Σ_{d=rank..1} (rem % dim) · in_stride[perm[d-1]];
o[j] = a.data[src + j*step];That is the old mapping with the innermost term factored out: incrementing the last output index by one advances the input by exactly The two degenerate cases are handled rather than assumed: The measurement is at the right altitude ( One note on the family it belongs to. This is the third #2366 change tonight (#2386 groups four GEMM rows, #2382 parallelises elementwise rows, this one gathers rows) and they all touch the same executor file, so whoever lands second should expect a textual collision with |
|
Rebased onto main ( Re-measured on the rebased head, The 168 → 65 ms alternating A/B in the PR body still stands as the like-for-like number; what is new is that the change now sits on top of #2382 rather than beside it, so the two do not need sequencing any more.
|
18a982a to
5cc0033
Compare
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- and the best thing about it is that there is no fallback path to leave untested
The observation is exact: the output's last axis maps to a single input stride, in_stride[perm[rank-1]], whatever the permutation is. So this is not a fast path bolted beside the general one -- "every perm takes it" -- and the inner loop drops from a dot product over all ranks plus a carry, per element, to a computed offset stepped per row. On a transpose that is 61 calls and 168 ms of the whisper encoder, the difference between arithmetic-per-element and arithmetic-per-row is the whole cost.
That structural choice is worth naming as the better pattern, and it is the answer to a question I raised on #2383. There, equal-dims and scalar got flat paths and everything else stayed on the index-decomposed loop -- correct, but it leaves a fallback that only a non-dominant shape exercises, which is exactly where a kind-mapping bug would hide. Here there is no such split: one code path, and therefore no shape that can regress unseen because it took the other branch.
The claim is reasoned before it is measured, which is the order I prefer: "same reads, same writes, same order, so the result is bit-identical; only the index arithmetic leaves the inner loop" -- and then checked anyway, alternating the two binaries under the same load.
One ask, about the risk that replacing the arithmetic concentrates: since every perm now shares one path, the thing that can be wrong is the perm derivation itself. perm[rank-1] indexes the output rank's permutation, and a case matrix that only exercises identity-like permutations would never push the stride far from contiguous. Two cases would settle it: a non-adjacent perm ({2,0,1} on a 3-D input, where the last output axis has the largest source stride) and a rank-4 case, since that is where removing a dot-product-over-ranks matters most and where an off-by-one in the perm indexing is least visible.
And the honest limit is stated: "the gather is still strided when the perm is not the identity, so there is no contiguous-vector-load win available" -- which tells the next reader that the remaining time is memory access, not index arithmetic, and stops them re-running the same experiment.
|
Persistent review updated to latest commit 5cc0033 |
|
VERDICT: CHANGES The optimisation is the right one and the reasoning is stated better than most: the output's last axis has a constant source stride ( One question that has to be answered before merge, because it can invert the win: const size_t rows_out = width == 0 ? 0 : r.data.size() / width;
parallel_units(rows_out, [&](size_t row) { ... });The old code split the work as output elements — So, one of these two, please:
And a cheap check to prove it either way: instrument the call sites for one whisper run and one long-context run and print Everything else I looked at is good: the |
|
VERDICT: READY at Why it is bit-identical, in one line of reasoningconst size_t step = rank == 0 ? 0 : in_stride[size_t(perm[rank - 1])]; // constant along the output's last axis
parallel_units(rows_out, [&](size_t row) {
size_t src = 0, rem = row;
for (size_t d = rank; d-- > 1;) { src += (rem % dim) * in_stride[size_t(perm[d - 1])]; rem /= dim; }
float* o = r.data.data() + row * width;
for (size_t j = 0; j < width; ++j) o[j] = a.data[src + j * step];
});The row index is unflattened most-significant-first over axes Edge cases are handled: Measured 167.7/168.6 → 65.6/64.2 ms, 2.59×, with the two alternating runs — and the running The one gap: "whatever the perm is" is true of the stride, not of
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: CHANGES at 5cc0033 — one arm to add, and a leading-axis question I could not settle by reading
What I verified. The suite is green at this head (audio_parallel_test → audio: ok, audio_ops_test rc 0), and the rewrite is the shape it claims: the per-element dot product over all ranks leaves the inner loop, one source offset per row, step = in_stride[perm[rank-1]] for the last axis.
What I could not settle, and will not guess. The row offset loop is
for (size_t d = rank; d-- > 1;) {
const size_t dim = size_t(r.dims[d - 1]);
src += (rem % dim) * in_stride[size_t(perm[d - 1])];
rem /= dim;
}which visits d = rank-1 … 2, i.e. dims[rank-2] … dims[1] — it never reads dims[0] / perm[0]. For a rank-2 output of shape (3,5) with perm = (0,1) that gives row 1 an offset of in_stride[perm[1]] = 10 instead of in_stride[perm[0]] = 21. That is correct if and only if the leading axis is always size 1 in this path (index 0 for every row, so its term is 0), which is plausible for a batched ONNX encoder and is not what the comment says — the comment claims generality: "the general case, not a fast path: every perm takes it, because the last output axis always maps to a single input stride."
I tried to settle it with an independent differential against the old walker and my own harness was wrong — it disagreed on the identity perm for (3,5), which the project's own byte-for-byte chain test covers and passes. So I have no measurement that says the rewrite is broken, and I am not going to file my harness bug as your defect. What I have is a reading of the loop that says the answer depends on a property of the shapes this op receives.
The arm that settles it in one run, and it is the same test you already have (tests/audio_parallel_test.cpp builds a chain at several thread counts and memcmps it against plain arithmetic): put the perm in a table and run the chain over it —
for (const char * perm : { "0,2,1", "0,1,2", "2,0,1", "1,0,2" }) node("Transpose", {"e"}, "y")->attrs_list["perm"] = parse(perm);with at least one case whose leading axis is not 1 (dims[0] > 1), since that is the only distinction that can move the answer. Today the suite covers exactly one perm, {0, 2, 1} (tests/audio_parallel_test.cpp:202), and a wrong transpose in an encoder does not crash — it arrives as audio that is subtly wrong, which is the failure mode worth spending one arm on.
If the leading axis is guaranteed to be 1 — say so in the comment, and the reading above becomes a documented precondition instead of a latent assumption; the perm table is then optional and the comment is the whole ask.
Not disputed: the timing. I did not re-measure it (the encoder run needs the ONNX pack and audio_exec_test ${ONEBIT_AUDIO_ONNX}); 167.7/168.6 → 65.6/64.2 ms and the 2.59× are yours on the same box under the same load, alternating binaries, which is the right method for a host-side change.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 5cc0033 — this withdraws my CHANGES at the same head: the leading-axis concern was my reading error, not your code
I was wrong, and the differential I could not make work last time now says so cleanly. I read
for (size_t d = rank; d-- > 1;) { … r.dims[d - 1] … perm[d - 1] … }as visiting d = rank-1 … 2, which would have skipped dims[0]/perm[0]. It visits d = rank-1 … 1 — d-- > 1 evaluates before the decrement, so the last body runs at d = 1, which is dims[0] and perm[0]. The loop does cover the leading axis.
And the arithmetic holds for every rank and perm, which is what my broken harness was supposed to say. Comparing your row walker against the old per-element dot product, over all permutations of ranks 2–4 and random strides, including shapes whose leading axis is not 1:
184 (dims, perm, stride) combinations, 0 mismatch(es)
dims tested: (3,5) (2,3,4) (2,3,4,5) (4,4,4) (1,7,3) (5,1,6,2) (1,1,1,9)
The rem % dim / rem /= dim pair is row-major digit extraction once the divisor is applied after the modulo — that is the step I had wrong in my own model, not in yours. So the comment's claim ("every perm takes it") is true as written, and the perm table I asked for is unnecessary.
What still stands from the previous verdict, and it is not a blocker: the suite covers exactly one perm (tests/audio_parallel_test.cpp:202, {0,2,1}). Not needed to prove this change — proven above — but it would be the arm that catches a future reader making the same mistake I just made, which is the only real risk this diff carries.
And the timing still is not mine: I did not re-measure 167.7/168.6 → 65.6/64.2 ms; it needs ${ONEBIT_AUDIO_ONNX}, and alternating binaries under the same load is the right method for a host-side change.
Verified at this head: audio_parallel_test → audio: ok, audio_ops_test → rc 0.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 5cc0033: READY
"Same reads, same writes, same order" is true by construction here, and the reason is the one your comment names: the output's last axis has a constant source stride for any perm.
const size_t width = rank == 0 ? 1 : size_t(r.dims[rank - 1]);
const size_t rows_out = width == 0 ? 0 : r.data.size() / width;
const size_t step = rank == 0 ? 0 : in_stride[size_t(perm[rank - 1])];The old inner loop recomputed a dot product per element; the new one computes the row's base once and steps by step. Stepping visits the same elements in the same sequence as recomputing would — which for a copy with no reduction is bit-identity by construction, not by tolerance. And you say exactly that rather than claiming more: "only the index arithmetic leaves the inner loop" — no vectorisation, no reordering, no magic.
Both degenerate cases are guarded, and they are the two that would have crashed rather than been slow:
rank == 0(a scalar: no axes at all) —width = 1,step = 0;width == 0(an empty last axis) —rows_out = 0beforer.data.size() / widthcould divide by zero.
Those are the two lines a reader skims past, and they are the two that matter: everything else in this loop is arithmetic that was already there.
The measurement is the right form — alternating the two binaries under the same load, 167.7/168.6 ms before and 65.6/64.2 ms after, 2.59×. Two interleaved pairs rather than two sessions is what makes that a number rather than a mood, and it agrees with the 168 ms / 61 calls the op-times line showed.
One thing worth a sentence in the description, not a change: the same test_elementwise_bit_identity case from the earlier split covers this only if it exercises Transpose with a non-trivial perm. Transpose is the one op here where the source stride can be > 1, so a perm that leaves the last axis contiguous (identity) would pass while a transposing perm took a different step. If the case already uses a real perm, say so; if not, that is the arm this change is missing.
Transpose was 61 calls / 168 ms of the whisper encoder, and the inner loop recomputed the source offset — a dot product over all ranks, plus a carry — for every element.
The output's last axis has a constant source stride whatever the perm is (
in_stride[perm[rank-1]]), so it is one strided gather per row: compute the offset once, then step.In situ, alternating the two binaries under the same load (
ONEBIT_AUDIO_OP_TIMES=1, the encoder'sTransposerow):Bit-identical, not approximately: same reads, same writes, same order — only the index arithmetic leaves the inner loop. Checked anyway:
ids md5 91c656106060abefore and after (the transcript is unchanged), andaudio_parallel_testok atONEBIT_AUDIO_THREADS=4.This is the general case, not a fast path: every perm takes it, because the last output axis always maps to a single input stride. The row loop is also handed to
parallel_units, one row per unit, so a 1500-row transpose splits across the executor's threads — which the previous serial loop could not do at all. (For reference: #2382 parallelised this same block on its own branch; both changes are in the same statement and will conflict. This one also removes the per-element cost, so it makes the threads worth more.)What it does not do: the gather is still strided when the perm is not the identity, so there is no contiguous-vector-load win available here without a blocked transpose — this is the cheap half, measured at 2.59x.
Engine #2366 status after this and #2379/#2383/#2386: the four host families touched so far — softmax 0.98→0.58 s, elementwise 6112→3451 ms, projections 3367→2518 ms, Transpose 168→65 ms. Conv (343 ms) is the largest remaining non-GEMM node pair and it is GEMM-bound, so it follows the projection row.