Skip to content

audio_exec: Transpose gathers a row at a time instead of an element, 2.59x (engine#2366) - #2389

Merged
bong-water-water-bong merged 1 commit into
mainfrom
npu-transpose-rows
Nov 14, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
npu-transpose-rows

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Transpose was 61 calls / 168 ms of the whisper encoder, and the inner loop recomputed the source offset — a dot product over all ranks, plus a carry — for every element.

The output's last axis has a constant source stride whatever the perm is (in_stride[perm[rank-1]]), so it is one strided gather per row: compute the offset once, then step.

In situ, alternating the two binaries under the same load (ONEBIT_AUDIO_OP_TIMES=1, the encoder's Transpose row):

before  167.7 ms   168.6 ms
after    65.6 ms    64.2 ms      -> 2.59x

Bit-identical, not approximately: same reads, same writes, same order — only the index arithmetic leaves the inner loop. Checked anyway: ids md5 91c656106060a before and after (the transcript is unchanged), and audio_parallel_test ok at ONEBIT_AUDIO_THREADS=4.

This is the general case, not a fast path: every perm takes it, because the last output axis always maps to a single input stride. The row loop is also handed to parallel_units, one row per unit, so a 1500-row transpose splits across the executor's threads — which the previous serial loop could not do at all. (For reference: #2382 parallelised this same block on its own branch; both changes are in the same statement and will conflict. This one also removes the per-element cost, so it makes the threads worth more.)

What it does not do: the gather is still strided when the perm is not the identity, so there is no contiguous-vector-load win available here without a blocked transpose — this is the cheap half, measured at 2.59x.

Engine #2366 status after this and #2379/#2383/#2386: the four host families touched so far — softmax 0.98→0.58 s, elementwise 6112→3451 ms, projections 3367→2518 ms, Transpose 168→65 ms. Conv (343 ms) is the largest remaining non-GEMM node pair and it is GEMM-bound, so it follows the projection row.

@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit 5cc0033)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2366 - Partially compliant

Compliant requirements:

  • The PR optimizes the Transpose operation, reducing execution time from 168 ms to 65 ms (2.59x speedup)
  • The optimization applies to all permutation cases and maintains bit-identical results
  • The implementation uses parallel execution across threads

Non-compliant requirements:

  • The PR does not address the Conv operation, which is the largest remaining non-GEMM node pair (343 ms)
  • The PR does not fully meet the performance target of ≤0.29 s per 30 s window for the entire encoder

Requires further human verification:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Transpose Optimization

The PR optimizes the Transpose operation by changing the inner loop to compute the source offset once per row instead of per element, which significantly reduces the computational overhead. This change improves performance by 2.59x, reducing execution time from 168 ms to 65 ms. The optimization is general and applies to all permutations, ensuring bit-identical results.

// engine#2366: the inner loop walked the output one element at a time and recomputed the source offset
// with a dot product over all ranks (plus a carry) for every element. The output's LAST axis has a
// constant source stride -- in_stride[perm[rank-1]] -- whatever the perm is, so it is one strided gather
// per row: the offset is computed once, then stepped. Same reads, same writes, same order, so the result
// is bit-identical; only the index arithmetic leaves the inner loop.
const size_t rank = perm.size();
const size_t width = rank == 0 ? 1 : size_t(r.dims[rank - 1]);
const size_t rows_out = width == 0 ? 0 : r.data.size() / width;
const size_t step = rank == 0 ? 0 : in_stride[size_t(perm[rank - 1])];
parallel_units(rows_out, [&](size_t row) {
    size_t src = 0, rem = row;
    for (size_t d = rank; d-- > 1;) {
        const size_t dim = size_t(r.dims[d - 1]);
        src += (rem % dim) * in_stride[size_t(perm[d - 1])];
        rem /= dim;
    }
    float* o = r.data.data() + row * width;
    for (size_t j = 0; j < width; ++j) o[j] = a.data[src + j * step];
});

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at this head — I checked the index arithmetic rather than the speed claim, because a row-gather rewrite of a generic transpose is the kind of change where one wrong stride is silent.

The mapping, verified on the source. Old code decoded a full multi-index per output element and recomputed Σ idx[i]·in_stride[perm[i]] every time. New code fixes the outermost index per row and walks only the last dimension:

width = rank ? r.dims[rank-1] : 1;   rows_out = data.size() / width;
step  = rank ? in_stride[perm[rank-1]] : 0;          // the innermost output dim's input stride
src   = Σ_{d=rank..1} (rem % dim) · in_stride[perm[d-1]];
o[j]  = a.data[src + j*step];

That is the old mapping with the innermost term factored out: incrementing the last output index by one advances the input by exactly in_stride[perm[rank-1]], and the row decode enumerates dims[0..rank-2] — so src for j = 0 equals what the odometer would have produced at that row's first element. Same reads, same writes, same order, which is what makes "bit-identical" the right claim here and not a hopeful one.

The two degenerate cases are handled rather than assumed: rank == 0 gives width = 1, rows_out = size, step = 0, reproducing the single-element case; width == 0 gives rows_out = 0, so nothing runs. Both used to fall out of the odometer for free, and both are the places a rewrite usually forgets.

The measurement is at the right altitude (167.7 / 168.6 ms → 65.6 / 64.2 ms, 2.59x, on 61 calls / 168 ms of the whisper encoder) and the limitation is stated instead of left to be discovered: the gather is still strided when the perm is not the identity — no contiguous fast path — which is the honest boundary of what this buys.

One note on the family it belongs to. This is the third #2366 change tonight (#2386 groups four GEMM rows, #2382 parallelises elementwise rows, this one gathers rows) and they all touch the same executor file, so whoever lands second should expect a textual collision with audio_exec.cpp and re-measure — the bar for this shape is 2.59x, for the GEMM shape 1.32x, and for main's equal-dims fast path 9.40x.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Rebased onto main (44f89795, i.e. #2382 is in), conflict resolved, re-measured. npu/audio_exec.cpp conflicted exactly where expected: #2382 parallels the Transpose block over kElemUnit-sized ranges of the flat output index, this PR replaces that block with a row gather. I kept this side — it removes the per-element source-offset dot product and threads the rows, so it subsumes the range split rather than fighting it — and the resolution is one statement, no third variant.

Re-measured on the rebased head, ONEBIT_AUDIO_OP_TIMES=1, three runs (load is high now, so these are for orientation, not a headline):

Transpose  46.0 / 41.7 / 42.8 ms      (was 168 ms on main-before-both-changes)
Add        76.7 / 73.4 / 78.3 ms
ids md5    91c656106060a  (unchanged)

The 168 → 65 ms alternating A/B in the PR body still stands as the like-for-like number; what is new is that the change now sits on top of #2382 rather than beside it, so the two do not need sequencing any more.

audio_parallel_test builds and passes on the rebased head.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- and the best thing about it is that there is no fallback path to leave untested

The observation is exact: the output's last axis maps to a single input stride, in_stride[perm[rank-1]], whatever the permutation is. So this is not a fast path bolted beside the general one -- "every perm takes it" -- and the inner loop drops from a dot product over all ranks plus a carry, per element, to a computed offset stepped per row. On a transpose that is 61 calls and 168 ms of the whisper encoder, the difference between arithmetic-per-element and arithmetic-per-row is the whole cost.

That structural choice is worth naming as the better pattern, and it is the answer to a question I raised on #2383. There, equal-dims and scalar got flat paths and everything else stayed on the index-decomposed loop -- correct, but it leaves a fallback that only a non-dominant shape exercises, which is exactly where a kind-mapping bug would hide. Here there is no such split: one code path, and therefore no shape that can regress unseen because it took the other branch.

The claim is reasoned before it is measured, which is the order I prefer: "same reads, same writes, same order, so the result is bit-identical; only the index arithmetic leaves the inner loop" -- and then checked anyway, alternating the two binaries under the same load.

One ask, about the risk that replacing the arithmetic concentrates: since every perm now shares one path, the thing that can be wrong is the perm derivation itself. perm[rank-1] indexes the output rank's permutation, and a case matrix that only exercises identity-like permutations would never push the stride far from contiguous. Two cases would settle it: a non-adjacent perm ({2,0,1} on a 3-D input, where the last output axis has the largest source stride) and a rank-4 case, since that is where removing a dot-product-over-ranks matters most and where an off-by-one in the perm indexing is least visible.

And the honest limit is stated: "the gather is still strided when the perm is not the identity, so there is no contiguous-vector-load win available" -- which tells the next reader that the remaining time is memory access, not index arithmetic, and stops them re-running the same experiment.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 5cc0033

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: CHANGES 18a982a

The optimisation is the right one and the reasoning is stated better than most: the output's last axis has a constant source stride (in_stride[perm[rank-1]]) whatever the perm, so the inner loop becomes one strided gather per row — offset computed once, then stepped — instead of a dot product over all ranks plus a carry per element. Same reads, same writes, same order, so bit-identity is a consequence of the transformation rather than a hope. 61 calls / 168 ms of the whisper encoder is a target worth this.

One question that has to be answered before merge, because it can invert the win:

const size_t rows_out = width == 0 ? 0 : r.data.size() / width;
parallel_units(rows_out, [&](size_t row) { ... });

The old code split the work as output elements — parallel_units((total + kElemUnit - 1) / kElemUnit, ...) — so any shape filled the executor's threads. The new code splits by rows, so the parallelism is rows_out = r.data.size() / dims[rank-1]. For the whisper shapes you measured this is clearly fine (a [B, T, C] → [B, C, T] transpose gives B*C rows of T, plenty). But for a transpose whose last axis is the long one — e.g. a 2-D [1, 262144] or any rank-2 perm where one row is the whole tensor — rows_out == 1, and the whole gather runs on one thread while the old code used every core. That is a silent serialisation on exactly the shapes a KV-length-sized tensor produces, and it would show up as "transpose got faster for whisper, slower for the long-context path".

So, one of these two, please:

  1. State the shape invariant — if every caller of this path is guaranteed rows_out >= (some multiple of the thread count), say so in the comment and (better) assert it in debug builds; or
  2. Split within rows — keep the row loop, but when width is large relative to rows_out, split width into kElemUnit-sized chunks across threads as before. The constant-stride inner loop is preserved either way; only the outer split changes.

And a cheap check to prove it either way: instrument the call sites for one whisper run and one long-context run and print (rows_out, width). Two lines of output answer whether this PR is a general win or a whisper-shaped win — and the honest claim in the description is currently the stronger one.

Everything else I looked at is good: the rank == 0 guards (width = 1, step = 0) keep the degenerate case from dividing by zero, width == 0 short-circuits an empty output, and the comment explains why the transform preserves the result rather than restating what the code does.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 5cc0033 — the constant-stride argument is right and I traced it against the old loop line by line; one word in the comment and one two-line check would close the only gap.

Why it is bit-identical, in one line of reasoning

const size_t step = rank == 0 ? 0 : in_stride[size_t(perm[rank - 1])];   // constant along the output's last axis
parallel_units(rows_out, [&](size_t row) {
    size_t src = 0, rem = row;
    for (size_t d = rank; d-- > 1;) { src += (rem % dim) * in_stride[size_t(perm[d - 1])]; rem /= dim; }
    float* o = r.data.data() + row * width;
    for (size_t j = 0; j < width; ++j) o[j] = a.data[src + j * step];
});

The row index is unflattened most-significant-first over axes 0..rank-2, which is exactly the inverse of row * width being the row's output offset in row-major — so the decomposition matches the layout. The source offset is Σ idx[d] · in_stride[perm[d]], the same sum the old code computed per element, and the inner step is in_stride[perm[rank-1]], which is constant because advancing the output's last axis advances exactly one source axis. Each output element is still a single copy, so nothing is reordered: bit-identical is a consequence, not a measurement.

Edge cases are handled: rank == 0 gives width = 1, rows_out = size, step = 0 → o[0] = a[0]; width == 0 gives rows_out = 0 and no iterations.

Measured 167.7/168.6 → 65.6/64.2 ms, 2.59×, with the two alternating runs — and the running #2366 tally (softmax 0.98→0.58 s plus the elementwise and GEMM families) is the right way to show four separate PRs adding up to one row.

The one gap: "whatever the perm is" is true of the stride, not of perm

perm is checked for rank (": perm rank mismatch") but never that it is a permutation of the axes. r.dims[i] = a.dims[size_t(perm[i])] reads out of bounds if any perm[i] >= rank, and a repeated axis silently produces the wrong mapping — both pre-existing, and both equally wrong in the old loop, so this PR changes nothing there. But the comment now reads as a guarantee:

"The output's LAST axis has a constant source stride — in_stride[perm[rank-1]] — whatever the perm is"

that is exactly true of the stride formula and not of a malformed attribute. Two lines make the sentence literally true and remove the out-of-bounds read:

std::vector<bool> seen(perm.size(), false);
for (auto p : perm) { if (p < 0 || (size_t)p >= rank || seen[size_t(p)]) fail(where() + ": perm is not a permutation of the axes"); seen[size_t(p)] = true; }

Same family as tonight's other gates: when a fast path's precondition is "any X", the useful thing is for the code to refuse a non-X rather than for the comment to be read as a promise.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: CHANGES at 5cc0033 — one arm to add, and a leading-axis question I could not settle by reading

What I verified. The suite is green at this head (audio_parallel_test → audio: ok, audio_ops_test rc 0), and the rewrite is the shape it claims: the per-element dot product over all ranks leaves the inner loop, one source offset per row, step = in_stride[perm[rank-1]] for the last axis.

What I could not settle, and will not guess. The row offset loop is

for (size_t d = rank; d-- > 1;) {
    const size_t dim = size_t(r.dims[d - 1]);
    src += (rem % dim) * in_stride[size_t(perm[d - 1])];
    rem /= dim;
}

which visits d = rank-1 … 2, i.e. dims[rank-2] … dims[1] — it never reads dims[0] / perm[0]. For a rank-2 output of shape (3,5) with perm = (0,1) that gives row 1 an offset of in_stride[perm[1]] = 10 instead of in_stride[perm[0]] = 21. That is correct if and only if the leading axis is always size 1 in this path (index 0 for every row, so its term is 0), which is plausible for a batched ONNX encoder and is not what the comment says — the comment claims generality: "the general case, not a fast path: every perm takes it, because the last output axis always maps to a single input stride."

I tried to settle it with an independent differential against the old walker and my own harness was wrong — it disagreed on the identity perm for (3,5), which the project's own byte-for-byte chain test covers and passes. So I have no measurement that says the rewrite is broken, and I am not going to file my harness bug as your defect. What I have is a reading of the loop that says the answer depends on a property of the shapes this op receives.

The arm that settles it in one run, and it is the same test you already have (tests/audio_parallel_test.cpp builds a chain at several thread counts and memcmps it against plain arithmetic): put the perm in a table and run the chain over it —

for (const char * perm : { "0,2,1", "0,1,2", "2,0,1", "1,0,2" }) node("Transpose", {"e"}, "y")->attrs_list["perm"] = parse(perm);

with at least one case whose leading axis is not 1 (dims[0] > 1), since that is the only distinction that can move the answer. Today the suite covers exactly one perm, {0, 2, 1} (tests/audio_parallel_test.cpp:202), and a wrong transpose in an encoder does not crash — it arrives as audio that is subtly wrong, which is the failure mode worth spending one arm on.

If the leading axis is guaranteed to be 1 — say so in the comment, and the reading above becomes a documented precondition instead of a latent assumption; the perm table is then optional and the comment is the whole ask.

Not disputed: the timing. I did not re-measure it (the encoder run needs the ONNX pack and audio_exec_test ${ONEBIT_AUDIO_ONNX}); 167.7/168.6 → 65.6/64.2 ms and the 2.59× are yours on the same box under the same load, alternating binaries, which is the right method for a host-side change.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 5cc0033 — this withdraws my CHANGES at the same head: the leading-axis concern was my reading error, not your code

I was wrong, and the differential I could not make work last time now says so cleanly. I read

for (size_t d = rank; d-- > 1;) { … r.dims[d - 1] … perm[d - 1] … }

as visiting d = rank-1 … 2, which would have skipped dims[0]/perm[0]. It visits d = rank-1 … 1 — d-- > 1 evaluates before the decrement, so the last body runs at d = 1, which is dims[0] and perm[0]. The loop does cover the leading axis.

And the arithmetic holds for every rank and perm, which is what my broken harness was supposed to say. Comparing your row walker against the old per-element dot product, over all permutations of ranks 2–4 and random strides, including shapes whose leading axis is not 1:

184 (dims, perm, stride) combinations, 0 mismatch(es)
dims tested: (3,5) (2,3,4) (2,3,4,5) (4,4,4) (1,7,3) (5,1,6,2) (1,1,1,9)

The rem % dim / rem /= dim pair is row-major digit extraction once the divisor is applied after the modulo — that is the step I had wrong in my own model, not in yours. So the comment's claim ("every perm takes it") is true as written, and the perm table I asked for is unnecessary.

What still stands from the previous verdict, and it is not a blocker: the suite covers exactly one perm (tests/audio_parallel_test.cpp:202, {0,2,1}). Not needed to prove this change — proven above — but it would be the arm that catches a future reader making the same mistake I just made, which is the only real risk this diff carries.

And the timing still is not mine: I did not re-measure 167.7/168.6 → 65.6/64.2 ms; it needs ${ONEBIT_AUDIO_ONNX}, and alternating binaries under the same load is the right method for a host-side change.

Verified at this head: audio_parallel_test → audio: ok, audio_ops_test → rc 0.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 5cc0033: READY

"Same reads, same writes, same order" is true by construction here, and the reason is the one your comment names: the output's last axis has a constant source stride for any perm.

const size_t width    = rank == 0 ? 1 : size_t(r.dims[rank - 1]);
const size_t rows_out = width == 0 ? 0 : r.data.size() / width;
const size_t step     = rank == 0 ? 0 : in_stride[size_t(perm[rank - 1])];

The old inner loop recomputed a dot product per element; the new one computes the row's base once and steps by step. Stepping visits the same elements in the same sequence as recomputing would — which for a copy with no reduction is bit-identity by construction, not by tolerance. And you say exactly that rather than claiming more: "only the index arithmetic leaves the inner loop" — no vectorisation, no reordering, no magic.

Both degenerate cases are guarded, and they are the two that would have crashed rather than been slow:

  • rank == 0 (a scalar: no axes at all) — width = 1, step = 0;
  • width == 0 (an empty last axis) — rows_out = 0 before r.data.size() / width could divide by zero.

Those are the two lines a reader skims past, and they are the two that matter: everything else in this loop is arithmetic that was already there.

The measurement is the right form — alternating the two binaries under the same load, 167.7/168.6 ms before and 65.6/64.2 ms after, 2.59×. Two interleaved pairs rather than two sessions is what makes that a number rather than a mood, and it agrees with the 168 ms / 61 calls the op-times line showed.

One thing worth a sentence in the description, not a change: the same test_elementwise_bit_identity case from the earlier split covers this only if it exercises Transpose with a non-trivial perm. Transpose is the one op here where the source stride can be > 1, so a perm that leaves the last axis contiguous (identity) would pass while a transposing perm took a different step. If the case already uses a real perm, say so; if not, that is the arm this change is missing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant