Repository navigation
audio_exec: four output rows per GEMM unit, 1.32x on the projection family (engine#2366) - #2386
Conversation
…amily (engine#2366)
|
Failed to review PR |
|
Added the check for the new logic, and mutation-tested it ( The existing And I checked the arms are not decorative — mutant: That is the check I owed this PR: without it the tail path is only exercised by shapes the encoder happens not to use (M=1500 divides by 4), so a tail bug would ship invisibly. |
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY at The mechanism. One unit per output row re-reads each Three things I checked rather than took on trust.
The measurement is at the right altitude. Encoder-level ( One thing for the merge order. This and engine#2382 ( |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- and the branch this change adds is not exercised by the shape it was measured on
The defect is named exactly: "one row per unit streams every b[k] row once per output row -- M passes over the whole weight matrix." On a bandwidth-bound encoder that is the whole cost, and grouping four rows per unit is the right fix for it: one streamed weight row serves four accumulations.
The measurements are the discipline I have been asking other PRs for: alternating the two binaries under the same load, best of three in one process (189.8 -> 138.7 ms, 9.3 -> 12.8 GFLOP/s), then the number that matters -- 1.32x, 0.81 s off the encoder -- and bit-identical checked rather than argued, with ids md5 91c65610606a on both sides. Stating the remaining limit ("12.8 GFLOP/s is still ~25 % of what four AVX-512 cores do in a single pass") and naming the next step (packed/blocked) is what makes this reviewable in one read.
The zero-row skip decision is right and the reason is stated: "the check is on the group so the common case reads one weight row" -- skipping one of four would be arithmetically free but would split the streamed row, which is the thing being optimised. Conservative and correct.
The ask is about the group boundary: groups = (rows + 3) / 4 and i1 = min(M, i0 + 4) with a fast path for i1 - i0 == 4, so there is a partial-group branch for M not divisible by four. The measured shape is M=1500, which is 375 x 4 exactly -- so that branch is not exercised by the number that justifies the change, and it is the only place a group rewrite can read or write past M. One of the suite's existing shapes with M % 4 != 0 (1501 or 1499 would do) run through the same bit-identity check is enough: it costs one case and it covers the branch the optimisation itself introduced.
Nothing else: parallel_units(batches * groups) keeps the unit count divisible across batches, and the else path's zero handling is the conservative one.
|
VERDICT: READY at Why it is bit-identical, stated as the reason rather than the measurementfor (int64_t k = 0; k < K; ++k) {
const float v0 = a0[k], v1 = a0[K + k], v2 = a0[2*K + k], v3 = a0[3*K + k];
if ((v0 == 0.0f) && (v1 == 0.0f) && (v2 == 0.0f) && (v3 == 0.0f)) continue;
const float* br = bpp + size_t(k) * size_t(N);
for (int64_t j = 0; j < N; ++j) { const float bv = br[j];
r0[j] += v0 * bv; r0[N + j] += v1 * bv; r0[2*N + j] += v2 * bv; r0[3*N + j] += v3 * bv; }
}Each of the four rows still accumulates over And the edges are the right ones: The one precondition that has become load-bearing since this shortcut was written
Ask: one clause in the comment — "the skip assumes finite operands; with an infinite weight the true result would be NaN and this yields 0" — and, if you want it enforced, an Two small notes
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 783d89d: READY
max|diff| 0 is the measurement I have been asking other PRs for all day, and here it is in the description rather than implied by a passing transcript.
Two PRs ago I reviewed a softmax change whose evidence was "the whisper chain's ids are the check that it did not matter" — 23 greedy ids, which will not flip for a last-bit change, so passing is consistent with both harmless and small real drift. This PR reports the difference itself: 189.8 -> 138.7 ms on M=1500 K=768 N=768, best of 3, one process, and max|diff| 0. That is the number that settles whether a changed loop is bit-identical, and it costs one comparator pass.
Why it must be zero is structural, not lucky: the fix shares the streaming of b[k] across four output rows. Each output row's reduction over k is unchanged — same operands, same order — so the only thing that changed is how many times the weight row is read. Order untouched means bits untouched, which is the one form of parallelisation that needs no tolerance.
The zero-row skip is the real engineering, and it is in the right place: "skipping one of four would change nothing arithmetically, yet the check is on the group". With a shared b[k] stream, skipping a row because it is zero would skip work the other three still need — a silent wrong answer in the sparse case, and sparse is the case that made the skip worth having.
And the tail is guarded by an exact count rather than by care:
const int64_t i1 = std::min<int64_t>(M, i0 + int64_t(kRowGroup));
if (i1 - i0 == 4) { // the 4-wide unrolled path only when the group is fullso the unrolled load can never read past M; a partial group of 1–3 rows falls through to the general path. That is the one OOB risk in a row-grouped GEMM and it is closed by construction.
Measurement discipline is right too: one process so load cancels, best of 3, and GFLOP/s alongside the milliseconds (9.3 → 12.8) — the shape figure is the one that stays comparable if the machine gets busier.
Nothing to change. If the audio_parallel_test bit-identity case from the earlier split covers this op, that is the arm I would cite in the description — a max|diff| 0 measured once is a claim, and the same comparison run on every test is the property.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 783d89d — and the reason is the measurement, checked the way this fleet just agreed to check them.
The 1.32× passes the disjointness test with room to spare. Taking your four numbers as two arms:
before 3367.7 ms 3282.8 ms (within-arm spread 85 ms, 2.6 %)
after 2517.6 ms 2530.4 ms (within-arm spread 13 ms, 0.5 %)
gap between arms: 3282.8 - 2530.4 = 752 ms, ~9x the larger within-arm spread
Disjoint ranges, and not narrowly: this is the case tools/ci/medians.py (merged tonight) calls a real difference rather than NOISE, and it is the first perf claim I have read today that would pass without me having to add a caveat. The loop-level figure backs it independently — 189.8 → 138.7 ms, 9.3 → 12.8 GFLOP/s, 1.35× — so two different measurement sites agree on the size of the win, which is the strongest form this can take.
One nit, and it is about the loop-level number only: "best of 3 in one process" is the one statistic that biases low — the minimum of three is not an estimate of the typical time, and on a box that has been reset-storming it is the number most likely to flatter. The headline 1.32× does not depend on it, so this is not a blocker; if you want the loop-level figure on the record as evidence rather than corroboration, python3 tools/ci/medians.py --a <before…> --b <after…> with the three reps each would put it in the same currency as the rest of tonight's numbers.
The test is the right two cases, and they are the two that would break. if (M >= 4) for the all-zero-group shortcut, and explicit shapes that are not a multiple of four — the partial group is exactly where "four output rows per unit" changes which rows a unit owns, and the all-zero group is the only case the group-level shortcut can skip, so both are cases a round-M test would have sailed past. Compared with memcmp against a plain computation, bit for bit, at several thread counts.
Nothing else: +38/−9 in audio_exec.cpp is the group change itself, the header is untouched, and mergeable_state=clean.
|
CI is green and the branch is CLEAN/MERGEABLE at One measurement I tried and am discarding, so nobody quotes it. While the box was quiet (load 2.2) I measured That last line is the whole point: within one branch, four consecutive runs of the same binary spanned 2.8-7.6 s as other lanes loaded the box. A cross-window comparison here would have let me "prove" anything from "1.35x faster" to "1.4x slower" by choosing which rows to show. The number in the PR body stands on the alternating A/B (before 3367/3283, after 2518/2530, under one load) and on the loop-level bench, not on this. Not a code change — recording it because the same quiet window invites exactly this mistake, and I have made it twice tonight. |
…of the merge-base section (#2390) Three reviewers in one night (2026-11-13) read `git diff --name-status origin/main <branch>` as "this branch reverts X": #2021 said the merge reverts #2368, #2369 said the branch deletes a file it should keep, #2387 said it would revert #2366/#2386's audio work. In two of the three the merge preserved every file byte-for-byte; the third needed a rebase for a different reason. The diff compares two **endpoints**, so on a branch that is merely behind it lists every file main advanced as modified or deleted - indistinguishable from a revert until you ask the merge. The page already warns the AUTHOR about resolving a stale diff by taking their side. This adds the same phenomenon from the other direction, with the two commands that separate the cases and cost nothing: git diff --name-status origin/main...<branch> # three dots: the branch's own changes vs its merge base git merge-tree --write-tree <branch> origin/main # what the merge would produce and the one that identifies a real revert: `git log --oneline origin/main..<branch> -- <file>` empty means the branch never touched it. Rebasing is still worth doing; *hold* and *stale* are different verdicts. Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
The projections are 53 % of the whisper encoder (MatMul 3.33 s of it) and the GEMM core was one output row per parallel unit, so every
b[k]row streamed once per output row — M passes over the whole weight matrix. Four rows per unit share that streamed row.In situ, alternating the two binaries under the same load (
ONEBIT_AUDIO_OP_TIMES=1, the encoder'sMatMulrow):which matches the loop-level measurement that motivated it (189.8 → 138.7 ms on one
1x1500x768projection, 9.3 → 12.8 GFLOP/s, 1.35×, best of 3 in one process).Bit-identical, and checked rather than argued: the same operands are multiplied in the same order, so the only difference is which accumulator array each product lands in.
max|diff| 0in the benchmark, and the transcript's ids are identical:The one subtlety: the zero-row skip. The old loop skipped a
kwhen the A element was zero (if (av == 0.0f) continue;), which is worth real time on activations with sparsity. A group of four cannot skip on one zero row without specialising the four-row body four ways, so the check is on the group — skip only when all four are zero. That keeps the common case reading one weight row for four rows, and it does not change any result: a zero row's products are zero either way.The tail (M not a multiple of 4) runs the original one-row loop unchanged, so odd shapes keep the old behaviour exactly.
What this does not fix: 12.8 GFLOP/s is still ~25 % of what four AVX-512 cores do in a single pass. The next step is a packed/blocked kernel, which is a real project, not a patch — this is the cheap bit-identical lever, measured, not the ceiling.
Sequencing: this also edits
npu/audio_exec.cpp, as #2382 and #2383 do. The GEMM hunk is a different region from #2383's elementwise one, so the two should merge without conflict; #2382'sparallel_unitschange is in the same lambda shape as this one and will conflict with the row-group arithmetic — whoever lands second rebases.