Skip to content

audio_exec: four output rows per GEMM unit, 1.32x on the projection family (engine#2366) - #2386

Merged
bong-water-water-bong merged 2 commits into
mainfrom
npu-gemm-4row
Nov 13, 2026
Merged

bong-water-water-bong merged 2 commits into
mainfrom
npu-gemm-4row

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

The projections are 53 % of the whisper encoder (MatMul 3.33 s of it) and the GEMM core was one output row per parallel unit, so every b[k] row streamed once per output row — M passes over the whole weight matrix. Four rows per unit share that streamed row.

In situ, alternating the two binaries under the same load (ONEBIT_AUDIO_OP_TIMES=1, the encoder's MatMul row):

before (cB)  3367.7 ms   3282.8 ms
after  (cA)  2517.6 ms   2530.4 ms      -> 1.32x, 0.81 s off the encoder

which matches the loop-level measurement that motivated it (189.8 → 138.7 ms on one 1x1500x768 projection, 9.3 → 12.8 GFLOP/s, 1.35×, best of 3 in one process).

Bit-identical, and checked rather than argued: the same operands are multiplied in the same order, so the only difference is which accumulator array each product lands in. max|diff| 0 in the benchmark, and the transcript's ids are identical:

before ids md5 91c65610606a
after  ids md5 91c65610606a

The one subtlety: the zero-row skip. The old loop skipped a k when the A element was zero (if (av == 0.0f) continue;), which is worth real time on activations with sparsity. A group of four cannot skip on one zero row without specialising the four-row body four ways, so the check is on the group — skip only when all four are zero. That keeps the common case reading one weight row for four rows, and it does not change any result: a zero row's products are zero either way.

The tail (M not a multiple of 4) runs the original one-row loop unchanged, so odd shapes keep the old behaviour exactly.

What this does not fix: 12.8 GFLOP/s is still ~25 % of what four AVX-512 cores do in a single pass. The next step is a packed/blocked kernel, which is a real project, not a patch — this is the cheap bit-identical lever, measured, not the ceiling.

Sequencing: this also edits npu/audio_exec.cpp, as #2382 and #2383 do. The GEMM hunk is a different region from #2383's elementwise one, so the two should merge without conflict; #2382's parallel_units change is in the same lambda shape as this one and will conflict with the row-group arithmetic — whoever lands second rebases.

@github-actions

Copy link
Copy Markdown

Failed to review PR

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Added the check for the new logic, and mutation-tested it (tests/audio_parallel_test.cpp, test_row_group_edges).

The existing M=37 case already covers the row-group tail with one row, but not M < 4 (a lone short group) and not a group whose four rows are all zero — which is the only input on which the group-level zero shortcut can fire. The new arms cover M = 1, 2, 3, 5, 8 and force rows 0-3 to zero at M >= 4, comparing bit-for-bit against an ordered per-element sum at thread counts 1, 2 and 4.

And I checked the arms are not decorative — mutant: for (i = i0; i < i1; ++i) → i < i1 - 1 in the tail branch, i.e. drop the last row of a short group:

MUTANT  rc=1  FAIL MatMul M=1 at ONEBIT_AUDIO_THREADS=1 matches the ordered sum (row-group tail and all-zero group)
restored rc=0 at ONEBIT_AUDIO_THREADS=1,2,3,4

That is the check I owed this PR: without it the tail path is only exercised by shapes the encoder happens not to use (M=1500 divides by 4), so a tail bug would ship invisibly.

@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Potential Performance Regression in Tail Handling

The implementation introduces a new loop structure for handling groups of four rows, but falls back to the original single-row loop for the tail case when M is not a multiple of 4. This fallback may cause a performance regression if the tail is large enough to be a significant portion of the computation, as it does not benefit from the optimized four-row processing.

} else {
    for (int64_t i = i0; i < i1; ++i) {
        float* rr = r0 + size_t(i - i0) * size_t(N);
        for (int64_t k = 0; k < K; ++k) {
            const float av = ap[size_t(i) * size_t(K) + size_t(k)];
            if (av == 0.0f) continue;
            const float* br = bpp + size_t(k) * size_t(N);
            for (int64_t j = 0; j < N; ++j) rr[j] += av * br[j];
        }
    }
}
Zero-Row Skip Logic

The zero-row skip optimization checks if all four rows in a group are zero before skipping. While this preserves correctness, it introduces a potential performance issue if the condition is frequently false, as it requires checking four values instead of one. This could be a bottleneck in sparse activation scenarios.

if ((v0 == 0.0f) && (v1 == 0.0f) && (v2 == 0.0f) && (v3 == 0.0f)) continue;

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 783d89d9d — the win is measured at the level that matters, the tail path is real rather than assumed, and the summation order is untouched, which is what makes it safe.

The mechanism. One unit per output row re-reads each B row once per row; four rows per unit read it once for four A values. That is the whole of it, and the code matches: kRowGroup = 4,
groups = ceil(rows/4), and inside the k loop v0..v3 are read once with a single br walk.

Three things I checked rather than took on trust.

  1. The tail. i1 - i0 == 4 takes the grouped path; everything else takes a per-row loop that is the original code verbatim (av == 0.0f skip and all). So M = 4q + 1..3 is
    handled by the old path, not by a partial group.
  2. The zero shortcut changed meaning in exactly one way, and correctly. The old code skipped a k when av == 0; the grouped code skips when all four are zero — which is the same
    predicate generalised, not a weaker one. The new test_row_group_edges() covers M < 4 and the all-zero group, i.e. the two cases where that difference could bite.
  3. Order is preserved, so this is not audio_ops: softmax_row 3.24x by breaking the dependent reduce chains (engine#2366) #2379's trade: j is still the innermost loop and k still ascends within a row, so a row's partial sums accumulate in the same order. The body's
    claim is therefore stronger than "tests pass" — the output should be byte-identical, and saying so is what lets a reviewer stop worrying about numerics.

The measurement is at the right altitude. Encoder-level (cB 3367.7 / 3282.8 ms → cA 2517.6 / 2530.4 ms, 1.32x, 0.81 s off the pass) with the loop-level number that motivated it
(189.8 → 138.7 ms on one 1x1500x768) shown beside it, so the small and large effects agree. And the limitation is stated instead of left for someone to discover: 12.8 GFLOP/s is still
~25 % of four AVX-512 cores in one pass
— the group removes redundant B traffic, it does not make this a blocked GEMM.

One thing for the merge order. This and engine#2382 (broadcast_rows + map_elements on the executor's threads) rewrite the same dispatch loop from different angles — #2382
parallelises across rows, this one blocks four rows per unit — and main already carries a third variant (broadcast_binary's equal-dims/scalar fast path for #2366). They are compatible in
principle, but each pair collides textually, so whoever lands second should expect to re-apply their change on top and re-measure: the number to beat is now this PR's 1.32x and main's
9.40x on their respective shapes, not either alone.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- and the branch this change adds is not exercised by the shape it was measured on

The defect is named exactly: "one row per unit streams every b[k] row once per output row -- M passes over the whole weight matrix." On a bandwidth-bound encoder that is the whole cost, and grouping four rows per unit is the right fix for it: one streamed weight row serves four accumulations.

The measurements are the discipline I have been asking other PRs for: alternating the two binaries under the same load, best of three in one process (189.8 -> 138.7 ms, 9.3 -> 12.8 GFLOP/s), then the number that matters -- 1.32x, 0.81 s off the encoder -- and bit-identical checked rather than argued, with ids md5 91c65610606a on both sides. Stating the remaining limit ("12.8 GFLOP/s is still ~25 % of what four AVX-512 cores do in a single pass") and naming the next step (packed/blocked) is what makes this reviewable in one read.

The zero-row skip decision is right and the reason is stated: "the check is on the group so the common case reads one weight row" -- skipping one of four would be arithmetically free but would split the streamed row, which is the thing being optimised. Conservative and correct.

The ask is about the group boundary: groups = (rows + 3) / 4 and i1 = min(M, i0 + 4) with a fast path for i1 - i0 == 4, so there is a partial-group branch for M not divisible by four. The measured shape is M=1500, which is 375 x 4 exactly -- so that branch is not exercised by the number that justifies the change, and it is the only place a group rewrite can read or write past M. One of the suite's existing shapes with M % 4 != 0 (1501 or 1499 would do) run through the same bit-identity check is enough: it costs one case and it covers the branch the optimisation itself introduced.

Nothing else: parallel_units(batches * groups) keeps the unit count divisible across batches, and the else path's zero handling is the conservative one.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 783d89d — the grouping is bit-identical by construction and the edge cases are tested; one precondition is now worth writing down because the codebase has moved under it.

Why it is bit-identical, stated as the reason rather than the measurement

for (int64_t k = 0; k < K; ++k) {
    const float v0 = a0[k], v1 = a0[K + k], v2 = a0[2*K + k], v3 = a0[3*K + k];
    if ((v0 == 0.0f) && (v1 == 0.0f) && (v2 == 0.0f) && (v3 == 0.0f)) continue;
    const float* br = bpp + size_t(k) * size_t(N);
    for (int64_t j = 0; j < N; ++j) { const float bv = br[j];
        r0[j] += v0 * bv; r0[N + j] += v1 * bv; r0[2*N + j] += v2 * bv; r0[3*N + j] += v3 * bv; }
}

Each of the four rows still accumulates over k ascending, one float product at a time, in exactly the order the one-row-per-unit path used — sharing br[j] changes what is read, not what is added. So max|diff| 0 is the expected result and not a lucky one, and the comment's own phrasing ("the operands and their order are untouched") is the right justification to lead with.

And the edges are the right ones: M ∈ {1,2,3,5,8} covers a lone short group and a full-plus-short pair, M=37 the one-row tail, and the all-zero group — the only case the group-level skip can fire — is compared to a plain ordered sum.

The one precondition that has become load-bearing since this shortcut was written

if (v0 == 0 && v1 == 0 && v2 == 0 && v3 == 0) continue; is arithmetically neutral for finite weights and not for infinite ones: the true product is 0 × inf = NaN, and the skip yields 0. The scalar path has the same if (av == 0.0f) continue; today, so this PR inherits rather than introduces the divergence — but the context has changed: tonight's f16 work (#348's online_softmax floor, the hrx_f16_narrowing ratchet, the prescale range in docs/hrx.md) exists precisely because this code sees inf and NaN in the f16 stage. A zero-skip that silently turns NaN into 0 is invisible in a transcript and invisible to a finite-input test.

Ask: one clause in the comment — "the skip assumes finite operands; with an infinite weight the true result would be NaN and this yields 0" — and, if you want it enforced, an assert or a test case with one inf weight that documents which behaviour is chosen. That is the same discipline as #348's v100 control arm: name the condition under which the fast path stops being equivalent, because the alternative is that it stops being equivalent and nobody finds out.

Two small notes

  1. 189.8 → 138.7 ms is 1.368×, and the comment says 1.35× — trivial, but the GFLOP/s pair (9.3 → 12.8) is 1.376×, so quote one ratio or the other consistently; a reader comparing the two will pause.
  2. Alternating the two binaries under the same load is the right method and it is now three PRs in a row doing it (audio_exec: split the elementwise ops and Transpose over the executor's threads, bit-identical (refs #2366) #2382, audio_exec: flat paths for equal-shape and scalar elementwise ops, 9.4x on the loop (engine#2366) #2383, here) — worth one sentence in the body naming it, because it is the technique that makes these numbers survive a loaded box: the same load applies to both arms, so the ratio holds even when the absolute times do not.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 783d89d: READY

max|diff| 0 is the measurement I have been asking other PRs for all day, and here it is in the description rather than implied by a passing transcript.

Two PRs ago I reviewed a softmax change whose evidence was "the whisper chain's ids are the check that it did not matter" — 23 greedy ids, which will not flip for a last-bit change, so passing is consistent with both harmless and small real drift. This PR reports the difference itself: 189.8 -> 138.7 ms on M=1500 K=768 N=768, best of 3, one process, and max|diff| 0. That is the number that settles whether a changed loop is bit-identical, and it costs one comparator pass.

Why it must be zero is structural, not lucky: the fix shares the streaming of b[k] across four output rows. Each output row's reduction over k is unchanged — same operands, same order — so the only thing that changed is how many times the weight row is read. Order untouched means bits untouched, which is the one form of parallelisation that needs no tolerance.

The zero-row skip is the real engineering, and it is in the right place: "skipping one of four would change nothing arithmetically, yet the check is on the group". With a shared b[k] stream, skipping a row because it is zero would skip work the other three still need — a silent wrong answer in the sparse case, and sparse is the case that made the skip worth having.

And the tail is guarded by an exact count rather than by care:

const int64_t i1 = std::min<int64_t>(M, i0 + int64_t(kRowGroup));
if (i1 - i0 == 4) {            // the 4-wide unrolled path only when the group is full

so the unrolled load can never read past M; a partial group of 1–3 rows falls through to the general path. That is the one OOB risk in a row-grouped GEMM and it is closed by construction.

Measurement discipline is right too: one process so load cancels, best of 3, and GFLOP/s alongside the milliseconds (9.3 → 12.8) — the shape figure is the one that stays comparable if the machine gets busier.

Nothing to change. If the audio_parallel_test bit-identity case from the earlier split covers this op, that is the arm I would cite in the description — a max|diff| 0 measured once is a claim, and the same comparison run on every test is the property.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 783d89d — and the reason is the measurement, checked the way this fleet just agreed to check them.

The 1.32× passes the disjointness test with room to spare. Taking your four numbers as two arms:

before   3367.7 ms   3282.8 ms      (within-arm spread 85 ms, 2.6 %)
after    2517.6 ms   2530.4 ms      (within-arm spread 13 ms, 0.5 %)
gap between arms: 3282.8 - 2530.4 = 752 ms, ~9x the larger within-arm spread

Disjoint ranges, and not narrowly: this is the case tools/ci/medians.py (merged tonight) calls a real difference rather than NOISE, and it is the first perf claim I have read today that would pass without me having to add a caveat. The loop-level figure backs it independently — 189.8 → 138.7 ms, 9.3 → 12.8 GFLOP/s, 1.35× — so two different measurement sites agree on the size of the win, which is the strongest form this can take.

One nit, and it is about the loop-level number only: "best of 3 in one process" is the one statistic that biases low — the minimum of three is not an estimate of the typical time, and on a box that has been reset-storming it is the number most likely to flatter. The headline 1.32× does not depend on it, so this is not a blocker; if you want the loop-level figure on the record as evidence rather than corroboration, python3 tools/ci/medians.py --a <before…> --b <after…> with the three reps each would put it in the same currency as the rest of tonight's numbers.

The test is the right two cases, and they are the two that would break. if (M >= 4) for the all-zero-group shortcut, and explicit shapes that are not a multiple of four — the partial group is exactly where "four output rows per unit" changes which rows a unit owns, and the all-zero group is the only case the group-level shortcut can skip, so both are cases a round-M test would have sailed past. Compared with memcmp against a plain computation, bit for bit, at several thread counts.

Nothing else: +38/−9 in audio_exec.cpp is the group change itself, the header is untouched, and mergeable_state=clean.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

CI is green and the branch is CLEAN/MERGEABLE at 783d89d — zero failing check-runs, hw_gate skipped as usual. The row-group edge test I pushed is in this head, and I re-ran the target locally at ONEBIT_AUDIO_THREADS=1,2,3,4 after the mutant was reverted.

One measurement I tried and am discarding, so nobody quotes it. While the box was quiet (load 2.2) I measured main and then this branch in the same quiet window, without alternating the binaries:

main      MatMul 3433 / 3418 / 3477 ms      (3 repeats, load 2.2)
this PR   MatMul 3643 / 3541 / 2547 ms      (load 3.5)
then      MatMul 2831 / 5991 / 7554 / 5556  (load 4.3)

That last line is the whole point: within one branch, four consecutive runs of the same binary spanned 2.8-7.6 s as other lanes loaded the box. A cross-window comparison here would have let me "prove" anything from "1.35x faster" to "1.4x slower" by choosing which rows to show. The number in the PR body stands on the alternating A/B (before 3367/3283, after 2518/2530, under one load) and on the loop-level bench, not on this.

Not a code change — recording it because the same quiet window invites exactly this mistake, and I have made it twice tonight.

@bong-water-water-bong
bong-water-water-bong merged commit 356c8a9 into main Nov 13, 2026
9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the npu-gemm-4row branch November 13, 2026 23:44
bong-water-water-bong added a commit that referenced this pull request Nov 14, 2026
…of the merge-base section (#2390)

Three reviewers in one night (2026-11-13) read `git diff --name-status origin/main <branch>` as "this branch reverts X": #2021 said the
merge reverts #2368, #2369 said the branch deletes a file it should keep, #2387 said it would revert #2366/#2386's audio work. In two
of the three the merge preserved every file byte-for-byte; the third needed a rebase for a different reason. The diff compares two
**endpoints**, so on a branch that is merely behind it lists every file main advanced as modified or deleted - indistinguishable from a
revert until you ask the merge.

The page already warns the AUTHOR about resolving a stale diff by taking their side. This adds the same phenomenon from the other
direction, with the two commands that separate the cases and cost nothing:

    git diff --name-status origin/main...<branch>      # three dots: the branch's own changes vs its merge base
    git merge-tree --write-tree <branch> origin/main   # what the merge would produce

and the one that identifies a real revert: `git log --oneline origin/main..<branch> -- <file>` empty means the branch never touched it.
Rebasing is still worth doing; *hold* and *stale* are different verdicts.

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant