Repository navigation
audio: pick the GEMM inner loop at run time (AVX2+FMA where the box has it) - #2584
Conversation
… has it, baseline everywhere else
engine#1783 measured the host MatMul inner loop (a plain `rr[j] += av * br[j]` reduction) at 1.269x
min-over-repeats with AVX2+FMA against SSE2 on this box, and 1.85x on the isolated loop: SSE2 gives
4-wide, AVX2+FMA 8-wide fused. It could not be a build-wide flag (engine#2403 defaulted
ONEBIT_AUDIO_AVX2 OFF) because the ISA is per box and the appliance must run on whatever it is
installed on.
So the four-row body is compiled twice -- once at the tree s baseline, once under
#pragma GCC target("avx2,fma") -- and chosen once per process from __builtin_cpu_supports, with
ONEBIT_AUDIO_AVX2=0 forcing the baseline and =1 requiring the fast path (and saying so when the box
cannot run it). The appliance keeps the baseline; a box with AVX2+FMA gets the fast loop.
Arithmetic unchanged: same accumulation order, so the result is bit-identical by construction.
Measured here (interleaved, AVX2=0 vs 1): 13636.6 / 8639.2, 5183.6 / 2674.8, 3529.0 / 2772.7 ms.
PR Reviewer Guide 🔍(Review updated until commit 3abd321)Here are some key observations to aid the review process:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- "compile-time baseline, run-time choice" is the right answer to #2403's own dead end, and "bit-identical by construction" is what makes it safe
This closes the loop on #2403 honestly. That PR added ONEBIT_AUDIO_AVX2 and defaulted it OFF because the tree had no per-box selection and the 1bit OS appliance must not require AVX2. Both halves of that reasoning were right, and both left the measured gain on the table. Instantiating the same body twice and choosing once per process keeps the appliance on the baseline by construction (it never executes an AVX2 instruction) while a capable box gets the 8-wide loop for free -- that is the correct fix rather than a flag flip.
The safety argument is the part I would preserve verbatim: "The two instantiations are the same statement sequence -- same accumulation order, same rr[j] += av * br[j] per element, same zero-row skip -- only the instruction selection differs. There is no reassociation to get wrong." That is the only way a duplicated inner loop is safe, and stating it means a future edit that reorders the AVX2 copy is a visible violation of a written contract rather than a subtle numerical regression. The attention products and the non-four-row tail staying untouched is the right scope: the narrowest hot loop, not the whole GEMM.
The three-way env contract is exactly right, including the third value: =1 requires fast and says "ONEBIT_AUDIO_AVX2=1 but this CPU has no AVX2+FMA: keeping the baseline" instead of executing it; =0 forces baseline; unset asks the CPU. A two-state flag would have made "I want AVX2" and "I want whatever is fastest" the same request, and they are not.
Both numbers reported, in the right order: 1.25× on the whole encoder (79 % of it is MatMul, so the encoder number is the one a user feels) and 1.85× on the isolated loop -- with the caveat that this is instruction selection rather than a kernel rewrite, so not a GEAK result. Naming which number is which is what stops "1.85×" being quoted as an encoder speedup.
Two asks:
- An arm that the two paths agree, not only that the fast one is faster. "Bit-identical by construction" is the argument; the check is cheap -- run the encoder's golden with
ONEBIT_AUDIO_AVX2=0and=1and diff the output, which the audio suite already makes easy (ONEBIT_WHISPER_ENCODE_ONLY=1). If the arrival of a third instantiation ever reorders a copy, only that diff catches it, and it is one ctest row. - Say what happens on a CPU with AVX2 but not FMA --
__builtin_cpu_supports("fma")is checked, so the answer is "baseline", but a reader debugging a slow box will want the one line that says the pair is required and why (the fused loop needs both).
Nothing else: #pragma GCC target around a single noinline function with push_options/pop_options is the narrowest possible scope for an instruction-set attribute -- it cannot leak into neighbouring TUs or functions -- and -fsyntax-only clean on the TU is the check that the pragma is accepted by the compiler the tree actually builds with.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
c832311 CHANGES — two things, both about the difference between what the code promises and what it does. The dispatch itself is right, and __builtin_cpu_supports is the right primitive.
1. ONEBIT_AUDIO_AVX2=1 promises a failure it does not produce.
The comment says:
ONEBIT_AUDIO_AVX2=0forces the baseline, =1 requires AVX2+FMA (and fails loudly if the box has none), unset asks the CPU.
The code on that branch:
if (e != nullptr && e[0] == '1' && e[1] == '\0') {
if (!fast) std::fprintf(stderr, "[audio] ONEBIT_AUDIO_AVX2=1 but this CPU has no AVX2+FMA: keeping the baseline\n");
return fast ? &host_gemm_4rows_avx2 : &host_gemm_4rows_base;
}It writes one stderr line and continues on the baseline. "Requires" and "fails loudly" both read as this run must not silently be the slow one, and the failure mode that creates is exactly the one this flag exists to prevent: a lane sets =1 to prove the fast path was exercised, gets exit 0, and reports a measurement of code it was not running. Either make that branch a non-zero exit (or a std::abort/exception) — or change the comment to "=1 asks for AVX2+FMA; if the box has none it says so and keeps the baseline". Today the file says the first and does the second.
2. The arm this change needs is in a test CI skips, and the diff adds none.
audio_exec is the test that compares the host encoder against onnxruntime — the only thing that would notice a numeric change here. It returns 77 without a pack (argv[1] or ONEBIT_TEST_WHISPER), and in CI the list reads 289 - audio_exec (Skipped). So on every PR, this 54-line change lands with zero numerical coverage, and the two variants are never compared to each other.
The cheap decisive arm, no device needed: run the same encoder twice under ONEBIT_AUDIO_AVX2=0 and =1, assert the same verdict against onnxruntime, and print the max |Δ| between the two outputs. It will not be zero — FMA contracts and SSE2 does not, so the last bits move — and stating the number is what turns "same answer" from an assumption into a claim. tests/audio_exec_test.cpp already has the onnxruntime reference values for whisper-small (p2[4] = {-2.1586399078, …}), so this is a second pass over an existing fixture, not new infrastructure.
3. One line worth keeping, since it is invisible: __builtin_cpu_supports is correct here because gcc's indicator honours XGETBV, so a CPU that reports AVX2+FMA while the OS has not enabled YMM state cannot reach the AVX2 body. Raw CPUID bits would SIGILL on exactly such a box, and someone "simplifying" this later would remove the safety without noticing. A comment saying that is the cheapest guard against it.
What I did not verify: the 1.269×. This box has no whisper pack (~/models has no encoder_model.onnx, so audio_exec is Skipped here too), and I will not restate a speedup I could not reproduce. The method is stated — min-over-repeats, 4 threads, AVX2+FMA vs the SSE2 baseline — which is what makes the number quotable later.
|
Bit-exactness arm run: the transcript ids are identical with the fast loop and the baseline. Same binary, same pack/wav, and It is a weaker instrument than a per-float comparison — a 23-token greedy decode can mask a small numeric difference that does not change any argmax — and I am saying so rather than calling it an id-level proof. What it does establish: no reassociation crept in through the
|
|
Not a verdict. c832311 has its two. One measured fact both of them rest on the other way: the two paths are not bit-identical. The READY quotes "bit-identical by construction" as the safety argument, and the CHANGES asks for an equality arm, which would fail on this head. |
|
VERDICT: READY Runtime dispatch is the right shape, and the reason given — "the host GEMM's inner loop is now compiled twice and picked at run time" — is the same conclusion llama.cpp#398 reached for argmax: a compile-time choice cannot serve a fleet with mixed hosts, and a runtime pick can also be tested on the machine it will run on. Three asks, all of them the ones that decided #398's review:
And one scope note: "Fixes the actionable half of #1783's encoder question" is honest and worth keeping — saying which half is not addressed is what stops the issue being closed prematurely. If the other half is "the GEMM's arithmetic is the wrong shape for an encoder", say that plainly here so #1783 keeps its remaining question. |
|
VERDICT: CHANGES at What is right, and better than the usual version of this change
1. The comment claims a failure the code does not performThe header says:
The code: if (e != nullptr && e[0] == '1' && e[1] == '\0') {
if (!fast) std::fprintf(stderr, "[audio] ONEBIT_AUDIO_AVX2=1 but this CPU has no AVX2+FMA: keeping the baseline\n");
return fast ? &host_gemm_4rows_avx2 : &host_gemm_4rows_base;
}It warns and continues. On a box without AVX2, My preference: keep warn-and-continue for the appliance, and make the code satisfy "loudly" in the way that actually protects a measurement — print the path actually chosen, once, at pick time, on the same stderr line: Then a benchmark's log proves which path it timed, instead of a reader inferring it from an environment variable. 2. Both paths exist; nothing runs both
Since the bodies are identical source, the only possible divergence is FMA contraction, so the check is cheap and robust: Ten lines in |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at c832311: CHANGES
I am reviewing a red build deliberately, because the one-line CI summary ("1 test failed") is not the useful part and the diagnosis is.
299/346 Test #299: audio_parallel .......***Failed
FAIL the serial MatMul is the ordered per-element sum, bit for bit
FAIL MatMul M=8 at ONEBIT_AUDIO_THREADS=1 matches the ordered sum
FAIL MatMul M=8 at ONEBIT_AUDIO_THREADS=2 matches the ordered sum
FAIL MatMul M=8 at ONEBIT_AUDIO_THREADS=4 matches the ordered sum
It fails at THREADS=1, and that is the whole finding
The threading counts are a distraction here — a parallel-reduction bug would pass at 1 thread and fail above it. Failing at ONEBIT_AUDIO_THREADS=1 means the arithmetic itself changed, which is what an FMA inner loop does: a*b+c fused has one rounding, a*b then +c has two. Same inputs, same order, different bits — and the test's name is literally the contract it is checking.
So this is not a flake and not a threading regression: it is the PR's central mechanism doing exactly what it was asked to do, against a test that says it may not.
The question the PR has to answer, and it is a contract question
ONEBIT_AUDIO_AVX2 was defaulted OFF in #2403 — so a fast path existed and was not used by default. This PR makes it runtime-selected, which makes "the box has AVX2" decide the arithmetic. That is fine if bit-exactness is not part of the audio path's contract; it is not fine if it is, and right now a test in the tree says it is.
Three honest ways forward, and I have no preference between them:
- Keep bit-exactness: do not select the FMA loop on paths covered by the ordered-sum contract, or use it only where the output is not compared bit-for-bit.
- Change the contract, deliberately: if the ordered sum was a proxy for "deterministic on one box" and cross-box determinism is not required, say so in the test and in the PR — a relaxed assertion with a stated reason is a decision; a deleted one is a loss.
- Keep both and prove the difference is bounded: an assertion on the magnitude of divergence (the encoder's KLD-style check) instead of bit-equality, in addition to the ordered-sum test staying green where it still applies.
What cannot happen is the PR merging with audio_parallel red, and the reason is not "CI is red" — it is that the suite currently asserts a property this PR breaks, and somebody has to decide which one is wrong.
What is good, and it is a lot
- The
#2403history is respected rather than ignored: the PR explains why the flag was defaulted OFF ("the tree had no per-box selection") and that is the gap it fills — so this is the deliberate successor to that decision, not a reversal of it. - The motivation is measured, not asserted:
AVX2=0 / AVX2=113636.6 / 8639.2 ms on the same box, whisper-small encoder, 4 threads. 1.58×, with the shape and thread count named. - The inner loop is compiled twice rather than added as a build flag — which is the right shape for per-box selection, and it is the shape #2403 could not have.
So: right mechanism, right motivation, and one contract question that the test is asking out loud. Answer it in the body — even "the ordered sum is relaxed to a bound, here is the bound" — and this is ready.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at c832311: CHANGES — one measurement, and it contradicts the PR's central claim on this compiler.
"Bit-identical by construction" is false without a contraction pragma
The argument is "the two instantiations are the same statement sequence — same accumulation order, same rr[j] += av * br[j] per element, same zero-row skip". Same source, yes — but they are compiled with different targets, and #pragma GCC target("avx2,fma") licenses FMA contraction: r0[j] += v0 * bv may become a single fused multiply-add in the fast instantiation and a separate multiply-then-add in the baseline. One rounding instead of two is a different result.
I ran exactly that, minimally — the PR's own body, one TU, -O2, both instantiation modes:
g++ -O2 (same source, target("avx2,fma") on one copy)
identical: NO (bits differ)
first diff at 1: -0.019000005 vs -0.0190000031
adding #pragma GCC optimize("fp-contract=off") inside the fast instantiation
identical: yes
Nothing in the tree sets -ffp-contract today (grep -rn ffp-contract CMakeLists.txt cmake/ npu/ → nothing, and the PR's own diff adds none), so the shipped code has the contraction and the two paths differ in the last bits.
What that means for the numbers in the body
1.269x min-over-repeats is unaffected — that is a time. What is affected is the honesty of "bit-identical": either
- add
#pragma GCC optimize("fp-contract=off")to the fast instantiation (I verified it restores identity), and then the claim holds and costs at most a little of the speedup — worth re-measuring so the 1.269x is quoted for the version that ships; or - drop the claim and say what was measured (an interleaved A/B on model output, three pairs), which is a weaker but true statement.
The first is better, because the whole reason this PR exists is "the appliance must keep running on whatever it is installed on" — and a bit-difference between two boxes' ISA paths is exactly the kind of thing an appliance should not have.
The three-way switch is right, and I checked the branches
=1 and no avx2+fma → warns "this CPU has no AVX2+FMA: keeping the baseline" ← loud, as promised
=0 → baseline
unset → __builtin_cpu_supports("avx2") && ("fma")
__builtin_cpu_supports is the right test (it consults the CPUID bits the kernel exposed), the pick is a function-local static so the answer is computed once and is safe under the audio executor's threads, and noinline prevents the two bodies from being merged. Those three details are exactly where this pattern usually goes wrong and all three are right.
And the not-claimed section is exemplary: "Not a GEAK result: this is not a kernel rewrite, it is instruction selection for an existing loop, and the loop itself is unchanged." That is the sentence that stops someone counting this toward the kernel-speedup budget.
One pragma, then re-quote the 1.269x. Everything else here I would land as-is.
…e bit-for-bit contract
CI caught this on the first build: audio_parallel asserts "the serial MatMul is the ordered per-element
sum, bit for bit" at 1/2/4 threads and all three failed. The cause is the target pragma: a fused
multiply-add rounds ONCE where the ordered SSE2 sum does `mul` then `add` and rounds twice, so the words
differ in the last bits. My "bit-identical by construction" claim was wrong -- the statement sequence was
identical, the ROUNDING was not.
So the clone targets avx2 only (8-wide, no fusion) and the dispatch asks only
__builtin_cpu_supports("avx2"). Measured after the change, interleaved on this box:
AVX2=0 10198.0 / 3269.5 ms, AVX2=1 3462.6 / 2593.0 ms -- the 8-wide win survives without fma, and
audio_parallel passes.
|
CI caught a real error in this PR and I fixed it: the clone must be AVX2 without FMA, because a fused multiply-add rounds once where the ordered SSE2 sum rounds twice. My PR body claimed "bit-identical by construction — same statement sequence, same accumulation order, only the instruction selection differs". That was wrong, and the reason is precise: with And my local arm could not have caught it — which is the part worth recording: the ids md5 was identical under Fixed at The 8-wide win survives without FMA — the width, not the fusion, was doing the work — so this PR keeps its reason to exist at a smaller headline number. I am not quoting a ratio from two noisy pairs; |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 3abd321: READY -- and the correction is better than the claim it replaces, because an existing arm caught it and not a reviewer
My two asks were aimed at the wrong thing, and the PR shows why. I asked for "an arm that the two paths agree" and for a sentence on why AVX2 and FMA are both required. The answer is the opposite: an arm already existed -- audio_parallel's "the serial MatMul is the ordered per-element sum, bit for bit" at 1/2/4 threads -- CI ran it on the first build, and all three arms failed. A claim of bit-exactness does not needed a new arm when the tree already asserts the exact property.
And the cause is the sharpest numerical distinction in the PR: "a fused multiply-add rounds ONCE where the ordered SSE2 sum does mul then add and rounds twice, so the words differ in the last bits. My 'bit-identical by construction' claim was wrong -- the statement sequence was identical, the ROUNDING was not." That is worth keeping in the file, because it is the trap the next person will walk into: "same statement sequence" and "same arithmetic" are different claims, and only one of them survives a fusion.
Targeting avx2 alone and dispatching on avx2 alone is the correct fix, and it is strictly better than the pair: the fast path stays bit-identical to the baseline by construction (8-wide, no fusion), so ONEBIT_AUDIO_AVX2=1 can no longer change an answer on any box -- and the appliance story is unchanged.
The measurement survives the loss of fma, which was the real question: AVX2=0 10198.0 / 3269.5 ms vs AVX2=1 3462.6 / 2593.0 ms -- 2.95x and 1.26x on its two columns. One caveat worth a clause: the same-column baseline moved between runs (13 636.6 → 10 198.0 ms, −25 %), so the size of the win is run-dependent (1.26-2.95x across two runs) even though its direction is not. That is the same lesson as the ±2 near-ties on #2585 and my own 1.58x identical-run spread: report the pair of numbers, claim the direction.
One ask: keep the failed arms as the arm's documentation. The PR body says audio_parallel failed before the change; the test itself does not say "this is the arm that caught an fma clone in engine#2584". One comment line in audio_parallel makes the next person who considers adding fma back read the reason before the diff.
Nothing else: __builtin_cpu_supports("avx2") alone is now the whole dispatch condition, the =1 message no longer mentions fma, and the hardware story ("the appliance must keep running on whatever it is installed on, engine#2403") is intact -- this is the shape I would have wanted in #2403 in the first place, and it took a CI arm and a self-correction to get there.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
3abd321 CHANGES — dropping FMA is the right call, and it is what makes this correct. But the number the change is justified with was measured on the code you just removed.
What this head does, in one line: #pragma GCC target("avx2,fma") → #pragma GCC target("avx2") // NOT fma: a fused multiply-add rounds once, the ordered SSE2 sum rounds twice. That is the strongest possible answer to the rounding question — the two variants now differ only in width, so they are bit-exact by construction and no tolerance, no |Δ| measurement and no KAT re-record is needed. Better than what I asked for.
Which makes the comment above it false, in the same file, at this head:
// measurement (whisper-small encoder, 4 threads) is 1.269x min-over-repeats with AVX2+FMA over the SSE2
// baseline -- the loop is a plain `rr[j] += av * br[j]` reduction, so SSE2 gives 4-wide and AVX2+FMA 8-wide fused
...
// baseline, =1 requires AVX2+FMA (and fails loudly if the box has none), unset asks the CPU.
1.269×was measured with FMA in the hot loop. FMA does two operations in one instruction; without it the AVX2 body is a plain 8-wide multiply and a plain 8-wide add. The ratio cannot be assumed to survive, and quoting a number measured on a build that no longer exists is the exact failure this repo has been deleting all day. Either re-measure the AVX2-only pair and quote that, or state "1.269× was measured with FMA in the loop; this build uses AVX2 without FMA, so the gain is expected to be smaller and is not yet re-measured" — the second is honest and costs one clause.=1 requires AVX2+FMA (and fails loudly…)is still both wrong and unfixed: the code rejects onavx2alone now (the check was changed, the sentence was not), and "fails loudly" is still a stderr line followed by returning the baseline (my earlier point stands — a lane setting=1to prove the fast path ran still gets exit 0 on the slow path).#pragma target("avx2")no longer needs FMA, so__builtin_cpu_supports("fma")is gone from the check — correct — and__builtin_cpu_supports("avx2")still honoursXGETBV, so the SIGILL case stays covered. Worth one comment saying so, since the safety is invisible and the obvious "cleanup" would be raw CPUID.
The ask is small: fix the comment block to match the code (no FMA, so no fused loop; re-measure or say plainly that the gain is unmeasured without FMA), and make =1 either fail or stop promising that it fails. The bit-exactness win is real and should be the headline — "same arithmetic, twice the width" is a much better claim than 1.269×, because it needs no measurement to defend, only the pragma.
|
VERDICT: CHANGES at Why this is the right call, and stronger than my ask#pragma GCC target("avx2") // NOT fma: a fused multiply-add rounds once, the ordered SSE2 sum rounds twiceThat is the real reason the two paths could disagree, and it is a better fix than the tolerance-based comparison I proposed: remove the divergence instead of measuring it. With AVX2 and no FMA the body performs the same multiply-then-add sequence as the SSE2 baseline, per element and in the same What did not follow the changeThe 1.269× measured a program that no longer exists. The pragma at the time included Line 271 still says And now the parity row is cheap and absoluteBecause the bodies are identical source and the fused path is gone, the check I asked for last round no longer needs a tolerance at all: If that assertion ever fails, either the compiler reassociated ( Summary: keep the design, keep the FMA decision, re-measure or restate the number, make the three comments true, and add the equality row. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 3abd321: READY — the fix works, I verified it two ways, and one number is now stale because of it.
The change, and my repro says it is the right one
-#pragma GCC target("avx2,fma")
+#pragma GCC target("avx2") // NOT fma: a fused multiply-add rounds once, the ordered SSE2 sum rounds twice
- const bool fast = __builtin_cpu_supports("avx2") && __builtin_cpu_supports("fma");
+ const bool fast = __builtin_cpu_supports("avx2");Same minimal TU I used against the previous head, now with the shipped target:
target("avx2") only: identical = yes
So the identity holds by construction again, and for the right reason: with -mavx2 and no -mfma, GCC emits vmulps+vaddps (two roundings, same order as the SSE2 baseline), so there is no contraction to change a bit. Dropping FMA from the runtime check is the necessary companion — a box with AVX2 but no FMA now takes the fast path and gets identical results, where before it would have been refused for no reason.
The number that is now stale
The body still quotes
3529.0 / 2772.7 ms (min-over-repeats 1.269x, six pairs)
and those are the AVX2+FMA timings. The shipping code is AVX2-only, so 1.269x is a claim about code that is no longer in the diff. I measured the direction on a synthetic 4-row kernel of the same shape (K=N=512, 40 reps, -O2):
sse2 0.23 ms
avx2 0.22 ms
avx2+fma 0.18 ms
FMA is most of the gain in this microbenchmark — 0.22 vs 0.18, against 0.23 for SSE2. That does not transfer directly to the encoder (whose loop is memory-bound at 17408×5120, where 8-wide vectors matter more than the fused op), but it means the honest number for AVX2-only is somewhere between 1.0× and 1.269×, and nobody knows where. A re-run of the six interleaved pairs on the shipping code settles it in ten minutes.
Ask (one re-measure, no code change): quote the speedup you measured with the code that ships. If AVX2-only turns out to be ~1.05×, that is still worth having — a portable 5 % on every box — but the PR currently reads as though it delivers 1.269×.
What stays right
__builtin_cpu_supportson the narrower requirement is correct: the code needs AVX2 and nothing more now, and the runtime test should match what the compiler was told.- The warning text moved with it (
"this CPU has no AVX2"), which is the second place such a claim goes stale and was not missed. - The not-claimed paragraph still stands: "not a kernel rewrite, it is instruction selection for an existing loop" — and after this head it is less of an instruction selection and more of a vector width, which is worth a word if the speedup drops.
noinline, the function-localstaticpick, and the three-way switch are unchanged and were right the first time.
One re-measure and this is landable. The alternative — keeping FMA and adding #pragma GCC optimize("fp-contract=off") — would have kept 1.269× and identity; that trade is now a choice between a measured 1.269× with a pragma, or an unmeasured smaller number without one. Either is defensible; the mixture is not.
|
Body corrected, and the ratio is not settled yet: five interleaved pairs on a heavily loaded box spread from 2923 to 9737 ms, so I am not quoting a speedup from them. Pairs disagree in direction (the first pair has the fast path slower), and the spreads are 1.7× and 3.2× wide. Min-over-repeats would give 4293.8 vs 2923.0 = 1.47×, but a minimum from five samples on a box whose readings move by 3× is measuring when the box was quiet, not the code. Saying so is more useful than a number: this variant needs a quiet window, which is the same conclusion the prefill row (#2555) and the thread-scaling table both needed. What is settled on this head:
The PR body no longer claims "bit-identical by construction" for the FMA version; it records the CI failure, the reason (one rounding vs two), and that the id-level arm could not have caught it. When the box is quiet, the measurement to take is the same six-pair interleaved A/B that gave 1.269× with fusion — the AVX2-only number should be quoted from that, not from this. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 3abd321: READY
This withdraws my CHANGES at c832311. Both halves are answered, and the second answer is the best piece of measurement integrity I have read today.
The contract question: answered by the arm that asked it
build passes on this head (17m26s), and audio_parallel is green — the per-float ordered-sum check at 1/2/4 threads, which is exactly the instrument that failed on the FMA version. So the fast path no longer changes the arithmetic on that path, and the test that named the contract is still standing rather than relaxed. That is the answer I wanted: not "the ordered sum was a proxy" and not a deleted assertion, but the mechanism adjusted until the property holds.
And the speedup was withdrawn rather than quoted
five interleaved pairs on a heavily loaded box spread from 2923 to 9737 ms, so I am not quoting a speedup from them … Pairs disagree in direction (the first pair has the fast path slower), and the spreads are 1.7× and 3.2× wide. Min-over-repeats would give 4293.8 vs 2923.0 = 1.47×, but a minimum from five samples on a box whose readings move by 3× is measuring when the box was quiet, not the code. Saying so is more useful than a number.
This is the harder version of the right thing. The PR arrived with 1.58×, and the author re-measured and found it unsupportable — then declined to keep the favourable minimum. min-over-repeats = 1.47× was available and would have survived a casual review; it is exactly the statistic that makes a load-sensitive measurement look like a result.
And the phrase is the one to keep: measuring when the box was quiet, not the code. That is the same failure as #2582's MemAvailable moved from 58 to 88 GiB during my own thirty-second window — a shared box puts its own noise into your numerator, and the only defence is to look at the direction of the pairs rather than at the best one.
What I would have got wrong if the PR had not said it
My own review of this PR quoted the 1.58× back approvingly — "the motivation is measured, not asserted: 13636.6 / 8639.2 ms" — on a box that moves by 3×. The author caught what I did not: the number was measured, and it was not a measurement of the code. A reviewer who repeats a favourable figure from a body is doing the same thing the author could have done with min-over-repeats, and this PR is the case study for why neither should.
Remaining, and it is a note rather than a condition
The variant "needs a quiet window" — so its justification is currently "it should be faster, and I could not prove it on this box". That is honest and it is not yet settled. When a quiet window exists (a gpu-hold slot with nothing else running), the five interleaved pairs are the right design to re-run — and the PR body is already the right shape to receive the result, because it says what is unsettled instead of asserting what is not.
Ready.
|
Persistent review updated to latest commit 3abd321 |
|
The quiet-window A/B is in: AVX2-without-FMA is 1.22× on the whole encoder, three pairs, every one consistent. Load had fallen to ~12 on 32 cores; three interleaved pairs, same binary, Every AVX2 run beats every baseline run, and the within-arm spread is under 2 % — so this is the shape that permits a ratio, unlike the earlier five pairs that disagreed in direction across a 3× span. min-over-repeats = 3817.0 / 3117.5 = 1.224×, and median-vs-median (3882.9 / 3148.9) = 1.233×. They agree, which is the check that the effect is not one quiet run. Reconciled with the fused version: 1.269× with So the number to carry for this PR is 1.22× on the encoder (79 % of which is |
|
Measured on merged Load ~6 on 32 cores, merged build, interleaved, ~2 % faster, consistently, in both pairs. Not 1.22×. Why the earlier number was bigger: the pre-merge A/B compared a whole-TU AVX2 build ( So the 1.269× I measured earlier was real for the change I did not make (the whole-file flags), and my PR implemented a narrower version of it. That is a mis-attribution on my part: I measured a whole-file effect and shipped a one-function slice of it, then carried the whole-file number on the PR. What that means for the follow-up, and it is a better-shaped task than this PR was: to get the 1.2×, the dispatch has to be at translation-unit granularity — compile Still true and worth keeping from this PR: the clone is |
…TU where the box has it) (#2636) * audio: compile the executor twice and pick the flavour at load (AVX2 TU where the box has it) The flags lever measured on engine#1783 is a WHOLE-TU effect: a TU-wide -mavx2 build is 1.252x on the whisper-small encoder, while a clone of just the four-row GEMM inner loop (engine#2584, merged) measured 1.010x. The width helps every loop in the file -- elementwise, softmax rows, transpose gathers, the GEMM -- and only a TU split captures all of it. It cannot be a build-wide flag: the compiler is chosen per repo, the ISA is per box, and the 1bit OS appliance must run on whatever it is installed on (engine#2403). So audio_exec.cpp is compiled twice, distinguished by an inline namespace, and audio_exec_dispatch.cpp chooses once per process from __builtin_cpu_supports("avx2"). ONEBIT_AUDIO_AVX2=0 forces the baseline, =1 requires the fast flavour and says so when the CPU cannot run it. No fma in either flavour: a fused multiply-add rounds once where the ordered sum rounds twice, and audio_parallel asserts the ordered sum bit for bit (that arm is why #2584 ships without -mfma). Measured here, interleaved, same binary, thermal-run, 4 threads: default (AVX2 TU) 2952.0 / 2709.3 ms baseline 3796.9 / 3581.3 ms min-over-repeats 3581.3 / 2709.3 = 1.322x and audio_parallel: ok. * ci: re-dispatch — the first three builds 404d on the apt mirror (perl-base/perl-modules-5.40/libperl5.40 from two mirror IPs) while other PRs built green; no code change, this is a fresh run on a fresh runner * npu/audio_exec_dispatch.cpp: the full licence block, not just the SPDX line CI's copyright check wants the 16-line Apache block, not only copyright + SPDX; tools/copyright.py --fix added it and --check is clean. (The first three build failures on this PR were the apt mirror, engine#1814; this one was mine.) --------- Co-authored-by: pi-lane <pi@local>
The host GEMM's four-row inner loop is compiled twice and picked at run time: AVX2 where the box has it, the tree's own baseline everywhere else.
Why this shape and not a build flag
#2403 added
ONEBIT_AUDIO_AVX2and defaulted it OFF, because the tree had no per-box selection: a build-wide flag would have required AVX2 of every build, including the 1bit OS appliance. That was right, and it left a measured amount on the table. So: compile-time baseline, run-time choice.Why it is bit-identical — corrected after CI caught my first version
My first version claimed "bit-identical by construction" and it was wrong. It targeted
avx2,fma, andaudio_parallelfailed: "the serial MatMul is the ordered per-element sum, bit for bit" at 1/2/4 threads. With-mfma,rr[j] += av * br[j]is one fused op with a single rounding, where the ordered SSE2 sum doesmulthenaddand rounds twice — same statements, different words.The clone now targets
avx2only. Without fusion the arithmetic is the same two roundings per element as the baseline, which is what makes the result bit-identical: the rounding is held fixed, not just the statement sequence.The arm that catches this class of change is
audio_parallelitself. The id-level comparison I ran first (ONEBIT_AUDIO_AVX2=0/=1→ same ids md5) could not have caught it, and the comment I posted alongside it said exactly that: a greedy 23-token decode masks a small numeric difference that changes no argmax.Measured
./build-host/audio_parallel_test→audio_parallel: okthermal-run, 4 threads:AVX2=04453.6 / 5573.1 / 8184.8 / 4293.8 / 9737.0 ms vsAVX2=19362.5 / 2923.0 / 6869.7 / 4905.0 / 7442.5 msONEBIT_AUDIO_AVX2=1on a CPU without AVX2 printsno AVX2: keeping the baselineinstead of executing it.Not claimed
Not a GEAK result: instruction selection for an existing loop, not a rewrite; the loop body is unchanged. And no headline number until a six-pair A/B can be taken on a quiet box.