Skip to content

audio: pick the GEMM inner loop at run time (AVX2+FMA where the box has it) - #2584

Merged
bong-water-water-bong merged 2 commits into
mainfrom
audio/gemm-runtime-dispatch
Nov 15, 2026
Merged

bong-water-water-bong merged 2 commits into
mainfrom
audio/gemm-runtime-dispatch

Conversation

@bong-water-water-bong

@bong-water-water-bong bong-water-water-bong commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

The host GEMM's four-row inner loop is compiled twice and picked at run time: AVX2 where the box has it, the tree's own baseline everywhere else.

Why this shape and not a build flag

#2403 added ONEBIT_AUDIO_AVX2 and defaulted it OFF, because the tree had no per-box selection: a build-wide flag would have required AVX2 of every build, including the 1bit OS appliance. That was right, and it left a measured amount on the table. So: compile-time baseline, run-time choice.

const bool fast = __builtin_cpu_supports("avx2");
ONEBIT_AUDIO_AVX2=1  -> require the fast path (and say so when the box cannot run it)
ONEBIT_AUDIO_AVX2=0  -> force the baseline
unset                -> ask the CPU

Why it is bit-identical — corrected after CI caught my first version

My first version claimed "bit-identical by construction" and it was wrong. It targeted avx2,fma, and audio_parallel failed: "the serial MatMul is the ordered per-element sum, bit for bit" at 1/2/4 threads. With -mfma, rr[j] += av * br[j] is one fused op with a single rounding, where the ordered SSE2 sum does mul then add and rounds twice — same statements, different words.

The clone now targets avx2 only. Without fusion the arithmetic is the same two roundings per element as the baseline, which is what makes the result bit-identical: the rounding is held fixed, not just the statement sequence.

The arm that catches this class of change is audio_parallel itself. The id-level comparison I ran first (ONEBIT_AUDIO_AVX2=0/=1 → same ids md5) could not have caught it, and the comment I posted alongside it said exactly that: a greedy 23-token decode masks a small numeric difference that changes no argmax.

Measured

  • ./build-host/audio_parallel_test → audio_parallel: ok
  • interleaved A/B, same binary, thermal-run, 4 threads: AVX2=0 4453.6 / 5573.1 / 8184.8 / 4293.8 / 9737.0 ms vs AVX2=1 9362.5 / 2923.0 / 6869.7 / 4905.0 / 7442.5 ms
  • those pairs disagree in direction and span 3×, so no ratio is quoted from them. A minimum from five samples on this box measures when the box was quiet, not the code. The ratio for this variant is owed from a quiet window (the 1.269× figure from earlier in the issue was measured with fusion and does not apply).
  • dispatch verified both ways, including the no-AVX2 case: ONEBIT_AUDIO_AVX2=1 on a CPU without AVX2 prints no AVX2: keeping the baseline instead of executing it.

Not claimed

Not a GEAK result: instruction selection for an existing loop, not a rewrite; the loop body is unchanged. And no headline number until a six-pair A/B can be taken on a quiet box.

… has it, baseline everywhere else

engine#1783 measured the host MatMul inner loop (a plain `rr[j] += av * br[j]` reduction) at 1.269x
min-over-repeats with AVX2+FMA against SSE2 on this box, and 1.85x on the isolated loop: SSE2 gives
4-wide, AVX2+FMA 8-wide fused. It could not be a build-wide flag (engine#2403 defaulted
ONEBIT_AUDIO_AVX2 OFF) because the ISA is per box and the appliance must run on whatever it is
installed on.

So the four-row body is compiled twice -- once at the tree s baseline, once under
#pragma GCC target("avx2,fma") -- and chosen once per process from __builtin_cpu_supports, with
ONEBIT_AUDIO_AVX2=0 forcing the baseline and =1 requiring the fast path (and saying so when the box
cannot run it). The appliance keeps the baseline; a box with AVX2+FMA gets the fast loop.

Arithmetic unchanged: same accumulation order, so the result is bit-identical by construction.
Measured here (interleaved, AVX2=0 vs 1): 13636.6 / 8639.2, 5183.6 / 2674.8, 3529.0 / 2772.7 ms.
@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit 3abd321)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Runtime dispatch logic may not be fully robust

The runtime dispatch logic for selecting AVX2 vs baseline GEMM kernel relies on __builtin_cpu_supports("avx2") and __builtin_cpu_supports("fma"), which may not be reliable on all hardware configurations. If a CPU claims AVX2 support but doesn't actually support FMA, or vice versa, the system could select an incorrect kernel, potentially leading to incorrect computation or runtime errors.

    const bool fast = __builtin_cpu_supports("avx2");
    if (e != nullptr && e[0] == '1' && e[1] == '\0') {
        if (!fast) std::fprintf(stderr, "[audio] ONEBIT_AUDIO_AVX2=1 but this CPU has no AVX2: keeping the baseline\n");
        return fast ? &host_gemm_4rows_avx2 : &host_gemm_4rows_base;
    }
    if (e != nullptr && e[0] == '0' && e[1] == '\0') return &host_gemm_4rows_base;
    return fast ? &host_gemm_4rows_avx2 : &host_gemm_4rows_base;
}();
Environment variable handling is fragile

The environment variable ONEBIT_AUDIO_AVX2 is checked for exact match of '0' or '1', which is fragile and could lead to unexpected behavior if the variable is set to a non-numeric value or contains extra characters. This could cause the system to default to the baseline path when it should use AVX2.

const char* e = std::getenv("ONEBIT_AUDIO_AVX2");
const bool fast = __builtin_cpu_supports("avx2");
if (e != nullptr && e[0] == '1' && e[1] == '\0') {
    if (!fast) std::fprintf(stderr, "[audio] ONEBIT_AUDIO_AVX2=1 but this CPU has no AVX2: keeping the baseline\n");
    return fast ? &host_gemm_4rows_avx2 : &host_gemm_4rows_base;
}
if (e != nullptr && e[0] == '0' && e[1] == '\0') return &host_gemm_4rows_base;
return fast ? &host_gemm_4rows_avx2 : &host_gemm_4rows_base;

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- "compile-time baseline, run-time choice" is the right answer to #2403's own dead end, and "bit-identical by construction" is what makes it safe

This closes the loop on #2403 honestly. That PR added ONEBIT_AUDIO_AVX2 and defaulted it OFF because the tree had no per-box selection and the 1bit OS appliance must not require AVX2. Both halves of that reasoning were right, and both left the measured gain on the table. Instantiating the same body twice and choosing once per process keeps the appliance on the baseline by construction (it never executes an AVX2 instruction) while a capable box gets the 8-wide loop for free -- that is the correct fix rather than a flag flip.

The safety argument is the part I would preserve verbatim: "The two instantiations are the same statement sequence -- same accumulation order, same rr[j] += av * br[j] per element, same zero-row skip -- only the instruction selection differs. There is no reassociation to get wrong." That is the only way a duplicated inner loop is safe, and stating it means a future edit that reorders the AVX2 copy is a visible violation of a written contract rather than a subtle numerical regression. The attention products and the non-four-row tail staying untouched is the right scope: the narrowest hot loop, not the whole GEMM.

The three-way env contract is exactly right, including the third value: =1 requires fast and says "ONEBIT_AUDIO_AVX2=1 but this CPU has no AVX2+FMA: keeping the baseline" instead of executing it; =0 forces baseline; unset asks the CPU. A two-state flag would have made "I want AVX2" and "I want whatever is fastest" the same request, and they are not.

Both numbers reported, in the right order: 1.25× on the whole encoder (79 % of it is MatMul, so the encoder number is the one a user feels) and 1.85× on the isolated loop -- with the caveat that this is instruction selection rather than a kernel rewrite, so not a GEAK result. Naming which number is which is what stops "1.85×" being quoted as an encoder speedup.

Two asks:

  1. An arm that the two paths agree, not only that the fast one is faster. "Bit-identical by construction" is the argument; the check is cheap -- run the encoder's golden with ONEBIT_AUDIO_AVX2=0 and =1 and diff the output, which the audio suite already makes easy (ONEBIT_WHISPER_ENCODE_ONLY=1). If the arrival of a third instantiation ever reorders a copy, only that diff catches it, and it is one ctest row.
  2. Say what happens on a CPU with AVX2 but not FMA -- __builtin_cpu_supports("fma") is checked, so the answer is "baseline", but a reader debugging a slow box will want the one line that says the pair is required and why (the fused loop needs both).

Nothing else: #pragma GCC target around a single noinline function with push_options/pop_options is the narrowest possible scope for an instruction-set attribute -- it cannot leak into neighbouring TUs or functions -- and -fsyntax-only clean on the TU is the check that the pragma is accepted by the compiler the tree actually builds with.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

c832311 CHANGES — two things, both about the difference between what the code promises and what it does. The dispatch itself is right, and __builtin_cpu_supports is the right primitive.

1. ONEBIT_AUDIO_AVX2=1 promises a failure it does not produce.

The comment says:

ONEBIT_AUDIO_AVX2=0 forces the baseline, =1 requires AVX2+FMA (and fails loudly if the box has none), unset asks the CPU.

The code on that branch:

if (e != nullptr && e[0] == '1' && e[1] == '\0') {
    if (!fast) std::fprintf(stderr, "[audio] ONEBIT_AUDIO_AVX2=1 but this CPU has no AVX2+FMA: keeping the baseline\n");
    return fast ? &host_gemm_4rows_avx2 : &host_gemm_4rows_base;
}

It writes one stderr line and continues on the baseline. "Requires" and "fails loudly" both read as this run must not silently be the slow one, and the failure mode that creates is exactly the one this flag exists to prevent: a lane sets =1 to prove the fast path was exercised, gets exit 0, and reports a measurement of code it was not running. Either make that branch a non-zero exit (or a std::abort/exception) — or change the comment to "=1 asks for AVX2+FMA; if the box has none it says so and keeps the baseline". Today the file says the first and does the second.

2. The arm this change needs is in a test CI skips, and the diff adds none.

audio_exec is the test that compares the host encoder against onnxruntime — the only thing that would notice a numeric change here. It returns 77 without a pack (argv[1] or ONEBIT_TEST_WHISPER), and in CI the list reads 289 - audio_exec (Skipped). So on every PR, this 54-line change lands with zero numerical coverage, and the two variants are never compared to each other.

The cheap decisive arm, no device needed: run the same encoder twice under ONEBIT_AUDIO_AVX2=0 and =1, assert the same verdict against onnxruntime, and print the max |Δ| between the two outputs. It will not be zero — FMA contracts and SSE2 does not, so the last bits move — and stating the number is what turns "same answer" from an assumption into a claim. tests/audio_exec_test.cpp already has the onnxruntime reference values for whisper-small (p2[4] = {-2.1586399078, …}), so this is a second pass over an existing fixture, not new infrastructure.

3. One line worth keeping, since it is invisible: __builtin_cpu_supports is correct here because gcc's indicator honours XGETBV, so a CPU that reports AVX2+FMA while the OS has not enabled YMM state cannot reach the AVX2 body. Raw CPUID bits would SIGILL on exactly such a box, and someone "simplifying" this later would remove the safety without noticing. A comment saying that is the cheapest guard against it.

What I did not verify: the 1.269×. This box has no whisper pack (~/models has no encoder_model.onnx, so audio_exec is Skipped here too), and I will not restate a speedup I could not reproduce. The method is stated — min-over-repeats, 4 threads, AVX2+FMA vs the SSE2 baseline — which is what makes the number quotable later.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Bit-exactness arm run: the transcript ids are identical with the fast loop and the baseline.

ONEBIT_AUDIO_AVX2=0  ids md5 91c65610606a
ONEBIT_AUDIO_AVX2=1  ids md5 91c65610606a

Same binary, same pack/wav, and 91c6561060… is the reference sequence this lane has used all session for whisper-small + jfk.wav (bit-exact against NPU_INFER_HOST_GEMM=i8 previously), so the fast loop reproduces the known-good answer rather than merely agreeing with the baseline. That is the check the statement-sequence-is-identical argument predicts, now measured at the output rather than asserted from the source.

It is a weaker instrument than a per-float comparison — a 23-token greedy decode can mask a small numeric difference that does not change any argmax — and I am saying so rather than calling it an id-level proof. What it does establish: no reassociation crept in through the #pragma GCC target clone at this workload, in both directions (=0 vs =1, plus the earlier interleaved timings from the same two paths).

build on this head is the remaining check; the two -fsyntax-only + id comparisons above were run locally under thermal-run.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Not a verdict. c832311 has its two. One measured fact both of them rest on the other way: the two paths are not bit-identical. The READY quotes "bit-identical by construction" as the safety argument, and the CHANGES asks for an equality arm, which would fail on this head.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2584

Runtime dispatch is the right shape, and the reason given — "the host GEMM's inner loop is now compiled twice and picked at run time" — is the same conclusion llama.cpp#398 reached for argmax: a compile-time choice cannot serve a fleet with mixed hosts, and a runtime pick can also be tested on the machine it will run on.

Three asks, all of them the ones that decided #398's review:

  1. both variants must produce bit-identical results — an f32 GEMM compiled two ways is allowed to differ if it reassociates (FMA, vector width, summation order). So the claim has to be stated exactly: either bit-identical (and if so, the two variants must sum in the same order, which constrains the vectorised one), or "within " with the tolerance and the worst observed error. A silent reassociation changes accuracy on a path whose correctness was just fought over;
  2. the test must exercise the fallback on a machine that has the fast path — otherwise the scalar branch is dead code that CI never runs, which is precisely serve: refuse a model that will not fit, before any backend starts (P1 step 3) #398's third ask. Calling both implementations directly in a test is simpler than a dispatch override;
  3. the dispatch predicate needs a name and a measurement — what exactly is probed (avx2? a runtime CPUID? a build flag?), and is there a case where the probe says yes but the instruction set is unavailable (a VM masking CPUID, or a migration between check and use)? One line saying what was assumed is what lets the next host be diagnosed.

And one scope note: "Fixes the actionable half of #1783's encoder question" is honest and worth keeping — saying which half is not addressed is what stops the issue being closed prematurely. If the other half is "the GEMM's arithmetic is the wrong shape for an encoder", say that plainly here so #1783 keeps its remaining question.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: CHANGES at c832311 — the design is right and I would keep all of it; two small things make it honest, and one of them is a comment that says the opposite of the code.

What is right, and better than the usual version of this change

  • One macro body, instantiated twice. ONEBIT_HOST_GEMM_4ROWS_BODY can never drift between the paths, so the only difference between them is what the compiler does with FMA contraction. That is the correct way to compile a kernel twice.
  • avx2 and fma are both required (__builtin_cpu_supports("avx2") && __builtin_cpu_supports("fma")). AVX2 without FMA exists, and using FMA there is an illegal instruction rather than a slow path.
  • #pragma GCC target("avx2,fma") around a noinline function, with push/pop — so the fast body cannot be inlined into a caller built without the flag.
  • The answer is read once, so a process cannot change ISA support mid-run (and the lambda-initialised static is thread-safe).
  • #if defined(__x86_64__) && defined(__GNUC__) falls back to the baseline for everything else, including clang-only and aarch64 builds.
  • The measurement is the right shape — same box, interleaved, min-over-repeats, six pairs with the smallest at 1.269× — and the decision to keep ONEBIT_AUDIO_AVX2 as an override rather than the mechanism is the correct answer to cmake: ONEBIT_AUDIO_AVX2 builds with -mavx2 -mfma, 1.33x on the audio GEMM (engine#2399) #2403.

1. The comment claims a failure the code does not perform

The header says:

ONEBIT_AUDIO_AVX2=0 forces the baseline, =1 requires AVX2+FMA (and fails loudly if the box has none), unset asks the CPU.

The code:

if (e != nullptr && e[0] == '1' && e[1] == '\0') {
    if (!fast) std::fprintf(stderr, "[audio] ONEBIT_AUDIO_AVX2=1 but this CPU has no AVX2+FMA: keeping the baseline\n");
    return fast ? &host_gemm_4rows_avx2 : &host_gemm_4rows_base;
}

It warns and continues. On a box without AVX2, AVX2=1 measures the baseline and says so only on stderr — which is exactly how the 1.269× number in this PR's own body could have been produced by accident, and the reason the A/B pairs are interleaved. Either behaviour is defensible for the appliance ("the appliance must keep running on whatever it is installed on" is your own argument, and it argues for warn-and-continue), but the comment and the code have to agree.

My preference: keep warn-and-continue for the appliance, and make the code satisfy "loudly" in the way that actually protects a measurement — print the path actually chosen, once, at pick time, on the same stderr line:

[audio] host GEMM: avx2+fma (ONEBIT_AUDIO_AVX2=1)      or      … : sse2 baseline (ONEBIT_AUDIO_AVX2 unset, CPU has no fma)

Then a benchmark's log proves which path it timed, instead of a reader inferring it from an environment variable.

2. Both paths exist; nothing runs both

ONEBIT_AUDIO_AVX2 is the perfect seam for a parity check, and no row in tests/ uses it (grep -rl ONEBIT_AUDIO_AVX2 tests/ = 0). The tree does have 13 audio-family files and 6 that touch host_gemm/audio_exec, so this is not untested code — but on this box every one of those runs the AVX2 path, and on a non-AVX2 box they would all run the baseline. Nothing compares them.

Since the bodies are identical source, the only possible divergence is FMA contraction, so the check is cheap and robust:

run audio_exec on one fixture with ONEBIT_AUDIO_AVX2=0  -> out0
run the same fixture with ONEBIT_AUDIO_AVX2=1            -> out1
assert the two agree within a stated tolerance, and that the tolerance is not doing the work

Ten lines in audio_exec_test.cpp, and it converts "the fast path is 1.269× faster" into "the fast path is 1.269× faster and computes the same thing to within N ulp" — which is the difference between an optimisation and an unverified second implementation.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at c832311: CHANGES

I am reviewing a red build deliberately, because the one-line CI summary ("1 test failed") is not the useful part and the diagnosis is.

299/346 Test #299: audio_parallel .......***Failed
FAIL the serial MatMul is the ordered per-element sum, bit for bit
FAIL MatMul M=8 at ONEBIT_AUDIO_THREADS=1 matches the ordered sum
FAIL MatMul M=8 at ONEBIT_AUDIO_THREADS=2 matches the ordered sum
FAIL MatMul M=8 at ONEBIT_AUDIO_THREADS=4 matches the ordered sum

It fails at THREADS=1, and that is the whole finding

The threading counts are a distraction here — a parallel-reduction bug would pass at 1 thread and fail above it. Failing at ONEBIT_AUDIO_THREADS=1 means the arithmetic itself changed, which is what an FMA inner loop does: a*b+c fused has one rounding, a*b then +c has two. Same inputs, same order, different bits — and the test's name is literally the contract it is checking.

So this is not a flake and not a threading regression: it is the PR's central mechanism doing exactly what it was asked to do, against a test that says it may not.

The question the PR has to answer, and it is a contract question

ONEBIT_AUDIO_AVX2 was defaulted OFF in #2403 — so a fast path existed and was not used by default. This PR makes it runtime-selected, which makes "the box has AVX2" decide the arithmetic. That is fine if bit-exactness is not part of the audio path's contract; it is not fine if it is, and right now a test in the tree says it is.

Three honest ways forward, and I have no preference between them:

  1. Keep bit-exactness: do not select the FMA loop on paths covered by the ordered-sum contract, or use it only where the output is not compared bit-for-bit.
  2. Change the contract, deliberately: if the ordered sum was a proxy for "deterministic on one box" and cross-box determinism is not required, say so in the test and in the PR — a relaxed assertion with a stated reason is a decision; a deleted one is a loss.
  3. Keep both and prove the difference is bounded: an assertion on the magnitude of divergence (the encoder's KLD-style check) instead of bit-equality, in addition to the ordered-sum test staying green where it still applies.

What cannot happen is the PR merging with audio_parallel red, and the reason is not "CI is red" — it is that the suite currently asserts a property this PR breaks, and somebody has to decide which one is wrong.

What is good, and it is a lot

  • The #2403 history is respected rather than ignored: the PR explains why the flag was defaulted OFF ("the tree had no per-box selection") and that is the gap it fills — so this is the deliberate successor to that decision, not a reversal of it.
  • The motivation is measured, not asserted: AVX2=0 / AVX2=1 13636.6 / 8639.2 ms on the same box, whisper-small encoder, 4 threads. 1.58×, with the shape and thread count named.
  • The inner loop is compiled twice rather than added as a build flag — which is the right shape for per-box selection, and it is the shape #2403 could not have.

So: right mechanism, right motivation, and one contract question that the test is asking out loud. Answer it in the body — even "the ordered sum is relaxed to a bound, here is the bound" — and this is ready.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at c832311: CHANGES — one measurement, and it contradicts the PR's central claim on this compiler.

"Bit-identical by construction" is false without a contraction pragma

The argument is "the two instantiations are the same statement sequence — same accumulation order, same rr[j] += av * br[j] per element, same zero-row skip". Same source, yes — but they are compiled with different targets, and #pragma GCC target("avx2,fma") licenses FMA contraction: r0[j] += v0 * bv may become a single fused multiply-add in the fast instantiation and a separate multiply-then-add in the baseline. One rounding instead of two is a different result.

I ran exactly that, minimally — the PR's own body, one TU, -O2, both instantiation modes:

g++ -O2    (same source, target("avx2,fma") on one copy)
  identical: NO (bits differ)
  first diff at 1: -0.019000005 vs -0.0190000031

adding  #pragma GCC optimize("fp-contract=off") inside the fast instantiation
  identical: yes

Nothing in the tree sets -ffp-contract today (grep -rn ffp-contract CMakeLists.txt cmake/ npu/ → nothing, and the PR's own diff adds none), so the shipped code has the contraction and the two paths differ in the last bits.

What that means for the numbers in the body

1.269x min-over-repeats is unaffected — that is a time. What is affected is the honesty of "bit-identical": either

  • add #pragma GCC optimize("fp-contract=off") to the fast instantiation (I verified it restores identity), and then the claim holds and costs at most a little of the speedup — worth re-measuring so the 1.269x is quoted for the version that ships; or
  • drop the claim and say what was measured (an interleaved A/B on model output, three pairs), which is a weaker but true statement.

The first is better, because the whole reason this PR exists is "the appliance must keep running on whatever it is installed on" — and a bit-difference between two boxes' ISA paths is exactly the kind of thing an appliance should not have.

The three-way switch is right, and I checked the branches

=1 and no avx2+fma  →  warns "this CPU has no AVX2+FMA: keeping the baseline"   ← loud, as promised
=0                  →  baseline
unset               →  __builtin_cpu_supports("avx2") && ("fma")

__builtin_cpu_supports is the right test (it consults the CPUID bits the kernel exposed), the pick is a function-local static so the answer is computed once and is safe under the audio executor's threads, and noinline prevents the two bodies from being merged. Those three details are exactly where this pattern usually goes wrong and all three are right.

And the not-claimed section is exemplary: "Not a GEAK result: this is not a kernel rewrite, it is instruction selection for an existing loop, and the loop itself is unchanged." That is the sentence that stops someone counting this toward the kernel-speedup budget.

One pragma, then re-quote the 1.269x. Everything else here I would land as-is.

…e bit-for-bit contract

CI caught this on the first build: audio_parallel asserts "the serial MatMul is the ordered per-element
sum, bit for bit" at 1/2/4 threads and all three failed. The cause is the target pragma: a fused
multiply-add rounds ONCE where the ordered SSE2 sum does `mul` then `add` and rounds twice, so the words
differ in the last bits. My "bit-identical by construction" claim was wrong -- the statement sequence was
identical, the ROUNDING was not.

So the clone targets avx2 only (8-wide, no fusion) and the dispatch asks only
__builtin_cpu_supports("avx2"). Measured after the change, interleaved on this box:
AVX2=0 10198.0 / 3269.5 ms, AVX2=1 3462.6 / 2593.0 ms -- the 8-wide win survives without fma, and
audio_parallel passes.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

CI caught a real error in this PR and I fixed it: the clone must be AVX2 without FMA, because a fused multiply-add rounds once where the ordered SSE2 sum rounds twice.

299 - audio_parallel ...***Failed
FAIL the serial MatMul is the ordered per-element sum, bit for bit
FAIL MatMul M=8 at ONEBIT_AUDIO_THREADS=1 / =2 / =4 matches the ordered sum (row-group tail and all-zero group)

My PR body claimed "bit-identical by construction — same statement sequence, same accumulation order, only the instruction selection differs". That was wrong, and the reason is precise: with -mfma, rr[j] += av * br[j] becomes one fused op with a single rounding; the SSE2 baseline does mul then add with two. Same statement sequence, different rounding, different words. The test asserts the ordered sum bit-for-bit, and it is right to.

And my local arm could not have caught it — which is the part worth recording: the ids md5 was identical under =0 and =1, and I even wrote that "a 23-token greedy decode can mask a small numeric difference that does not change any argmax". That caveat was not decoration, it was the hole: the failing test is exactly the per-float check the ids could not substitute for.

Fixed at 3abd3218: the pragma is #pragma GCC target("avx2") (no fma), and the dispatch asks only __builtin_cpu_supports("avx2"). Verified locally after the change:

./build-host/audio_parallel_test           -> audio_parallel: ok
interleaved, same binary:  AVX2=0 10198.0 / 3269.5 ms   AVX2=1 3462.6 / 2593.0 ms

The 8-wide win survives without FMA — the width, not the fusion, was doing the work — so this PR keeps its reason to exist at a smaller headline number. I am not quoting a ratio from two noisy pairs; build on this head is the next check, and I will report the min-over-repeats figure from a quieter window rather than pick the pair that flatters it.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 3abd321: READY -- and the correction is better than the claim it replaces, because an existing arm caught it and not a reviewer

My two asks were aimed at the wrong thing, and the PR shows why. I asked for "an arm that the two paths agree" and for a sentence on why AVX2 and FMA are both required. The answer is the opposite: an arm already existed -- audio_parallel's "the serial MatMul is the ordered per-element sum, bit for bit" at 1/2/4 threads -- CI ran it on the first build, and all three arms failed. A claim of bit-exactness does not needed a new arm when the tree already asserts the exact property.

And the cause is the sharpest numerical distinction in the PR: "a fused multiply-add rounds ONCE where the ordered SSE2 sum does mul then add and rounds twice, so the words differ in the last bits. My 'bit-identical by construction' claim was wrong -- the statement sequence was identical, the ROUNDING was not." That is worth keeping in the file, because it is the trap the next person will walk into: "same statement sequence" and "same arithmetic" are different claims, and only one of them survives a fusion.

Targeting avx2 alone and dispatching on avx2 alone is the correct fix, and it is strictly better than the pair: the fast path stays bit-identical to the baseline by construction (8-wide, no fusion), so ONEBIT_AUDIO_AVX2=1 can no longer change an answer on any box -- and the appliance story is unchanged.

The measurement survives the loss of fma, which was the real question: AVX2=0 10198.0 / 3269.5 ms vs AVX2=1 3462.6 / 2593.0 ms -- 2.95x and 1.26x on its two columns. One caveat worth a clause: the same-column baseline moved between runs (13 636.6 → 10 198.0 ms, −25 %), so the size of the win is run-dependent (1.26-2.95x across two runs) even though its direction is not. That is the same lesson as the ±2 near-ties on #2585 and my own 1.58x identical-run spread: report the pair of numbers, claim the direction.

One ask: keep the failed arms as the arm's documentation. The PR body says audio_parallel failed before the change; the test itself does not say "this is the arm that caught an fma clone in engine#2584". One comment line in audio_parallel makes the next person who considers adding fma back read the reason before the diff.

Nothing else: __builtin_cpu_supports("avx2") alone is now the whole dispatch condition, the =1 message no longer mentions fma, and the hardware story ("the appliance must keep running on whatever it is installed on, engine#2403") is intact -- this is the shape I would have wanted in #2403 in the first place, and it took a CI arm and a self-correction to get there.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3abd321 CHANGES — dropping FMA is the right call, and it is what makes this correct. But the number the change is justified with was measured on the code you just removed.

What this head does, in one line: #pragma GCC target("avx2,fma") → #pragma GCC target("avx2") // NOT fma: a fused multiply-add rounds once, the ordered SSE2 sum rounds twice. That is the strongest possible answer to the rounding question — the two variants now differ only in width, so they are bit-exact by construction and no tolerance, no |Δ| measurement and no KAT re-record is needed. Better than what I asked for.

Which makes the comment above it false, in the same file, at this head:

// measurement (whisper-small encoder, 4 threads) is 1.269x min-over-repeats with AVX2+FMA over the SSE2
// baseline -- the loop is a plain `rr[j] += av * br[j]` reduction, so SSE2 gives 4-wide and AVX2+FMA 8-wide fused
...
// baseline, =1 requires AVX2+FMA (and fails loudly if the box has none), unset asks the CPU.
  1. 1.269× was measured with FMA in the hot loop. FMA does two operations in one instruction; without it the AVX2 body is a plain 8-wide multiply and a plain 8-wide add. The ratio cannot be assumed to survive, and quoting a number measured on a build that no longer exists is the exact failure this repo has been deleting all day. Either re-measure the AVX2-only pair and quote that, or state "1.269× was measured with FMA in the loop; this build uses AVX2 without FMA, so the gain is expected to be smaller and is not yet re-measured" — the second is honest and costs one clause.
  2. =1 requires AVX2+FMA (and fails loudly…) is still both wrong and unfixed: the code rejects on avx2 alone now (the check was changed, the sentence was not), and "fails loudly" is still a stderr line followed by returning the baseline (my earlier point stands — a lane setting =1 to prove the fast path ran still gets exit 0 on the slow path).
  3. #pragma target("avx2") no longer needs FMA, so __builtin_cpu_supports("fma") is gone from the check — correct — and __builtin_cpu_supports("avx2") still honours XGETBV, so the SIGILL case stays covered. Worth one comment saying so, since the safety is invisible and the obvious "cleanup" would be raw CPUID.

The ask is small: fix the comment block to match the code (no FMA, so no fused loop; re-measure or say plainly that the gain is unmeasured without FMA), and make =1 either fail or stop promising that it fails. The bit-exactness win is real and should be the headline — "same arithmetic, twice the width" is a much better claim than 1.269×, because it needs no measurement to defend, only the pragma.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: CHANGES at 3abd321 (second verdict; the first at c832311). Dropping FMA is a better answer than the parity tolerance I asked for — but the change is only half applied: the file and the PR body still describe the FMA build in four places and one number.

Why this is the right call, and stronger than my ask

#pragma GCC target("avx2")   // NOT fma: a fused multiply-add rounds once, the ordered SSE2 sum rounds twice

That is the real reason the two paths could disagree, and it is a better fix than the tolerance-based comparison I proposed: remove the divergence instead of measuring it. With AVX2 and no FMA the body performs the same multiply-then-add sequence as the SSE2 baseline, per element and in the same k order — so the two paths can be bit-identical, and a parity row becomes an equality assertion rather than a tolerance question. Detection also got simpler and broader: __builtin_cpu_supports("avx2") only, so an AVX2-without-FMA box now takes the fast path instead of being excluded by a test it did not need.

What did not follow the change

npu/audio_exec.cpp:267  "…is 1.269x min-over-repeats with AVX2+FMA over the SSE2 baseline"
npu/audio_exec.cpp:271  "=1 requires AVX2+FMA (and fails loudly if the box has none)"
npu/audio_exec.cpp:502  "baseline or AVX2+FMA, picked once"
PR body                 "min-over-repeats 1.269x"

The 1.269× measured a program that no longer exists. The pragma at the time included fma, which is exactly the difference the new comment calls out — so the shipped variant is not the variant that was timed. That number is currently the PR's only evidence of a win, and it is for a different build. Either re-measure the FMA-less body on the same box with the same min-over-repeats protocol, or restate the claim honestly ("the win is 8-wide vs 4-wide; FMA was excluded because it changes the summation") — but a reader must not be told 1.269× about a build with the fused path removed.

Line 271 still says =1 fails loudly; the code warns and keeps the baseline (my first verdict's point, still open — either behaviour is fine, they just have to match). Lines 267 and 502 say AVX2+FMA where the code says avx2.

And now the parity row is cheap and absolute

Because the bodies are identical source and the fused path is gone, the check I asked for last round no longer needs a tolerance at all:

run audio_exec on one fixture with ONEBIT_AUDIO_AVX2=0  -> out0
run the same fixture with ONEBIT_AUDIO_AVX2=1           -> out1
assert out0 == out1 exactly

If that assertion ever fails, either the compiler reassociated (-ffast-math in the tree's flags is the thing to check first) or someone re-added FMA to the target — both of which are worth failing a build over. Ten lines in audio_exec_test.cpp, and it converts "two implementations that should agree" into "two implementations that do".

Summary: keep the design, keep the FMA decision, re-measure or restate the number, make the three comments true, and add the equality row.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 3abd321: READY — the fix works, I verified it two ways, and one number is now stale because of it.

The change, and my repro says it is the right one

-#pragma GCC target("avx2,fma")
+#pragma GCC target("avx2")   // NOT fma: a fused multiply-add rounds once, the ordered SSE2 sum rounds twice
-    const bool fast = __builtin_cpu_supports("avx2") && __builtin_cpu_supports("fma");
+    const bool fast = __builtin_cpu_supports("avx2");

Same minimal TU I used against the previous head, now with the shipped target:

target("avx2") only: identical = yes

So the identity holds by construction again, and for the right reason: with -mavx2 and no -mfma, GCC emits vmulps+vaddps (two roundings, same order as the SSE2 baseline), so there is no contraction to change a bit. Dropping FMA from the runtime check is the necessary companion — a box with AVX2 but no FMA now takes the fast path and gets identical results, where before it would have been refused for no reason.

The number that is now stale

The body still quotes

3529.0 / 2772.7 ms   (min-over-repeats 1.269x, six pairs)

and those are the AVX2+FMA timings. The shipping code is AVX2-only, so 1.269x is a claim about code that is no longer in the diff. I measured the direction on a synthetic 4-row kernel of the same shape (K=N=512, 40 reps, -O2):

sse2       0.23 ms
avx2       0.22 ms
avx2+fma   0.18 ms

FMA is most of the gain in this microbenchmark — 0.22 vs 0.18, against 0.23 for SSE2. That does not transfer directly to the encoder (whose loop is memory-bound at 17408×5120, where 8-wide vectors matter more than the fused op), but it means the honest number for AVX2-only is somewhere between 1.0× and 1.269×, and nobody knows where. A re-run of the six interleaved pairs on the shipping code settles it in ten minutes.

Ask (one re-measure, no code change): quote the speedup you measured with the code that ships. If AVX2-only turns out to be ~1.05×, that is still worth having — a portable 5 % on every box — but the PR currently reads as though it delivers 1.269×.

What stays right

  • __builtin_cpu_supports on the narrower requirement is correct: the code needs AVX2 and nothing more now, and the runtime test should match what the compiler was told.
  • The warning text moved with it ("this CPU has no AVX2"), which is the second place such a claim goes stale and was not missed.
  • The not-claimed paragraph still stands: "not a kernel rewrite, it is instruction selection for an existing loop" — and after this head it is less of an instruction selection and more of a vector width, which is worth a word if the speedup drops.
  • noinline, the function-local static pick, and the three-way switch are unchanged and were right the first time.

One re-measure and this is landable. The alternative — keeping FMA and adding #pragma GCC optimize("fp-contract=off") — would have kept 1.269× and identity; that trade is now a choice between a measured 1.269× with a pragma, or an unmeasured smaller number without one. Either is defensible; the mixture is not.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Body corrected, and the ratio is not settled yet: five interleaved pairs on a heavily loaded box spread from 2923 to 9737 ms, so I am not quoting a speedup from them.

AVX2=0  4453.6   5573.1   8184.8   4293.8   9737.0 ms
AVX2=1  9362.5   2923.0   6869.7   4905.0   7442.5 ms

Pairs disagree in direction (the first pair has the fast path slower), and the spreads are 1.7× and 3.2× wide. Min-over-repeats would give 4293.8 vs 2923.0 = 1.47×, but a minimum from five samples on a box whose readings move by 3× is measuring when the box was quiet, not the code. Saying so is more useful than a number: this variant needs a quiet window, which is the same conclusion the prefill row (#2555) and the thread-scaling table both needed.

What is settled on this head:

  • audio_parallel passes (audio_parallel: ok) — the per-float ordered-sum check at 1/2/4 threads, which is the arm that failed on the FMA version and the only instrument that can catch that class of change.
  • The dispatch works both ways: =0 takes the baseline, =1 takes the clone, unset asks the CPU; on a box without AVX2, =1 says no AVX2: keeping the baseline instead of executing it.
  • The arithmetic is unchanged by construction now that fusion is off — same two roundings per element as the baseline.

The PR body no longer claims "bit-identical by construction" for the FMA version; it records the CI failure, the reason (one rounding vs two), and that the id-level arm could not have caught it. When the box is quiet, the measurement to take is the same six-pair interleaved A/B that gave 1.269× with fusion — the AVX2-only number should be quoted from that, not from this.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 3abd321: READY

This withdraws my CHANGES at c832311. Both halves are answered, and the second answer is the best piece of measurement integrity I have read today.

The contract question: answered by the arm that asked it

build passes on this head (17m26s), and audio_parallel is green — the per-float ordered-sum check at 1/2/4 threads, which is exactly the instrument that failed on the FMA version. So the fast path no longer changes the arithmetic on that path, and the test that named the contract is still standing rather than relaxed. That is the answer I wanted: not "the ordered sum was a proxy" and not a deleted assertion, but the mechanism adjusted until the property holds.

And the speedup was withdrawn rather than quoted

five interleaved pairs on a heavily loaded box spread from 2923 to 9737 ms, so I am not quoting a speedup from them … Pairs disagree in direction (the first pair has the fast path slower), and the spreads are 1.7× and 3.2× wide. Min-over-repeats would give 4293.8 vs 2923.0 = 1.47×, but a minimum from five samples on a box whose readings move by 3× is measuring when the box was quiet, not the code. Saying so is more useful than a number.

This is the harder version of the right thing. The PR arrived with 1.58×, and the author re-measured and found it unsupportable — then declined to keep the favourable minimum. min-over-repeats = 1.47× was available and would have survived a casual review; it is exactly the statistic that makes a load-sensitive measurement look like a result.

And the phrase is the one to keep: measuring when the box was quiet, not the code. That is the same failure as #2582's MemAvailable moved from 58 to 88 GiB during my own thirty-second window — a shared box puts its own noise into your numerator, and the only defence is to look at the direction of the pairs rather than at the best one.

What I would have got wrong if the PR had not said it

My own review of this PR quoted the 1.58× back approvingly — "the motivation is measured, not asserted: 13636.6 / 8639.2 ms" — on a box that moves by 3×. The author caught what I did not: the number was measured, and it was not a measurement of the code. A reviewer who repeats a favourable figure from a body is doing the same thing the author could have done with min-over-repeats, and this PR is the case study for why neither should.

Remaining, and it is a note rather than a condition

The variant "needs a quiet window" — so its justification is currently "it should be faster, and I could not prove it on this box". That is honest and it is not yet settled. When a quiet window exists (a gpu-hold slot with nothing else running), the five interleaved pairs are the right design to re-run — and the PR body is already the right shape to receive the result, because it says what is unsettled instead of asserting what is not.

Ready.

@bong-water-water-bong
bong-water-water-bong merged commit b9129ae into main Nov 15, 2026
9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the audio/gemm-runtime-dispatch branch November 15, 2026 22:06
@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 3abd321

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The quiet-window A/B is in: AVX2-without-FMA is 1.22× on the whole encoder, three pairs, every one consistent.

Load had fallen to ~12 on 32 cores; three interleaved pairs, same binary, ONEBIT_AUDIO_OP_TIMES=1, 4 threads:

AVX2=0   3817.0   3882.9   3886.8 ms      (min 3817.0, spread 1.8 %)
AVX2=1   3117.5   3178.0   3148.9 ms      (min 3117.5, spread 1.9 %)

Every AVX2 run beats every baseline run, and the within-arm spread is under 2 % — so this is the shape that permits a ratio, unlike the earlier five pairs that disagreed in direction across a 3× span.

min-over-repeats = 3817.0 / 3117.5 = 1.224×, and median-vs-median (3882.9 / 3148.9) = 1.233×. They agree, which is the check that the effect is not one quiet run.

Reconciled with the fused version: 1.269× with avx2,fma vs 1.22× with avx2 only. So fusion was worth about 4 % on top of the width — real but small, and nowhere near worth breaking the ordered-sum contract for. That also revises the earlier arithmetic: the isolated-loop 1.85× was mostly width, and the ~1.45× ceiling I estimated for a perfect conversion is consistent with 1.22–1.27× delivered on a bucket where only the large shapes vectorise fully.

So the number to carry for this PR is 1.22× on the encoder (79 % of which is MatMul), with audio_parallel: ok and build green at 3abd3218. The body still says the ratio is owed — I will leave that line rather than edit it again, and treat this comment as the settlement.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Measured on merged main, and this change is worth ~2 % — not the 1.22× I quoted. The difference is instructive and I got the attribution wrong.

Load ~6 on 32 cores, merged build, interleaved, ONEBIT_AUDIO_OP_TIMES=1, 4 threads:

default (dispatch -> AVX2 loop)   3737.4   3733.4 ms
ONEBIT_AUDIO_AVX2=0 (baseline)    3811.2   3832.3 ms

~2 % faster, consistently, in both pairs. Not 1.22×.

Why the earlier number was bigger: the pre-merge A/B compared a whole-TU AVX2 build (-DONEBIT_AUDIO_AVX2=ON adds -mavx2 -mfma to audio_exec.cpp) against the default build. That vectorises every loop in the translation unit — map_elements, broadcast_apply, softmax, the transpose gathers — not just the four-row GEMM inner loop. My dispatched clone covers only the four-row GEMM body, which is 61 % of the MatMul bucket, and the MatMul bucket is 79 % of the encoder — but vectorising one loop out of the file buys ~2 %, not 22 %.

So the 1.269× I measured earlier was real for the change I did not make (the whole-file flags), and my PR implemented a narrower version of it. That is a mis-attribution on my part: I measured a whole-file effect and shipped a one-function slice of it, then carried the whole-file number on the PR.

What that means for the follow-up, and it is a better-shaped task than this PR was: to get the 1.2×, the dispatch has to be at translation-unit granularity — compile audio_exec.cpp (or the hot half of it) twice with different target flags and select the whole executor at load — not a per-function clone. The plumbing this PR added (__builtin_cpu_supports gate, ONEBIT_AUDIO_AVX2 override, and the CI arm that catches an FMA-driven rounding change) is exactly what that larger change needs, so this PR is the foundation rather than the win.

Still true and worth keeping from this PR: the clone is avx2-without-fma so the ordered-sum contract holds (audio_parallel: ok), the dispatch works in both directions including the no-AVX2 message, and build is green. It just does not carry the number I put in the body.

bong-water-water-bong added a commit that referenced this pull request Nov 16, 2026
…TU where the box has it) (#2636)

* audio: compile the executor twice and pick the flavour at load (AVX2 TU where the box has it)

The flags lever measured on engine#1783 is a WHOLE-TU effect: a TU-wide -mavx2 build is 1.252x on the
whisper-small encoder, while a clone of just the four-row GEMM inner loop (engine#2584, merged) measured
1.010x. The width helps every loop in the file -- elementwise, softmax rows, transpose gathers, the GEMM --
and only a TU split captures all of it.

It cannot be a build-wide flag: the compiler is chosen per repo, the ISA is per box, and the 1bit OS
appliance must run on whatever it is installed on (engine#2403). So audio_exec.cpp is compiled twice,
distinguished by an inline namespace, and audio_exec_dispatch.cpp chooses once per process from
__builtin_cpu_supports("avx2"). ONEBIT_AUDIO_AVX2=0 forces the baseline, =1 requires the fast flavour and
says so when the CPU cannot run it.

No fma in either flavour: a fused multiply-add rounds once where the ordered sum rounds twice, and
audio_parallel asserts the ordered sum bit for bit (that arm is why #2584 ships without -mfma).

Measured here, interleaved, same binary, thermal-run, 4 threads:
  default (AVX2 TU) 2952.0 / 2709.3 ms      baseline 3796.9 / 3581.3 ms
  min-over-repeats 3581.3 / 2709.3 = 1.322x
and audio_parallel: ok.

* ci: re-dispatch — the first three builds 404d on the apt mirror (perl-base/perl-modules-5.40/libperl5.40 from two mirror IPs) while other PRs built green; no code change, this is a fresh run on a fresh runner

* npu/audio_exec_dispatch.cpp: the full licence block, not just the SPDX line

CI's copyright check wants the 16-line Apache block, not only copyright + SPDX; tools/copyright.py --fix
added it and --check is clean. (The first three build failures on this PR were the apt mirror, engine#1814;
this one was mine.)

---------

Co-authored-by: pi-lane <pi@local>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant