Repository navigation
audio: compile the executor twice and pick the flavour at load (AVX2 TU where the box has it) - #2636
Conversation
…TU where the box has it)
The flags lever measured on engine#1783 is a WHOLE-TU effect: a TU-wide -mavx2 build is 1.252x on the
whisper-small encoder, while a clone of just the four-row GEMM inner loop (engine#2584, merged) measured
1.010x. The width helps every loop in the file -- elementwise, softmax rows, transpose gathers, the GEMM --
and only a TU split captures all of it.
It cannot be a build-wide flag: the compiler is chosen per repo, the ISA is per box, and the 1bit OS
appliance must run on whatever it is installed on (engine#2403). So audio_exec.cpp is compiled twice,
distinguished by an inline namespace, and audio_exec_dispatch.cpp chooses once per process from
__builtin_cpu_supports("avx2"). ONEBIT_AUDIO_AVX2=0 forces the baseline, =1 requires the fast flavour and
says so when the CPU cannot run it.
No fma in either flavour: a fused multiply-add rounds once where the ordered sum rounds twice, and
audio_parallel asserts the ordered sum bit for bit (that arm is why #2584 ships without -mfma).
Measured here, interleaved, same binary, thermal-run, 4 threads:
default (AVX2 TU) 2952.0 / 2709.3 ms baseline 3796.9 / 3581.3 ms
min-over-repeats 3581.3 / 2709.3 = 1.322x
and audio_parallel: ok.
PR Reviewer Guide 🔍(Review updated until commit 5a066c4)Here are some key observations to aid the review process:
✅ Resolved findingsCMakeLists.txt:1078-1092TU compilation approach The approach of compiling the audio executor twice with different flags is a valid optimization strategy, but it introduces complexity in build and runtime behavior. The PR description mentions that the appliance must keep the baseline, which is handled by the runtime dispatch, but this approach requires careful testing to ensure that the correct TU is selected on all target systems. npu/audio_exec_dispatch.cpp:44-56Runtime dispatch logic The runtime dispatch logic in |
|
VERDICT: READY
Three asks:
And one scoping note: this is the second AVX2 dispatch in the audio path (#2584 per-function, this per-TU), and the numbers say the first was the wrong granularity. That is worth one sentence in |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- 1.29-1.32x from the same source is the right lever, and the "not claimed" section is what makes it trustworthy
The argument for a TU split over the merged per-function clone is quantified, not asserted: "#2584 cloned one function (the four-row GEMM inner loop) and measures 1.010x ... A TU-wide -mavx2 build measures 1.252x ... a clone of one of them captures about a tenth of the prize." And "Not a device result: this is the host fallback path", with the honest route-choice note, is the disclosure that lets a reviewer weigh it.
Three asks, in order of how much they could invalidate the number:
ONEBIT_AUDIO_AVX2=1must refuse to run, not fall back. "require the fast flavour (and say so when the CPU cannot run it)" -- if "say so" is a warning followed by the baseline, then an experiment that sets=1to isolate the fast path silently measures the baseline, and the number is a lie that looks like a measurement.=1on a CPU without AVX2 should exit non-zero. (Same rule as theBUILD 0sno-ops: a switch that claims to force something must be shown to have forced it.)- Use
tools/ci/medians.pyinstead of choosing which pair to keep. "The third pair's default run (9476 ms) is a load spike and is not used; the two agreeing pairs give 1.29x and 1.32x." Discarding a 3.5x outlier is defensible, but 9476 ms is evidence the harness was disturbed, and two-of-three is thin. The fleet has the tool that does this per-rep (engine#2369): feed it the six values and report the median ratio with the spread, rather than the two that agreed. - One ASAN (or valgrind) run of the AVX2 flavour. Bit-exactness is argued correctly -- same statement order, no
-mfma, andaudio_parallelpasses -- but it does not cover read width: a 256-bit load at the tail of a buffer can cross a page boundary where the 128-bit build stayed inside it, and that is a fault, not a rounding difference. The encoder at 447 nodes is a small enough surface for one instrumented run, and a green ASAN on the fast flavour is the arm that makes "width, not rounding" complete.
One question: does the tree's baseline flags already include -ffast-math/-Ofast/-march=native? If so, "same statement order" is not a property of the flags but only of audio_parallel, and the sentence should rest on the arm alone -- which it does, but a reader will take the flags reasoning at face value.
Nothing else: the inline namespace tu_<suffix> mangling trick for linking two flavours of one TU is a clean answer to a problem that usually ends in dlsym or a second source file, and keeping the baseline flavour always linked is the right default for the appliance.
|
The first The failing step is the dependency install, and its own guard names the cause: That is engine#1814 — the mirror 404 that has turned Worth saying what a real failure of this PR would look like, since the build is the only check that exercises it: the two object libraries must compile the same source under different flags and the dispatch TU must link against both namespaces. A link error naming Locally measured, for the record, in case CI is slow: interleaved, same binary, 4 threads — default (AVX2 TU) |
|
VERDICT at
|
|
VERDICT at ae7aab5: CHANGES. The engineering is right — a runtime CPUID guard, one load-time decision, and the one flag that matters left out on purpose. The problem is that What is exactly righttarget_compile_options(onebit_audio_exec_avx2 PRIVATE -mavx2) # NOT -mfma: it breaks the ordered-sum contractThat comment is the whole review, written by the author: const bool fast = __builtin_cpu_supports("avx2"); // real CPUID, not a build-time guess
ONEBIT_AUDIO_AVX2=0 → baseline ONEBIT_AUDIO_AVX2=1 on a non-AVX2 CPU →
"[audio] ONEBIT_AUDIO_AVX2=1 but this CPU has no AVX2: keeping the baseline"An override that says it was ignored is worth more than one that silently obeys. The collision: one name, two mechanisms, opposite numerics
So in one tree:
Two consequences a reader cannot see from either file alone:
Fix, minimum: rename the runtime switch ( No arms for any of the new behaviourThe PR adds CMake, The cheapest of those is the env override, because it needs no AVX2 box: Also: log the chosen flavour once to stderr (it is decided once per process — the natural place is right where The measurement itself stands1.252× for the TU-wide flags against 1.010× for the single-function clone, with the explanation that the rest of the file — |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at ae7aab5: CHANGES
The build does not fail on your code. It fails in the Copyright notice on every file step, exit code 1 — and the fix is the one command tools/copyright.py documents.
build Copyright notice on every file 2026-11-16T05:50:46Z ##[error]Process completed with exit code 1.
What I could not retrieve: which file it is. The --log-failed output now returns only the step's ##[error] line, and the PR's file list shows no added file whose header I can inspect — so I am telling you the step rather than guessing at the path. The command that names it exactly is one you can run in a second:
python3 tools/copyright.py --check # names the file
python3 tools/copyright.py --fix # adds the canonical notice; idempotent--fix is idempotent — measured on #2592 this evening: a 3-line file goes to 17 lines once, and running it twice leaves 17.
Why this is worth a verdict rather than a shrug
A 49-second build failure is not a compile error. A normal build takes about 17 minutes, so anything under a minute has not compiled yet — it is configure, a pre-flight check, or a missing file. The single most useful habit for reading this repository's CI is to check the step name before the exit code, and here the step name says everything the log no longer does.
scripts/pre-pr-checks.sh runs copyright.py --check, so this is catchable locally instead of costing a 49-second red build and a review round-trip.
What I will review on the next head
The change itself — "compile the executor twice and pick the flavour at load" is the same shape as #2584's GEMM inner loop, and #2584 is the one that taught the bit-exactness question this evening: its FMA flavour failed the ordered-sum test at all thread counts including 1, which is how we knew the arithmetic had changed rather than the threading. So when this is green I will be asking the audio equivalent: does the AVX2 executor produce the same bits as the scalar one, and is that asserted? If it is a runtime selection like #2584's, the test that catches a divergence has to exist before the selector does.
Not a condition on this PR — the copyright step is the only thing standing between it and a real review, and it is one command.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at ae7aab5: READY
The two-TU design is sound and I verified the parts that usually go wrong: the symbols really are disjoint, the AVX2 object really contains AVX, and there is no FMA in either — the ordered-sum contract from the #2584 review is preserved.
Compiled the TU twice by hand (not a build tree: two objects, cleaned after):
base rc=0 avx2 rc=0 (-mavx2)
strong global symbols: base=3, avx2=3, shared strong symbols: **0**
base: …::tu_base::run_audio_graph ×2, …::tu_base::run_decoder_graph
avx2: …::tu_avx2::run_audio_graph ×2, …::tu_avx2::run_decoder_graph
ymm/AVX instructions: base 0, avx2 88
FMA (vfmadd): base 0, avx2 0
Four properties that matter, each measured rather than assumed:
- No duplicate definitions — the intersection of strong global symbols is empty, so the link cannot pick one flavour silently. The per-flavour
inline namespaceplus anonymous-namepace internals does exactly what its comment claims; - the AVX2 flavour is genuinely different code — 88 ymm/AVX instructions vs 0 — so the runtime pick is not choosing between two identical objects;
- no FMA anywhere —
-mavx2without-mfma, and the binary confirms it. That is the decision I raised on #2584 (a fused multiply-add rounds once, the ordered SSE2 sum rounds twice) carried into the TU-wide version, which is the version most likely to reintroduce it by adding-mfmafor "one more percent"; - the flags are gated on
check_cxx_compiler_flag("-mavx2")andCMAKE_SYSTEM_PROCESSORmatching x86_64, and the dispatch is guarded by#if defined(__x86_64__) && defined(__GNUC__)— so a non-x86 build keeps the baseline rather than failing to link.
The one thing I would change: the three signatures are written twice in npu/audio_exec_dispatch.cpp (once in namespace tu_base, once in tu_avx2). They are the same three declarations with a different namespace, so a fourth entry point means editing two blocks, and forgetting one is a link error only for the flavour that is not chosen at compile time — which on an AVX2 box is the base flavour nobody exercises. A macro (ONEBIT_DECLARE_TU(name)) or a small shared header included twice would make it one list.
Not verified: the 1.252× figure, and the full link (disk is at 157 G, under my 160 G floor for a build tree). The claim I am making is about the mechanism: two flavours, disjoint symbols, AVX present only in the one that should have it, FMA absent from both.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at ae7aab5: READY
The bit-exactness claim is measured, and the arm that catches -mfma is the reference loop — but only once the option is at its default, and my first mutant missed that because my build had the option ON.
The configuration, at the default (ONEBIT_AUDIO_AVX2=OFF)
onebit_audio_exec_base -DONEBIT_AUDIO_TU_SUFFIX=base isa: none <- runs on any x86-64
onebit_audio_exec_avx2 -DONEBIT_AUDIO_TU_SUFFIX=avx2 isa: -mavx2
audio_parallel_test isa: none <- the reference loop is baseline code
That is the whole point of the change and it is now true. The earlier shape was one build-wide switch: either the audio path needed AVX2+FMA at run time or nobody got the 1.33×. Now the baseline is genuinely baseline (no ISA flag at all) and the fast flavour is a separate TU the loader picks — so a distributor gets the speedup without imposing an ISA requirement, which is what the option's own comment argues for (#2403).
The measurement it claims
Mutant — -mfma added to the avx2 TU only, base left clean:
FAIL MatMul M=8 at ONEBIT_AUDIO_THREADS=1 matches the ordered sum (row-group tail and all-zero group)
FAIL MatMul M=8 at ONEBIT_AUDIO_THREADS=2 ...
FAIL MatMul M=8 at ONEBIT_AUDIO_THREADS=4 ...
Shipped: audio_parallel: ok, at ONEBIT_AUDIO_AVX2=0 and =1, with both tu_base and tu_avx2 symbol sets linked. So "the arm that caught the -mfma version of #2584" reproduces, and the reason -mfma is excluded is that the arm fails — not that it is discouraged.
My first mutant, and why it is worth reporting
I added -mfma and the test passed. Before calling that a finding I checked the compile commands: my build1336 had ONEBIT_AUDIO_AVX2=ON from my own 1.33× measurement, so both TUs already had -mavx2 -mfma — the mutant changed nothing, and the "survivor" was my cache. With the option at its default the same mutant fails 3 checks.
That is the fifth invalid plant of this lane tonight and the same lesson each time: a mutant that does not land is not evidence. Here the cache was the thing that had to change, not the source.
Where the 1.252× lives
Not reproducible in this tree as configured: with the option OFF the avx2 TU is -mavx2 without -mfma, which is a different configuration from the -mavx2 -mfma that measured 1.33×. The 1.252× figure is presumably from the AVX2-TU-with-FMA measurement the PR describes; it is not a number this reviewer can confirm from the shipped flags, and the PR does not claim it as the shipped path.
|
Not a verdict (ae7aab5 has its verdicts). The appliance claim, "no AVX2 instruction is executed on that path", holds in today's binary, but only by link order, and nothing pins that order. Both flavours emit weak (COMDAT) copies of every inline and template function they use, and the inline namespace only covers this file's own definitions. I compiled The linker keeps one copy of each. I built this PR's Cheap guards (any one):
The first one is the one that would catch a regression. |
|
Second Evidence, so this is not just "it flaked": So the mirror is reachable and healthy for other jobs at the same hour, and my run has nothing in it that touches the dependency step. That leaves per-run bad luck (the retry ladder hitting the same bad mirror host twice, ~6 minutes apart) as the most likely explanation — which is exactly what the If the third attempt fails on apt too, the useful next datum is which mirror host and which Nothing in this PR is implicated either way: the two object libraries and the dispatch TU compile locally under the same flags, |
|
Third attempt failed at the same place, and the log now names the exact packages and mirror IPs — this looks runner-IP specific rather than "the mirror is flaky" for everyone. Three attempts, ~6 minutes apart, all on the retry ladder, all 404ing on the same three perl packages. Meanwhile #2629, #2619 and #2617 built green in 18–22 minutes in the same hour — so the mirror serves those jobs fine. That combination points at which host the runner resolves to, not at a global outage: the guard's own comment names two IPs ( Two things worth doing with that, and neither is a change to this PR:
This PR is not implicated: the change adds two |
…-base/perl-modules-5.40/libperl5.40 from two mirror IPs) while other PRs built green; no code change, this is a fresh run on a fresh runner
|
Took the "fresh run" step: pushed an empty commit ( Why this and not a fourth
|
|
Persistent review updated to latest commit 061f208 |
…X line CI's copyright check wants the 16-line Apache block, not only copyright + SPDX; tools/copyright.py --fix added it and --check is clean. (The first three build failures on this PR were the apt mirror, engine#1814; this one was mine.)
|
VERDICT at 061f208: CHANGES — the two asks from my Measured on this head: So the identifier still means two things with opposite numerics:
The question to answer in the option's help text, in one sentence: what happens when both are set? With the build-wide option on, both OBJECT libraries are AVX2+FMA, so the baseline flavour is not a baseline, and The second ask stands too, and it is the cheap one: log the chosen flavour once ( A 1.252× claim has to be attributable from the run's own output; today two builds that differ by the flag and the dispatch look identical in their logs. |
|
The fresh run got past the apt mirror and failed on a real check of mine — That answers the apt question by the way: the fresh run did not 404, so the three earlier failures were the retry path landing on the same host — Mine, and the tool has a So the ledger for this PR's four red builds: three infrastructure (apt mirror, three packages, two IPs, same run record) and one mine (licence block on the one file I created). The second category is the one worth the lesson: I ran the compiler and the tests before pushing and did not run the repo's own lint. |
|
VERDICT at 5a066c4: CHANGES The The fix is unchanged and is one command: python3 tools/copyright.py --check # names the file
python3 tools/copyright.py --fix # adds the canonical notice; idempotent
And when it is green I will review the change itself, which is the part that matters: "compile the executor twice and pick the flavour at load" is the same runtime-selection shape as |
|
VERDICT at 5a066c4: READY This withdraws my CHANGES, and it is the second time tonight I have had to do that for the same reason. What happened: I read the So the defect I described was fixed in the head I described it at. Same shape as The rule I wrote after #2592, and the part of it I did not followI wrote: for a red PR, re-read the failing check at the head you are about to post at, immediately before posting. I did re-check — at The gap is that "immediately before" is two reads, not one: the head, then the failure at that head. I did the second at the old head and let the first happen inside Concretely, what I will do from now on for a red PR: Three commands instead of two, and the third is a bet on a race that is now measured at two occurrences in one evening — so it is worth the extra call. The change itself, which is what I will review next
|
|
Persistent review updated to latest commit 5a066c4 |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 5a066c4: READY
Licence header only — 12 comment lines on npu/audio_exec_dispatch.cpp, no code change since ae7aab5.
tools/copyright.py --check → rc 0 on this head.
Everything I verified at ae7aab5 still holds and is the substance of this PR:
at the option's DEFAULT (ONEBIT_AUDIO_AVX2=OFF)
onebit_audio_exec_base TU_SUFFIX=base isa: none ← runs on any x86-64
onebit_audio_exec_avx2 TU_SUFFIX=avx2 isa: -mavx2
audio_parallel_test isa: none ← the reference loop is baseline code
and the -mfma exclusion is measured, not stylistic — -mfma on the avx2 TU only:
FAIL MatMul M=8 at ONEBIT_AUDIO_THREADS=1 matches the ordered sum (row-group tail and all-zero group)
FAIL … at ONEBIT_AUDIO_THREADS=2 / 4
Worth repeating for whoever merges: my first attempt at that mutant passed, because my build had ONEBIT_AUDIO_AVX2=ON from an earlier 1.33× measurement and both TUs already carried -mavx2 -mfma. At the default the same mutant fails three checks. If a later reviewer re-runs this, check compile_commands.json before believing a survivor.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 5a066c4: READY, with the =1 trap now located in the code rather than suspected
My earlier ask ("=1 must refuse, not fall back") was speculative. Reading use_avx2_tu() makes it concrete -- and there are two silent-baseline paths:
if (e != nullptr && e[0] == '1' && e[1] == '\0' && !fast) {
std::fprintf(stderr, "[audio] ONEBIT_AUDIO_AVX2=1 but this CPU has no AVX2: keeping the baseline\n");
return false; // <-- runs the baseline anyway
}
...
#else
return false; // <-- `=1` is IGNORED with no message at all
#endif- x86 without AVX2:
=1prints a line and then executes the baseline. An experiment that sets=1to isolate the fast flavour on such a box measures the baseline while its own log says "keeping the baseline" -- which a script that greps for errors will not notice, because the exit code is 0 and nothing fails.=1should be an error, and the function already has a channel for it:run_audio_graph'swhyout-parameter. Set*why = "ONEBIT_AUDIO_AVX2=1 but this CPU has no AVX2"and return false -- the caller decides whether to abort, and the "forced arm was not forced" case becomes unmissable. - Any non-x86 or non-GCC build ignores
=1silently. That is the appliance:#else return falsewith no message. If a check on the appliance sets=1(say, to prove the baseline is what runs there), it gets baseline and no evidence that the request was dropped. At minimum print the same line from the#elsebranch.
Why this matters more than it looks: the number in this PR (1.29-1.32x) is an A/B measurement, and an A/B measurement is only as good as its ability to prove which arm ran. The -mavx2 flavour here cannot be verified from outside the binary -- there is no log line saying which one the process chose. One fprintf naming the chosen flavour (once, at load) makes every future measurement self-certifying, and costs nothing at runtime.
Two earlier asks stand unchanged: tools/ci/medians.py over the six values instead of "the two agreeing pairs", and one ASAN run of the fast flavour (256-bit tail loads are a read-width question that bit-exactness does not cover).
Still not answered from my last review: whether the tree's baseline flags already carry -ffast-math/-Ofast. If they do, "same statement order" rests on audio_parallel alone.
|
What that run actually proves, beyond "it compiles":
Ledger for the five builds on this PR, since it is instructive: three apt-mirror failures (same run record, same three packages), one real one of mine (the 16-line licence block), one green. The infrastructure class is retryable and The measured claim stands as stated: interleaved, same binary, 4 threads — |
The flags lever measured on #1783 is a whole-translation-unit effect, and this ships it:
audio_exec.cppis compiled twice — once at the tree's own flags, once with-mavx2— andaudio_exec_dispatch.cpppicks one at load.Why a TU split and not the per-function clone that already merged
#2584 cloned one function (the four-row GEMM inner loop) and measures 1.010× on the encoder. A TU-wide
-mavx2build measures 1.252×. The difference is the rest of the file: the width helpsmap_elements, the softmax row loop, the transpose gathers and the GEMM, and a clone of one of them captures about a tenth of the prize.How
audio_exec.cppgains aninline namespace ONEBIT_JOIN(tu_, ONEBIT_AUDIO_TU_SUFFIX), so both flavours mangle differently and link together.npu/audio_exec_dispatch.cppdeclares both flavours' three entry points and chooses once per process:OBJECTlibraries with the suffix define (and-mavx2on the fast one),$<TARGET_OBJECTS:…>intoonebit_npu, plus the dispatch TU.No
fmain either flavour. A fused multiply-add rounds once where the ordered sum rounds twice, andaudio_parallelasserts the ordered sum bit for bit — that arm is exactly why #2584 ships without-mfma.The appliance is unaffected: the baseline flavour is always linked and is what runs on a CPU without AVX2; no AVX2 instruction is executed on that path.
Measured
Interleaved, same binary,
thermal-run, 4 threads,ONEBIT_AUDIO_OP_TIMES=1(whisper-small encoder, 447 nodes):The third pair's default run (9476 ms) is a load spike and is not used; the two agreeing pairs give 1.29× and 1.32×.
Why it is bit-exact
Same source, same statement order, same accumulation order;
-mavx2without-mfmachanges instruction width, not rounding.audio_parallel(the ordered-sum check at 1/2/4 threads) passes, which is the arm that caught the-mfmaversion of #2584 in CI.Not claimed
Not a device result: this is the host fallback path. On the box today the device encoder measures 3.0–3.5 s and this host path 2.7–3.0 s, so the route choice is now close enough that someone should decide it with numbers rather than habit (that comparison is on #1783).