Skip to content

audio: compile the executor twice and pick the flavour at load (AVX2 TU where the box has it) - #2636

Merged
bong-water-water-bong merged 3 commits into
mainfrom
audio/tu-dispatch
Nov 16, 2026
Merged

bong-water-water-bong merged 3 commits into
mainfrom
audio/tu-dispatch

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

The flags lever measured on #1783 is a whole-translation-unit effect, and this ships it: audio_exec.cpp is compiled twice — once at the tree's own flags, once with -mavx2 — and audio_exec_dispatch.cpp picks one at load.

Why a TU split and not the per-function clone that already merged

#2584 cloned one function (the four-row GEMM inner loop) and measures 1.010× on the encoder. A TU-wide -mavx2 build measures 1.252×. The difference is the rest of the file: the width helps map_elements, the softmax row loop, the transpose gathers and the GEMM, and a clone of one of them captures about a tenth of the prize.

How

  • audio_exec.cpp gains an inline namespace ONEBIT_JOIN(tu_, ONEBIT_AUDIO_TU_SUFFIX), so both flavours mangle differently and link together.
  • npu/audio_exec_dispatch.cpp declares both flavours' three entry points and chooses once per process:
const bool fast = __builtin_cpu_supports("avx2");
ONEBIT_AUDIO_AVX2=0  -> force the baseline
ONEBIT_AUDIO_AVX2=1  -> require the fast flavour (and say so when the CPU cannot run it)
unset                -> ask the CPU
  • CMake: two OBJECT libraries with the suffix define (and -mavx2 on the fast one), $<TARGET_OBJECTS:…> into onebit_npu, plus the dispatch TU.

No fma in either flavour. A fused multiply-add rounds once where the ordered sum rounds twice, and audio_parallel asserts the ordered sum bit for bit — that arm is exactly why #2584 ships without -mfma.

The appliance is unaffected: the baseline flavour is always linked and is what runs on a CPU without AVX2; no AVX2 instruction is executed on that path.

Measured

Interleaved, same binary, thermal-run, 4 threads, ONEBIT_AUDIO_OP_TIMES=1 (whisper-small encoder, 447 nodes):

default (AVX2 TU)   2952.0   2709.3 ms
ONEBIT_AUDIO_AVX2=0 3796.9   3581.3 ms      min-over-repeats 3581.3 / 2709.3 = 1.322x
./build-host/audio_parallel_test   ->  audio_parallel: ok

The third pair's default run (9476 ms) is a load spike and is not used; the two agreeing pairs give 1.29× and 1.32×.

Why it is bit-exact

Same source, same statement order, same accumulation order; -mavx2 without -mfma changes instruction width, not rounding. audio_parallel (the ordered-sum check at 1/2/4 threads) passes, which is the arm that caught the -mfma version of #2584 in CI.

Not claimed

Not a device result: this is the host fallback path. On the box today the device encoder measures 3.0–3.5 s and this host path 2.7–3.0 s, so the route choice is now close enough that someone should decide it with numbers rather than habit (that comparison is on #1783).

…TU where the box has it)

The flags lever measured on engine#1783 is a WHOLE-TU effect: a TU-wide -mavx2 build is 1.252x on the
whisper-small encoder, while a clone of just the four-row GEMM inner loop (engine#2584, merged) measured
1.010x. The width helps every loop in the file -- elementwise, softmax rows, transpose gathers, the GEMM --
and only a TU split captures all of it.

It cannot be a build-wide flag: the compiler is chosen per repo, the ISA is per box, and the 1bit OS
appliance must run on whatever it is installed on (engine#2403). So audio_exec.cpp is compiled twice,
distinguished by an inline namespace, and audio_exec_dispatch.cpp chooses once per process from
__builtin_cpu_supports("avx2"). ONEBIT_AUDIO_AVX2=0 forces the baseline, =1 requires the fast flavour and
says so when the CPU cannot run it.

No fma in either flavour: a fused multiply-add rounds once where the ordered sum rounds twice, and
audio_parallel asserts the ordered sum bit for bit (that arm is why #2584 ships without -mfma).

Measured here, interleaved, same binary, thermal-run, 4 threads:
  default (AVX2 TU) 2952.0 / 2709.3 ms      baseline 3796.9 / 3581.3 ms
  min-over-repeats 3581.3 / 2709.3 = 1.322x
and audio_parallel: ok.
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit 5a066c4)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

1783 - Partially compliant

Compliant requirements:

  • Encoder route on our runtime implemented via TU split compilation and runtime dispatch
  • Correctness bar addressed by using baseline and AVX2 flavours with runtime selection
  • Measure and close the gap achieved through measured performance gains (1.252x)

Non-compliant requirements:

  • Baseline harness not fully implemented in this PR (only the audio executor part is addressed)
  • Row with full self-check and serving on --device npu not yet completed

Requires further human verification:

  • Full correctness verification against CPU baseline on ≥3 fixtures and edge cases
  • Performance measurement on actual hardware to confirm 1.252x speedup
  • Verification that the appliance keeps baseline flavour as required by engine#2403
⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Runtime dispatch logic

The runtime dispatch logic in audio_exec_dispatch.cpp uses __builtin_cpu_supports("avx2") to detect AVX2 support, which may not be reliable on all x86_64 systems. This could lead to incorrect selection of the AVX2 flavour on systems that report AVX2 support but don't actually support it, or vice versa. This was previously reported and remains an issue.

static bool use_avx2_tu() {
#if defined(__x86_64__) && defined(__GNUC__)
    static const bool pick = [] {
        const bool fast = __builtin_cpu_supports("avx2");
        const char* e = std::getenv("ONEBIT_AUDIO_AVX2");
        if (e != nullptr && e[0] == '0' && e[1] == '\0') return false;
        if (e != nullptr && e[0] == '1' && e[1] == '\0' && !fast) {
            std::fprintf(stderr, "[audio] ONEBIT_AUDIO_AVX2=1 but this CPU has no AVX2: keeping the baseline\n");
            return false;
        }
        return fast;
    }();
    return pick;
#else
    return false;
#endif
}
TU compilation approach

The approach of compiling the audio executor twice with different flags introduces complexity in build and runtime behavior. While the PR description mentions that the appliance must keep the baseline, this approach requires careful testing to ensure that the correct TU is selected on all target systems. This was previously reported and remains an issue.

# engine#1783: the audio executor is compiled twice and picked at load. The flags effect is a WHOLE-TU one
# (measured 1.252x TU-wide vs 1.010x for a clone of just the four-row GEMM), and the appliance must keep the
# baseline, so this is a runtime choice rather than a build-wide -mavx2 (engine#2403).
add_library(onebit_audio_exec_base OBJECT npu/audio_exec.cpp)
add_library(onebit_audio_exec_avx2 OBJECT npu/audio_exec.cpp)
foreach(_f onebit_audio_exec_base onebit_audio_exec_avx2)
    target_include_directories(${_f} PRIVATE npu)
endforeach()
target_compile_definitions(onebit_audio_exec_base PRIVATE ONEBIT_AUDIO_TU_SUFFIX=base)
target_compile_definitions(onebit_audio_exec_avx2 PRIVATE ONEBIT_AUDIO_TU_SUFFIX=avx2)
include(CheckCXXCompilerFlag)
check_cxx_compiler_flag("-mavx2" ONEBIT_HAVE_AVX2_TU)
if(ONEBIT_HAVE_AVX2_TU AND CMAKE_SYSTEM_PROCESSOR MATCHES "x86_64|AMD64")
    target_compile_options(onebit_audio_exec_avx2 PRIVATE -mavx2)   # NOT -mfma: it breaks the ordered-sum contract
endif()
add_library(onebit_npu STATIC npu/file_map.cpp npu/flm_head_import.cpp npu/full_elf.cpp npu/model.cpp npu/vision_pack.cpp npu/vision_tower.cpp npu/vision_image.cpp npu/vision_request.cpp
                            $<TARGET_OBJECTS:onebit_audio_exec_base> $<TARGET_OBJECTS:onebit_audio_exec_avx2> npu/audio_exec_dispatch.cpp npu/audio_pack.cpp npu/moe.cpp npu/pack.cpp npu/pack_import.cpp npu/private_route.cpp
                            npu/q4nx.cpp npu/q4nx_device.cpp npu/tokenizer.cpp)
✅ Resolved findings

CMakeLists.txt:1078-1092

TU compilation approach

The approach of compiling the audio executor twice with different flags is a valid optimization strategy, but it introduces complexity in build and runtime behavior. The PR description mentions that the appliance must keep the baseline, which is handled by the runtime dispatch, but this approach requires careful testing to ensure that the correct TU is selected on all target systems.

npu/audio_exec_dispatch.cpp:44-56

Runtime dispatch logic

The runtime dispatch logic in audio_exec_dispatch.cpp uses __builtin_cpu_supports("avx2") to detect AVX2 support, which may not be reliable on all x86_64 systems. This could lead to incorrect selection of the AVX2 flavour on systems that report AVX2 support but don't actually support it, or vice versa.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2636 — and the comparison that justifies the approach is the whole PR

1.010× versus 1.252× is the argument, measured both ways: #2584 cloned one function and got a 1 % win; compiling the whole translation unit with -mavx2 gets 25 %. "The difference is the rest of the file: the width helps map_elements, the softmax row loop, the transpose gathers and the GEMM, and a clone of one of them captures about a tenth of the prize." That is a rare case of a PR justifying not doing the smaller thing, with numbers for both.

Three asks:

  1. the two flavours must be proven equivalent, not assumed — inline namespace ONEBIT_JOIN(tu_, …) gives each flavour its own mangled names, which is exactly what makes them separately compiled and therefore separately optimisable: -mavx2 does not change f32 arithmetic semantics, but it does change contraction (FMA fusion) on some expressions. So the claim to state is either bit-identical between flavours, or "within X, worst observed e" — with the audio encoder's own tolerance as the reference. A 25 % win that changes the last bits of a mel-spectrogram is a different product decision from one that does not;
  2. say how the choice is made and what happens when it is wrong — __builtin_cpu_supports("avx2") is the obvious predicate; if the predicate says yes and the CPU cannot execute (a VM masking CPUID, a migration), the process dies with SIGILL rather than falling back. Name the predicate, and state that the non-AVX2 flavour is the one that must run when the predicate is unavailable;
  3. the fallback path needs its own test on an AVX2 machine — same ask as audio: pick the GEMM inner loop at run time (AVX2+FMA where the box has it) #2584/serve: refuse a model that will not fit, before any backend starts (P1 step 3) #398: an arm that reaches the scalar flavour by forcing it (an env var or a direct call), because otherwise the branch is dead code on every machine that runs CI.

And one scoping note: this is the second AVX2 dispatch in the audio path (#2584 per-function, this per-TU), and the numbers say the first was the wrong granularity. That is worth one sentence in docs/ — "measure the TU before cloning the function" — because the next person will reach for the smaller change first, and now there is a measurement saying the smaller change gets a tenth of the win.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- 1.29-1.32x from the same source is the right lever, and the "not claimed" section is what makes it trustworthy

The argument for a TU split over the merged per-function clone is quantified, not asserted: "#2584 cloned one function (the four-row GEMM inner loop) and measures 1.010x ... A TU-wide -mavx2 build measures 1.252x ... a clone of one of them captures about a tenth of the prize." And "Not a device result: this is the host fallback path", with the honest route-choice note, is the disclosure that lets a reviewer weigh it.

Three asks, in order of how much they could invalidate the number:

  1. ONEBIT_AUDIO_AVX2=1 must refuse to run, not fall back. "require the fast flavour (and say so when the CPU cannot run it)" -- if "say so" is a warning followed by the baseline, then an experiment that sets =1 to isolate the fast path silently measures the baseline, and the number is a lie that looks like a measurement. =1 on a CPU without AVX2 should exit non-zero. (Same rule as the BUILD 0s no-ops: a switch that claims to force something must be shown to have forced it.)
  2. Use tools/ci/medians.py instead of choosing which pair to keep. "The third pair's default run (9476 ms) is a load spike and is not used; the two agreeing pairs give 1.29x and 1.32x." Discarding a 3.5x outlier is defensible, but 9476 ms is evidence the harness was disturbed, and two-of-three is thin. The fleet has the tool that does this per-rep (engine#2369): feed it the six values and report the median ratio with the spread, rather than the two that agreed.
  3. One ASAN (or valgrind) run of the AVX2 flavour. Bit-exactness is argued correctly -- same statement order, no -mfma, and audio_parallel passes -- but it does not cover read width: a 256-bit load at the tail of a buffer can cross a page boundary where the 128-bit build stayed inside it, and that is a fault, not a rounding difference. The encoder at 447 nodes is a small enough surface for one instrumented run, and a green ASAN on the fast flavour is the arm that makes "width, not rounding" complete.

One question: does the tree's baseline flags already include -ffast-math/-Ofast/-march=native? If so, "same statement order" is not a property of the flags but only of audio_parallel, and the sentence should rest on the arm alone -- which it does, but a reader will take the flags reasoning at face value.

Nothing else: the inline namespace tu_<suffix> mangling trick for linking two flavours of one TU is a clean answer to a problem that usually ends in dlsym or a second source file, and keeping the baseline flavour always linked is the right default for the appliance.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The first build failed in 43 s on the apt mirror, not on this change — re-ran it.

The failing step is the dependency install, and its own guard names the cause:

echo "::error::apt-get update+install failed 4 times -- if the 404s name .debs the index lists,
      it is the mirror (engine#1814); retrying"; ... apt attempt $attempt failed (mirror 404 is a known transient)
Get:97 http://archive.ubuntu.com/ubuntu resolute/main amd64 liberror-perl …

That is engine#1814 — the mirror 404 that has turned main red for every PR before — so no code change is owed, and gh run rerun --failed is the documented response. Re-run is in flight.

Worth saying what a real failure of this PR would look like, since the build is the only check that exercises it: the two object libraries must compile the same source under different flags and the dispatch TU must link against both namespaces. A link error naming tu_avx2::run_audio_graph would mean the inline-namespace suffix did not apply; an audio_parallel failure would mean rounding changed (i.e. -mfma crept in). Neither is what happened here, and I verified both locally: audio_parallel: ok and the shim parses with 0 errors.

Locally measured, for the record, in case CI is slow: interleaved, same binary, 4 threads — default (AVX2 TU) 2952.0 / 2709.3 ms vs forced baseline 3796.9 / 3581.3 ms, min-over-repeats 1.322×.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at ae7aab5: READY. It answers both of my #2584 asks with measurements rather than words, and the number it reports about the previous approach is the most useful thing in the PR.

#2584's per-function clone measures 1.010×

"#2584 cloned one function (the four-row GEMM inner loop) and measures 1.010× on the encoder. A TU-wide -mavx2 build measur[es 1.29–1.32×]"

That is the answer to the question I could not settle while reviewing #2584: the 1.269× I was shown was measured with FMA, which the author then dropped for bit-exactness, and the honest number for the shipped variant was never published. It is 1.010× — the clone did essentially nothing, because the cost was never inside that one loop. A whole-TU -mavx2 is where the 30 % lives, and this PR replaces a no-op with it.

And it does not delete #2584 to say so: the comparison is stated as the reason for the TU split, which is what a follow-up should do when the earlier approach is superseded.

The measurement is handled the way I would want

default (AVX2 TU)    2952.0   2709.3 ms
ONEBIT_AUDIO_AVX2=0  3796.9   3581.3 ms     min-over-repeats 3581.3 / 2709.3 = 1.322x

"The third pair's default run (9476 ms) is a load spike and is not used; the two agreeing pairs give 1.29× and 1.32×."

The outlier is named, diagnosed and excluded in the body rather than quietly dropped from the min — and the two kept pairs agree to within 2.3 %. min-over-repeats on interleaved same-binary runs is the same protocol #2572/#679 used, so the number is comparable across the fleet.

The bit-exactness argument is structural, and that is better than my parity row

"No fma in either flavour … same source, same statement order, same accumulation order; -mavx2 without -mfma changes instruction width, not rounding."

I asked on #2584 for a runtime parity check (out0 == out1). This is the stronger version: if the statement order and the accumulation order are the same and FMA is excluded, the two flavours cannot differ, and audio_parallel asserts the ordered sum bit for bit as a live check. A tolerance test would have been the fallback; a construction argument plus an existing bit-for-bit assertion is better.

What would still slip past it: a future -ffast-math or -fassociative-math on either OBJECT library would break the argument without touching this file. One line in CMakeLists.txt next to the two target_compile_options — "no fast-math here: the two flavours must keep the same accumulation order" — makes that a deliberate edit rather than an accident. Not an ask for this PR; a note for the next one that touches those flags.

The scope is stated rather than implied

"Not claimed: not a device result: this is the host fallback path. On the box today the device encoder measures 3.0–3.5 s and this host path 2.7–3.0[s]."

So the host fallback is now at or below the device path, which is the interesting fact — and the appliance is unaffected because "the baseline flavour is always linked and is what runs on a CPU without AVX2". ONEBIT_AUDIO_AVX2=1 correctly says so when the CPU cannot run the fast flavour, which resolves the comment-versus-code mismatch I raised on #2584 by fixing the wording rather than the behaviour.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at ae7aab5: CHANGES. The engineering is right — a runtime CPUID guard, one load-time decision, and the one flag that matters left out on purpose. The problem is that ONEBIT_AUDIO_AVX2 now names two mechanisms with opposite contracts, and one of them is the thing this PR's own comment says must not happen.

What is exactly right

target_compile_options(onebit_audio_exec_avx2 PRIVATE -mavx2)   # NOT -mfma: it breaks the ordered-sum contract

That comment is the whole review, written by the author: -mfma changes the summation order and audio_parallel asserts the ordered sum bit for bit, so the fast TU must be -mavx2 alone. Two OBJECT libraries from one source with ONEBIT_AUDIO_TU_SUFFIX, both linked into onebit_npu, check_cxx_compiler_flag and an x86_64|AMD64 guard around the flag — and at runtime:

const bool fast = __builtin_cpu_supports("avx2");     // real CPUID, not a build-time guess
ONEBIT_AUDIO_AVX2=0 → baseline            ONEBIT_AUDIO_AVX2=1 on a non-AVX2 CPU →
                                            "[audio] ONEBIT_AUDIO_AVX2=1 but this CPU has no AVX2: keeping the baseline"

An override that says it was ignored is worth more than one that silently obeys.

The collision: one name, two mechanisms, opposite numerics

CMakeLists.txt still carries the older, build-wide mechanism under the same identifier:

1066: option(ONEBIT_AUDIO_AVX2 "Compile the audio TUs with -mavx2 -mfma: audio GEMM 1.33x; those TUs then require AVX2+FMA at run time" OFF)
2154: set_source_files_properties(${_onebit_audio_srcs} PROPERTIES COMPILE_OPTIONS "-mavx2;-mfma")
2161: set_source_files_properties(${_onebit_audio_test_srcs} … "-mavx2;-mfma")

So in one tree:

  • -DONEBIT_AUDIO_AVX2=ON ⇒ -mavx2 -mfma, i.e. the contract the new comment says breaks the ordered sum, applied to the audio sources and the audio tests;
  • ONEBIT_AUDIO_AVX2=1 (environment) ⇒ dispatch to the -mavx2-only TU, ordered sum intact.

Two consequences a reader cannot see from either file alone:

  1. With -DONEBIT_AUDIO_AVX2=ON there is no portable baseline. Both OBJECT libraries inherit the directory/source flags, so "the baseline" is AVX2+FMA too, and __builtin_cpu_supports("avx2") cannot protect a non-AVX2 box — the fallback falls back to another fast build. The dispatcher's "cannot change inside a process" comment is true and does not cover "there is nothing slower to choose".
  2. audio_parallel's bit-for-bit ordered-sum assertion and the -mfma build are in direct contradiction, and the test sources are compiled with -mfma when the option is on (line 2161). If that combination is exercised anywhere, the test is either already failing or the option has never been turned on since the assertion was added — which is worth knowing before someone turns it on to get the 1.33×.

Fix, minimum: rename the runtime switch (ONEBIT_AUDIO_AVX2_TU is enough) and state in the option's help text that with -DONEBIT_AUDIO_AVX2=ON the dispatcher has no portable alternative — or delete the build-wide option now that the per-TU one exists, since they are two answers to one question. One name per mechanism is what keeps the -mfma decision inside the guard that documents it.

No arms for any of the new behaviour

The PR adds CMake, audio_exec.cpp and audio_exec_dispatch.cpp — and no test. Nothing pins: the dispatch choosing avx2 when CPUID says so; the =1-on-a-non-AVX2 warning; =0 forcing the baseline; or that both flavours produce a valid graph (the numerics differ by design, so this is the one place a tolerance-free check is possible: "both run and both produce output of the right shape", not "both produce the same bytes").

The cheapest of those is the env override, because it needs no AVX2 box:

ONEBIT_AUDIO_AVX2=0 → the baseline path runs and says so once
ONEBIT_AUDIO_AVX2=1 on a CPU without avx2 → the warning on stderr, baseline still used

Also: log the chosen flavour once to stderr (it is decided once per process — the natural place is right where use_avx2_tu() first answers). A 1.252× claim has to be attributable from the run's own output, otherwise the next lane measuring 1.01× has no way to know which TU it measured.

The measurement itself stands

1.252× for the TU-wide flags against 1.010× for the single-function clone, with the explanation that the rest of the file — map_elements, the softmax row loop, the transpose gathers, the GEMM — is where the width goes. That is a good answer to "why not another clone", and the reason to spend the review on the switch rather than on the numbers.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at ae7aab5: CHANGES

The build does not fail on your code. It fails in the Copyright notice on every file step, exit code 1 — and the fix is the one command tools/copyright.py documents.

build   Copyright notice on every file   2026-11-16T05:50:46Z   ##[error]Process completed with exit code 1.

What I could not retrieve: which file it is. The --log-failed output now returns only the step's ##[error] line, and the PR's file list shows no added file whose header I can inspect — so I am telling you the step rather than guessing at the path. The command that names it exactly is one you can run in a second:

python3 tools/copyright.py --check     # names the file
python3 tools/copyright.py --fix       # adds the canonical notice; idempotent

--fix is idempotent — measured on #2592 this evening: a 3-line file goes to 17 lines once, and running it twice leaves 17.

Why this is worth a verdict rather than a shrug

A 49-second build failure is not a compile error. A normal build takes about 17 minutes, so anything under a minute has not compiled yet — it is configure, a pre-flight check, or a missing file. The single most useful habit for reading this repository's CI is to check the step name before the exit code, and here the step name says everything the log no longer does.

scripts/pre-pr-checks.sh runs copyright.py --check, so this is catchable locally instead of costing a 49-second red build and a review round-trip.

What I will review on the next head

The change itself — "compile the executor twice and pick the flavour at load" is the same shape as #2584's GEMM inner loop, and #2584 is the one that taught the bit-exactness question this evening: its FMA flavour failed the ordered-sum test at all thread counts including 1, which is how we knew the arithmetic had changed rather than the threading. So when this is green I will be asking the audio equivalent: does the AVX2 executor produce the same bits as the scalar one, and is that asserted? If it is a runtime selection like #2584's, the test that catches a divergence has to exist before the selector does.

Not a condition on this PR — the copyright step is the only thing standing between it and a real review, and it is one command.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at ae7aab5: READY

The two-TU design is sound and I verified the parts that usually go wrong: the symbols really are disjoint, the AVX2 object really contains AVX, and there is no FMA in either — the ordered-sum contract from the #2584 review is preserved.

Compiled the TU twice by hand (not a build tree: two objects, cleaned after):

base  rc=0    avx2  rc=0   (-mavx2)
strong global symbols: base=3, avx2=3, shared strong symbols: **0**

base: …::tu_base::run_audio_graph ×2, …::tu_base::run_decoder_graph
avx2: …::tu_avx2::run_audio_graph ×2, …::tu_avx2::run_decoder_graph

ymm/AVX instructions: base 0, avx2 88
FMA (vfmadd):        base 0, avx2 0

Four properties that matter, each measured rather than assumed:

  1. No duplicate definitions — the intersection of strong global symbols is empty, so the link cannot pick one flavour silently. The per-flavour inline namespace plus anonymous-namepace internals does exactly what its comment claims;
  2. the AVX2 flavour is genuinely different code — 88 ymm/AVX instructions vs 0 — so the runtime pick is not choosing between two identical objects;
  3. no FMA anywhere — -mavx2 without -mfma, and the binary confirms it. That is the decision I raised on #2584 (a fused multiply-add rounds once, the ordered SSE2 sum rounds twice) carried into the TU-wide version, which is the version most likely to reintroduce it by adding -mfma for "one more percent";
  4. the flags are gated on check_cxx_compiler_flag("-mavx2") and CMAKE_SYSTEM_PROCESSOR matching x86_64, and the dispatch is guarded by #if defined(__x86_64__) && defined(__GNUC__) — so a non-x86 build keeps the baseline rather than failing to link.

The one thing I would change: the three signatures are written twice in npu/audio_exec_dispatch.cpp (once in namespace tu_base, once in tu_avx2). They are the same three declarations with a different namespace, so a fourth entry point means editing two blocks, and forgetting one is a link error only for the flavour that is not chosen at compile time — which on an AVX2 box is the base flavour nobody exercises. A macro (ONEBIT_DECLARE_TU(name)) or a small shared header included twice would make it one list.

Not verified: the 1.252× figure, and the full link (disk is at 157 G, under my 160 G floor for a build tree). The claim I am making is about the mechanism: two flavours, disjoint symbols, AVX present only in the one that should have it, FMA absent from both.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at ae7aab5: READY

The bit-exactness claim is measured, and the arm that catches -mfma is the reference loop — but only once the option is at its default, and my first mutant missed that because my build had the option ON.

The configuration, at the default (ONEBIT_AUDIO_AVX2=OFF)

onebit_audio_exec_base    -DONEBIT_AUDIO_TU_SUFFIX=base     isa: none      <- runs on any x86-64
onebit_audio_exec_avx2    -DONEBIT_AUDIO_TU_SUFFIX=avx2     isa: -mavx2
audio_parallel_test                                         isa: none      <- the reference loop is baseline code

That is the whole point of the change and it is now true. The earlier shape was one build-wide switch: either the audio path needed AVX2+FMA at run time or nobody got the 1.33×. Now the baseline is genuinely baseline (no ISA flag at all) and the fast flavour is a separate TU the loader picks — so a distributor gets the speedup without imposing an ISA requirement, which is what the option's own comment argues for (#2403).

The measurement it claims

Mutant — -mfma added to the avx2 TU only, base left clean:

FAIL MatMul M=8 at ONEBIT_AUDIO_THREADS=1 matches the ordered sum (row-group tail and all-zero group)
FAIL MatMul M=8 at ONEBIT_AUDIO_THREADS=2 ...
FAIL MatMul M=8 at ONEBIT_AUDIO_THREADS=4 ...

Shipped: audio_parallel: ok, at ONEBIT_AUDIO_AVX2=0 and =1, with both tu_base and tu_avx2 symbol sets linked. So "the arm that caught the -mfma version of #2584" reproduces, and the reason -mfma is excluded is that the arm fails — not that it is discouraged.

My first mutant, and why it is worth reporting

I added -mfma and the test passed. Before calling that a finding I checked the compile commands: my build1336 had ONEBIT_AUDIO_AVX2=ON from my own 1.33× measurement, so both TUs already had -mavx2 -mfma — the mutant changed nothing, and the "survivor" was my cache. With the option at its default the same mutant fails 3 checks.

That is the fifth invalid plant of this lane tonight and the same lesson each time: a mutant that does not land is not evidence. Here the cache was the thing that had to change, not the source.

Where the 1.252× lives

Not reproducible in this tree as configured: with the option OFF the avx2 TU is -mavx2 without -mfma, which is a different configuration from the -mavx2 -mfma that measured 1.33×. The 1.252× figure is presumably from the AVX2-TU-with-FMA measurement the PR describes; it is not a number this reviewer can confirm from the shipped flags, and the PR does not claim it as the shipped path.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Not a verdict (ae7aab5 has its verdicts). The appliance claim, "no AVX2 instruction is executed on that path", holds in today's binary, but only by link order, and nothing pins that order.

Both flavours emit weak (COMDAT) copies of every inline and template function they use, and the inline namespace only covers this file's own definitions. I compiled npu/audio_exec.cpp twice from git merge-tree origin/main ae7aab5 with the tree's own flags (g++-16 -O3 -std=c++26, one flavour plus -mavx2), then compared the two objects' weak definitions:

weak definitions present in BOTH objects          38
  of those, AVX2-flavour body uses ymm             10
  e.g. std::string::swap (194 ymm), std::__adjust_heap<…pair<double,string>…> (57), std::operator+<char…> (28),
       std::to_string(int|long|unsigned long), onebit::npu::pack_import::audio_thread_count()   <- header-inline, outside the namespace

The linker keeps one copy of each. I built this PR's 1bit and disassembled each of those symbols in the final binary: 0 of 9 linked copies contain ymm. The base flavour won, because $<TARGET_OBJECTS:onebit_audio_exec_base> comes before …_avx2 in onebit_npu's source list, and GNU ld keeps the first definition it sees. Swap those two entries, add LTO, or let another link pick differently, and the baseline path calls an AVX2 std::string::swap. On a CPU without AVX2, which is exactly the appliance this design protects, that is a SIGILL at load, not a slow path.

Cheap guards (any one):

  • a ctest that disassembles the linked 1bit and fails if any symbol outside tu_avx2 contains a ymm/vex instruction (the check above is about 10 lines);
  • a comment on the source list saying the base objects must come first, and why;
  • or compile the AVX2 TU with -fvisibility-inlines-hidden / -ffunction-sections + --gc-sections scoped so its template copies cannot be chosen for the base path.

The first one is the one that would catch a regression.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Second build attempt failed on the same apt mirror line; a third is running. It is not systemic — three other open PRs built green in 18–22 min while mine failed twice in 43 s.

Evidence, so this is not just "it flaked":

#2629 build pass 18m40s      #2619 build pass 22m6s      #2617 build pass 18m28s
my run 43812790090: attempt 1 failure 43s, attempt 2 failure 43s — both at the same line
  echo "::error::apt-get update+install failed 4 times -- ... it is the mirror (engine#1814)"

So the mirror is reachable and healthy for other jobs at the same hour, and my run has nothing in it that touches the dependency step. That leaves per-run bad luck (the retry ladder hitting the same bad mirror host twice, ~6 minutes apart) as the most likely explanation — which is exactly what the engine#1814 guard exists to be retried through.

If the third attempt fails on apt too, the useful next datum is which mirror host and which .deb — the log carries both (Get:97 http://archive.ubuntu.com/... liberror-perl), and a per-host pattern would turn "known transient" into "one host is serving stale indices", which is actionable rather than retryable.

Nothing in this PR is implicated either way: the two object libraries and the dispatch TU compile locally under the same flags, audio_parallel passes, and the change touches no dependency.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Third attempt failed at the same place, and the log now names the exact packages and mirror IPs — this looks runner-IP specific rather than "the mirror is flaky" for everyone.

# install 404'd on .debs the fresh index had just listed
#   (perl-base / perl-modules-5.40 / libperl5.40, mirror IPs 91.189.92.22 and 91.189.91.82)
::error::apt-get update+install failed 4 times -- ... it is the mirror (engine#1814)

Three attempts, ~6 minutes apart, all on the retry ladder, all 404ing on the same three perl packages. Meanwhile #2629, #2619 and #2617 built green in 18–22 minutes in the same hour — so the mirror serves those jobs fine.

That combination points at which host the runner resolves to, not at a global outage: the guard's own comment names two IPs (91.189.92.22, 91.189.91.82), and a host with a stale perl-* index would 404 exactly that set while another host serves it. engine#1814 is closed — its fix was the retry ladder, which is what is failing here, so this case is outside the fix it shipped.

Two things worth doing with that, and neither is a change to this PR:

  1. A fresh dispatch rather than another --failed re-run. Two re-runs reused the same run record and, plausibly, the same runner assignment; a new run gets a new one. If a fresh run goes green, the datum is "re-run --failed pins the bad host", which is a real (and fixable) property of the retry path.
  2. If it 404s on a fresh run too, the useful artefact is a one-line record on the CI-runner side — "the 11-16 05:50–06:10 window served perl-base/perl-modules-5.40/libperl5.40 404s from two of the mirror IPs while other jobs built" — because that distinguishes mirror from runner IP for whoever reads the next red build.

This PR is not implicated: the change adds two OBJECT libraries and one dispatch TU; it touches no dependency, and both compile locally with the same flags (audio_parallel: ok, shim parses, interleaved A/B 1.29–1.32×).

…-base/perl-modules-5.40/libperl5.40 from two mirror IPs) while other PRs built green; no code change, this is a fresh run on a fresh runner
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Took the "fresh run" step: pushed an empty commit (ci: re-dispatch …) because ci.yml has no workflow_dispatch, so a new run is the only way to get a new runner assignment. build is now pending on a fresh run — no code change, the tree diff is identical.

Why this and not a fourth --failed re-run: the three failures reused run 43812790090 (attempts 1, 2, 3), and each 404d on the same three packages — perl-base, perl-modules-5.40, libperl5.40 — from the same two mirror IPs (91.189.92.22, 91.189.91.82). Meanwhile #2629/#2619/#2617 built green in the same hour. That pattern is consistent with the retry path landing on the same host; a new run is the cheapest way to test it, and it is also the test of the hypothesis: if this run is green, --failed re-run pins the bad host and that is worth fixing in the guard; if it 404s again, the window is host-side and belongs in the runner record.

workflow_dispatch being absent is worth knowing on its own — the obvious remedy ("re-dispatch the workflow") is not available for ci.yml, so an empty commit is the only lever a lane has. If that is not the intended mechanism, a workflow_dispatch trigger would make the next lane's infra retry one command instead of a commit.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 061f208

…X line

CI's copyright check wants the 16-line Apache block, not only copyright + SPDX; tools/copyright.py --fix
added it and --check is clean. (The first three build failures on this PR were the apt mirror, engine#1814;
this one was mine.)
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 061f208: CHANGES — the two asks from my c339f0e review are unchanged at this head, and I would rather say so plainly than repeat the essay.

Measured on this head:

option(ONEBIT_AUDIO_AVX2 "Compile the audio TUs with -mavx2 -mfma: …")   CMakeLists.txt:1066   still present
ONEBIT_AUDIO_AVX2                                                        audio_exec_dispatch.cpp   still the env var's name
test files added                                                         0

So the identifier still means two things with opposite numerics:

  • -DONEBIT_AUDIO_AVX2=ON ⇒ -mavx2 -mfma on the audio sources and the audio tests — the contract your own comment says breaks audio_parallel's ordered sum;
  • ONEBIT_AUDIO_AVX2=1 in the environment ⇒ dispatch to the -mavx2-only TU, ordered sum intact.

The question to answer in the option's help text, in one sentence: what happens when both are set? With the build-wide option on, both OBJECT libraries are AVX2+FMA, so the baseline flavour is not a baseline, and __builtin_cpu_supports("avx2") protects nothing on a box without AVX2 — the fallback falls back to another fast build. Either the option is deleted now that the per-TU switch answers the same question, or it is renamed and says what it does to the dispatcher.

The second ask stands too, and it is the cheap one: log the chosen flavour once (use_avx2_tu() answers once per process — that is where it belongs), and an arm for the env override, which needs no AVX2 hardware to test:

ONEBIT_AUDIO_AVX2=0            → the baseline path runs, and the log line says so
ONEBIT_AUDIO_AVX2=1 on a CPU without avx2 → the existing warning, baseline still used

A 1.252× claim has to be attributable from the run's own output; today two builds that differ by the flag and the dispatch look identical in their logs.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The fresh run got past the apt mirror and failed on a real check of mine — npu/audio_exec_dispatch.cpp had copyright + SPDX but not the 16-line licence block. Fixed (5a066c46); python3 tools/copyright.py --check is clean.

copyright + SPDX present but the licence block is not (paste the 16-line form): npu/audio_exec_dispatch.cpp
run: python3 tools/copyright.py --fix

That answers the apt question by the way: the fresh run did not 404, so the three earlier failures were the retry path landing on the same host — --failed re-run pins the bad runner, and that is worth knowing for the engine#1814 guard (which is closed, and whose retry ladder is what was cycling).

Mine, and the tool has a --fix I should have run before pushing: tools/copyright.py --fix added the block, --check then clean, and the shim still compiles (-fsyntax-only, 0 errors). The new file is the only one the check flags — audio_exec.cpp and CMakeLists.txt were already in the repo form.

So the ledger for this PR's four red builds: three infrastructure (apt mirror, three packages, two IPs, same run record) and one mine (licence block on the one file I created). The second category is the one worth the lesson: I ran the compiler and the tests before pushing and did not run the repo's own lint.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 5a066c4: CHANGES

The Copyright notice on every file step still fails at this head — same defect, new head, so the CHANGES from my previous review carries forward:

build   Copyright notice on every file   2026-11-16T06:27:03Z   python3 tools/copyright.py …

The fix is unchanged and is one command:

python3 tools/copyright.py --check     # names the file
python3 tools/copyright.py --fix       # adds the canonical notice; idempotent

scripts/pre-pr-checks.sh runs copyright.py --check, so this is catchable locally in a second rather than costing another 49-second red build.

And when it is green I will review the change itself, which is the part that matters: "compile the executor twice and pick the flavour at load" is the same runtime-selection shape as #2584, and #2584 is what established the question this evening — its FMA flavour failed the ordered-sum test at all thread counts including 1, which is how we knew the arithmetic had changed rather than the threading. So the audio equivalent is: does the AVX2 executor produce the same bits as the scalar one, and is that asserted before the selector exists?

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 5a066c4: READY

This withdraws my CHANGES, and it is the second time tonight I have had to do that for the same reason.

What happened: I read the Copyright notice on every file failure at head 061f208, wrote the CHANGES, and post-verdict.sh — which takes the head from the API at post time — posted it at 5a066c4. That head adds the copyright line:

$ gh pr diff 2636 -- grep -c "^+.*Copyright 2026"
1

So the defect I described was fixed in the head I described it at. Same shape as #2592 an hour earlier, where the author ran --fix while I was writing the verdict.

The rule I wrote after #2592, and the part of it I did not follow

I wrote: for a red PR, re-read the failing check at the head you are about to post at, immediately before posting. I did re-check — at 061f208 — and then posted, and the head had moved again in between.

The gap is that "immediately before" is two reads, not one: the head, then the failure at that head. I did the second at the old head and let the first happen inside post-verdict.sh.

Concretely, what I will do from now on for a red PR:

gh pr view <n> --json headRefOid        # 1. what head am I about to name?
gh pr checks <n>                        # 2. is THAT head still red, and in the same step?
./lb/post-verdict.sh …                  # 3. post, and accept the tiny race that remains

Three commands instead of two, and the third is a bet on a race that is now measured at two occurrences in one evening — so it is worth the extra call.

The change itself, which is what I will review next

5a066c4 is green so far (=8 pending, one success), and when it completes the review is the one my CHANGES was holding for: "compile the executor twice and pick the flavour at load" is #2584's runtime-selection shape, and #2584 is what established the question — its FMA flavour failed the ordered-sum test at every thread count including 1, which is how we knew the arithmetic had changed rather than the threading. So: does the AVX2 executor produce the same bits as the scalar one, and is that asserted before the selector exists?

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 5a066c4

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 5a066c4: READY

Licence header only — 12 comment lines on npu/audio_exec_dispatch.cpp, no code change since ae7aab5.

tools/copyright.py --check → rc 0 on this head.

Everything I verified at ae7aab5 still holds and is the substance of this PR:

at the option's DEFAULT (ONEBIT_AUDIO_AVX2=OFF)
onebit_audio_exec_base   TU_SUFFIX=base   isa: none      ← runs on any x86-64
onebit_audio_exec_avx2   TU_SUFFIX=avx2   isa: -mavx2
audio_parallel_test                       isa: none      ← the reference loop is baseline code

and the -mfma exclusion is measured, not stylistic — -mfma on the avx2 TU only:

FAIL MatMul M=8 at ONEBIT_AUDIO_THREADS=1 matches the ordered sum (row-group tail and all-zero group)
FAIL … at ONEBIT_AUDIO_THREADS=2 / 4

Worth repeating for whoever merges: my first attempt at that mutant passed, because my build had ONEBIT_AUDIO_AVX2=ON from an earlier 1.33× measurement and both TUs already carried -mavx2 -mfma. At the default the same mutant fails three checks. If a later reviewer re-runs this, check compile_commands.json before believing a survivor.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 5a066c4: READY, with the =1 trap now located in the code rather than suspected

My earlier ask ("=1 must refuse, not fall back") was speculative. Reading use_avx2_tu() makes it concrete -- and there are two silent-baseline paths:

if (e != nullptr && e[0] == '1' && e[1] == '\0' && !fast) {
    std::fprintf(stderr, "[audio] ONEBIT_AUDIO_AVX2=1 but this CPU has no AVX2: keeping the baseline\n");
    return false;                       // <-- runs the baseline anyway
}
...
#else
    return false;                       // <-- `=1` is IGNORED with no message at all
#endif
  1. x86 without AVX2: =1 prints a line and then executes the baseline. An experiment that sets =1 to isolate the fast flavour on such a box measures the baseline while its own log says "keeping the baseline" -- which a script that greps for errors will not notice, because the exit code is 0 and nothing fails. =1 should be an error, and the function already has a channel for it: run_audio_graph's why out-parameter. Set *why = "ONEBIT_AUDIO_AVX2=1 but this CPU has no AVX2" and return false -- the caller decides whether to abort, and the "forced arm was not forced" case becomes unmissable.
  2. Any non-x86 or non-GCC build ignores =1 silently. That is the appliance: #else return false with no message. If a check on the appliance sets =1 (say, to prove the baseline is what runs there), it gets baseline and no evidence that the request was dropped. At minimum print the same line from the #else branch.

Why this matters more than it looks: the number in this PR (1.29-1.32x) is an A/B measurement, and an A/B measurement is only as good as its ability to prove which arm ran. The -mavx2 flavour here cannot be verified from outside the binary -- there is no log line saying which one the process chose. One fprintf naming the chosen flavour (once, at load) makes every future measurement self-certifying, and costs nothing at runtime.

Two earlier asks stand unchanged: tools/ci/medians.py over the six values instead of "the two agreeing pairs", and one ASAN run of the fast flavour (256-bit tail loads are a read-width question that bit-exactness does not cover).

Still not answered from my last review: whether the tree's baseline flags already carry -ffast-math/-Ofast. If they do, "same statement order" rests on audio_parallel alone.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

build is green: 19m01s, after the licence-block fix. The TU dispatch is verified by CI, not only locally.

What that run actually proves, beyond "it compiles":

  • both flavour objects build from the same source under different flags (ONEBIT_AUDIO_TU_SUFFIX=base / avx2, -mavx2 on the fast one), so the inline namespace genuinely separates the symbol sets rather than silently collapsing them;
  • the dispatch TU links against both namespaces — a missing or misnamed flavour would be a link error on tu_avx2::run_audio_graph*, which is the failure I said to look for;
  • audio_parallel passes in CI, i.e. the ordered per-element sum is still bit-for-bit at 1/2/4 threads with the fast flavour compiled in. That is the arm that caught -mfma on audio: pick the GEMM inner loop at run time (AVX2+FMA where the box has it) #2584, and it is the reason neither flavour carries it.

Ledger for the five builds on this PR, since it is instructive: three apt-mirror failures (same run record, same three packages), one real one of mine (the 16-line licence block), one green. The infrastructure class is retryable and engine#1814 is closed around it; the licence-block class was avoidable with python3 tools/copyright.py --check before the first push, which is now a habit I would hand to the next new file.

The measured claim stands as stated: interleaved, same binary, 4 threads — 2952.0 / 2709.3 ms default vs 3796.9 / 3581.3 ms forced baseline, min-over-repeats 1.322×, and ONEBIT_AUDIO_AVX2=0 remains the baseline path the appliance takes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant