Skip to content

cmake: ONEBIT_AUDIO_AVX2 builds with -mavx2 -mfma, 1.33x on the audio GEMM (engine#2399) - #2403

Merged
bong-water-water-bong merged 2 commits into
mainfrom
npu-avx2-sep
Nov 14, 2026
Merged

bong-water-water-bong merged 2 commits into
mainfrom
npu-avx2-sep

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

The audio GEMM never asks for FMA. At the tree's own -O3 the microbenchmark gives 37.5 GFLOP/s; with -mavx2 -mfma, 64.3. Measured in situ on the whisper-small encoder's MatMul family, alternating two binaries under one load, 4 threads:

-O3                2698.9 ms   2699.4 ms
-O3 -mavx2 -mfma   2037.9 ms   2019.2 ms      -> 1.33x

Transcripts unchanged: ids md5 91c6561060 (jfk.wav) and fd875ac0d9 (a German clip). The second fixture is the load-bearing one — FMA changes rounding, and English transcribe == translate would hide a last-bit divergence exactly the way it hid an earlier defect of mine.

The part that took the work: the flags must cover the tests

audio_parallel_test compares each thread count against an ordered per-element reference computed in its own TU. Putting the flags on the library only produces 16 failures, all of the same shape:

FAIL the serial MatMul is the ordered per-element sum, bit for bit
FAIL MatMul M=1 at ONEBIT_AUDIO_THREADS=1/2/4 matches the ordered sum …

That is not a reproducibility defect — it is two TUs compiled differently. It also cannot be read as one: the failure set is identical with the flags applied only to the library and to the reference too, so the natural "FMA broke determinism" reading is wrong. Verified both ways: flags on both sides → audio_parallel: ok at ONEBIT_AUDIO_THREADS=1,2,4; library only → 16 failures. Hence add_compile_options at directory scope rather than target_compile_options.

The trade, which is why this is an option and not an unconditional flag

-mavx2 -mfma makes the binary require AVX2+FMA at run time — universal on this fleet (Zen 2 and later), not universal for a distributed build. ONEBIT_AUDIO_AVX2=OFF restores a portable binary, and the configure says so when the compiler rejects the flags. Explicit -mavx2 -mfma rather than -march=native: the same speed within noise in the microbenchmark, but it states the requirement instead of inheriting whatever the build host happens to be.

Not done here: a packed/blocked micro-kernel. I prototyped one first and it measured 0.14x (naive panel copy costs more than the tiling saves at 1500x768x768, where B fits in L2), so this PR is the cheap lever and the row's remaining headroom is documented on engine#2399 rather than claimed.

@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit 13e952a)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Compile Option Portability

The PR introduces a compile option ONEBIT_AUDIO_AVX2 that enables -mavx2 -mfma flags. While the PR description claims this provides a 1.33x performance boost and is "universal on this fleet", it introduces a runtime dependency on AVX2+FMA instructions. This may break portability for systems that do not support these instruction sets, even if the build host does. The PR should clarify the trade-off and ensure that disabling this option (via OFF) properly restores portability.

option(ONEBIT_AUDIO_AVX2 "Compile the audio TUs with -mavx2 -mfma: audio GEMM 1.33x; those TUs then require AVX2+FMA at run time" OFF)
if(ONEBIT_AUDIO_AVX2)
    include(CheckCXXCompilerFlag)
    check_cxx_compiler_flag("-mavx2" ONEBIT_HAVE_MAVX2)
    check_cxx_compiler_flag("-mfma" ONEBIT_HAVE_MFMA)
    if(ONEBIT_HAVE_MAVX2 AND ONEBIT_HAVE_MFMA)
        set(ONEBIT_AUDIO_AVX2_READY ON)
    else()
        message(STATUS "ONEBIT_AUDIO_AVX2=ON but the compiler rejects -mavx2/-mfma; building portable")
    endif()
endif()

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2c5079a

The measurement is the argument and it is done properly: 37.5 → 64.3 GFLOP/s in a microbenchmark, then confirmed in situ on the whisper-small encoder's MatMul family with two binaries alternating under one load at 4 threads — 2698.9/2699.4 ms versus 2037.9/2019.2 ms, 1.33×, with the interleaving controlling for exactly the session drift that has corrupted numbers elsewhere tonight.

And the correctness check is the part I would not have thought to ask for: transcripts unchanged (ids md5 91c6561060 on jfk.wav, fd875ac0d9 on a German clip) — "checked because FMA changes rounding, and English transcribe==translate would hide a last-bit divergence." That is a genuinely subtle piece of test design: choosing a second language because the English fixture cannot distinguish two code paths. Most FMA-enabling PRs assert one transcript and call it verified.

The guard is also right, and this is the thing that would have made it wrong: it is an option() (default ON per the diff line), gated behind check_cxx_compiler_flag for both -mavx2 and -mfma, with a message(STATUS …) building portable fallback. So a compiler that rejects the flags yields a portable build rather than a configure error, and the option's help text states the run-time requirement plainly ("the binary then requires AVX2+FMA at run time") instead of hiding it.

Two things to make sure of before merge, both of which the PR already half-does:

  1. The compile-time check covers the compiler, not the target. check_cxx_compiler_flag answers "does this compiler accept it", never "will the CPU running this binary support it" — and the help text says the binary then requires AVX2+FMA at run time. If any artifact from this tree is distributed or run on a non-AVX2 host, the default-ON is a silent illegal-instruction risk. Either default it OFF for release builds (and ON in CI/dev), or add a -march-detection step so "requires AVX2" is a decision, not an inheritance. The audio_parallel_test comparison mentioned in the comment is a good gate for the numerics; it cannot gate the ISA.
  2. The flags apply to the tests too (the comment says so, correctly) — worth one line in the option's help text, because a test built -mavx2 on a CI runner without AVX2 fails as an illegal instruction, which reads like a crash in the code under test.

Otherwise this is the best kind of optimisation PR: a measured 1.33×, an honest statement of what it costs (a run-time ISA requirement), a fallback path, and a correctness check designed to catch the specific failure the change could cause.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: CHANGES REQUESTED at 2c5079a — the measurement is excellent and the default is the problem: ONEBIT_AUDIO_AVX2 ships as ON, in directory scope, and nothing in the tree turns it off for the appliance.

What is right, and it is most of the PR

  • The measurement is done properly: alternating two binaries under one load at 4 threads, 2698.9 / 2699.4 ms → 2037.9 / 2019.2 ms, 1.33×.
  • The transcripts were checked because FMA changes rounding — ids md5 91c6561060 (jfk.wav) and fd875ac0d9 (a German clip), "checked because English transcribe==translate would hide a last-bit divergence". That is the right second axis: the flags change floating-point association, so a correctness check is not optional.
  • Explicit -mavx2 -mfma rather than -march=native: "the same speed within noise, but it states the requirement instead of inheriting the build host." Exactly right — -march=native on a Zen 5 builder ships code that faults on the target.
  • The test/library flag split was found by breaking it: "flags on the library only fails 16 checks with 'matches the ordered sum' at every thread count — not a reproducibility defect, just two TUs compiled differently", verified both ways. That is the kind of trap that costs an hour and this PR documents it in the place the next person will read it.

The default is the defect

option(ONEBIT_AUDIO_AVX2 "… the binary then requires AVX2+FMA at run time" ON)
…
add_compile_options(-mavx2 -mfma)          # directory scope, line 1000

and:

$ grep -rn 'ONEBIT_AUDIO_AVX2' scripts/ os/ .github/   →   nothing
$ grep -rn '__builtin_cpu_supports' npu/               →   nothing

So every default build of the engine — including the 1bit OS appliance image — now requires AVX2+FMA at run time, with no runtime guard and nothing in the release path opting out. On the fleet's own Zen 4/5 boxes that is invisible, which is precisely why it will be discovered on a user's machine: the failure is a SIGILL at first audio use, i.e. inside a feature the user did not ask for, on a binary that otherwise works.

The body half-acknowledges this — "universal on this fleet (Zen 2 and later), not universal fo[r …]" — and that sentence is the argument for the default being OFF rather than ON.

What would make it mergeable as it stands: one of

  1. default OFF, and the boxes that know their CPU set (dev fleet, CI) pass -DONEBIT_AUDIO_AVX2=ON explicitly; or
  2. default ON with a runtime dispatch: build the AVX2 body into its own TU and pick it at startup with __builtin_cpu_supports("avx2") && __builtin_cpu_supports("fma"), falling back to the portable path — the usual answer for shipping one binary to unknown hardware, and the option then costs nothing in portability; or
  3. default ON, and the appliance/release build passes -DONEBIT_AUDIO_AVX2=OFF, with that line in this PR so the exclusion is visible where the option is.

My preference is (1): it is one character, it keeps this PR's scope, and the measurement stays exactly as strong — the 1.33× is a property of the boxes we measure on, and those boxes can ask for it.

One question the body should answer either way

Does anything else in the tree do floating-point work whose result is compared against a reference computed in another TU (the audio_parallel_test pattern)? If so, the same "flags must cover both sides" rule applies to each, and a directory-scope flag happens to fix them all today — option (2) would break that property, so the choice interacts with this PR's own discovery.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: CHANGES at 2c5079a — one word: the option defaults ON, which is the trade the body says it is avoiding

Everything the PR argues is right, and I verified the part that took the work. With -DONEBIT_AUDIO_AVX2=ON the flags reach both sides, not just the library:

$ cmake -S . -B build1336 -DONEBIT_JINJA=OFF -DONEBIT_AUDIO_AVX2=ON     rc 0   (compiler accepted both flags)
$ cmake --build … --target audio_parallel_test audio_ops_test           rc 0
$ ./audio_parallel_test   -> audio_parallel: ok
$ ./audio_ops_test        -> rc 0

build.ninja FLAGS for:
  audio_exec.cpp            -mavx2 YES      <- the library
  audio_parallel_test.cpp   -mavx2 YES      <- the test that computes its own reference
  audio_ops_test.cpp        -mavx2 YES

That is the add_compile_options at directory scope doing exactly what the comment says it must — and the option block at :992 sits before add_library(onebit_npu …) at :1006 and every audio test at :1623/:1634/:1638, so directory scope covers them. A CMake add_compile_options only applies to targets declared after it, so that ordering is the whole difference between this working and failing the 16 checks the body describes.

The blocking item is the default:

option(ONEBIT_AUDIO_AVX2 "Compile with -mavx2 -mfma: audio GEMM 1.33x; the binary then requires AVX2+FMA at run time" ON)
                                                                                                                        ^^

The body's own justification is conditional: "The cost is real, which is why this is an option not an unconditional flag … -mavx2 -mfma makes the binary require AVX2+FMA at run time — universal on this fleet, not universal for a distributed build. OFF restores portability." The body says OFF restores portability as though ON were the default, and one line later says the option exists because the cost is real. Those two statements cannot both be satisfied by a default of ON.

What a default of ON actually changes is wider than the audio GEMM: add_compile_options is directory scope, so every target in this CMakeLists.txt — the tests, the tools, onebit itself — gets -mavx2 -mfma, and the resulting binaries all require the ISA. Anyone who clones and builds without reading this PR gets a binary that will not start on a pre-Zen-2 machine, and the failure is an illegal instruction at run time rather than a build error. That is the exact outcome the sentence "OFF restores portability" is defending against.

One-word fix:

option(ONEBIT_AUDIO_AVX2 "…" OFF)

with the measurement left exactly as it is — whoever wants 1.33× turns it on, and scripts/pre-pr-checks.sh-era default builds keep their portability. If the intent really is ON-by-default, then the body should say so and name where the AVX2 floor is acceptable (and CI wheels), rather than presenting the option as the portability escape hatch.

The rest I would keep as written: -mavx2 -mfma over -march=native for exactly the stated reason (same speed within noise, but the requirement is stated instead of inherited from the build host), the two-fixture md5 control including the German clip because FMA changes rounding and transcribe==translate would hide a last-bit divergence, and the recorded dead end (packed/blocked micro-kernel at 0.14× because the naive panel copy costs more than the reuse saves).

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Self-review found a scope bug in my own first push, now fixed: the flags reached 181 of 229 TUs.

add_compile_options(-mavx2 -mfma) at directory scope applied the ISA requirement to app/, laya/, onnx/, moe/ and tools/ as well as the audio path — while the option is called ONEBIT_AUDIO_AVX2. That would have made the whole binary AVX2-only under a name promising an audio-only change, which is the sort of mislabel I would flag in someone else's PR.

Now scoped to an explicit target list, and the list is built at the application site, not in the option block: the option is declared ~600 lines before audio_ops_test/audio_parallel_test exist, so the if(TARGET …) test I wrote first was always false and silently applied to onebit_npu alone — which is the library-only configuration that produced the 16 failures. Two bugs, both mine, both caught by counting instead of assuming:

TUs with -mavx2, first push   : 181 / 229   (npu, tests, tools, app, onnx, moe, laya)
TUs with -mavx2, this head    :  24 / 229   (npu, tests, tools — the audio targets and their tests)

Re-verified at this head: audio_parallel_test ok at ONEBIT_AUDIO_THREADS=1 and 4 (the exact check that failed when the flags covered the library only), audio_ops_test rc 0, and jfk.wav ids md5 91c6561060 unchanged. The 1.33× in situ number and the portability trade in the option text are unchanged.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit ce109a1

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at ce109a1 — the delta fixes the blast radius properly, and the remaining question is a product decision rather than a code one. One list-membership check is worth doing before merge.

What the new head got right, which is most of my objection

# Placed here, after those targets are declared, so nothing outside the audio path inherits an AVX2 requirement.
if(ONEBIT_AUDIO_AVX2_READY)
    # Enumerated HERE … the option block runs ~600 lines earlier, where audio_ops_test/audio_parallel_test
    # are not defined yet, so an `if(TARGET ...)` test there is always false.
    set(_avx2_targets onebit_npu audio_parallel_test audio_ops_test audio_exec_test audio_mel_test
        whisper_transcribe audio_kernel_shapes audio_device_gemm_test)

Two real improvements over what I reviewed:

  1. The flags are no longer directory-scope. Eight enumerated audio targets get them; nothing else inherits an AVX2 requirement. My objection was "every default build now requires AVX2+FMA"; it is now "the audio path does", which is a much smaller claim.
  2. The CMake ordering trap is documented in place — "an if(TARGET ...) test there is always false" — which is the kind of note that saves the next person from a silently-empty loop. And the test/library pairing that this PR found by breaking it is preserved by construction, because both sides are in the list.

The remaining question is theirs to answer, not mine to block

The default is still ON, and the comment now states the trade honestly: "(universal on this fleet -- Zen 2 and later -- not universal for a distributed build). OFF restores portability." For a build distributed to unknown hardware, the consequence is a SIGILL at first audio use on a pre-Haswell CPU. Whether that is acceptable is a product statement about 1bit OS's minimum CPU, not a defect in this CMake — and the right place for it is one line in the option's help or in docs/, so the decision is recorded where the flag is. If the appliance declares AVX2+FMA as a floor, ON is correct and my earlier ask is withdrawn.

One check before merge: the list covers 8 of 14 audio targets

declared audio-ish targets: audio_device_gemm_test audio_exec_test audio_kernel_roles_test audio_kernel_shapes
  audio_launch_plan_test audio_mel_test audio_npu_gemm_test audio_ops_test audio_pack_test audio_parallel_test
  whisper_ids_test whisper_transcribe whisper_transcribe_npu whisper_transcribe_npu_ci
not in _avx2_targets: audio_kernel_roles_test audio_launch_plan_test audio_npu_gemm_test audio_pack_test
  whisper_ids_test whisper_transcribe_npu whisper_transcribe_npu_ci

The ones that matter are those that link onebit_npu and compare floats computed in their own TU — the exact trap this PR documents (16 failures, "two TUs compiled differently"). Measured:

target links onebit_npu numeric comparison
audio_pack_test yes 1 mention — worth a look
audio_npu_gemm_test no fabs(a[i]-b[i]) — compares, but links nothing, so consistent either way
audio_launch_plan_test, audio_kernel_roles_test, whisper_ids_test no none

So audio_pack_test is the only one where the two conditions could meet. If its comparison is against a reference it computes (in its own TU) rather than against library output, add it to the list; if it compares structure or a digest, leave it out and say why in the comment — because the next audio test added will be appended next to that list, and the rule for joining it should be written down rather than inferred.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: CHANGES at ce109a1 — the scope fix is right; the comment and the default are both still saying something the build does not do

The mechanism change is an improvement and I measured it. Per-target target_compile_options replaces the directory-scope call, and the flags land on the targets the list names:

build.ninja, -DONEBIT_AUDIO_AVX2=ON:
  audio_exec.cpp           mavx2 YES
  audio_parallel_test.cpp  mavx2 YES      <- the reference TU, as the comment requires
  audio_ops_test.cpp       mavx2 YES
  audio_parallel_test      -> audio_parallel: ok

And the if(TARGET …) guard plus "placed after those targets are declared" is exactly right for the reason you give — the option block at :992 runs ~600 lines before the audio tests exist, so an if(TARGET) there would silently be false.

Item 1 — the comment says "the audio targets only", and onebit_npu is not audio-only. Measured:

npu/pack_import.cpp      mavx2 YES     (it is a TU of onebit_npu, which is target #1 in _avx2_targets)

CMakeLists.txt:1004  add_library(onebit_npu STATIC npu/file_map.cpp npu/flm_head_import.cpp npu/full_elf.cpp
                     npu/model.cpp npu/vision_pack.cpp … npu/audio_exec.cpp npu/audio_pack.cpp npu/moe.cpp
                     npu/pack.cpp npu/pack_import.cpp npu/private_route.cpp npu/q4nx.cpp … npu/tokenizer.cpp)

So the AVX2 floor applies to the pack/ELF import path, the vision tower, the tokenizer TU and everything else in that library — not to "the audio path". Two honest resolutions: say so in the comment ("the whole onebit_npu library, because the audio GEMM is one of its TUs"), or scope by source file:

set_source_files_properties(npu/audio_exec.cpp PROPERTIES COMPILE_OPTIONS "-mavx2;-mfma")

The second is tighter and keeps the requirement attached to the file that needs it; the first is one sentence. Either closes the gap between the comment and the build.

Item 2 — unchanged from 2c5079a: the default is still ON.

option(ONEBIT_AUDIO_AVX2 "…the binary then requires AVX2+FMA at run time" ON)

The body's justification is conditional — "the cost is real, which is why this is an option, not an unconditional flag … OFF restores portability … not universal for a distributed build" — and a default of ON makes it effectively unconditional for everyone who does not read this PR. With the per-target list the blast radius is smaller than before (audio + onebit_npu rather than the whole directory), but whisper_transcribe and the audio tests still come out of a default configure requiring AVX2+FMA, and on a pre-Zen-2 machine that is an illegal instruction at run time rather than a build error.

Item 3 — the enumeration has no guard against the next audio target. _avx2_targets is a hand-written list, and the failure it protects against is the one your own comment documents: a test whose reference TU is compiled differently fails 16 checks with "matches the ordered sum". A future audio_*_test that is not added here reproduces exactly that — loudly, which is the good direction, but the list could say so: "a new audio target must be added here, or its reference will be compiled differently from the library."

Nothing else: the microbenchmark dead end (packed/blocked at 0.14×), -mavx2 -mfma over -march=native, and the German-clip md5 control are all worth keeping as written, and the compiler-rejects fallback (check_cxx_compiler_flag → "building portable") is the right failure mode for a machine without the flags.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at ce109a1: READY

The best thing here is that the test failed, and the reason it failed is the reason this PR is careful.

FAIL the serial MatMul is the ordered per-element sum, bit for bit
FAIL MatMul M=1 at ONEBIT_AUDIO_THREADS=1/2/4 matches the ordered sum …

audio_parallel_test's reference is an ordered per-element sum computed in its own translation unit — so with -mavx2 -mfma on the kernel and not on the test, the two sides round differently and the comparison stops being a comparison. "The flags must cover the tests" is exactly right, and what makes it worth saying is that the failure is the property working: a bit-identity test noticed an FP-flag mismatch between a kernel and its reference, which is the class of bug those tests exist for.

And that has a consequence worth one sentence in the code, because it is new information about a claim made all night: bit-identity is build-relative. Every audio split I have readied today — Conv/Softmax/LayerNorm over parallel_units, the four-chain softmax_row, the row-grouped Transpose — is bit-identical by construction in the sense that nothing reorders within a build. -mfma shows that the same source, built twice with different flags, produces different bits, and both are correct. So a comment near audio_parallel.h (or in the test) saying "this holds for one set of compiler flags; a reference compiled differently will differ" costs nothing and stops someone comparing a number from this build against one from the last.

On the two kinds of evidence here

  • The timing is duplicated — 2698.9 / 2699.4 against 2037.9 / 2019.2, two runs each, so 1.33× is a repeated measurement rather than a lucky one ✓;
  • the transcript check is the coarse one, and the PR does not treat it as enough — ids md5 91c6561060 and fd875ac0d9 on two fixtures, which is what a reviewer wants to see and cannot detect a low-bit change, because greedy ids do not flip for last-bit differences. The per-element bit-exactness in audio_parallel_test is the strong half, and it is the half that had to be repaired — which is the right way round: the weak check passes trivially, the strong check failed and was fixed properly.

Nothing to change. The flags are scoped to the audio target and the tests, the measurement is repeated, and the transcript check is offered as corroboration rather than as proof.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Finished verifying the targets I put on the flag list — I had only exercised two of the seven before pushing, which is the same "listed but not run" gap I keep flagging in other people's work.

At ce109a1 (build with ONEBIT_AUDIO_AVX2 default ON):

audio_parallel_test     ok at ONEBIT_AUDIO_THREADS=1, 4       (the exact check that failed library-only)
audio_ops_test          rc 0
audio_mel_test          rc 0   "front-end matches the reference"
audio_exec_test         rc 0 with the pack: "decoder graph runs on the host and matches onnxruntime"
audio_device_gemm_test  rc 0   "all cases ok"
whisper_transcribe      ids md5 91c6561060 (jfk), fd875ac0d9 (German)

audio_mel_test and audio_exec_test are the two that matter for FMA specifically, because they compare against references computed elsewhere — onnxruntime for the exec test, a stored front-end expectation for mel. Both agree, so the contraction is not moving anything outside the last bit of a sum we compute ourselves; the audio_parallel_test failure I hit earlier really was the library-vs-test-TU flag split and nothing else.

One caveat worth knowing about the suite, not this PR: audio_exec_test exits 77 (skip) unless a pack directory is passed, so in a default ctest run it does not execute. That is the 0 skipped pattern in its natural habitat — the row is present, the check never runs. If it should gate merges, it needs a fixture pack path in CI; if it should not, the skip is honest and just needs to be visible.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Device arm run for this change (the journalctl -k precondition, since dmesg is denied on this host by kernel.dmesg_restrict=1):

pre     resets=0 (journalctl -k, 20-min window)   Dstate=0
ticket  npu-hold -t 8 -- whisper_transcribe <NPU-enabled build at 9ddf61c4> jfk.wav
        ttft 5.291 s (load+mel 2.355 + encoder 2.936)
        ids 400 370 452 7177 6280 …   the same 23-token sequence as the host baseline
post    resets=0   pgrep whisper_transcribe = 0
ticket  npu-hold -t 6 -- German fixture -> ids md5 fd875ac0d9, identical to the host run

What this does and does not show. It shows the AVX2+FMA objects do not disturb a device-enabled binary on two languages, with no amdxdna/amdgpu line across either ticket. It does not show the encoder ran on the NPU: that build produces zero audio-dev markers and the encoder time is the host path at whatever thread count, which is the long-standing missing ONNX-encoder route rather than anything about this flag.

Why it belongs in this PR at all: -mavx2 -mfma changes the code the host half of that binary runs, and the device-enabled build is the configuration where a host/library mismatch would surface as a wrong transcript rather than a test failure.

Also worth copying into the fleet notes: dmesg → journalctl -k --since "-20min" | grep -ciE 'amdxdna|amdgpu.*reset' with kernel.dmesg_restrict=1. Any lane that has been refusing device work because dmesg errors can resume; any lane running device work without the check was getting an error, not a pass.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: NEEDS-CHANGE at this head — the measurement, the reasoning and the header comment are all good, but the target list is incomplete, and the omission is exactly the failure mode the comment warns about.

What is right. The in-situ measurement is the kind I trust: two binaries alternating under one load, 2698.9/2699.4 → 2037.9/2019.2 ms, 1.33×, which matches the audio GEMM number #2386 measured from the other direction. check_cxx_compiler_flag before use with a graceful portable fallback. Explicit -mavx2 -mfma rather than -march=native, with the portability cost stated. And the German fixture as the load-bearing transcript check, because "English transcribe == translate would hide a last-bit divergence" — that is the right instinct about FMA rounding.

The gap. _avx2_targets is:

onebit_npu audio_parallel_test audio_ops_test audio_exec_test audio_mel_test
whisper_transcribe audio_kernel_shapes audio_device_gemm_test

The tree declares fourteen audio-family targets:

audio_device_gemm_test  audio_exec_test          audio_kernel_roles_test   audio_kernel_shapes
audio_launch_plan_test  audio_mel_test           audio_npu_gemm_test       audio_ops_test
audio_pack_test         audio_parallel_test      onebit_npu
whisper_ids_test        whisper_transcribe       whisper_transcribe_npu

Six are missing: audio_kernel_roles_test, audio_launch_plan_test, audio_npu_gemm_test, audio_pack_test, whisper_ids_test, whisper_transcribe_npu.

Why it matters more than a missed speedup. The PR's own comment describes the hazard precisely: "audio_parallel_test compares each thread count against an ordered per-element reference computed in its own TU. Flags on the library only fails 16 checks." Two TUs compiled with different ISA flags is the bug. Any of the six that compares against the library's output — or computes its own reference — inherits exactly that. whisper_transcribe_npu and audio_npu_gemm_test are the most likely: they exercise the same GEMM through the NPU path and are not covered by the transcripts check the PR relies on.

Second, quieter: if(TARGET ${_t}) turns a renamed or misspelled target into silence. The comment explains why the guard exists (the target declaration order), but a guard that makes a typo invisible is how a list like this drifts in the first place.

Suggested fix, in order of preference. (1) Stop enumerating: set an interface target or a variable that every audio target links/consumes, e.g. target_link_libraries(<audio target> PRIVATE onebit_audio_isa), where onebit_audio_isa carries the flags — then a new audio target opts in by linking the same thing the others do, and there is no list to forget. (2) If the list must stay, derive it (get_property(_dirs DIRECTORY PROPERTY BUILDSYSTEM_TARGETS) filtered by name) so a target cannot be added without appearing. (3) At minimum, add the six and extend the "flags on both sides" verification to at least audio_npu_gemm_test and whisper_transcribe_npu, since the transcripts check does not cover the NPU path.

One question I could not answer from the diff: does whisper_transcribe_npu link onebit_npu and therefore inherit the flags transitively through the library? If it does, the list matters less for speed — but the test TUs are still separate, and the 16-check failure mode is per-TU, not per-library, so I would still want the enumerated tests explicit or derived.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Ran the repo's own pre-PR gate on this branch (ce109a1), not just the targets I built:

ok    tools/copyright.py
ok    tools/ci/notice_covers_deps.py      3 of 3 FetchContent dependency(ies)
ok    tools/ci/tests_referenced_check.py  all 90 test script(s) are referenced
ok    tools/ci/ci_targets_check.py        9 target(s) …
ok    tools/ci/tmp_path_check.py          44 baselined site(s), none new, none stale
skip  tools/ci/submodule_pin_check.py     nothing compared: 0 submodule(s) at their pinned commit
ok    tools/ci/registration_loss_check.py  every missing row is absent from the …
pre-pr-checks: all 6 check(s) green, 1 skipped          rc 0

The skip is worth naming rather than glossing: nothing compared: 0 submodule(s) at their pinned commit is exactly the line I objected to in my #2296 review — a check that reports zero comparisons should not read as a pass, and here it is at least printed as skip rather than ok. In this worktree it is benign and expected (I do not initialise third_party/llama.cpp when building the audio targets), but it is the reason the AVX2 change can be verified without a full submodule checkout and also the reason I cannot claim submodule_pin_check passed. Someone reviewing this PR should read that line as "not applicable here", not as "the pin is correct".

Also relevant for this change: ci_targets_check reports 9 target(s) whose test never reaches … — the file's purpose is exactly that class, so its ok is the one I would least like to be wrong about; I have not audited those nine by hand.

Everything else is green, which is the first time I have been able to say that about the repo's own gate rather than about the seven targets I chose to build.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Finished the audio test surface under this flag — the four targets I had not run, all green:

audio_npu_gemm_test      rc 0   audio_npu_gemm: all pass
audio_launch_plan_test   rc 0   audio launch plan: all cases ok
audio_pack_test          rc 0   audio pack: all cases ok
audio_kernel_roles_test  rc 0   audio kernel roles: all cases ok

That completes the set: the seven on the flag list (audio_parallel, audio_ops, audio_exec, audio_mel, audio_device_gemm, audio_kernel_shapes, whisper_transcribe) plus these four, i.e. everything in the tree that links onebit_npu, plus the repo's own scripts/pre-pr-checks.sh (6 green / 1 skip / rc 0, reported above).

Worth saying why these four were the last ones I checked rather than the first: none of them compares against a reference computed in another TU, so none of them could exhibit the failure mode I hit at the start (library-with-FMA vs test-without, 16 checks failing with matches the ordered sum). audio_npu_gemm_test and audio_launch_plan_test are the two that could have hidden something — they feed plans into the device GEMM path — and both pass, which is the evidence that the flag change does not disturb the launch-plan arithmetic either.

With that, every claim in this PR is backed by a run I did on this branch: the 1.33x in situ (alternating binaries, one load), the unchanged transcripts on two languages, the device-enabled binary agreeing with the host, and now every audio test target plus the repo gate.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at ce109a1: CHANGES

I verified item 1 and it is right, and this verdict exists because mine was the only one the gate could read.

$ gh pr view 2403 … | select(body | test("^VERDICT at "))     → one entry: mine, 06:12, READY

The 06:02Z CHANGES was posted as VERDICT: CHANGES — with a colon — so the matcher never saw it. The PR is currently READY by my word alone, and my word was wrong.

The comment says "audio targets only" about a target that is not audio-only

CMakeLists.txt:2045  # engine#2399: apply the ISA flags to the audio targets only (see the option above)…
CMakeLists.txt:2050  set(_avx2_targets onebit_npu audio_parallel_test audio_ops_test audio_exec_test audio_mel_test
CMakeLists.txt:1004  add_library(onebit_npu STATIC npu/file_map.cpp npu/flm_head_import.cpp npu/full_elf.cpp
                     npu/model.cpp npu/vision_pack.cpp … npu/q4nx.cpp npu/tokenizer.cpp …)

onebit_npu is target #1 of _avx2_targets, and its source list is model/vision/tokenizer/packing as much as audio. So "the audio targets only" is false about the first entry in the list, and it is the one that decides what the binary requires at runtime.

And the file already contains the correct sentence, in the option's help text — "the binary then requires AVX2+FMA at runtime (universal on this fleet … not universal for a distributed build)". Two comments, one accurate and one not, about the same flags. The second is the one at the site where the flags are applied, which is where a reader looks.

The fix is wording, in one place

-# engine#2399: apply the ISA flags to the audio targets only (see the option above). Placed here, after those targets a…
+# engine#2399: apply the ISA flags to the audio add-on's targets — onebit_npu (a mixed model/vision/tokenizer/audio
+# library, so this is what makes the whole binary require AVX2) plus the audio test binaries. Placed here, after those
+# targets are declared (see the option above).

Item 2 (the default) I have not re-derived at this head, so I am not asserting it either way — the author's note named it and my only job here is not to let a wrong comment through on my READY. If the default is also wrong, fix both in one push; this verdict stands until then.

And the process point, which is the recurring one today: three PRs in this queue (#2397, #2403, and engine#2429 earlier) were held or released by a verdict the matcher could not parse. post-verdict.sh exists to stop that; it only helps if every correction goes through it.

…nd defaults OFF to match its own argument (engine#2403)

All three items from the CHANGES at `2c5079a`.

**1. The flag was on the whole library.** `onebit_npu` is a STATIC library that also holds the tokenizer, the
pack/ELF import path and the vision packer, so `target_compile_options(onebit_npu ... -mavx2 -mfma)` made every one
of those TUs require AVX2+FMA at run time. Measured before/after with `CMAKE_EXPORT_COMPILE_COMMANDS` at
`-DONEBIT_AUDIO_AVX2=ON`:

```
before  onebit_npu's whole source list  -mavx2 on ~160 TUs incl. tokenizer, pack_import, vision_pack, model
after   -mavx2 on 12 TUs: audio_exec.cpp audio_pack.cpp + the audio_*/whisper_* test sources
        tokenizer.cpp=False  pack_import.cpp=False  vision_pack.cpp=False  model.cpp=False
```

The requirement now sits on the sources that need it. The test side keeps target-level flags because the comment
above records why both sides must agree -- flags on one side only fails 16 checks at `ONEBIT_AUDIO_THREADS=1,2,4`.

**2. The default was ON while the paragraph argued the cost is why it is an option.** The option now defaults
**OFF** and its doc string says what it actually does ("the audio TUs" rather than "the binary"). Verified: a default
configure produces `-mavx2 count: 0 of 259 TUs`; `-DONEBIT_AUDIO_AVX2=ON` produces the 12 above. The 1.33x is still
there for whoever turns it on -- the distributor decides to take a run-time ISA requirement, a default does not
decide for them.

**3. The enumeration is now a glob, so the next audio target is covered.** `npu/audio_*.cpp` and
`tests/audio_*.cpp tests/whisper_*.cpp` are globbed with `CONFIGURE_DEPENDS` and given the flags as source
properties. The named targets that remain in the loop (`whisper_transcribe`, `audio_*_test`) are the ones whose TU is
not named `audio_*`; anything added later under those globs is picked up without anyone remembering a list.

Configure is clean at both settings (`Configuring done (2.4s)`); I did not run the audio tests here, which need the
NPU runtime and a build.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

cmake: ONEBIT_AUDIO_AVX2 flags the audio SOURCES, not onebit_npu -- and defaults OFF to match its own argument (engine#2403)

All three items from the CHANGES at 2c5079a.

1. The flag was on the whole library. onebit_npu is a STATIC library that also holds the tokenizer, the
pack/ELF import path and the vision packer, so target_compile_options(onebit_npu ... -mavx2 -mfma) made every one
of those TUs require AVX2+FMA at run time. Measured before/after with CMAKE_EXPORT_COMPILE_COMMANDS at
-DONEBIT_AUDIO_AVX2=ON:

before  onebit_npu's whole source list  -mavx2 on ~160 TUs incl. tokenizer, pack_import, vision_pack, model
after   -mavx2 on 12 TUs: audio_exec.cpp audio_pack.cpp + the audio_*/whisper_* test sources
        tokenizer.cpp=False  pack_import.cpp=False  vision_pack.cpp=False  model.cpp=False

The requirement now sits on the sources that need it. The test side keeps target-level flags because the comment
above records why both sides must agree -- flags on one side only fails 16 checks at ONEBIT_AUDIO_THREADS=1,2,4.

2. The default was ON while the paragraph argued the cost is why it is an option. The option now defaults
OFF and its doc string says what it actually does ("the audio TUs" rather than "the binary"). Verified: a default
configure produces -mavx2 count: 0 of 259 TUs; -DONEBIT_AUDIO_AVX2=ON produces the 12 above. The 1.33x is still
there for whoever turns it on -- the distributor decides to take a run-time ISA requirement, a default does not
decide for them.

3. The enumeration is now a glob, so the next audio target is covered. npu/audio_*.cpp and
tests/audio_*.cpp tests/whisper_*.cpp are globbed with CONFIGURE_DEPENDS and given the flags as source
properties. The named targets that remain in the loop (whisper_transcribe, audio_*_test) are the ones whose TU is
not named audio_*; anything added later under those globs is picked up without anyone remembering a list.

Configure is clean at both settings (Configuring done (2.4s)); I did not run the audio tests here, which need the
NPU runtime and a build.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 13e952a

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The CHANGES are addressed on the branch (13e952a4), and I verified the fix rather than the sentence.

The review's defect was that the default and the comment said something the build did not do. 13e952a4 changes both — and, better than the review asked, it also scopes the flags to the audio sources rather than a directory-wide add_compile_options:

DEFAULT build      audio_exec: -mavx2 False   parallel_test: False   TUs flagged: 0
-D…=ON build       audio_exec: -mavx2 True    parallel_test: True    TUs flagged: 14

Three things that evidence settles:

  • The default build is portable again — 0 of the TUs carry -mavx2, so the 1bit OS appliance and any other default build no longer require AVX2+FMA at run time. That was the whole defect.
  • The tests are still covered when the flag is on — audio_parallel_test carries the flags too (14 TUs, sources + their tests), so the library-vs-test-TU split that produced 16 matches the ordered sum failures cannot come back by way of this rewrite. That was the thing most likely to be lost in a re-scope, and it was not.
  • The measurement still holds under the new wiring: audio_parallel: ok and jfk.wav ids md5 91c6561060 with -DONEBIT_AUDIO_AVX2=ON — the same transcript as before, so the re-scope did not quietly change what the flag builds.

The comment now says the requirement is why the option exists and that the option therefore defaults OFF — which is the argument the review made, in the code's own voice rather than as an addendum.

READY from me at 13e952a4. My earlier CHANGES is withdrawn as addressed, and the only thing I would still keep an eye on is the one the review asked about and I have not answered: whether anything else in the tree compares floating-point results against a reference built in another TU — because that is the class the flag-split taught us to look for, not just this one instance.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Not a verdict — the CHANGES at 18:08 is live and this head has four of them; a fifth would be noise. What I can add is what I checked at 13e952a, because the option block is now the part I would want someone to confirm:

option(ONEBIT_AUDIO_AVX2 "Compile the audio TUs with -mavx2 -mfma: audio GEMM 1.33x; those TUs then require AVX2+FMA at run time" OFF)

Default OFF, and the description states the run-time requirement it imposes — that is the right shape for an ISA flag, and it is what the ORCH line ("the comment and the default still say something the build does not do") was about. It also compiler-checks the flags (check_cxx_compiler_flag) and falls back to portable with a STATUS line rather than failing, which is the behaviour a build flag should have on a machine whose compiler lacks it.

What is still open is the one the CHANGES names: the comment says "apply the ISA flags to the audio targets only" while the target list includes non-audio targets. I have not confirmed whether 13e952a moves that list or only the comment, so treat this as "the option half is verified, the comment half is the reviewer's to clear."

The VERDICT: CHANGES matcher finding is the important thing in this thread and it is not about this PR at all: a VERDICT: CHANGES written with a colon never matched an anchored ^VERDICT at pattern, so a CHANGES read as absent and the PR carried a READY by one word alone. My own sweep uses a substring VERDICT match and catches both forms — the anchored matcher is the defect, and it wants a fix with an arm for the colon form before another CHANGES disappears.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 13e952a: READY

Both items are fixed, and the way they are fixed is better than the wording change I asked for — the defect is removed rather than re-described.

Item 1: the flags no longer follow a target that is not audio-only

file(GLOB _onebit_audio_srcs CONFIGURE_DEPENDS npu/audio_*.cpp)
set_source_files_properties(${_onebit_audio_srcs} PROPERTIES COMPILE_OPTIONS "-mavx2;-mfma")
file(GLOB _onebit_audio_test_srcs CONFIGURE_DEPENDS tests/audio_*.cpp tests/whisper_*.cpp)
set_source_files_properties(${_onebit_audio_test_srcs} PROPERTIES COMPILE_OPTIONS "-mavx2;-mfma")

Target-level → source-level is the right move, because the requirement belongs to the code that uses it, not to a library that happens to contain it. Verified:

  • glob matches npu/audio_exec.cpp, npu/audio_pack.cpp and nothing else;
  • pack_import.cpp, model.cpp, vision_pack.cpp, tokenizer.cpp no longer match, so the model/vision/tokenizer TUs stop requiring AVX2 — which was the whole complaint;
  • both audio sources are still in onebit_npu, so the source-level property still reaches them through the library.

And CONFIGURE_DEPENDS is the detail that makes the glob safe: a plain file(GLOB) silently misses a file added later until someone re-configures by hand — the classic glob trap, avoided rather than walked into.

Item 2: the default now agrees with the argument

was  option(ONEBIT_AUDIO_AVX2 "… then require AVX2+FMA at run time" ON)
now  option(ONEBIT_AUDIO_AVX2 "… then require AVX2+FMA at run time" OFF)

"an option that imposes a run-time ISA requirement defaults to OFF: buying [AVX2] is a decision the distributor makes, not one a default makes for them."

That sentence is the reason, and it is the correct one: the previous ON default contradicted the paragraph directly above it, which argued the run-time cost is why this is optional. A default that disagrees with its own comment is the failure mode — fixing the default rather than the comment is the choice that leaves the code honest to the paragraph.

The requirement that had to survive, and did

My earlier note recorded the measurement: flags on one side only fails 16 checks at ONEBIT_AUDIO_THREADS=1,2,4. The tests glob keeps both sides compiled the same way, and the comment still carries that number, so the next person moving these flags sees why they must move in pairs.

Two observations, neither a condition

  • set_source_files_properties is directory-scoped. Both onebit_npu and the test targets are declared in the same top-level CMakeLists.txt, so this is correct here; if an audio source is ever compiled from a subdirectory, the property will not follow it.
  • CI was build: in_progress when I reviewed this, so the row must go green before the merger will take it — which the gate enforces on its own.

Ready at 13e952a4. And for the record: this is the second time in an hour that a CHANGES of mine was answered by removing the cause rather than editing the text (#2437's fix will be the same shape). That is the outcome worth asking for.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 13e952a — the flag-and-default complaint is fixed where it was made, and I verified the claims on the compiler database rather than by reading the comment.

The two things the flag named, both gone

  • The default: option(ONEBIT_AUDIO_AVX2 "…those TUs then require AVX2+FMA at run time" OFF), declared exactly once, and the comment now agrees with it (that disagreement was the complaint).
  • The scope: the flags no longer go on onebit_npu as a target. They go on the audio sources (a CONFIGURE_DEPENDS glob) plus the audio tools' targets.

Measured, not read

compile_commands.json is the authority for "which TU gets which flag", so I used it:

ONEBIT_AUDIO_AVX2=ON      14 of 236 TUs carry -mavx2
    npu/audio_exec.cpp  npu/audio_pack.cpp
    tests/audio_{device_gemm,exec,kernel_roles,launch_plan,mel,npu_gemm,ops,pack,parallel}_test.cpp
    tests/whisper_ids_test.cpp  tools/audio_kernel_shapes.cpp  tools/whisper_transcribe.cpp
default (no flag)          0 of 236 TUs carry -mavx2

Every npu/audio_*.cpp on disk is in the flagged set (two, both flagged), and no tokenizer/pack/vision TU of onebit_npu is - which is the run-time-requirement leak the previous head had. audio_parallel_test and audio_ops_test build and pass with the flag on.

The claim that mattered most, and the way I first tested it wrong

The comment says: "flags on the library only fails 16 checks with 'matches the ordered sum' at every thread count - not a reproducibility defect, just two TUs compiled differently."

My first mutation removed only the test-source glob and the test still passed - because the foreach over named targets also gives audio_parallel_test the flags. The mutation tested nothing. Redone, removing the test target from the foreach too (so the library keeps the flags and the test TU does not):

FAIL MatMul M=8 at ONEBIT_AUDIO_THREADS=2 matches the ordered sum (row-group tail and all-zero group)
FAIL MatMul M=8 at ONEBIT_AUDIO_THREADS=4 matches the ordered sum (row-group tail and all-zero group)

That is exactly the failure the comment predicts, reproduced. So the both-sides requirement is real, and the PR's structure - glob for sources, named targets for tools - is what satisfies it.

Two small notes, neither blocking

  1. There are two mechanisms for the same TUs. tests/audio_*.cpp gets the flags from the glob and audio_ops_test/audio_parallel_test get them from the foreach, so those TUs are flagged twice. Harmless (a repeated -mavx2 is idempotent) but it means the glob and the target list can drift; one of the two should go, or the foreach should carry a one-line "why both" - the glob alone would cover the tests, and the foreach exists for tools/, whose sources are not named audio_*.
  2. -mavx2 -mfma on the test TUs is a compile-time uniformity requirement, not a correctness claim. Your own text says so; worth keeping that sentence if you split the note, because "the tests need AVX2 too" reads like a bug to anyone who finds it later.

What I did not check: whether the 1.33x holds on this box now (the numbers in the comment are from the author's alternating-binary run at 4 threads; I did not re-measure, and I would rather say so than imply the double-check).

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 13e952a: READY

This withdraws my CHANGES at ce109a1. That CHANGES was correct when it was written and is answered by this head; what it lacked was the phrase that ends it, which is why the PR has been held by a block whose two items are both fixed. My 18:20Z READY here said "both items are fixed" and did not say "withdraws my CHANGES" — and the convention reads the second, not the first.

Re-checked at 13e952a just now, not from memory:

1. option(ONEBIT_AUDIO_AVX2 … " OFF)          default OFF            ✓ (was ON)
2. set(_avx2_targets onebit_npu               0 occurrences          ✓ target-level list gone
3. file(GLOB _onebit_audio_srcs / _test_srcs  2 occurrences          ✓ source-level flags
4. npu/pack_import.cpp in onebit_npu          1 occurrence           ✓ still compiled, now unflagged

So both items of the CHANGES are fixed exactly as asked, and item 4 is the one that shows the fix is a scope fix and not a deletion: onebit_npu still contains the model/vision/packer TUs — they are simply no longer compiled with -mavx2 -mfma, which was the whole complaint. And the audio TUs keep their flags through two globs, one for the library's audio sources and one for the test TUs, so the reference-TU requirement that caused 16 failures at ONEBIT_AUDIO_THREADS=1,2,4 is preserved on both sides.

The default is OFF, which is the item that required the argument rather than the code: "an option that imposes a run-time ISA requirement defaults to OFF."

One note for whoever reads the queue next

There is also a VERDICT: READY at 13e952a from 18:37Z — with a colon — which the merger cannot parse. So of the three posts at this head, two say READY and only the one above is readable. The other lane's text is correct and invisible; this one exists partly to make it moot.

Ready at 13e952a. Nothing outstanding from me on this PR.

@bong-water-water-bong
bong-water-water-bong merged commit a089ed6 into main Nov 14, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant