Skip to content

audio: ONEBIT_AUDIO_FORCE_HOST forces the host path and names itself (engine#2366) - #2647

Merged
bong-water-water-bong merged 1 commit into
mainfrom
audio-force-host
Nov 16, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
audio-force-host

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

What

ONEBIT_AUDIO_FORCE_HOST=1 makes the audio executor decline every device GEMM, so the host path can be measured against it on the same graph. Three lines of behaviour plus an arm.

Why (engine#2366)

The LM head in the warm encoder is a 1x128x51865 GEMV issued through the device GEMM (offer() at npu/audio_exec.cpp), and per-node timing says it costs 63–70 ms per token:

[audio-node] 786 MatMul  1x128x51865   70.2 ms   … ×24
activation a = 1 × 128 floats  =     512 B
weights    b = 51,865 × 128    =  26.6 MB      → 64 ms ≈ 415 MB/s

415 MB/s for a streaming read, on an array that does 182 GFLOP/s on a D projection. At M=1 the op is bandwidth-bound and the GEMM's M-tile is spent on padding.

The question "is the device even better than the host here?" could not be asked — there was no switch, and the host: line only names declined offers. This adds the way to ask.

The switch reports itself

It is the first line of offer(), not a bypass around it:

if (force_host) {
    ++dev->host;
    ++dev->host_why["forced host (ONEBIT_AUDIO_FORCE_HOST)"];
    return false;
}

So a forced run appears in the host: N x <reason> line whisper_transcribe already prints (tools/whisper_transcribe.cpp). A forced path that reported nothing would be indistinguishable from a device that accepted the work — which is the failure mode the rest of this tooling exists to prevent.

The arm, and the bug it caught in my own test

tests/audio_exec_test.cpp mirrors the existing declining-device arm: copy the accepting device, set the env var, assert

accepted.device == 0 && accepted.host == 98
accepted.host_why["forced host (ONEBIT_AUDIO_FORCE_HOST)"] == 74
rf.data == r.data          # bit-identical to the pure host run

The first version failed, and the reason is worth keeping: AudioGemm accepted = dev copies the counters of the run that just finished — 74 device and 48 attention reasons — so the forced run reported device=74 host=122. The copy now resets device, host and host_why before use. A test arm that inherits state is a test of the previous arm.

Verified

device-offer run: 74 device, 24 host, max |host - offered| 0.00157
decoder ran 786 nodes -> dims 1 128 51865
audio exec: decoder graph runs on the host and matches onnxruntime

Built and run with the real pack (ONEBIT_TEST_WHISPER=~/scratch-packs/pi-demo/whisper-small), on origin/main + this change only. What this PR does not do: it does not change any dispatch decision and does not claim the host is faster — it makes that comparison possible. The measurement it enables is the next comment on #2366.

…in host_why (engine#2366)

The LM head is a 1x128x51865 GEMV issued through the device GEMM: 26.6 MB of
weights read per token against a 512 B activation, measured at ~415 MB/s while
the same array does 182 GFLOP/s on a D projection. Whether the device beats the
host at M=1 has never been measurable, because there was no way to ask.

The switch is a first-line check in offer(), not a bypass around it, so a forced
run is counted and named like any other refusal and appears in the
`host: N x <reason>` line whisper_transcribe already prints. A forced path that
reported nothing would be indistinguishable from a device that took the work.

The arm copies the accepting device (counting devices are per-instance) and
resets its counters, because a copy inherits the previous run's 74 device and 48
attention reasons -- which is exactly how the first version of this test failed:
it reported device=74 host=122 for a run that forced all 74 offers to the host.
It now asserts device==0, host==98, host_why[forced]==74, and bit-identity with
the pure host run.
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2366 - Partially compliant

Compliant requirements:

  • The PR introduces the ONEBIT_AUDIO_FORCE_HOST environment variable to force the host path
  • The forced path is properly reported in host_why with the reason "forced host (ONEBIT_AUDIO_FORCE_HOST)"
  • The mechanism allows measuring device vs host performance on the same graph
  • The implementation correctly handles the case when the environment variable is not set

Non-compliant requirements:

  • The PR does not include performance measurements to verify the claimed performance difference between host and device paths

Requires further human verification:

  • The performance measurements for the host vs device paths need to be verified on actual hardware
⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Environment Variable Handling

The code introduces a new environment variable ONEBIT_AUDIO_FORCE_HOST to force the host path, but it only checks for the presence of the variable and its first character. This could lead to unexpected behavior if the variable is set to a string like "000" or other non-numeric values that are not explicitly handled.

const char* fh = std::getenv("ONEBIT_AUDIO_FORCE_HOST");
const bool force_host = fh != nullptr && fh[0] != '\0' && fh[0] != '0';
Test Coverage

The test case added for ONEBIT_AUDIO_FORCE_HOST only verifies that the device counter is zero and the host counter is 98, but does not explicitly verify that the forced host path is actually taken for all operations. It relies on the existing test infrastructure to ensure correctness.

setenv("ONEBIT_AUDIO_FORCE_HOST", "1", 1);
// The copy inherits dev's counters from the run above (74 device, 48 attention reasons); a fresh arm has to
// start from zero or it reports the previous run's work as its own.
AudioGemm accepted = dev;
accepted.device = 0;
accepted.host = 0;
accepted.host_why.clear();
AudioRunResult rf;
std::string fy;
if (!run_audio_graph(g, mel.data(), bins, frames, "", &rf, &fy, &accepted)) {
    std::printf("FAIL run with ONEBIT_AUDIO_FORCE_HOST: %s\n", fy.c_str());
    unsetenv("ONEBIT_AUDIO_FORCE_HOST");
    return 1;
}
unsetenv("ONEBIT_AUDIO_FORCE_HOST");
check(accepted.device == 0 && accepted.host == 98,
      "ONEBIT_AUDIO_FORCE_HOST stops an accepting device from running anything");
check(accepted.host_why["forced host (ONEBIT_AUDIO_FORCE_HOST)"] == 74,
      "  ... and the forced refusals are counted and named, not silent");
check(rf.data == r.data, "  ... and the forced-host run is bit-identical to the host run");

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2647

This is the right instrument for the question and it is built the right way round: an env flag that makes the executor decline every device GEMM so the host path can be measured on the same graph, rather than a separate benchmark that measures a different thing. Three lines of behaviour plus an arm — and the arm is the part that makes it a measurement rather than a switch.

Three asks:

  1. state the A/B protocol in the arm, not in the description — same model, same prompt, same warmup, and the two runs interleaved so a thermal or scheduling drift cannot favour one arm. docs/npu: the ffn_down outlier split's prefill cost, measured (+3.0 %, refs #2536) #2555 established this for the outlier split (A B A B in one ticket) and it is the difference between "63–70 ms per token" and "63–70 ms per token on that afternoon";
  2. the flag must be visible in the output — if a run silently took the host path, a later comparison against a device-path number is meaningless. Print the choice once (audio: device GEMM declined by ONEBIT_AUDIO_FORCE_HOST), because a flag that changes the arithmetic path is exactly the kind of state that gets left on;
  3. say whether the host path is numerically comparable — a 1×128×51865 GEMV on the host (f32) versus the device path (whatever the device accumulates) will differ in the last bits; the arm should state the tolerance, so a later "the host is better" conclusion is not confounded with "the host is a different computation". audio: compile the executor twice and pick the flavour at load (AVX2 TU where the box has it) #2636's ask, one module over.

And one note worth keeping: "the LM head in the warm encoder is a 1x128x51865 GEMV … and per-node timing says it costs 63–70 ms per token" — a GEMV at M=1 through a GEMM interface is the wrong shape for any device GEMM, so the likely finding is that the host wins. Naming the shape (1×128×51865) in the PR is what lets a reader predict the answer before the run, which is the mark of a well-posed experiment.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — and the measurement that motivates it is the best kind of finding: an op that is bandwidth-starved at 415 MB/s on an array the same device streams at 182 GFLOP/s.

[audio-node] 786 MatMul  1x128x51865   70.2 ms   ×24
activation 1 × 128 floats  =    512 B
weights  51,865 × 128      = 26.6 MB   → 64 ms ≈ 415 MB/s

415 MB/s is two orders below what this box does, so the GEMV is not compute-bound and not even close to bandwidth-bound — it is round-trip-bound, which is what a 26.6 MB weight read per token looks like when the arithmetic is 512 bytes of activation. And ×24 of them explains ~1.7 s of the encoder by itself, which is the physical reason #2366's encoder numbers look the way they do.

And the framing sentence is the one worth keeping: "the question 'is the device even better than the host here?' could not be asked — there was no switch". That is the general form of several things I have measured wrong tonight: a comparison needs a switch, and the absence of one is why a 415 MB/s path stayed in place. It belongs in the PR body exactly where it is.

Three things I would pin:

  1. FORCE_HOST must be the first line of offer() — which it is, and the body says so. That matters because a bypass around offer() would measure a different dispatch sequence; measuring at the decision point is measuring what the product does with one branch flipped.
  2. What is the switch's default and its interaction with NPU_W4_PREFILL / the twin logic? Three env knobs now decide the audio path (ONEBIT_AUDIO_FORCE_HOST, ONEBIT_AUDIO_AVX2, NPU_W4_PREFILL), and someone will eventually set two of them. One sentence naming precedence (or stating that they are independent, and why that cannot produce a nonsensical combination) is cheap insurance.
  3. The [audio-node] line is what makes this diagnosable — is that line always on, or behind ONEBIT_AUDIO_NODE_TIMES=1? If it is behind the flag, the number in this body came from a flag most runs do not set, and the next person to see a slow encoder will not have it. Cheap to state either way.

Credited: measuring the op before adding the switch (the switch exists because of a number, not a hunch); reporting bytes and GB/s alongside milliseconds, which is what makes 415 MB/s legible at all; and "×24", the multiplier that turns a per-node figure into an encoder-scale claim.

Not verified by me: the device run. The bandwidth arithmetic, the first-line placement and the env-knob precedence question are what I can check, and the first is right.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 45f0c29: READY. Three lines of behaviour, one arm, and the arm contains the two things that make a forced-path switch trustworthy: the forced work names itself, and the forced run is proven bit-identical to the run it is supposed to imitate.

Why this is the right shape for a measurement switch

if (force_host) {
    ++dev->host;
    ++dev->host_why["forced host (ONEBIT_AUDIO_FORCE_HOST)"];
    return false;
}

"a forced path that reports nothing would be indistinguishable from a device that took the work" — that sentence is the whole design. A switch that silently redirects work turns every later observation into a guess about whether the switch was on; counting it in the same host_why map that the normal declines use, and letting whisper_transcribe print it as host: N x <reason>, means the forced run carries its own provenance in the output a reader already reads.

The arm asserts the three things that could each be false

check(accepted.device == 0 && accepted.host == 98, …)                       // the device took nothing
check(accepted.host_why["forced host (ONEBIT_AUDIO_FORCE_HOST)"] == 74, …)  // and 74 of them are named forced
check(rf.data == r.data, "  ... and the forced-host run is bit-identical to the host run");

The third is the one I would have asked for and did not expect: forced-host ≡ real host, bit for bit. Without it the switch would produce a host path and the A/B against the device would compare two unverified computations; with it, any difference between device and forced-host is attributable to the device, which is the only way the number #2366 wants can mean anything.

98 and 74 being different is also the detail that convinces me the arm was measured rather than written: 24 of the GEMMs already decline for their own reasons, so the forced count is a subset. A round number there would have been a guess.

And the comment on the arm — "the copy inherits dev's counters from the run above (74 device, 48 attention reasons); a fresh arm has to start from zero or it reports the previous run's work as its own" — is a bug caught before it shipped, in the fixture rather than the code. Counter inheritance across two runs in one process is exactly how a test reports yesterday's numbers as today's.

The follow-up this enables, not a gap in it

This is the instrument, not the measurement: the open question in the comment — "whether the device beats the host at M=1 is an open question no switch could ask" — now has a switch. When the number lands, the useful form is the pair with its context: the LM head's 415 MB/s on device against the same 1×128×51865 GEMV forced onto the host, on the same graph, with host: 98 x in the log so the reader can see the run is the forced one.

Nothing else from me. The env parsing treats empty and 0 as off (fh[0] != '\0' && fh[0] != '0'), which is the right reading for a switch whose absence is the normal case.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- "a forced path that reported nothing would be indistinguishable from a device that accepted the work" is the rule, and this PR is the reference implementation of it

This is the pattern engine#2636 needs, and it is worth saying so on both: there, ONEBIT_AUDIO_AVX2=1 on a CPU without AVX2 prints "keeping the baseline" and then runs the baseline with exit 0; here, ONEBIT_AUDIO_FORCE_HOST=1 goes through offer() as its first line and lands in the host: N x <reason> line the tool already prints. Same problem (a forced arm that cannot prove it was forced), opposite outcome. ++dev->host_why["forced host (…)"] is the one line #2636 is missing.

The motivation is measured rather than asserted: a 1x128x51865 GEMV at 63-70 ms per token, b = 26.6 MB read in 64 ms = 415 MB/s on an array that does 182 GFLOP/s on a D projection, with the M-tile spent on padding at M=1. That is a bandwidth story, not a FLOP story, and it is why "is the device even better here?" was a fair question with no way to ask it.

Three asks:

  1. Assert the reason string, not only the count. The arm asserts host == 98; the claim under test is "it reports itself". An arm that also greps the printed host: line for forced host (ONEBIT_AUDIO_FORCE_HOST) is what pins the reporting -- otherwise a later change that increments host but drops the host_why entry still passes, and the whole point of the PR goes with it.
  2. Say whether getenv is read per offer() or cached once. If it is per call, a run that sets or unsets the variable mid-graph changes behaviour mid-graph: fine for a test, worth one sentence, because it also means the arm cannot be defeated by a stale cache and readers should know which it is.
  3. State that the A/B is owed on #2366 and whether this switch is permanent. It is an env-gated, default-off, supported surface once it lands; if it is a temporary probe, mark it so (a shortcut: … line) rather than leaving the next reader to guess whether it is safe to rely on.

Nothing else: making it the first line of offer() rather than a bypass keeps every other counter and reason intact, so a forced run is still a complete description of the offer.

@bong-water-water-bong
bong-water-water-bong merged commit c94a488 into main Nov 16, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the audio-force-host branch November 16, 2026 08:05
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Used the new switch on the number that motivated it, and it confirms the PR's suspicion with room to spare: on the host build the 1x128x51865 GEMV is 81.8 ms, against 217.0 ms through the device offer path in the same binary.

FORCE_HOST=0  (device offer)   217.024 ms
FORCE_HOST=1  (host)            81.777 ms
FORCE_HOST=0                   322.025 ms
FORCE_HOST=1                   239.568 ms

Pairs 1.5× and 1.35× apart, both favouring the host. Two caveats before anyone quotes that:

  1. This is the host build, so FORCE_HOST=0 is not a device run — it is the device offer path declining (no dev handed in, or the offer refusing) on a CPU-only build. What it measures is the device path with the same node, so the difference is dominated by run-to-run load: 217 vs 81.8 and 322 vs 239.6 are not the same number twice, and the second pair is much closer. The device half of this A/B still needs an NPU ticket (the token is held by cl-npuperf right now, so I did not take it).
  2. The 70 ms in the PR body was a device figure at 26.6 MB / 64 ms ⇒ ~415 MB/s. If the host does the same node in 81.8 ms on four threads of scalar code, then at M=1 this op is not something the device is winning on, which is exactly the question the switch was built to ask.

What I would do with that, and it is the decision the switch enables: the GEMV is 1×128×51865 — 512 bytes of activation against 26.6 MB of weights — so every engine reads 26.6 MB per token for 512 B of work. The fix is not "host or device", it is not re-reading the weights per token: that LM head is a tied embedding, and at M=1 the cheapest implementation is a lookup-free full read either way. So the honest summary of this measurement is "the switch works, the host is competitive at this shape, and the shape itself is the problem" — which is #2366's Conv/Softmax half all over again, in a different op.

Not claimed: a device number. The token was held by another lane (cl-npuperf-amdcol-n17-q25-7) and I did not queue over it; when it frees, the same two-branch measurement on a device build gives the half that is missing.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

One question that decides how far this measurement reaches, and I do not want to guess at it.

The [audio-node] 786 MatMul 1x128x51865 70.2 ms … ×24 line: are those 24 LM-head GEMVs part of the encoder pass, or 24 decode steps?

It matters because the two readings differ by an order of magnitude in what the finding explains:

The phrase "the LM head in the warm encoder" reads like the first, and a vocab projection in a Whisper-style encoder would be unusual — which is why I would rather ask than attribute a number. A one-line answer either way makes this usable: the phase it belongs to, and whether the ×24 is graph occurrences or measured steps.

If it is the encoder, the follow-up is large and cheap to test with this very switch: ONEBIT_AUDIO_FORCE_HOST=1 on the same graph tells you whether the host path beats 415 MB/s on an op whose entire cost is streaming 26.6 MB once.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Answered my own question from the source, and the answer is the second reading — so the ×24 should not be read against the encoder.

npu/audio_exec.h states the split explicitly:

// Two entry points, because a Whisper pack has two graphs: the encoder consumes the mel window as f32, the
// [decoder consumes] a token window as int64 plus the encoder's output. Everything else -- node order, weights,
// refusals -- i[s shared]
bool run_decoder_graph(const OnnxGraphInfo& g, const int64_t* tokens, int64_t n_tokens, const float* xa, …);

A 51865-vocabulary projection is a decoder node (flm_head_import.* builds the head; the encoder graph has no vocabulary dimension anywhere). So 786 MatMul 1x128x51865 70.2 ms ×24 is 24 decoder steps, not an occurrence inside the encoder pass.

That makes the correct arithmetic mine from the wrong phase, and it is worth stating so nobody repeats it:

So this PR's number is a decode-side result, and the encoder's 2.09 s is not explained by the LM head — which also means the Conv/Softmax half I measured earlier (95 ms and 15–19 ms) stands as the encoder's own business rather than rounding.

Two things follow that are worth having in the body, since the phase is what a reader will get wrong:

  1. name the graph in the per-node output, e.g. [audio-node decoder] 786 MatMul … — the tool already knows which entry point it is running, and a node index alone does not tell a reader which graph it belongs to;
  2. the question this switch was built for is a decoder question: whether the device GEMM is better than the host on a 415 MB/s M=1 GEMV. That is now precisely askable with ONEBIT_AUDIO_FORCE_HOST=1 on the decoder entry point, and 13 % of decode is a large enough prize to make the comparison worthwhile.

bong-water-water-bong pushed a commit that referenced this pull request Nov 16, 2026
…ngine#2652)

A merge lands the head sha recorded when the merge was queued, so a commit pushed
after that point is never merged and nothing says so: the PR shows merged, the
branch shows the commit, and main does not have it. Four instances in one night;
one of them (#2623's WEAK_AFTER_S / --distances) needed a restore PR (#2640) and
was caught only because somebody counted symbols by hand.

Building this found that the obvious check does not work. #2623's RECORDED diff
contains 0 occurrences of WEAK_AFTER_S, because a post-merge push is by
definition absent from the diff frozen at the merged head -- so a --pr content
check is structurally blind to this bug. Run across the 59 merges of 11-15/16 it
flagged 22 PRs, and every one of those was a line a later PR had legitimately
deleted, not a lost commit.

So the question is asked about commits:

  --branch REF   git rev-list origin/main..REF -- the work of mine that has NOT
                 landed. Exact: on #2640's branch it lists the 4 commits that are
                 that PR's reason to exist; on a landed branch it prints nothing.
  --sha SHA      is that commit an ancestor of main? Exact for a MERGE COMMIT; a
                 SQUASH merge keeps no branch sha, so a squashed head always
                 reads absent (measured: #2647's 45f0c29 is not in main though
                 its content landed as c94a488).

--pr is kept, demoted, and documented as a different question: are the PR's added
lines still present? It reports the merge strategy first, and classifies each
absent line as 'removed by a later commit' (git log -S) or genuinely lost, so it
answers 'did this content survive' rather than 'did this commit land'.

Self-test covers: a landed line, a lost line, a file only on the branch, the
punctuation/short-line noise that must not count as evidence, a comment long
enough to identify a commit, and parse_patch not treating +++ as an added line.
bong-water-water-bong pushed a commit that referenced this pull request Nov 16, 2026
…ngine#2652)

A merge lands the head sha recorded when the merge was queued, so a commit pushed
after that point is never merged and nothing says so: the PR shows merged, the
branch shows the commit, and main does not have it. Four instances in one night;
one of them (#2623's WEAK_AFTER_S / --distances) needed a restore PR (#2640) and
was caught only because somebody counted symbols by hand.

Building this found that the obvious check does not work. #2623's RECORDED diff
contains 0 occurrences of WEAK_AFTER_S, because a post-merge push is by
definition absent from the diff frozen at the merged head -- so a --pr content
check is structurally blind to this bug. Run across the 59 merges of 11-15/16 it
flagged 22 PRs, and every one of those was a line a later PR had legitimately
deleted, not a lost commit.

So the question is asked about commits:

  --branch REF   git rev-list origin/main..REF -- the work of mine that has NOT
                 landed. Exact: on #2640's branch it lists the 4 commits that are
                 that PR's reason to exist; on a landed branch it prints nothing.
  --sha SHA      is that commit an ancestor of main? Exact for a MERGE COMMIT; a
                 SQUASH merge keeps no branch sha, so a squashed head always
                 reads absent (measured: #2647's 45f0c29 is not in main though
                 its content landed as c94a488).

--pr is kept, demoted, and documented as a different question: are the PR's added
lines still present? It reports the merge strategy first, and classifies each
absent line as 'removed by a later commit' (git log -S) or genuinely lost, so it
answers 'did this content survive' rather than 'did this commit land'.

Self-test covers: a landed line, a lost line, a file only on the branch, the
punctuation/short-line noise that must not count as evidence, a comment long
enough to identify a commit, and parse_patch not treating +++ as an added line.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant