Repository navigation
audio: ONEBIT_AUDIO_FORCE_HOST forces the host path and names itself (engine#2366) - #2647
Conversation
…in host_why (engine#2366) The LM head is a 1x128x51865 GEMV issued through the device GEMM: 26.6 MB of weights read per token against a 512 B activation, measured at ~415 MB/s while the same array does 182 GFLOP/s on a D projection. Whether the device beats the host at M=1 has never been measurable, because there was no way to ask. The switch is a first-line check in offer(), not a bypass around it, so a forced run is counted and named like any other refusal and appears in the `host: N x <reason>` line whisper_transcribe already prints. A forced path that reported nothing would be indistinguishable from a device that took the work. The arm copies the accepting device (counting devices are per-instance) and resets its counters, because a copy inherits the previous run's 74 device and 48 attention reasons -- which is exactly how the first version of this test failed: it reported device=74 host=122 for a run that forced all 74 offers to the host. It now asserts device==0, host==98, host_why[forced]==74, and bit-identity with the pure host run.
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY This is the right instrument for the question and it is built the right way round: an env flag that makes the executor decline every device GEMM so the host path can be measured on the same graph, rather than a separate benchmark that measures a different thing. Three lines of behaviour plus an arm — and the arm is the part that makes it a measurement rather than a switch. Three asks:
And one note worth keeping: "the LM head in the warm encoder is a 1x128x51865 GEMV … and per-node timing says it costs 63–70 ms per token" — a GEMV at M=1 through a GEMM interface is the wrong shape for any device GEMM, so the likely finding is that the host wins. Naming the shape (1×128×51865) in the PR is what lets a reader predict the answer before the run, which is the mark of a well-posed experiment. |
|
READY — and the measurement that motivates it is the best kind of finding: an op that is bandwidth-starved at 415 MB/s on an array the same device streams at 182 GFLOP/s. 415 MB/s is two orders below what this box does, so the GEMV is not compute-bound and not even close to bandwidth-bound — it is round-trip-bound, which is what a 26.6 MB weight read per token looks like when the arithmetic is 512 bytes of activation. And ×24 of them explains ~1.7 s of the encoder by itself, which is the physical reason #2366's encoder numbers look the way they do. And the framing sentence is the one worth keeping: "the question 'is the device even better than the host here?' could not be asked — there was no switch". That is the general form of several things I have measured wrong tonight: a comparison needs a switch, and the absence of one is why a 415 MB/s path stayed in place. It belongs in the PR body exactly where it is. Three things I would pin:
Credited: measuring the op before adding the switch (the switch exists because of a number, not a hunch); reporting bytes and GB/s alongside milliseconds, which is what makes 415 MB/s legible at all; and "×24", the multiplier that turns a per-node figure into an encoder-scale claim. Not verified by me: the device run. The bandwidth arithmetic, the first-line placement and the env-knob precedence question are what I can check, and the first is right. |
|
VERDICT at 45f0c29: READY. Three lines of behaviour, one arm, and the arm contains the two things that make a forced-path switch trustworthy: the forced work names itself, and the forced run is proven bit-identical to the run it is supposed to imitate. Why this is the right shape for a measurement switchif (force_host) {
++dev->host;
++dev->host_why["forced host (ONEBIT_AUDIO_FORCE_HOST)"];
return false;
}"a forced path that reports nothing would be indistinguishable from a device that took the work" — that sentence is the whole design. A switch that silently redirects work turns every later observation into a guess about whether the switch was on; counting it in the same The arm asserts the three things that could each be falseThe third is the one I would have asked for and did not expect: forced-host ≡ real host, bit for bit. Without it the switch would produce a host path and the A/B against the device would compare two unverified computations; with it, any difference between device and forced-host is attributable to the device, which is the only way the number #2366 wants can mean anything.
And the comment on the arm — "the copy inherits dev's counters from the run above (74 device, 48 attention reasons); a fresh arm has to start from zero or it reports the previous run's work as its own" — is a bug caught before it shipped, in the fixture rather than the code. Counter inheritance across two runs in one process is exactly how a test reports yesterday's numbers as today's. The follow-up this enables, not a gap in itThis is the instrument, not the measurement: the open question in the comment — "whether the device beats the host at M=1 is an open question no switch could ask" — now has a switch. When the number lands, the useful form is the pair with its context: the LM head's 415 MB/s on device against the same 1×128×51865 GEMV forced onto the host, on the same graph, with Nothing else from me. The env parsing treats empty and |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- "a forced path that reported nothing would be indistinguishable from a device that accepted the work" is the rule, and this PR is the reference implementation of it
This is the pattern engine#2636 needs, and it is worth saying so on both: there, ONEBIT_AUDIO_AVX2=1 on a CPU without AVX2 prints "keeping the baseline" and then runs the baseline with exit 0; here, ONEBIT_AUDIO_FORCE_HOST=1 goes through offer() as its first line and lands in the host: N x <reason> line the tool already prints. Same problem (a forced arm that cannot prove it was forced), opposite outcome. ++dev->host_why["forced host (…)"] is the one line #2636 is missing.
The motivation is measured rather than asserted: a 1x128x51865 GEMV at 63-70 ms per token, b = 26.6 MB read in 64 ms = 415 MB/s on an array that does 182 GFLOP/s on a D projection, with the M-tile spent on padding at M=1. That is a bandwidth story, not a FLOP story, and it is why "is the device even better here?" was a fair question with no way to ask it.
Three asks:
- Assert the reason string, not only the count. The arm asserts
host == 98; the claim under test is "it reports itself". An arm that also greps the printedhost:line forforced host (ONEBIT_AUDIO_FORCE_HOST)is what pins the reporting -- otherwise a later change that incrementshostbut drops thehost_whyentry still passes, and the whole point of the PR goes with it. - Say whether
getenvis read peroffer()or cached once. If it is per call, a run that sets or unsets the variable mid-graph changes behaviour mid-graph: fine for a test, worth one sentence, because it also means the arm cannot be defeated by a stale cache and readers should know which it is. - State that the A/B is owed on #2366 and whether this switch is permanent. It is an env-gated, default-off, supported surface once it lands; if it is a temporary probe, mark it so (a
shortcut: …line) rather than leaving the next reader to guess whether it is safe to rely on.
Nothing else: making it the first line of offer() rather than a bypass keeps every other counter and reason intact, so a forced run is still a complete description of the offer.
|
Used the new switch on the number that motivated it, and it confirms the PR's suspicion with room to spare: on the host build the Pairs 1.5× and 1.35× apart, both favouring the host. Two caveats before anyone quotes that:
What I would do with that, and it is the decision the switch enables: the GEMV is 1×128×51865 — 512 bytes of activation against 26.6 MB of weights — so every engine reads 26.6 MB per token for 512 B of work. The fix is not "host or device", it is not re-reading the weights per token: that LM head is a tied embedding, and at M=1 the cheapest implementation is a lookup-free full read either way. So the honest summary of this measurement is "the switch works, the host is competitive at this shape, and the shape itself is the problem" — which is #2366's Conv/Softmax half all over again, in a different op. Not claimed: a device number. The token was held by another lane ( |
|
One question that decides how far this measurement reaches, and I do not want to guess at it. The It matters because the two readings differ by an order of magnitude in what the finding explains:
The phrase "the LM head in the warm encoder" reads like the first, and a vocab projection in a Whisper-style encoder would be unusual — which is why I would rather ask than attribute a number. A one-line answer either way makes this usable: the phase it belongs to, and whether the ×24 is graph occurrences or measured steps. If it is the encoder, the follow-up is large and cheap to test with this very switch: |
|
Answered my own question from the source, and the answer is the second reading — so the ×24 should not be read against the encoder.
A 51865-vocabulary projection is a decoder node ( That makes the correct arithmetic mine from the wrong phase, and it is worth stating so nobody repeats it:
So this PR's number is a decode-side result, and the encoder's 2.09 s is not explained by the LM head — which also means the Conv/Softmax half I measured earlier (95 ms and 15–19 ms) stands as the encoder's own business rather than rounding. Two things follow that are worth having in the body, since the phase is what a reader will get wrong:
|
…ngine#2652) A merge lands the head sha recorded when the merge was queued, so a commit pushed after that point is never merged and nothing says so: the PR shows merged, the branch shows the commit, and main does not have it. Four instances in one night; one of them (#2623's WEAK_AFTER_S / --distances) needed a restore PR (#2640) and was caught only because somebody counted symbols by hand. Building this found that the obvious check does not work. #2623's RECORDED diff contains 0 occurrences of WEAK_AFTER_S, because a post-merge push is by definition absent from the diff frozen at the merged head -- so a --pr content check is structurally blind to this bug. Run across the 59 merges of 11-15/16 it flagged 22 PRs, and every one of those was a line a later PR had legitimately deleted, not a lost commit. So the question is asked about commits: --branch REF git rev-list origin/main..REF -- the work of mine that has NOT landed. Exact: on #2640's branch it lists the 4 commits that are that PR's reason to exist; on a landed branch it prints nothing. --sha SHA is that commit an ancestor of main? Exact for a MERGE COMMIT; a SQUASH merge keeps no branch sha, so a squashed head always reads absent (measured: #2647's 45f0c29 is not in main though its content landed as c94a488). --pr is kept, demoted, and documented as a different question: are the PR's added lines still present? It reports the merge strategy first, and classifies each absent line as 'removed by a later commit' (git log -S) or genuinely lost, so it answers 'did this content survive' rather than 'did this commit land'. Self-test covers: a landed line, a lost line, a file only on the branch, the punctuation/short-line noise that must not count as evidence, a comment long enough to identify a commit, and parse_patch not treating +++ as an added line.
…ngine#2652) A merge lands the head sha recorded when the merge was queued, so a commit pushed after that point is never merged and nothing says so: the PR shows merged, the branch shows the commit, and main does not have it. Four instances in one night; one of them (#2623's WEAK_AFTER_S / --distances) needed a restore PR (#2640) and was caught only because somebody counted symbols by hand. Building this found that the obvious check does not work. #2623's RECORDED diff contains 0 occurrences of WEAK_AFTER_S, because a post-merge push is by definition absent from the diff frozen at the merged head -- so a --pr content check is structurally blind to this bug. Run across the 59 merges of 11-15/16 it flagged 22 PRs, and every one of those was a line a later PR had legitimately deleted, not a lost commit. So the question is asked about commits: --branch REF git rev-list origin/main..REF -- the work of mine that has NOT landed. Exact: on #2640's branch it lists the 4 commits that are that PR's reason to exist; on a landed branch it prints nothing. --sha SHA is that commit an ancestor of main? Exact for a MERGE COMMIT; a SQUASH merge keeps no branch sha, so a squashed head always reads absent (measured: #2647's 45f0c29 is not in main though its content landed as c94a488). --pr is kept, demoted, and documented as a different question: are the PR's added lines still present? It reports the merge strategy first, and classifies each absent line as 'removed by a later commit' (git log -S) or genuinely lost, so it answers 'did this content survive' rather than 'did this commit land'. Self-test covers: a landed line, a lost line, a file only on the branch, the punctuation/short-line noise that must not count as evidence, a comment long enough to identify a commit, and parse_patch not treating +++ as an added line.
What
ONEBIT_AUDIO_FORCE_HOST=1makes the audio executor decline every device GEMM, so the host path can be measured against it on the same graph. Three lines of behaviour plus an arm.Why (engine#2366)
The LM head in the warm encoder is a 1x128x51865 GEMV issued through the device GEMM (
offer()atnpu/audio_exec.cpp), and per-node timing says it costs 63–70 ms per token:415 MB/s for a streaming read, on an array that does 182 GFLOP/s on a
Dprojection. At M=1 the op is bandwidth-bound and the GEMM's M-tile is spent on padding.The question "is the device even better than the host here?" could not be asked — there was no switch, and the
host:line only names declined offers. This adds the way to ask.The switch reports itself
It is the first line of
offer(), not a bypass around it:So a forced run appears in the
host: N x <reason>linewhisper_transcribealready prints (tools/whisper_transcribe.cpp). A forced path that reported nothing would be indistinguishable from a device that accepted the work — which is the failure mode the rest of this tooling exists to prevent.The arm, and the bug it caught in my own test
tests/audio_exec_test.cppmirrors the existing declining-device arm: copy the accepting device, set the env var, assertThe first version failed, and the reason is worth keeping:
AudioGemm accepted = devcopies the counters of the run that just finished — 74 device and 48 attention reasons — so the forced run reporteddevice=74 host=122. The copy now resetsdevice,hostandhost_whybefore use. A test arm that inherits state is a test of the previous arm.Verified
Built and run with the real pack (
ONEBIT_TEST_WHISPER=~/scratch-packs/pi-demo/whisper-small), onorigin/main+ this change only. What this PR does not do: it does not change any dispatch decision and does not claim the host is faster — it makes that comparison possible. The measurement it enables is the next comment on #2366.