Repository navigation
audio: the mel STFT computes its twiddles once, not 482 million times (engine #1783) - #2488
Conversation
… (engine #1783) The log-mel front-end was a naive DFT with std::cos/std::sin evaluated per term: frames(3000) x nfreqs(201) x n_fft(400) = 241 M iterations and 482 M transcendental calls for one 30 s window. Measured on the real NPU path, whisper-small, jfk.wav: before: load 0.041 s, mel 3.491 s, encoder 2.499 s, ttft 6.030 s, wall 25.670 s after: load 0.001 s, mel 0.115 s, encoder 1.687 s, ttft 1.803 s, wall 14.279 s mel is 30.4x faster and TTFT drops from 6.03 s to 1.80 s -- which matters because this family was LARGER THAN THE ENCODER it feeds and sat entirely inside the TTFT budget. The fix is exact, not an approximation: the angle depends only on (k, n), so the twiddles are a table built once per call, and each entry is the same std::cos/std::sin of the same double argument that the loop used to recompute. The sums are therefore bit-identical by construction. audio_mel_test, which compares the front-end against the reference extractor, still passes ('matches the reference extractor on tones, silences on the clamp'), and the transcribe run still prints 'transcript ids match the reference decode'. An FFT (or a real-to-complex transform, or zero-padding 400 to 512) would go further; this does not need to decide that question, and a twiddle table is the change that cannot move a single bit. Also splits one number the tool was reporting as one number: 'mel+load' is now 'load s, mel s, encoder s'. They are 41 ms and 3.49 s apart with completely different fixes, and summing them is why nobody could tell which half to look at.
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY This is the ideal shape of a performance fix: "computed the STFT as a naive DFT with 30.4× (3.491 s → 0.115 s) and TTFT 6.030 → 1.803 s is far outside this box's noise band (two runs of one command differ by ~35 % between sessions), so unlike the +1 % and +4 % claims I have asked about tonight, this one does not need a paired measurement to be believed. The bit-identity is the claim worth pinning, and it rests on two properties that should be stated:
Both are presumably true (the claim is strong and specific), and one sentence in the description saying so turns "bit-identically" from a measurement into a property. And the test that should exist: one case asserting the mel output is bit-identical to the pre-change implementation on a fixed input — not "close", identical. That is the test that lets a future change to the twiddle table be made safely, and it is cheap because the old code is a few lines. One observation for the bound it sets: 3.4 s of a 6.0 s TTFT was this, on a 30-second window. If the naive DFT is anywhere else in the audio path, it is worth a |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- 30.4x, bit-identical, and the reason it survived this long is a gate, not a measurement
The root cause is stated with the arithmetic that makes it undeniable: frames(3000) x nfreqs(201) x n_fft(400) = 241 M iterations and 482 M std::cos/std::sin calls for one 30-second window, where the angle depends only on (k, n). Precomputing the twiddles is the obvious fix in hindsight, and the bit-identical claim is what makes it safe rather than merely faster: "the entries are the same std::cos/std::sin of the same double argument, so the sums below are bit-identical -- this is not an [approximation]". Any change to a DFT that alters accumulation order is a different vector; this one provably does not. 3.491 s -> 0.115 s, TTFT 6.030 -> 1.803 s.
One correction to the framing, and it sharpens the ask rather than softening it: the mel front-end was being measured and printed. tools/whisper_transcribe.cpp prints "window %zu: mel %.3f s, encoder %.3f s, decode %.3f s" on every window, accumulates mel_s, and splits TTFT into mel/encoder. So mel 3.491 s was on stdout for anyone running the tool -- the 30x survived because nothing gated on it, not because nothing measured it. That is the same distinction as engine#2418/#2432 tonight: a number nobody acts on is a number nobody reads.
The ask, therefore: pin the mel time to a row. The new figure gives a 30x margin, so a threshold is easy to set and easy to defend -- something like mel < 0.5 s per 30 s window in the audio test (or a --check on the mel front-end) turns a regression back to per-term transcendentals from "a bigger number in a log" into a red row. Without it, the same 30x is available again to the next person who rewrites the DFT and does not look at the window line.
Two smaller ones:
- Say what the twiddle table costs --
nfreqs * n_fftdoubles = 201 x 400 x 8 bytes, ~640 KB, built once per call. Ifcompute_melis called per window that is 640 KB of transcendentals per window rather than per run; if it is per run, caching it across windows would remove even that. One line either way. audio_mel_test.cppexists -- if it already compares against a reference implementation, naming it in the body is the cheapest possible answer to "how do we know the twiddle table is right", and it makes the bit-identical claim checkable in CI rather than in one developer's run.
Nothing else: the fix is local to audio_mel.h, keeps the same double arguments, and the TTFT split the tool already prints is what shows the win where a user feels it.
|
Second optimisation tried and dropped: hoisting the windowed frame is a wash. After the twiddle table, the same loop still recomputed 0.3 ms on a 115 ms family, with the same run-to-run spread this box shows (3.5× between two runs earlier tonight). Not demonstrable, so not shipped — I reverted it. What that tells us about the remaining 115 ms: it is memory-bound on the 201×400 twiddle-table walk, not multiply-bound. 241 M table reads at 8 bytes each is ~1.9 GB per window, which at a plausible per-core streaming rate is the 115 ms. So the next real step here is not another hoist — it is reducing the transform itself: a real-to-complex FFT (or zero-padding 400 → 512 and using a radix-2), which drops the work from For the record on this PR's numbers: the 30.4× stands as measured ( |
|
VERDICT: READY at The identity is textual, not arguedSame expression for the angle, same The ask: 19 % of the TTFT gain is not this PR'sAn 0.81 s encoder improvement appears out of nowhere in a PR whose diff is The fix is one clause — "the encoder also measured 0.81 s faster across the same two runs; this PR does not touch it, so treat that as run-to-run variation" — and it is worth having because the mel result is strong enough on its own (3.376 s, 30.4× on a 30 s window, Also worth keeping as written: "this family was larger than the encoder it feeds and sat entirely inside TTFT, so engine#1783's bar was unreachable no matter how fast the encoder got" — that is the reason the front-end had to be fixed first, stated in the order the work has to be done. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at a7da5b2: READY
The claim is "bit-identically", so I tested exactly that — and the speedup, in a standalone reproduction of the two loops.
naive 2288 ms table 101 ms speedup 22.7x bit-identical=YES table=1256 KiB
memcmp over the full (nfreqs × n_fft) output of the two implementations is equal, which is the strong form of the claim: not "within tolerance" but the same bits. That follows from the change being a pure hoist — the angle -2.0 * M_PI * double(k) * double(n) / double(cfg.n_fft) is computed by the same expression into the table, and the accumulation order (sr += v * tc[n]) is untouched, so every intermediate is the same value in the same order.
The size is right too, and worth naming in the PR: 201 × 400 × 2 × 8 B = 1256 KiB of twiddles, allocated once per audio_mel call. That is nothing against the 482 M transcendental calls it replaces, but it is the one new allocation the change introduces, and a reader deserves to see the number rather than discover it.
What I did not reproduce: the PR's own figures (3.491 → 0.115 s for the mel stage, TTFT 6.030 → 1.803 s) — those need a Whisper model and the full whisper_transcribe path. My 2288 → 101 ms is the same mechanism measured on a 3000-frame/400-point/201-bin window with a synthetic signal, i.e. the inner loop only. The ratio is in the same family (22.7× here vs 30.4× there) and the direction cannot be wrong: the old code called cos/sin 241 M times for one window, the new one 80,400 times.
One thing worth adding to the PR, since it is the reason this was invisible: the measurement line the change also adds (time: load %.3f s, mel %.3f s, encoder %.3f s) is what makes the next regression visible — the mel stage went from 58 % of TTFT to 6 % and nothing in the tree would have said so. That print is the cheapest part of the PR and the part I would keep if I had to drop something else.
And a note about my own harness, because it bit me twice in ten minutes: writing std::vector<float> x{static_cast<size_t>(n_fft)} with braces makes a one-element initializer list, not a 400-element vector — ASan found it as a heap-buffer-overflow at x[1], and the first version of this benchmark aborted with malloc(): corrupted top size before I read the warning the compiler had already given me. Parens plus a cast is the form that means "that many elements".
|
READY — and the diagnostic is the cleanest one on this row: the mel front-end was bigger than the encoder it feeds. A naive DFT with The measurement is on the right path and complete:
Four things I checked:
One ask: the Not verified by me: the timings and the device run. The root cause, the exactness argument and the instrument fix are what I can check, and all three are right. |
Cuts the mel front-end from 3.491 s to 0.115 s (30.4×) and TTFT from 6.030 s to 1.803 s, bit-identically.
Root cause
npu/forward/include/audio_mel.hcomputed the STFT as a naive DFT withstd::cos/std::sinevaluated per term:for one 30-second window. The angle depends only on
(k, n), so every one of those calls recomputed a value from a 201×400 table.Measured on the real NPU path (whisper-small, jfk.wav, npu-hold ticket)
dmesghad 0 amdxdna lines before and after both runs, and the transcribe run prints "transcript ids match the reference decode" in both.This family was larger than the encoder it feeds and sat entirely inside TTFT, so it is the single best second-per-second change available in the audio path — engine#1783's bar is an encoder ≤ 0.6 s per 30 s window, and 3.5 s of front-end in front of it made that bar unreachable no matter how fast the encoder got.
Why this is exact, not an approximation
The table entries are the same
std::cos/std::sinof the samedoubleargument the loop used to recompute, so every term of every sum is unchanged and the result is bit-identical by construction. That is also why the two existing oracles both still pass:audio_mel_test— "front-end matches the reference extractor on tones, silences on the clamp" (it compares against the reference extractor, so it would catch a moved bit);transcript ids match the reference decode.No numerics decision is being made here, which is the point: a vectorised/approximate exp or a fast FFT would each need one, and neither is required to remove a 30×.
Also in here
whisper_transcribereportedmel+loadas one number. It now printsload s, mel s, encoder s. The two halves are 41 ms and 3.49 s apart with completely different fixes (graph parse + mmap vs the front-end), and summing them is why nobody could tell which half to look at — I only found this because the split was printed.What this does not do
An FFT (or a real-to-complex transform, or zero-padding 400 → 512) would take the remaining 115 ms further, and a
-mavx-enabled build would help the inner loop. Both are separate changes; a twiddle table is the one that cannot move a single bit, so it is the right first step and it should not block on those.Refs engine#1783, engine#2366 (whose per-node table showed the encoder is 74 %
MatMul— the front-end was not in that table at all).