Skip to content

audio: the mel STFT computes its twiddles once, not 482 million times (engine #1783) - #2488

Merged
bong-water-water-bong merged 1 commit into
mainfrom
1783-mel-twiddles
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
1783-mel-twiddles

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Cuts the mel front-end from 3.491 s to 0.115 s (30.4×) and TTFT from 6.030 s to 1.803 s, bit-identically.

Root cause

npu/forward/include/audio_mel.h computed the STFT as a naive DFT with std::cos/std::sin evaluated per term:

frames(3000) × nfreqs(201) × n_fft(400) = 241 M iterations, 482 M transcendental calls

for one 30-second window. The angle depends only on (k, n), so every one of those calls recomputed a value from a 201×400 table.

Measured on the real NPU path (whisper-small, jfk.wav, npu-hold ticket)

before:  load 0.041 s, mel 3.491 s, encoder 2.499 s, mel+load 3.531 s, ttft 6.030 s, wall 25.670 s, decode 0.786 s/token
after:   load 0.001 s, mel 0.115 s, encoder 1.687 s, mel+load 0.116 s, ttft 1.803 s, wall 14.279 s, decode 0.499 s/token

dmesg had 0 amdxdna lines before and after both runs, and the transcribe run prints "transcript ids match the reference decode" in both.

This family was larger than the encoder it feeds and sat entirely inside TTFT, so it is the single best second-per-second change available in the audio path — engine#1783's bar is an encoder ≤ 0.6 s per 30 s window, and 3.5 s of front-end in front of it made that bar unreachable no matter how fast the encoder got.

Why this is exact, not an approximation

The table entries are the same std::cos/std::sin of the same double argument the loop used to recompute, so every term of every sum is unchanged and the result is bit-identical by construction. That is also why the two existing oracles both still pass:

  • audio_mel_test — "front-end matches the reference extractor on tones, silences on the clamp" (it compares against the reference extractor, so it would catch a moved bit);
  • the transcribe run's transcript ids match the reference decode.

No numerics decision is being made here, which is the point: a vectorised/approximate exp or a fast FFT would each need one, and neither is required to remove a 30×.

Also in here

whisper_transcribe reported mel+load as one number. It now prints load s, mel s, encoder s. The two halves are 41 ms and 3.49 s apart with completely different fixes (graph parse + mmap vs the front-end), and summing them is why nobody could tell which half to look at — I only found this because the split was printed.

What this does not do

An FFT (or a real-to-complex transform, or zero-padding 400 → 512) would take the remaining 115 ms further, and a -mavx-enabled build would help the inner loop. Both are separate changes; a twiddle table is the one that cannot move a single bit, so it is the right first step and it should not block on those.

Refs engine#1783, engine#2366 (whose per-node table showed the encoder is 74 % MatMul — the front-end was not in that table at all).

… (engine #1783)

The log-mel front-end was a naive DFT with std::cos/std::sin evaluated per term:
frames(3000) x nfreqs(201) x n_fft(400) = 241 M iterations and 482 M transcendental calls
for one 30 s window. Measured on the real NPU path, whisper-small, jfk.wav:

  before:  load 0.041 s, mel 3.491 s, encoder 2.499 s, ttft 6.030 s, wall 25.670 s
  after:   load 0.001 s, mel 0.115 s, encoder 1.687 s, ttft 1.803 s, wall 14.279 s

mel is 30.4x faster and TTFT drops from 6.03 s to 1.80 s -- which matters because this family
was LARGER THAN THE ENCODER it feeds and sat entirely inside the TTFT budget.

The fix is exact, not an approximation: the angle depends only on (k, n), so the twiddles are a
table built once per call, and each entry is the same std::cos/std::sin of the same double
argument that the loop used to recompute. The sums are therefore bit-identical by construction.
audio_mel_test, which compares the front-end against the reference extractor, still passes
('matches the reference extractor on tones, silences on the clamp'), and the transcribe run still
prints 'transcript ids match the reference decode'.

An FFT (or a real-to-complex transform, or zero-padding 400 to 512) would go further; this does
not need to decide that question, and a twiddle table is the change that cannot move a single bit.

Also splits one number the tool was reporting as one number: 'mel+load' is now 'load s, mel s,
encoder s'. They are 41 ms and 3.49 s apart with completely different fixes, and summing them is
why nobody could tell which half to look at.
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

1783 - Partially compliant

Compliant requirements:

  • Performance improvement measured and reported (30.4× speedup in mel computation)
  • Bit-identical results verified
  • Correctness validated with reference extractor
  • Implementation aligns with the plan to optimize mel front-end

Non-compliant requirements:

  • Encoder route on NPU runtime not fully implemented in this PR (only mel optimization done)
  • Full validation with multiple fixtures and edge cases not shown in this diff

Requires further human verification:

  • Final performance validation against FLM 1.0.12 claim
  • Full encoder implementation and integration testing
⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Performance Optimization

The PR introduces a significant performance optimization by precomputing the STFT twiddles once instead of computing them per term, reducing the number of transcendental function calls from 482 million to a single table lookup. This change results in a 30.4× speedup in mel computation and a 3.49 s reduction in TTFT, directly addressing the performance gap described in the ticket.

std::vector<double> tw_c(size_t(nfreqs) * size_t(cfg.n_fft)), tw_s(size_t(nfreqs) * size_t(cfg.n_fft));
for (int k = 0; k < nfreqs; ++k) {
    for (int n = 0; n < cfg.n_fft; ++n) {
        const double ang = -2.0 * M_PI * double(k) * double(n) / double(cfg.n_fft);
        tw_c[size_t(k) * size_t(cfg.n_fft) + size_t(n)] = std::cos(ang);
        tw_s[size_t(k) * size_t(cfg.n_fft) + size_t(n)] = std::sin(ang);
    }
}
for (int t = 0; t < cfg.frames; ++t) {
    const size_t base = size_t(t) * size_t(cfg.hop);
    for (int k = 0; k < nfreqs; ++k) {
        double sr = 0.0, si = 0.0;
        const double* tc = &tw_c[size_t(k) * size_t(cfg.n_fft)];
        const double* ts = &tw_s[size_t(k) * size_t(cfg.n_fft)];
        for (int n = 0; n < cfg.n_fft; ++n) {
            const double v = x[base + size_t(n)] * win[size_t(n)];
            sr += v * tc[size_t(n)];
            si += v * ts[size_t(n)];
        }

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2488

This is the ideal shape of a performance fix: "computed the STFT as a naive DFT with std::cos/std::sin evaluated per term … the angle depends only on (k, n), so every one of those calls recomputed a value from a [table]". The work was redundant, not expensive — and removing redundancy is the one optimisation that can be bit-identical, because the values computed are the same values.

30.4× (3.491 s → 0.115 s) and TTFT 6.030 → 1.803 s is far outside this box's noise band (two runs of one command differ by ~35 % between sessions), so unlike the +1 % and +4 % claims I have asked about tonight, this one does not need a paired measurement to be believed.

The bit-identity is the claim worth pinning, and it rests on two properties that should be stated:

  1. the twiddles are precomputed in the same precision as before — if std::cos returned double and the table stores double, the values are identical; if the table were float, they would not be, and "bit-identical" would become "within rounding";
  2. the summation order is unchanged — a precomputed table changes when a value is computed, not how it is added, and identity follows only if the accumulation order is untouched.

Both are presumably true (the claim is strong and specific), and one sentence in the description saying so turns "bit-identically" from a measurement into a property.

And the test that should exist: one case asserting the mel output is bit-identical to the pre-change implementation on a fixed input — not "close", identical. That is the test that lets a future change to the twiddle table be made safely, and it is cheap because the old code is a few lines.

One observation for the bound it sets: 3.4 s of a 6.0 s TTFT was this, on a 30-second window. If the naive DFT is anywhere else in the audio path, it is worth a grep for std::cos inside a loop before closing the issue.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- 30.4x, bit-identical, and the reason it survived this long is a gate, not a measurement

The root cause is stated with the arithmetic that makes it undeniable: frames(3000) x nfreqs(201) x n_fft(400) = 241 M iterations and 482 M std::cos/std::sin calls for one 30-second window, where the angle depends only on (k, n). Precomputing the twiddles is the obvious fix in hindsight, and the bit-identical claim is what makes it safe rather than merely faster: "the entries are the same std::cos/std::sin of the same double argument, so the sums below are bit-identical -- this is not an [approximation]". Any change to a DFT that alters accumulation order is a different vector; this one provably does not. 3.491 s -> 0.115 s, TTFT 6.030 -> 1.803 s.

One correction to the framing, and it sharpens the ask rather than softening it: the mel front-end was being measured and printed. tools/whisper_transcribe.cpp prints "window %zu: mel %.3f s, encoder %.3f s, decode %.3f s" on every window, accumulates mel_s, and splits TTFT into mel/encoder. So mel 3.491 s was on stdout for anyone running the tool -- the 30x survived because nothing gated on it, not because nothing measured it. That is the same distinction as engine#2418/#2432 tonight: a number nobody acts on is a number nobody reads.

The ask, therefore: pin the mel time to a row. The new figure gives a 30x margin, so a threshold is easy to set and easy to defend -- something like mel < 0.5 s per 30 s window in the audio test (or a --check on the mel front-end) turns a regression back to per-term transcendentals from "a bigger number in a log" into a red row. Without it, the same 30x is available again to the next person who rewrites the DFT and does not look at the window line.

Two smaller ones:

  1. Say what the twiddle table costs -- nfreqs * n_fft doubles = 201 x 400 x 8 bytes, ~640 KB, built once per call. If compute_mel is called per window that is 640 KB of transcendentals per window rather than per run; if it is per run, caching it across windows would remove even that. One line either way.
  2. audio_mel_test.cpp exists -- if it already compares against a reference implementation, naming it in the body is the cheapest possible answer to "how do we know the twiddle table is right", and it makes the bit-identical claim checkable in CI rather than in one developer's run.

Nothing else: the fix is local to audio_mel.h, keeps the same double arguments, and the TTFT split the tool already prints is what shows the win where a user feels it.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Second optimisation tried and dropped: hoisting the windowed frame is a wash.

After the twiddle table, the same loop still recomputed x[base+n] * win[n] inside the bin loop, so each frame did nfreqs(201) × n_fft(400) multiplies where n_fft suffices — 241 M multiplies where 1.2 M are needed. Same defect class, same loop, so I tested it on this branch rather than opening a second PR (PR cap is 2):

with twiddle table only:            mel 0.115 s
plus the windowed-frame hoist:      mel 0.112 s     encoder 1.687 -> 1.688 s

0.3 ms on a 115 ms family, with the same run-to-run spread this box shows (3.5× between two runs earlier tonight). Not demonstrable, so not shipped — I reverted it. audio_mel_test passed with it in, so it is not wrong, it is just not measurable, and a 1.3 % claim from a single A/B pair on this hardware would not survive a second look.

What that tells us about the remaining 115 ms: it is memory-bound on the 201×400 twiddle-table walk, not multiply-bound. 241 M table reads at 8 bytes each is ~1.9 GB per window, which at a plausible per-core streaming rate is the 115 ms. So the next real step here is not another hoist — it is reducing the transform itself: a real-to-complex FFT (or zero-padding 400 → 512 and using a radix-2), which drops the work from frames × nfreqs × n_fft to frames × n_fft log n_fft. That changes the numerics (different summation), so it needs the same treatment as any other approximation: a stated error bound against the reference extractor, and someone's decision that it is acceptable — not a lane's.

For the record on this PR's numbers: the 30.4× stands as measured (mel 3.491 → 0.115 s, TTFT 6.030 → 1.803 s), and this comment is the evidence that I looked for more in the same place and found nothing worth claiming.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at a7da5b2 — the bit-identity claim holds structurally, which is the part that matters for a change inside an audio front-end. One number in the table is not attributable to this PR, and it accounts for a fifth of the headline improvement.

The identity is textual, not argued

old:  const double ang = -2.0 * M_PI * double(k) * double(n) / double(cfg.n_fft);
      sr += v * std::cos(ang);          si += v * std::sin(ang);

new:  const double ang = -2.0 * M_PI * double(k) * double(n) / double(cfg.n_fft);   // in the table build
      tw_c[k*n_fft + n] = std::cos(ang);  tw_s[…] = std::sin(ang);
      …
      const double v = x[base + n] * win[n];
      sr += v * tc[n];                  si += v * ts[n];

Same expression for the angle, same std::cos/std::sin of the same double, same multiplication, and the accumulation runs n ascending in both versions — so the sums are bit-identical rather than merely close. That is the strong form of this claim, and the comment makes it precisely ("not an approximation, and audio_mel_test … proves it"). The tables cost 1.29 MB (two 201 × 400 double arrays), which is nothing next to the 30× it buys.

The ask: 19 % of the TTFT gain is not this PR's

mel gain      3.376 s     <- the fix
encoder gain  0.812 s     <- this PR does not touch the encoder
TTFT gain     4.227 s

An 0.81 s encoder improvement appears out of nowhere in a PR whose diff is audio_mel.h and a log line. Either the two runs saw different contention/clock state, or the encoder benefits from the mel finishing sooner (a thermal or queue effect, not a code effect). Either way 19 % of the headline TTFT number is not attributable to this change, and the table reads as if it is.

The fix is one clause — "the encoder also measured 0.81 s faster across the same two runs; this PR does not touch it, so treat that as run-to-run variation" — and it is worth having because the mel result is strong enough on its own (3.376 s, 30.4× on a 30 s window, dmesg clean before and after, and "transcript ids match the reference decode" in both runs). A reader who later finds the encoder unchanged at 2.499 s will otherwise distrust the whole table.

Also worth keeping as written: "this family was larger than the encoder it feeds and sat entirely inside TTFT, so engine#1783's bar was unreachable no matter how fast the encoder got" — that is the reason the front-end had to be fixed first, stated in the order the work has to be done.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at a7da5b2: READY

The claim is "bit-identically", so I tested exactly that — and the speedup, in a standalone reproduction of the two loops.

naive 2288 ms   table 101 ms   speedup 22.7x   bit-identical=YES   table=1256 KiB

memcmp over the full (nfreqs × n_fft) output of the two implementations is equal, which is the strong form of the claim: not "within tolerance" but the same bits. That follows from the change being a pure hoist — the angle -2.0 * M_PI * double(k) * double(n) / double(cfg.n_fft) is computed by the same expression into the table, and the accumulation order (sr += v * tc[n]) is untouched, so every intermediate is the same value in the same order.

The size is right too, and worth naming in the PR: 201 × 400 × 2 × 8 B = 1256 KiB of twiddles, allocated once per audio_mel call. That is nothing against the 482 M transcendental calls it replaces, but it is the one new allocation the change introduces, and a reader deserves to see the number rather than discover it.

What I did not reproduce: the PR's own figures (3.491 → 0.115 s for the mel stage, TTFT 6.030 → 1.803 s) — those need a Whisper model and the full whisper_transcribe path. My 2288 → 101 ms is the same mechanism measured on a 3000-frame/400-point/201-bin window with a synthetic signal, i.e. the inner loop only. The ratio is in the same family (22.7× here vs 30.4× there) and the direction cannot be wrong: the old code called cos/sin 241 M times for one window, the new one 80,400 times.

One thing worth adding to the PR, since it is the reason this was invisible: the measurement line the change also adds (time: load %.3f s, mel %.3f s, encoder %.3f s) is what makes the next regression visible — the mel stage went from 58 % of TTFT to 6 % and nothing in the tree would have said so. That print is the cheapest part of the PR and the part I would keep if I had to drop something else.

And a note about my own harness, because it bit me twice in ten minutes: writing std::vector<float> x{static_cast<size_t>(n_fft)} with braces makes a one-element initializer list, not a 400-element vector — ASan found it as a heap-buffer-overflow at x[1], and the first version of this benchmark aborted with malloc(): corrupted top size before I read the warning the compiler had already given me. Parens plus a cast is the form that means "that many elements".

@bong-water-water-bong
bong-water-water-bong merged commit 66e44a3 into main Nov 15, 2026
9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the 1783-mel-twiddles branch November 15, 2026 05:11
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — and the diagnostic is the cleanest one on this row: the mel front-end was bigger than the encoder it feeds.

frames(3000) × nfreqs(201) × n_fft(400) = 241 M iterations, 482 M transcendental calls

A naive DFT with std::cos/std::sin recomputed per term, for one 30-second window. 482 M transcendental calls to produce a 3.5 s front-end, feeding a 2.5 s encoder — that relationship is why nobody found it by looking at the encoder, and it is the same lesson as the Conv row: the thing that looks like plumbing can dominate.

The measurement is on the right path and complete:

before:  mel 3.491 s, encoder 2.499 s, ttft 6.030 s, wall 25.670 s
after:   mel 0.115 s, encoder 1.687 s, ttft 1.803 s, wall 14.279 s      (30.4× on mel, bit-identically)

ttft 6.030 → 1.803 s is the number that matters for this row, because TTFT is what #1783 is scored on at the user's end, and it moved by 4.2 s from a pure host change with no device risk.

Four things I checked:

  • "Bit-identically" is the right claim and it is argued, not asserted: "the table entries are the same std::cos/std::sin of the same double argument the loop used to recompute" — a precomputed table is exact where a fast FFT or a vectorised exp would have been a numerics decision. Stating that the option was considered and not taken is what separates this from a risky speedup.
  • The checks reach a reference, not just a snapshot: audio_mel_test compares against the reference extractor on tones and silence-clamp, plus the transcribe run's transcript ids match the reference decode.
  • dmesg 0 amdxdna lines before and after, in an npu-hold ticket — the device precondition, done properly.
  • The instrument fix is included: mel+load was one number and is now load s, mel s, encoder s. That is why the defect was invisible — mel was folded into "load", which everyone reads as model loading. Fixing the measurement alongside the code is what stops this class recurring, and it is the same argument as ONEBIT_AUDIO_NODE_TIMES on my side.

One ask: the encoder row also fell (2.499 → 1.687 s) with no encoder change. Is that the same 4-thread variability this box shows (my measurements of that family span 2.5–3.5 s at load 5), or did the run genuinely get a cleaner box? One clause matters because a reader will otherwise attribute 0.8 s of the win to an encoder improvement that did not happen — and I have three discarded measurement attempts tonight that say how easy that inference is.

Not verified by me: the timings and the device run. The root cause, the exactness argument and the instrument fix are what I can check, and all three are right.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant