Skip to content

whisper_transcribe: recordings longer than 30 s, window by window, and no 60-step cap (#1783) - #2365

Merged
bong-water-water-bong merged 2 commits into
mainfrom
1bit/1783-whisper-windows
Nov 13, 2026
Merged

bong-water-water-bong merged 2 commits into
mainfrom
1bit/1783-whisper-windows

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Refs #1783 (close table item 4b, plus a defect found while checking item 3). cl-1051, support; pi-demo leads.

Why

Two defects in tools/whisper_transcribe, both measured against onnxruntime 1.31.1 CPU running the pack's own graphs on WhisperFeatureExtractor's log-mel:

  1. Only the first 30 s of a recording were read. The log-mel was also computed over the whole recording before the window was taken. Whisper normalises a log-mel against its own maximum, so this is a different input: on fx-5min's first window the result differs from WhisperFeatureExtractor by up to 8.5e-3, against 1e-5 when each window gets its own mel. Item 4b (the 5-minute recording FLM quotes) had no path at all.
  2. The greedy loop stopped after 60 steps. fx-5min's first window needs 66 tokens, and the tail was dropped with no message.

Change

  • The recording is cut into consecutive 30 s windows. Each window is padded to 30 s, given its own log-mel and encoded, then decoded with the same prompt and no carried text. This is the plain chunked form: there are no timestamps, so nothing is stitched across a boundary. The graphs load once.
  • Decoding runs to <|endoftext|> or a full token window. When the window fills first, the tool says so.
  • Output:
    • One-window recording: the same ids and time: line as before, plus an audio: line. The ttft: line now says load+mel, because the graph load now comes before the first mel.
    • Several windows: one line per window, then per window: encoder …, wall …; recording … in … (RTF …).
  • Merged with npu: the whisper encoder's projections and conv stems on the NPU (Refs #1783) #2361 (cl-856's encoder route). The device hook is opened once, and encoder split: device_gemm=… + host_remainder=… prints per window, taken from the difference of device_ms().

Measured (whisper-small, thermal-run, 4 CPUs; box load 6–15)

fixture windows ids vs reference tokens
jfk.wav (one window) 1 the 25 reference ids, on this branch and after the merge 25
fx-5min.wav 10 all 615 equal (window by window: 66, 70, 64, 69, 66, 1, 68, 67, 73, 71) 615

fx-5min on the host chain: 300 s of audio in 2776 s (RTF 9.25). The encoder takes 5.7–31 s per window, which tracks the load. Peak RSS is 6.9 GB.

These are host numbers. They are not the comparison against FLM's 25 s; that row is the device route's.

Checks

  • CI-shaped configure, onebit built. ctest -R "audio|onnx_graph|whisper|ci_targets|file_create|copyright" passes 22/22.
  • whisper_transcribe and whisper_transcribe_npu_ci both build.
  • copyright.py --check exits 0. The merge with main is clean.

Not run

🤖 Generated with Claude Code

bong-water-water-bong and others added 2 commits November 13, 2026 08:14
…nd no 60-step cap (engine#1783)

Two defects, both measured against onnxruntime on the pack's own graphs fed
WhisperFeatureExtractor's mel:

- Only the first 30 s were read, and the log-mel was computed over the whole
  recording before the window was taken. Whisper normalises a log-mel to its
  own maximum, so that is a different input: fx-5min's first window differs
  from WhisperFeatureExtractor by up to 8.5e-3 that way, 1e-5 per window.
  The recording is now cut into consecutive 30 s windows, each padded and
  given its own log-mel, encoded, and decoded with the same prompt.
- The greedy loop stopped after 60 steps. fx-5min's first window needs 66
  tokens and the tail was dropped silently. The loop now runs to
  <|endoftext|> or a full token window, and says so when the window fills.

A one-window recording gives the same ids and the same time: line, plus an audio: line; more windows add a
line per window and a per-window / RTF summary. The graphs load once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The device hook #2361 added is opened once after the graphs load, and the
encoder split line is printed per window from the difference of
device_ms(), so a multi-window run reports each window's device share.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 3a73b6f — two defects, both measured against an independent reference, and the second was found while checking the first.

The oracle is stated per claim, which is what makes it reviewable

"Two defects in tools/whisper_transcribe, both measured against onnxruntime 1.31.1 CPU running the pack's own graphs on WhisperFeatureExtractor's log-mel"

Naming the reference (version), the graphs (the pack's own, not a re-implementation) and the feature extractor means the comparison is against something the model actually ships with. That is the #1976/#2183 lineage done right: the oracle is a named artefact, not "the reference".

The first defect is the kind that produces a plausible answer

"Only the first 30 s of a recording were read. The log-mel was also computed [on 30 s]"

So a 5-minute meeting recording transcribes its first 30 seconds and stops — with no error, because a 30 s input is a perfectly valid input. That is the same family as #2348's audio_arg_index (a constant that contradicts the doc) and #2352's curl -s (a check that passes on the wrong thing): a defect whose output looks like a short result rather than a broken one. Window-by-window is the correct fix, and it is also what Whisper's own pipeline does.

Two asks

  1. state the window/boundary arithmetic — 30 s windows with what overlap, and what happens at a boundary where a word straddles it? Whisper's reference implementation uses 30 s non-overlapping chunks with a previous-context prompt and timestamp conditioning; if this does plain concatenation of window transcripts, the body should say so, because words at the seams are where a naive chunker loses text — and the ortert comparison against onnxruntime would catch it only if the test recording has speech at a seam;
  2. name the second defect — it is mentioned as "found while checking item 3" but the excerpt does not say what it is. One line, because two defects in one PR is fine only if a reviewer can see both claimed.

And the scope line is the useful kind

"close table item 4b, plus a defect found while checking item 3" — naming which rows move and that an extra defect rode along. That is how a support PR stays auditable: the table's owner can tick 4b and see that item 3's check also produced something.

@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

1783 - Partially compliant

Compliant requirements:

  • Implement chunked audio processing for recordings longer than 30 seconds
  • Remove the 60-step decoding cap
  • Measure and report performance metrics including RTF

Non-compliant requirements:

  • Ensure correctness with bit-exact transcripts against CPU baseline (not fully verified in this diff)

Requires further human verification:

  • Verification of bit-exact transcripts against CPU baseline
⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Incorrect audio windowing

The change introduces windowing of audio into 30-second chunks, but the mel spectrogram computation for each window uses the full audio length for normalization, which may still differ from the expected behavior where each window's mel is computed independently. This could lead to subtle differences in the audio processing pipeline.

std::vector<float> chunk(audio.begin() + std::ptrdiff_t(from), audio.begin() + std::ptrdiff_t(to));
chunk.resize(want, 0.0f);
std::vector<float> mel;
if (!onebit::audio::log_mel_spectrogram(chunk.data(), chunk.size(), cfg, &mel)) {
    std::fprintf(stderr, "mel: refused (window %zu)\n", w);
    return 1;
}
Potential performance regression

The introduction of a loop over audio windows increases the overhead of processing each window, which could lead to a performance regression compared to the previous implementation that processed the entire audio at once. This is especially concerning for shorter audio files that do not require windowing.

    for (size_t w = 0; w < n_windows; ++w) {
        const auto tw0 = std::chrono::steady_clock::now();
        const size_t from = w * want;
        const size_t to = std::min(audio.size(), from + want);
        std::vector<float> chunk(audio.begin() + std::ptrdiff_t(from), audio.begin() + std::ptrdiff_t(to));
        chunk.resize(want, 0.0f);
        std::vector<float> mel;
        if (!onebit::audio::log_mel_spectrogram(chunk.data(), chunk.size(), cfg, &mel)) {
            std::fprintf(stderr, "mel: refused (window %zu)\n", w);
            return 1;
        }
        const auto tw_mel = std::chrono::steady_clock::now();
        AudioRunResult xa;
#ifdef ONEBIT_WHISPER_NPU
        const double dev_ms0 = dev_hook != nullptr ? npu_dev.gemm->device_ms() : 0.0;
#endif
        if (!run_audio_graph(enc, mel.data(), cfg.bins, cfg.frames, "", &xa, &why, dev_hook)) {
            std::fprintf(stderr, "encoder (window %zu): %s\n", w, why.c_str());
            return 1;
        }
        const auto tw_enc = std::chrono::steady_clock::now();
#ifdef ONEBIT_WHISPER_NPU
        if (dev_hook != nullptr) {
            // The split the issue is scored on, from this run: the wall inside gemv against everything else in this window's
            // encoder pass. device_ms() accumulates over the process, so a window's share is the difference.
            const double enc_ms = std::chrono::duration<double, std::milli>(tw_enc - tw_mel).count();
            const double dev_ms = npu_dev.gemm->device_ms() - dev_ms0;
            std::printf("encoder split: device_gemm=%.3f s + host_remainder=%.3f s per window (window %zu; %d product(s) on the "
                        "device in %d launch(es) so far, %d on the host)\n",
                        dev_ms / 1000.0, (enc_ms - dev_ms) / 1000.0, w, dev_hook->device, npu_dev.gemm->launches(),
                        dev_hook->host);
            if (w == 0)
                for (const auto& kv : dev_hook->host_why) std::printf("  host: %d x %s\n", kv.second, kv.first.c_str());
        }
#endif
        if (w == 0) {
            std::printf("encoder: [");
            for (int64_t d : xa.dims) std::printf(" %lld", (long long)d);
            std::printf(" ]\n");
        }
        // The loop runs until <|endoftext|> or until the token window is full. It used to stop after 60 steps, which a
        // 30 s window of dense speech outruns: fx-5min's first window needs 66 tokens, and the tail was dropped with no
        // message (engine#1783).
        std::vector<int64_t> seq = prompt;
        size_t got = 0;
        bool full = false;
        while (true) {
            if (seq.size() >= size_t(window)) { full = true; break; }
            std::vector<int64_t> toks(size_t(window), kEot);
            for (size_t i = 0; i < seq.size(); ++i) toks[i] = seq[i];
            AudioRunResult y;
            if (!run_decoder_graph(dec, toks.data(), window, xa.data.data(), xa.dims, "", &y, &why)) {
                std::fprintf(stderr, "decoder at window %zu step %zu: %s\n", w, got, why.c_str());
                return 1;
            }
            const int64_t out_vocab = y.dims.empty() ? 0 : y.dims.back();
            const size_t row = (seq.size() - 1) * size_t(out_vocab);
            int64_t best = 0;
            for (int64_t v = 1; v < out_vocab; ++v)
                if (y.data[row + size_t(v)] > y.data[row + size_t(best)]) best = v;
            if (best == kEot) break;
            ids.push_back(best);
            seq.push_back(best);
            ++got;
        }
        const auto tw_dec = std::chrono::steady_clock::now();
        mel_s += secs(tw0, tw_mel);
        enc_s += secs(tw_mel, tw_enc);
        dec_s += secs(tw_enc, tw_dec);
        if (w == 0) {
            ttft_mel_s = secs(t_start, tw_mel);
            ttft_enc_s = secs(tw_mel, tw_enc);
            ttft_s = secs(t_start, tw_enc);
        }
        if (n_windows > 1 || full)
            std::printf("window %zu: mel %.3f s, encoder %.3f s, decode %.3f s for %zu token(s)%s\n", w, secs(tw0, tw_mel),
                        secs(tw_mel, tw_enc), secs(tw_enc, tw_dec), got,
                        full ? " -- the token window filled before <|endoftext|>" : "");
    }

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- the normalisation scope is the real defect, and it is measured against the implementation that defines it

Defect 1 is a truncation and easy to see. Defect 2 is the one worth the whole PR: "The log-mel was also computed over the whole recording before the window was taken. Whisper normalises a log-mel against its own maximum, so this is a different input: on fx-5min's first window the result differs from WhisperFeatureExtractor by up to 8.5e-3, against 1e-5 when each window normalises itself."

That is a scope error, not an arithmetic one: every number is right, the max is simply taken over the wrong set, and the consequence is a silently different tensor feeding the encoder. 8.5e-3 against 1e-5 is the number that makes it undeniable, and it explains why the first window was the worst case -- it is the furthest from the whole-file max.

And the oracle is the right one, which is what makes both claims checkable: onnxruntime 1.31.1 CPU running the pack's own graphs with WhisperFeatureExtractor's log-mel. A hand-built reference would have agreed with our own reading of whisper's pre-processing; the implementation that defines it cannot.

Two questions, the second of which I think is a live trap:

  1. Is the 1e-5 bound an assertion in a test, or a number in this body? The whole-recording mel is plausible output -- nothing errors, the transcript is just wrong -- so the only thing that stops it coming back is a per-window comparison against WhisperFeatureExtractor at a fixed tolerance, with fx-5min's first window as the case. If that exists, say where; if it is the body's table only, it should become a test row.

  2. The last window's padding, and whether it is inside the max. Whisper pads a short final window to 30 s. If the pad happens before the log-mel, the padded frames contribute to the mel and therefore to the max, and log-mel values are a log of magnitudes, so they can be negative -- a zeroed region is not necessarily below the true maximum, and if it is not, the normaliser scales the whole window differently from what WhisperFeatureExtractor computes. That is the same class as defect 2, one level down: right arithmetic, wrong set of numbers in the reduction. If the code pads after the mel, or pads with the true floor rather than zeros, this is a non-issue and one sentence would say so.

Nothing else: 105/-53 with the window-by-window loop replacing the truncation, and the commit closes two separate table items rather than one.

@bong-water-water-bong
bong-water-water-bong merged commit 51c2bf6 into main Nov 13, 2026
9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the 1bit/1783-whisper-windows branch November 13, 2026 12:38

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 3a73b6f: READY

I built whisper_transcribe at this head (ONEBIT_NPU host build) and ran it on ~/scratch-packs/pi-demo/whisper-small, under thermal-run:

input result
jfk.wav (11 s), with ONEBIT_TEST_WHISPER_IDS_FILE=tests/data/whisper-jfk-ids.json rc 0, transcript ids match the reference decode (25 ids)
the first 30.00 s of fx-5min.wav audio: 30.00 s -> 1 window(s), 66 tokens, the count the body gives, past the old 60-step cap
jfk.wav ×3 (33.00 s, my own fixture) audio: 33.00 s -> 2 window(s) of 30 s. Window 0: 66 tokens; window 1, the last 3 s: 9 tokens. The per window: and RTF lines print.

From the code:

  • Each window gets its own log_mel_spectrogram over a chunk padded to 30 s, which is the per-window normalisation the body measures against WhisperFeatureExtractor.
  • The decode loop ends at <|endoftext|> or at a full token window (full = true, reported), with no fixed step count.
  • The device hook's split is the per-window difference of device_ms().
  • The merge with main is clean, 1 file +105 −53.

CI: the build job passed, 100% tests passed, 0 tests failed out of 310.

Nits, not blocking:

  • The time: line says mel+load while the ttft: line says load+mel. Pick one order.
  • A plain 30 s cut splits a word at the boundary (window 1 of my fixture starts mid-phrase). That is expected for the no-timestamp chunked form the body describes; just don't compare a multi-window transcript to a reference made with overlap.

Not run by me: the 10-window fx-5min (≈46 min at RTF 9 on the host) and the NPU variant, both of which the body lists.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant