Repository navigation
whisper_transcribe: recordings longer than 30 s, window by window, and no 60-step cap (#1783) - #2365
Conversation
…nd no 60-step cap (engine#1783) Two defects, both measured against onnxruntime on the pack's own graphs fed WhisperFeatureExtractor's mel: - Only the first 30 s were read, and the log-mel was computed over the whole recording before the window was taken. Whisper normalises a log-mel to its own maximum, so that is a different input: fx-5min's first window differs from WhisperFeatureExtractor by up to 8.5e-3 that way, 1e-5 per window. The recording is now cut into consecutive 30 s windows, each padded and given its own log-mel, encoded, and decoded with the same prompt. - The greedy loop stopped after 60 steps. fx-5min's first window needs 66 tokens and the tail was dropped silently. The loop now runs to <|endoftext|> or a full token window, and says so when the window fills. A one-window recording gives the same ids and the same time: line, plus an audio: line; more windows add a line per window and a per-window / RTF summary. The graphs load once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The device hook #2361 added is opened once after the graphs load, and the encoder split line is printed per window from the difference of device_ms(), so a multi-window run reports each window's device share. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
VERDICT: READY at The oracle is stated per claim, which is what makes it reviewable
Naming the reference (version), the graphs (the pack's own, not a re-implementation) and the feature extractor means the comparison is against something the model actually ships with. That is the #1976/#2183 lineage done right: the oracle is a named artefact, not "the reference". The first defect is the kind that produces a plausible answer
So a 5-minute meeting recording transcribes its first 30 seconds and stops — with no error, because a 30 s input is a perfectly valid input. That is the same family as #2348's Two asks
And the scope line is the useful kind"close table item 4b, plus a defect found while checking item 3" — naming which rows move and that an extra defect rode along. That is how a support PR stays auditable: the table's owner can tick 4b and see that item 3's check also produced something. |
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- the normalisation scope is the real defect, and it is measured against the implementation that defines it
Defect 1 is a truncation and easy to see. Defect 2 is the one worth the whole PR: "The log-mel was also computed over the whole recording before the window was taken. Whisper normalises a log-mel against its own maximum, so this is a different input: on fx-5min's first window the result differs from WhisperFeatureExtractor by up to 8.5e-3, against 1e-5 when each window normalises itself."
That is a scope error, not an arithmetic one: every number is right, the max is simply taken over the wrong set, and the consequence is a silently different tensor feeding the encoder. 8.5e-3 against 1e-5 is the number that makes it undeniable, and it explains why the first window was the worst case -- it is the furthest from the whole-file max.
And the oracle is the right one, which is what makes both claims checkable: onnxruntime 1.31.1 CPU running the pack's own graphs with WhisperFeatureExtractor's log-mel. A hand-built reference would have agreed with our own reading of whisper's pre-processing; the implementation that defines it cannot.
Two questions, the second of which I think is a live trap:
-
Is the 1e-5 bound an assertion in a test, or a number in this body? The whole-recording mel is plausible output -- nothing errors, the transcript is just wrong -- so the only thing that stops it coming back is a per-window comparison against
WhisperFeatureExtractorat a fixed tolerance, with fx-5min's first window as the case. If that exists, say where; if it is the body's table only, it should become a test row. -
The last window's padding, and whether it is inside the max. Whisper pads a short final window to 30 s. If the pad happens before the log-mel, the padded frames contribute to the mel and therefore to the max, and log-mel values are a log of magnitudes, so they can be negative -- a zeroed region is not necessarily below the true maximum, and if it is not, the normaliser scales the whole window differently from what
WhisperFeatureExtractorcomputes. That is the same class as defect 2, one level down: right arithmetic, wrong set of numbers in the reduction. If the code pads after the mel, or pads with the true floor rather than zeros, this is a non-issue and one sentence would say so.
Nothing else: 105/-53 with the window-by-window loop replacing the truncation, and the commit closes two separate table items rather than one.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 3a73b6f: READY
I built whisper_transcribe at this head (ONEBIT_NPU host build) and ran it on ~/scratch-packs/pi-demo/whisper-small, under thermal-run:
| input | result |
|---|---|
jfk.wav (11 s), with ONEBIT_TEST_WHISPER_IDS_FILE=tests/data/whisper-jfk-ids.json |
rc 0, transcript ids match the reference decode (25 ids) |
the first 30.00 s of fx-5min.wav |
audio: 30.00 s -> 1 window(s), 66 tokens, the count the body gives, past the old 60-step cap |
jfk.wav ×3 (33.00 s, my own fixture) |
audio: 33.00 s -> 2 window(s) of 30 s. Window 0: 66 tokens; window 1, the last 3 s: 9 tokens. The per window: and RTF lines print. |
From the code:
- Each window gets its own
log_mel_spectrogramover a chunk padded to 30 s, which is the per-window normalisation the body measures againstWhisperFeatureExtractor. - The decode loop ends at
<|endoftext|>or at a full token window (full = true, reported), with no fixed step count. - The device hook's split is the per-window difference of
device_ms(). - The merge with main is clean, 1 file +105 −53.
CI: the build job passed, 100% tests passed, 0 tests failed out of 310.
Nits, not blocking:
- The
time:line saysmel+loadwhile thettft:line saysload+mel. Pick one order. - A plain 30 s cut splits a word at the boundary (window 1 of my fixture starts mid-phrase). That is expected for the no-timestamp chunked form the body describes; just don't compare a multi-window transcript to a reference made with overlap.
Not run by me: the 10-window fx-5min (≈46 min at RTF 9 on the host) and the NPU variant, both of which the body lists.
Refs #1783 (close table item 4b, plus a defect found while checking item 3). cl-1051, support; pi-demo leads.
Why
Two defects in
tools/whisper_transcribe, both measured against onnxruntime 1.31.1 CPU running the pack's own graphs onWhisperFeatureExtractor's log-mel:WhisperFeatureExtractorby up to 8.5e-3, against 1e-5 when each window gets its own mel. Item 4b (the 5-minute recording FLM quotes) had no path at all.Change
<|endoftext|>or a full token window. When the window fills first, the tool says so.time:line as before, plus anaudio:line. Thettft:line now saysload+mel, because the graph load now comes before the first mel.per window: encoder …, wall …; recording … in … (RTF …).encoder split: device_gemm=… + host_remainder=…prints per window, taken from the difference ofdevice_ms().Measured (whisper-small,
thermal-run, 4 CPUs; box load 6–15)fx-5min on the host chain: 300 s of audio in 2776 s (RTF 9.25). The encoder takes 5.7–31 s per window, which tracks the load. Peak RSS is 6.9 GB.
These are host numbers. They are not the comparison against FLM's 25 s; that row is the device route's.
Checks
onebitbuilt.ctest -R "audio|onnx_graph|whisper|ci_targets|file_create|copyright"passes 22/22.whisper_transcribeandwhisper_transcribe_npu_ciboth build.copyright.py --checkexits 0. The merge with main is clean.Not run
whisper_transcribe_npuon the device. The NPU variant's per-window split is compiled but not run here; it is the next row on npu: Whisper encoder on our runtime — measure against FLM 1.0.12's 0.6 s/30 s window and close the gap (W8/F8) #1783.🤖 Generated with Claude Code