fix(dictation): warm the mic so push-to-talk doesn't clip the first words#910
fix(dictation): warm the mic so push-to-talk doesn't clip the first words#910mcox5 wants to merge 1 commit into
Conversation
…ords `useAudioRecording` opened a fresh `getUserMedia` stream on every `startRecording` and stopped its tracks on every stop. On macOS that open takes several hundred ms — up to a second cold — and `MediaRecorder` only begins capturing after it resolves, so the first ~1s of speech (often the first several words) is lost on every push-to-talk dictation. Add an opt-in `keepWarm` mode that keeps the capture `MediaStream` open between recordings and pre-opens it on mount, so the hot path (chord-down → record) never waits on `getUserMedia` and captures from the first frame. The floating dictate window (always mounted) opts in via a new `keepMicWarm` option on `useCaptureRecordingSession`; the warm stream self-releases after `WARM_IDLE_RELEASE_MS` (60s) of inactivity so the OS mic indicator clears when the user isn't dictating. Behavior is unchanged for the voice-clone sample recorders and the main-window Dictate button (they don't pass `keepWarm`/`keepMicWarm`), which record once and should release the device immediately. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
📝 WalkthroughWalkthrough
ChangesWarm microphone capture
Estimated code review effort: 3 (Moderate) | ~30 minutes Sequence Diagram(s)sequenceDiagram
participant DictateWindow
participant useCaptureRecordingSession
participant useAudioRecording
participant MediaStream
DictateWindow->>useCaptureRecordingSession: configure keepMicWarm
useCaptureRecordingSession->>useAudioRecording: prewarm()
useAudioRecording->>MediaStream: open microphone stream
useAudioRecording->>MediaStream: reuse stream for recording
useAudioRecording->>useAudioRecording: schedule idle release after stop or cancel
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@app/src/components/DictateWindow/DictateWindow.tsx`:
- Around line 45-48: Update useAudioRecording’s microphone acquisition flow so
prewarm() and startRecording() share a single in-progress getUserMedia() promise
or otherwise serialize acquisition. Ensure startRecording awaits the existing
prewarm request when present, and retain/use the same resolved stream in
warmStreamRef so concurrent calls cannot create or leak separate microphone
streams.
In `@app/src/lib/hooks/useAudioRecording.ts`:
- Around line 111-140: Serialize stream acquisition in acquireStream and prewarm
using a shared in-flight promise so concurrent prewarming and recording reuse
the same getUserMedia operation and cannot overwrite or leak streams. Update
prewarm to schedule idle release only when no recording or start operation is
active, preserving the existing keepWarm and error-handling behavior.
- Around line 184-188: Update the recorder stop and cancel cleanup callbacks to
capture their own recorder and stream references, then only clear streamRef or
schedule/release cleanup when those references still match the currently active
recording. Preserve cleanup for the owning callback while preventing stale
callbacks from affecting a newer recording; apply this consistently to the
warm-release logic near scheduleIdleRelease and the corresponding cancel path.
- Around line 148-151: Update the recorder startup error path around
acquireStream, new MediaRecorder, and mediaRecorder.start so a failed startup
cleans up the acquired stream. In the catch handler, reschedule the warm release
when keepWarm is enabled; otherwise stop all acquired stream tracks, preserving
the existing successful startup behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: e9982f97-d405-4d7b-81c9-944a48fd66d5
📒 Files selected for processing (3)
app/src/components/DictateWindow/DictateWindow.tsxapp/src/lib/hooks/useAudioRecording.tsapp/src/lib/hooks/useCaptureRecordingSession.ts
| // This floating window is always mounted, so keeping the mic warm here | ||
| // removes the first-words clipping on push-to-talk without holding the | ||
| // device open from the main app window. | ||
| keepMicWarm: true, |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
Prevent overlapping microphone acquisition.
prewarm() runs asynchronously on mount, while dictate:start can immediately invoke startRecording(). Both can observe an empty warmStreamRef and call getUserMedia() concurrently; resolution order may leave the recorder using a different stream than the retained warm stream, leaking a live microphone stream or breaking reuse. Make acquisition single-flight in useAudioRecording, or ensure recording awaits the in-progress prewarm.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@app/src/components/DictateWindow/DictateWindow.tsx` around lines 45 - 48,
Update useAudioRecording’s microphone acquisition flow so prewarm() and
startRecording() share a single in-progress getUserMedia() promise or otherwise
serialize acquisition. Ensure startRecording awaits the existing prewarm request
when present, and retain/use the same resolved stream in warmStreamRef so
concurrent calls cannot create or leak separate microphone streams.
| const acquireStream = useCallback(async (): Promise<MediaStream> => { | ||
| if (streamHasLiveAudio(warmStreamRef.current)) { | ||
| return warmStreamRef.current; | ||
| } | ||
| // A dead warm stream (device unplugged / tracks ended) — drop it and reopen. | ||
| if (warmStreamRef.current) releaseWarmStream(); | ||
| await assertMediaDevices(); | ||
| const stream = await navigator.mediaDevices.getUserMedia({ | ||
| audio: AUDIO_CONSTRAINTS, | ||
| }); | ||
| if (keepWarm) warmStreamRef.current = stream; | ||
| return stream; | ||
| }, [assertMediaDevices, keepWarm, releaseWarmStream]); | ||
|
|
||
| /** | ||
| * Open the microphone ahead of the first recording so the initial dictation | ||
| * doesn't clip. No-op unless ``keepWarm`` is set. Safe to call repeatedly and | ||
| * safe to fail (e.g. permission not yet granted) — ``startRecording`` still | ||
| * surfaces a real error if capture is genuinely unavailable. | ||
| */ | ||
| const prewarm = useCallback(async () => { | ||
| if (!keepWarm) return; | ||
| clearIdleRelease(); | ||
| try { | ||
| await acquireStream(); | ||
| } catch { | ||
| // Permission missing / device busy — recording will report it properly. | ||
| } | ||
| scheduleIdleRelease(); | ||
| }, [keepWarm, clearIdleRelease, acquireStream, scheduleIdleRelease]); |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Serialize prewarming and recording stream acquisition.
Mount-time prewarm() and an immediate recording can both observe an empty warmStreamRef and call getUserMedia. The later result overwrites the first, potentially leaking one stream; additionally, prewarm() can arm its idle timer after recording has started and stop the active stream 60 seconds later.
Deduplicate acquisition with an in-flight promise and only schedule idle release when no recording/start operation is active.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@app/src/lib/hooks/useAudioRecording.ts` around lines 111 - 140, Serialize
stream acquisition in acquireStream and prewarm using a shared in-flight promise
so concurrent prewarming and recording reuse the same getUserMedia operation and
cannot overwrite or leak streams. Update prewarm to schedule idle release only
when no recording or start operation is active, preserving the existing keepWarm
and error-handling behavior.
| clearIdleRelease(); | ||
|
|
||
| // Reuse the warm stream when present (instant); otherwise open one now. | ||
| const stream = await acquireStream(); |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Restore stream cleanup when recorder startup fails.
Line 148 cancels the idle release before acquisition, but failures from new MediaRecorder(...) or mediaRecorder.start() reach the catch without releasing or rescheduling the stream. With keepWarm, this leaves the microphone open indefinitely.
Add catch-path cleanup: reschedule warm release when opted in, otherwise stop the acquired tracks.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@app/src/lib/hooks/useAudioRecording.ts` around lines 148 - 151, Update the
recorder startup error path around acquireStream, new MediaRecorder, and
mediaRecorder.start so a failed startup cleans up the acquired stream. In the
catch handler, reschedule the warm release when keepWarm is enabled; otherwise
stop all acquired stream tracks, preserving the existing successful startup
behavior.
| // Release the device unless we're keeping it warm for the next capture. | ||
| // When warm, the tracks stay live and an idle timer will reclaim them. | ||
| if (keepWarm) { | ||
| streamRef.current = null; | ||
| scheduleIdleRelease(); |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Prevent stale cleanup from releasing a newer recording.
A user can start another recording before the previous recorder's queued onstop runs. That stale handler then clears streamRef and schedules release of the shared warm stream, terminating the newer recording after 60 seconds. The cancel path has the same ownership race.
Capture the recorder/stream in each callback and only perform cleanup if it still owns the current recording.
Also applies to: 283-287
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@app/src/lib/hooks/useAudioRecording.ts` around lines 184 - 188, Update the
recorder stop and cancel cleanup callbacks to capture their own recorder and
stream references, then only clear streamRef or schedule/release cleanup when
those references still match the currently active recording. Preserve cleanup
for the owning callback while preventing stale callbacks from affecting a newer
recording; apply this consistently to the warm-release logic near
scheduleIdleRelease and the corresponding cancel path.
|
Confirming this on Voicebox 0.5.0 / macOS 26.5.1 / Apple Silicon. The first ~1s (first few words) of every push-to-talk capture is clipped, exactly as described — cold |
|
Confirming this on Windows 11 / WebView2 / CUDA with measurements — same ~1 s clip, every capture, and I can add device-exonerating data that supports keeping the fix at the Environment
Finding 1 — every capture carries exactly 1.0 s of dead audio at the headRMS analysis (100 ms windows) of 14 consecutive dictation WAVs from Short utterances land entirely inside the dead window: the file is ~1 s of silence and Whisper hallucinates its classic On Windows the lost second is recorded as zeros (the stream resolves, then delivers zero-fill while Chromium's audio pipeline spins up) rather than absent, but the user-facing effect is identical to your macOS trace: everything spoken in the first second of a push-to-talk is gone, on every capture, exactly as your root-cause section describes (fresh Finding 2 — the device/OS stack is not a factor (supports the warm-stream approach)Raw WASAPI capture on the same mic via PortAudio/sounddevice, bypassing WebView2:
Live room audio from the first 100 ms window in all three runs; USB power management on the device's audio interface node is not in play. So the entire 1.0 s is inside the WebView2 Minor related observationMost captures also show 0.0 s trailing silence — speech runs flush to the end of the file, so releasing the chord mid-final-word clips the tail. If the Happy to test a Windows build of this branch if useful — this is currently the biggest UX gap for our team's dictation use, and the warm-stream approach matches everything we measured. |
3 participants
Problem
On push-to-talk dictation the first ~1 second of speech — often the first few words — is clipped on every capture. Steps to reproduce (macOS, Apple Silicon):
Root cause
useAudioRecording.startRecording()opens a freshgetUserMediastream on every recording and stops its tracks again on every stop:On macOS opening the input device takes several hundred ms — up to ~1s cold — and
MediaRecorderonly begins capturing once the promise resolves. Everything spoken in that window is lost, and because the stream is torn down after each capture, every dictation pays the cost, not just the first.This is distinct from the fullscreen bug (#836 / #848, where
getUserMedianever resolves at all in a native-fullscreen Space) and from #843 (raw-audio constraints / waveform stream) — neither keeps the dictation mic warm.Changes
useAudioRecording— new opt-inkeepWarmoption. When set, the captureMediaStreamis kept open between recordings and reused, so the hot path (chord-down → record) never awaitsgetUserMediaandMediaRecordercaptures from the first frame. Adds aprewarm()that pre-opens the device, and a dead-stream guard that re-acquires if the tracks have ended (device unplugged / switched). The warm stream self-releases afterWARM_IDLE_RELEASE_MS(60s) of inactivity so the OS mic indicator clears when the user isn't dictating.useCaptureRecordingSession— newkeepMicWarmoption, threaded through touseAudioRecording, plus a mount effect that callsprewarm()so even the first dictation is warm.DictateWindow— the always-mounted floating pill opts in withkeepMicWarm. Warming lives here (not the main window) so the mic is only held open by the surface that actually services the global shortcut.Behavior is unchanged for the voice-clone sample recorders (
ProfileForm,SampleUpload) and the main-window Dictate button — they don't passkeepWarm/keepMicWarm, so they still open once and release the device immediately.Trade-off / notes
keepWarmthe mic is held open (indicator lit) during active dictation and for up toWARM_IDLE_RELEASE_MSafter — the price of zero clipping. Happy to make the timeout configurable or gate warming behind a setting if you'd prefer.prewarm()on it, fully eliminating clipping without holding the device open at all.Verification
bunx tsc -p app/tsconfig.json --noEmit— passes.biome linton the changed files — clean (no new warnings; the pre-existinguseExhaustiveDependencieswarnings inDictateWindoware untouched).Summary by CodeRabbit