Skip to content

fix(web): voice input starts recording on press instead of waiting on the server - #115

Merged
nullStack65 merged 7 commits into
release/native-0.1.10from
fix/clo-919-transcribe-latency
Nov 20, 2026
Merged

nullStack65 merged 7 commits into
release/native-0.1.10from
fix/clo-919-transcribe-latency

Conversation

@nullStack65

Copy link
Copy Markdown
Owner

Kameron reported that the composer transcribe button shows "starting" for 1 to 3 seconds. The goal is under 400 ms from press to recording. This is item 4 of the shared dictation issue (CLO-919).

Cause

Before it opened the microphone, a press waited for two or three server round trips in a row: the hook's status refresh, the status again inside prepare, and the media stream ticket. After that it created a new AudioContext and loaded the worklet module, one step at a time.

Fix

  • VoiceInputController takes a new captureDuringPreparation option, and web turns it on. Capture starts at once and the phase becomes recording while the server preparation runs alongside it. Releasing before preparation finishes waits for it. If preparation fails, the recording stops with the server's error. Mobile keeps its sequential start.
  • browserPcmCapture buffers PCM until the stream is ready, then flushes it into the stream. If the recording ended first, it cancels the stream and uses buffered transcription. One shared AudioContext with the loaded worklet is warmed when the composer mounts and is suspended between recordings. The microphone stream is never kept open, so the mic indicator only shows while recording.
  • A worklet processor that was released or stopped now returns false, so old processors don't pile up on the shared context.
  • When a status is already cached, the hook's status refresh runs alongside capture. A missing model still routes to setup, and an unsupported destination still shows its toast.

Latency gate (runs in ARC CI)

  • packages/client-runtime/src/voice-input/controller.test.ts: recording must begin within 400 ms while prepare never answers. It fails on the old sequential path.
  • apps/web/src/speech/voiceInputLatency.browser.test.ts uses the reusable apps/web/test/voiceLatencyHarness.ts. It drives real Chromium capture with a fake mic device, and the server is held open until after the press. It fails if press-to-recording or release-to-text goes over 400 ms, for both buffered and streaming. The browser project now launches Chromium with fake media and --disable-audio-output, so the audio clock runs headless without a sound server.

Measured on the ASUS (headless Chromium, simulated server round trip)

before after
press→recording, 50 ms round trip 231–642 ms 22–61 ms
press→recording, 300 ms round trip 790–1,924 ms 16–75 ms (cold, no prewarm: 18–333 ms)
release→text (client side), 300 ms round trip 16–184 ms 4–27 ms

The number on Kameron's laptop is unverified. The settings test CloudVoiceConfiguration.browser.test.tsx already fails on the base branch, so those failures are not from this change.

Deploy: none

Model: claude-opus-5-5
Task: coding

🤖 Generated with Claude Code

nullStack65 and others added 5 commits November 20, 2026 01:38
… the server

Pressing the composer transcribe button waited for two or three server
round trips (a status refresh, the status again inside prepare, and the
stream ticket) before it opened the microphone. It then created a fresh
AudioContext and loaded the worklet module one step at a time. Over a
300 ms link that is 0.8 to 1.9 s of "starting".

The web controller now opens capture right away and buffers PCM while
the server prepares. It flushes the buffer into the live stream when the
stream is ready, or falls back to buffered transcription if the
recording ended first. The status refresh runs alongside capture. One
AudioContext with the loaded worklet stays warm and suspended between
recordings. The microphone stream is never kept open. Released worklet
processors now end instead of lingering. Mobile keeps its sequential
start.

Model: claude-opus-5-5
Task: coding
Agent: T3 Claude Code (CLO-919 button lane)
Lane: magpie
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the hook

Review fixes for the press-to-recording change:

- If the server preparation fails after recording has started, the
  controller now stops the capture and keeps the audio. The next press
  prepares again and transcribes that audio, without recording again.
- The composer starts capture at once even before the first status
  answer, unless a known status says the model is missing or the
  destination is unsupported.
- The warmed AudioContext is suspended once its worklet loads, and it is
  closed if loading fails. Cleanup suspends it even when a resume is still
  in flight. The worklet release message is no longer cut off by closing
  the port.
- A hook-level browser test fails if a press waits for the status
  refresh, and it runs in ARC CI with the latency test.

Model: claude-opus-5-5
Task: coding
Agent: T3 Claude Code (CLO-919 button lane)
Lane: magpie
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The ARC pod image has no Chromium system libraries and cannot use sudo,
so headless Chromium exited with "libnspr4.so: cannot open shared object
file". Download the Ubuntu packages Chromium needs with apt as the runner
user, using private state directories so apt still resolves dependencies
and verifies the archive signatures, extract them, and add them to
LD_LIBRARY_PATH, as the libatomic step already does for Node.

Model: claude-opus-5-5
Task: coding
Agent: T3 Claude Code (CLO-919 button lane)
Lane: magpie
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Starting capture before the first status answer opened the microphone for
environments still missing a model, and on a slow runner it raced the
mount probe, so setup-wizard tests saw capture instead of setup. Only a
known ready status takes the fast path again; an unknown status waits for
its check, as before this change.

Model: claude-opus-5-5
Task: coding
Agent: T3 Claude Code (CLO-919 button lane)
Lane: magpie
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Running the whole composer voice hook test file on ARC also ran three
setup-wizard tests that ARC had never run before. On the runner, their
dialog content renders but is reported as not visible, while the same
tests pass locally. Select the press-to-recording and release-to-text
tests by name so the latency gate stays and those wizard tests stay out
of this step, as they were before.

Model: claude-opus-5-5
Task: coding
Agent: T3 Claude Code (CLO-919 button lane)
Lane: magpie
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

nullStack65 and others added 2 commits November 20, 2026 18:07
The retry read the recorder uri after releaseAudioSession, whose cancel
clears it, so the next press retried with no audio. Read the uri first.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@nullStack65

Copy link
Copy Markdown
Owner Author
{"reviewed_head":"0a9091d312e89b29164f227bf5d8ab853c0ad215","reviewer":"grok-4.7-build (claudeAgent, read-only)","findings":[{"id":"late-prep-failure-drops-retry-audio","severity":"important","summary":"finishRecording released the audio session before reading the recorder uri, and that release clears the uri, so a preparation failure after capture started left the retry with no audio. The uri is now read and stored before the release, and the regression test nulls the uri inside releaseRecording.","outcome":"confirmed","rechecked_head":"7e22c515a0c58a0a23b3aed881b2b977f95bb7fa"}]}

@nullStack65
nullStack65 merged commit c27299d into release/native-0.1.10 Nov 20, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant