Skip to content

feat(voice): Groq whisper as the primary dictation backend, streamed in pieces - #120

Merged
nullStack65 merged 2 commits into
release/native-0.1.10from
feat/clo-919-groq-primary
Nov 20, 2026
Merged

nullStack65 merged 2 commits into
release/native-0.1.10from
feat/clo-919-groq-primary

Conversation

@nullStack65

@nullStack65 nullStack65 commented Nov 20, 2026 •

Copy link
Copy Markdown
Owner

Dictation needs under 400 ms from key release to text (the shared dictation work, Linear CLO-919). The CPU reference was a p50 of 1431 ms, and every cloud provider here uploaded only after release.

What changed

  1. Groq whisper-large-v3-turbo is now a cloud speech provider and the recommended backend. Its key comes from settings or from a host-injected GROQ_API_KEY, which is removed from the server's environment so agent subprocesses never inherit it, the same way ASSEMBLYAI_API_KEY works. Custom words go in as Whisper's prompt. AssemblyAI is unchanged.
  2. On web and desktop, a Groq recording streams over its own socket, /ws/voice/cloud/groq, using the existing speech frame protocol. The server cuts audio at pauses once a piece reaches 4 s (forced at 12 s, at the quietest frame) and uploads each piece while the person is still talking. Release then waits only for the tail. The 4 s floor keeps one continuous dictation at or under 15 requests a minute, inside the free tier's 20. Silence and stray clicks are dropped, but a quiet microphone's audio is still sent. A rate-limited or unavailable piece is retried twice. A permanent failure fails the stream on the next audio frame, so it doesn't wait for release. With live transcription off, Groq uses the batch endpoint. Groq errors reach the user as their own message.
  3. Mobile uses Groq through the existing batch endpoint and gains no streaming yet.
  4. CI gate: GroqSpeechStream.test.ts feeds 30 dictations of 3 to 70 s in real time on TestClock through the real segmenter and stream. It asserts structurally that any tail left at release is at most 10 s of audio. It then fails if release-to-text exceeds 400 ms against a modelled Groq latency of 230 ms + 12 ms per audio second. Modelled result: p50 0 ms, p95 261 ms, max 264 ms. The model is an assumption. It leaves out real Groq variance and the socket round trip, so it does not prove 400 ms live. Live numbers wait on a Groq key. Other tests cover piece bounds, no lost speech, silence, clicks, quiet microphones, forced cuts, retries and failing early.
  5. Docs for users and internals are updated.
  6. Known limit: each piece is transcribed and filler-stripped on its own, so a forced 12 s cut can leave an odd capital or full stop at the join.

Verified locally (ASUS, Node 26): typecheck passes for contracts, shared, client-runtime, server (t3), web and mobile. The Groq, stream, CloudSpeech, settings and secrets tests pass. The web voiceBackends test passes. Two existing failures happen identically on the unchanged base, and CI runs neither: CloudVoiceConfiguration.browser.test.tsx fails 7/7 with a speechCustomWords flatMap error, and AgentContextBudget.test.ts measures 55,240 MCP bytes against a budget of 53,783. Neither is changed here.

Not verified: live Groq latency, rate limits and WER all wait on a free Groq key in Infisical.

Deploy: fleet

Model: claude-opus-5-5
Task: CLO-919

🤖 Generated with Claude Code

nullStack65 and others added 2 commits November 20, 2026 03:39
…in pieces

Groq whisper-large-v3-turbo joins the cloud speech providers and becomes the
recommended backend. Its key comes from settings or a host-injected
GROQ_API_KEY, like AssemblyAI. On web and desktop, a Groq recording streams
over its own socket; the server cuts the audio at pauses once a piece reaches
eight seconds and uploads each piece while the person is still talking, so
release waits only for the last piece. A CI test feeds 30 dictations in real
time against a modelled Groq latency and fails above 400 ms release-to-text.

Model: claude-opus-5-5
Task: CLO-919
Agent: Claude Code
Lane: magpie
Harness: t3/claude-code
Model-family: Claude
Model-generation: 5.5
Skills-SHA: 7a14a4240d1e
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review fixes on the Groq dictation stream:
- uploads fork into the stream's scope, so feed needs no Scope, and the
  route path is typed as a router path; the server typecheck passes again
- a rate-limited or unavailable piece is retried twice; a permanent failure
  fails the stream on the next audio frame instead of at release
- the floor drops from 8 s to 4 s and forced cuts from 20 s to 12 s, so short
  dictations upload during speech and the tail stays bounded
- a quiet microphone's audio is still sent; stray clicks and long silences
  inside a piece are dropped
- with live transcription off, Groq uses the batch endpoint, and the
  disclosure no longer claims streaming on mobile
- the context budget edit is reverted; that drift predates this branch

Model: claude-opus-5-5
Task: CLO-919
Agent: Claude Code
Lane: magpie
Harness: t3/claude-code
Model-family: Claude
Model-generation: 5.5
Skills-SHA: 7a14a4240d1e
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nullStack65

Copy link
Copy Markdown
Owner Author
{"reviewed_head":"14e158c2ba0a4f6c165a0f264febb2889d39ec8f","reviewer":"grok-4.7-build (claudeAgent, read-only)","findings":[{"id":"cached-config-keeps-host-key","severity":"nit","summary":"Effect's default ConfigProvider copies process.env on its first read, which startup config does before the speech layer deletes GROQ_API_KEY, so a later Config read could still see it. Nothing reads that variable again after the delete, and a child process inherits process.env, which the delete does clear.","outcome":"skipped"}]}

@nullStack65
nullStack65 merged commit de208f5 into release/native-0.1.10 Nov 20, 2026
4 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant