feat(stt): Use the dictation API for transcription and LLM cleanup#79
Draft
aurpsis-aai wants to merge 2 commits into
Draft
feat(stt): Use the dictation API for transcription and LLM cleanup#79aurpsis-aai wants to merge 2 commits into
aurpsis-aai wants to merge 2 commits into
Conversation
Point AssemblyAITranscriber at dictation.assemblyai.com/transcribe, which runs the sync STT transcription and a server-side LLM cleanup rewrite (disfluencies out, punctuation fixed) in the same single request: - config gains an empty `llm` block, selecting the service's default cleanup instruction and its server-owned guardrails - the response's `llm_response` is what gets pasted; a null rewrite (best-effort, 5 s server budget) degrades to the verbatim `text` with the `llm_error` logged, never a user-facing failure - drop the `X-AAI-Model` header — the dictation service pins the STT model server-side - set the API's documented 90 s client timeout on the request TranscriptionPrompt still rides along as `config.prompt`, now purely transcription steering; the pipeline stays one request per utterance with no client-side LLM pass. Docs updated to match. Co-Authored-By: Claude Fable 5 (Jeeves) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR migrates AssemblyAITranscriber in BlurtEngine from the Sync STT endpoint to AssemblyAI’s dictation API, so each utterance is handled by a single request that returns both the verbatim transcript and (when available) a server-side LLM cleanup rewrite that the app pastes.
Changes:
- Switch STT transport to
https://dictation.assemblyai.com/transcribe, removeX-AAI-Model, and set a 90s client timeout. - Always request the default server-side rewrite via an empty
config.llmobject and fall back totextwhenllm_responseis missing/null (loggingllm_erroras degradation). - Update engine/app/docs/tests wording and coverage to reflect “dictation API + rewrite” behavior.
Reviewed changes
Copilot reviewed 18 out of 18 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| Tests/BlurtEngineTests/TranscriptionPromptTests.swift | Updates test descriptions to reflect transcription prompt usage (not Sync-specific). |
| Tests/BlurtEngineTests/DictationSessionTests.swift | Updates comments to reflect dictation API semantics and unchanged short-clip behavior. |
| Tests/BlurtEngineTests/AssemblyAITranscriberTests.swift | Adds/updates tests for dictation endpoint, rewrite preference, fallback behavior, header/timeout contract, and config llm presence. |
| Sources/BlurtEngine/STT/TranscriptionPrompt.swift | Updates docs to describe dictation API prompt usage. |
| Sources/BlurtEngine/STT/TranscriptionContext.swift | Updates docs to remove Sync-specific wording. |
| Sources/BlurtEngine/STT/TranscriberProtocol.swift | Updates docs to reflect dictation API behavior (single-shot result). |
| Sources/BlurtEngine/STT/SyncSTTLimits.swift | Clarifies that Sync model limits still apply behind dictation API forwarding. |
| Sources/BlurtEngine/STT/AssemblyAITranscriber.swift | Core change: dictation endpoint, 90s timeout, remove model header, request rewrite via llm: {}, decode llm_response/llm_error, fallback to verbatim text. |
| Sources/BlurtEngine/Pipeline/DictationSession+Pipeline.swift | Updates pipeline comments to reflect dictation API and “no client-side styling pass”. |
| Sources/BlurtEngine/Pipeline/DictationSession.swift | Updates comments around recording cap and connection warm-up for dictation host. |
| Sources/BlurtEngine/Config/KeyTermsStore.swift | Updates docs to refer to dictation request priming. |
| Sources/BlurtEngine/Audio/MicCaptureProtocol.swift | Updates docs to refer to dictation request upload format. |
| Sources/BlurtEngine/Audio/MicCapture.swift | Updates docs to refer to dictation API upload expectations. |
| README.md | Updates product overview and architecture summary to reflect dictation API + rewrite. |
| BLURTENGINE.md | Updates engine developer guide to reflect dictation API request/response and server-side rewrite behavior. |
| App/Blurt/Blurt/Overlay/OverlayView.swift | Updates UI copy comments to refer to dictation API latency. |
| App/Blurt/Blurt/DictationComposition.swift | Updates composition docs to refer to dictation transcriber. |
| AGENTS.md | Updates canonical repo/architecture notes to reflect dictation API + server-side rewrite contract. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-Authored-By: Claude Fable 5 (Jeeves) <noreply@anthropic.com>
Comment on lines
86
to
88
| - **Fast** — the model responds in under 100 ms. Blurt pre-warms the HTTPS | ||
| connection while you're still speaking and flips to "transcribing" at | ||
| key-up, so text lands about as soon as you stop talking. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Switches
AssemblyAITranscriberfrom the raw Sync STT endpoint (sync.assemblyai.com,X-AAI-Model: u3-sync-pro) to the dictation API (dictation.assemblyai.com/transcribe), so one request per utterance now covers both transcription and the post-processing LLM cleanup step server-side.How
configpart gains an emptyllmblock — per the dictation API contract,{ "llm": {} }requests the service's default cleanup rewrite (remove disfluencies, fix punctuation) under its server-owned guardrail scaffold. The rewrite instruction deliberately stays server-side; nothing is duplicated in the client.text(verbatim),llm_response, andllm_error. The engine pastesllm_responseand falls back totextwhen it's null — the rewrite is best-effort (5 s server budget), sollm_erroris logged as a degradation, never surfaced as a failure (required client behavior per the API doc).X-AAI-Modelis no longer sent — the dictation service pins the STT model when it forwards to sync.TranscriptionPromptstill rides along asconfig.prompt, unchanged — it steers transcription (contextual priming, key terms); cleanup is now thellmblock's job. The dictation service forwardsprompt/sample_rate/channelsto sync as-is, and the audio geometry/limits (16 kHz S16LE, ~0.1–120 s) are unchanged, soMicCaptureandSyncSTTLimitsbehavior is untouched.AGENTS.md,BLURTENGINE.md,README.md) updated, including the "no separate LLM cleanup pass" invariant, which is now "no client-side LLM cleanup pass — the rewrite rides the same single request".Architecture invariants preserved
One POST per utterance (no second request for the cleanup), no streaming, no client-side LLM pass, no new dependencies,
TranscriberProtocolshape unchanged.Tests
llm_response(with and withoutllm_error) degrades totext.llmis always present and always empty.X-AAI-Model, 90 s timeout.Verification
swift build -Xswiftc -warnings-as-errorsclean;swift format lint --strictclean;scripts/check.sh --portablegreen.swift testcould not run locally (no Xcode toolchain on this machine — noTestingmodule); per AGENTS.md, CI'scheck.ymlon macos-26 is the authority on green.Notes for reviewers
detail("Invalid API key"), so the user still sees a meaningful message.🤖 Generated with Claude Code