Skip to content

feat(stt): Use the dictation API for transcription and LLM cleanup#79

Draft
aurpsis-aai wants to merge 2 commits into
mainfrom
aurpsis/dictation-api
Draft

feat(stt): Use the dictation API for transcription and LLM cleanup#79
aurpsis-aai wants to merge 2 commits into
mainfrom
aurpsis/dictation-api

Conversation

@aurpsis-aai

Copy link
Copy Markdown

What

Switches AssemblyAITranscriber from the raw Sync STT endpoint (sync.assemblyai.com, X-AAI-Model: u3-sync-pro) to the dictation API (dictation.assemblyai.com/transcribe), so one request per utterance now covers both transcription and the post-processing LLM cleanup step server-side.

How

  • The JSON config part gains an empty llm block — per the dictation API contract, { "llm": {} } requests the service's default cleanup rewrite (remove disfluencies, fix punctuation) under its server-owned guardrail scaffold. The rewrite instruction deliberately stays server-side; nothing is duplicated in the client.
  • The response now decodes text (verbatim), llm_response, and llm_error. The engine pastes llm_response and falls back to text when it's null — the rewrite is best-effort (5 s server budget), so llm_error is logged as a degradation, never surfaced as a failure (required client behavior per the API doc).
  • X-AAI-Model is no longer sent — the dictation service pins the STT model when it forwards to sync.
  • The request carries the API's documented 90 s client timeout.
  • TranscriptionPrompt still rides along as config.prompt, unchanged — it steers transcription (contextual priming, key terms); cleanup is now the llm block's job. The dictation service forwards prompt/sample_rate/channels to sync as-is, and the audio geometry/limits (16 kHz S16LE, ~0.1–120 s) are unchanged, so MicCapture and SyncSTTLimits behavior is untouched.
  • Docs (AGENTS.md, BLURTENGINE.md, README.md) updated, including the "no separate LLM cleanup pass" invariant, which is now "no client-side LLM cleanup pass — the rewrite rides the same single request".

Architecture invariants preserved

One POST per utterance (no second request for the cleanup), no streaming, no client-side LLM pass, no new dependencies, TranscriberProtocol shape unchanged.

Tests

  • Happy path asserts the rewrite (not the verbatim transcript) is returned, still in a single round-trip.
  • New fallback test: null/absent llm_response (with and without llm_error) degrades to text.
  • New config test: llm is always present and always empty.
  • Wire-contract test updated: raw key auth, no X-AAI-Model, 90 s timeout.

Verification

  • swift build -Xswiftc -warnings-as-errors clean; swift format lint --strict clean; scripts/check.sh --portable green.
  • swift test could not run locally (no Xcode toolchain on this machine — no Testing module); per AGENTS.md, CI's check.yml on macos-26 is the authority on green.

Notes for reviewers

  • The dictation API returns 404 (not 401) for an invalid key; the existing error path already extracts the body's detail ("Invalid API key"), so the user still sees a meaningful message.
  • Latency: the rewrite adds LLM time to the round-trip (bounded at 5 s server-side; typical short clips well under that). The warm-up path is unchanged and now pre-pools the dictation host connection.

🤖 Generated with Claude Code

Point AssemblyAITranscriber at dictation.assemblyai.com/transcribe, which
runs the sync STT transcription and a server-side LLM cleanup rewrite
(disfluencies out, punctuation fixed) in the same single request:

- config gains an empty `llm` block, selecting the service's default
  cleanup instruction and its server-owned guardrails
- the response's `llm_response` is what gets pasted; a null rewrite
  (best-effort, 5 s server budget) degrades to the verbatim `text` with
  the `llm_error` logged, never a user-facing failure
- drop the `X-AAI-Model` header — the dictation service pins the STT
  model server-side
- set the API's documented 90 s client timeout on the request

TranscriptionPrompt still rides along as `config.prompt`, now purely
transcription steering; the pipeline stays one request per utterance
with no client-side LLM pass. Docs updated to match.

Co-Authored-By: Claude Fable 5 (Jeeves) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 24, 2026 15:06

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR migrates AssemblyAITranscriber in BlurtEngine from the Sync STT endpoint to AssemblyAI’s dictation API, so each utterance is handled by a single request that returns both the verbatim transcript and (when available) a server-side LLM cleanup rewrite that the app pastes.

Changes:

  • Switch STT transport to https://dictation.assemblyai.com/transcribe, remove X-AAI-Model, and set a 90s client timeout.
  • Always request the default server-side rewrite via an empty config.llm object and fall back to text when llm_response is missing/null (logging llm_error as degradation).
  • Update engine/app/docs/tests wording and coverage to reflect “dictation API + rewrite” behavior.

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated no comments.

Show a summary per file
File Description
Tests/BlurtEngineTests/TranscriptionPromptTests.swift Updates test descriptions to reflect transcription prompt usage (not Sync-specific).
Tests/BlurtEngineTests/DictationSessionTests.swift Updates comments to reflect dictation API semantics and unchanged short-clip behavior.
Tests/BlurtEngineTests/AssemblyAITranscriberTests.swift Adds/updates tests for dictation endpoint, rewrite preference, fallback behavior, header/timeout contract, and config llm presence.
Sources/BlurtEngine/STT/TranscriptionPrompt.swift Updates docs to describe dictation API prompt usage.
Sources/BlurtEngine/STT/TranscriptionContext.swift Updates docs to remove Sync-specific wording.
Sources/BlurtEngine/STT/TranscriberProtocol.swift Updates docs to reflect dictation API behavior (single-shot result).
Sources/BlurtEngine/STT/SyncSTTLimits.swift Clarifies that Sync model limits still apply behind dictation API forwarding.
Sources/BlurtEngine/STT/AssemblyAITranscriber.swift Core change: dictation endpoint, 90s timeout, remove model header, request rewrite via llm: {}, decode llm_response/llm_error, fallback to verbatim text.
Sources/BlurtEngine/Pipeline/DictationSession+Pipeline.swift Updates pipeline comments to reflect dictation API and “no client-side styling pass”.
Sources/BlurtEngine/Pipeline/DictationSession.swift Updates comments around recording cap and connection warm-up for dictation host.
Sources/BlurtEngine/Config/KeyTermsStore.swift Updates docs to refer to dictation request priming.
Sources/BlurtEngine/Audio/MicCaptureProtocol.swift Updates docs to refer to dictation request upload format.
Sources/BlurtEngine/Audio/MicCapture.swift Updates docs to refer to dictation API upload expectations.
README.md Updates product overview and architecture summary to reflect dictation API + rewrite.
BLURTENGINE.md Updates engine developer guide to reflect dictation API request/response and server-side rewrite behavior.
App/Blurt/Blurt/Overlay/OverlayView.swift Updates UI copy comments to refer to dictation API latency.
App/Blurt/Blurt/DictationComposition.swift Updates composition docs to refer to dictation transcriber.
AGENTS.md Updates canonical repo/architecture notes to reflect dictation API + server-side rewrite contract.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-Authored-By: Claude Fable 5 (Jeeves) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 24, 2026 15:16

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 1 comment.

Comment thread README.md
Comment on lines 86 to 88
- **Fast** — the model responds in under 100 ms. Blurt pre-warms the HTTPS
connection while you're still speaking and flips to "transcribing" at
key-up, so text lands about as soon as you stop talking.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants