diff --git a/AGENTS.md b/AGENTS.md index 1705b9d..c281b11 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,7 +4,7 @@ This file is the canonical agent guide for working with this repository. ## What this is -Blurt is a macOS dictation app powered by [AssemblyAI](https://www.assemblyai.com). Tap or hold a trigger key, speak, and polished text is typed into the focused app. Transcription happens in a single remote AssemblyAI Sync STT call — a per-utterance `prompt` (a transcription directive plus contextual priming built from the focused app/window/field and the user's key terms) rides along with the request, so the transcript comes back already polished (there is no separate LLM pass). The user supplies an API key. Two layers: +Blurt is a macOS dictation app powered by [AssemblyAI](https://www.assemblyai.com). Tap or hold a trigger key, speak, and polished text is typed into the focused app. Transcription happens in a single remote AssemblyAI dictation API call — a per-utterance `prompt` (a transcription directive plus contextual priming built from the focused app/window/field and the user's key terms) rides along with the request, and the same request asks the service for its server-side LLM cleanup rewrite (`config.llm`), so the text that comes back is already polished. The user supplies an API key. Two layers: - **`Sources/BlurtEngine/`** — Swift Package (`swift-tools-version:6.2`, macOS 26+) owning the pipeline. Pure logic behind protocol seams, no AppKit-shell dependencies and **no external SPM dependencies** (just Foundation/Security/AVFoundation plus toolchain modules like Synchronization). - **`App/Blurt/`** — AppKit/SwiftUI shell (Xcode project generated by XcodeGen) that wires the engine to an overlay window, a single setup/settings window, and a hotkey. It has **no external SPM dependencies** — its only package is the local `BlurtEngine` (declared in `App/Blurt/project.yml`); the former `mxcl/AppUpdater` dependency and its in-place self-updater were removed (see [Updates](#updates)). The rebindable dictation trigger is **home-grown** — a `CGEventTap` (`DictationKeyTap`) over a pure engine state machine (`DictationKeyGate`), no package. The engine's hard no-external-dependencies rule is a **rule**; the app merely happens to carry none today. @@ -90,14 +90,14 @@ Update checking is **manual and download-only** — there is no in-place install ```text DictationKeyTap (CGEventTap + DictationKeyGate) → AppCoordinator → DictationSession (actor) → MicCapture ↓ - AssemblyAITranscriber (STT + cleanup, AssemblyAI Sync API: one POST /transcribe) + AssemblyAITranscriber (STT + LLM cleanup, AssemblyAI dictation API: one POST /transcribe) ↓ KeyInjector → focused app (clipboard-paste via a synthesized Cmd-V CGEvent) ``` -**`MicCapture`** (`Sources/BlurtEngine/Audio/MicCapture.swift`) is an actor implementing `MicCaptureProtocol`. It captures with **`AVAudioRecorder`**, recording straight to a temp 16 kHz / mono / 16-bit PCM WAV — the exact geometry the Sync API wants, so `stop()` reads the file back as raw S16LE bytes (`Data`, via `AVAudioFile`'s int16 common format) with no resampling or float-conversion pass — the very blob the Sync request uploads. A **fresh recorder per session** resolves the current default input device at `record()` time; this is deliberate. A long-lived `AVAudioEngine` was used previously and its input graph bound to one device, going stale on a device switch (mic ↔ built-in) — raising `-10868` (`kAudioUnitErr_FormatNotSupported`) or capturing all-zero buffers. **Don't reintroduce an `AVAudioEngine`/`installTap` capture path.** The overlay meter (`levels`) comes from the recorder's dBFS power on a ~30 Hz timer, mapped to `0…1` by `linearLevel(fromPowerDB:)` (floored so room ambient reads as empty bars rather than a meter that never rests). `levels` and `warmUp()` are part of `MicCaptureProtocol` itself, with an empty-stream / no-op default each, so stubs conform with just `start()`/`stop()` and hosts read the meter through the seam they inject. +**`MicCapture`** (`Sources/BlurtEngine/Audio/MicCapture.swift`) is an actor implementing `MicCaptureProtocol`. It captures with **`AVAudioRecorder`**, recording straight to a temp 16 kHz / mono / 16-bit PCM WAV — the exact geometry the dictation API wants, so `stop()` reads the file back as raw S16LE bytes (`Data`, via `AVAudioFile`'s int16 common format) with no resampling or float-conversion pass — the very blob the dictation request uploads. A **fresh recorder per session** resolves the current default input device at `record()` time; this is deliberate. A long-lived `AVAudioEngine` was used previously and its input graph bound to one device, going stale on a device switch (mic ↔ built-in) — raising `-10868` (`kAudioUnitErr_FormatNotSupported`) or capturing all-zero buffers. **Don't reintroduce an `AVAudioEngine`/`installTap` capture path.** The overlay meter (`levels`) comes from the recorder's dBFS power on a ~30 Hz timer, mapped to `0…1` by `linearLevel(fromPowerDB:)` (floored so room ambient reads as empty bars rather than a meter that never rests). `levels` and `warmUp()` are part of `MicCaptureProtocol` itself, with an empty-stream / no-op default each, so stubs conform with just `start()`/`stop()` and hosts read the meter through the seam they inject. -**`AssemblyAITranscriber`** (`Sources/BlurtEngine/STT/AssemblyAITranscriber.swift`) implements `TranscriberProtocol` against AssemblyAI's **Sync** STT API (the endpoint `assembly dictate` uses): a single `POST https://sync.assemblyai.com/transcribe` with the captured audio as a raw S16LE PCM blob (exactly the bytes `MicCapture.stop()` returned — there is no encoding pass) in the `audio` multipart part plus a JSON `config` part (`sample_rate`, `channels`, and a `prompt`), `X-AAI-Model: u3-sync-pro`. The `prompt` (built per utterance by `TranscriptionPrompt.build(context:)`) is prepended to the model's system prompt, so the transcript that comes back is **already polished** — there is no separate LLM post-processing step. The finished transcript comes back in the response body — no `/v2/upload`, no job submission, no polling. Truly synchronous: `TranscriberProtocol.transcribe(pcm:sampleRate:)` is a single `async throws -> String` that returns the whole transcript at once (no streaming/deltas). The sync model handles audio from ~80 ms up to 120 s (server-side ~30 s inference deadline); those limits live in `SyncSTTLimits` and back `DictationSession`'s auto-release timeout (so a held hotkey stops before the cap). +**`AssemblyAITranscriber`** (`Sources/BlurtEngine/STT/AssemblyAITranscriber.swift`) implements `TranscriberProtocol` against AssemblyAI's **dictation** API: a single `POST https://dictation.assemblyai.com/transcribe` with the captured audio as a raw S16LE PCM blob (exactly the bytes `MicCapture.stop()` returned — there is no encoding pass) in the `audio` multipart part plus a JSON `config` part (`sample_rate`, `channels`, a `prompt`, and an empty `llm` block). No model header — the dictation service pins the STT model server-side. The `prompt` (built per utterance by `TranscriptionPrompt.build(context:)`) steers _transcription_; the `llm` block asks the service to run its default LLM cleanup rewrite (remove disfluencies, fix punctuation) over the verbatim transcript, all inside the same request. The response carries both `text` (verbatim) and `llm_response` (the rewrite); the transcriber returns the rewrite and falls back to `text` when `llm_response` is null — the rewrite is best-effort (5 s server-side budget), so a rewrite failure (`llm_error`) is a logged degradation, never a user-facing error. Still no `/v2/upload`, no job submission, no polling: `TranscriberProtocol.transcribe(pcm:sampleRate:)` is a single `async throws -> String` that returns the whole polished text at once (no streaming/deltas). The underlying sync model handles audio from ~80 ms up to 120 s (server-side ~30 s inference deadline); those limits live in `SyncSTTLimits` and back `DictationSession`'s auto-release timeout (so a held hotkey stops before the cap). **`DictationSession`** (`Sources/BlurtEngine/Pipeline/DictationSession.swift`, with the command surface in `+Commands.swift`, the phase stream / signposts in `+Observation.swift`, and the post-release transcribe→inject pipeline in `+Pipeline.swift`) is the central actor. It exposes `press()` / `release()` / `cancel()` / `cancelRecording()`, a synchronous fire-and-forget `submit(_: Command)` mirroring those four for callback-shaped hosts (commands run in exact emit order — the tap wires straight into it, no per-callback `Task` spawning), and a `phase: PipelinePhase` (`idle | recording | transcribing | injecting | failed | cancelled`). A host-supplied `readinessCheck` closure can refuse a press before any capture begins (the app passes a key-presence check, so a missing API key surfaces as `.failed(.apiKeyMissing)` at press time — never after the user has spoken). Its three collaborators are protocol-typed (`MicCaptureProtocol`, `TranscriberProtocol`, `InjectorProtocol`) so tests can stub them — see `Tests/BlurtEngineTests/Stubs/`. STT errors that are already `BlurtError` (e.g. `.apiKeyMissing`) surface verbatim; other errors are wrapped in `.sttFailed`. The pipeline is just transcribe → inject (the cleanup happens server-side), and an empty transcript returns to `.idle` without injecting. Two perceived-latency choices to preserve: `release()` claims `.transcribing` _before_ `mic.stop()` (the stop chime and pill switch fire at key-up, not after the recording is read back — this ordering also closes the double-release window `ReleaseRaceTests` pins), and the press-time AX context read is consumed with a bounded wait (`contextWaitBudget`, 500 ms) so an unresponsive frontmost app costs the transcript its priming, never a multi-second stall. @@ -120,7 +120,7 @@ The trigger is editable in the Shortcut section of the setup/settings window (`H ## Transcription prompt -`Sources/BlurtEngine/STT/TranscriptionPrompt.swift` builds the instruction passed as the Sync STT request's `config.prompt` (see `AssemblyAITranscriber`). It's unit-tested in `Tests/BlurtEngineTests/TranscriptionPromptTests.swift`. Every built prompt opens with the fixed `baseInstruction` — `"Transcribe without speaker labels, audio event descriptions, or emotion markers."` — a negative-exclusion clause that suppresses the annotation markers (`[Speaker]`, `[door creaks]`, `[laughing]`) the model would otherwise paste into the user's text. There is deliberately **no** language directive: pinning the prompt to English hurt non-English transcription, so language is left to the model's own detection — don't reintroduce a "Transcribe in English"-style clause. `build(context:)` wraps that pivot in _contextual priming_: prior-cursor text, the selected text (the highlighted run the dictation will replace, so the model is primed on what's being rewritten — read via `kAXSelectedTextAttribute` and skipped in secure fields), a topic hint from the window title, a destination sentence from the app/field, and inline keyword boosting from the user's key terms. It's phrased per AssemblyAI's Universal-3 Pro prompting guidance (positive/authoritative wording, no "Don't"/"Avoid"/"Never") and stays under the API's 4096-character cap. Note: a "remove filler words (um, uh, like)" _content_ directive is **not** in the model's trained instruction set, so it's a no-op and was deliberately dropped — don't reintroduce it. `build(context:)` returns `nil` when there's no usable context, and passing `prompt: nil` to the transcriber omits the field so the server applies its own default prompt. +`Sources/BlurtEngine/STT/TranscriptionPrompt.swift` builds the instruction passed as the dictation request's `config.prompt` (see `AssemblyAITranscriber`) — it steers the _transcription_, not the LLM rewrite (that's the request's separate `llm` block). It's unit-tested in `Tests/BlurtEngineTests/TranscriptionPromptTests.swift`. Every built prompt opens with the fixed `baseInstruction` — `"Transcribe without speaker labels, audio event descriptions, or emotion markers."` — a negative-exclusion clause that suppresses the annotation markers (`[Speaker]`, `[door creaks]`, `[laughing]`) the model would otherwise paste into the user's text. There is deliberately **no** language directive: pinning the prompt to English hurt non-English transcription, so language is left to the model's own detection — don't reintroduce a "Transcribe in English"-style clause. `build(context:)` wraps that pivot in _contextual priming_: prior-cursor text, the selected text (the highlighted run the dictation will replace, so the model is primed on what's being rewritten — read via `kAXSelectedTextAttribute` and skipped in secure fields), a topic hint from the window title, a destination sentence from the app/field, and inline keyword boosting from the user's key terms. It's phrased per AssemblyAI's Universal-3 Pro prompting guidance (positive/authoritative wording, no "Don't"/"Avoid"/"Never") and stays under the API's 4096-character cap. Note: a "remove filler words (um, uh, like)" _content_ directive is **not** in the STT model's trained instruction set, so it's a no-op and was deliberately dropped — don't reintroduce it here; disfluency removal is the server-side LLM rewrite's job. `build(context:)` returns `nil` when there's no usable context, and passing `prompt: nil` to the transcriber omits the field so the server applies its own default prompt. ## Conventions @@ -174,7 +174,7 @@ regenerated where XcodeGen runs). ## Things deliberately not here - **No `LSUIElement`, no menu-bar-_only_ mode** — Blurt is a Dock app first. It _does_ have a `MenuBarExtra` status item (a live dictation indicator plus a discoverability menu for the otherwise-invisible hotkey — see `MenuBar/MenuBarScene.swift`), but that's a convenience layered on the Dock icon, not a replacement. The macOS notch can hide a status item on a crowded menu bar, so the Dock icon stays the guaranteed entry point and nothing depends on the menu bar item being visible. A menu-bar-_only_ variant (no Dock icon) was tried and reverted twice for exactly that reason — don't drop the Dock icon or add `LSUIElement`. -- **No streaming STT** — the AssemblyAI Sync API returns the full transcript in one response, so the overlay shows "Transcribing…" then jumps to the full text. +- **No streaming STT** — the AssemblyAI dictation API returns the full (already rewritten) text in one response, so the overlay shows "Transcribing…" then jumps to the full text. - **No local models, no model downloads** — transcription (with server-side cleanup) is a remote AssemblyAI API call. There is no on-device ASR or LLM, no model cache, and no download progress UI. The setup/settings window's API-key section (`APIKeyStepView`) saves the key, gated by `hasAPIKey`; the hotkey is disabled until a key is saved. Don't reintroduce a local-model path. - **No in-place / automatic self-update** — the update flow is manual: check → open the DMG in the browser → the user installs it themselves (see [Updates](#updates)). The `mxcl/AppUpdater` dependency and the in-place self-updater it powered were removed, so the app no longer replaces itself on disk or checks in the background. Don't reintroduce a background/auto-updater, a self-replacing install path, or that dependency; extend `UpdateChecker` / `UpdateCheckModel` instead. -- **No separate LLM cleanup pass** — cleanup is folded into the Sync STT call via its `prompt` config field. There is no LLM Gateway client, no `StylerProtocol`, and no post-transcription styling stage. Don't reintroduce one; adjust `TranscriptionPrompt.baseInstruction` (or `build(context:)`) instead. +- **No client-side LLM cleanup pass** — cleanup is the dictation API's server-side rewrite, requested by the `llm` block that rides on the same `/transcribe` call. There is no LLM Gateway client, no `StylerProtocol`, no post-transcription styling stage, and no second request. Don't reintroduce one; the rewrite instruction is server-owned (the empty `llm` block selects it), and transcription steering belongs in `TranscriptionPrompt`. diff --git a/App/Blurt/Blurt/DictationComposition.swift b/App/Blurt/Blurt/DictationComposition.swift index e158612..c0ea6b8 100644 --- a/App/Blurt/Blurt/DictationComposition.swift +++ b/App/Blurt/Blurt/DictationComposition.swift @@ -12,7 +12,7 @@ struct DictationComponents { let transcriber: any TranscriberProtocol let injector: any InjectorProtocol - /// The real pipeline: a fresh `MicCapture`, the AssemblyAI Sync transcriber, + /// The real pipeline: a fresh `MicCapture`, the AssemblyAI dictation transcriber, /// and the clipboard-paste injector. This is what `AppCoordinator` builds, so /// production behavior is unchanged by the test seam existing. static func production() -> DictationComponents { diff --git a/App/Blurt/Blurt/Overlay/OverlayView.swift b/App/Blurt/Blurt/Overlay/OverlayView.swift index a00832e..20b4549 100644 --- a/App/Blurt/Blurt/Overlay/OverlayView.swift +++ b/App/Blurt/Blurt/Overlay/OverlayView.swift @@ -102,7 +102,7 @@ struct OverlayView: View { case .processing: // Matches the site demo's "Transcribing…" label (the demo cross-fades REC → // Transcribing); cyan echoes the demo's --ice. Cross-fades like the bars. - // The label breathes (slow opacity pulse) so the wait for the Sync API + + // The label breathes (slow opacity pulse) so the wait for the dictation API + // paste reads as active work rather than a frozen pill. TranscribingLabel(animated: !reduceMotion) .transition(.opacity) @@ -178,7 +178,7 @@ private struct StatusLineText: View { /// The "Transcribing…" status line with a slow breathing pulse — the processing /// counterpart of the recording bars' idle shimmer, so the pill keeps visibly -/// working while the app waits on the Sync API and pastes the result. Driven by +/// working while the app waits on the dictation API and pastes the result. Driven by /// the same continuous-clock `TimelineView` pattern as `WaveformBars` (never a /// one-shot state toggle). Under Reduce Motion the label holds steady at full /// opacity — exactly the pre-animation rendering. diff --git a/BLURTENGINE.md b/BLURTENGINE.md index f6b28da..18b6b9a 100644 --- a/BLURTENGINE.md +++ b/BLURTENGINE.md @@ -1,6 +1,6 @@ # Building on BlurtEngine -BlurtEngine is the Swift package that powers [Blurt](README.md)'s dictation pipeline: capture speech from the microphone, transcribe it in a single AssemblyAI Sync STT call (with server-side cleanup driven by a per-utterance prompt), and paste the polished text into the focused app. This guide is for developers embedding the engine in their own macOS app or extending it inside this repository. For repo-wide conventions and agent workflow, see [AGENTS.md](AGENTS.md). +BlurtEngine is the Swift package that powers [Blurt](README.md)'s dictation pipeline: capture speech from the microphone, transcribe it in a single AssemblyAI dictation API call (transcription plus a server-side LLM cleanup rewrite, contextually primed by a per-utterance prompt), and paste the polished text into the focused app. This guide is for developers embedding the engine in their own macOS app or extending it inside this repository. For repo-wide conventions and agent workflow, see [AGENTS.md](AGENTS.md). ## What you get @@ -51,17 +51,17 @@ Before the first dictation can succeed the host must have: ```text press() ──▶ MicCapture.start() release() ──▶ MicCapture.stop() → Data (raw S16LE PCM) (16 kHz mono 16-bit PCM) AssemblyAITranscriber.transcribe(pcm:sampleRate:context:) - + focus/context capture (one POST sync.assemblyai.com/transcribe, X-AAI-Model: u3-sync-pro) + + focus/context capture (one POST dictation.assemblyai.com/transcribe: STT + LLM rewrite) + connection warm-up KeyInjector.insert(text, after: priorText) (clipboard paste via synthesized ⌘V) ``` Key properties of the design, which your integration can rely on: -- **One request per utterance, no streaming.** The Sync API returns the complete transcript in the response body — no upload step, no job polling, no incremental deltas. `TranscriberProtocol.transcribe` is a single `async throws -> String`. UIs should show a "transcribing…" state and then the whole result; there is nothing to stream. -- **Cleanup happens server-side.** The per-utterance `config.prompt` (built by `TranscriptionPrompt` from the captured context) rides along with the request, so the transcript comes back already polished. There is no separate LLM pass, no styling stage, and deliberately no hook for one — adjust the prompt instead. +- **One request per utterance, no streaming.** The dictation API returns the complete transcript — and its LLM-rewritten form — in the response body: no upload step, no job polling, no incremental deltas, no second request for the cleanup. `TranscriberProtocol.transcribe` is a single `async throws -> String`. UIs should show a "transcribing…" state and then the whole result; there is nothing to stream. +- **Cleanup happens server-side.** The request's empty `llm` block asks the service for its default cleanup rewrite (remove disfluencies, fix punctuation), applied to the verbatim transcript inside the same call; the per-utterance `config.prompt` (built by `TranscriptionPrompt` from the captured context) primes the _transcription_. The engine pastes `llm_response`, falling back to the verbatim `text` when the best-effort rewrite failed (`llm_error`) — a degradation, never a user-facing error. There is no client-side LLM pass, no styling stage, and deliberately no hook for one. - **Latency is pre-paid where possible.** `press()` fires a detached `warmUp()` at the transcriber (pre-opening the HTTPS connection while the user speaks, ~170 ms saved cold) and kicks off the cross-process accessibility read of the focused field without awaiting it — the read is then consumed at transcribe time with a bounded wait (`DictationSession.contextWaitBudget`, 500 ms), so an unresponsive frontmost app costs the transcript its priming, never a multi-second stall — and never delays the recording indicator. On the way out, `release()` flips the phase to `.transcribing` _before_ reading the recorded audio back, so a host's stop cue fires at key-up rather than after the disk read. -- **A held trigger auto-releases.** `DictationSession` stops recording after `maxRecordingSeconds` (default `SyncSTTLimits.autoReleaseSeconds`, 115 s) so audio never exceeds what the Sync endpoint accepts, and transcribes what it has. Clips shorter than `SyncSTTLimits.minPCMBytes` (~100 ms of audio — an accidental tap) are dropped as a silent no-op rather than sent to earn a 400. +- **A held trigger auto-releases.** `DictationSession` stops recording after `maxRecordingSeconds` (default `SyncSTTLimits.autoReleaseSeconds`, 115 s) so audio never exceeds what the endpoint accepts, and transcribes what it has. Clips shorter than `SyncSTTLimits.minPCMBytes` (~100 ms of audio — an accidental tap) are dropped as a silent no-op rather than sent to earn a 400. ## DictationSession @@ -101,7 +101,7 @@ Failures surface as `PipelinePhase.failed(BlurtError)`: | `.apiKeyMissing` | No AssemblyAI key stored — point the user at your key-entry UI. With a key-presence `readinessCheck`, this surfaces at press time, before any recording. | | `.microphonePermissionDenied` / `.accessibilityPermissionMissing` | Permission gaps; `PermissionsChecker` has openers for the right Settings panes. | | `.audioCaptureFailed(underlying:)` | The mic couldn't start, or captured audio couldn't be processed. | -| `.sttFailed(underlying:)` | The Sync request failed; the underlying error carries the HTTP status and the server's message when available. | +| `.sttFailed(underlying:)` | The dictation request failed; the underlying error carries the HTTP status and the server's message when available. | | `.targetAppLost` / `.noEditableTarget` | Paste-side outcomes. When thrown by `KeyInjector` the transcript is already on the clipboard, and the session degrades them to `.noTarget` rather than a failure. | All cases are `LocalizedError` with user-ready `errorDescription` strings, and `BlurtError` is `Equatable` (wrapped errors compare by NSError domain + code), so phase equality is test-friendly. @@ -121,7 +121,7 @@ func warmUp() async // pre-open the device; default: no-op Only `start()`/`stop()` must be implemented — `levels` and `warmUp()` have defaults, so a stub or headless capture conforms for free while hosts still read the meter and warm the device through the same seam they inject. -`MicCapture` records with `AVAudioRecorder` straight to a temp 16 kHz / mono / 16-bit PCM WAV — exactly the geometry the Sync API wants — and reads it back as raw S16LE bytes on `stop()` (no float detour; the blob uploads as-is). A **fresh recorder per session** resolves the current default input device at `record()` time, which is why device switches (headset ↔ built-in) just work. Do **not** replace this with a long-lived `AVAudioEngine`/`installTap` graph: that design was tried, bound itself to one device, and failed with `-10868` or all-zero buffers on device switches. +`MicCapture` records with `AVAudioRecorder` straight to a temp 16 kHz / mono / 16-bit PCM WAV — exactly the geometry the dictation API wants — and reads it back as raw S16LE bytes on `stop()` (no float detour; the blob uploads as-is). A **fresh recorder per session** resolves the current default input device at `record()` time, which is why device switches (headset ↔ built-in) just work. Do **not** replace this with a long-lived `AVAudioEngine`/`installTap` graph: that design was tried, bound itself to one device, and failed with `-10868` or all-zero buffers on device switches. `MicCapture`'s `levels` is a ~30 Hz meter of the recorder's dBFS power mapped to `0…1` (floored at −50 dBFS so room ambient reads as silence) — feed it to a voice-bars view; it costs nothing when unobserved. Its `warmUp()` pre-creates and prepares a recorder so the first `start()` skips hardware route discovery (Blurt calls it at launch, once mic permission is granted, so warming never triggers the permission prompt). @@ -132,9 +132,9 @@ func transcribe(pcm: Data, sampleRate: Int, context: TranscriptionContext?) asyn func warmUp() async // optional; no-op default ``` -`AssemblyAITranscriber` is a stateless `Sendable` struct. One `POST https://sync.assemblyai.com/transcribe` per utterance: the audio as raw S16LE PCM (the `pcm` blob, byte-for-byte) in the `audio` multipart part, plus a JSON `config` part (`sample_rate`, `channels`, and the rendered `prompt`), with `X-AAI-Model: u3-sync-pro` and the API key in `Authorization`. Its initializer takes an `apiKeyProvider` closure (defaults to `APIKeyStore.get`), a `baseURL`, and an `HTTPTransport` — inject a fake transport (see `Tests/BlurtEngineTests/Stubs/FakeHTTPTransport.swift`) to test against canned responses. `warmUp()` fires a throwaway GET at the host root to pre-pool the connection; it never throws and any failure just means the real request pays connection setup as before. +`AssemblyAITranscriber` is a stateless `Sendable` struct. One `POST https://dictation.assemblyai.com/transcribe` per utterance: the audio as raw S16LE PCM (the `pcm` blob, byte-for-byte) in the `audio` multipart part, plus a JSON `config` part (`sample_rate`, `channels`, the rendered `prompt`, and an empty `llm` block requesting the service's default cleanup rewrite), with the API key in `Authorization` (no model header — the service pins the STT model server-side). The response carries the verbatim `text` and the rewritten `llm_response`; the transcriber returns the rewrite and falls back to `text` when it is null (the rewrite is best-effort — `llm_error` is logged, never surfaced as a failure). Its initializer takes an `apiKeyProvider` closure (defaults to `APIKeyStore.get`), a `baseURL`, and an `HTTPTransport` — inject a fake transport (see `Tests/BlurtEngineTests/Stubs/FakeHTTPTransport.swift`) to test against canned responses. `warmUp()` fires a throwaway GET at the host root to pre-pool the connection; it never throws and any failure just means the real request pays connection setup as before. -The model's limits live in `SyncSTTLimits` (16 kHz sample rate, ~0.1 s–120 s audio, and the auto-release math) — the single source shared by the mic, the session, and the request so recorded and declared geometry can't drift. +The model's limits live in `SyncSTTLimits` (16 kHz sample rate, ~0.1 s–120 s audio, and the auto-release math — the sync STT model behind the dictation service) — the single source shared by the mic, the session, and the request so recorded and declared geometry can't drift. ### `InjectorProtocol` → `KeyInjector` @@ -152,7 +152,7 @@ The session calls `setTargetApp` at press time with the app that was frontmost w Recognition quality comes from per-utterance priming, assembled automatically inside `press()` — hosts don't call these APIs directly, but should know what's collected: - **`TranscriptionContext`** carries the frontmost app name, window title, focused-field label, the text before the caret, the selected text (which a paste will replace), and the user's key terms. It's captured via Accessibility at press time (skipped in secure fields), off the hot path — and consumed at transcribe time with a bounded wait (`DictationSession.contextWaitBudget`), so a hung read is abandoned rather than stalling the transcript. -- **`TranscriptionPrompt.build(context:)`** renders that into the Sync request's `config.prompt`, opening with the fixed `baseInstruction` ("Transcribe without speaker labels, audio event descriptions, or emotion markers.") and staying under the API's 4096-character cap. An empty context yields `nil`, which omits the field so the server applies its own default. Two deliberate omissions, both regression-tested: no language directive (pinning to English hurt non-English speech) and no "remove filler words" clause (not in the model's trained instruction set — a no-op). Don't reintroduce either. +- **`TranscriptionPrompt.build(context:)`** renders that into the dictation request's `config.prompt` (transcription steering only — the cleanup rewrite is the separate `llm` block), opening with the fixed `baseInstruction` ("Transcribe without speaker labels, audio event descriptions, or emotion markers.") and staying under the API's 4096-character cap. An empty context yields `nil`, which omits the field so the server applies its own default. Two deliberate omissions, both regression-tested: no language directive (pinning to English hurt non-English speech) and no "remove filler words" clause (not in the model's trained instruction set — a no-op). Don't reintroduce either. - **`KeyTermsStore`** persists the user's domain vocabulary (names, jargon) in `UserDefaults`; `DictationSession` re-reads it at every press via its `keyTermsProvider` closure, so Settings edits apply to the next utterance without rebuilding the session. Pass your own provider to source terms from elsewhere. For key storage, compose against **`APIKeyGateway`** — the injectable get/set/`hasKey` seam over the key store. `ProductionAPIKeyStore` forwards to the Keychain-backed `APIKeyStore`; `InMemoryAPIKeyStore` is a ready-made in-memory conformance for tests and harnesses (Blurt's XCUITest runs use it so the real Keychain item is never touched, and its `hasKey` backs the session's `readinessCheck`). For a settings UI, **`APIKeySubmission`** wraps the gateway with the validate-then-save flow (`submit(_:)` → valid / invalid / unreachable / saveFailed, via `APIKeyValidator`): it saves only a key AssemblyAI actively accepts, so an unverified key never persists. @@ -192,7 +192,7 @@ Run `swift test` for the engine suites (`--filter DictationSessionTests` for one Each of these was tried the other way and reverted; the longer stories are in [AGENTS.md](AGENTS.md) and the source comments: - **No external SPM dependencies in the engine.** Foundation/Security/AVFoundation only. -- **No streaming STT, no local models, no separate LLM cleanup pass.** One Sync request per utterance is the architecture; cleanup belongs in `TranscriptionPrompt`. +- **No streaming STT, no local models, no client-side LLM cleanup pass.** One dictation request per utterance is the architecture; the cleanup rewrite is server-side (the request's `llm` block), and transcription steering belongs in `TranscriptionPrompt`. - **No `AVAudioEngine`/`installTap` capture path.** Fresh `AVAudioRecorder` per session, resolved at record time. - **Paste is always clipboard-based** (save → write → ⌘V → settle → restore), with the copied-to-clipboard degradation for lost targets. - **No English-pinning or filler-word clauses in the prompt.** diff --git a/README.md b/README.md index ccee886..0ed02ee 100644 --- a/README.md +++ b/README.md @@ -63,7 +63,7 @@ more or less what this app lets you do to your Mac. Built entirely native (AppKit + SwiftUI — no Electron, no web views). The whole pipeline lives in **BlurtEngine**, a dependency-free Swift 6 package: mic capture, one synchronous `POST` to -[AssemblyAI's Sync STT API](https://www.assemblyai.com), and a clipboard paste +[AssemblyAI's dictation API](https://www.assemblyai.com), and a clipboard paste into the focused app. No local models, no upload-then-poll job queue, no background daemons — audio in, polished text out, one HTTP request per utterance. @@ -78,10 +78,11 @@ utterance. (right ⌘ by default; right ⌥ and `fn` also available). Tap to toggle, hold for push-to-talk. The event tap swallows nothing: a lone modifier types nothing anyway, and combos like ⌘C pass through untouched. -- **Polished in one step** — each utterance rides to AssemblyAI's Sync STT API +- **Polished in one step** — each utterance rides to AssemblyAI's dictation API with a contextual prompt built from the focused app, window, and field, the - text around your cursor, and your own key terms — so the transcript comes - back already cleaned up. No separate LLM pass, no model downloads. + text around your cursor, and your own key terms; the same call runs a + server-side LLM cleanup (disfluencies out, punctuation fixed), so the text + comes back already polished. No second request, no model downloads. - **Fast** — the model responds in under 100 ms. Blurt pre-warms the HTTPS connection while you're still speaking and flips to "transcribing" at key-up, so text lands about as soon as you stop talking. @@ -162,8 +163,8 @@ directories, so the app needs a stable install path to be usable at all. Sources/BlurtEngine/ Swift 6 package owning the pipeline — no external dependencies Audio/ MicCapture: fresh AVAudioRecorder per session, 16 kHz mono PCM, live level meter; DX7/Juno-106 sound packs - STT/ AssemblyAITranscriber: one POST to sync.assemblyai.com/transcribe - (u3-sync-pro) + TranscriptionPrompt contextual priming + STT/ AssemblyAITranscriber: one POST to dictation.assemblyai.com/transcribe + (STT + LLM rewrite) + TranscriptionPrompt contextual priming Pipeline/ DictationSession actor: press/release/cancel commands, phase stream, auto-release before the API's recording cap Hotkey/ DictationKeyGate/Router: pure, unit-tested state machine for the @@ -181,7 +182,7 @@ App/Blurt/ AppKit/SwiftUI shell (Xcode project generated by XcodeG ``` The engine is a standalone package you can embed to build your own dictation -app — mic capture, Sync transcription, and paste-into-the-focused-app behind +app — mic capture, dictation-API transcription, and paste-into-the-focused-app behind three protocol seams, fully stubbed in tests. [`BLURTENGINE.md`](./BLURTENGINE.md) is the developer guide. diff --git a/Sources/BlurtEngine/Audio/MicCapture.swift b/Sources/BlurtEngine/Audio/MicCapture.swift index e4399b6..db31415 100644 --- a/Sources/BlurtEngine/Audio/MicCapture.swift +++ b/Sources/BlurtEngine/Audio/MicCapture.swift @@ -3,7 +3,7 @@ import Foundation import os /// Captures mic audio with `AVAudioRecorder`, which records straight to the -/// 16 kHz / mono / 16-bit PCM the Sync STT API wants — so there's no manual tap, +/// 16 kHz / mono / 16-bit PCM the dictation API wants — so there's no manual tap, /// sample-rate conversion, or PCM plumbing here. Each session uses a freshly /// created recorder, which resolves the *current* default input device at /// `record()` time. That's the whole reason this is no longer an `AVAudioEngine`: @@ -20,7 +20,7 @@ public actor MicCapture: MicCaptureProtocol { public nonisolated let levels: AsyncStream private nonisolated let levelsContinuation: AsyncStream.Continuation - /// The geometry the recorder converts hardware audio to on the fly. The Sync + /// The geometry the recorder converts hardware audio to on the fly. The dictation /// API's rate (`SyncSTTLimits.sampleRate`) — the same one the pipeline hands /// the transcriber — so `stop()` returns bytes ready to upload with no /// resampling or re-encoding pass. @@ -159,13 +159,13 @@ public actor MicCapture: MicCaptureProtocol { // MARK: - File helpers - /// Read a recorded PCM file back as raw S16LE bytes — the Sync API's upload + /// Read a recorded PCM file back as raw S16LE bytes — the dictation API's upload /// encoding. The on-disk WAV already holds 16-bit int samples, so asking /// `AVAudioFile` for the int16 common format makes this a straight copy-out: /// no detour through Float32 (which the default `processingFormat` would /// impose, and which the transcriber would only convert straight back). Int16 /// is host-endian; Apple platforms (arm64/x86_64) are little-endian, so the - /// bytes are already the S16LE the Sync API expects. + /// bytes are already the S16LE the dictation API expects. static func decodePCM(fromFileAt url: URL) throws -> Data { let file = try AVAudioFile(forReading: url, commonFormat: .pcmFormatInt16, interleaved: true) let frameCount = AVAudioFrameCount(file.length) diff --git a/Sources/BlurtEngine/Audio/MicCaptureProtocol.swift b/Sources/BlurtEngine/Audio/MicCaptureProtocol.swift index 458cb92..9323dae 100644 --- a/Sources/BlurtEngine/Audio/MicCaptureProtocol.swift +++ b/Sources/BlurtEngine/Audio/MicCaptureProtocol.swift @@ -4,7 +4,7 @@ public protocol MicCaptureProtocol: Sendable { /// Begin capturing 16 kHz mono 16-bit PCM. Throws on permission/device failure. func start() async throws /// Stop capture and return the captured audio as raw S16LE PCM bytes — the - /// exact encoding the Sync STT request uploads, so no conversion pass sits on + /// exact encoding the dictation request uploads, so no conversion pass sits on /// the release hot path. Throws if the captured audio couldn't be read back, /// so the pipeline can surface an error instead of silently dropping speech. func stop() async throws -> Data diff --git a/Sources/BlurtEngine/Config/KeyTermsStore.swift b/Sources/BlurtEngine/Config/KeyTermsStore.swift index f5c92f2..4573a19 100644 --- a/Sources/BlurtEngine/Config/KeyTermsStore.swift +++ b/Sources/BlurtEngine/Config/KeyTermsStore.swift @@ -1,7 +1,7 @@ import Foundation /// Storage for the user's dictation "key terms" — a comma-separated list of -/// domain words (names, jargon, product names) that get folded into the Sync STT +/// domain words (names, jargon, product names) that get folded into the dictation /// request `prompt` as vocabulary priming, so the model is more likely to spell /// them correctly (see `TranscriptionPrompt.build`). /// diff --git a/Sources/BlurtEngine/Pipeline/DictationSession+Pipeline.swift b/Sources/BlurtEngine/Pipeline/DictationSession+Pipeline.swift index d34bbfb..a5afb66 100644 --- a/Sources/BlurtEngine/Pipeline/DictationSession+Pipeline.swift +++ b/Sources/BlurtEngine/Pipeline/DictationSession+Pipeline.swift @@ -5,7 +5,7 @@ import os // the press-time context read — split from `DictationSession.swift` to stay // within the lint file-length budget, like `+Commands` and `+Observation`. extension DictationSession { - /// Sample rate the mic delivers and the Sync request declares (`SyncSTTLimits`). + /// Sample rate the mic delivers and the dictation request declares (`SyncSTTLimits`). private static let captureSampleRate = SyncSTTLimits.sampleRate /// The longest `runTranscribeInject` waits for the press-time AX @@ -19,12 +19,12 @@ extension DictationSession { static let contextWaitBudget: Duration = .milliseconds(500) func runTranscribeInject(pcm: Data) async { - // Times the full post-release hot path — Sync STT round trip plus the paste + // Times the full post-release hot path — dictation round trip plus the paste // (including the clipboard settle) — across every exit (short-clip no-op, // empty transcript, failure, cancel, or a completed paste). let pipelineInterval = Self.signposter.beginInterval(Self.pipelineSignpostName) defer { Self.signposter.endInterval(Self.pipelineSignpostName, pipelineInterval) } - // A clip too short for the Sync model (an accidental brief tap) would only + // A clip too short for the STT model (an accidental brief tap) would only // earn a 400 — drop it as a silent no-op, like an empty transcript, rather // than calling the API and surfacing an error. guard pcm.count >= SyncSTTLimits.minPCMBytes else { @@ -45,8 +45,8 @@ extension DictationSession { } contextStream = nil - // The Sync STT API applies the cleanup prompt server-side, so the transcript - // it returns is already the final, polished text — there is no separate + // The dictation API runs the cleanup rewrite server-side, so the text it + // returns is already the final, polished text — there is no client-side // styling pass. guard let text = await transcribe(pcm: pcm) else { return } @@ -90,7 +90,7 @@ extension DictationSession { } } - /// Runs the single Sync STT request. Returns the transcript, or nil if it + /// Runs the single dictation request. Returns the transcript, or nil if it /// failed (phase set to `.failed`). private func transcribe(pcm: Data) async -> String? { do { diff --git a/Sources/BlurtEngine/Pipeline/DictationSession.swift b/Sources/BlurtEngine/Pipeline/DictationSession.swift index aa4b341..c26bd59 100644 --- a/Sources/BlurtEngine/Pipeline/DictationSession.swift +++ b/Sources/BlurtEngine/Pipeline/DictationSession.swift @@ -30,8 +30,8 @@ public actor DictationSession { /// rebuilding the session. Defaults to reading `KeyTermsStore`. private let keyTermsProvider: @Sendable () -> [String] /// Auto-releases the hotkey after this long so a held key can't run forever. - /// Defaults to just under the AssemblyAI Sync STT audio cap (see - /// `SyncSTTLimits`) — recording past it would only produce audio the sync + /// Defaults to just under the dictation API's audio cap (see + /// `SyncSTTLimits`) — recording past it would only produce audio the /// endpoint rejects, so we stop early and transcribe what we have. private let maxRecordingSeconds: Double /// Clock the auto-release timer and the context-wait budget (`+Pipeline`) @@ -162,7 +162,7 @@ public actor DictationSession { // mutually exclusive). let pressInterval = Self.signposter.beginInterval(Self.pressSignpostName) do { - // Pre-open the Sync connection while the user speaks, so the first dictation after an idle + // Pre-open the dictation connection while the user speaks, so the first dictation after an idle // gap doesn't pay DNS+TCP+TLS on the transcribe hot path (~170 ms cold, measured). Detached // + fire-and-forget: it must never delay recording, and a failure is harmless (the request // just pays setup as before); warming every press is cheap since a hot pool just reuses it. diff --git a/Sources/BlurtEngine/STT/AssemblyAITranscriber.swift b/Sources/BlurtEngine/STT/AssemblyAITranscriber.swift index fd3937d..e5e9db1 100644 --- a/Sources/BlurtEngine/STT/AssemblyAITranscriber.swift +++ b/Sources/BlurtEngine/STT/AssemblyAITranscriber.swift @@ -1,32 +1,37 @@ import Foundation import os -/// Latency instrumentation for the Sync round-trip. Findable via: +/// Latency instrumentation for the dictation round-trip. Findable via: /// log show --predicate 'subsystem == "dev.alex.blurt" && category == "Transcriber"' --last 1h /// File-scoped so both `send(_:body:audioDurationMs:)` (wall-clock) and /// `MetricsLogger` (the DNS/TCP/TLS/TTFB split) can write to it. private let transcriberLog = Logger(subsystem: BlurtIdentity.subsystem, category: "Transcriber") -/// `TranscriberProtocol` backed by AssemblyAI's **Sync** Speech-to-Text API. +/// `TranscriberProtocol` backed by AssemblyAI's **dictation** API. /// -/// Mirrors the endpoint used by the `assembly dictate` CLI command: a single -/// `POST sync.assemblyai.com/transcribe` carries the captured audio (raw S16LE -/// PCM, exactly the bytes the mic recorded — there is no re-encoding pass) -/// plus a JSON `config` part, and the finished transcript comes back in the -/// response body. No upload step, no job submission, no polling — one request -/// per utterance. The Universal-3 sync model (`u3-sync-pro`) handles audio from -/// ~80 ms up to 120 s with a server-side inference deadline of ~30 s. +/// A single `POST dictation.assemblyai.com/transcribe` carries the captured +/// audio (raw S16LE PCM, exactly the bytes the mic recorded — there is no +/// re-encoding pass) plus a JSON `config` part, and the response body carries +/// both the verbatim transcript and — because the config requests one via its +/// `llm` block — an LLM-rewritten version with disfluencies removed and +/// punctuation fixed. No upload step, no job submission, no polling — one +/// request per utterance covers transcription *and* cleanup. The service picks +/// the STT model server-side and handles audio from ~80 ms up to 120 s; the +/// rewrite is best-effort with a ~5 s server-side deadline, so a rewrite +/// failure still returns the verbatim transcript (`llm_response` null). public struct AssemblyAITranscriber: TranscriberProtocol { private let apiKeyProvider: @Sendable () -> String? private let baseURL: URL private let transport: any HTTPTransport - /// Required on every Sync API request — selects the synchronous STT model. - private static let syncModel = "u3-sync-pro" + /// Client timeout the dictation API documents: generous over the sync + /// upstream's ~30 s inference deadline plus the rewrite's 5 s budget, so the + /// server — not the client — decides when a slow request has failed. + private static let requestTimeout: TimeInterval = 90 public init( apiKeyProvider: @escaping @Sendable () -> String? = { APIKeyStore.get() }, - baseURL: URL = URL(staticString: "https://sync.assemblyai.com"), + baseURL: URL = URL(staticString: "https://dictation.assemblyai.com"), transport: any HTTPTransport = URLSession.shared ) { self.apiKeyProvider = apiKeyProvider @@ -34,7 +39,7 @@ public struct AssemblyAITranscriber: TranscriberProtocol { self.transport = transport } - // MARK: - Sync request + // MARK: - Dictation request public func transcribe( pcm: Data, sampleRate: Int, context: TranscriptionContext? @@ -48,8 +53,8 @@ public struct AssemblyAITranscriber: TranscriberProtocol { var request = URLRequest(url: baseURL.appendingPathComponent("transcribe")) request.httpMethod = "POST" + request.timeoutInterval = Self.requestTimeout request.setValue(apiKey, forHTTPHeaderField: "Authorization") - request.setValue(Self.syncModel, forHTTPHeaderField: "X-AAI-Model") request.setValue( "multipart/form-data; boundary=\(boundary)", forHTTPHeaderField: "Content-Type") @@ -57,20 +62,26 @@ public struct AssemblyAITranscriber: TranscriberProtocol { let sampleCount = pcm.count / SyncSTTLimits.bytesPerSample let audioDurationMs = Int((Double(sampleCount) / Double(sampleRate)) * 1000) let data = try await send(request, body: body, audioDurationMs: audioDurationMs) - guard let response = try? JSONDecoder().decode(SyncTranscriptResponse.self, from: data) else { + guard let response = try? JSONDecoder().decode(DictationResponse.self, from: data) else { throw AssemblyAIError.malformedResponse } - return response.text + if let error = response.llmError { + transcriberLog.warning( + "llm rewrite unavailable (\(error, privacy: .public)); using verbatim transcript") + } + // The rewrite is best-effort: a null `llm_response` (rewrite failed or + // timed out) degrades to the verbatim transcript rather than an error. + return response.llmResponse ?? response.text } - /// Pre-open and pool a connection to the Sync host so the next `transcribe` - /// reuses it instead of paying DNS+TCP+TLS on the hot path (~170 ms cold, more - /// on mobile — measured). A throwaway GET to the host root is enough to - /// establish the HTTP/2 connection `URLSession` then reuses for the POST to - /// `/transcribe`; the response (an auth-less 4xx) is discarded. No `X-AAI-Model` - /// or key, so it never reaches the model or counts as a transcription. A short - /// timeout keeps a dead network from leaving the task hanging. Fire-and-forget: - /// any error is swallowed, since the real request degrades to the old behavior. + /// Pre-open and pool a connection to the dictation host so the next + /// `transcribe` reuses it instead of paying DNS+TCP+TLS on the hot path + /// (~170 ms cold, more on mobile — measured). A throwaway GET to the host + /// root is enough to establish the HTTP/2 connection `URLSession` then reuses + /// for the POST to `/transcribe`; the response (an auth-less 4xx) is + /// discarded. No key, so it never counts as a transcription. A short timeout + /// keeps a dead network from leaving the task hanging. Fire-and-forget: any + /// error is swallowed, since the real request degrades to the old behavior. public func warmUp() async { var request = URLRequest(url: baseURL) request.httpMethod = "GET" @@ -84,12 +95,14 @@ public struct AssemblyAITranscriber: TranscriberProtocol { /// Builds the JSON `config` part sent alongside the audio. The context /// `prompt` is included only when non-empty; a nil or blank prompt omits the - /// field so the server applies its default prompt. Internal so tests can - /// assert the prompt wiring without inspecting the multipart upload body - /// (which `URLProtocol` mocks can't observe reliably for `upload(from:)`). + /// field so the server applies its default prompt. The `llm` block is always + /// present: an empty object requests the service's default cleanup rewrite + /// (remove disfluencies, fix punctuation). Internal so tests can assert the + /// prompt wiring without inspecting the multipart upload body (which + /// `URLProtocol` mocks can't observe reliably for `upload(from:)`). func makeConfigData(sampleRate: Int, prompt: String?) throws -> Data { try JSONEncoder().encode( - SyncConfig( + DictationConfig( sampleRate: sampleRate, channels: 1, prompt: prompt.trimmedNonEmpty() @@ -97,8 +110,8 @@ public struct AssemblyAITranscriber: TranscriberProtocol { ) } - /// Builds the `audio` (raw PCM) + `config` (JSON) multipart payload the Sync - /// API expects, matching the field names `assembly dictate` sends. + /// Builds the `audio` (raw PCM) + `config` (JSON) multipart payload the + /// dictation API expects. private func multipartBody(pcm: Data, config: Data, boundary: String) -> Data { var body = Data() // Reserve up front (payload + a generous allowance for the boundary/header @@ -136,7 +149,7 @@ public struct AssemblyAITranscriber: TranscriberProtocol { let (data, response) = try await transport.upload(for: request, from: body, delegate: metrics) let wallMs = (clock.now - start).milliseconds transcriberLog.info( - "sync round-trip audioMs=\(audioDurationMs, privacy: .public) wallMs=\(wallMs, format: .fixed(precision: 0), privacy: .public)" + "dictation round-trip audioMs=\(audioDurationMs, privacy: .public) wallMs=\(wallMs, format: .fixed(precision: 0), privacy: .public)" ) guard let http = response as? HTTPURLResponse else { return data } guard (200..<300).contains(http.statusCode) else { @@ -145,7 +158,7 @@ public struct AssemblyAITranscriber: TranscriberProtocol { return data } - /// Best human-readable explanation for a non-2xx response. The Sync API isn't + /// Best human-readable explanation for a non-2xx response. The API isn't /// consistent about the field name across error classes (`error`, `message`, /// and `detail` have all been seen), so try each; failing that, fall back to /// the raw body text (trimmed and capped) so a failure never reaches the user @@ -162,25 +175,46 @@ public struct AssemblyAITranscriber: TranscriberProtocol { // MARK: - Wire types - private struct SyncConfig: Encodable { + private struct DictationConfig: Encodable { let sampleRate: Int let channels: Int /// Custom transcription instruction. Encoded only when non-nil (the /// synthesized `encode` uses `encodeIfPresent` for optionals), so omitting - /// it falls back to the server's default prompt. + /// it falls back to the server's default prompt. Steers *transcription*; + /// the cleanup rewrite is the `llm` block's job. let prompt: String? + /// The rewrite request. An empty object selects the service's default + /// cleanup instruction; per the API's `instruction`-mode rules, output + /// format and don't-answer-the-text safeguards are enforced server-side, + /// so nothing rides along here. + let llm = LLMRewrite() enum CodingKeys: String, CodingKey { case sampleRate = "sample_rate" case channels case prompt + case llm } } - private struct SyncTranscriptResponse: Decodable { + /// Encodes as `{}` — the default-cleanup rewrite request. + private struct LLMRewrite: Encodable {} + + private struct DictationResponse: Decodable { + /// The verbatim transcript — always present, never altered by the LLM. let text: String + /// The rewritten transcript, or nil when the rewrite failed or timed out. + /// Never empty when present: the service nulls out empty rewrites. + let llmResponse: String? + /// `"timeout"` or `"error"` when a requested rewrite failed. + let llmError: String? + enum CodingKeys: String, CodingKey { + case text + case llmResponse = "llm_response" + case llmError = "llm_error" + } } - /// A Sync API failure body. The endpoint labels the explanation differently + /// A dictation API failure body. The endpoint labels the explanation differently /// across error classes, so pull it from whichever of `error` / `message` / /// `detail` is present and string-valued (a non-string `detail`, e.g. FastAPI's /// validation array, is ignored — the caller then falls back to the raw body). @@ -201,7 +235,7 @@ public struct AssemblyAITranscriber: TranscriberProtocol { } } -/// Per-request `URLSessionTaskDelegate` that logs the Sync round-trip's latency +/// Per-request `URLSessionTaskDelegate` that logs the dictation round-trip's latency /// breakdown from `URLSessionTaskMetrics`: how much was connection setup /// (DNS/TCP/TLS — warmable by pre-connecting at record-start) versus server /// inference (`ttfbMs` ≈ requestStart→responseStart). `reused=true` means the @@ -222,7 +256,7 @@ private final class MetricsLogger: NSObject, URLSessionTaskDelegate, @unchecked } transcriberLog.info( """ - sync metrics audioMs=\(self.audioDurationMs, privacy: .public) \ + dictation metrics audioMs=\(self.audioDurationMs, privacy: .public) \ reused=\(t.isReusedConnection, privacy: .public) \ dnsMs=\(ms(t.domainLookupStartDate, t.domainLookupEndDate), privacy: .public) \ connectMs=\(ms(t.connectStartDate, t.connectEndDate), privacy: .public) \ diff --git a/Sources/BlurtEngine/STT/SyncSTTLimits.swift b/Sources/BlurtEngine/STT/SyncSTTLimits.swift index 32c3651..7a120d4 100644 --- a/Sources/BlurtEngine/STT/SyncSTTLimits.swift +++ b/Sources/BlurtEngine/STT/SyncSTTLimits.swift @@ -1,8 +1,9 @@ -/// Limits imposed by AssemblyAI's Sync STT API. `DictationSession` auto-releases -/// a held hotkey just before the cap so a long press never records audio the -/// endpoint would reject. +/// Limits imposed by the Sync STT model behind AssemblyAI's dictation API +/// (the dictation service forwards audio to Sync unchanged, so its caps apply +/// end to end). `DictationSession` auto-releases a held hotkey just before the +/// cap so a long press never records audio the endpoint would reject. public enum SyncSTTLimits { - /// Sample rate of the 16 kHz / mono / 16-bit PCM geometry the Sync API + /// Sample rate of the 16 kHz / mono / 16-bit PCM geometry the API /// expects. The single definition shared by `MicCapture` (which records at /// this rate) and `DictationSession` (which declares it on the request), so /// the recorded and declared rates can't drift apart. diff --git a/Sources/BlurtEngine/STT/TranscriberProtocol.swift b/Sources/BlurtEngine/STT/TranscriberProtocol.swift index 1295073..c13ca5a 100644 --- a/Sources/BlurtEngine/STT/TranscriberProtocol.swift +++ b/Sources/BlurtEngine/STT/TranscriberProtocol.swift @@ -3,7 +3,7 @@ import Foundation public protocol TranscriberProtocol: Sendable { /// Transcribe captured audio (raw S16LE mono PCM at `sampleRate` — the bytes /// `MicCaptureProtocol.stop()` returns, uploaded as-is) into text. - /// The Sync STT API resolves an utterance to a single final transcript, so + /// The dictation API resolves an utterance to a single final transcript, so /// this returns that transcript in one shot (no incremental deltas). /// /// `context` carries per-utterance priming (focused app + text before the diff --git a/Sources/BlurtEngine/STT/TranscriptionContext.swift b/Sources/BlurtEngine/STT/TranscriptionContext.swift index 2d08878..5a5fc3e 100644 --- a/Sources/BlurtEngine/STT/TranscriptionContext.swift +++ b/Sources/BlurtEngine/STT/TranscriptionContext.swift @@ -1,4 +1,4 @@ -/// Per-utterance context the Sync STT model is trained to use as *contextual* +/// Per-utterance context the STT model is trained to use as *contextual* /// priming — it improves recognition accuracy (vocabulary, continuity, /// capitalization) without changing the output format. Gathered at dictation /// start from the focused app and the text preceding the cursor, then rendered diff --git a/Sources/BlurtEngine/STT/TranscriptionPrompt.swift b/Sources/BlurtEngine/STT/TranscriptionPrompt.swift index 9ff6170..02dfc6d 100644 --- a/Sources/BlurtEngine/STT/TranscriptionPrompt.swift +++ b/Sources/BlurtEngine/STT/TranscriptionPrompt.swift @@ -1,5 +1,5 @@ -/// Builds the instruction sent to AssemblyAI's Sync STT API as the `prompt` -/// field of the request `config` (see `AssemblyAITranscriber`). The Sync model +/// Builds the instruction sent to the dictation API as the `prompt` +/// field of the request `config` (see `AssemblyAITranscriber`). The STT model /// prepends this to its own system prompt. /// /// Every built prompt opens with a fixed `baseInstruction` — a plain-text @@ -42,13 +42,13 @@ enum TranscriptionPrompt { static let baseInstruction = "Transcribe without speaker labels, audio event descriptions, or emotion markers." - /// Hard cap the Sync API places on `config.prompt`; a longer prompt fails + /// Hard cap the API places on `config.prompt`; a longer prompt fails /// the whole request, so `build` must never exceed it. The contextual blocks /// are all clipped upstream in `FocusCapture`; the user's key terms are the /// one unbounded input, so `build` fits them to whatever budget remains. static let characterCap = 4096 - /// Renders `context` into a Sync STT prompt, or `nil` when there is no usable + /// Renders `context` into a transcription prompt, or `nil` when there is no usable /// context (the server then applies its own default prompt). static func build(context: TranscriptionContext?) -> String? { guard let context else { return nil } diff --git a/Tests/BlurtEngineTests/AssemblyAITranscriberTests.swift b/Tests/BlurtEngineTests/AssemblyAITranscriberTests.swift index df63efd..14d0ce7 100644 --- a/Tests/BlurtEngineTests/AssemblyAITranscriberTests.swift +++ b/Tests/BlurtEngineTests/AssemblyAITranscriberTests.swift @@ -11,7 +11,7 @@ import Testing @Suite("HTTP network clients") struct HTTPClientTests { - @Test("transcriber posts to the sync endpoint and returns the transcript") + @Test("transcriber posts to the dictation endpoint and returns the rewritten text") func transcribeHappyPath() async throws { let hits = Counter() let transport = FakeHTTPTransport { request in @@ -19,15 +19,30 @@ struct HTTPClientTests { guard request.url?.path.hasSuffix("/transcribe") == true, request.httpMethod == "POST" else { return (404, Data()) } - return (200, json(["text": "hello world"])) + return (200, json(["text": "um hello world", "llm_response": "Hello world."])) } let result = try await collectTranscript(makeTranscriber(apiKey: "test-key", transport: transport)) - #expect(result == "hello world") - // Single round-trip: no upload/submit/poll fan-out. + // The LLM rewrite — not the verbatim transcript — is what gets pasted. + #expect(result == "Hello world.") + // Single round-trip: transcription + rewrite ride one request, no fan-out. #expect(hits.value == 1) } + @Test("transcriber falls back to the verbatim transcript when no rewrite came back") + func transcribeFallsBackWithoutRewrite() async throws { + // A null / absent `llm_response` (the rewrite is best-effort) must degrade + // to the verbatim transcript, never to an error or an empty paste. + for body in [ + Data(#"{"text":"hello world","llm_response":null,"llm_error":"timeout"}"#.utf8), + json(["text": "hello world"]), + ] { + let transport = FakeHTTPTransport { _ in (200, body) } + let result = try await collectTranscript(makeTranscriber(apiKey: "test-key", transport: transport)) + #expect(result == "hello world") + } + } + @Test("transcriber succeeds with a real context (builds and sends a prompt)") func transcribeWithContext() async throws { let transport = FakeHTTPTransport { request in @@ -47,15 +62,17 @@ struct HTTPClientTests { #expect(result == "hello world") } - @Test("transcribe sends the raw key and the sync model selector as headers") + @Test("transcribe sends the raw key, no model header, and the documented timeout") func transcribeSendsAuthAndModelHeaders() async throws { let transport = FakeHTTPTransport { request in - // The wire contract: the raw key in Authorization (no "Bearer" prefix), - // the sync model selector, and a boundary-tagged multipart body. Anything - // else gets a 400 so a header regression fails loudly here. + // The wire contract: the raw key in Authorization (no "Bearer" prefix), a + // boundary-tagged multipart body, no `X-AAI-Model` (the dictation service + // pins the STT model server-side), and the API's documented 90 s client + // timeout. Anything else gets a 400 so a regression fails loudly here. guard request.value(forHTTPHeaderField: "Authorization") == "test-key", - request.value(forHTTPHeaderField: "X-AAI-Model") == "u3-sync-pro", - request.value(forHTTPHeaderField: "Content-Type")?.hasPrefix("multipart/form-data; boundary=") == true + request.value(forHTTPHeaderField: "X-AAI-Model") == nil, + request.value(forHTTPHeaderField: "Content-Type")?.hasPrefix("multipart/form-data; boundary=") == true, + request.timeoutInterval == 90 else { return (400, Data()) } return (200, json(["text": "ok"])) } @@ -70,10 +87,9 @@ struct HTTPClientTests { let transport = FakeHTTPTransport { request in _ = hits.next() // The warm-up must be a bare, auth-less GET off the /transcribe path — - // carrying the key or model header would make it count as a transcription. + // carrying the key would make it count as a transcription. if request.httpMethod == "GET", request.url?.path.hasSuffix("/transcribe") == false, - request.value(forHTTPHeaderField: "Authorization") == nil, - request.value(forHTTPHeaderField: "X-AAI-Model") == nil + request.value(forHTTPHeaderField: "Authorization") == nil { _ = getHits.next() } @@ -140,6 +156,21 @@ struct HTTPClientTests { #expect(object?["channels"] as? Int == 1) } + @Test("config part always requests the default cleanup rewrite") + func configRequestsDefaultRewrite() throws { + // `llm` must be present and empty on every request: present so the service + // runs the rewrite at all, empty so the server-owned default cleanup + // instruction (and its guardrails) applies rather than a client-side copy. + for prompt in ["CONTEXT. Transcribe.", nil] { + let config = try makeTranscriber(apiKey: "test-key") + .makeConfigData(sampleRate: 16_000, prompt: prompt) + let object = try JSONSerialization.jsonObject(with: config) as? [String: Any] + let llm = object?["llm"] as? [String: Any] + #expect(llm != nil) + #expect(llm?.isEmpty == true) + } + } + @Test("config part omits the prompt field when there is no context") func configOmitsPromptWhenNil() throws { let config = try makeTranscriber(apiKey: "test-key") diff --git a/Tests/BlurtEngineTests/DictationSessionTests.swift b/Tests/BlurtEngineTests/DictationSessionTests.swift index 422a0fc..17b7925 100644 --- a/Tests/BlurtEngineTests/DictationSessionTests.swift +++ b/Tests/BlurtEngineTests/DictationSessionTests.swift @@ -20,7 +20,7 @@ struct DictationSessionTests { extension DictationSessionTests { @Test("happy path: press → release → transcribe → inject") func happyPath() async throws { - // The Sync API returns the already-cleaned transcript in one response. + // The dictation API returns the already-cleaned text in one response. let fixture = makeSession(mode: .transcript("Hello world.")) await fixture.session.press() @@ -38,7 +38,7 @@ extension DictationSessionTests { func tooShortClipNoOps() async throws { let fixture = makeSession(mode: .transcript("should not be used")) // Below SyncSTTLimits.minPCMBytes (3200 at 16 kHz) — an accidental brief - // tap the Sync endpoint would reject with a 400. + // tap the endpoint would reject with a 400. await fixture.mic.setPCM(Data(count: 6)) await fixture.session.press() @@ -120,7 +120,7 @@ extension DictationSessionTests { @Test("empty transcript returns to idle without injecting") func emptyTranscriptReturnsToIdle() async throws { - // Sync API yielded only whitespace (e.g. silence) — nothing to inject. + // The API yielded only whitespace (e.g. silence) — nothing to inject. let fixture = makeSession(mode: .transcript(" ")) await fixture.session.press() diff --git a/Tests/BlurtEngineTests/TranscriptionPromptTests.swift b/Tests/BlurtEngineTests/TranscriptionPromptTests.swift index f624289..260465a 100644 --- a/Tests/BlurtEngineTests/TranscriptionPromptTests.swift +++ b/Tests/BlurtEngineTests/TranscriptionPromptTests.swift @@ -108,12 +108,12 @@ struct TranscriptionPromptTests { expected: "Dictated into Notes. \(base)"), ] - @Test("build maps focus context to the Sync prompt", arguments: cases) + @Test("build maps focus context to the transcription prompt", arguments: cases) func build(_ c: Case) { #expect(TranscriptionPrompt.build(context: c.context) == c.expected) } - @Test("built prompt fits within the Sync API 4096-character cap for capped prior text") + @Test("built prompt fits within the API's 4096-character cap for capped prior text") func withinCap() { let longPrior = String(repeating: "word ", count: 200) let prompt = TranscriptionPrompt.build(