fix(desktop): end an on-device transcription window when the speaker pauses

[aryanorastar]

Summary

On Apple Silicon, STTSessionState.resolveMode selects the on-device Parakeet path by default for microphone and system audio — cloud STT is the fallback, not the default. So ambient transcription runs through LocalTranscriptionService, which only closed a transcription window on a fixed 10-second boundary:

/// Window length transcribed at a time. Not real-time — gives a ~10 s "lag" like the user wants.
private let windowSeconds = 10.0

An utterance therefore waited for wherever it happened to land inside that window. Uniform arrival means ~5 s expected and ~10 s worst case, for speech that finished seconds earlier.

This closes a window when the speaker pauses instead. The 10 s cap is unchanged and still bounds continuous speech.

What changed

Three changes, all inside the on-device path:

1. A window closes on a pause. Once the buffer holds ≥1 s of voiced audio followed by 0.6 s of quiet under the shared noise floor, transcribe it. The minimum is measured in voiced samples, not buffer length — a mostly-quiet window with one blip in it is exactly what Parakeet answers with a hallucinated word (a 1.1 s window at rms 0.0067 decoded to "Yeah." live).
2. Leading silence is dropped, advancing the emitted-seconds cursor over it so absolute timestamps stay exact. Without this, quiet counts against the 10 s cap and the window fills partway through the next sentence.
3. Pump tick 1 s → 0.5 s, since it is now the floor on how soon a finished utterance can be transcribed rather than a poll for a full window.

Measured

Real microphone, named dev bundle, on-device Parakeet, MacBook Air. Latency is speech-end wall clock to the transcription log line.

Before — fixed 10 s window, three identical utterances:

utterance	latency	window
Omi, what time is it now	1.08 s	[90.0s–100.0s]
Omi, what time is it now	7.65 s	[120.0s–130.0s]
Omi, what time is it now	6.18 s	[140.0s–150.0s]

Identical speech, 1.08 s vs 7.65 s — the only variable is where the utterance fell inside the window. Every window is exactly 10.0 s wide and boundary-aligned, which is the tell.

After — 14 utterances across two builds:

	before	after
mean	4.97 s	0.74 s
min	1.08 s	0.53 s
max	7.65 s	1.04 s
spread	6.57 s	0.51 s
theoretical worst case	10.0 s	1.2 s
window width emitted	10.0 s fixed	2.1–3.4 s, the utterance itself

The spread collapsing from 6.6 s to 0.5 s is the part that matters: transcription arrives at the same moment every time instead of anywhere in a ten-second range.

Tests

LocalTranscriptionEndpointTests — 13 new cases pinning each half of the rule in both directions: speech-then-pause closes, speech-still-running does not, a pause shorter than the tail does not, pure silence never drains, a blip in a mostly-quiet window does not, voiced audio accumulates across short gaps, room tone under the noise floor reads as a pause, leading silence is trimmed with lead-in kept, a buffer opening with speech is not trimmed, and a pause between two utterances is never trimmed.

Verified on this branch, which contains no other change:

• xcrun swift build -c debug --package-path Desktop — clean
• LocalTranscriptionEndpointTests + LocalTranscriptionDuplicatePolicyTests — 18 tests, 0 failures

Why this is its own PR

Split out of #11801 (wake word) at the reviewer's request. Pause endpointing applies to every on-device transcription session, whether or not anyone enables the wake word, so it should not wait on a product decision about an opt-in toggle. It stands alone: this branch is main plus one commit, and the endpointing tests pass on it with no wake-word code present.

Honest gaps

• Parakeet still hallucinates a single word on quiet windows — twice in ~3 minutes of near-silence it emitted "Yeah." at rms 0.027–0.030, conf 0.60–0.61. Shorter windows surface these more often than 10 s windows did, so this change makes an existing behaviour more visible. I did not add a confidence floor: real speech in the same session ran conf 0.78–0.98 and the quiet-window hallucinations ran 0.35–0.79, so the distributions overlap and any threshold I picked would be fitted to one room.
• One machine, one room, one voice. 21 timed utterances is a clean before/after on a large effect, not a population.
• BLE/pendant capture is unaffected — resolveMode sends .bleDevice to the cloud path, untouched here.

Failure-Class: none

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[aryanorastar]

The four red checks here are not this diff. This PR is main plus one commit touching three desktop files and no backend at all.

gcp-sa-key-ratchet (landed in #11125) declares two trigger paths that do not exist, and run_checks.py:187 fails the whole manifest resolution when a non-glob trigger is missing — before any check runs:

FAIL: could not resolve manifest checks: gcp-sa-key-ratchet: explicit trigger path does not exist: backend/agent-proxy/main.py; gcp-sa-key-ratchet: explicit trigger path does not exist: backend/agent-proxy/Dockerfile

Reproduces on a pristine checkout of main with no branch involved. Filed as #12182, along with a second defect underneath it — the ratchet has never actually passed its own baseline, which the resolution failure was hiding.

I have the one-line fix but am deliberately not sending it: on its own it turns a broken-manifest error into a red credential check on every PR, which is worse. #12182 has the detail and the two honest resolutions, both of which need someone who owns that area.

This PR's own content is verified independently of CI: swift build clean, LocalTranscriptionEndpointTests + LocalTranscriptionDuplicatePolicyTests 18/18, and scripts/pr-preflight 22/22 locally against 009132dbb3 before #11125 landed.

[Git-on-my-level]

Verified the premise first: STTSessionState.resolveMode does select .local for mic/system audio on Apple Silicon, so this touches the default transcription path — worth the extra care, and it holds up.

desktop/macos/Desktop/Sources/LocalTranscriptionService.swift

• isEndpointed(_:) is the right shape: a 0.6 s quiet tail under the shared 0.004 floor plus >=1 s of voiced audio, counted in 100 ms chunks by voicedSamples. Measuring the minimum in voiced samples rather than buffer length is what actually blocks the mostly-quiet-window hallucination case documented in the PR body.
• leadingSilenceSamples trims only ahead of the first speech (the scan stops at the first chunk over the floor, keeping a 200 ms lead-in), and drain advances emittedSeconds over the trimmed amount so absolute timestamps stay exact. The pause-between-utterances case staying untrimmed is the important subtlety, and it's pinned by a test.
• drain's (force || endpointed) ? available : windowSamples preserves the 10 s cap semantics for continuous speech while letting a pause consume the whole buffer — no arbitrary mid-utterance split.
• Hoisting the inline 0.004 in drain's RMS guard into the shared speechFloor constant is a good cleanup.

desktop/macos/Desktop/Tests/LocalTranscriptionEndpointTests.swift

• All 13 expectations are consistent with the implementation semantics — I re-derived the endpointing/trim arithmetic independently against each case (speech-then-pause closes; running speech, a pause shorter than the tail, pure silence, a sub-second blip, and a blip inside a mostly-quiet window don't close; voiced audio accumulates across short gaps; room tone under the floor reads as a pause; and the four leading-silence cases including the keep-two-chunks lead-in arithmetic). The blip-in-a-mostly-quiet-window test is the one that pins the real regression risk here.

desktop/macos/changelog/unreleased/20260823-local-transcription-pause-endpointing.json

• Accurate and user-true ("about a second later" matches the measured 0.53-1.04 s) and matches the repo's desktop changelog schema.

The four red checks are not this diff. Confirmed independently: backend/agent-proxy/main.py and backend/agent-proxy/Dockerfile don't exist on main, gcp-sa-key-ratchet landed in #11125 at 09:04Z — three minutes before this PR opened — and Hygiene / Detect Desktop Swift Changes / PR Metadata Preflight / (cascading) Desktop Swift Build & Tests all die at the same could not resolve manifest checks line before any check body runs. That's the #12182 report, and declining to send the one-line fix from this branch was the right call.

Splitting this out of #11801 was also correct — pause endpointing applies to every on-device session, with or without the wake word.

What's left for a maintainer is judgment rather than correctness: 0.6 s tail / 1.0 s min-utterance are single-room, single-voice tuned constants on the default transcription path, and the disclosed trade-off — shorter windows surface Parakeet's quiet-window single-word hallucinations more often — is a product-behavior call. Merge will stay blocked until the #12182 manifest fix lands so CI can actually run these tests.

---

by AI on behalf of David — if you need David’s attention urgently, please @Git-on-my-level and escalate with need human response.

[kodjima33]

Desktop bug-fix fast lane (owner-override) — well-verified pause-endpointing fix for on-device transcription. Holding merge: PR Metadata Preflight fails on a stale frozen merge-ref referencing a since-deleted backend/agent-proxy path (gcp-sa-key-ratchet check) — unrelated to this PR's Swift-only diff, matches the known frozen-merge-ref CI pattern. Reruns won't clear it; needs a branch update to pick up current main.

[kodjima33]

Desktop bug fix: pause-based endpointing cuts on-device transcription latency from ~5s avg/10s worst-case to <1s, with 13 pinned tests. CLEAN, all CI green (PR Metadata Preflight now passes — prior frozen-ref block cleared). Confirmed unfixed on main. Owner-override merge.