Feat/macos chat voice controls - #12290
Conversation
…k create Add a discard control while chat PTT is listening, a stop-speaking affordance for spoken AI answers, and stable Idempotency-Key headers when syncing local action items so retries do not mint duplicates. Co-authored-by: Cursor <cursoragent@cursor.com>
Unblock desktop-e2e-flow-coverage pre-push for the new stop-speaking helper. Co-authored-by: Cursor <cursoragent@cursor.com>
Satisfy desktop-swiftlint force_unwrapping on the new create-item capture stub. Co-authored-by: Cursor <cursoragent@cursor.com>
desktop-test-quality only accepts the annotation on the read line or the immediately preceding line. Co-authored-by: Cursor <cursoragent@cursor.com>
Git-on-my-level
left a comment
There was a problem hiding this comment.
Thanks @Yny4ii — this is carefully built work: the cancel affordance shows real attention to hit-target stability, the idempotency key is threaded through the right seams, and the tests (including the source-layout tripwire) are unusually thoughtful. One verified gap needs fixing before merge; the rest is minor.
Required: the Stop-speaking control never appears for native realtime voice answers.
VoiceResponsePlaybackMonitor is only refreshed from the TTS lane (FloatingBarVoicePlaybackService.swift:526/559/763/771) and from native stop (RealtimeHubController+VoiceOutput.swift:96 inside stopNativePlayback). Nothing refreshes it when native PCM playback starts — RealtimeHubController+SessionDelegate.swift:711 acquires the .nativeRealtime lease and enqueues audio with no refresh() — nor when playback drains naturally (onPlaybackIdle → releaseOutput, RealtimeHubController+VoiceOutput.swift:16-31). Every composer gates the new button on the monitor's cached @Published isActive (ChatInputView.swift, QueryHeroBar.swift, AskAIInputView.swift, and DashboardPage.swift through HomeAskBarControls.resolve(isSpeaking:)), so the button will not render while a native realtime answer is speaking, and the cached value can go stale afterwards. Escape still works because FloatingControlBarWindow.handleEscapeKey() calls VoiceResponsePlayback.interrupt() directly — which is also why this escapes casual testing.
Suggested fix: call VoiceResponsePlaybackMonitor.shared.refresh() after a successful acquireVoiceOutput(.nativeRealtime, …) and in onPlaybackIdle after releaseOutput. That also matches the test gap: VoiceResponsePlaybackTests covers only the idle path (testInterruptWhenIdleReturnsFalse) — please add a test that drives isActive true (e.g. coordinator acquires a .nativeRealtime lease → monitor refresh → isActive == true), which would have caught the missing start-side refresh.
Everything else is in good shape:
VoiceResponsePlayback.swift— good consolidation of stop into one entry point across TTS and native lanes; lease-awareinterrupt()(native stop before TTS interrupt) is the right ordering.PushToTalkMicButton.swift/PushToTalkButtonTrigger.swift— cancel trailing the mic so the send hit-target doesn't jump is exactly the right call, andshowsCancelRecordingControl(phase:)correctly limits it to.recording/.lockedRecording/.pendingLockDecision.FloatingControlBarView.swift— both overlays (chat surface and pill) share the same cancel action, keep the container accessibility labels, and the chat overlay correctly dropsallowsHitTesting(false)so the cancel button is clickable.PushToTalkManager.cancelPushToTalkForAutomation()and theptt_cancelaction inDesktopAutomationBridge.swift— mirrors theptt_stoppattern faithfully and resetsautomationCaptureBypass.APIClient.swift/APIClient+TaskCatalog.swift/TasksStore.swift— theIdempotency-Keychain is clean: always-sent key (web parity), stabledesktop-action-item:<rowid>for durable local rows, andActionItemCreateIdempotencyTestsverifies all three properties via URLProtocol capture.ChatProvider.swift(~line 3261) — stopping an agent turn now also silences spoken audio after the owner check; the comment explains the intent.FloatingBarVoicePlaybackService.swift— thesetFloatingPillResponseGlowrefactor (if/else instead of early return) keeps both lanes publishing; note this file carries four of the five refresh sites, so centralizing the refresh (or deriving the monitor from the coordinator's output snapshot) might be a worthwhile follow-up.
Non-blocking, for a maintainer: the always-on Idempotency-Key header lands ahead of its backend dedupe counterpart (as the description notes), and the cancel-vs-send semantics on the floating bar are a product call worth a quick human pass.
Requesting changes for the native-lane refresh wiring plus the one test; voice-UX semantics and header sequencing also deserve human maintainer sign-off before merge.
🤖 Automated maintainer review on behalf of @Git-on-my-level. AI-generated feedback - human review required before merge.
by AI on behalf of David — if you need David’s attention urgently, please @Git-on-my-level and escalate with need human response.
What changed and why
Add cancel-while-recording for chat PTT (mic still sends; cancel trails so the hit target does not jump), a stop-speaking control for spoken AI answers (composers, Escape, agent Stop), and a stable
Idempotency-Keyon action-item create keyed to the local SQLite row so sync retries do not duplicate tasks. Backend dedupe for that key lands with the web duplicate-task fix branch and is not in this PR.Product invariants affected
How it was verified
PushToTalkButtonTriggerTests,VoiceResponsePlaybackTests,ActionItemCreateIdempotencyTests(and Home ask-bar stop control coverage) — 14/14 PASS./Applications/omi-voice-controls.app: bridgeptt_start→ptt_cancel→cancelled=true,terminal_reason=cancelled.Tests
PushToTalkButtonTriggerTests(including cancel-trails-mic layout tripwire).VoiceResponsePlaybackTests.ActionItemCreateIdempotencyTests.VoiceResponsePlayback.swiftcovered infloating-bar-functional.yamlfor e2e flow coverage.Failure class (fixes)
Failure-Class: none
Scoped cleanups (optional)
coversforVoiceResponsePlayback, SwiftLint force-unwrap fix in the idempotency URL stub, source-inspection escape placement for the layout tripwire.Line-Count-Exception: desktop/macos/Desktop/Sources/DesktopAutomationBridge.swift | 4679 -> 4687 | Register ptt_cancel automation action next to existing PTT bridge actions
Line-Count-Exception: desktop/macos/Desktop/Sources/FloatingControlBar/FloatingControlBarView.swift | 3012 -> 3038 | Add interactive cancel on floating/notch PTT recording overlays
Line-Count-Exception: desktop/macos/Desktop/Sources/FloatingControlBar/PushToTalkManager.swift | 2807 -> 2821 | Expose cancelPushToTalkForAutomation for the bridge cancel path
Line-Count-Exception: desktop/macos/Desktop/Sources/MainWindow/Pages/DashboardPage.swift | 4297 -> 4303 | Wire Home ask bar stop-speaking through existing trailing controls
Line-Count-Exception: desktop/macos/Desktop/Sources/Providers/ChatProvider.swift | 6858 -> 6860 | Interrupt spoken playback when the user stops an in-flight agent turn
Line-Count-Exception: desktop/macos/Desktop/Sources/Stores/TasksStore.swift | 3676 -> 3684 | Pass stable Idempotency-Key when syncing local-first task creates