Skip to content

Feat/macos chat voice controls - #12290

Open
Yny4ii wants to merge 4 commits into
BasedHardware:mainfrom
Yny4ii:feat/macos-chat-voice-controls
Open

Feat/macos chat voice controls#12290
Yny4ii wants to merge 4 commits into
BasedHardware:mainfrom
Yny4ii:feat/macos-chat-voice-controls

Conversation

@Yny4ii

@Yny4ii Yny4ii commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

What changed and why

Add cancel-while-recording for chat PTT (mic still sends; cancel trails so the hit target does not jump), a stop-speaking control for spoken AI answers (composers, Escape, agent Stop), and a stable Idempotency-Key on action-item create keyed to the local SQLite row so sync retries do not duplicate tasks. Backend dedupe for that key lands with the web duplicate-task fix branch and is not in this PR.

Product invariants affected

  • INV-AUTH-1
  • INV-CHAT-1
  • INV-VOICE-1
  • INV-TASK-1

How it was verified

  • Focused Swift tests: PushToTalkButtonTriggerTests, VoiceResponsePlaybackTests, ActionItemCreateIdempotencyTests (and Home ask-bar stop control coverage) — 14/14 PASS.
  • Named bundle /Applications/omi-voice-controls.app: bridge ptt_startptt_cancelcancelled=true, terminal_reason=cancelled.
  • Pre-push / local PR preflight passed with this body (line-count exceptions + invariant citations).
  • Not re-verified in this session: mid-TTS Stop in the live UI (spoken playback was already silent on this machine and on the site download build — treated as environmental, not introduced here). Manual cancel → re-record and live task create with Idempotency-Key still worth a quick pass on reviewer’s machine.

Tests

  • Core cancel path + post-cancel restart: PushToTalkButtonTriggerTests (including cancel-trails-mic layout tripwire).
  • Stop-speaking helper: VoiceResponsePlaybackTests.
  • Idempotency-Key always sent / caller key reused / stable local-row key: ActionItemCreateIdempotencyTests.
  • VoiceResponsePlayback.swift covered in floating-bar-functional.yaml for e2e flow coverage.

Failure class (fixes)

Failure-Class: none

Scoped cleanups (optional)

  • Pre-push unblock commits: e2e covers for VoiceResponsePlayback, SwiftLint force-unwrap fix in the idempotency URL stub, source-inspection escape placement for the layout tripwire.

Line-Count-Exception: desktop/macos/Desktop/Sources/DesktopAutomationBridge.swift | 4679 -> 4687 | Register ptt_cancel automation action next to existing PTT bridge actions
Line-Count-Exception: desktop/macos/Desktop/Sources/FloatingControlBar/FloatingControlBarView.swift | 3012 -> 3038 | Add interactive cancel on floating/notch PTT recording overlays
Line-Count-Exception: desktop/macos/Desktop/Sources/FloatingControlBar/PushToTalkManager.swift | 2807 -> 2821 | Expose cancelPushToTalkForAutomation for the bridge cancel path
Line-Count-Exception: desktop/macos/Desktop/Sources/MainWindow/Pages/DashboardPage.swift | 4297 -> 4303 | Wire Home ask bar stop-speaking through existing trailing controls
Line-Count-Exception: desktop/macos/Desktop/Sources/Providers/ChatProvider.swift | 6858 -> 6860 | Interrupt spoken playback when the user stops an in-flight agent turn
Line-Count-Exception: desktop/macos/Desktop/Sources/Stores/TasksStore.swift | 3676 -> 3684 | Pass stable Idempotency-Key when syncing local-first task creates

Review in cubic

Yny4ii and others added 4 commits August 27, 2026 12:23
…k create

Add a discard control while chat PTT is listening, a stop-speaking affordance
for spoken AI answers, and stable Idempotency-Key headers when syncing local
action items so retries do not mint duplicates.

Co-authored-by: Cursor <cursoragent@cursor.com>
Unblock desktop-e2e-flow-coverage pre-push for the new stop-speaking helper.

Co-authored-by: Cursor <cursoragent@cursor.com>
Satisfy desktop-swiftlint force_unwrapping on the new create-item capture stub.

Co-authored-by: Cursor <cursoragent@cursor.com>
desktop-test-quality only accepts the annotation on the read line or the
immediately preceding line.

Co-authored-by: Cursor <cursoragent@cursor.com>

@Git-on-my-level Git-on-my-level left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Yny4ii — this is carefully built work: the cancel affordance shows real attention to hit-target stability, the idempotency key is threaded through the right seams, and the tests (including the source-layout tripwire) are unusually thoughtful. One verified gap needs fixing before merge; the rest is minor.

Required: the Stop-speaking control never appears for native realtime voice answers.

VoiceResponsePlaybackMonitor is only refreshed from the TTS lane (FloatingBarVoicePlaybackService.swift:526/559/763/771) and from native stop (RealtimeHubController+VoiceOutput.swift:96 inside stopNativePlayback). Nothing refreshes it when native PCM playback startsRealtimeHubController+SessionDelegate.swift:711 acquires the .nativeRealtime lease and enqueues audio with no refresh() — nor when playback drains naturally (onPlaybackIdlereleaseOutput, RealtimeHubController+VoiceOutput.swift:16-31). Every composer gates the new button on the monitor's cached @Published isActive (ChatInputView.swift, QueryHeroBar.swift, AskAIInputView.swift, and DashboardPage.swift through HomeAskBarControls.resolve(isSpeaking:)), so the button will not render while a native realtime answer is speaking, and the cached value can go stale afterwards. Escape still works because FloatingControlBarWindow.handleEscapeKey() calls VoiceResponsePlayback.interrupt() directly — which is also why this escapes casual testing.

Suggested fix: call VoiceResponsePlaybackMonitor.shared.refresh() after a successful acquireVoiceOutput(.nativeRealtime, …) and in onPlaybackIdle after releaseOutput. That also matches the test gap: VoiceResponsePlaybackTests covers only the idle path (testInterruptWhenIdleReturnsFalse) — please add a test that drives isActive true (e.g. coordinator acquires a .nativeRealtime lease → monitor refresh → isActive == true), which would have caught the missing start-side refresh.

Everything else is in good shape:

  • VoiceResponsePlayback.swift — good consolidation of stop into one entry point across TTS and native lanes; lease-aware interrupt() (native stop before TTS interrupt) is the right ordering.
  • PushToTalkMicButton.swift / PushToTalkButtonTrigger.swift — cancel trailing the mic so the send hit-target doesn't jump is exactly the right call, and showsCancelRecordingControl(phase:) correctly limits it to .recording / .lockedRecording / .pendingLockDecision.
  • FloatingControlBarView.swift — both overlays (chat surface and pill) share the same cancel action, keep the container accessibility labels, and the chat overlay correctly drops allowsHitTesting(false) so the cancel button is clickable.
  • PushToTalkManager.cancelPushToTalkForAutomation() and the ptt_cancel action in DesktopAutomationBridge.swift — mirrors the ptt_stop pattern faithfully and resets automationCaptureBypass.
  • APIClient.swift / APIClient+TaskCatalog.swift / TasksStore.swift — the Idempotency-Key chain is clean: always-sent key (web parity), stable desktop-action-item:<rowid> for durable local rows, and ActionItemCreateIdempotencyTests verifies all three properties via URLProtocol capture.
  • ChatProvider.swift (~line 3261) — stopping an agent turn now also silences spoken audio after the owner check; the comment explains the intent.
  • FloatingBarVoicePlaybackService.swift — the setFloatingPillResponseGlow refactor (if/else instead of early return) keeps both lanes publishing; note this file carries four of the five refresh sites, so centralizing the refresh (or deriving the monitor from the coordinator's output snapshot) might be a worthwhile follow-up.

Non-blocking, for a maintainer: the always-on Idempotency-Key header lands ahead of its backend dedupe counterpart (as the description notes), and the cancel-vs-send semantics on the floating bar are a product call worth a quick human pass.

Requesting changes for the native-lane refresh wiring plus the one test; voice-UX semantics and header sequencing also deserve human maintainer sign-off before merge.


🤖 Automated maintainer review on behalf of @Git-on-my-level. AI-generated feedback - human review required before merge.


by AI on behalf of David — if you need David’s attention urgently, please @Git-on-my-level and escalate with need human response.

@Git-on-my-level Git-on-my-level added macOS needs-maintainer-review Needs a human maintainer to sign off before merge needs-tests PR introduces logic that should be covered by tests labels Aug 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

macOS needs-maintainer-review Needs a human maintainer to sign off before merge needs-tests PR introduces logic that should be covered by tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants