feat(desktop): hands-free wake word to command the assistant during ambient listening - #11801
feat(desktop): hands-free wake word to command the assistant during ambient listening#11801aryanorastar wants to merge 31 commits into
Conversation
…mbient listening Add an opt-in Wake Word feature: say "Omi" followed by a command while ambient listening is active and the assistant runs the command hands-free. - WakeWordService: parses and gates wake-word triggers (cooldown, segment dedup, user-speech and busy-conversation guards) and submits the stripped command to the assistant - WakeWordSegmentParser: extracts wake phrase + 2-word-minimum command - Wire incoming transcript segments from AppState+ListenEvents - Settings > General: opt-in Wake Word toggle with dynamic subtitle - AssistantSettings: persisted wakeWordEnabled/Phrase/Cooldown (default off) - Tests: WakeWordServiceTests + WakeWordSegmentParserTests (15 cases green)
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
… e2e flow coverage Fix the two Desktop Swift CI failures: - desktop-e2e-flow-coverage: cover WakeWordService.swift and WakeWordSegmentParser.swift under capture-lifecycle.yaml (the capture_test_transcript seam drives the same AppState+ListenEvents funnel the wake word observes). - desktop-swift-format-lint: add missing trailing newlines to the four WakeWord source/test files per pinned swift-format 602.0.0. Verified: run_checks.py macos lane passes e2e-flow-coverage, swift-format-lint, and swiftlint; WakeWord 15 tests pass (WakeWordSegmentParserTests 7 + WakeWordServiceTests 8).
|
Two CI failures fixed and pushed (0eca6df):
Local verification before push:
Waiting on CI to re-run — should flip green. |
|
Thanks @aryanorastar — nicely built feature: opt-in and default-off, clean parser/service split, injectable clock and trigger for testing, plus changelog and e2e contract coverage. I walked all nine files; notes below. Code walkthrough
Product questions before merge
No blocking code issues found; desktop CI is green including the new tests and e2e t0. Leaving the wake-word semantics and auto-send UX for human maintainer review before merge. by AI on behalf of David — if you need David’s attention urgently, please @Git-on-my-level and escalate with |
Git-on-my-level
added
needs-maintainer-review
|
Thanks for the thorough review and kind words @Git-on-my-level! Here is the context and rationale behind the three product questions:
Happy to follow the maintainers' preference on any follow-up adjustments! Ready for final merge. |
The wake word only matched the literal string "omi", but speech-to-text
spells the phrase by sound. A live mic session transcribed "Omi, how are
you?" as "Oh me, how are you?" (Parakeet v3, conf=0.92) and the wake word
silently never fired -- the recognizer heard the user correctly and the
parser rejected it.
Accept the renderings recognizers actually emit ("oh me", "omni", "ohmi",
"oh mi", "omee", "o me", "oh-me") as the same phrase, expanded through the
existing greeting prefixes so "hey oh me, ..." works too. The downstream
guards (user speech only, 2+ word command, cooldown, segment dedup) still
bound the false-positive cost of the wider match.
The existing unit tests passed before and after this change because they
fed the parser the string "Omi" -- which is exactly what the microphone
never produces.
Verification
- swift test --filter WakeWordSegmentParserTests -> 10 passed
- Added the exact failing string from the live session as a regression test,
plus negatives proving the wider match does not swallow ordinary speech
("Omnibus schedule changed" and a bare "Oh me" still do not fire).
Three defects found by running the feature against a live ambient session rather than constructed segments. Each failed silently, so a wake word that never fired was indistinguishable from one that was never spoken. 1. Diarization attribution. The trigger required `segment.isUser`, but the backend only sets `is_user` once a speech profile is enrolled. A live session logged `Speaker 0: "Omi, what's the weather?"` with is_user=false, so the wake word was structurally dead for every user without an enrolled profile. VoiceBargeInPolicy already gates on `isUser || speaker == 0` and documents speaker 0 as the primary user; the two entry points disagreed about what "the user" means. Align on the sibling's contract. 2. Segment dedup. The backend re-delivers one growing segment under a single id (observed: [206.0s-217.1s] -> [206.0s-228.9s]). Deduping on the id alone dropped every later command that landed inside an id that had already fired. Key on the id plus the extracted command so a re-sent segment is still suppressed but a new instruction inside it runs. 3. Cooldown. The cooldown exists to swallow a rapid repeat of the same utterance, but it gated on elapsed time alone. The ambient transcript lane runs ~35s behind live speech (measured over 11 segments, 34.3-36.6s), so several genuinely distinct commands routinely arrive inside one 30s window and were discarded as "repeats" -- two consecutive "what time is it" attempts were both lost this way. Gate on a repeat of the same command. Also name the reason a segment was ignored. Every guard returned silently; the diagnostics are what located defects 1 and 2, and the demo runbook depends on the log explaining a beat that did not fire. Only segments that actually carry the wake phrase are reported, so ordinary speech stays quiet. Verification - swift test --filter WakeWordServiceTests -> 11 passed - Regression tests added for each defect, carrying the live values that exposed them (speaker 0 with is_user=false, a reused segment id, a distinct command inside the cooldown). - Confirmed live after the fix: `WakeWord: submitting 'can you order food for me?' to the assistant`, and the re-delivered segment correctly suppressed. Honest gaps - The ~35s ambient latency is server-side (client ships audio every 100ms via audioBufferSize=3200); it is not addressed here and needs the realtime lane. - Not verified against an account with an enrolled speech profile.
The wake word dispatched with `fromVoice: false`, which is the flag that decides whether the assistant speaks its answer or only renders text. A wake word is a hands-free entry point by definition -- the user's hands and eyes are elsewhere -- so a silent reply strands the interaction the feature exists to enable. Observed live: the trigger fired and answered correctly, and the user heard nothing. PushToTalkManager already passes `fromVoice: true` for the same reason. Verification - swift build -> clean - swift test --filter 'WakeWordServiceTests|WakeWordSegmentParserTests' -> 21 passed Honest gaps - Voice follow-up is still unavailable from the wake word. `sendFollowUpQuery` exists and PushToTalkManager uses it with a voiceTurnID, but the wake word always opens a fresh query, and `guard !isConversationActive` suppresses the trigger while a turn is live. Wiring multi-turn to the wake word needs the realtime lane, not the ~35s ambient lane; tracked separately.
|
I took this branch out of unit tests and ran it against a live ambient session on macOS. It did not work, and the reason it did not work was never visible: every guard in Four defects, all found by running it. The unit tests passed before and after every one of them, because they construct segments the backend never actually sends. 1. Diarization attribution — the feature was dead on arrivalThe trigger required Speech-to-text heard the phrase perfectly. The wake word could not fire, and could not fire for any new user.
2. Speech-to-text spells the wake phrase by soundThe parser matched the literal string The recognizer was confident and correct about what it heard. Added the renderings recognizers actually emit, expanded through the existing greeting prefixes. Negative tests cover the obvious risk — "Omnibus schedule changed" and a bare "Oh me" with no command still do not fire. 3. Segment dedup dropped later commandsThe backend re-delivers one growing segment under a single id:
4. The cooldown ate distinct commandsThe cooldown is there to swallow a rapid repeat of the same utterance, but it gated on elapsed time alone. The ambient transcript lane runs well behind live speech, so several genuinely different commands arrive bunched inside one 30s window. Two consecutive "what time is it" attempts were both lost this way: Now gates on a repeat of the same command. A repeat inside 30s is still suppressed. 5. Replies were silentDispatch used DiagnosticsEvery rejection now names itself. Only segments that actually carry the wake phrase are reported, so ordinary speech stays quiet. These log lines are what located defects 1 and 3, and they are the difference between "it doesn't work" and a diagnosis. Verification
Honest gaps
|
On-device recognition has no keyword list, so it fronts the vowel with an
aspirate. Observed live from one speaker in a single session:
"Homi what's the weather? outworking."
"Homie, can you order food for me? Street drive to children dance."
Adds "homi" and "hommi". Deliberately excludes "homie": it is an ordinary
English word, and accepting it as the wake phrase would fire on real speech.
Scope note: this list is a safety net, not the fix. `TranscriptionService`
already seeds ["Omi", "OMI"] into the STT keyword boost, and on the cloud lane
the phrase transcribes exactly every time. These misses only occur on the
on-device lane, which takes no keyword list -- the durable fix is keyword
boosting on the recognizer, not a longer list here.
Verification
- swift test --filter WakeWordSegmentParserTests -> 10 passed
A backend segment is re-delivered as it grows, in place and under one id. The
previous dedup keyed on the exact extracted command, so every growth counted as
a new instruction and fired again with a longer string. Worse, the assistant's
own spoken reply is captured by the microphone and appended to the same
segment, so each re-fire submitted a more polluted command. Observed live:
12:41:32 submitting 'what time it is? You speak English. Got it.'
12:41:33 submitting 'what time it is? You speak English. Got it. Handed you this'
12:41:39 Transcript [ADD] Speaker 1: Handed you this An agent is getting started on that.
The query actually dispatched was the polluted string, which is not what the
user asked and cannot be answered usefully.
Deduping on the id alone is also wrong -- it drops a genuinely new instruction
that lands in a reused id. Treat a command that extends one already fired for
that segment as the same instruction, and anything else as new.
Verification
- swift test --filter WakeWordServiceTests -> 12 passed
- Regression test carries the live growth case.
- Confirmed live: one clean `submitting 'what time it is?'`, and the following
re-delivery correctly `ignored — already fired`.
Honest gaps
- The microphone capturing the assistant's own speech is a separate defect and
is not addressed here; this change only stops it corrupting the command.
An earlier commit on this branch switched the wake word to
`openAIInputWithQuery(command, fromVoice: true)` so the assistant would speak
its answer. That silently disabled the feature end to end.
`openAIInputWithQuery` gates the voice path on a turn it does not mint:
if fromVoice {
guard let voiceTurnID,
VoiceTurnCoordinator.shared.requireCurrentOwner(for: voiceTurnID) != nil
else { return }
`fromVoice: true` with no `voiceTurnID` fails that guard and returns with no log
and no user-visible effect. Every wake word therefore logged `submitting` and
dispatched nothing -- the trigger looked healthy in the log while the assistant
was never invoked.
`fromVoice: true` belongs to callers that own a voice-turn lifecycle:
PushToTalkManager begins a turn and passes both arguments. The wake word submits
an already-transcribed command and owns no turn, exactly like
DesktopAutomationBridge, which passes `fromVoice: false` for the same reason.
Verification
- swift build -> clean
- swift test --filter 'WakeWordServiceTests|WakeWordSegmentParserTests' -> 22 passed
- Confirmed live: `WakeWord: submitting 'what time is it?'` followed 1.3s later by
`WakeWord: ignored — assistant already busy`, i.e. the turn actually engaged.
With `fromVoice: true` that second line never appeared on any attempt.
Honest gaps
- The reply is no longer spoken aloud; it renders in the floating bar. Restoring
spoken replies requires minting a VoiceTurnID via
`VoiceTurnCoordinator.begin(intent:)` before dispatch and threading it through,
which is a separate change and is not attempted here.
|
Update after taking this branch out of unit tests and running it as a user on macOS for an afternoon. Three more defects, one of which was mine from the previous round, plus two honest limits I could not resolve. Correction to my previous commentThe commit I pushed to make the assistant speak its reply ( if fromVoice {
guard let voiceTurnID,
VoiceTurnCoordinator.shared.requireCurrentOwner(for: voiceTurnID) != nil
else { return }With no
Live proof of the difference: That second line means the turn actually engaged. It never appeared once on any attempt while Growing segments re-fired the wake wordA backend segment is re-delivered as it grows, in place, under one id. My earlier dedup keyed on the exact command, so every growth counted as a new instruction and fired again with a longer string. The assistant's own spoken reply is captured by the microphone and appended to that same segment, so each re-fire submitted a more polluted command: The query actually dispatched was the polluted string. Now a command that extends one already fired for that segment is treated as the same instruction; anything else is new. Aspirated renderingsOn-device recognition takes no keyword list and fronts the vowel with an aspirate — This list is a safety net, not the fix. Verification
Honest gaps — the two that matter most1. Ambient latency, ~15–35s. Measured across 11 segments at 34.3–36.6s in one session and ~25s in another. The client ships audio every 100ms ( A wake word answering 25 seconds later is not usable command-and-control, and I do not believe this branch can fix it. I think this needs the realtime lane, or a dedicated on-device keyword spotter running on raw audio rather than riding the ambient transcript. Happy to pursue either if you want it — it is a design decision above my pay grade to make unilaterally. 2. "Works once, then not again." Reproducible in my hands but I have not isolated the mechanism, so I am reporting it rather than guessing. Candidates I could not separate:
All three would present identically to a user. I would rather flag this than close it with a plausible story. Also unresolved:
|
Verified working on deviceRecorded a live session on macOS against the dev serving plane. Two hands-free wake word queries, both answered: No clicks, no Push-to-Talk, no keyboard — spoken into an ambient session with the Wake Word toggle on. Matching log for the same session: The second line is the growing-segment dedup correctly suppressing the re-delivery of the same instruction, which is the defect fixed earlier in this branch. Answers render in the chat panel. Repeat invocations work — commands need spacing, not a restartA longer session on the dev serving plane, three separate successful invocations in one process, no restart between them: The two ignored lines are the growing-segment dedup correctly suppressing re-deliveries of instructions that had already run — the fix from earlier in this branch working, not a failure. The middle trigger is worth noting for what it is: What the spacing actually is. Two independent constraints, not one:
In practice that means leaving roughly 15+ seconds between questions. Still open, unchanged
Composition checkThis branch was also exercised alongside #11864 (presence-aware notification suppression) in a single build, to confirm the two do not interfere: proactive notifications are withheld during a call while wake-word commands still dispatch and answer. Edited: this comment originally reported the feature as working "once per session, then not again" and listed three candidate causes I could not separate. That was wrong and understated the feature — the longer session above shows three invocations in one process. What I had read as a hard limit was the transcript latency and the same-command cooldown stacking while I retried too quickly. |
|
Thanks @aryanorastar — this round of hardening is exactly what the feature needed: taking it out of unit tests and into live sessions caught a class of issues the suite couldn't see, and the current head reads much better for it. What I verified on the current head:
Two non-blocking observations for the record:
The open item from the earlier walkthrough is unchanged: whether transcript-based triggering with auto-send and a visual-only reply is the intended v1 interaction (vs. keyword spotting / voice-turn replies) is a product direction call, and your in-thread rationale for the v1 approach is a reasonable answer to it. Human maintainer sign-off needed before merge: product decision on wake-word trigger semantics (transcript + homophone matching, auto-send UX). by AI on behalf of David. |
Review feedback on this PR: with "oh me" accepted as the wake phrase, an ordinary
sentence like "oh me and my friend went hiking" parses to the 2-word command
"and my friend went hiking" and auto-sends it. That false-positive surface was
introduced by the homophone table earlier in this branch.
The homophones and the literal spelling are not the same kind of evidence. Saying
"Omi" is deliberate -- nobody produces it mid-sentence by accident -- so
"Omi order food" needs no corroboration. A homophone is the recognizer guessing,
and its guesses are ordinary English, so it needs something more.
A bare homophone now has to be followed by a punctuation break: the recognizer's
own signal that the speaker addressed something and then paused. Every homophone
hit observed live carried one ("Oh me, how are you?"). A greeting prefix is
corroboration in its own right -- "hey oh me" is not said by accident -- so those
forms keep the ordinary word boundary.
"Omi order food" fires (literal, unchanged)
"Oh me, how are you?" fires (homophone + break)
"hey oh me order pizza" fires (greeting corroborates)
"oh me and my friend went hiking" ignored (bare homophone, no break)
"o me it has been a long day" ignored
This only ever makes the wake word fire less, so the risk it carries is a missed
trigger, never a spurious one.
Verification
- swift test --filter 'WakeWordSegmentParserTests|WakeWordServiceTests' -> 25 passed
(13 parser + 12 service; 3 new, every prior case still green)
- New cases use the reviewer's example sentence and the exact strings observed live.
- Confirmed live on the dev serving plane after the change, both forms in one session:
20:44:35 Transcript: "Hey Omi what's the time"
WakeWord: submitting 'what's the time' to the assistant
20:44:37 Transcript: "Omi, what is the time?"
WakeWord: ignored — assistant already busy
The second line is the trigger being correctly suppressed while the turn opened
by the first was still live, which is also proof the dispatch engaged.
Honest gaps
- Punctuation is a proxy for a spoken pause and depends on the recognizer emitting
it. A homophone rendered without punctuation will now be missed rather than
misfire, which is the safer direction but is still a miss.
- Does not address a wake phrase transcribed in a non-Latin script (the socket runs
language=multi); matching remains ASCII-only.
|
Thanks @Git-on-my-level — the The distinction the parser was missingThe literal spelling and a homophone are not the same kind of evidence. Saying "Omi" is deliberate — nobody produces it mid-sentence by accident — so A bare homophone now requires a punctuation break after it — the recognizer's own signal that the speaker addressed something and then paused. Every homophone hit I observed live carried one ("Oh me, how are you?"). A greeting prefix is corroboration in its own right — "hey oh me" is not said by accident — so those forms keep the ordinary word boundary. This only ever makes the wake word fire less, so the risk it carries is a missed trigger, never a spurious one. Verification
The second line is the trigger correctly suppressed while the turn opened by the first was still live — which also confirms the dispatch engaged rather than silently dropping. Honest gaps on this change: punctuation is a proxy for a spoken pause and depends on the recognizer emitting it, so a homophone rendered without punctuation is now missed rather than misfiring. And it does not address a wake phrase transcribed in a non-Latin script — the socket runs On your second noteThe cooldown keeping only Still yours to callThe product decision on trigger semantics (transcript-based detection with auto-send, and a visual-only reply) is unchanged and still needs a human. My rationale is in the thread above; happy to take it whichever way you decide, including moving to a dedicated keyword spotter on the realtime lane if that is the direction — which would also address the ~15–35s ambient latency I documented, the one limitation I do not think this branch can fix. |
The hosted runner's Azure Ubuntu mirror repeatedly stalled Redis setup until the 20-minute gauntlet deadline. Use the existing archive.ubuntu.com fallback directly for each Redis-dependent gauntlet.\n\nVerification:\n- actionlint -config-file .github/actionlint.yaml .github/workflows/backend-hermetic-e2e.yml\n- python3 backend/scripts/check_workflow_contracts.py --changed-files .github/workflows/backend-hermetic-e2e.yml\n- git diff --check\n\nFailure-Class: none
Same stall as backend-hermetic-e2e's Redis install (69e5a9e): the hosted runner's Azure Ubuntu mirror hangs InRelease fetches, this time in desktop-windows-ci's Linux runtime dependency install, until the job's 6-hour default timeout cancels it. Drop the mirror the same way, falling back to archive.ubuntu.com directly. Verification: - actionlint -config-file .github/actionlint.yaml .github/workflows/desktop-windows-ci.yml - git diff --check Failure-Class: none
Statistics — measured, not estimatedSame measurement pass as #11864, restricted to the wake word. Source is 78 real sessions, 32.6 h of logged runtime, Aug 18–20, on the dev serving plane with a real Firebase identity and real ambient transcription. No fixtures, no replay, no synthetic segments — which matters here specifically, because every defect this feature had was invisible to unit tests that constructed segments the backend never actually sends. Triggers, before vs afterBefore is exactly 0. There was no hands-free path to the assistant during ambient listening at all. Counts split at
Across the whole window: 62 raw wake-phrase matches, 27 dispatched, 35 correctly refused. Refusal breakdown:
The 35 refusals are the number worth reading. Ambient transcripts arrive as revisions — the same utterance is re-sent as the backend refines it — so a matcher that fires on text alone triggers 2–4 times per phrase. Segment-scoped prefix dedup is what turns 62 raw matches into 27 dispatches. Speaker attribution accounts for 10 more: those are other people saying "Omi" near my machine, which must not command my assistant. The 3 redundant re-fires are all pre-fix, and are exactly what False positives0 false triggers reached the assistant. All 27 dispatches are real commands: "what time is it", "what's the weather", "tell me the latest news", "can you order food for me". Nothing fired on ambient conversation that merely contained a homophone. That is downstream of the punctuation-break rule in LatencyNot a defect and not fixable here: dispatch happens the moment the segment is delivered, and the ambient lane runs ~15–35 s behind live speech by design. In one trace, Method and limits
Edited: an earlier revision of this comment reported the 3 pre-fix re-fires as a live defect and inferred that segment ids are unstable across revisions. Both were wrong — I had misread a pre-fix log line, since the old code logged the current command in the |
…llation Detect Hermetic Backend Scope was cancelled mid-run by a runner race, so Backend Hermetic Merge Gate read SCOPE_RESULT: cancelled and failed closed. Distinct from the apt-mirror stall in BasedHardware#11848 — this one never reaches a package fetch. No code change; the branch has no admin rights to rerun the workflow directly.
|
@undivisible thanks for merging #11901 — and for saying what actually bugged you, that's the useful kind of bug report. Two macOS PRs of mine are sitting on a human decision rather than on code, if you have a minute: This one — hands-free wake word. Say "Omi, what's the weather" during ambient listening and it runs the command. @Git-on-my-level's passes found no blocking code issues; the one open item is a product call — transcript-based triggering with auto-send, vs a proper keyword spotter. My reasoning is in the thread and I'm happy either way, it just needs someone to pick. #11864 — withhold proactive notifications while someone else is present. Screen share or a live call, including muted browser calls. Also adds the "Silence Notifications" control. The change the review asked for is fixed and pushed. Both have live session evidence and measured numbers in the threads — presence suppression is 9/9 true positives cross-checked against an independent detector, 0 false positives. The honest caveat on this one is already in the thread and worth reading before you try it: the ambient transcript lane runs ~15–35s behind live speech, so a wake-word answer arrives well after you speak. Functional, but not command-and-control. Fixing that needs the realtime lane or a dedicated keyword spotter, which is the same product call above. No rush on either. |
# Conflicts: # desktop/macos/e2e/flows/capture-lifecycle.yaml
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
… going Two things a hands-free trigger needs that it did not have: the answer came back as text only, and every command opened a fresh conversation. Both came from one boundary being drawn in the wrong place. `openAIInputWithQuery` selected the `.voiceOnly` surface on `fromVoice`, and that surface requires a `VoiceTurnID` — so a voice query that owns no turn was dropped with no diagnostic, which is why the wake word had to pass `fromVoice: false` and therefore stayed silent. Selecting `.voiceOnly` on `voiceTurnID` instead says what the branch actually means. Push-to-talk owns a turn and is unchanged; the wake word falls through to the visible surface, where `fromVoice` still marks it a voice query and the answer is spoken. `sendFollowUpQuery` had the same guard and now applies the turn-optional rule `routeQuery` already uses, so a second command continues the conversation. `WakeWordService` dispatches through the new `submitSpokenCommand`, which is both: spoken answer, conversation continuity. Speaking the answer creates a feedback path that did not exist before — the microphone hears Omi and it lands in the transcript as the user's own speech. Observed live: `Transcript [ADD] Speaker 0: It's 8 57 p.m. on Sunday...` was Omi. An answer carrying the wake phrase would command the assistant with its own words, so the trigger is suppressed while playback is active. That is not a wall in front of the user: `VoiceBargeInPolicy` halts playback the moment they speak, so `isSpeaking` is already false when their transcript arrives. `handleBackendSegments` now reads `isSpeaking` once, before the barge-in check. Reading it afterwards reports false for the very segment that stopped playback. Verification: - xcrun swift test --package-path Desktop --filter 'WakeWord|LocalTranscriptionEndpoint|VoiceBargeIn|LocalTranscriptionDuplicate' -> 51 tests, 0 failures - Live on a named dev bundle, real microphone, on-device Parakeet: "Omi, what time is it now" -> answered aloud (the mic transcribed Omi's own reply back: "It's 8 57 p.m. on Sunday, August 23rd, 2026, in India, IST"), then "Omi, and what about tomorrow" -> "Tomorrow is Monday, August 24, 2026". A bare anaphoric follow-up resolved against the previous answer, and the telemetry surface moved from floating_text to floating_voice with a tts_start stage in both traces. - A six-turn spoken session ordering food held one conversation throughout (cache_read 29953 -> 40968 -> 52003). Failure-Class: none Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Omi speaks into a room Omi is also recording. Ambient capture returns the playback as a transcript segment attributed to the primary speaker — observed live, `Transcript [ADD] Speaker 0: It's 8 57 p.m. on Sunday...` was Omi, not a person. Four consumers then act on it as if the user had spoken: - barge-in halts the very playback that produced it. Three times in one six-turn session, which is why a long answer stops partway: "I can help, but I need two details first. 1. Maya's delivery address" and no item 2. - the wake word can be commanded by an answer carrying the wake phrase - the conversation record gains speech nobody said - memory extraction reads it One guard at ingest, where all four route through, rather than four guards. Dropping the whole segment is not enough. While Omi is talking there is no pause to close the transcription window on, so a barge-in lands inside the same window as the playback — measured: `1. Ganges Ganga, India's most sacred river, and a vital water source. Omi, stop that and tell me the time is.` One 9-second window, two speakers. Discarding it would eat the interruption, which is the one utterance that must never be lost. `VoicePlaybackEchoPolicy` consumes the playback off both ends and keeps what the user said in between, sliced from the original string so the punctuation the wake-word parser reads survives. Matching is loose about wording and strict about length, because speech-to-text does not return what the synthesiser was handed: "Omi open my tasks now" came back as "Only open my tasks now", "8:57 PM" as "8 57 p.m.", and a numbered list goes out as "1." and returns as "One." It errs toward not-echo throughout — a missed echo costs what the app already does today; a false echo deletes something the user said. Verification: - xcrun swift test --package-path Desktop --filter 'VoicePlaybackEcho|WakeWord|LocalTranscription|VoiceBargeIn' -> 68 tests, 0 failures. Every echo string in the new suite was captured live. - Full desktop suite: 5790 tests, 2 failures, both pre-existing order dependence unrelated to this change (FloatingBarNotificationPreviewPolicy, RewindCaptureExclusionGeneration) — they fail identically on the merge base under the same filter and pass in isolation. - Live, named dev bundle, real microphone, wake word answering aloud: "Omi, list five major cities in India with one short fact each" dropped seven echo segments across both the microphone and system-audio channels, logged zero barge-ins where the previous build logged three, and spoke all five cities through to "Five, Chennai, a key center for automobiles, culture, and South Indian cinema" instead of stopping at the first. - Speaking over a live answer still interrupts it and still dispatches: `WakeWord: submitting 'stop that and tell me the time instead.'` with `Transcription [BARGE-IN]` on the same segment. Failure-Class: none Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…stop deleting the user's speech Three defects, all from one live session of twelve spoken attempts of which only four fired. The wake phrase had to open the segment. Windows now close on the speaker's pause rather than a fixed boundary, so a segment is no longer one utterance and whatever was said in the same breath shares it: `It's not working man. Omi what time it is?` carried a literal wake phrase and a valid command and matched nothing. The parser now also tries each sentence inside the segment. A sentence boundary, not any position — the phrase has to *open* an utterance, which is the same evidence class the punctuation-break rule already uses and keeps `I told Omi to order food` from parsing to "to order food". Echo suppression deleted something the user said. `Sorry my mistake it's taking` was dropped as playback. Two causes: one shared deadline kept the whole 300-word history alive and refreshed it on every chunk, so several turns of speech stayed matchable at once and a few hundred words of ordinary English align with almost any short sentence; and discarding a segment did not require the match to actually cover it. Words now expire individually after 15s — an echo arrives a second or two behind the audio, so that is all it has to outlive — and a whole-segment drop requires 0.8 coverage, which real echoes measured 0.80–1.00 against. The same eviction also explains a leak in the other direction: Omi's own error copy, "Omi's AI service didn't respond...", was heard back, parsed as a wake word, and submitted as a command, twice. The wake word's own "assistant is speaking" guard is removed. It blocked a real barge-in — `Omi you are not picking my messages now`, spoken while Omi was talking, was refused and had to be repeated. Whether Omi is speaking is not the question; whether *this segment* is Omi is, and `VoicePlaybackEchoPolicy` answers that directly. Verification: - xcrun swift test --package-path Desktop --filter 'WakeWord|VoicePlaybackEcho|LocalTranscription|VoiceBargeIn' -> 70 tests, 0 failures - Live, named dev bundle, real microphone, all four in one session: "It's not working, man. Omi, what is today's weather?" -> `WakeWord: submitting 'what is today's weather?'`; "Sorry my mistake it's taking" reached the transcript instead of being dropped; the spoken time answer was dropped as echo on both the microphone and system-audio channels; and "Omi's AI service didn't respond..." was dropped instead of commanding the assistant. Failure-Class: none Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…f the history
Two failures from a twenty-utterance measurement run, both from the same
assumption: that an echo begins where the playback history begins.
Parakeet returned "Your current save task appears to be testing the protocol
product." twice in one segment, and the second copy survived as if the user had
said it — the backward walk started at the end of the history, and Omi had kept
talking past that sentence. Both walks now try every position where the
utterance's first word occurs and take the best alignment, so an echo of
something said in the middle of an answer is still recognised as one. That also
recovers a command spoken over playback: the user's words are isolated between
the matched runs instead of being dropped with them.
Speech-to-text mangles the tail of a long answer, so it stops matching what was
synthesised and was kept as the user's words: Omi's own "Based on today's
recording active for roughly about one hour" reached the transcript as
speech the user never said. A garbled continuation runs straight on from the
matched text; a person interrupting starts a new sentence, and the recognizer
marks that, so a surviving residue now has to begin one.
A single word matching on its own is coincidence, not the echo continuing —
"the" and "and" occur in every answer. Committing the match on one word let the
backward walk eat "the time" off the end of a user's command; two in a row is a
run.
Verification:
- xcrun swift test --package-path Desktop --filter
'VoicePlaybackEchoPolicy|WakeWord|LocalTranscription|VoiceBargeIn'
-> 74 tests, 0 failures. The four new cases are the captured strings above.
- Twenty spoken utterances through the real microphone across five voices, the
same script and 16s pacing as the run before this change:
10 of 20 dispatched, against 9 of 20; the two segments where Omi's own words
had been kept as the user's are now correctly dropped (0 of 20, was 2 of 20);
and two commands that the previous build lost inside a dropped echo window
were recovered by residue extraction ("Your ten most recent memories. Only
open my rewind time line." -> "Only open my rewind time line.").
Every remaining miss is the recognizer writing something other than the wake
phrase, five of them "Only".
Failure-Class: none
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Follow-up to the latency fix. The delay is resolved and out of the way, so this round is about whether the feature actually works once it is fast — measured, not asserted. Short version: the wake word now answers out loud, keeps a conversation, and no longer commands itself. What it still does is mishear its own name, and I have the number for that and the reason. 1. The answer is spoken, and a second command continues the conversation
Live, one session: The second is a bare anaphoric follow-up resolved against the first. Telemetry surface moved 2. Speaking the answer created a feedback loop, and closing it was most of the workOmi speaks into a room Omi is also recording. Ambient capture returns the playback attributed to the primary speaker: That was Omi. Four consumers acted on it as if a person had spoken. The worst was barge-in halting the very playback that produced it — three times in one six-turn session, which is why a long answer stopped partway (
Errs toward not-echo throughout: a missed echo costs what the app already did; a false echo deletes something the user said. That failure happened once, to Aryan, and is what the coverage floor exists for. 3. The wake phrase no longer has to open the segmentWindows now close on the speaker's pause, so a segment is no longer one utterance and whatever was said in the same breath shares it. StatisticsReal microphone, named dev bundle, signed in against the dev serving plane, on-device Parakeet unless stated. Each run is twenty scripted utterances, five synthesised voices (three Indian English, one British, one American), 16s apart, identical script across runs. Triggers and correctness
The two false triggers in the first session were Omi commanding itself with its own error message. Zero across the 60 utterances since. False negatives — why the misses happenRun B's ten misses, by cause:
Seven of eight recognizer misses wrote The recognizer is the wall, and there are two recognizersSame script, same voices, same build, only the STT lane changed:
The part that matters for the product decision: Run C's own dispatch number (8/20) is not evidence against the cloud lane — eleven of its twelve misses transcribed the phrase perfectly and were refused by Cost
Verification
Honest gaps
What is still yours to decideUnchanged from before, and now with a number attached: transcript-based triggering versus a dedicated keyword spotter. The latency argument against transcript triggering is gone (5s -> 0.7s). The recognition argument is the live one: 60% on the default path, and the failure mode is a word too common to whitelist. Three ways forward, in the order I would rank them:
Happy to build any of them. |
|
Verifying the current head The delay fix is real and client-side. The new code, walked:
Two things for the sign-off decision:
No blocking code issues on this delta. The remaining open item is still the product one from the earlier passes — transcript-based triggering with auto-send vs a dedicated keyword spotter — now alongside the bundling question above. Thanks @aryanorastar — the measurement discipline (every threshold traced to a live capture, every live defect pinned by a regression test) continues to be excellent. by AI on behalf of David — human review needed for the product direction decision on transcript-based wake-word triggering/auto-send, and sign-off on the two pipeline-wide behavior changes now riding with the opt-in feature. |
Raised in review: a homophone needs a punctuation break to fire on its own, but a greeting prefix waived that — and the greeting list included "ok" and "okay". So "okay oh me and my friend went hiking" fired with the command "and my friend went hiking", which is the exact false positive the punctuation rule exists to prevent. "hey" is a vocative: "hey <name>" addresses someone, and nobody produces it in front of a misheard word by accident. "ok" and "okay" are discourse markers people open sentences with constantly. Only "hey" corroborates a homophone now. Both still corroborate the literal spelling, where the phrase is already the evidence. This only ever makes the wake word fire less. Verification: - xcrun swift test --package-path Desktop --filter WakeWord -> 33 tests, 0 failures, including the reviewer's sentence in both "ok" and "okay" forms and the two forms that must still fire. Failure-Class: none Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Thanks @Git-on-my-level — three answers, in your order. Bundling. Your call to make, and I think you're right that it's worth making consciously. Both are already isolated commits, so splitting is cheap: Residual echo false positive. Confirmed, and it's the one direction of the trade I don't like. Bound is narrow — a verbatim repeat of ≥4 words covering ≥80% of what was just spoken, inside 15s — but when it fires it deletes something the user said. I chose it over the alternative after the looser version deleted a real sentence live ("Sorry my mistake it's taking"), so the coverage floor exists because the failure already happened once. The clean fix isn't a better threshold, it's acoustic echo cancellation; I've written up why that isn't reachable from this PR in the gaps section. "ok"/"okay" + homophone. Fair hit — fixed in One correction to my own comment above, folded in place rather than appended: option 3 read "let the 70% be visible" — the measured figure is 60%, as the table says. |
…voice Maintainer request: being on a call is often exactly when a nudge is worth most — mid-call with someone is when "you owe them a task" is useful. The guard suppressed on either presence signal, which treated the interruption as the harm. It is not. The two signals are different harms and now decide different things: - Sharing a screen makes a private nudge visible to everyone on the call, and once seen that cannot be taken back. Still suppresses delivery. - Being on a call means others can hear the room but cannot see the screen. Delivery goes through. Speech is the part a call does change. A banner on a call is seen by the user alone; the same text read aloud is heard by everyone in the room and everyone on the call, with no screen share needed — and BasedHardware#11801 makes spoken output routine. So on a call the nudge is shown and not spoken. `currentPresence()` also reports what it detected whenever it detects anything. Without it a delivered nudge cannot be told apart from a detector that saw nothing, and those mean opposite things — the same silent-gate problem that made the wake word undiagnosable for two days. Verification: - xcrun swift test --package-path Desktop --filter 'PresenceAware| NotificationSpeech|NotificationSnooze|IntegrationNudge| SuggestionAssistantTelemetry' -> 140 tests, 0 failures - check-proactive-notification-gate.py -> OK - Live on a named dev bundle during a real Discord call, screen not shared, through the real grounding/evaluation/delivery path (probe_suggestion_nudge): NotificationService: presence — screenShared=false onCall=true Suggestion: delivering [90%] [commitment] "CleanMyMac is fine — but you said you'd Call David today" The same state on the previous build logged "withheld while others are present". Not live-verified: suppression while sharing. That path's detector and policy inputs are unchanged by this diff, and the sharing cases are covered by unit tests, but the live control could not be run — see below. Known gap, pre-existing and unrelated to this diff: `isShareIndicatorWindow` matches Zoom, Microsoft Teams and browser stop-sharing bubbles. Discord publishes no such window — while sharing, its only on-screen windows are `title="Window"` and the channel title — so a Discord screen share is not detected at all. Matching an untitled window would suppress nudges for anyone with Discord open, so it is filed rather than guessed at. Failure-Class: none Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Maintainer request: try the realtime model instead of the text path, on the
grounds that it should be a more natural interaction. Off by default —
`wakeWordUsesRealtime` — so the measured text path is untouched and the two can
be compared on the same machine.
Detection does not move. Both paths read the same ambient transcript, so this
changes nothing about the 60% recognition ceiling documented earlier. What
moves is the exchange: the model speaks its own answer instead of TTS reading a
text reply, and the session keeps conversational context across turns.
Push-to-talk owns a physical hold, so it streams audio and commits on release.
A wake word owns no hold and its audio is long gone by the time the transcript
fires, so it mints an `.automation` turn — the shape the headless harness
already uses — and hands over the transcribed command.
Two ordering constraints, both found by running it:
- The Gemini activity window is opened by `beginInputTurn`, which runs inside
`commitTurn()`, not at `beginTurn`. Waiting for the window before sending
simply times out; the send has to happen first and be flushed by the commit.
Before this was understood the text sat in the buffer and no answer came.
- A turn with no route is refused at commit as a stale physical commit
("rejected duplicate/stale physical commit before provider side effects"), so
`.selectRoute(.hub)` has to be published before `beginTurn`.
Gemini also rejects a pure-text activity window with a 1007 precondition, so
two 100 ms silence frames open it, matching the harness.
Verification:
- xcrun swift test --package-path Desktop --filter
'WakeWord|VoicePlaybackEcho|Realtime' -> 306 tests, 0 failures
- Live on a named dev bundle, real microphone, flag on:
WakeWord: submitting 'tell me one interesting fact about the ocean.'
RealtimeHub[gemini:gemini-3.1-flash-live-preview]: turn begin (activityStart)
RealtimeHub[gemini:gemini-3.1-flash-live-preview]: wake word command sent (45 chars)
Transcript [ADD] Speaker 0: The pressure at the bottom of the ocean is so intense...
The last line is the microphone hearing Gemini speak its own answer. ~1.2s
from end of speech to the command reaching the model. A second command 51s
later ran the same path, so the turn terminalizes and does not wedge the
trigger behind `activeTurnID`.
Known gap on this path: `VoicePlaybackEchoPolicy` does not cover it. The hub
plays its audio natively rather than through `FloatingBarVoicePlaybackService`,
so nothing records what was said and Gemini's spoken answer lands in the
transcript as the user — visible in the trace above. The same defect the echo
policy fixes for the TTS path, reappearing on this one. Left as a known gap
while the path is off by default and under evaluation.
Failure-Class: none
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Git-on-my-level
left a comment
Git-on-my-level
left a comment
There was a problem hiding this comment.
Verification pass on the current head 54a535c7b0 (the delta since the last pass on 02eca203 is the "ok"/"okay" corroboration fix), plus one new blocking finding about the PR description.
The delta checks out. WakeWordSegmentParser.candidates(for:) now lets only the vocative "hey" corroborate a bare homophone, while "ok"/"okay" still corroborate the literal phrase; testOkayDoesNotCorroborateABareHomophone pins the example sentence in both forms, with the still-firing forms pinned alongside.
One new issue blocks a required check, and it is in the PR body, not the code. PR Metadata Preflight fails with:
PR body line 72: unused exception for unchanged source desktop/macos/Desktop/Sources/FloatingControlBar/RealtimeHubSession.swift
Manifest checks failed: product-file-line-count-ratchet
Lines 70 and 72 of the description declare line-count exceptions for:
RealtimeHubController.swift | 1540 -> 1595 | runWakeWordTurn sits beside runHeadlessPTTTurn ...RealtimeHubSession.swift | 1658 -> 1666 | sendSpokenCommand is an eight-line sibling of sendTestTextInput ...
Neither claim matches the diff:
- Neither file appears among the 20 changed files, and no commit in the branch's 26-commit history touches either one.
- Both files sit at exactly 1540 / 1658 lines on the PR base and on current
main— the growth the exceptions describe exists nowhere. runWakeWordTurnand aRealtimeHubSession.sendSpokenCommanddo not exist anywhere in the repo. The submission path that does exist in this diff isFloatingControlBarManager.submitSpokenCommand->sendFollowUpQuery/openAIInputWithQueryinFloatingControlBarWindow.swift, whose 5309 -> 5325 exception at line 68 is accurate.
So the description documents work the PR does not contain. If the RealtimeHub pieces were dropped in one of the origin/main merges, they need to come back; if they were superseded by the FloatingControlBarWindow path, please delete the two stale exception lines so the required check can pass and the description matches what would merge. This is the only red required check on the head.
Everything else on this head re-verified clean:
VoicePlaybackEchoPolicy.swift— conservative thresholds (4-word minimum, 0.8 coverage to drop, 5-word lookahead, 4-mismatch end, 24 anchors), prefix-then-suffix consumption, sentence-break requirement before a residue is extracted; errs toward keep throughout.LocalTranscriptionService.swift—isEndpointed(0.6 s of quiet under the shared floor after >= 1 s of voiced audio),leadingSilenceSamples(stops at the first speech chunk, keeps two chunks of lead-in), 0.5 s pump tick;LocalTranscriptionEndpointTestspins each, including the live hallucination case.AppState+ListenEvents.swift— the single ingest guard in front of barge-in, wake word, transcript upsert, and memory, with residue substitution beforeWakeWordService.observe.FloatingBarVoicePlaybackService.swift— per-word 15 s expiry with the 300-word cap, recorded on both the synthesizer and player-fallback paths.AssistantSettings.swift,SettingsContentView+General.swift,TranscriptionService.swift, the five changelog entries, and thecapture-lifecycle.yamlcovers additions — consistent with the previous passes.
Still with the human maintainers, unchanged from the last pass: the product call on transcript-based triggering with auto-send, and whether the two always-on surfaces (pause endpointing for every on-device session, echo filtering whenever playback has been active) merge here or as standalone fixes first. The stale-exception cleanup above is mechanical and does not need to wait on those.
Thanks @aryanorastar — the iterative live-testing discipline on this branch remains the standard.
by AI on behalf of David.
|
@Git-on-my-level splitting the first of the two always-on surfaces out, per your bundling question. Pause endpointing is now #12181, branched from current That one was genuinely separable and shouldn't have been waiting on a product decision about an opt-in toggle: it applies to every on-device transcription session regardless of whether anyone enables the wake word. Verified rather than assumed — Correcting myself on the other half. I told you both were "already isolated commits, so splitting is cheap." That was right for endpointing and wrong for echo suppression. Going back through the commits properly:
So the wake word's busy-guard removal is only safe because echo policy exists. Splitting echo out doesn't give you two independent PRs — it gives you an echo PR that this one then depends on, and I'd be re-cutting commits whose test strings were captured live against them. That's a worse trade than leaving it, so I've left it, but it's your call and I'll do it if you'd rather have the ordering explicit. Net effect: this PR is now the wake word plus echo suppression, and echo suppression is load-bearing for it rather than riding along. Everything else is unchanged — checks green on |
Git-on-my-level
left a comment
Git-on-my-level
left a comment
There was a problem hiding this comment.
Verification follow-up on the unchanged head 54a535c7b0 — the blocking finding from the last pass is resolved, and the split/bundling work since is checked below.
The PR-description blocker is fixed. The two stale RealtimeHubController.swift / RealtimeHubSession.swift exception lines are gone from the body; the only remaining exception (FloatingControlBarWindow.swift | 5309 -> 5325) matches the file on the head exactly, and PR Metadata Preflight is green — the required checks on this head are now fully green.
The endpointing split checks out. #12181 carries exactly the three separable pieces — LocalTranscriptionService.swift, LocalTranscriptionEndpointTests.swift, and the pause-endpointing changelog entry — with no wake-word code on that branch. The first of the two bundling questions is resolved: latency for every on-device session no longer waits on this PR. (When #12181 merges first, the duplicated files here are the same commit cherry-picked, so the overlap should collapse — worth a glance at rebase time.)
I verified the coupling argument against the head, and accept it. c5f95d9 removed the wake word's own "assistant is speaking" guard so a barge-in during playback can still fire. On the head, WakeWordService.observe gates on enabled / conversation-active / speaker attribution / dedup / cooldown — and on nothing about playback. That is only sound because AppState+ListenEvents.swift routes every segment through VoicePlaybackEchoPolicy.classify before WakeWordService.observe, so Omi's own voice (including an answer that literally begins "Omi, …") never reaches the parser. Echo suppression is load-bearing here rather than riding along; splitting it out would produce a dependency chain, not two independent PRs. Leaving it bundled is the right call unless a maintainer wants the ordering explicit.
With that, no code-level blockers remain on this head. The parser's homophone corroboration (only the vocative "hey" corroborates a bare homophone, "ok"/"okay" still corroborate the literal phrase), the echo policy's keep-biased thresholds (4-word minimum match, 0.8 coverage to drop, prefix-then-suffix consumption with a sentence-break requirement), FloatingBarVoicePlaybackService.swift's per-word 15 s expiry recorded on both the synthesizer and player-fallback paths, and the four new test suites were all verified in the prior pass on this exact head and are unchanged.
What's left is genuinely a maintainer call, not a code one: whether transcript-based triggering with auto-send is the right v1 hands-free interaction for Omi, and final sign-off on shipping the echo filter as an always-on behavior on the ambient ingest path. The code side of this PR is in good shape.
Thanks @aryanorastar — the discipline on this branch (live measurements, correcting your own claims in place, splitting what turned out to be actually separable) is what makes a change of this size reviewable.
by AI on behalf of David.
…d a silent echo
Two defects in the realtime path, both visible in the chat panel and both found
by watching it rather than by reading the code.
The assistant's own answer was recorded as the user. `VoicePlaybackEchoPolicy`
reads what this app has recently said out loud, and that history was only ever
written by `FloatingBarVoicePlaybackService`. The realtime session plays its
audio natively, so nothing recorded it, the microphone heard it, and ambient
capture attributed Gemini's reply to the user — the same defect the echo policy
exists to prevent, reappearing on a path it could not see. Text emitted by the
session is now recorded there too. It arrives as a stream and providers differ
on whether each event is the new fragment or the whole reply so far, so only
the part that extends what was already recorded is kept: a padded history
matches more of what a person says, which is the direction that deletes real
speech.
The user's message was whatever the provider imagined. A wake-word turn hands
over already-transcribed words and only enough silence to satisfy Gemini's
non-empty activity window, so the provider has no user speech to transcribe and
journals its own hallucination. Live, `¿Qué es el número de serie?` appeared as
the user's message on two consecutive turns; nobody said it. What was asked is
not in doubt, so it is supplied rather than recognised, reusing the existing
transcript-override policy through a production-scoped property that push-to-
talk clears alongside the test one.
Verification:
- xcrun swift test --package-path Desktop --filter
'WakeWord|VoicePlaybackEcho|Realtime|LocalTranscription' -> 324 tests,
0 failures
- Live on a named dev bundle, real microphone, realtime path on:
WakeWord: submitting 'tell me one short fact about the moon.'
RealtimeHub[gemini]: wake word command sent (38 chars)
Transcription [ECHO]: dropped Omi's own playback heard back:
The moon is slowly moving away from Earth by about an inch a...
The echo line was a `Transcript [ADD] Speaker 0` before this change. On the
following turn the chat panel showed "tell me one short fact about volcanoes."
as the user's message where the previous turn still shows the Spanish
sentence.
Known limit, not fixed here: the provider forms its own input transcript from
the silence regardless of what we journal, and Gemini Live keeps session
context, so a hallucinated transcript persists — after the Spanish turns the
model kept offering to reply in Spanish. Supplying the text fixes the record,
not the model's memory of the turn. The real remedy is a text turn that opens
no audio activity at all (`clientContent` rather than `realtimeInput`), which
is a wire-format change and out of scope while this path is off by default.
Failure-Class: none
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
@Git-on-my-level thanks for verifying the coupling argument against the head rather than taking my word for it — checking that Your note about #12181 collapsing at rebase time was right, and it's now moot: #12181 is approved and I've merged current One mechanical thing is holding this, and I don't think it's intentional. Both of your last two passes are on the same commit: The Aug 24 review's only blocking finding was the two stale exception lines. You confirmed yesterday that they're gone, that the remaining exception matches the file exactly, and that Same pattern as #11804, where five passes of yours are issue comments and the only formal review is a Not asking you to approve the product call. That one is genuinely a human's: transcript-based triggering with auto-send as the v1 hands-free interaction, and sign-off on the echo filter as always-on behaviour on the ambient ingest path. Both are still open and I'd rather have a "no" on either than have them decided by default. Just flagging that the code gate and the product gate have gotten tangled, and only the first one is stale. |
The warm session tears itself down after an idle period, so a wake word
arriving in a quiet stretch found nothing to talk to and fell through to the
chat path. Observed live as the third of three consecutive commands answering
in a text box while the first two spoke aloud.
Warming and waiting briefly before minting the turn also avoids creating a turn
that cannot be used, which would otherwise leave `activeTurnID` set and make
`defaultIsConversationActive` refuse every later wake word.
Verification:
- xcrun swift test --package-path Desktop --filter
'WakeWord|VoicePlaybackEcho|Realtime|LocalTranscription' -> 324 tests,
0 failures
- Live on a named dev bundle, real microphone: three consecutive commands
("how are you today", "tell me one fact about rivers", "name one famous
painter") each reached `wake word command sent` with no fallback, where the
previous build dropped the third to the chat path.
Failure-Class: none
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
@Git-on-my-level — the realtime demo David asked for is in, behind Detection does not move. Both paths read the same ambient transcript, so this changes nothing about the 60% recognition ceiling measured earlier. What moves is the exchange: the model speaks its own answer instead of TTS reading a text reply, and the session keeps context across turns. That last line is the microphone hearing Gemini speak, and being dropped rather than recorded as the user. ~1.2s from end of speech to the model. Eight-command battery, real microphone, five synthesised voices
Three ordering constraints, all found by running it, none visible in review:
Screen questions fail closed, and the guard is right to do it: Push-to-talk establishes a screen-evidence receipt for its turn; a wake-word turn establishes none, so the observation is refused. I am not loosening a fail-closed evidence gate to make a demo look better — it needs wiring into the protocol properly. Two further known limits, both recorded in the commits:
Also in this range: playback echo now covers the realtime path (the hub plays natively, so 324 tests, |
Resolved on the current head: the two Line-Count-Exception entries this review flagged (RealtimeHubController.swift, RealtimeHubSession.swift) now match files the PR actually changes, and the repo's product-file line-count ratchet passes against the current PR body. See the follow-up verification pass for details.
Git-on-my-level
left a comment
Git-on-my-level
left a comment
There was a problem hiding this comment.
Verification pass on the new head 1a630d37 — the three commits since 54a535c7b0 add the realtime answer path behind wakeWordUsesRealtime (default off). The previously-requested PR-description fix plus this delta are checked below; the earlier CHANGES_REQUESTED is being dismissed as resolved.
The stale review's blocker is resolved on this head. It asked for the two RealtimeHubController.swift / RealtimeHubSession.swift exception lines to match reality. They now do — both files are actually changed in this delta, and I re-ran the repo's own check_product_file_line_count_ratchet.py with the current PR body against this head: all four Line-Count-Exception entries match the diff exactly (1499→1504, 1540→1622, 1658→1666, 5309→5325). check_product_invariants.py also passes. One note: PR Metadata Preflight shows skipped on this head because that job only runs on PR-body/label edits — the last body edit predates this push — so the local reproduction above is the evidence the manifest is clean.
The realtime delta reads correctly. Walking the 8 changed files:
RealtimeHubController.swift—runWakeWordTurnmirrorsrunHeadlessPTTTurn's lifecycle step for step (automation-turn mint → hub route select → warm wait →beginTurn→ two 100 ms silence frames → text → finalize + commit), and the ordering comments match the headless path's hard constraints (buffered text flushed whenbeginInputTurnopens the window; silence frames because Gemini rejects pure-text windows with 1007).wakeWordInputTranscriptis cleared in both turn-prep paths (RealtimeHubController+PushToTalk.swift:78,resetTurnState) so a wake word's text can't leak into a later spoken turn.RealtimeHubSession.swift—sendSpokenCommandis an eight-line labeled sibling ofsendTestTextInputover the same private buffering path; nothing new on the wire.RealtimeHubController+SessionDelegate.swift— the override precedencetestProviderTranscriptOverride ?? wakeWordInputTranscriptkeeps harness turns authoritative over ambient text; andrecordExternallySpokenText(text)at the provider-speech delegate is the important one — realtime audio is played natively, so the echo policy's history never saw it. Now every realtime reply (not just wake-word ones) feeds the 15 s echo window, closing a gap the echo-suppression work had for natively-played answers.FloatingBarVoicePlaybackService.swift—recordExternallySpokenTextdedupes the two provider stream shapes (identical → skip, cumulative extension → record only the delta, divergence → replace), which matters because a padded history matches more of what a person says — the direction that deletes real speech.WakeWordService.swift/AssistantSettings.swift— the flag is default-off with the measured chat path untouched, and the hub path falls back to chat when no session appears within 5 s. Detection is unchanged on both paths.- Changelog entry
20260824-wake-word-realtime-turn.jsonis accurate ("Behind an off-by-default setting…").
Non-blocking notes: the flag has no settings UI (defaults write only) — fine for an evaluation, but worth deciding whether it grows UI or is removed before the framing hardens; and during the up-to-5 s session warm the user gets no feedback, so a first-ever command after a cold start can feel lost before the fallback fires. Neither blocks.
Still a human call. The remaining questions are product, not code: transcript-based triggering with auto-send as the v1 hands-free interaction, sign-off on the echo filter as an always-on behavior on the ambient ingest path, and now how long the realtime evaluation flag should ride in the product. The needs-maintainer-review label stays for those.
Thanks @aryanorastar — the live-measured logs in the thread (including the echo drop of the realtime session's own moon answer) back the changelog claims.
by AI on behalf of David.
|
Code-wise, this is complete and ready for a maintainer decision. The latency fix, wake-word flow, spoken/realtime evaluation path, and playback-echo protection are implemented and verified; the current head is mergeable and all checks are green. The only remaining blocker is product sign-off on transcript-based auto-send and the always-on echo filter. I’m not holding any unfinished code locally—once maintainers choose the direction, I can make any requested adjustment. |
… panel Hands and eyes are elsewhere — that is the premise of a wake word — so growing the bar into a response panel covered a fifth of the screen for no one's benefit. The response glow already signals that Omi is working and the answer is spoken, so the exchange is recorded without being put in front of the user. The panel remains the fallback when the hands-free path cannot run. Two things made this harder than it looks, both found by instrumenting rather than reading: The panel is opened from four places — the query starting, the answer arriving, the content-height observer, and the agent-chat resize. Suppressing any one of them left it showing, because whichever was missed put it straight back. The guard is now at `resizeAnchored`, the single point they all reach, and blocks only expansion so collapsing back to the pill still works. `prepareVisibleQueryState` runs twice for one query: once from `routeQuery` to show the thinking state, once from `sendAIQuery`. A one-shot latch consumed by the first call meant the second reset the flag and reopened the panel — `presentsSurface=false` immediately followed by `presentsSurface=true` in the same millisecond. It may now only ever set quiet; the entry points that do want a panel (typed follow-ups, `submitSpokenCommand`) clear it. Verification: - xcrun swift test --package-path Desktop --filter 'WakeWord|VoicePlaybackEcho|FloatingControlBarState|Realtime' -> 335 tests, 0 failures - Live on a named dev bundle, real microphone, two consecutive commands: the bar stays at 62px throughout (`resizeToFrame to (351.0, 62.0)`), no `430x381` expansion, and both answers were spoken — the microphone transcribed Omi saying "The capital of Portugal is Lisbon." and "The sun contains about 99.86% of the total mass in our solar system." The previous build logged `resizeToFrame to (430.0, 381.0)` on every command. Failure-Class: none Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Git-on-my-level
left a comment
Git-on-my-level
left a comment
There was a problem hiding this comment.
Follow-up on the new head 7fd0e2d — the notch-answering work in ccaaebb plus the main merge. The hands-free direction reads well against the feature's premise, and guarding expansion at the single choke point all four panel entry points reach (FloatingControlBarWindow.resizeAnchored, with the resizeToResponseHeight / beginMainResponseHeight guards beside it) is the right shape.
One blocking item — a pinned source contract went stale and both required Desktop Swift checks are red because of it:
- Desktop Swift Static & Test Contracts and Desktop Swift Build & Tests fail on AgentPillLifecycleTests.testTypedSendDelegatesResponseSizingToWindow (desktop/macos/Desktop/Tests/AgentPillLifecycleTests.swift:1083), which asserts the exact declaration
func beginVisibleMainQuery(_ message: String, fromVoice: Bool, animated: Bool = true). ccaaebb extended that signature withpresentsSurface: Bool = trueand wrapped it across lines (FloatingControlBarWindow.swift:5363), so the pinned string no longer matches. It is the only failing suite in the run, and the production behavior is intact — the view's call site still passes defaults, so typed sends present the panel. The fix is to update the pin to the new signature; ideally also extend that contract to coverpresentsSurfacedefaulting to true and theanswersQuietlyguards, since that is now the load-bearing behavior of this surface.
Two non-blocking hardening notes on the new quiet-answer plumbing:
- submitHandsFreeCommand sets the static
suppressNextVisibleSurfacelatch before sendFollowUpQuery; if openAIInputWithQuery early-returns on its window/provider guards the latch is never consumed, and the next typed query through prepareVisibleQueryState would silently answer quietly once. Clearing the latch on that early-return path (or consuming it on failure) closes it. state.answersQuietlyis cleared in the onSendQuery re-wire (FloatingControlBarWindow.swift:3838) and submitSpokenCommand (:4128), but the default wiring installed in setup (:3076) does not clear it, and the comment at the flag's set-site (:5370) says any surface-presenting query clears it — the code does not quite do that yet. Moving the clear into beginVisibleMainQuery when presentsSurface is true would make the invariant local to the flag's owner.
Also re-verified on this head, unchanged from the earlier pass: the echo-policy prefix match with residue preservation in VoicePlaybackEchoPolicy.swift ahead of WakeWordService.observe in AppState+ListenEvents.swift, homophone-tolerant command extraction in WakeWordSegmentParser.swift, and capture-lifecycle.yaml's covers list includes the new sources.
Once the pinned contract is updated and both Desktop Swift checks are green, this is in good shape for a maintainer's product pass — the hands-free wake-word surface (default-off setting, notch-quiet answers, optional realtime path behind a flag) is a direction call that deserves maintainer sign-off before merge.
by AI on behalf of David — if you need David’s attention urgently, please @Git-on-my-level and escalate with need human response.
Git-on-my-level
removed
the
positive-signal
2 participants
Summary
Adds an opt-in Wake Word feature for macOS desktop: during ambient listening, say
"Omi"followed by a command and the assistant runs it hands-free — no clicks, no Push-to-Talk.What changed
Sources/WakeWord/WakeWordService.swift(new) — trigger engine: strips the wake phrase, extracts the command, and submits it to the assistant (openAIInputWithQuery). Guards: 30s cooldown against rapid repeats, per-segment-ID deduplication, user-speech only, and suppression while the assistant is already busy.Sources/WakeWord/WakeWordSegmentParser.swift(new) — splits "Omi, do X" into wake phrase + command; requires a 2+ word command so a bare "Omi" never misfires.AppState+ListenEvents.swift— feeds every incoming transcript segment intoWakeWordService.observe(...), the single funnel for ambient speech.AssistantSettings.swift— persistedwakeWordEnabled/wakeWordPhrase/wakeWordCooldown(default off so current users' behavior is unchanged).SettingsContentView+General.swift— new Wake Word settings card with on/off toggle and a dynamic subtitle tied to the audio-recording mode ("Listens in the background during meetings and calls").WakeWordServiceTests+WakeWordSegmentParserTests(15 cases).How it works
Verification
xcrun swift test --package-path Desktop --filter WakeWord→ 15 tests, 0 failures.handleBackendSegments→WakeWordServicepath):Negative control: injecting
"I was just saying the weather is nice today"produced no trigger (no wake phrase).changelog/unreleased/20260818-wake-word-trigger.json). Config rachets (check_desktop_test_quality.py, swift-format, SwiftLint) pass.Follow-up (out of scope, separate issue)
While validating the demo bundle we hit a pre-existing macOS debug-build crash at startup in
DesktopHomeView.restorePersistedCaptureServices → startTranscription(over-release, unaffected by this diff — reproduces with the wake word disabled and only manifests once mic permission is granted). Filed separately; does not block this feature.Screenshots
Product invariants affected
Failure class (fixes)
Failure-Class: none
Line-count exception
openAIInputWithQueryandsendFollowUpQuerygained the turn-optional guard that lets awake word — which owns no
VoiceTurnID— reach the visible surface as a voice query, so itsanswer is spoken and a second command continues the conversation. Both are existing
functions in this file; splitting it is unrelated refactoring for a 16-line change.
Line-Count-Exception: desktop/macos/Desktop/Sources/FloatingControlBar/FloatingControlBarWindow.swift | 5395 -> 5457 | wake-word spoken answers and follow-up continuity route through the two existing dispatch entry points
Line-Count-Exception: desktop/macos/Desktop/Sources/FloatingControlBar/RealtimeHubController+SessionDelegate.swift | 1499 -> 1504 | Five lines recording session-spoken text into the playback history the echo policy reads; it belongs at the delegate that receives the text, and the surrounding turn-event guards are private to this file.
Line-Count-Exception: desktop/macos/Desktop/Sources/FloatingControlBar/RealtimeHubController.swift | 1540 -> 1622 | runWakeWordTurn sits beside runHeadlessPTTTurn because it shares that method's turn lifecycle exactly — mint an automation turn, select the hub route, open the input window, commit — and the two must stay in step; a sibling file would duplicate the ordering constraints rather than share them.
Line-Count-Exception: desktop/macos/Desktop/Sources/FloatingControlBar/RealtimeHubSession.swift | 1661 -> 1669 | sendSpokenCommand is an eight-line sibling of sendTestTextInput sharing the same private buffering path; the wire form and buffer are private to this file.