fix: preserve adaptive interruption across tool calls - #2290
fix: preserve adaptive interruption across tool calls#2290rosetta-livekit-bot[bot] wants to merge 1 commit into
Conversation
🦋 Changeset detectedLatest commit: 12829f9 The changes in this PR will be included in the next version bump. This PR includes changesets to release 39 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| this.audioRecognition.onEndOfAgentSpeech( | ||
| options?.ignoreUserTranscriptUntil ?? Date.now(), | ||
| { paused: true }, | ||
| ); | ||
| } |
There was a problem hiding this comment.
🟡 Pausing the agent's voice now reports a bogus "user talked over the agent" event every time
An overlap is opened and then immediately closed as agent-ended (onStartOfOverlapSpeech at agents/src/voice/agent_activity.ts:1684-1689 followed by onEndOfAgentSpeech at agents/src/voice/agent_activity.ts:1696) whenever agent playout is paused, so listeners receive an extra overlapping-speech notification and a zero-valued measurement for a pause that produced no real judgement.
Impact: Applications and dashboards subscribed to overlapping-speech events see spurious entries (and zeroed interruption metrics) each time the agent's voice is paused.
Mechanism: overlap opened right before the boundary that closes it
Before this PR the pause path passed { paused: true }, so no agent-speech-ended sentinel was written and the overlap opened at agents/src/voice/agent_activity.ts:1684 stayed open across the pause (an event was produced only on a real verdict or user-speech end).
Now onEndOfAgentSpeech (agents/src/voice/audio_recognition.ts:784-831) always closes any open overlap via closeOverlap(now, undefined, true) and then sends agent-speech-ended. Since the pause path opens the overlap synchronously one statement earlier, the queued sentinel pair is overlap-speech-started + overlap-speech-ended(agentEnded: true), and the transform at agents/src/inference/interruption/interruption_stream.ts:274-300 enqueues an OverlappingSpeechEvent built from InterruptionCacheEntry.default() (all zeros, isInterruption: false). That event is emitted publicly (this.model.emit('overlapping_speech', chunk) at agents/src/inference/interruption/interruption_stream.ts:331) and forwarded to the session, plus an interruption_metrics event with numInterruptions: 0 / numBackchannels: 0.
If the intent is that pausing is a plain speech boundary, the overlap open at agents/src/voice/agent_activity.ts:1683-1689 is now redundant and should be dropped (the overlap is re-opened by onStartOfAgentSpeech when playout resumes while the user is still speaking, agents/src/voice/audio_recognition.ts:770-775).
Prompt for agents
In AgentActivity.interruptByAudioActivity (agents/src/voice/agent_activity.ts, pause branch), an overlap inference is started via audioRecognition.onStartOfOverlapSpeech(...) immediately before audioRecognition.onEndOfAgentSpeech(...) is called for the same pause. With the new semantics in AudioRecognition.onEndOfAgentSpeech (which now always closes an open overlap with agentEnded=true and then resets the detector), this pair produces an immediate, meaningless overlapping_speech event with all-zero timings plus a zeroed interruption_metrics event on every pause. Decide whether the overlap should still be opened at pause time at all: since AudioRecognition.onStartOfAgentSpeech now re-opens the overlap when playout resumes while the user is still speaking, the pause-time open appears redundant. Verify against the upstream Python implementation and remove the redundant open (or suppress emitting an event for an overlap that is closed with no inference requests).
Was this helpful? React with 👍 or 👎 to provide feedback.
Ports livekit/agents#6564 to preserve adaptive interruption when audible playout pauses for tool execution or resumes during active user speech.
@livekit/agents.Source diff coverage
livekit-agents/livekit/agents/inference/interruption.py->agents/src/inference/interruption/interruption_stream.ts. ExcludesagentEndedoverlaps from backchannel metrics; target event types already carry the marker.livekit-agents/livekit/agents/voice/agent_activity.py->agents/src/voice/agent_activity.ts. Removes pause/resume lifecycle exceptions, ends active speech at tool-call thinking gaps, rearms interruption handling, and removes duplicate pause-cancel teardown. The equivalent target realtime tool path is updated because JS separates pipeline and realtime execution.livekit-agents/livekit/agents/voice/audio_recognition.py->agents/src/voice/audio_recognition.ts. Closes overlap before detector reset, restarts inference when playout resumes over active user speech, preserves endpointing state, and serializes JS async sentinel batches to match Pythonsend_nowaitordering.livekit-agents/livekit/agents/voice/remote_session.py->agents/src/voice/remote_session.ts. Mirrors the source TODO because the remote-session protobuf cannot yet carryagentEnded. No unsupported wire field is invented.tests/test_agent_session.py->agents/src/voice/agent_activity_tool_output_commit.test.ts. Verifies speech end is observed after entering the tool-callthinkingstate and later ends occur while listening.tests/test_false_interruption_resume.py->agents/src/voice/agent_activity.test.ts. Verifies cancelling already-paused speech does not report a duplicate speech end.tests/test_interruption/test_overlapping_speech_event.py->agents/src/inference/interruption/interruption_stream.test.ts. Ports the added agent-ended metric assertion. The existing Python/Pydantic serialization assertion is not applicable to a compile-time TypeScript interface.tests/test_realtime_adaptive_interruption.py->agents/src/voice/realtime_adaptive_interruption.test.ts. Ports overlap-before-reset ordering, idempotent user end, resumed detector restart, and real speech-end assertions.Protocol gap
agentEndedstill cannot be forwarded by remote sessions because the shared AgentSessionEvent protobuf has no field for it. This matches the authoritative source PR, which documents the same limitation withTODO(AGT-3180). Local adaptive interruption behavior and metrics preserve the marker.Validation
pnpm test agents(2120 passed, 5 skipped)pnpm buildpnpm build:agentsafter final refinementpnpm --filter @livekit/agents typecheckpnpm --filter @livekit/agents lintpnpm format:checkcue-clivoice run: post-tool playout was adaptively interrupted,getWeatherexecuted for the replacement turn, and a fresh assistant response was committedSource: livekit/agents#6564
Ported from livekit/agents#6564
Original PR description
Bug: Adaptive interruption could silently drop a user overlap when agent playout resumed after a tool call because the interruption stream received mismatched speech boundaries.
Treat thinking gaps as real speech boundaries and restart overlap inference when playout resumes mid-utterance. Agent-ended overlaps remain inconclusive and no longer count as backchannels; remote forwarding is noted until the protocol carries that marker.
Fixes livekit/agents#6548. Fixes livekit/agents#6548.