fix: preserve adaptive interruption across tool calls

[chenghao-mou]

Bug: Adaptive interruption could silently drop a user overlap when agent playout resumed after a tool call because the interruption stream received mismatched speech boundaries.

Treat thinking gaps as real speech boundaries and restart overlap inference when playout resumes mid-utterance. Agent-ended overlaps remain inconclusive and no longer count as backchannels; remote forwarding is noted until the protocol carries that marker.

Fixes #6548. Fixes #6548.

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

[biztex]

This looks like the right fix — treating the thinking gap as a real speech boundary and restarting overlap inference when playout resumes addresses the root cause at the signal source, where my #6550 only made the stream tolerate the redundant start sentinel. The metrics and remote-session coverage go beyond what mine attempted, too.

Closing #6550 in favor of this. If the hermetic stream-level regression tests from it would be useful here (a fake bargein gateway driving InterruptionWebSocketStream through a mid-overlap second segment), happy to adapt them to the new sentinel semantics — just say the word.

[longcw]

question: what is the difference between session.agent_state and self._agent_speaking?

[biztex]

(Chiming in from #6550, since this distinction was the heart of #6548 — hope it helps.)

They live at different layers and deliberately diverge mid-turn:

• session.agent_state is the session-level lifecycle (listening / thinking / speaking), updated via AgentActivity._update_agent_state(). A reply that produces tool calls flips it to "thinking" between segments; it is what applications observe through agent_state_changed.

• AudioRecognition._agent_speaking is a private recognition-side flag meaning "an agent playout window is open for interruption/endpointing purposes". It is set by _on_start_of_agent_speech() (first audio frame of playout) and cleared only inside _on_end_of_agent_speech() — which is intentionally not called when the reply produced tool calls, so it stays True straight through the thinking gap. It gates overlap tracking in _on_start_of_speech(), the transcript-ignore window, and _should_ignore_input_audio().

So during a tool-call gap the two disagree by design: agent_state == "thinking" while _agent_speaking == True. was_speaking as written captures "playout was audibly active at reply-done" (session view) rather than "the recognition-side speech window is still open" — which one is wanted here is of course the author's call.

[longcw]

from claude code:

#6662 section (b) is the same bug with a different trigger: a second agent-speech boundary inside one turn resets the detector stream while an overlap is open. It corrected the false-interruption pause. It did not touch the tool branch:

agents/livekit-agents/livekit/agents/voice/agent_activity.py

Lines 3443 to 3452 in 59c2fae

	if not speech_handle.interrupted and len(tool_output.output) > 0:
	self._session._update_agent_state("thinking")
	elif self._session.agent_state == "speaking":
	self._session._update_agent_state("listening")
	if self._audio_recognition:
	self._audio_recognition._on_end_of_agent_speech(
	ignore_user_transcript_until=time.time()
	)
	if self.interruption_enabled:
	self._restore_interruption_by_audio_activity()

The tool-response reply is a new _pipeline_reply_task on the same speech_handle, so its playout start reaches the same call site as a first segment, where resumed defaults to False. Second start sentinel, _reset_state(), verdict gone. _restore_interruption_by_audio_activity() is in the skipped elif, so there is no VAD fallback either.

Two edits instead:

1. Call _on_end_of_agent_speech(..., paused=True) and _restore_interruption_by_audio_activity() in the tool branch. A pause keeps _overlap_open and sends no reset, so user audio keeps reaching the gateway while the tools run.
2. Add resumed = resumed or self._overlap_open in _on_start_of_agent_speech. _overlap_open is set only while an overlap is unresolved, so a new turn still resets. This covers the queued say() case too.

Then the synthesized _on_start_of_speech, the _agent_speaking guard, both was_speaking gates and the self._speaking and condition can all go. Three of them have side effects: the synthesized onset overwrites _utterance_started_at with the resume time, the guard lets _turn_tracker reset mid-utterance, and speech_duration=0.0 with _overlap_count = 0 shifts the buffer down to _prefix_size.

Keep the sentinel reorder. It fixes a separate bug, and it is real: the reset currently goes out before the agent_ended=True sentinel, so the stream drops that event and audio_recognition.py:502 never emits. The num_backchannels change needs the reorder to reach the metrics at all.

[devin-ai-integration]

Devin Review found 1 new potential issue.

View 5 additional findings in Devin Review.

[devin-ai-integration]

🟡 Inconclusive overlap results are now reported to remote sessions as if the user only backchanneled

Overlaps that end because the agent stopped talking are now published (_on_overlapping_speech at livekit-agents/livekit/agents/voice/remote_session.py:602-606) with no marker that the result is inconclusive, so remote listeners read them as confirmed "user did not interrupt" results.

Impact: Remote dashboards/consumers will count these inconclusive overlaps as backchannels, inflating backchannel statistics for turns where the user may actually have been interrupting.

Why these events start reaching the remote session only now

Before this PR, _on_end_of_agent_speech sent _AgentSpeechEndedSentinel before _on_end_of_overlap_speech, so the detector stream had already reset (_reset_state() clears _overlap_started) and the subsequent _OverlapSpeechEndedSentinel produced no OverlappingSpeechEvent (see livekit-agents/livekit/agents/inference/interruption.py:621-640). The reordering in livekit-agents/livekit/agents/voice/audio_recognition.py:509-514 now emits the event with agent_ended=True, is_interruption=False.

Locally this was compensated for: livekit-agents/livekit/agents/inference/interruption.py:670 no longer counts agent_ended events as backchannels, and AudioRecognition._on_overlap_speech_event skips the backchannel latch for them (livekit-agents/livekit/agents/voice/audio_recognition.py:1480). The remote forwarding path has no such compensation — it only forwards is_interruption, detection_delay, detected_at, overlap_started_at. Until the protocol carries agent_ended (the TODO), an option is to skip forwarding agent_ended events rather than forwarding them as non-interruptions.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[biztex]

Verified this against the branch head (217d25f) — the finding is real, and it is newly reachable behavior rather than a pre-existing gap:

• Before this PR, _on_end_of_agent_speech pushed _AgentSpeechEndedSentinel before _on_end_of_overlap_speech, so _reset_state() had already cleared _overlap_started and the detector's _OverlapSpeechEndedSentinel arm produced no event. With the reordering (audio_recognition.py:509-514), the detector now emits OverlappingSpeechEvent(is_interruption=False, agent_ended=True) (interruption.py:621-641) — an event that simply never fired before.
• AgentActivity._on_overlap_speech_ended forwards every event to the session unconditionally (agent_activity.py:1887-1889), so the remote forwarder sees them all.
• Both local consumers were taught the distinction (interruption.py:670 excludes agent_ended from num_backchannels; audio_recognition.py:1480 skips the backchannel latch), but the wire message cannot express it: OverlappingSpeech in livekit/protocol (protobufs/agent/livekit_agent_session.proto) carries only is_interruption / overlap_started_at / detection_delay / detected_at. A remote consumer therefore has no choice but to read these as confirmed backchannels.

Since the proto change lives in livekit/protocol (an optional bool agent_ended = 5 would be backward-compatible there), the self-contained interim fix is to skip forwarding inconclusive verdicts, in line with the existing TODO:

Suggested change

	# TODO(AGT-3180): Forward agent_ended when the remote-session protocol supports it.
	# TODO(AGT-3180): Forward agent_ended when the remote-session protocol supports it.
	if event.agent_ended:
	# an inconclusive verdict is indistinguishable from a confirmed backchannel
	# on the wire; skip it until the protocol can carry the marker
	return

Suppressing the event entirely matches pre-PR remote behavior (these overlaps produced no event at all), so nothing downstream regresses in the meantime, and the TODO keeps the protocol follow-up visible.