fix(voice): keep the agent thinking while a tool is still running - #6908
Open
VihaanAgarwal wants to merge 1 commit into
Open
fix(voice): keep the agent thinking while a tool is still running#6908VihaanAgarwal wants to merge 1 commit into
VihaanAgarwal wants to merge 1 commit into
Conversation
longcw
reviewed
Aug 20, 2026
longcw
left a comment
Contributor
There was a problem hiding this comment.
looks good to me, but one thing I am not sure is
| current_span.set_attribute(trace_types.ATTR_RESPONSE_TEXT, forwarded_text) | ||
|
|
||
| if not speech_handle.interrupted and len(tool_output.output) > 0: | ||
| if not speech_handle.interrupted and (len(tool_output.output) > 0 or not exe_task.done()): |
Contributor
There was a problem hiding this comment.
this branch also calls _on_end_of_agent_speech and _restore_interruption_by_audio_activity
Contributor
There was a problem hiding this comment.
maybe move it to the other branch, but may need to take care if there is no tool reply after exe_task, the agent state should be back to listening
elif self._session.agent_state == "speaking":
# a tool still running keeps the agent busy: "listening" here would arm the
# user-away timer mid-tool (#6904)
still_running = not speech_handle.interrupted and not exe_task.done()
self._session._update_agent_state("thinking" if still_running else "listening")
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
with_fillerspeech during a long tool call knocks the session back tolistening, which arms the user-away timer while the tool is still running. Withuser_away_timeoutset, the user gets markedawaymid-lookup and the app's inactivity handling fires over the agent's own tool call.Two paths reset the state when a speech ends, and neither knows a tool is in flight:
_tts_task_implsetslisteningwhen the filler finishes playing_on_pipeline_reply_donedoes the same as the say task's done callbackThe pipeline reply has the same gap without any filler: at playout end it only picks
thinkingwhen a tool already finished (len(tool_output.output) > 0), so a slow tool drops the state tolisteninguntil its result comes back.The fix consults in-flight tool executions at those three spots.
_background_speechesalready tracks speeches that are awaiting their tools, so the two speech-end paths hand backthinkingwhen it's non-empty. The reply task also stays inthinkingwhenexe_taskhasn't completed yet.Added a test that runs a slow tool with a filler and a short
user_away_timeout. On main the filler ends inlisteningand the user flips toawaymid-tool. With the fix both speech ends return tothinkingand the away timer stays disarmed. Unit suite, ruff and mypy are green.Fixes #6904