Skip to content

fix(voice): keep the agent thinking while a tool is still running - #6908

Open
VihaanAgarwal wants to merge 1 commit into
livekit:mainfrom
VihaanAgarwal:fix/filler-away-timer-during-tool
Open

fix(voice): keep the agent thinking while a tool is still running#6908
VihaanAgarwal wants to merge 1 commit into
livekit:mainfrom
VihaanAgarwal:fix/filler-away-timer-during-tool

Conversation

@VihaanAgarwal

Copy link
Copy Markdown

with_filler speech during a long tool call knocks the session back to listening, which arms the user-away timer while the tool is still running. With user_away_timeout set, the user gets marked away mid-lookup and the app's inactivity handling fires over the agent's own tool call.

Two paths reset the state when a speech ends, and neither knows a tool is in flight:

  • _tts_task_impl sets listening when the filler finishes playing
  • _on_pipeline_reply_done does the same as the say task's done callback

The pipeline reply has the same gap without any filler: at playout end it only picks thinking when a tool already finished (len(tool_output.output) > 0), so a slow tool drops the state to listening until its result comes back.

The fix consults in-flight tool executions at those three spots. _background_speeches already tracks speeches that are awaiting their tools, so the two speech-end paths hand back thinking when it's non-empty. The reply task also stays in thinking when exe_task hasn't completed yet.

Added a test that runs a slow tool with a filler and a short user_away_timeout. On main the filler ends in listening and the user flips to away mid-tool. With the fix both speech ends return to thinking and the away timer stays disarmed. Unit suite, ruff and mypy are green.

Fixes #6904

@VihaanAgarwal
VihaanAgarwal requested a review from a team as a code owner August 19, 2026 14:18

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 2 additional findings.

Open in Devin Review

@longcw longcw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me, but one thing I am not sure is

current_span.set_attribute(trace_types.ATTR_RESPONSE_TEXT, forwarded_text)

if not speech_handle.interrupted and len(tool_output.output) > 0:
if not speech_handle.interrupted and (len(tool_output.output) > 0 or not exe_task.done()):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this branch also calls _on_end_of_agent_speech and _restore_interruption_by_audio_activity

@longcw longcw Aug 20, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe move it to the other branch, but may need to take care if there is no tool reply after exe_task, the agent state should be back to listening

elif self._session.agent_state == "speaking":
    # a tool still running keeps the agent busy: "listening" here would arm the
    # user-away timer mid-tool (#6904)
    still_running = not speech_handle.interrupted and not exe_task.done()
    self._session._update_agent_state("thinking" if still_running else "listening")

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Filler speech during a long tool call resets agent_state to listening, arming the user-away timer mid-tool

2 participants