Filler speech during a long tool call resets `agent_state` to `listening`, arming the user-away timer mid-tool

[kucharzyk-sebastian]

Bug Description

with_filler exists to cover long-running tool calls. Using it during one causes the session to mark the user away while the tool is still running, which fires whatever inactivity handling the app has attached — in our case a "are you still there?" re-prompt spoken over a lookup the agent had itself just announced.

The user-away timer is armed only when both sides are idle:

# agent_session.py (main) — _update_agent_state at L1966, _update_user_state at L2010
if state == "listening" and self._agent_state == "listening":
    self._set_user_away_timer()
else:
    self._cancel_user_away_timer()

A running tool is supposed to be protected by the transition to thinking:

# agent_activity.py (main, ~L4197)
# wait for the tool execution to complete
tool_output.first_tool_started_fut.add_done_callback(
    lambda _: self._session._update_agent_state("thinking")
)

But thinking is asserted once, when the first tool starts. When a filler plays, the session goes to speaking, and at the end of that speech two separate paths put it back to listening, neither aware that a tool is still running:

# 1. agent_activity.py, in _tts_task_impl        (main L3002 / 1.6.9 L2948)
if self._session.agent_state == "speaking":
    self._session._update_agent_state("listening")

# 2. agent_activity.py, _on_pipeline_reply_done  (main L2769 / 1.6.9 L2719)
#    registered as the done callback for the say() task itself
if not self._speech_q and (not self._current_speech or self._current_speech.done()):
    self._session._update_agent_state("listening")

Nothing restores thinking afterwards — the next transition to thinking (main L3498 / 1.6.9 L3448) requires len(tool_output.output) > 0, i.e. the tool has already finished. So for the remainder of the tool call the session sits in listening, the away timer arms, and after user_away_timeout the user is marked away even though the agent is demonstrably busy.

For a non-realtime session, say() dispatches to _tts_task → _tts_task_impl (main L1490-1502), so filler speech reaches both paths above. There is no tool-in-flight state anywhere in agent_activity.py for either path to consult.

Sequence:

1. Tool starts → agent_state = "thinking" → away timer cancelled. Correct.
2. Filler plays → agent_state = "speaking".
3. Filler ends → agent_state = "listening" (L3002).
4. Both sides listening → _set_user_away_timer() arms.
5. Tool still running → user_away_timeout elapses → user_state = "away".

The result is that the feature built for long tool calls disables the state protection that makes long tool calls safe. An agent that says nothing during the tool is protected; an agent that uses with_filler as documented is not.

Real transcript shape:

Agent:  "One moment. I'm checking."
        ...lookup runs, >5s...
Agent:  "Hello? Can you hear me?"

Expected Behavior

A tool call in flight keeps the user-away timer disarmed, whether or not filler speech played during it. Either:

• filler speech restores the previous agent_state when it ends, rather than hardcoding listening; or
• the away timer is not armed while any tool is in flight, independently of agent_state.

Reproduction Steps

1. Create an AgentSession with user_away_timeout set (we use 5.0; the default 15.0
   reproduces with a correspondingly slower tool).
2. Attach a user_state_changed handler that logs new_state == "away".
3. Define a function tool that wraps an operation slower than user_away_timeout in
   ctx.with_filler(...).
4. Trigger the tool and stay silent.
5. The filler plays; user_state flips to "away" while the tool is still running.
   Without the with_filler block, "away" does not fire — the session stays in
   "thinking" for the whole tool call.

- Minimal example -

session = AgentSession(..., user_away_timeout=5.0)

@session.on("user_state_changed")
def _on_user_state(ev):
    print("user_state:", ev.new_state, "agent_state:", session.agent_state)

class Assistant(Agent):
    @function_tool()
    async def slow_lookup(self, ctx: RunContext, q: str) -> str:
        """A tool slower than the away timeout."""
        async with ctx.with_filler("Just a second.", delay=0.6):
            await asyncio.sleep(10)
        return "done"

# Logs: user_state: away agent_state: listening   <-- while the tool is still running

Operating System

Linux 6.6.114.1-microsoft-standard-WSL2 (WSL2 on Windows 11)

Models Used

No response

Package Versions

livekit==1.1.12
livekit-agents==1.6.4
livekit-api==1.1.1
livekit-plugins-soniox==1.6.4
python==3.14.4

Session/Room/Call IDs

No response

Proposed Solution

# Option 1 — restore the state the session held before the filler spoke, instead of
# assuming a filler always returns the session to listening.
# In _tts_task_impl, capture agent_state before speaking and restore it after, or skip
# the transition entirely when a tool is still in flight:

if self._session.agent_state == "speaking":
    if self._tools_in_flight():          # tool still running -> go back to thinking
        self._session._update_agent_state("thinking")
    else:
        self._session._update_agent_state("listening")

# Option 2 — decouple the away timer from agent_state for this case: do not arm it
# while any tool is in flight, so no speech during a tool call can arm it.

Additional Context

No response

Screenshots and Recordings

No response