user_away_timeout getting triggered for SIP participant on telephony calls even when user is not speaking and taking background noise and conversations as user state speaking.

[abhishekdeepmindz]

Bug Description

So, in my telephony voice AI agent application i have implemented a logic where after the user is away for configured time it will prompt the user whether they are still there with messages like are you still there? initially i had implemented it via llm, stt nodes by resetting a timer everytime a new speech or response was detected, purely on transripts, if no new transcripts then it would prompt the message. We moved to user_away_timeout session variables for easier calculation. But now this is causing the prompt message to not be called because we keep getting the state changes every second if there is background noise or background inaudible speech.
How does this work internally?
Following is my session parameters.

{ "allow_interruptions": true, "discard_audio_if_uninterruptible": true, "min_interruption_duration": 0.3, "min_interruption_words": 0, "min_endpointing_delay": 0.4, "max_endpointing_delay": 2, "max_tool_steps": 3, "user_away_timeout": 10, "min_consecutive_speech_delay": 0, "preemptive_generation": { "enabled": true, "preemptive_tts": true, "max_speech_duration": 10, "max_retries": 3 } }

Self hosted Livekit Instance, livekit agents version ~=1.5, using BVC Telephony

[EVENT] type='user_state_changed' old_state='speaking' new_state='listening' created_at=1780986023.7001052
[EVENT] type='user_state_changed' old_state='listening' new_state='speaking' created_at=1780985971.4367104
[EVENT] type='user_state_changed' old_state='listening' new_state='speaking' created_at=1780985971.4367104
[EVENT] type='user_state_changed' old_state='speaking' new_state='listening' created_at=1780986026.7517605
[EVENT] type='user_state_changed' old_state='listening' new_state='away' created_at=1780986036.752435

Expected Behavior

Should it be activated on background noise as well?

Reproduction Steps

@session.on("user_state_changed")
        def on_user_state_changed(ev):
            logger.info(f"[EVENT] {ev}")

            if ev.new_state == "away":
                logger.info("User is inactive")
                self.user_is_away = True

                # prevent duplicate loops
                if self.retry_task is None or self.retry_task.done():
                    self.retry_task = asyncio.create_task(self.handle_user_away(session))

            elif ev.new_state in ["speaking", "listening"]:
                logger.info("User is back")
                self.user_is_away = False
                self.idle_warning_count  = 0

                # cancel retry loop immediately
                if self.retry_task:
                    self.retry_task.cancel()
                    self.retry_task = None

Operating System

Linux VM

Models Used

Deepgram nova-3, gpt-oss 120B

Package Versions

livekit>=1.1.5
livekit-agents[azure,cartesia,deepgram,elevenlabs,google,groq,openai,sarvam]~=1.5

Session/Room/Call IDs

No response

Proposed Solution

Additional Context

No response

Screenshots and Recordings

No response

[longcw]

so you want to ignore the user turn without transcription (caused by noise) when tracking the user away state?

[abhishekdeepmindz]

If i am not using any VAD and vad is kept as None in AgentSession and using stt as turn_detection provider in the config. Then how is turns getting triggered via noise? Shouldnt it be dependent on user transcripts? Or am I missing any configuration?

[longcw]

oh wait, you don't have an VAD? what is your turn detection mode then

[acharan-tech-200037]

A couple of things might be worth trying to narrow down where the state changes are coming from:

1. Enable BVC Telephony noise cancellation (if not already enabled)

Since this is a telephony use case, background TV audio, nearby conversations, road noise, etc. may be reaching the agent and being interpreted as speech activity. It would be useful to test with BVC's noise suppression/noise cancellation enabled and compare the frequency of listening -> speaking transitions.

2. Try Adaptive Interruption Handling

If you're not already using adaptive interruption handling, it may help distinguish actual user intent from short background utterances/backchannel audio better than pure audio activity detection.

3. Consider tracking inactivity using agent states as well

If the noisy telephony environment is causing frequent user_state_changed transitions, another approach could be to track inactivity from the agent side.

For example, if the agent has not been in a speaking or thinking state for a configured duration, it may indicate that the conversation has stalled and the user is not actively responding.

Something along the lines of:

• Agent finishes speaking.
• No meaningful user response/transcript is received.
• Agent is neither speaking nor thinking for N seconds.
• Trigger a "Are you still there?" prompt.

This may be more reliable in noisy telephony environments where background audio keeps resetting user activity detection.

[dorukdumlu]

Dug into this a bit — I think the root cause is that away tracking is keyed off raw audio-activity state flips, and every flip back to listening re-arms the timer with the full timeout.

The chain in turn_detection="stt" mode (the reporter's setup, Deepgram + no VAD):

1. Deepgram's SpeechStarted is an energy-based signal that fires on background noise without ever producing a transcript — the plugin itself notes it's not correlated with transcripts:

   agents/livekit-plugins/livekit-plugins-deepgram/livekit/plugins/deepgram/stt.py

   Lines 793 to 803 in f8ad46d

   	if data["type"] == "SpeechStarted":
   	# This is a normal case. Deepgram's SpeechStarted events
   	# are not correlated with speech_final or utterance end.
   	# It's possible that we receive two in a row without an endpoint
   	# It's also possible we receive a transcript without a SpeechStarted event.
   	if self._speaking:
   	return
   	self._speaking = True
   	start_event = stt.SpeechEvent(type=stt.SpeechEventType.START_OF_SPEECH)
   	self._event_ch.send_nowait(start_event)

2. In stt turn-detection mode that becomes START_OF_SPEECH → on_start_of_speech → user_state = "speaking", which cancels the away timer:

   agents/livekit-agents/livekit/agents/voice/audio_recognition.py

   Lines 1244 to 1251 in f8ad46d

   	elif ev.type == stt.SpeechEventType.START_OF_SPEECH and self._turn_detection_mode == "stt":
   	# If the plugin provided a server onset timestamp, use it;
   	# otherwise fall back to message arrival time.
   	if self._speech_start_time is None:
   	self._speech_start_time = ev.speech_start_time or time.time()
   	with trace.use_span(self._ensure_user_turn_span(start_time=self._speech_start_time)):
   	self._hooks.on_start_of_speech(None, speech_start_time=self._speech_start_time)

3. When the (noise) segment ends, the state flips back to listening and _set_user_away_timer() re-arms from scratch with the full user_away_timeout:

   agents/livekit-agents/livekit/agents/voice/agent_session.py

   Lines 1764 to 1767 in f8ad46d

   	if state == "listening" and self._agent_state == "listening":
   	self._set_user_away_timer()
   	else:
   	self._cancel_user_away_timer()

So with telephony background noise triggering a segment every few seconds, the away deadline gets pushed out indefinitely even though the user never actually said anything. Same story for VAD-based setups on a noisy line — BVC helps but doesn't fix the tracking logic itself.

Interestingly the codebase already treats transcripts as the trustworthy signal in the opposite direction: leaving the away state only happens on a final transcript ("in case VAD has a miss detection"):

agents/livekit-agents/livekit/agents/voice/agent_session.py

Lines 1789 to 1791 in f8ad46d

	if self.user_state == "away" and ev.is_final:
	# reset user state from away to listening in case VAD has a miss detection
	self._update_user_state("listening")

Proposal: make away tracking deadline-based instead of reset-based, anchored to meaningful activity rather than raw state flips:

• track a last_activity_time on the session, updated when (a) a non-empty user transcript arrives (interim or final, via _user_input_transcribed), and (b) the agent leaves the listening state
• when both user and agent return to listening, arm the timer with the remaining time: max(0, user_away_timeout - (now - last_activity_time)) instead of the full timeout
• timer fire behavior unchanged (user_state = "away")

Effect: speech segments that never produce a transcript (noise) no longer defer "away", while real speech does — because it produces transcripts. user_state semantics stay exactly as they are (speaking/listening flips, interruption behavior untouched), no new config needed. Setups without any transcription would keep the current behavior as a fallback, since there's no signal to distinguish noise there.

Happy to put up a PR with a regression test (noise-only segments deferring the away deadline) if this direction sounds right.

[dorukdumlu]

Following up on the proposal above — unless you'd rather a different direction, I'm going to open a draft PR with the deadline-based away tracking (last_activity_time + remaining timeout) and a regression test for noise-only segments. Happy to change course if that framing is wrong.

[dorukdumlu]

Draft PR up with the deadline-based approach + regression tests: #6499 — happy to adjust if you'd rather a different activity signal.

[dorukdumlu]

Update on #6499: the cancel policy you asked for is in. Default stays presence. For this telephony case set user_away_on=conversation so background noise does not hold the away timer. Rebased onto current main.

@longcw