VAD/STT stop reacting mid-call (user_state stuck in "listening") on telephony (SIP) calls

[ullaimlu]

Bug Description

Description

The agent responds correctly on the first conversational turn. At some later point in the call (the exact point varies between calls, not consistently tied to a specific turn or event), VAD and STT stop generating any events entirely — no user_state_changed events, no STT transcripts (partial or final) — even though audio frames from the participant continue to arrive (confirmed separately with an independent audio track subscription that keeps receiving frames throughout).

As a result, our inactivity-prompt logic (based on agent_state_changed → "listening") repeatedly says filler prompts like "I can't hear you, please repeat" and eventually hangs up the call due to perceived silence — even though the user is actively speaking.

Expected Behavior

Looks isolated to VAD: it stops emitting speech/silence transitions
mid-call while audio keeps arriving.

The expected behavior should be the VAD user state returning to speaking so the STT can work and the agent do not fall on hanging up for inactivity.

Reproduction Steps

We can't see any log or exception when this happens, and it only occurs
in some calls, so we don't have a reliable way to reproduce it on demand.

Operating System

• OS / deployment: self-hosted - Call type: outbound SIP telephony call

Models Used

Deepgram nova 3, gpt 4.6 luna, Deepgram aura 2

Package Versions

- livekit==1.1.14
- livekit-agents[openai,deepgram,cartesia,silero,turn_detector]==1.6.9

Session/Room/Call IDs

No response

Proposed Solution

Additional Context

What we've already ruled out

• Removed allow_interruptions=False from all session.say() / generate_reply() calls, replacing it with a temporary elevation of the session's interruption.min_words threshold instead — no change.
• Switched from the deprecated silero.VAD.load() plugin to livekit.agents.inference.VAD(model="silero") — no change.
• Switched turn detection from the older livekit.plugins.turn_detector.multilingual.MultilingualModel (text-based) to the newer inference.TurnDetector(version="v1-mini") (audio-based) — no change.
• Confirmed via a separate rtc.AudioStream.from_participant() subscription that audio frames are continuously arriving on the participant's track throughout the freeze — so it does not appear to be a SIP/network/one-way-audio issue.

Screenshots and Recordings

Turn detection / VAD config in use

turn_handling=TurnHandlingOptions(
    turn_detection=inference.TurnDetector(version="v1-mini"),
    endpointing={"min_delay": 0.5, "max_delay": 1},
    interruption={
        "mode": "vad",
        "min_words": 3,
        "discard_audio_if_uninterruptible": False,
    },
    preemptive_generation={"preemptive_stt": False, "enabled": False},
),
vad=inference.VAD(
    model="silero",
    min_speech_duration=0.05,
    min_silence_duration=0.25,
    activation_threshold=0.4,
),

Log example

Timeline:
18:14:37 - call answered, AMD detects human
18:14:38 - audio input attached, STT/TTS pipelines connected
18:14:49 - agent speaks first line
18:14:49 - user_state_changed: listening -> speaking (VAD working)
18:14:51 - STT final transcript received
18:14:51 - user_state_changed: speaking -> listening (VAD working)
18:14:56 - function_tool executes
18:14:59 - agent speaks a short confirmation line
18:14:59 - "reusing STT pipeline from previous activity" logged
18:15:02 - agent delivers a longer message and waits for the user response
From here on the user keeps speaking (confirmed live, and separately
verified via an independent audio track subscription that keeps
receiving frames), but VAD never fires another user_state_changed
event for the rest of the call (~75s). No STT transcripts either,
which looks like a downstream effect of VAD not detecting speech.
VAD state of the user gets stuck on "listening" although the user is speaking.
18:15:42 / 18:15:49 / 18:15:56 - inactivity prompts fire
18:16:10 - agent hangs up, no response detected