Realtime voice pipeline: Silero VAD false triggers and ERROR after response generation

[jimdoughty65-bot]

I am building a local home voice assistant using LocalAI v4.10.0 on Windows 11 with Docker Desktop.

Hardware:

Minisforum UM890 Pro, Ryzen 9 8945HS, 64 GB RAM

NVIDIA RTX 5060 Ti 16 GB

ReSpeaker XVF3800 4-microphone USB array

My modular LocalAI Voice Pipeline is named nova-voice:

VAD: audio-cpp-silero-vad

Transcription: whisper-small-en

LLM: spark-x2.5-4b

TTS: kokoro

All four components can load and show Running.

Actual speech is working surprisingly well. Whisper normally transcribes my voice correctly and the LLM produces sensible responses.

I have two problems and would appreciate guidance on the correct LocalAI configuration rather than continuing to change settings blindly.

1. VAD is much too easily triggered by non-speech sounds

The ReSpeaker is a sensitive far-field microphone. Small real sounds such as:

keyboard clicks

shirt rustling

movement

a squeak from my swivel chair

can cause the realtime pipeline to leave LISTENING and process a turn.

Whisper then often produces things such as:

[BLANK_AUDIO]

[LAUGHTER]

[GASP]

[Music playing]

or occasionally hallucinated text.

For example, my chair squeaked and Whisper classified it as [LAUGHTER]. The LLM then naturally replied that it sounded like I was having a wonderful time laughing.

If I sit completely still in a quiet room, the system can remain in LISTENING for several minutes. So it appears that actual small noises are triggering the VAD rather than the pipeline triggering with no input.

What is the recommended way in LocalAI v4.10.0 to make realtime Silero VAD less sensitive to these short non-speech noises while preserving far-field voice pickup?

Should this be adjusted through pipeline.turn_detection, a Silero probability/threshold setting, eagerness, vad_window_sec, or another setting?

If possible, could someone provide an example YAML configuration appropriate for this use case?

2. Realtime pipeline appears to enter ERROR after response generation

A normal spoken turn goes approximately:

LISTENING → Hearing you speak → Processing → Generating Response → ERROR

Transcription generally takes about 2–3 seconds. LLM generation is inconsistent: sometimes a few seconds and sometimes much longer, even for simple questions.

The text response is generated successfully, but the UI appears to enter ERROR immediately afterward.

Kokoro itself loads successfully, but audible TTS output has not yet been confirmed. An earlier Talk-page Test Tone also produced no audible sound.

API Tracing is enabled, but the Traces page only shows:

POST /v1/realtime/calls → 201

The trace confirms "model": "nova-voice", but individual VAD/STT/LLM/TTS operations do not appear there.

What is the best log or diagnostic location for identifying which stage is causing the post-generation ERROR?

I would especially appreciate guidance on the intended LocalAI configuration before replacing models or changing microphone hardware. The goal is a continuous, natural, far-field home voice assistant.

Thank you for any guidance.