Switch VideoAvatars for AgentSession with multiple agents

[Viktoriagrg]

Bug Description

When using VideoAvatars with multiple agents under a single AgentSession, the audio input/output (QueueAudioOutput) is initialized once when the avatar starts. Currently, there’s no way to reset or reassign the audio buffer for a different avatar of the same AgentSession.

This means we cannot smoothly switch avatars without restarting the AvatarSession, which causes a brief moment of the video closing.

Expected Behavior

• The audio buffer should support resetting or reassigning per avatar.
• Switching avatars should happen smoothly with no video interruption.

Reproduction Steps

Some have tried to switch avatars by reassigning the destination_identity of the audio output:

self.session.output.audio.destination_identity = avatar._avatar_participant_identity

However, this does not solve the problem. The reason is that the audio buffers for already-started avatars have already been created and tied to the initial avatar session.

self._audio_buffer = QueueAudioOutput(sample_rate=SAMPLE_RATE) await self._audio_buffer.start() self._audio_buffer.on("clear_buffer", self._on_clear_buffer) agent_session.output.audio = self._audio_buffer

• Once the avatar is started, the _audio_buffer is bound to the session and cannot be reset or reassigned to another avatar dynamically.
• Simply changing destination_identity does not create a new audio buffer or detach the old one.

Operating System

macOS

Models Used

No response

Package Versions

livekit==1.0.20
livekit-agents==1.3.6

Session/Room/Call IDs

No response

Proposed Solution

add restart() method to AvatarSession, something like:

async def restart(self):
        self._audio_buffer.on("clear_buffer", self._on_clear_buffer)
        self._agent_session.output.audio = self._audio_buffer


also update identity in `switch` method

Additional Context

No response

Screenshots and Recordings

No response

[chenghao-mou]

If I understand your ask correctly, you want something like update_options for the AvatarSession, right? (We do have something similar in AgentSession and AgentServer already, but not with avatars)

[Viktoriagrg]

Yeah, something like this. The idea is to introduce a way to control the avatar’s active/inactive state so agent speech can be handled correctly within a single AgentSession

[chenghao-mou]

Unless the avatar vendor you use supports updating settings mid-streaming, this is no way for us to update or change without a reconnection. By design, there should be one instance of AvatarSession and AgentSession for each session. More specifically, AvatarSession is actually tied to the session/room, not really the agent.

I can see a use case like an interactive game where you have different NPCs each have their own avatars in one session. But I don't think we designed it with that kind of feature in mind. We can keep this open to see if there is more interest in this.

[Arjun-A-I]

I do see use cases here, for a multi agentic flow where we could have different voices for different agents (restaurant_agent as ex). It would be weird to see the same agent avatar for all of them. An option to update the avatar session settings would be awesome (through which we can change the avatar id). Avatar session restart would be a tradeoff tho which is fine.

[dimasimonowich]

Hi, @chenghao-mou !
Thank you for the response
Also interested in this issue

LiveKit provides the Meet repository as a reference for integrating agents into real, user-facing products. In these scenarios, tearing down and restarting an AvatarSession to make it active and switch avatars causes visible video interruptions. However, this is a common requirement for agent-based meeting applications, not an edge case.

Enforcing “one AvatarSession per session” effectively means avatar switching requires reconnection, making multi-agent or multi-avatar experiences within a single room impractical.

From my point of view, smooth avatar switching within a session would be highly beneficial for LiveKit usage for multi-agent meet-like scenarios. Given this, it would be helpful to understand the role Meet is expected to play in agent-driven products if dynamic avatar switching within a session is not supported, or if it is merely a self-hosted version of LiveKit Playground.

Keeping it open makes sense, but for me, it seems like a product limitation right now.

[chenghao-mou]

To understand your ask better, does this mean you are actually fine with spending N times of avatar cost (N is the number of avatars you want to run in parallel) so that they can switch smoothly?

I had a rough idea about the cost, but didn't know it made economic sense to run them in parallel in 2025 2026. If that's the case, we can just expose a switch avatar function at the agent session level, assuming they are all up and running already.

[dimasimonowich]

@chenghao-mou
It would be great to have this opportunity to switch between avatars. Yes — the expectation is to run multiple avatars in parallel and incur the associated costs. Smooth switching is more important than minimizing avatar runtime cost in these scenarios.

[chenghao-mou]

It is doable, but the client also needs to change to select the "active" one automatically.

This is me switching manually in the playground, but the active avatar and audio are done with a transfer call.

output.mp4

I will talk to the team about how to proceed.

[Viktoriagrg]

Hi @chenghao-mou , thanks for the clarification that this is feasible with client-side support and an active-avatar transfer.

Just checking if there are any updates on how the team plans to move forward with this (API changes or recommended approach). This would be very useful for enabling seamless avatar switching without restarting the AvatarSession.

Thanks.

[chenghao-mou]

Unfortunately, we don't have a timeline for this atm. Will share more once I have it.

[longcw]

hot swap avatar sessions is supported in #5863