Support multiple participants in a single AgentSession with a shared chat context

[Swayam-Hingu-PE]

Feature Request: Support multiple participants in a single AgentSession

Problem

The current SDK assumes a single active participant per AgentSession. While the existing workarounds are sufficient for transcription, they don't support a single-session, multi-participant conversation with a shared chat context.

Use case

We're building AI-conducted interviews with human takeover.

1. An AI agent interviews a candidate in a 1:1 conversation.
2. A human interviewer joins the room and takes over questioning.
3. The agent switches to a silent "observer" mode—it stops speaking but continues listening.
4. While silent, it must continue hearing both the interviewer and the candidate, appending both to the same chat context.
5. When the interviewer leaves or hands control back, the agent resumes the interview with the complete conversation history.

The requirement is therefore:

N participants → 1 AgentSession → 1 shared chat context → 1 LLM/TTS, with the agent's speaking ability toggleable independently of its listening.

Why the current approaches don't work

• RoomIO.set_participant() / toggle_io.py

  • Only one participant can be attached at a time.
  • Participants that are not currently attached are not included in the conversation history, resulting in a one-sided context.

• Multiple AgentSessions

  • This creates one session per participant, which also means one chat context per participant.
  • Our use case requires a single shared context and a single LLM/TTS pipeline.

SDK limitation (agents 1.6.7)

From our investigation, the current implementation appears to make two single-participant assumptions:

• _participant_available_fut latches onto the first participant (room_io.py:378-396).
• _ParticipantInputStream maintains a single _stream / _publication (_input.py:202-204), so a second participant's track replaces the existing one instead of being added alongside it.

Expected behavior

It should be possible for a single AgentSession to receive audio from multiple participants while maintaining a single conversation context. Each completed user turn should also include the speaker identity so the application can attribute transcripts correctly while preserving one shared chat history.

This would enable use cases such as:

• AI interviews with human takeover
• Meeting copilots
• Customer support escalations
• Collaborative conversations where the agent needs to listen to multiple participants while speaking selectively

[Darshak03]

@longcw I've opened #6855 implementing this via participant audio mixing (AudioInputOptions.mix_participants=True), can you pls review it

[Darshak03]

@longcw pls suggest if you have any other approach
#6855 has one limitation: it publishes the transcript on behalf of the linked participant

[longcw]

thanks for the pr, but I think the composition point should be an AudioInput instead of mixing in RoomIO.

audio_input: AudioInput
                   ├─ 1 participant → _ParticipantAudioInputStream   (unchanged)
                   └─ N            → _MultiParticipantAudioInput
                                       ├─ _ParticipantAudioInputStream(A)   ← unchanged
                                       ├─ _ParticipantAudioInputStream(B)
                                       └─ strategy: mix | pick

RoomIO keeps the participant lifecycle it already owns (_on_participant_connected/_disconnected plus the kind and ATTRIBUTE_PUBLISH_ON_BEHALF filtering) and just calls add(participant)/remove(identity).

[Darshak03]

@longcw can u pls check this #6884 PR, pls suggest if any changes needed

[longcw]

@Darshak03 before moving forward, I'd like to understand why you don't want to run STT separately and feed the transcripts from multiple participants to the AgentSession. Mixing the audio lost the participant info per stream, and you may need STT diarization in that case.