Skip to content

Add opt-in DirectPolling microphone capture mode (Quest capture reliability) - #379

Open
faryal-9 wants to merge 1 commit into
livekit:mainfrom
faryal-9:faryal/direct-polling-mic-capture
Open

Add opt-in DirectPolling microphone capture mode (Quest capture reliability)#379
faryal-9 wants to merge 1 commit into
livekit:mainfrom
faryal-9:faryal/direct-polling-mic-capture

Conversation

@faryal-9

Copy link
Copy Markdown

Problem

The default microphone capture path plays the mic clip through an AudioSource and captures it via AudioProbe's OnAudioFilterRead. That couples two clocks: the microphone driver writes the clip ring at the input device's rate, while the AudioSource reads it at the output pipeline's rate. On devices where those clocks drift, the filter's read position slides against the microphone's write position until it consumes stale or not-yet-written regions of the ring — which surfaces as intermittent silence and crackle.

We measured this in production on Meta Quest 3, whose capture pipeline runs at 24 kHz (the device-format mismatch family reported in #172; the format half was fixed by #304, but the two-clock transport remains).

Fix

An opt-in MicrophoneCaptureMode.DirectPolling that reads 10 ms frames straight out of the microphone clip with Microphone.GetPosition/AudioClip.GetData on the main thread and raises AudioRead directly — no AudioSource or AudioProbe involved, so the microphone's own write position is the only clock and drift cannot accumulate. This is the same capture strategy Photon Voice and Mumble use on the same hardware.

Default behavior is unchanged: the existing constructor keeps the filter-based path; the new mode is selected via a constructor overload.

Details

  • Ring wrap-around is handled modularly; a transient GetPosition() == -1 (observed on some Android device states) is skipped rather than fed into GetData.
  • After a main-thread hitch (GC, scene load), backlog beyond a 300 ms catch-up budget is dropped oldest-first: the native audio source ingests 10 ms frames into a bounded queue at real time, so emitting hundreds of ms of stale audio in one tick overflows it — and real-time voice cannot use that audio anyway.
  • Channel shape adapts to the source's resolved device format: mono → N duplication (the common Android mono-clip case), N → mono averaging, first-channel fallback otherwise (warned once).
  • A throwing AudioRead subscriber is contained with a throttled warning: the loop is the sole frame producer, so an escaped exception would leave the microphone permanently silent.
  • Stop/restart — including the existing iOS pause/resume path — invalidate the loop via a generation counter.

Tested

This ports the capture loop we have been running in production in our Quest app (as a vendored SDK modification) onto the current main layout.

  • Meta Quest 3 (Unity 2022.3, Android, 24 kHz mono capture → stereo source): multi-hour sessions including repeated network drops and SDK reconnects show steady 10 ms frame production (~250 packets per 5 s confirmed via OutboundRtp packet counters) with no silence and no drift — where the filter-based path previously degraded.
  • Editor (Windows, 48 kHz): verified against a remote peer for parity with the default path.

Fixes #172 (for apps that opt in).

…ility)

The default capture path plays the microphone clip through an AudioSource
and captures it via AudioProbe's OnAudioFilterRead. That couples two
clocks: the microphone driver writes the clip ring at the input device's
rate while the AudioSource reads it at the output pipeline's rate. On
devices where those clocks drift, the filter's read position slides
against the microphone's write position until it consumes stale or
not-yet-written regions of the ring, which surfaces as intermittent
silence and crackle. We measured this in production on Meta Quest 3
(whose capture pipeline runs at 24 kHz).

This adds an opt-in MicrophoneCaptureMode.DirectPolling that reads 10 ms
frames straight out of the clip with Microphone.GetPosition/GetData on
the main thread and raises AudioRead directly - no AudioSource or
AudioProbe involved, so the microphone's own write position is the only
clock and drift cannot accumulate. This is the same capture strategy
Photon Voice and Mumble use on the same hardware.

Details:
- Default behavior is unchanged: the existing constructor keeps the
  filter-based path; the new mode is selected via a constructor overload.
- Ring wrap-around is handled modularly; a transient GetPosition() == -1
  (observed on some Android device states) is skipped rather than fed
  into GetData.
- After a main-thread hitch (GC, scene load) the backlog beyond a 300 ms
  catch-up budget is dropped oldest-first: the native audio source
  ingests 10 ms frames into a bounded queue at real time, so emitting
  hundreds of ms of stale audio in one tick overflows it, and real-time
  voice cannot use that audio anyway.
- Channel shape is adapted to the source's resolved device format
  (mono -> N duplication for the common Android mono-clip case, N -> mono
  averaging, first-channel fallback otherwise).
- A throwing AudioRead subscriber is contained (throttled warning): the
  loop is the sole frame producer, so an escaped exception would leave
  the microphone permanently silent.
- Stop/restart (including the iOS pause/resume path) invalidate the loop
  via a generation counter.

Tested on Meta Quest 3 (Unity 2022.3, Android, 24 kHz mono capture,
stereo source): multi-hour sessions with repeated network drops and SDK
reconnects show steady 10 ms frame production (~250 packets per 5 s via
OutboundRtp counters) with no silence or drift, where the filter-based
path previously degraded. Editor (Windows, 48 kHz) verified against a
remote peer for parity.
@CLAassistant

CLAassistant commented Aug 27, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants