diff --git a/README.md b/README.md index 284dee4c..ff35979c 100644 --- a/README.md +++ b/README.md @@ -330,8 +330,6 @@ void TrackSubscribed(IRemoteTrack track, RemoteTrackPublication publication, Rem With Platform Audio, the audio input and output are managed by the native ADM of WebRTC. This unlocks echo cancellation, noise suppression, auto gain control and hardware processing if available. There are some known issues with Platform Audio, that we are working on resolving: -- On iOS, disposing of Platform Audio object stops Unity audio output -- On iOS and Unity 6, backgrounding the app breaks Platform Audio - On MacOS with bluetooth headset, unmuting can break audio output #### Initialize Platform Audio @@ -417,6 +415,72 @@ IEnumerator PublishLocalMicrophonePlatform(PlatformAudio platformAudio, Room roo Using Platform Audio, for audio output of subscribed remote audio tracks you don't need any Unity handling. +#### Audio Output Routing + +On mobile, the OS decides where call audio plays (Bluetooth headset, wired headset, loudspeaker, earpiece). `PlatformAudio` exposes a routing policy on top of that: + +```cs +// Automatic policy: route to the best available output kind, most preferred first. +// The default ranking is Bluetooth > WiredHeadset > Speaker > Earpiece. +platformAudio.OutputPreference = new[] { AudioOutputKind.Bluetooth, AudioOutputKind.WiredHeadset, AudioOutputKind.Speaker }; + +// Convenience toggle for the built-in outputs: reorders Speaker/Earpiece inside +// OutputPreference (the list is the single source of truth, there is no separate state). +platformAudio.IsSpeakerOutputPreferred = false; // prefer the earpiece + +// Sticky override: audio stays routed to the device until the override is cleared +// or the device disappears (then the automatic policy resumes). +var (recording, playout) = platformAudio.GetDevices(); +platformAudio.SelectOutput(playout[0]); +platformAudio.ClearOutputOverride(); + +// Observability: raised on the Unity main thread whenever the available devices or +// the active route change. AudioDevice.Kind and AudioDevice.IsSelected tell you what +// each entry is and which one is playing. +platformAudio.DevicesChanged += (playoutDevices, recordingDevices) => { /* refresh your device UI */ }; +``` + +##### The call audio session + +Routing is only asserted while a call is in progress. `SetSessionAudioEnabled` is that switch, and it starts out enabled, so an app that creates `PlatformAudio` once at startup — the usual pattern, to keep a single ADM alive across calls — should hand the session back until it is needed: + +```cs +var platformAudio = new PlatformAudio(); +platformAudio.SetSessionAudioEnabled(false); // no call yet + +// ... a call starts: +platformAudio.SetSessionAudioEnabled(true); +yield return platformAudio.StartRecording(); + +// ... the call ends: +platformAudio.StopRecording(); +platformAudio.SetSessionAudioEnabled(false); +``` + +While disabled, the SDK holds no call audio session: on iOS WebRTC's voice-processing unit is off and the session sits in a music-friendly idle state, and on Android 12+ the SDK requests neither `MODE_IN_COMMUNICATION` nor the output route pin, so the platform's normal routing applies. On Android the session is also acquired lazily: constructing `PlatformAudio` issues no audio-mode traffic even though session audio starts enabled — the first action that needs the session while it is enabled takes it (enabling it explicitly, changing the output preference or selection, or starting capture) — so the disable-right-after-creation pattern above is completely silent at startup. Device enumeration and `DevicesChanged` keep working on both platforms, so a device picker can be populated before the first call. Leaving session audio enabled outside a call asserts the call session (on Android: from the first routing action on) for as long as the instance lives, which is rarely what an app wants: the OS treats the app as being in a call, and the route stays pinned to the call policy instead of following the platform. + +Unity's own audio engine is a separate layer that the SDK does not touch, and it needs a little care from an app that plays its own audio (music, SFX) alongside calls. When an output device is added or removed, Unity reinitializes its engine, which **stops every `AudioSource`** — and it raises `AudioSettings.OnAudioConfigurationChanged` only afterwards, so by the time the app is notified there is nothing left playing to inspect. What should still be audible therefore has to be remembered from before the change and restarted in that callback. On Android the callback's `deviceWasChanged` argument is `false` even for a real device change, so it cannot be used to filter these events. The Meet sample's `PlatformAudioController` shows the whole pattern. + +**Known limitation — the platform's Bluetooth SCO state can get stuck.** Android brings a Bluetooth headset's *call* link up asynchronously, and its SCO state machine can be left in a pending state that never resolves. While it is, the platform accepts `setCommunicationDevice` but never applies it (`AS.BtHelper: requestScoState: failed to connect in state 1`, `preferredCommunicationDevice: null`), so a call's audio — and any media the app plays alongside it — stays on the loudspeaker for the whole call and returns to the headset when the call ends. It is platform state, not app state: it survives the app being restarted, and the SDK cannot clear it (the outstanding request belongs to another client in the process). The SDK logs a warning naming this and retries with backoff. + +Two things are known to provoke or reveal it, device-verified on a Pixel 8a (Android 16): + +- Unity's audio engine claims the call link itself through the deprecated `AudioManager.startBluetoothSco()` when it initializes with a headset already connected — about 3 s before this SDK creates its ADM, and not triggered by anything in the SDK or the samples. Present in 2022.3 and Unity 6 alike; neither version uses the Android 12 communication-device API, which is why the two collide. +- Once stuck, only the platform clears it: disconnecting and reconnecting the headset (which triggers the platform's own `resetBluetoothSco`), toggling Bluetooth, or restarting the phone. After that, routing works normally — the call link comes up in well under a second. + +The reliable workaround is to connect the headset *after* the app has started, or to reconnect it once if a call has landed on the loudspeaker. + +Do **not** call `AudioSettings.Reset` as part of that recovery on Android. Unity has already reopened its output by the time it notifies you, so a reset adds nothing — and reinitializing the engine makes Unity claim a Bluetooth headset's call link through the deprecated `AudioManager.startBluetoothSco()`, which evicts the `setCommunicationDevice` route pin the SDK holds and can leave the platform's SCO state machine unable to connect at all (`AS.BtHelper: requestScoState: failed to connect in state 1` on every subsequent attempt). Call audio and game audio then both stay on the loudspeaker for the rest of the session, no matter how often the route is re-pinned. Restarting the app's own `AudioSource`s is enough and stays out of the platform's way. + +One consequence to design around on Android: while a call session is active on a classic (BR/EDR) Bluetooth headset, the platform suspends the headset's A2DP media link and routes *all* output — the app's own media included — over the headset's call link. Observed on a Pixel 8a (Android 16) with `adb shell dumpsys audio`: `STREAM_MUSIC` moves to `bt_sco_hs` while the call is active and back to `bt_a2dp` afterwards. Game audio therefore keeps playing during a call, but at the call link's quality, and it returns to full quality when the session is disabled — one more reason to hold the session only for the duration of a call. This is a platform property of classic Bluetooth, not something the routing API can override. + +Per-platform behavior: + +- **Android 12+ (API 31)**: the full `OutputPreference` ranking applies — the SDK routes to the highest-ranked available kind and re-routes on device changes; kinds missing from the list are never auto-selected (when nothing ranked is available, the OS default route applies). `SelectOutput` pins a device from `GetDevices().Playout` as the communication device; the pin is dropped once that device disappears. While session audio is disabled, `SelectOutput` only records the choice — it is applied when the session is next enabled, and until then `GetDevices`/`DevicesChanged` keep reporting the platform's own route. There is deliberately no pending flag for that deferral: a pre-call device picker should treat its own last `SelectOutput` call as the pending choice and confirm application via the `IsSelected` flip in `GetDevices`/`DevicesChanged` once the session is enabled; a deferred choice whose device disappears first is dropped for good (same rule as an active pin), observable as the device leaving the playout list. `DevicesChanged` is raised on communication-device changes; changes that fire no OS event are caught by a poll with roughly 1.5 s of latency. Requires the `MODIFY_AUDIO_SETTINGS` permission in your `AndroidManifest.xml`. Routing is asserted only while session audio is enabled: the session is first taken by the first trigger that needs it while enabled — an explicit enable, an output preference/selection change, or capture starting; never by construction alone — the SDK then holds `MODE_IN_COMMUNICATION` with the route pinned, and clears the pin and restores the mode it replaced on disable, while enumeration and `DevicesChanged` stay live either way. Note: since Android 13 the OS only honors the app's communication-mode request — and with it the route pin — while the app has an active voice-communication capture, so keep the mic capture running for the whole call, even while muted with the track unpublished (see `PlatformAudioController` in the Meet sample); an active capture without an enabled session hands routing back to the platform, so pair the two at the call boundaries. +- **Older Android**: no routing backend — `OutputPreference` is stored and round-trips but has no routing effect, and `SelectOutput` throws `NotSupportedException`. `DevicesChanged` is never raised. +- **iOS**: external devices (Bluetooth, wired) always take priority over the built-in outputs, so the Speaker/Earpiece relative order — `IsSpeakerOutputPreferred` — is the only part of the ranking with an effect. It decides where audio goes when no external device is connected, is applied through the audio session mode (never by overriding the output port), and takes effect immediately, including mid-call. `SelectOutput` throws `NotSupportedException` — the OS owns route selection on iOS; present the system route picker (`AVRoutePickerView`) instead. `GetDevices().Playout` is the audio session's current output route (iOS does not enumerate every reachable device), and `DevicesChanged` is raised when that route changes. +- **Desktop (Windows/macOS/Linux)**: output is selected per device — `SelectOutput` selects the playout device like `SetPlayoutDevice`, and the `OutputPreference` ranking has no routing effect. `DevicesChanged` is never raised (no hot-plug events yet). + ### RPC Perform your own predefined method calls from one participant to another. diff --git a/Runtime/Plugins/iOS/LiveKitAudioSession.mm b/Runtime/Plugins/iOS/LiveKitAudioSession.mm index f341fc56..958fb3a7 100644 --- a/Runtime/Plugins/iOS/LiveKitAudioSession.mm +++ b/Runtime/Plugins/iOS/LiveKitAudioSession.mm @@ -15,55 +15,590 @@ */ #import +#import + +#include +#include + +// This plugin coordinates the single shared AVAudioSession with WebRTC's iOS +// Audio Device Module (ADM). WebRTC ships an RTCAudioSession proxy that, left in +// its default "automatic" mode, reconfigures the category/route and *deactivates* +// the session whenever a call's playout/recording starts and stops. That fights +// with Unity/FMOD: on join the app's other audio (e.g. ambient music) is rerouted +// to the earpiece and attenuated, and on hang-up the session is deactivated out +// from under Unity so its audio dies. +// +// To make playout stable and keep Unity audio alive across call state, we put +// RTCAudioSession into MANUAL mode and have the app own the session: +// * We hold exactly one permanent activation (setActive:YES). Because +// RTCAudioSession ref-counts activation, WebRTC's per-call setActive:YES/NO +// only cycles the count and never actually deactivates the hardware session. +// * The category/mode/options are derived from a session STATE machine driven +// from C# (PlatformAudio knows the call's recording state) plus a +// speaker-vs-earpiece preference (see the table below). Every apply is also +// mirrored into WebRTC's RTCAudioSessionConfiguration snapshot so the ADM +// re-applies the same config on its own restarts. +// * The speaker preference is expressed via the session MODE only (VideoChat +// routes to the loudspeaker by default, VoiceChat to the receiver), never via +// overrideOutputAudioPort, so connected wired/Bluetooth devices always win. +// * The VPIO voice-processing unit (hardware AEC/AGC/NS) is gated by +// isAudioEnabled. It defaults to YES so call audio works out of the box (the +// unit still only initializes once a call actually has an audio track, so +// pre-call audio is unaffected). Callers can toggle it via +// LiveKit_SetAudioEnabled -- e.g. OFF on hang-up so the unit stops between +// calls while our held activation keeps the session alive for Unity. +// * Backgrounding interrupts the session and WebRTC stops its audio unit. On +// foreground, RTCAudioSession's own recovery restarts the unit exactly once, +// with no retry -- and Unity/FMOD restarts *its* audio around the same moment, +// reconfiguring the shared session (observed with Unity 6). Whoever loses that +// race stays broken, so we observe foreground/interruption-end ourselves and, +// after Unity's restart has settled, re-assert the current state's config and +// cycle isAudioEnabled to force a clean rebuild of the audio unit. +// * Route changes (headset plug/unplug, Bluetooth connect, mode switches) are +// observed via AVAudioSessionRouteChangeNotification and forwarded to C# +// through a registered callback so the SDK can raise its DevicesChanged event. +// +// Session state table (state is set from C# via LiveKit_SetSessionState): +// +// state category mode options +// 0 idle PlayAndRecord Default BT | A2DP | MixWithOthers +// | DefaultToSpeaker* +// 1 playout-only PlayAndRecord Default same as idle +// 2 recording PlayAndRecord VideoChat (speaker) / BT | A2DP +// VoiceChat (earpiece) +// +// *DefaultToSpeaker only while the speaker is preferred. In the recording state +// the speaker preference is carried by the mode alone. Idle and playout-only +// share a config: PlayAndRecord stays because the ADM initializes its VPIO unit +// with input disabled for playout-only (InitPlayOrRecord(false)) but nothing +// guarantees VPIO under the Playback category; mode Default + MixWithOthers is +// the music-friendliest config the ADM demonstrably supports. The states stay +// distinct so the mapping can diverge without touching the C# driver. +// +// RTCAudioSession lives inside the statically-linked liblivekit_ffi; we reach it +// dynamically via NSClassFromString + a protocol-typed id so this file never +// creates a link-time dependency on the class. If the class can't be found we +// fall back to configuring AVAudioSession directly (legacy behavior). + +/// Minimal subset of WebRTC's RTCAudioSession that we message dynamically. +@protocol LiveKitRTCAudioSession +@property(nonatomic, assign) BOOL useManualAudio; +@property(nonatomic, assign) BOOL isAudioEnabled; +@property(nonatomic, readonly) int activationCount; +- (void)lockForConfiguration; +- (void)unlockForConfiguration; +- (BOOL)setActive:(BOOL)active error:(NSError**)outError; +- (BOOL)setCategory:(AVAudioSessionCategory)category + mode:(AVAudioSessionMode)mode + options:(AVAudioSessionCategoryOptions)options + error:(NSError**)outError; +@end + +/// Minimal subset of WebRTC's RTCAudioSessionConfiguration (the snapshot the ADM +/// re-applies on its own restarts), messaged dynamically like RTCAudioSession. +@protocol LiveKitRTCAudioSessionConfiguration +@property(nonatomic, strong) NSString* category; +@property(nonatomic, assign) AVAudioSessionCategoryOptions categoryOptions; +@property(nonatomic, strong) NSString* mode; +@end + +/// Session states, mirroring PlatformAudio's driver in C#. Do not renumber. +enum { + kLiveKitSessionStateIdle = 0, + kLiveKitSessionStatePlayoutOnly = 1, + kLiveKitSessionStateRecording = 2, +}; + +typedef void (*LiveKitRouteChangeCallback)(void); + +// Tracks whether *we* currently hold the one app-owned activation, so we add and +// release it exactly once regardless of how many times configure/restore run. +static BOOL s_liveKitHoldsActivation = NO; + +// Snapshot of the AVAudioSession configuration as it was the first time LiveKit +// touched the session (i.e. whatever Unity set up from its iOS Player Settings). +// Captured lazily in LiveKit_ConfigureAudioSessionForVoIP and re-applied by +// LiveKit_RestoreDefaultAudioSession when the last PlatformAudio is disposed. +static BOOL s_hasCachedState = NO; +static NSString* s_cachedCategory = nil; +static NSString* s_cachedMode = nil; +static AVAudioSessionCategoryOptions s_cachedCategoryOptions = 0; + +// YES between configure and restore: gates the foreground-recovery observers so +// they no-op once LiveKit has handed the session back to the app. +static BOOL s_liveKitConfigured = NO; +// The isAudioEnabled state the caller wants (updated by LiveKit_SetAudioEnabled), +// so recovery knows whether to restart the audio unit after re-asserting config. +static BOOL s_audioDesired = NO; +// Coalesces recovery requests (didBecomeActive and interruption-ended both fire +// on foreground) into one delayed pass. +static BOOL s_recoveryPending = NO; + +// The state machine inputs (see the table above). The defaults match what a fresh +// PlatformAudio pushes right after construction, so the config applied by +// configure is already the one the C# driver expects. +static int s_sessionState = kLiveKitSessionStatePlayoutOnly; +static BOOL s_speakerPreferred = YES; + +// Invoked (on the main queue) whenever the audio route changes, so the C# side +// can re-query the route and raise DevicesChanged. +static LiveKitRouteChangeCallback s_routeChangeCallback = NULL; + +// AllowBluetooth was renamed AllowBluetoothHFP in the iOS 26 SDK; same guard the +// WebRTC fork uses. The speaker preference never rides on these options. +#if defined(__IPHONE_26_0) && __IPHONE_OS_VERSION_MAX_ALLOWED >= __IPHONE_26_0 +static const AVAudioSessionCategoryOptions kLiveKitBluetoothOptions = + AVAudioSessionCategoryOptionAllowBluetoothHFP | + AVAudioSessionCategoryOptionAllowBluetoothA2DP; +#else +static const AVAudioSessionCategoryOptions kLiveKitBluetoothOptions = + AVAudioSessionCategoryOptionAllowBluetooth | + AVAudioSessionCategoryOptionAllowBluetoothA2DP; +#endif + +/// Returns WebRTC's shared RTCAudioSession if it's present in the linked binary, +/// or nil if the class can't be found (in which case callers use AVAudioSession). +static id LiveKit_RTCSession() { + Class cls = NSClassFromString(@"RTCAudioSession"); + if (!cls || ![cls respondsToSelector:@selector(sharedInstance)]) { + return nil; + } +#pragma clang diagnostic push +#pragma clang diagnostic ignored "-Warc-performSelector-leaks" + return (id)[cls performSelector:@selector(sharedInstance)]; +#pragma clang diagnostic pop +} + +static NSString* LiveKit_DesiredMode() { + if (s_sessionState == kLiveKitSessionStateRecording) { + return s_speakerPreferred ? AVAudioSessionModeVideoChat + : AVAudioSessionModeVoiceChat; + } + return AVAudioSessionModeDefault; +} + +static AVAudioSessionCategoryOptions LiveKit_DesiredOptions() { + AVAudioSessionCategoryOptions options = kLiveKitBluetoothOptions; + if (s_sessionState != kLiveKitSessionStateRecording) { + options |= AVAudioSessionCategoryOptionMixWithOthers; + // Mode Default routes PlayAndRecord to the receiver; outside a call there + // is no mode that both prefers the speaker and leaves music processing + // alone, so here -- and only here -- the preference rides on an option. + if (s_speakerPreferred) { + options |= AVAudioSessionCategoryOptionDefaultToSpeaker; + } + } + return options; +} + +/// Mirrors our category/mode/options into WebRTC's RTCAudioSessionConfiguration +/// snapshot so the ADM re-applies the same config whenever it (re)configures the +/// session itself (audio unit init, interruption recovery). +static void LiveKit_MirrorWebRTCConfiguration(NSString* mode, + AVAudioSessionCategoryOptions options) { + Class cls = NSClassFromString(@"RTCAudioSessionConfiguration"); + if (!cls || ![cls respondsToSelector:@selector(webRTCConfiguration)] || + ![cls respondsToSelector:@selector(setWebRTCConfiguration:)]) { + return; + } +#pragma clang diagnostic push +#pragma clang diagnostic ignored "-Warc-performSelector-leaks" + id config = + (id)[cls performSelector:@selector(webRTCConfiguration)]; + if (config == nil) { + return; + } + config.category = AVAudioSessionCategoryPlayAndRecord; + config.mode = mode; + config.categoryOptions = options; + [cls performSelector:@selector(setWebRTCConfiguration:) withObject:config]; +#pragma clang diagnostic pop +} + +/// Applies the config derived from (s_sessionState, s_speakerPreferred) to the +/// session and mirrors it into the WebRTC snapshot. Logs expected vs. actual so +/// device tests can see who won when something else reconfigures the session. +/// +/// rebuildAudioUnitOnModeChange: the ADM does not rebuild its VPIO unit on a +/// route change that keeps the hardware sample rate (HandleValidRouteChange -> +/// HandleSampleRateChange no-ops when the audio parameters are intact), so a +/// live mode switch leaves the unit calibrated for the previous route -- +/// device-observed as an attenuated loudspeaker after an earpiece -> speaker +/// toggle. Passing YES cycles isAudioEnabled after a mode change to force a +/// clean rebuild against the new route, at the cost of a brief audio gap. +/// Callers that handle the rebuild themselves (foreground recovery) or run +/// before the unit exists (configure) pass NO. +static void LiveKit_ApplySessionConfig(NSString* reason, BOOL rebuildAudioUnitOnModeChange) { + NSString* mode = LiveKit_DesiredMode(); + AVAudioSessionCategoryOptions options = LiveKit_DesiredOptions(); + BOOL modeChanged = ![[AVAudioSession sharedInstance].mode isEqualToString:mode]; + + id rtc = LiveKit_RTCSession(); + NSError* error = nil; + if (rtc != nil) { + [rtc lockForConfiguration]; + if (![rtc setCategory:AVAudioSessionCategoryPlayAndRecord + mode:mode + options:options + error:&error] || error) { + NSLog(@"LiveKit: failed to apply session config (%@): %@", + reason, error.localizedDescription); + } + [rtc unlockForConfiguration]; + } else { + AVAudioSession* session = [AVAudioSession sharedInstance]; + if (![session setCategory:AVAudioSessionCategoryPlayAndRecord + mode:mode + options:options + error:&error] || error) { + NSLog(@"LiveKit: failed to apply session config (%@): %@", + reason, error.localizedDescription); + } + } + + LiveKit_MirrorWebRTCConfiguration(mode, options); + + AVAudioSession* current = [AVAudioSession sharedInstance]; + NSLog(@"LiveKit: session config (%@): state=%d speakerPreferred=%d expected mode=%@ options=%lu" + " -> actual category=%@ mode=%@ options=%lu", + reason, s_sessionState, s_speakerPreferred, mode, (unsigned long)options, + current.category, current.mode, (unsigned long)current.categoryOptions); + + if (rebuildAudioUnitOnModeChange && modeChanged && s_audioDesired && rtc != nil) { + rtc.isAudioEnabled = NO; + rtc.isAudioEnabled = YES; + NSLog(@"LiveKit: cycled isAudioEnabled to rebuild the audio unit after mode change (%@)", + reason); + } +} + +/// Re-applies the current state's config and reactivates the session, then +/// cycles isAudioEnabled to force WebRTC to rebuild its VPIO audio unit. Runs on +/// a delay so it lands after Unity/FMOD's own foreground audio restart (which is +/// itself delayed and can reconfigure the shared session underneath WebRTC's +/// one-shot, no-retry interruption recovery -- the Unity 6 focus race). +static void LiveKit_ScheduleSessionRecovery() { + if (!s_liveKitConfigured || s_recoveryPending) { + return; + } + s_recoveryPending = YES; + dispatch_after(dispatch_time(DISPATCH_TIME_NOW, (int64_t)(0.5 * NSEC_PER_SEC)), + dispatch_get_main_queue(), ^{ + s_recoveryPending = NO; + if (!s_liveKitConfigured) { + return; // restored/disposed while the recovery was pending + } + + AVAudioSession* session = [AVAudioSession sharedInstance]; + // "Who won" the focus race: what Unity/FMOD left the session as. + NSLog(@"LiveKit: foreground recovery; session before re-assert: category=%@ mode=%@ options=%lu", + session.category, session.mode, (unsigned long)session.categoryOptions); + + LiveKit_ApplySessionConfig(@"foreground recovery", NO); + + // Reactivate directly on AVAudioSession: the OS deactivated the hardware + // session during the interruption, but RTCAudioSession's activation + // ref-count still includes our held activation, so reactivating through + // the proxy would double-count it. + NSError* error = nil; + if (![session setActive:YES error:&error] || error) { + NSLog(@"LiveKit: recovery failed to reactivate session: %@", error.localizedDescription); + } + + // Cycle isAudioEnabled to rebuild the VPIO unit against the re-asserted + // session (WebRTC's own foreground restart may have failed, or been undone + // by Unity's). Harmless when no call audio is active: with playout and + // recording uninitialized WebRTC ignores the change. + id rtc = LiveKit_RTCSession(); + if (rtc != nil && s_audioDesired) { + rtc.isAudioEnabled = NO; + rtc.isAudioEnabled = YES; + } + + NSLog(@"LiveKit: foreground recovery done (audioDesired=%d, activationCount=%d)", + s_audioDesired, rtc != nil ? rtc.activationCount : -1); + }); +} + +/// Registers app-lifetime observers for foreground/interruption recovery and for +/// route-change forwarding. Registered once on first configure; the handlers +/// no-op while LiveKit is not configured. +static void LiveKit_RegisterLifecycleObserversIfNeeded() { + static BOOL s_observersRegistered = NO; + if (s_observersRegistered) { + return; + } + s_observersRegistered = YES; + + NSNotificationCenter* center = [NSNotificationCenter defaultCenter]; + [center addObserverForName:UIApplicationDidBecomeActiveNotification + object:nil + queue:[NSOperationQueue mainQueue] + usingBlock:^(NSNotification* note) { + LiveKit_ScheduleSessionRecovery(); + }]; + [center addObserverForName:AVAudioSessionInterruptionNotification + object:nil + queue:[NSOperationQueue mainQueue] + usingBlock:^(NSNotification* note) { + NSNumber* type = note.userInfo[AVAudioSessionInterruptionTypeKey]; + if (type.unsignedIntegerValue == AVAudioSessionInterruptionTypeEnded) { + LiveKit_ScheduleSessionRecovery(); + } + }]; + [center addObserverForName:AVAudioSessionRouteChangeNotification + object:nil + queue:[NSOperationQueue mainQueue] + usingBlock:^(NSNotification* note) { + if (!s_liveKitConfigured) { + return; + } + NSNumber* reason = note.userInfo[AVAudioSessionRouteChangeReasonKey]; + NSMutableArray* outputs = [NSMutableArray array]; + for (AVAudioSessionPortDescription* port in + [AVAudioSession sharedInstance].currentRoute.outputs) { + [outputs addObject:[NSString stringWithFormat:@"%@ (%@)", port.portName, port.portType]]; + } + NSLog(@"LiveKit: route changed (reason=%lu) outputs=%@", + (unsigned long)reason.unsignedIntegerValue, + [outputs componentsJoinedByString:@", "]); + LiveKitRouteChangeCallback callback = s_routeChangeCallback; + if (callback != NULL) { + callback(); + } + }]; +} + +/// Captures the current audio session category/mode/options exactly once, before +/// LiveKit reconfigures the session for VoIP. Subsequent calls are no-ops so the +/// snapshot always reflects the pristine, pre-LiveKit (Unity-configured) state. +static void LiveKit_CacheSessionStateIfNeeded() { + if (s_hasCachedState) { + return; + } + AVAudioSession* session = [AVAudioSession sharedInstance]; + // copy so the strings persist for the app lifetime regardless of ARC/MRC. + s_cachedCategory = [session.category copy]; + s_cachedMode = [session.mode copy]; + s_cachedCategoryOptions = session.categoryOptions; + s_hasCachedState = YES; + NSLog(@"LiveKit: cached audio session state: category=%@, mode=%@, options=%lu", + s_cachedCategory, s_cachedMode, (unsigned long)s_cachedCategoryOptions); +} + +/// Maps an AVAudioSessionPort type to the C# AudioOutputKind numbering +/// (Unknown=0, Earpiece=1, Speaker=2, WiredHeadset=3, Bluetooth=4, Usb=5, +/// HearingAid=6). Do not renumber. AVAudioSession has no dedicated hearing-aid +/// port type, so 6 is never produced here; AirPlay/HDMI/CarAudio and other +/// unroutable-by-us ports map to Unknown. +static int LiveKit_OutputKindForPortType(NSString* portType) { + if ([portType isEqualToString:AVAudioSessionPortBuiltInReceiver]) return 1; + if ([portType isEqualToString:AVAudioSessionPortBuiltInSpeaker]) return 2; + if ([portType isEqualToString:AVAudioSessionPortHeadphones]) return 3; + if ([portType isEqualToString:AVAudioSessionPortBluetoothA2DP] || + [portType isEqualToString:AVAudioSessionPortBluetoothHFP] || + [portType isEqualToString:AVAudioSessionPortBluetoothLE]) return 4; + if ([portType isEqualToString:AVAudioSessionPortUSBAudio]) return 5; + return 0; +} extern "C" { -/// Configures the iOS audio session for VoIP/WebRTC use. -/// This sets AVAudioSessionCategoryPlayAndRecord with VoiceChat mode, -/// which enables the VPIO (Voice Processing IO) AudioUnit for: -/// - Hardware echo cancellation (AEC) -/// - Automatic gain control (AGC) -/// - Noise suppression (NS) +/// Configures the iOS audio session for VoIP/WebRTC use and takes app ownership +/// of the shared AVAudioSession. +/// +/// This applies the config for the current session state (playout-only for a +/// fresh PlatformAudio; see the state table at the top of this file), puts +/// RTCAudioSession into manual mode, and holds a single permanent activation so +/// WebRTC never deactivates the session on its own. /// -/// Call this before creating PlatformAudio to ensure WebRTC can -/// properly initialize the microphone and speaker. +/// Call this before creating PlatformAudio. Call audio is enabled by default, so +/// no further call is required for it to work; use LiveKit_SetAudioEnabled(false) +/// to stop the VPIO unit between calls (e.g. on hang-up). void LiveKit_ConfigureAudioSessionForVoIP() { - AVAudioSession* session = [AVAudioSession sharedInstance]; - NSError* error = nil; + // Snapshot the pristine (Unity Player Settings) session before we change it. + LiveKit_CacheSessionStateIfNeeded(); + + LiveKit_RegisterLifecycleObserversIfNeeded(); + s_liveKitConfigured = YES; + s_audioDesired = YES; // mirrors the isAudioEnabled default set below + + id rtc = LiveKit_RTCSession(); + + // Manual mode: WebRTC won't activate/deactivate the session on its own, and + // won't initialize the VPIO unit until we grant permission via isAudioEnabled + // (set below). This is what lets us own activation and gate the unit. Set + // before the first apply so the ADM never races the initial configuration. + if (rtc != nil) { + rtc.useManualAudio = YES; + } - // Configure for VoIP with echo cancellation - BOOL success = [session setCategory:AVAudioSessionCategoryPlayAndRecord - mode:AVAudioSessionModeVoiceChat - options:AVAudioSessionCategoryOptionDefaultToSpeaker | - AVAudioSessionCategoryOptionAllowBluetooth | - AVAudioSessionCategoryOptionAllowBluetoothA2DP - error:&error]; + LiveKit_ApplySessionConfig(@"configure", NO); - if (!success || error) { - NSLog(@"LiveKit: Failed to configure VoIP audio session: %@", error.localizedDescription); + if (rtc == nil) { + // RTCAudioSession unavailable: activate AVAudioSession directly (legacy). + AVAudioSession* session = [AVAudioSession sharedInstance]; + NSError* error = nil; + if (![session setActive:YES error:&error] || error) { + NSLog(@"LiveKit: Failed to activate audio session: %@", error.localizedDescription); + return; + } + NSLog(@"LiveKit: Audio session configured (AVAudioSession fallback)"); return; } - // Activate the audio session - success = [session setActive:YES error:&error]; - if (!success || error) { - NSLog(@"LiveKit: Failed to activate audio session: %@", error.localizedDescription); + // Hold exactly one app-owned activation. RTCAudioSession ref-counts activation, + // so WebRTC's balanced setActive:YES/NO during a call never drops the real + // session below active while we hold this. + if (!s_liveKitHoldsActivation) { + [rtc lockForConfiguration]; + NSError* error = nil; + if ([rtc setActive:YES error:&error] && !error) { + s_liveKitHoldsActivation = YES; + } else { + NSLog(@"LiveKit: Failed to activate audio session: %@", error.localizedDescription); + } + [rtc unlockForConfiguration]; + } + + // Grant WebRTC permission to initialize its audio unit by default so call audio + // works without an explicit LiveKit_SetAudioEnabled(true). The unit is only + // actually created once a call has an audio track, so pre-call audio is + // unaffected. Callers may still disable it (e.g. on hang-up) via + // LiveKit_SetAudioEnabled(false). + rtc.isAudioEnabled = YES; + + NSLog(@"LiveKit: Audio session configured for VoIP (manual mode, activationCount=%d)", + rtc.activationCount); +} + +/// Enables or disables WebRTC's VPIO audio unit while the app keeps ownership of +/// the session. Pass true when a call connects and false when it ends. +/// +/// This is only effective in manual mode (set up by LiveKit_ConfigureAudioSessionForVoIP). +/// Disabling on hang-up stops incoming/outgoing call audio and the VPIO processing, +/// but leaves the session active (via the app's held activation), so Unity audio +/// keeps playing. +void LiveKit_SetAudioEnabled(bool enabled) { + s_audioDesired = enabled ? YES : NO; + id rtc = LiveKit_RTCSession(); + if (rtc == nil) { return; } + rtc.isAudioEnabled = enabled ? YES : NO; + NSLog(@"LiveKit: isAudioEnabled=%@ (activationCount=%d)", enabled ? @"YES" : @"NO", rtc.activationCount); +} + +/// Sets whether the loudspeaker is preferred over the earpiece for the built-in +/// outputs and live-applies the resulting config (see the state table). External +/// devices (wired, Bluetooth) always take priority over both; this only decides +/// where audio goes when no external device is connected. +void LiveKit_SetSpeakerPreferred(bool preferred) { + BOOL value = preferred ? YES : NO; + if (s_speakerPreferred == value) { + return; + } + s_speakerPreferred = value; + if (s_liveKitConfigured) { + LiveKit_ApplySessionConfig(@"speaker preference", YES); + } +} - NSLog(@"LiveKit: Audio session configured for VoIP (PlayAndRecord + VoiceChat mode)"); +/// Sets the session state (0 idle, 1 playout-only, 2 recording; see the state +/// table) and live-applies the resulting config. Driven from C#: PlatformAudio +/// knows whether recording is active and whether call audio is wanted. +void LiveKit_SetSessionState(int state) { + if (state < kLiveKitSessionStateIdle || state > kLiveKitSessionStateRecording) { + NSLog(@"LiveKit: ignoring unknown session state %d", state); + return; + } + if (s_sessionState == state) { + return; + } + s_sessionState = state; + if (s_liveKitConfigured) { + LiveKit_ApplySessionConfig(@"session state", YES); + } } -/// Restores the audio session to the default ambient category. -/// Call this when PlatformAudio is disposed if you want to restore -/// the original audio behavior. +/// Registers (or clears, with NULL) the callback invoked on the main queue +/// whenever the audio route changes. The callback carries no payload; the C# +/// side re-queries LiveKit_GetCurrentOutputRoutes. +void LiveKit_SetRouteChangeCallback(LiveKitRouteChangeCallback callback) { + s_routeChangeCallback = callback; +} + +/// Returns the current output route as newline-separated "kind\tname\tuid" +/// entries (kind per LiveKit_OutputKindForPortType). The caller must release the +/// returned buffer with LiveKit_FreeRouteString. +char* LiveKit_GetCurrentOutputRoutes() { + NSMutableString* result = [NSMutableString string]; + for (AVAudioSessionPortDescription* port in + [AVAudioSession sharedInstance].currentRoute.outputs) { + [result appendFormat:@"%d\t%@\t%@\n", + LiveKit_OutputKindForPortType(port.portType), + port.portName ?: @"", + port.UID ?: @""]; + } + return strdup(result.UTF8String); +} + +/// Frees a buffer returned by LiveKit_GetCurrentOutputRoutes. +void LiveKit_FreeRouteString(char* str) { + free(str); +} + +/// Restores the audio session Unity had before LiveKit touched it (or the ambient +/// category if LiveKit never configured it), relinquishes the app-owned activation +/// and manual mode, and reactivates the session so Unity audio output resumes. +/// Call this when the last PlatformAudio is disposed. void LiveKit_RestoreDefaultAudioSession() { + // Stand down the foreground-recovery observers before touching the session. + s_liveKitConfigured = NO; + s_audioDesired = NO; + // Reset the state machine to the defaults a fresh PlatformAudio expects, so a + // later reconfigure starts from the same config it will be driven to. + s_sessionState = kLiveKitSessionStatePlayoutOnly; + s_speakerPreferred = YES; + + id rtc = LiveKit_RTCSession(); + + if (rtc != nil) { + // Stop the VPIO unit and release our activation before handing control back. + rtc.isAudioEnabled = NO; + if (s_liveKitHoldsActivation) { + NSError* error = nil; + if (![rtc setActive:NO error:&error] || error) { + NSLog(@"LiveKit: Failed to deactivate audio session: %@", error.localizedDescription); + } + s_liveKitHoldsActivation = NO; + } + rtc.useManualAudio = NO; + } + AVAudioSession* session = [AVAudioSession sharedInstance]; NSError* error = nil; + if (s_hasCachedState) { + if (![session setCategory:s_cachedCategory + mode:s_cachedMode + options:s_cachedCategoryOptions + error:&error] || error) { + NSLog(@"LiveKit: Failed to restore cached audio session (category=%@, mode=%@): %@", + s_cachedCategory, s_cachedMode, error.localizedDescription); + } + } else { + // Configure was never called, so we have nothing to restore to; fall back + // to the ambient category. + [session setCategory:AVAudioSessionCategoryAmbient error:&error]; + if (error) { + NSLog(@"LiveKit: Failed to restore default audio session: %@", error.localizedDescription); + } + } - [session setCategory:AVAudioSessionCategoryAmbient error:&error]; - if (error) { - NSLog(@"LiveKit: Failed to restore default audio session: %@", error.localizedDescription); + // Hand an active session back to Unity so its audio output resumes. + error = nil; + if (![session setActive:YES error:&error] || error) { + NSLog(@"LiveKit: Failed to reactivate audio session: %@", error.localizedDescription); } } diff --git a/Runtime/Scripts/Audio/AndroidRouteController.cs b/Runtime/Scripts/Audio/AndroidRouteController.cs new file mode 100644 index 00000000..ca763805 --- /dev/null +++ b/Runtime/Scripts/Audio/AndroidRouteController.cs @@ -0,0 +1,860 @@ +#if UNITY_ANDROID && !UNITY_EDITOR +using System; +using System.Collections.Generic; +using System.Globalization; +using System.Threading; +using LiveKit.Internal; +using UnityEngine; + +namespace LiveKit +{ + /// + /// Android routing backend for , built on the + /// communication-device APIs introduced in Android 12 (API 31): + /// AudioManager.getAvailableCommunicationDevices / + /// setCommunicationDevice / clearCommunicationDevice. + /// + /// The controller owns the voice-communication audio session while it holds it — + /// session audio enabled ( — i.e. a call is in + /// progress) and acquired by a first trigger, see the lazy-acquisition paragraph: + /// it enters MODE_IN_COMMUNICATION (saving and restoring the prior mode) and + /// keeps the output route pinned to the best device — the sticky + /// override while its device is still available, otherwise + /// the highest-ranked available kind per the current + /// . Owning the mode is what makes the + /// pin authoritative: without it the platform periodically reasserts its own default + /// route (observed on Pixel 8a: Telecom flipped playout back to the earpiece every + /// ~6 s after a Bluetooth session ended). Note that since Android 13 the mode request + /// is only honored while the app has active voice-communication capture, so + /// re-asserts the policy when capture + /// (re)starts — through , which like every other + /// re-evaluation path pins nothing while session audio is disabled. + /// + /// The session is acquired lazily: construction issues no setMode and no pin + /// even though session audio starts out enabled. The first trigger that needs the + /// session while it is enabled acquires it — an explicit + /// call with true, an + /// (which includes the + /// re-assert) or a + /// . Apps that disable session audio right after + /// construction therefore cause no audio-mode traffic at startup at all; the eager + /// constructor acquisition produced a take → pin → clear transient there, and with + /// a Bluetooth headset connected it started an asynchronous SCO activation only to + /// clear it mid-negotiation. The exposure is a receive-only app that never records, + /// never touches routing and never calls : it + /// no longer gets the mode and pin from construction, and opts back in by calling + /// with true at its call boundary. + /// + /// While the session is not held — session audio disabled, or enabled but nothing + /// has needed it yet — it belongs to the platform (communication device cleared, + /// prior mode restored on release), so the mode request and the route pin cover the + /// call rather than the lifetime of the instance. Enumeration, the change listener + /// and the poll thread stay alive regardless, so and + /// keep reporting the platform's own routing while + /// idle. + /// + /// Route changes are detected two ways, both required (device-verified in the + /// sample hotfix this backend is hardened from, PR #364): + /// - OnCommunicationDeviceChangedListener — fires when the OS changes or + /// clears the pin (e.g. the pinned device disconnected). + /// - A poll thread (every 1.5 s) — covers transitions that fire no event: a device + /// added while a pin is active, and the trace-verified teardown where a powered-off + /// Bluetooth headset stays in the available list up to ~10 s after the route + /// already fell back to the earpiece, then leaves the list without another + /// communication-device change. + /// + /// Threading: re-evaluation runs on whichever thread triggered it (Unity main, + /// the Android main executor, or the poll thread — all JVM-attached) behind one + /// lock. may therefore be raised from any of them; + /// marshals it to the Unity main thread. + /// + internal sealed class AndroidRouteController : IRouteController + { + // android.media.AudioManager / AudioAttributes constants. + private const int ModeInCommunication = 3; // AudioManager.MODE_IN_COMMUNICATION + private const int AudioFocusGain = 1; // AudioManager.AUDIOFOCUS_GAIN + private const int AudioFocusRequestGranted = 1; // AudioManager.AUDIOFOCUS_REQUEST_GRANTED + private const int UsageVoiceCommunication = 2; // AudioAttributes.USAGE_VOICE_COMMUNICATION + private const int ContentTypeSpeech = 1; // AudioAttributes.CONTENT_TYPE_SPEECH + + private const int MinSupportedApiLevel = 31; + private static readonly TimeSpan PollInterval = TimeSpan.FromSeconds(1.5); + // How long a pin is given to take effect before it is issued again. Selecting a + // Bluetooth device starts an asynchronous SCO negotiation, and until it completes + // the platform keeps reporting the previous communication device — so without this + // the poll re-issues the pin into its own pending activation, which the platform + // refuses ("BtHelper: requestScoState: failed to connect in state 1", device-verified + // on a Pixel 8a / Android 16) and the route never arrives at all. Real route changes + // come through the change listener, so this only slows down recovering from a pin the + // platform dropped silently. + private static readonly TimeSpan PinSettleTimeout = TimeSpan.FromSeconds(6); + // Ceiling for the backoff applied when the platform keeps taking the pin without + // acting on it — a state this SDK cannot clear (see the Bluetooth note in README). + private static readonly TimeSpan PinSettleTimeoutMax = TimeSpan.FromSeconds(30); + + private readonly PlatformAudio _owner; + private readonly object _gate = new object(); + private readonly ManualResetEventSlim _stopPoll = new ManualResetEventSlim(false); + private readonly List _recordingSnapshot; + private readonly Thread _pollThread; + + private List _ranked; + private int _stickyDeviceId = -1; + private int _pinnedDeviceId = -1; + // When the outstanding pin was last issued — Stopwatch ticks, monotonic, so a + // wall-clock step can neither cut the settle window short nor stretch it — to + // give it _pinSettleTimeout to take effect, and whether the platform has been + // seen honoring it since. + private long _pinIssuedAtTimestamp; + private bool _pinApplied; + private TimeSpan _pinSettleTimeout = PinSettleTimeout; + // Session audio starts enabled, matching the documented default of + // PlatformAudio.SetSessionAudioEnabled (uniform with iOS). + private bool _sessionAudioEnabled = true; + // Whether this controller currently holds the call session (mode entered, pin + // allowed). Never true while _sessionAudioEnabled is false. Acquisition is + // lazy: despite the enabled default, nothing is taken until the first trigger + // that needs the session — see AcquireSessionIfNeeded and the class doc. + private bool _sessionAcquired; + private int _savedAudioMode; + private bool _audioModeSaved; + // Set when an enter/leave transition failed (JNI unavailable, platform error) so + // Reevaluate retries it: without the retry, one transient failure on disable + // would leave the platform in MODE_IN_COMMUNICATION with the route pinned until + // the next call boundary, while this controller reports the session released. + private bool _sessionTransitionPending; + private CommunicationDeviceListener _listener; + private AndroidJavaObject _audioFocusRequest; + private bool _audioFocusEnabled; + private List<(int Id, AudioOutputKind Kind, bool IsSelected)> _lastSignature; + private bool _disposed; + + public event Action, IReadOnlyList> DevicesChanged; + + /// + /// Creates the Android backend, or an on + /// Android versions below 12 (API 31), which lack the communication-device APIs + /// this backend is built on. On those versions the routing verbs are documented + /// no-ops/throws, matching the gate the sample hotfix carried. + /// + internal static IRouteController Create(PlatformAudio owner, IReadOnlyList initialPreference) + { + int sdkInt; + try + { + sdkInt = AndroidSdkInt(); + } + catch (Exception e) + { + Utils.Warning($"AndroidRouteController: failed to read Build.VERSION.SDK_INT, routing disabled: {e.Message}"); + return new UnsupportedRouteController(owner, "this Android device"); + } + + if (sdkInt < MinSupportedApiLevel) + return new UnsupportedRouteController(owner, $"Android API {sdkInt} (routing requires API {MinSupportedApiLevel})"); + + return new AndroidRouteController(owner, initialPreference); + } + + private AndroidRouteController(PlatformAudio owner, IReadOnlyList initialPreference) + { + _owner = owner; + _ranked = new List(initialPreference); + + // The FFI exposes a single placeholder entry for the OS default input on + // Android; input routing follows the communication device, so this list is + // static and can back every DevicesChanged payload. Fetched before any + // session state is touched so a failure here has no side effects. + _recordingSnapshot = owner.GetDevicesViaFfi().Recording; + + // Session audio defaults to enabled, but the call session is NOT taken + // here: acquisition waits for the first trigger that needs it (see the + // class doc), and the prior audio mode is saved at that acquisition, where + // it reflects the state actually being replaced. This initial Reevaluate is + // therefore observation-only — it seeds the device signature and reports + // the platform's own route. + RegisterListener(); + Reevaluate(); + + _pollThread = new Thread(PollLoop) + { + IsBackground = true, + Name = "LiveKitAndroidRoutePoll", + }; + _pollThread.Start(); + } + + public (List Recording, List Playout) GetDevices() + { + var recording = _owner.GetDevicesViaFfi().Recording; + var playout = new List(); + try + { + using var audioManager = GetAudioManager(); + using var current = audioManager.Call("getCommunicationDevice"); + var currentId = current != null ? current.Call("getId") : -1; + + using var available = audioManager.Call("getAvailableCommunicationDevices"); + var count = available.Call("size"); + for (var i = 0; i < count; i++) + { + using var device = available.Call("get", i); + playout.Add(ToAudioDevice(device, (uint)i, currentId)); + } + } + catch (Exception e) + { + Utils.Warning($"AndroidRouteController: device enumeration failed: {e.Message}"); + } + return (recording, playout); + } + + public void ApplyOutputPreference(IReadOnlyList ranked) + { + lock (_gate) + { + _ranked = new List(ranked); + AcquireSessionIfNeeded(); + } + Reevaluate(); + } + + public void SelectOutput(AudioDevice device) + { + if (string.IsNullOrEmpty(device.Guid) + || !int.TryParse(device.Guid, NumberStyles.Integer, CultureInfo.InvariantCulture, out var id)) + throw new ArgumentException( + $"Device '{device.Name}' does not carry an Android device id; " + + "pass an entry from GetDevices().Playout", nameof(device)); + + lock (_gate) + { + _stickyDeviceId = id; + AcquireSessionIfNeeded(); + } + Reevaluate(); + } + + public void ClearOutputOverride() + { + lock (_gate) + { + if (_stickyDeviceId == -1) + return; + _stickyDeviceId = -1; + } + Reevaluate(); + } + + /// + /// Takes or hands back the call audio session: enabling enters + /// MODE_IN_COMMUNICATION and lets the policy pin the route, disabling + /// clears the pin and restores the mode this controller replaced. An explicit + /// enable acquires the session even when the state was already enabled — the + /// lazy default means "enabled but nothing has needed the session yet" is a real + /// state, and this call is the documented way for a receive-only app to take the + /// session at its call boundary. Disabling before anything acquired the session + /// releases nothing: there is nothing to release, and issuing a clear/restore + /// there would be exactly the startup transient lazy acquisition removes. + /// The ranked preference survives the transition unconditionally. The sticky + /// override survives it only while its device stays available: the + /// drop-on-disappear bookkeeping keeps running while the session is disabled, so + /// a device that leaves the list between calls (a headset powered off) clears + /// the override for good, and the next call routes by the ranked preference. + /// + public void SetSessionAudioEnabled(bool enabled) + { + lock (_gate) + { + if (_disposed || (_sessionAudioEnabled == enabled && _sessionAcquired == enabled)) + return; + _sessionAudioEnabled = enabled; + if (enabled) + { + AcquireSessionIfNeeded(); + } + else if (_sessionAcquired) + { + _sessionAcquired = false; + LeaveCommunicationMode(); + } + } + + // Re-evaluate outside the lock (Reevaluate takes it): pin the policy's target + // on enable, report the platform's own route on disable. + Reevaluate(); + } + + // Called under _gate. The first routing trigger while session audio is enabled + // takes the call session (lazy acquisition — see the class doc); every later + // call is a no-op. Routing verbs express the intent to route, which is what the + // session exists for, so all of them funnel through here: an explicit enable, + // ApplyOutputPreference (including the StartRecording re-assert) and + // SelectOutput. + private void AcquireSessionIfNeeded() + { + if (_disposed || !_sessionAudioEnabled || _sessionAcquired) + return; + _sessionAcquired = true; + EnterCommunicationMode(); + } + + /// + /// Optional audio-focus request (AUDIOFOCUS_GAIN with voice-communication + /// attributes) held while enabled. Off by default. Not exposed on the public + /// API surface (PAR-019 defines it once); flip it here when embedding scenarios + /// need focus, until a supported knob exists. + /// + internal bool AudioFocusEnabled + { + get + { + lock (_gate) return _audioFocusEnabled; + } + set + { + lock (_gate) + { + if (_disposed || _audioFocusEnabled == value) + return; + _audioFocusEnabled = value; + if (value) + RequestAudioFocus(); + else + AbandonAudioFocus(); + } + } + } + + public void Dispose() + { + lock (_gate) + { + if (_disposed) + return; + _disposed = true; + } + + // Stop the poll first so no re-evaluation runs concurrently with teardown. + _stopPoll.Set(); + if (_pollThread.Join(TimeSpan.FromSeconds(3))) + _stopPoll.Dispose(); + else + Utils.Warning("AndroidRouteController: poll thread did not stop in time"); + + // Unregister BEFORE clearing the pin: clearCommunicationDevice fires the + // change event, and a still-registered listener would immediately re-pin. + UnregisterListener(); + + lock (_gate) + { + AbandonAudioFocus(); + // Same idempotent release as a session-audio disable: only a session + // this controller holds (or a transition still pending retry) is handed + // back — a never-acquired session leaves the platform untouched, so + // creating and disposing an instance without a call issues no audio + // traffic at all. + if (_sessionAcquired || _sessionTransitionPending) + LeaveCommunicationMode(); + _sessionAcquired = false; + _sessionAudioEnabled = false; + } + } + + /// + /// Single policy pass: picks the target device (sticky override while its device + /// is still available — dropped for good once it disappears — else the best + /// available kind by rank), pins it when it differs from the active route, and + /// raises when the observable list (ids, kinds, + /// selection) changed since the last pass. Re-pinning is skipped when the target + /// is already active: our own setCommunicationDevice fires the change listener, + /// and that no-op check is what stops the feedback loop. When nothing sticky or + /// ranked is available, an existing pin is released so the OS default applies; + /// kinds missing from the ranking are never auto-selected. + /// + /// While the session is not held — session audio disabled, or enabled but not + /// yet acquired — the pass is observation-only: it enumerates, keeps the sticky + /// bookkeeping current and still raises , but issues + /// no setCommunicationDevice / clearCommunicationDevice and reports the + /// platform's own communication device as the selected one. The change listener + /// and the poll thread run through here without acquiring anything, so neither + /// can resurrect a released session nor take a lazily-deferred one; the + /// re-assert acquires first (in + /// ) and then runs through here like the + /// rest. + /// + private void Reevaluate() + { + List playout = null; + lock (_gate) + { + if (_disposed) + return; + // A failed enter/leave transition is retried from here: every trigger — + // the poll, the change listener, the StartRecording re-assert — funnels + // through this pass, so a transient JNI failure cannot leave the + // platform holding (or missing) the call session until the next call + // boundary. Retries the transition for the CURRENT desired state, so a + // flip that happened in between is never undone. + if (_sessionTransitionPending) + { + if (_sessionAcquired) + EnterCommunicationMode(); + else + LeaveCommunicationMode(); + } + try + { + using var audioManager = GetAudioManager(); + using var current = audioManager.Call("getCommunicationDevice"); + var currentId = current != null ? current.Call("getId") : -1; + + using var available = audioManager.Call("getAvailableCommunicationDevices"); + var count = available.Call("size"); + var devices = new List<(AndroidJavaObject Device, int Id, AudioOutputKind Kind)>(count); + try + { + for (var i = 0; i < count; i++) + { + var device = available.Call("get", i); + devices.Add((device, device.Call("getId"), KindFromDeviceType(device.Call("getType")))); + } + + var targetIndex = -1; + if (_stickyDeviceId != -1) + { + targetIndex = devices.FindIndex(d => d.Id == _stickyDeviceId); + if (targetIndex < 0) + { + Utils.Debug("AndroidRouteController: sticky output device disappeared; reverting to automatic policy"); + _stickyDeviceId = -1; + } + } + + if (targetIndex < 0) + { + var bestRank = int.MaxValue; + for (var i = 0; i < devices.Count; i++) + { + var rank = _ranked.IndexOf(devices[i].Kind); + if (rank >= 0 && rank < bestRank) + { + bestRank = rank; + targetIndex = i; + } + } + } + + int selectedId; + if (!_sessionAcquired) + { + // No session held (no call in progress, or nothing has + // needed the session yet): report which device the platform + // would use for communication audio, and touch nothing. The + // target computed above is still worth running — it keeps + // the sticky override's "dropped once the device disappears" + // bookkeeping alive while idle — but it is only applied once + // the session is acquired. + selectedId = currentId; + } + else if (targetIndex >= 0) + { + var target = devices[targetIndex]; + if (currentId == _pinnedDeviceId) + _pinApplied = true; + if (target.Id != currentId) + { + // A Bluetooth pin that has not taken effect yet is left to + // finish: setCommunicationDevice starts an asynchronous SCO + // negotiation there, and re-issuing lands in the platform's + // own pending activation and gets refused, so hammering it + // keeps the route from ever arriving. Once the pin has been + // seen applied, a later divergence is the platform dropping + // it (what the poll exists for) and is re-pinned at once. + // The other kinds apply without a negotiation, so a + // divergence there is always a dropped or ignored pin and + // is re-issued immediately, as before the settle window + // existed. See PinSettleTimeout, and _pinSettleTimeout for + // the backoff applied when the platform takes a Bluetooth + // pin but never acts on it. + var retry = _pinnedDeviceId == target.Id && !_pinApplied + && target.Kind == AudioOutputKind.Bluetooth; + var settling = retry && ElapsedSincePinIssued() < _pinSettleTimeout; + if (settling) + { + // Waiting on the negotiation: report the device the + // platform still has, never the one merely requested. + // The change listener re-runs this pass the moment the + // pin lands, and the selection flip raises the + // DevicesChanged for the real arrival. + selectedId = currentId; + } + else + { + var ok = audioManager.Call("setCommunicationDevice", target.Device); + Utils.Debug($"AndroidRouteController: setCommunicationDevice(kind={target.Kind}) -> {ok}"); + // Stamped on every attempt, not only on success: + // measured from a stale issue time the settle window + // expires for good after one refused re-issue, and the + // backoff decays into a warn+re-issue every poll tick. + _pinIssuedAtTimestamp = System.Diagnostics.Stopwatch.GetTimestamp(); + if (ok) + { + _pinnedDeviceId = target.Id; + _pinApplied = false; + } + if (retry) + { + // The platform is taking the request and not acting on + // it. Back off rather than keep asking: retrying into + // an activation the platform will not start achieves + // nothing, and the cause is usually outside this SDK + // (see the Bluetooth note in the README). + Utils.Warning( + $"AndroidRouteController: the platform is not applying the route pin for " + + $"{target.Kind} after {_pinSettleTimeout.TotalSeconds:0}s. If this is a " + + "Bluetooth headset, another component in this process (Unity's audio engine " + + "does this when it initializes with a headset connected) may hold an " + + "outstanding startBluetoothSco request, which blocks the call link until it " + + "resolves. Call audio stays on the previous output until then."); + var next = TimeSpan.FromTicks(_pinSettleTimeout.Ticks * 2); + _pinSettleTimeout = next > PinSettleTimeoutMax ? PinSettleTimeoutMax : next; + } + else + { + _pinSettleTimeout = PinSettleTimeout; + } + // Report the platform's answer, not the request: the + // synchronous kinds are visible in this re-read right + // away, while a pending Bluetooth pin must not be + // announced as selected before it lands — GetDevices() + // reads the same truth, and a premature "selected" + // would also swallow the arrival event, because the + // signature would never change again. + using var applied = audioManager.Call("getCommunicationDevice"); + selectedId = applied != null ? applied.Call("getId") : -1; + if (ok && selectedId == target.Id) + _pinApplied = true; + } + } + else + { + selectedId = currentId; + } + } + else + { + if (_pinnedDeviceId != -1) + { + audioManager.Call("clearCommunicationDevice"); + ResetPinTracking(); + Utils.Debug("AndroidRouteController: no ranked device available; cleared pin, OS default applies"); + using var fallback = audioManager.Call("getCommunicationDevice"); + selectedId = fallback != null ? fallback.Call("getId") : -1; + } + else + { + selectedId = currentId; + } + } + + var signature = new List<(int Id, AudioOutputKind Kind, bool IsSelected)>(devices.Count); + foreach (var d in devices) + signature.Add((d.Id, d.Kind, d.Id == selectedId)); + + if (SignatureChanged(signature)) + { + _lastSignature = signature; + playout = new List(devices.Count); + for (var i = 0; i < devices.Count; i++) + playout.Add(ToAudioDevice(devices[i].Device, (uint)i, selectedId)); + } + } + finally + { + foreach (var d in devices) + d.Device.Dispose(); + } + } + catch (Exception e) + { + Utils.Warning($"AndroidRouteController: route evaluation failed: {e.Message}"); + } + } + + // Raised outside the lock; PlatformAudio marshals to the Unity main thread. + if (playout != null) + DevicesChanged?.Invoke(playout, new List(_recordingSnapshot)); + } + + // Called under _gate from both pin-release sites. Clears everything the + // settle/backoff logic keys on; anything left behind resurfaces on the next + // pin as a spurious settle-skip or backoff warning. + private void ResetPinTracking() + { + _pinnedDeviceId = -1; + _pinApplied = false; + _pinSettleTimeout = PinSettleTimeout; + } + + private TimeSpan ElapsedSincePinIssued() + { + var elapsedTicks = System.Diagnostics.Stopwatch.GetTimestamp() - _pinIssuedAtTimestamp; + return TimeSpan.FromSeconds((double)elapsedTicks / System.Diagnostics.Stopwatch.Frequency); + } + + private bool SignatureChanged(List<(int Id, AudioOutputKind Kind, bool IsSelected)> signature) + { + if (_lastSignature == null || _lastSignature.Count != signature.Count) + return true; + for (var i = 0; i < signature.Count; i++) + { + if (!_lastSignature[i].Equals(signature[i])) + return true; + } + return false; + } + + private void PollLoop() + { + if (AndroidJNI.AttachCurrentThread() != 0) + { + Utils.Warning("AndroidRouteController: failed to attach poll thread to the JVM; poll disabled, only OS events will re-route"); + return; + } + try + { + while (!_stopPoll.Wait(PollInterval)) + Reevaluate(); + } + finally + { + AndroidJNI.DetachCurrentThread(); + } + } + + // Both mode methods are called under _gate. The save/restore pairs up per + // acquire -> release transition and is idempotent in both directions: the prior + // mode is only captured when we do not already hold one, and it is only restored + // when it was actually read from the platform — a failed read must never turn + // into an unconditional MODE_NORMAL, which would stomp a mode this app does not + // own (the rule the PAR-000 hotfix established). A failure marks the transition + // pending, and Reevaluate retries it (warned once, retries silent) — both + // methods are safe to re-run partially completed. + private void EnterCommunicationMode() + { + try + { + using var audioManager = GetAudioManager(); + if (!_audioModeSaved) + { + _savedAudioMode = audioManager.Call("getMode"); + _audioModeSaved = true; + } + audioManager.Call("setMode", ModeInCommunication); + _sessionTransitionPending = false; + Utils.Debug($"AndroidRouteController: audio mode -> MODE_IN_COMMUNICATION (was {_savedAudioMode})"); + } + catch (Exception e) + { + if (!_sessionTransitionPending) + Utils.Warning($"AndroidRouteController: failed to enter communication mode (will retry): {e.Message}"); + _sessionTransitionPending = true; + } + } + + private void LeaveCommunicationMode() + { + try + { + using var audioManager = GetAudioManager(); + audioManager.Call("clearCommunicationDevice"); + ResetPinTracking(); + if (_audioModeSaved) + { + audioManager.Call("setMode", _savedAudioMode); + _audioModeSaved = false; + Utils.Debug($"AndroidRouteController: route cleared, audio mode restored ({_savedAudioMode})"); + } + else + { + Utils.Debug("AndroidRouteController: route cleared, no saved audio mode to restore"); + } + _sessionTransitionPending = false; + } + catch (Exception e) + { + if (!_sessionTransitionPending) + Utils.Warning($"AndroidRouteController: failed to release the audio session (will retry): {e.Message}"); + _sessionTransitionPending = true; + } + } + + private void RegisterListener() + { + try + { + using var unityPlayer = new AndroidJavaClass("com.unity3d.player.UnityPlayer"); + using var activity = unityPlayer.GetStatic("currentActivity"); + using var audioManager = activity.Call("getSystemService", "audio"); + using var executor = activity.Call("getMainExecutor"); + + _listener = new CommunicationDeviceListener(this); + audioManager.Call("addOnCommunicationDeviceChangedListener", executor, _listener); + } + catch (Exception e) + { + _listener = null; + Utils.Warning($"AndroidRouteController: failed to register device listener, falling back to polling only: {e.Message}"); + } + } + + private void UnregisterListener() + { + if (_listener == null) + return; + try + { + using var audioManager = GetAudioManager(); + audioManager.Call("removeOnCommunicationDeviceChangedListener", _listener); + } + catch (Exception e) + { + Utils.Warning($"AndroidRouteController: failed to unregister device listener: {e.Message}"); + } + _listener = null; + } + + // Both focus methods are called under _gate. + private void RequestAudioFocus() + { + try + { + using var attributesBuilder = new AndroidJavaObject("android.media.AudioAttributes$Builder"); + using var withUsage = attributesBuilder.Call("setUsage", UsageVoiceCommunication); + using var withContentType = withUsage.Call("setContentType", ContentTypeSpeech); + using var attributes = withContentType.Call("build"); + using var focusBuilder = new AndroidJavaObject("android.media.AudioFocusRequest$Builder", AudioFocusGain); + using var withAttributes = focusBuilder.Call("setAudioAttributes", attributes); + _audioFocusRequest = withAttributes.Call("build"); + + using var audioManager = GetAudioManager(); + var result = audioManager.Call("requestAudioFocus", _audioFocusRequest); + Utils.Debug($"AndroidRouteController: requestAudioFocus -> {(result == AudioFocusRequestGranted ? "granted" : result.ToString())}"); + } + catch (Exception e) + { + _audioFocusRequest?.Dispose(); + _audioFocusRequest = null; + Utils.Warning($"AndroidRouteController: audio focus request failed: {e.Message}"); + } + } + + private void AbandonAudioFocus() + { + if (_audioFocusRequest == null) + return; + try + { + using var audioManager = GetAudioManager(); + audioManager.Call("abandonAudioFocusRequest", _audioFocusRequest); + } + catch (Exception e) + { + Utils.Warning($"AndroidRouteController: failed to abandon audio focus: {e.Message}"); + } + _audioFocusRequest.Dispose(); + _audioFocusRequest = null; + } + + private void OnCommunicationDeviceChangedFromJava() + { + try + { + Reevaluate(); + } + catch (Exception e) + { + Utils.Warning($"AndroidRouteController: listener re-evaluation failed: {e.Message}"); + } + } + + private static AudioDevice ToAudioDevice(AndroidJavaObject device, uint index, int selectedId) + { + var id = device.Call("getId"); + using var productName = device.Call("getProductName"); + return new AudioDevice + { + Index = index, + Name = productName?.Call("toString") ?? string.Empty, + Guid = id.ToString(CultureInfo.InvariantCulture), + Kind = KindFromDeviceType(device.Call("getType")), + IsSelected = id == selectedId, + }; + } + + // AudioDeviceInfo.TYPE_* to AudioOutputKind, mirroring the planned FFI mapping. + private static AudioOutputKind KindFromDeviceType(int deviceType) + { + switch (deviceType) + { + case 1: // TYPE_BUILTIN_EARPIECE + return AudioOutputKind.Earpiece; + case 2: // TYPE_BUILTIN_SPEAKER + return AudioOutputKind.Speaker; + case 3: // TYPE_WIRED_HEADSET + case 4: // TYPE_WIRED_HEADPHONES + return AudioOutputKind.WiredHeadset; + case 7: // TYPE_BLUETOOTH_SCO + case 26: // TYPE_BLE_HEADSET + case 27: // TYPE_BLE_SPEAKER + return AudioOutputKind.Bluetooth; + case 22: // TYPE_USB_HEADSET + return AudioOutputKind.Usb; + case 23: // TYPE_HEARING_AID + return AudioOutputKind.HearingAid; + default: + return AudioOutputKind.Unknown; + } + } + + private static int AndroidSdkInt() + { + using var version = new AndroidJavaClass("android.os.Build$VERSION"); + return version.GetStatic("SDK_INT"); + } + + // Caller owns the returned object (wrap it in `using var`). + private static AndroidJavaObject GetAudioManager() + { + using var unityPlayer = new AndroidJavaClass("com.unity3d.player.UnityPlayer"); + using var activity = unityPlayer.GetStatic("currentActivity"); + return activity.Call("getSystemService", "audio"); + } + + // C#-side implementation of the Java callback interface. AndroidJavaProxy can + // only implement interfaces, which is why this listens for communication-device + // changes rather than subclassing android.media.AudioDeviceCallback (an abstract + // class); list add/remove transitions that fire no communication-device event + // are covered by the poll thread instead. + private sealed class CommunicationDeviceListener : AndroidJavaProxy + { + private readonly AndroidRouteController _controller; + + public CommunicationDeviceListener(AndroidRouteController controller) + : base("android.media.AudioManager$OnCommunicationDeviceChangedListener") + { + _controller = controller; + } + + // Invoked by Android on the activity's main executor — a JVM-attached + // thread, but not the Unity main thread. + public void onCommunicationDeviceChanged(AndroidJavaObject device) + { + device?.Dispose(); + _controller.OnCommunicationDeviceChangedFromJava(); + } + } + } +} +#endif diff --git a/Runtime/Scripts/Audio/AndroidRouteController.cs.meta b/Runtime/Scripts/Audio/AndroidRouteController.cs.meta new file mode 100644 index 00000000..1f1eb67b --- /dev/null +++ b/Runtime/Scripts/Audio/AndroidRouteController.cs.meta @@ -0,0 +1,11 @@ +fileFormatVersion: 2 +guid: f96269970c2ac4b4ea77f794848cafae +MonoImporter: + externalObjects: {} + serializedVersion: 2 + defaultReferences: [] + executionOrder: 0 + icon: {instanceID: 0} + userData: + assetBundleName: + assetBundleVariant: diff --git a/Runtime/Scripts/Audio/IosRouteController.cs b/Runtime/Scripts/Audio/IosRouteController.cs new file mode 100644 index 00000000..2fc9ff2c --- /dev/null +++ b/Runtime/Scripts/Audio/IosRouteController.cs @@ -0,0 +1,223 @@ +#if UNITY_IOS && !UNITY_EDITOR +using System; +using System.Collections.Generic; +using System.Runtime.InteropServices; +using System.Text; +using LiveKit.Internal; + +namespace LiveKit +{ + /// + /// iOS routing backend over the LiveKitAudioSession.mm plugin. The OS owns output + /// route selection on iOS, so this backend does not pick devices: it reduces + /// to the speaker-vs-earpiece relative + /// order (applied as the audio session mode by the plugin; external devices always + /// take priority over both built-ins), reports the session's current output route + /// as the playout device list, and raises from the + /// plugin's route-change observation. throws: apps that + /// want explicit device picking should present the system route picker + /// (AVRoutePickerView). + /// + /// All plugin P/Invoke for route observation stays inside this class; the session + /// state machine itself is driven by (which knows the + /// recording state) through . + /// + internal sealed class IosRouteController : IRouteController + { + private delegate void RouteChangeDelegate(); + + [DllImport("__Internal")] + private static extern void LiveKit_SetRouteChangeCallback(RouteChangeDelegate callback); + + [DllImport("__Internal")] + private static extern void LiveKit_SetSpeakerPreferred([MarshalAs(UnmanagedType.I1)] bool preferred); + + [DllImport("__Internal")] + private static extern IntPtr LiveKit_GetCurrentOutputRoutes(); + + [DllImport("__Internal")] + private static extern void LiveKit_FreeRouteString(IntPtr routes); + + // The native callback slot is registered once for the app lifetime (matching + // the plugin's app-lifetime notification observers) and fans out to the live + // controllers; keeping the delegate in a static field pins it for the native + // side. Instances add and remove themselves under StaticGate. + private static readonly object StaticGate = new object(); + private static readonly List LiveControllers = new List(); + private static readonly RouteChangeDelegate NativeRouteChanged = OnNativeRouteChanged; + private static bool _callbackRegistered; + + private readonly object _gate = new object(); + // The FFI recording list (a single placeholder for the OS default input), + // captured once: route changes never affect it and re-querying the FFI from + // the route callback would be wasted work. + private readonly List _recordingSnapshot; + private string _lastSignature; + private bool _disposed; + + public event Action, IReadOnlyList> DevicesChanged; + + internal IosRouteController(PlatformAudio owner, IReadOnlyList initialPreference) + { + _recordingSnapshot = owner.GetDevicesViaFfi().Recording; + + ApplyOutputPreference(initialPreference); + _lastSignature = Signature(QueryCurrentOutputs()); + + lock (StaticGate) + { + LiveControllers.Add(this); + if (!_callbackRegistered) + { + LiveKit_SetRouteChangeCallback(NativeRouteChanged); + _callbackRegistered = true; + } + } + } + + public (List Recording, List Playout) GetDevices() + { + return (new List(_recordingSnapshot), QueryCurrentOutputs()); + } + + public void ApplyOutputPreference(IReadOnlyList ranked) + { + // Reduce the ranked list per the PAR-019 precedence rule: the only part of + // the ranking iOS can express is whether Speaker outranks Earpiece. + var speaker = -1; + var earpiece = -1; + for (var i = 0; i < ranked.Count; i++) + { + if (ranked[i] == AudioOutputKind.Speaker) speaker = i; + else if (ranked[i] == AudioOutputKind.Earpiece) earpiece = i; + } + var speakerPreferred = speaker >= 0 && (earpiece < 0 || speaker < earpiece); + LiveKit_SetSpeakerPreferred(speakerPreferred); + } + + public void SelectOutput(AudioDevice device) + { + throw new NotSupportedException( + "SelectOutput is not supported on iOS: the OS owns output route selection. " + + "Present the system route picker (AVRoutePickerView) instead, or use " + + "OutputPreference / IsSpeakerOutputPreferred for the built-in outputs."); + } + + public void ClearOutputOverride() + { + // No override can exist on iOS: SelectOutput throws. + } + + public void SetSessionAudioEnabled(bool enabled) + { + // Handled by PlatformAudio itself on iOS: it drives the session state machine + // (which needs the recording state this backend does not know) through + // IOSAudioSessionHelper. Nothing route-specific to gate here. + } + + public void Dispose() + { + lock (StaticGate) + { + LiveControllers.Remove(this); + } + lock (_gate) + { + _disposed = true; + } + } + + /// + /// Native route-change entry point, invoked by the plugin on the iOS main + /// queue (not the Unity main thread; marshals the + /// public event). + /// + [AOT.MonoPInvokeCallback(typeof(RouteChangeDelegate))] + private static void OnNativeRouteChanged() + { + IosRouteController[] controllers; + lock (StaticGate) + { + controllers = LiveControllers.ToArray(); + } + foreach (var controller in controllers) + controller.HandleRouteChanged(); + } + + private void HandleRouteChanged() + { + List playout; + lock (_gate) + { + if (_disposed) return; + + playout = QueryCurrentOutputs(); + var signature = Signature(playout); + if (signature == _lastSignature) return; + _lastSignature = signature; + } + + DevicesChanged?.Invoke(playout, new List(_recordingSnapshot)); + } + + /// + /// The current output route reported by the audio session. On iOS this is the + /// active route (usually one device), not an enumeration of every reachable + /// device — AVAudioSession exposes no such list for outputs. + /// + private static List QueryCurrentOutputs() + { + var devices = new List(); + + var routesPtr = LiveKit_GetCurrentOutputRoutes(); + if (routesPtr == IntPtr.Zero) return devices; + + string routes; + try + { + routes = Marshal.PtrToStringUTF8(routesPtr); + } + finally + { + LiveKit_FreeRouteString(routesPtr); + } + if (string.IsNullOrEmpty(routes)) return devices; + + foreach (var line in routes.Split('\n')) + { + if (line.Length == 0) continue; + var fields = line.Split('\t'); + if (fields.Length != 3) + { + Utils.Warning($"IosRouteController: malformed route entry '{line}'"); + continue; + } + + var kind = int.TryParse(fields[0], out var rawKind) + && Enum.IsDefined(typeof(AudioOutputKind), rawKind) + ? (AudioOutputKind)rawKind + : AudioOutputKind.Unknown; + devices.Add(new AudioDevice + { + Index = (uint)devices.Count, + Name = fields[1], + Guid = fields[2], + Kind = kind, + // Everything in the current route is live output by definition. + IsSelected = true, + }); + } + + return devices; + } + + private static string Signature(List playout) + { + var builder = new StringBuilder(); + foreach (var device in playout) + builder.Append(device.Guid).Append('\u001f').Append((int)device.Kind).Append('\u001e'); + return builder.ToString(); + } + } +} +#endif diff --git a/Runtime/Scripts/Audio/IosRouteController.cs.meta b/Runtime/Scripts/Audio/IosRouteController.cs.meta new file mode 100644 index 00000000..069ce527 --- /dev/null +++ b/Runtime/Scripts/Audio/IosRouteController.cs.meta @@ -0,0 +1,11 @@ +fileFormatVersion: 2 +guid: 2b7a83f0ed8eb4ce1bbd824fdf80c424 +MonoImporter: + externalObjects: {} + serializedVersion: 2 + defaultReferences: [] + executionOrder: 0 + icon: {instanceID: 0} + userData: + assetBundleName: + assetBundleVariant: diff --git a/Runtime/Scripts/Audio/PlatformAudio.cs b/Runtime/Scripts/Audio/PlatformAudio.cs index 7c113e20..62381372 100644 --- a/Runtime/Scripts/Audio/PlatformAudio.cs +++ b/Runtime/Scripts/Audio/PlatformAudio.cs @@ -1,6 +1,7 @@ using System; using System.Collections; using System.Collections.Generic; +using System.Threading; using LiveKit.Proto; using LiveKit.Internal; using LiveKit.Internal.FFI.Requests; @@ -27,13 +28,58 @@ internal static class IOSAudioSessionHelper internal static extern void LiveKit_ConfigureAudioSessionForVoIP(); /// - /// Restores the iOS audio session to ambient mode. + /// Restores the audio session Unity had before LiveKit configured it + /// (or the ambient category as a fallback) and reactivates it so Unity + /// audio output resumes. Called when the last PlatformAudio is disposed. /// [DllImport("__Internal")] internal static extern void LiveKit_RestoreDefaultAudioSession(); + + /// + /// Enables or disables WebRTC's VPIO audio unit while the app keeps + /// ownership of the audio session. Enable when a call connects, disable + /// when it ends. Disabling on hang-up stops call audio without + /// deactivating the session, so other app audio keeps playing. + /// + [DllImport("__Internal")] + internal static extern void LiveKit_SetAudioEnabled([MarshalAs(UnmanagedType.I1)] bool enabled); + + /// + /// Sets the audio session state (0 idle, 1 playout-only, 2 recording) so the + /// plugin can apply the matching category/mode/options (see the state table in + /// LiveKitAudioSession.mm). Driven by PlatformAudio, which knows whether + /// recording is active and whether call audio is wanted. + /// + [DllImport("__Internal")] + internal static extern void LiveKit_SetSessionState(int state); } #endif + /// + /// The kind of audio output device, used for ranked routing policies on mobile + /// platforms (see ). + /// + /// The numeric values mirror the planned FFI protocol enum (AudioDeviceKind) one-to-one + /// so a future FFI-backed implementation maps without translation. Do not renumber. + /// + public enum AudioOutputKind + { + /// The platform did not report a device type. + Unknown = 0, + /// The phone's built-in earpiece (receiver). + Earpiece = 1, + /// The built-in loudspeaker. + Speaker = 2, + /// A wired headset or headphones. + WiredHeadset = 3, + /// A Bluetooth audio device. + Bluetooth = 4, + /// A USB audio device. + Usb = 5, + /// A hearing aid. + HearingAid = 6, + } + /// /// Information about an audio device (microphone or speaker). /// @@ -49,6 +95,21 @@ public struct AudioDevice /// over index for device selection. /// public string Guid; + /// + /// The kind of output this device represents. Classified by the routing backend + /// for playout devices — on iOS from the audio session's current route, on + /// Android 12 (API 31) and newer from the communication-device list; where the platform does not report a type + /// (recording devices, desktop, older Android). + /// + public AudioOutputKind Kind; + /// + /// Whether this device is the active output route. Reported by the routing + /// backend for playout devices on iOS and on Android 12 (API 31) and newer; + /// always false where no backend reports selection state (recording devices, + /// desktop, older Android). + /// + public bool IsSelected; } /// @@ -73,17 +134,71 @@ public sealed class PlatformAudio : IDisposable { internal readonly FfiHandle Handle; private readonly PlatformAudioInfo _info; + private readonly IRouteController _routeController; + private readonly SynchronizationContext _syncContext; + private List _outputPreference = new List(DefaultOutputPreference); private bool _disposed = false; +#if UNITY_IOS && !UNITY_EDITOR + // Tracks live PlatformAudio instances so the iOS audio session is restored + // only when the last one is disposed (aligned with the native ADM ref-count). + private static int _instanceCount; + + // Inputs of the iOS session-state machine (see the state table in + // LiveKitAudioSession.mm). PlatformAudio is the driver because it is the one + // that knows both: whether recording is active (its own StartRecording/ + // StopRecording calls) and whether call audio is wanted (SetSessionAudioEnabled). + private const int IosSessionStateIdle = 0; + private const int IosSessionStatePlayoutOnly = 1; + private const int IosSessionStateRecording = 2; + private bool _iosRecordingActive; + private bool _iosSessionAudioEnabled = true; + + private void UpdateIosSessionState() + { + var state = !_iosSessionAudioEnabled ? IosSessionStateIdle + : _iosRecordingActive ? IosSessionStateRecording + : IosSessionStatePlayoutOnly; + IOSAudioSessionHelper.LiveKit_SetSessionState(state); + } +#endif + + private static readonly AudioOutputKind[] DefaultOutputPreference = + { + AudioOutputKind.Bluetooth, + AudioOutputKind.WiredHeadset, + AudioOutputKind.Speaker, + AudioOutputKind.Earpiece, + }; /// /// Number of available recording (microphone) devices. /// - public int RecordingDeviceCount => _info.RecordingDeviceCount; + public int RecordingDeviceCount + { + get + { + ThrowIfDisposed(); + return _info.RecordingDeviceCount; + } + } /// /// Number of available playout (speaker) devices. /// - public int PlayoutDeviceCount => _info.PlayoutDeviceCount; + public int PlayoutDeviceCount + { + get + { + ThrowIfDisposed(); + return _info.PlayoutDeviceCount; + } + } + + private void ThrowIfDisposed() + { + if (_disposed) + throw new ObjectDisposedException(nameof(PlatformAudio)); + } /// /// Creates a new PlatformAudio instance, enabling the platform ADM. @@ -91,9 +206,23 @@ public sealed class PlatformAudio : IDisposable /// This must be called before creating any PlatformAudioSource or connecting /// to a room if you want automatic speaker playout for remote audio. /// - /// On iOS, this automatically configures the audio session for VoIP mode - /// (PlayAndRecord category with VoiceChat mode) to enable hardware echo - /// cancellation and microphone input. + /// On iOS, this automatically configures the audio session for VoIP use and + /// takes app ownership of it. The session's mode follows the call state: a + /// voice/video-chat mode (enabling hardware echo cancellation) while recording + /// is active, and a music-friendly default mode otherwise (see + /// / / + /// ). + /// + /// Session audio starts out enabled on every platform, but on Android the call + /// session itself is acquired lazily: construction changes no audio mode and + /// pins no route — the first routing action while session audio is enabled + /// takes the session (an explicit enable, + /// an output preference or selection change, or the + /// re-assert). Apps that create PlatformAudio before their first call should + /// still call with false right + /// after construction and enable it when a call starts, so the call session + /// covers calls rather than the app's lifetime — on iOS that is also what keeps + /// the idle session in its music-friendly state. /// /// /// Thrown if the platform ADM could not be initialized (e.g., no audio devices, @@ -103,7 +232,7 @@ public PlatformAudio() { #if UNITY_IOS && !UNITY_EDITOR // Configure iOS audio session for VoIP before initializing WebRTC ADM. - // This sets PlayAndRecord category with VoiceChat mode for hardware AEC. + // This sets PlayAndRecord category with VideoChat mode for hardware AEC. IOSAudioSessionHelper.LiveKit_ConfigureAudioSessionForVoIP(); #endif @@ -118,7 +247,46 @@ public PlatformAudio() Handle = FfiHandle.FromOwnedHandle(platformAudio.Handle); _info = platformAudio.Info; + try + { + _syncContext = SynchronizationContext.Current; + _routeController = CreateRouteController(); + _routeController.DevicesChanged += OnRouteControllerDevicesChanged; + +#if UNITY_IOS && !UNITY_EDITOR + // A fresh instance starts in the playout-only state (recording has not + // been started); this matches the plugin's post-configure default, so the + // call is a no-op unless an earlier instance left another state behind. + UpdateIosSessionState(); +#endif + } + catch + { + // Without this, a route-controller failure would leak the FFI handle + // until the SafeHandle finalizer eventually reclaims it. + _routeController?.Dispose(); + Handle.Dispose(); + throw; + } + Utils.Debug($"PlatformAudio created: {RecordingDeviceCount} recording devices, {PlayoutDeviceCount} playout devices"); + +#if UNITY_IOS && !UNITY_EDITOR + // Count this instance only after successful construction so a failed + // ctor never leaves the counter stuck above zero. + System.Threading.Interlocked.Increment(ref _instanceCount); +#endif + } + + private IRouteController CreateRouteController() + { +#if UNITY_ANDROID && !UNITY_EDITOR + return AndroidRouteController.Create(this, _outputPreference); +#elif UNITY_IOS && !UNITY_EDITOR + return new IosRouteController(this, _outputPreference); +#else + return new DesktopRouteController(this); +#endif } /// @@ -128,24 +296,46 @@ public PlatformAudio() /// - Desktop (Windows/macOS/Linux): returns the full list of microphones and /// speakers reported by the OS. Devices can be selected with /// / . - /// - iOS and Android: returns a single placeholder entry at index 0 for each - /// list, representing the system's currently selected default input/output. - /// The OS owns audio routing on these platforms (AVAudioSession on iOS, - /// AudioManager on Android), so individual devices are not enumerated and - /// selecting one is a no-op (see / + /// - iOS: the playout list is the audio session's current output route (usually + /// one device, with and + /// set) — iOS does not enumerate every + /// reachable output device. The recording list is a single placeholder entry + /// for the OS default input. + /// - Android 12 (API 31) and newer: the playout list contains the available + /// communication devices with and + /// set; entries can be routed to with + /// . The recording list stays a single placeholder + /// entry for the OS default input — input routing follows the selected + /// communication device. + /// - Older Android: returns a single placeholder entry at index 0 for each list, + /// representing the system's currently selected default input/output. The OS + /// owns audio routing (AudioManager), so individual devices are not enumerated + /// and selecting one is a no-op (see / /// ). /// /// /// A tuple containing: /// - Recording: List of available microphones (on iOS/Android, a single /// placeholder for the OS default input) - /// - Playout: List of available speakers/headphones (on iOS/Android, a single - /// placeholder for the OS default output) + /// - Playout: List of available speakers/headphones (on iOS, the current output + /// route; on pre-API-31 Android, a single placeholder for the OS default + /// output) /// /// /// Thrown if device enumeration failed. /// public (List Recording, List Playout) GetDevices() + { + ThrowIfDisposed(); + return _routeController.GetDevices(); + } + + /// + /// Device enumeration through the FFI, shared by the route controllers. + /// and are not + /// reported by the FFI and stay at their defaults (Unknown / false). + /// + internal (List Recording, List Playout) GetDevicesViaFfi() { using var request = FFIBridge.Instance.NewRequest(); request.request.PlatformAudioHandle = (ulong)Handle.DangerousGetHandle(); @@ -179,6 +369,241 @@ public PlatformAudio() return (recording, playout); } + /// + /// Ranked automatic output routing policy, most preferred first. When no explicit + /// output override is active (), the platform routes to + /// the highest-ranked kind that has a connected device. + /// + /// Default: Bluetooth > WiredHeadset > Speaker > Earpiece. + /// + /// Precedence with : this list is the single + /// source of truth; the bool is convenience sugar that only rewrites the relative + /// order of and + /// inside this list, and reading the bool + /// reads their current relative order. There is no separate speaker-preference state. + /// + /// Platform notes: on iOS, external devices (Bluetooth, wired) always take priority + /// over the built-in outputs, so the Speaker/Earpiece relative order — i.e. + /// — is the only part of the ranking with an + /// effect; it is applied through the audio session mode and takes effect + /// immediately, including mid-call. On Android the full ranking applies: the + /// backend routes to the highest-ranked available kind on Android 12 (API 31) + /// and newer, and kinds missing from the list are never auto-selected (when + /// nothing ranked is available the OS default route applies). On desktop, output + /// is selected per device ( / + /// ) and the ranking has no routing effect. + /// On older Android versions (routing backend not implemented there) the value + /// is stored and round-trips, but has no routing effect either. + /// + /// Thrown if set to null. + /// + /// Thrown if the list contains or duplicates. + /// + public IReadOnlyList OutputPreference + { + get + { + ThrowIfDisposed(); + return _outputPreference.AsReadOnly(); + } + set + { + ThrowIfDisposed(); + if (value == null) + throw new ArgumentNullException(nameof(value)); + + var ranked = new List(value.Count); + foreach (var kind in value) + { + if (kind == AudioOutputKind.Unknown) + throw new ArgumentException( + "OutputPreference cannot contain AudioOutputKind.Unknown", nameof(value)); + if (ranked.Contains(kind)) + throw new ArgumentException( + $"OutputPreference contains {kind} more than once", nameof(value)); + ranked.Add(kind); + } + + _outputPreference = ranked; + _routeController.ApplyOutputPreference(_outputPreference.AsReadOnly()); + } + } + + /// + /// Whether the loudspeaker is preferred over the earpiece for automatic routing. + /// + /// Precedence with : the list is the single source of + /// truth; this bool is convenience sugar that only rewrites the relative order of + /// and + /// inside , and reading it reads their current + /// relative order. There is no separate speaker-preference state. Reading returns + /// true when Speaker ranks ahead of Earpiece (or Earpiece is absent), false when + /// Speaker is absent. Setting reorders the pair in place at the position of + /// whichever currently ranks first, inserting a missing kind next to the present + /// one (or appending both when neither is listed) so the value round-trips. + /// + /// Platform notes: on iOS, external devices (Bluetooth, wired) always take priority + /// over the built-in outputs, so this bool is the only part of the ranking with an + /// effect. It decides where audio goes when no external device is connected, is + /// applied through the audio session mode (never by overriding the output port), + /// and takes effect immediately, including mid-call. On Android the full ranking + /// applies: the backend routes to the highest-ranked available kind on Android 12 + /// (API 31) and newer. On desktop, output is selected per device + /// ( / ) and the + /// ranking has no routing effect. On older Android versions (routing backend not + /// implemented there) the value is stored and round-trips, but has no routing + /// effect either. + /// + public bool IsSpeakerOutputPreferred + { + get + { + ThrowIfDisposed(); + var speaker = _outputPreference.IndexOf(AudioOutputKind.Speaker); + var earpiece = _outputPreference.IndexOf(AudioOutputKind.Earpiece); + if (speaker < 0) return false; + return earpiece < 0 || speaker < earpiece; + } + set + { + ThrowIfDisposed(); + var first = value ? AudioOutputKind.Speaker : AudioOutputKind.Earpiece; + var second = value ? AudioOutputKind.Earpiece : AudioOutputKind.Speaker; + + var reordered = new List(_outputPreference.Count + 2); + var pairInserted = false; + foreach (var kind in _outputPreference) + { + if (kind == AudioOutputKind.Speaker || kind == AudioOutputKind.Earpiece) + { + if (!pairInserted) + { + reordered.Add(first); + reordered.Add(second); + pairInserted = true; + } + continue; + } + reordered.Add(kind); + } + if (!pairInserted) + { + reordered.Add(first); + reordered.Add(second); + } + + _outputPreference = reordered; + _routeController.ApplyOutputPreference(_outputPreference.AsReadOnly()); + } + } + + /// + /// Routes audio output to the given device as a sticky override of the automatic + /// policy: the route stays on the device until + /// is called. The device is matched against the + /// current playout list by + /// when set, otherwise by index and name. + /// + /// Platform notes: on desktop this selects the device like + /// . On Android 12 (API 31) and newer the + /// device is pinned as the communication device; the override is dropped once the + /// device disappears from the playout list (automatic policy resumes). While + /// session audio is disabled () the choice is + /// only recorded — no pin is issued, and / + /// keep reporting the platform's own route — until a + /// call enables the session. There is deliberately no pending flag for that + /// deferral: the app holds both inputs (its own SelectOutput call and its own + /// session-enable state), so a pre-call device picker should treat its last + /// selection as the pending choice and confirm application through the existing + /// surface — once the session is enabled and the pin lands, the device's + /// flips in / + /// . A deferred choice is dropped for good when its + /// device disappears before the session is enabled (the same drop-on-disappear + /// rule as an active pin), observable as the device leaving the playout list in + /// the same events. On iOS the + /// OS owns output route selection and this method throws + /// — present the system route picker + /// (AVRoutePickerView) instead, or use / + /// for the built-in outputs. On older + /// Android versions (routing backend not implemented there) this method also + /// throws. + /// + /// A playout device from . + /// + /// Thrown if the device does not match any current playout device. + /// + /// + /// Thrown on iOS (the OS owns route selection) and on Android below API 31. + /// + public void SelectOutput(AudioDevice device) + { + ThrowIfDisposed(); + var (_, playout) = GetDevices(); + foreach (var candidate in playout) + { + var matches = !string.IsNullOrEmpty(device.Guid) + ? candidate.Guid == device.Guid + : candidate.Index == device.Index && candidate.Name == device.Name; + if (!matches) continue; + + _routeController.SelectOutput(candidate); + return; + } + + throw new ArgumentException( + $"Device '{device.Name}' (index {device.Index}, guid {device.Guid ?? "none"}) " + + "is not a current playout device", nameof(device)); + } + + /// + /// Clears the sticky override set by so the automatic + /// policy applies again. + /// + /// Platform notes: on desktop there is no automatic policy to fall back to yet, so + /// clearing keeps the currently selected device (no-op). On Android 12 (API 31) + /// and newer the automatic policy re-routes immediately. On older Android + /// versions and on iOS no override can exist ( throws), + /// so this is a no-op there. + /// + public void ClearOutputOverride() + { + ThrowIfDisposed(); + _routeController.ClearOutputOverride(); + } + + /// + /// Raised when the set of available audio devices changes, with the current playout + /// and recording device lists. Raised on the Unity main thread. + /// + /// On iOS this fires when the audio session's output route changes (headset + /// plugged/unplugged, Bluetooth connected, speaker/earpiece switches); the playout + /// list is the new route. On Android it is raised by the routing backend + /// (Android 12/API 31 and newer) when the available communication devices or the + /// active route change; changes that fire no OS event are detected by a poll with + /// roughly 1.5 s of latency. Desktop hot-plug events are not implemented yet in + /// this version, so the event is never raised there. Subscribing and + /// unsubscribing is safe at any time, including after . + /// + public event Action, IReadOnlyList> DevicesChanged; + + private void OnRouteControllerDevicesChanged( + IReadOnlyList playout, IReadOnlyList recording) + { + if (_disposed) return; + + if (_syncContext != null && _syncContext != SynchronizationContext.Current) + { + _syncContext.Post(_ => + { + if (!_disposed) + DevicesChanged?.Invoke(playout, recording); + }, null); + return; + } + + DevicesChanged?.Invoke(playout, recording); + } + /// /// Sets the recording device (microphone) by index. /// @@ -192,6 +617,7 @@ public PlatformAudio() /// public void SetRecordingDevice(uint index) { + ThrowIfDisposed(); var (recording, _) = GetDevices(); if (index >= recording.Count) throw new InvalidOperationException($"Recording device index {index} out of range (max: {recording.Count - 1})"); @@ -215,6 +641,7 @@ public void SetRecordingDevice(uint index) /// public void SetRecordingDevice(string deviceId) { + ThrowIfDisposed(); using var request = FFIBridge.Instance.NewRequest(); request.request.PlatformAudioHandle = (ulong)Handle.DangerousGetHandle(); request.request.DeviceId = deviceId; @@ -241,6 +668,7 @@ public void SetRecordingDevice(string deviceId) /// public void SetPlayoutDevice(uint index) { + ThrowIfDisposed(); var (_, playout) = GetDevices(); if (index >= playout.Count) throw new InvalidOperationException($"Playout device index {index} out of range (max: {playout.Count - 1})"); @@ -264,6 +692,7 @@ public void SetPlayoutDevice(uint index) /// public void SetPlayoutDevice(string deviceId) { + ThrowIfDisposed(); using var request = FFIBridge.Instance.NewRequest(); request.request.PlatformAudioHandle = (ulong)Handle.DangerousGetHandle(); request.request.DeviceId = deviceId; @@ -280,15 +709,25 @@ public void SetPlayoutDevice(string deviceId) /// /// Starts recording from the microphone. /// - /// Recording is started automatically when PlatformAudio is created. - /// Use this to resume recording after calling StopRecording. + /// Recording does not start on its own when PlatformAudio is created — call + /// this to start capturing, and again to resume after . + /// On Android and iOS the coroutine first awaits the OS microphone-permission + /// dialog when the permission has not been granted yet, and only then opens the + /// capture — a capture opened while the prompt is pending would record silence. /// This turns on the system's recording privacy indicator (e.g., on macOS/iOS). + /// On iOS this also switches the audio session to its recording state + /// (voice/video-chat mode per , enabling + /// hardware echo cancellation). /// /// /// Thrown if the operation failed. /// public IEnumerator StartRecording() { + // Iterator method: this throws on the first MoveNext, like the other + // exceptions below — Unity's StartCoroutine runs that synchronously. + ThrowIfDisposed(); + #if PLATFORM_ANDROID if (!Permission.HasUserAuthorizedPermission(Permission.Microphone)) { @@ -313,6 +752,22 @@ public IEnumerator StartRecording() } #endif +#if UNITY_IOS && !UNITY_EDITOR + if (!UnityEngine.Application.HasUserAuthorization(UnityEngine.UserAuthorization.Microphone)) + { + // Ask for the record permission BEFORE the ADM opens the input unit. The + // system prompt is asynchronous: a capture opened while it is still + // pending records silence, and nothing reopens the input after the user + // grants — so without this gate the first run of an app publishes a + // silent microphone track. + yield return UnityEngine.Application.RequestUserAuthorization( + UnityEngine.UserAuthorization.Microphone); + if (!UnityEngine.Application.HasUserAuthorization(UnityEngine.UserAuthorization.Microphone)) + throw new InvalidOperationException( + "Microphone permission denied by user; cannot start recording."); + } +#endif + using var request = FFIBridge.Instance.NewRequest(); request.request.PlatformAudioHandle = (ulong)Handle.DangerousGetHandle(); @@ -322,8 +777,22 @@ public IEnumerator StartRecording() if (res.StartRecording.HasError && !string.IsNullOrEmpty(res.StartRecording.Error)) throw new InvalidOperationException($"Failed to start recording: {res.StartRecording.Error}"); +#if UNITY_IOS && !UNITY_EDITOR + _iosRecordingActive = true; + UpdateIosSessionState(); +#endif + Utils.Debug("PlatformAudio: started recording"); + // Re-assert the routing policy now that capture is active. Since Android 13 + // the app's MODE_IN_COMMUNICATION request — and with it the + // communication-device pin — is only honored while the app has active + // voice-communication capture, so the platform may have moved the route + // while it was un-owned. On Android this is also where a lazily-deferred + // call session is first acquired (see SetSessionAudioEnabled). No-op on the + // other backends. + _routeController.ApplyOutputPreference(_outputPreference.AsReadOnly()); + // Ensures this method is always a valid iterator even when the PLATFORM_ANDROID // branch is compiled out (no `yield return` would otherwise be reachable on // non-Android builds, which is a compile error for IEnumerator-returning methods). @@ -336,12 +805,15 @@ public IEnumerator StartRecording() /// Use this to temporarily stop recording without disposing PlatformAudio. /// This turns off the system's recording privacy indicator (e.g., on macOS/iOS). /// Call StartRecording to resume recording. + /// On iOS this also switches the audio session back to its playout-only state + /// (music-friendly default mode). /// /// /// Thrown if the operation failed. /// public void StopRecording() { + ThrowIfDisposed(); using var request = FFIBridge.Instance.NewRequest(); request.request.PlatformAudioHandle = (ulong)Handle.DangerousGetHandle(); @@ -351,20 +823,93 @@ public void StopRecording() if (res.StopRecording.HasError && !string.IsNullOrEmpty(res.StopRecording.Error)) throw new InvalidOperationException($"Failed to stop recording: {res.StopRecording.Error}"); +#if UNITY_IOS && !UNITY_EDITOR + _iosRecordingActive = false; + UpdateIosSessionState(); +#endif + Utils.Debug("PlatformAudio: stopped recording"); } + /// + /// Signals whether call audio should be active on the platform audio session, + /// i.e. whether a call is in progress. Enabled by default when PlatformAudio is + /// created, so it only needs to be called to false when leaving a room + /// (and back to true when rejoining) — but an app that creates + /// PlatformAudio well before its first call (e.g. at startup, to keep the ADM + /// alive) should disable it right after creation, so the platform's call audio + /// session is only held for the duration of an actual call. + /// + /// On iOS this gates WebRTC's VPIO audio unit while the app retains ownership + /// of the shared AVAudioSession. Disabling stops the microphone/remote audio + /// path and the hardware voice processing, and drops the session to its idle + /// state (music-friendly default mode), but keeps the audio session active so + /// other Unity audio (e.g. background music) is not interrupted — which is why + /// Unity audio survives a hang-up. + /// + /// On Android 12 (API 31) and newer this gates the voice-communication audio + /// session the routing backend holds: while enabled the SDK requests + /// MODE_IN_COMMUNICATION and keeps the output route pinned per + /// ; while disabled it holds neither, so the OS + /// applies its normal routing and the call session covers the call rather than + /// the lifetime of this instance. The session is acquired lazily: despite the + /// enabled default, creating the instance takes nothing — the first routing + /// action while enabled takes it (calling this method with true, even + /// when already enabled; changing / + /// ; ; or the + /// re-assert). A receive-only app that never + /// records and never touches routing therefore keeps the platform's own routing + /// until it calls this method with true at its call boundary. Device + /// enumeration and keep working while disabled. Unlike iOS, + /// disabling does not stop the ADM: pair it with + /// / at the call + /// boundaries — an active capture without the session is what lets the platform + /// take routing back (see ). + /// + /// On the remaining platforms this is a no-op: the OS/ADM manages the session + /// directly. + /// + /// True while a call is active, false otherwise. + public void SetSessionAudioEnabled(bool enabled) + { + ThrowIfDisposed(); +#if UNITY_IOS && !UNITY_EDITOR + IOSAudioSessionHelper.LiveKit_SetAudioEnabled(enabled); + _iosSessionAudioEnabled = enabled; + UpdateIosSessionState(); +#endif + _routeController.SetSessionAudioEnabled(enabled); + Utils.Debug($"PlatformAudio: session audio enabled={enabled}"); + } + /// /// Releases the PlatformAudio resources. /// /// When disposed, the platform ADM may be disabled if this was the last /// PlatformAudio instance. + /// + /// Disposing is idempotent. After disposal every public member throws + /// , except subscribing to / + /// unsubscribing from , which stays safe. /// public void Dispose() { if (_disposed) return; - Handle.Dispose(); _disposed = true; + _routeController.DevicesChanged -= OnRouteControllerDevicesChanged; + _routeController.Dispose(); + Handle.Dispose(); + +#if UNITY_IOS && !UNITY_EDITOR + // Once the last instance is gone, relinquish the app-owned audio session: + // disable call audio, release our activation, leave manual mode, restore + // the session Unity had before LiveKit touched it, and reactivate it so + // Unity audio output resumes. Balances LiveKit_ConfigureAudioSessionForVoIP() + // in the constructor so the session isn't left stuck in PlayAndRecord. + if (System.Threading.Interlocked.Decrement(ref _instanceCount) == 0) + IOSAudioSessionHelper.LiveKit_RestoreDefaultAudioSession(); +#endif + Utils.Debug("PlatformAudio disposed"); } } diff --git a/Runtime/Scripts/Audio/RouteController.cs b/Runtime/Scripts/Audio/RouteController.cs new file mode 100644 index 00000000..aef51735 --- /dev/null +++ b/Runtime/Scripts/Audio/RouteController.cs @@ -0,0 +1,155 @@ +using System; +using System.Collections.Generic; + +namespace LiveKit +{ + /// + /// Backend seam for audio output routing. registers one + /// implementation per platform and forwards its public routing API + /// (, , + /// , , + /// ) through it, so the plumbing can be swapped + /// per platform — and later wholesale for an FFI-backed implementation — without changing + /// a public signature. + /// + internal interface IRouteController : IDisposable + { + /// Snapshot of the current recording and playout device lists. + (List Recording, List Playout) GetDevices(); + + /// Applies the ranked automatic output policy, most preferred first. + void ApplyOutputPreference(IReadOnlyList ranked); + + /// + /// Routes output to the given device as a sticky override of the automatic policy. + /// The device has already been validated against the current playout snapshot. + /// + void SelectOutput(AudioDevice device); + + /// Clears the sticky override so the automatic policy applies again. + void ClearOutputOverride(); + + /// + /// Signals whether a call is in progress, i.e. whether the backend may hold the + /// platform's voice-communication audio session. Device enumeration and + /// must keep working while disabled. + /// + void SetSessionAudioEnabled(bool enabled); + + /// + /// Raised when the available devices change, with the current (playout, recording) + /// lists. May be raised from any thread; marshals it to + /// the Unity main thread before re-raising publicly. + /// + event Action, IReadOnlyList> DevicesChanged; + } + + /// + /// Desktop routing backend: wraps the FFI device enumeration and per-device GUID + /// selection. Ranked-kind policy is not implemented on desktop (output is chosen per + /// device), and no desktop hot-plug events exist yet, so + /// is never raised. + /// + internal sealed class DesktopRouteController : IRouteController + { + private readonly PlatformAudio _owner; + + public DesktopRouteController(PlatformAudio owner) + { + _owner = owner; + } + + public (List Recording, List Playout) GetDevices() + { + return _owner.GetDevicesViaFfi(); + } + + public void ApplyOutputPreference(IReadOnlyList ranked) + { + // No routing effect on desktop: output is selected per device, not by kind. + } + + public void SelectOutput(AudioDevice device) + { + if (!string.IsNullOrEmpty(device.Guid)) + _owner.SetPlayoutDevice(device.Guid); + else + _owner.SetPlayoutDevice(device.Index); + } + + public void ClearOutputOverride() + { + // No automatic policy to fall back to on desktop; the selected device stays. + } + + public void SetSessionAudioEnabled(bool enabled) + { + // No call session to hold on desktop: the ADM owns the devices directly. + } + + public event Action, IReadOnlyList> DevicesChanged + { + add { } + remove { } + } + + public void Dispose() + { + } + } + + /// + /// Placeholder backend for platforms without a routing implementation: Android below + /// API 31 (which lacks the communication-device APIs the Android backend is built + /// on). Device snapshots still work through the FFI (a single placeholder entry for + /// the OS default input/output); the routing verbs throw or no-op as documented on + /// the public API. + /// + internal sealed class UnsupportedRouteController : IRouteController + { + private readonly PlatformAudio _owner; + private readonly string _platform; + + public UnsupportedRouteController(PlatformAudio owner, string platform) + { + _owner = owner; + _platform = platform; + } + + public (List Recording, List Playout) GetDevices() + { + return _owner.GetDevicesViaFfi(); + } + + public void ApplyOutputPreference(IReadOnlyList ranked) + { + // Stored by PlatformAudio; no routing effect until this platform's backend lands. + } + + public void SelectOutput(AudioDevice device) + { + throw new NotSupportedException( + $"SelectOutput is not supported on {_platform}"); + } + + public void ClearOutputOverride() + { + // No override can exist on this platform: SelectOutput throws. + } + + public void SetSessionAudioEnabled(bool enabled) + { + // Nothing to gate: this platform has no routing backend holding a session. + } + + public event Action, IReadOnlyList> DevicesChanged + { + add { } + remove { } + } + + public void Dispose() + { + } + } +} diff --git a/Runtime/Scripts/Audio/RouteController.cs.meta b/Runtime/Scripts/Audio/RouteController.cs.meta new file mode 100644 index 00000000..82c5747b --- /dev/null +++ b/Runtime/Scripts/Audio/RouteController.cs.meta @@ -0,0 +1,11 @@ +fileFormatVersion: 2 +guid: 3a99955b361ca4e5aa765a7e6dfc9e73 +MonoImporter: + externalObjects: {} + serializedVersion: 2 + defaultReferences: [] + executionOrder: 0 + icon: {instanceID: 0} + userData: + assetBundleName: + assetBundleVariant: diff --git a/Samples~/Agents/Assets/Build.meta b/Samples~/Agents/Assets/Build.meta new file mode 100644 index 00000000..b3c9e97a --- /dev/null +++ b/Samples~/Agents/Assets/Build.meta @@ -0,0 +1,8 @@ +fileFormatVersion: 2 +guid: b0d7497a6950e4f02aa4b812de50fd9f +folderAsset: yes +DefaultImporter: + externalObjects: {} + userData: + assetBundleName: + assetBundleVariant: diff --git a/Samples~/Agents/Assets/Plugins/Android/AndroidManifest.xml b/Samples~/Agents/Assets/Plugins/Android/AndroidManifest.xml index abc4bbc0..de1869eb 100644 --- a/Samples~/Agents/Assets/Plugins/Android/AndroidManifest.xml +++ b/Samples~/Agents/Assets/Plugins/Android/AndroidManifest.xml @@ -7,6 +7,7 @@ + wired headset > speaker > +// earpiece) and keeps the route pinned across device changes while a call is in +// progress. This controller only demonstrates the observability side by logging +// DevicesChanged. On Android an active mic capture is what keeps the SDK's route +// authoritative (since Android 13 the OS only honors an app's communication-mode request +// while it has active voice-communication capture), so start the capture with +// StartCapture when the call begins (even when joining muted) — it then stays open +// across mute cycles until StopCapture when the call ends. See StartCapture and +// Unpublish. +// +// The call audio session itself is gated by SetSessionAudioEnabled: the ADM is created +// once at app start and kept alive, but the platform's call session is only held for the +// duration of a call. See Initialize and SetSessionAudioEnabled. public sealed class PlatformAudioController : IDisposable { - const string MicTrackName = "player-mic"; + readonly string _trackName; + readonly AudioProcessingOptions _audioOptions; PlatformAudio _platformAudio; PlatformAudioSource _source; LocalAudioTrack _track; Room _room; + bool _isRecording; + // What should be audible after an output device change, remembered from before it. + readonly Dictionary _audibleSources = new Dictionary(); + // Whether a remember pass has ever swept the scene; gates the adopt-loops fallback + // in RestartAudibleSources to the very first switch. + bool _sceneSwept; + public bool IsInitialized => _platformAudio != null; public bool IsPublished { get; private set; } + public PlatformAudioController(string trackName, AudioProcessingOptions audioOptions) + { + _trackName = trackName; + _audioOptions = audioOptions; + } + // Creates the WebRTC ADM. This MUST run before Room.Connect so the SDK wires automatic - // speaker playout for remote tracks to this ADM — otherwise remote (agent) audio is never + // speaker playout for remote tracks to this ADM — otherwise remote audio is never // routed to an output and stays silent. Returns false if the ADM could not be created. public bool Initialize() { - return InitializePlatformAudio(); + if (!InitializePlatformAudio()) + return false; + + // The SDK routes output automatically from here on; the default + // PlatformAudio.OutputPreference ranking is already what a call app wants. + // A custom ranking would be a one-liner: + // _platformAudio.OutputPreference = new[] { AudioOutputKind.WiredHeadset, AudioOutputKind.Speaker }; + _platformAudio.DevicesChanged += OnDevicesChanged; + AudioSettings.OnAudioConfigurationChanged += OnUnityAudioConfigurationChanged; + + // Session audio defaults to enabled, so declare "no call yet" right away: the + // platform's call audio session should only be held while a call is actually in + // progress — enabled means "in a call". On iOS this drops the session to its + // music-friendly idle state; on Android — where the session is only taken by + // the first action that needs it, never by construction — it keeps a later + // routing action from taking the call session outside a call. The caller must + // re-enable it when its call starts and disable it again when the call ends + // (MeetManager does so on join/leave, LiveKitAgentSession around + // Connect/EndSession). + _platformAudio.SetSessionAudioEnabled(false); + return true; } - // Starts recording and publishes the mic track into the room. Initialize() must have been - // called (before the room connected) first. On any failure it disposes whatever was - // constructed and leaves IsPublished false; the caller should tear the rest down. + // Starts recording and publishes the mic track into the room. Initialize() must have + // been called (before the room connected) first. On any failure it unpublishes whatever + // was constructed and leaves IsPublished false; the ADM stays alive so a later Publish + // can retry. public IEnumerator Publish(Room room) { _room = room; @@ -39,19 +92,18 @@ public IEnumerator Publish(Room room) Debug.LogError("[PlatformAudioController] Publish called before Initialize(); aborting."); yield break; } + if (IsPublished) + yield break; - // Begin capturing from the default microphone. On macOS/iOS this turns on the - // recording privacy indicator and triggers the OS permission prompt; on Android - // it awaits the RECORD_AUDIO runtime permission dialog. - Debug.Log("[PlatformAudioController] Starting platform recording."); - yield return _platformAudio.StartRecording(); + // No-op when StartCapture already ran at call start (the normal case on + // Android) or when the capture was kept running across a mute cycle (see + // Unpublish). + yield return StartCapture(); - // AudioProcessingOptions.Default enables AEC, noise suppression, auto gain control - // and prefers hardware processing. - _source = new PlatformAudioSource(_platformAudio, AudioProcessingOptions.Default); - _track = LocalAudioTrack.CreateAudioTrack(MicTrackName, _source, _room); + _source = new PlatformAudioSource(_platformAudio, _audioOptions); + _track = LocalAudioTrack.CreateAudioTrack(_trackName, _source, _room); - Debug.Log($"[PlatformAudioController] Publishing mic track '{MicTrackName}'..."); + Debug.Log($"[PlatformAudioController] Publishing mic track '{_trackName}'..."); var options = new TrackPublishOptions { AudioEncoding = new AudioEncoding { MaxBitrate = 64000 }, @@ -62,12 +114,96 @@ public IEnumerator Publish(Room room) if (publish.IsError) { Debug.LogError("[PlatformAudioController] Failed to publish microphone track."); - Dispose(); + Unpublish(); yield break; } IsPublished = true; - Debug.Log("[PlatformAudioController] Microphone track published (AEC enabled)."); + Debug.Log($"[PlatformAudioController] Microphone track '{_trackName}' published."); + } + + // Starts the microphone capture without publishing a track; Publish() reuses the + // running capture. On macOS this turns on the recording privacy indicator; on iOS + // and Android the coroutine first awaits the OS microphone-permission dialog and + // only then opens the capture. On Android call this as soon as the call starts, even when + // joining muted: since Android 13 the app's communication-mode request — and with + // it the SDK's output route pin — is only honored while the app has ACTIVE + // voice-communication capture or playback, and the ADM's playout stream does not + // register as active, only the recorder does. The SDK re-asserts its routing policy + // whenever the capture (re)starts. + public IEnumerator StartCapture() + { + if (_platformAudio == null) + { + Debug.LogError("[PlatformAudioController] StartCapture called before Initialize(); aborting."); + yield break; + } + if (_isRecording) + yield break; + + Debug.Log("[PlatformAudioController] Starting platform recording."); + yield return _platformAudio.StartRecording(); + _isRecording = true; + } + + // Tears down the mic capture and track but keeps the ADM alive: remote playout + // continues and a later Publish() reuses it (e.g. a mute/unmute toggle). + public void Unpublish() + { + IsPublished = false; + + if (_track != null && _room != null) + { + Debug.Log("[PlatformAudioController] Unpublishing microphone track."); + _room.LocalParticipant.UnpublishTrack(_track, stopOnUnpublish: false); + } + _track = null; + +#if UNITY_ANDROID && !UNITY_EDITOR + // Keep the capture stream open while muted. Since Android 13, AudioService only + // honors this app's communication-mode request — and with it the SDK's output + // route pin — while the app has ACTIVE voice-communication capture or playback: + // with the recorder stopped, the mode drops back to MODE_NORMAL and the platform + // re-asserts the earpiece route. The track is unpublished and its source + // disposed below, so no audio reaches the room, but the OS mic-in-use indicator + // stays on while muted — same as other conferencing apps. Recording stops in + // StopCapture (call end) or Dispose. +#else + StopCapture(); +#endif + + _source?.Dispose(); + _source = null; + } + + // Gates the platform's call audio session: enabled means a call is in progress. + // On iOS it switches WebRTC's VPIO unit on/off while the app keeps ownership of the + // audio session, on Android 12+ it takes and releases the communication mode plus + // the SDK's output route pin — both so other Unity audio (e.g. background music) + // keeps playing outside a call. Call with true after joining a room and false when + // leaving it; Initialize() already disabled it for the idle app. + public void SetSessionAudioEnabled(bool enabled) + { + _platformAudio?.SetSessionAudioEnabled(enabled); + } + + // Stops the microphone capture if it is running. Only call this once the call has + // ended (after Unpublish): on Android, stopping the capture while still in a call + // hands routing authority back to the platform — see StartCapture. The next + // StartCapture (or Publish) restarts it. + public void StopCapture() + { + if (_platformAudio == null || !_isRecording) + return; + try + { + _platformAudio.StopRecording(); + } + catch (Exception e) + { + Debug.LogWarning($"[PlatformAudioController] Failed to stop recording: {e.Message}"); + } + _isRecording = false; } // Sets up PlatformAudio with the default recording/playout devices. @@ -80,6 +216,9 @@ bool InitializePlatformAudio() $"[PlatformAudioController] PlatformAudio initialized " + $"({_platformAudio.RecordingDeviceCount} mic(s), {_platformAudio.PlayoutDeviceCount} speaker(s))."); + var (recording, playout) = _platformAudio.GetDevices(); + Debug.Log(FormatDeviceLists(playout, recording)); + if (_platformAudio.RecordingDeviceCount > 0) _platformAudio.SetRecordingDevice(0); if (_platformAudio.PlayoutDeviceCount > 0) @@ -96,34 +235,163 @@ bool InitializePlatformAudio() } } - public void Dispose() + // Demonstrates the SDK's routing observability: the routing backend raises + // DevicesChanged (on the Unity main thread) whenever the available devices or the + // active route change — headset plugged/unplugged, Bluetooth connected, the route + // re-pinned after a device disappeared. An app would refresh its device picker here. + // This sample also uses it as the early warning for the Unity-audio recovery below. + void OnDevicesChanged(IReadOnlyList playout, IReadOnlyList recording) { - IsPublished = false; + Debug.Log("[PlatformAudioController] Audio devices changed.\n" + + FormatDeviceLists(playout, recording)); - if (_track != null && _room != null) + // Note what is audible while the engine is still healthy, but do not touch it: + // this event arrives early in a device switch (device-verified on a Pixel 8a / + // Android 16, where a Bluetooth headset's call profile appears ~650 ms before the + // media profile takes over), so reopening the engine here would reopen it onto the + // output the platform is about to leave. + RememberAudibleSources(forgetStopped: true); + } + + // Unity's audio engine opens an output device when the app starts. When that device + // goes away or another one takes over (Bluetooth connect or disconnect, wired + // plug/unplug), Unity reinitializes the engine, which stops every AudioSource, and + // raises this callback afterwards — device-verified on a Pixel 8a (Android 16): + // + // AudioTrack stop(11092): called with 92104 frames delivered <- sources stopped + // [PlatformAudioController] Unity audio configuration changed <- 25 ms later + // + // So the app has to restart its audio here, and it cannot learn what to restart from + // the scene at this point: everything is already stopped. What should be audible has + // to be remembered from before the switch (RememberAudibleSources) and put back now. + // Leaving that out is exactly how game audio ends up silent on the new device. + // + // deviceWasChanged is false even for a real device change on Android, so it cannot be + // used to filter these callbacks; the recovery reacts to all of them and stays safe + // through idempotence instead (a source already playing is left alone). + void OnUnityAudioConfigurationChanged(bool deviceWasChanged) + { + Debug.Log("[PlatformAudioController] Unity audio configuration changed " + + $"(deviceWasChanged={deviceWasChanged}, outputSampleRate={AudioSettings.outputSampleRate}, " + + $"speakerMode={AudioSettings.speakerMode})."); + + // Restore FIRST: the engine reinit has already stopped every source, so a + // remember pass at this point would see nothing playing and forget the very + // sources — one-shots above all — that it is supposed to put back. The refresh + // afterwards carries the restarted positions into the next switch without + // evicting anything, so a Play() the engine rejected mid-teardown keeps its + // slot for the next callback of the same switch. + RestartAudibleSources(); + RememberAudibleSources(forgetStopped: false); + } + + // Records what this sample intends to keep audible, so a device change can put it + // back. With forgetStopped, a source that is not playing is dropped from the set — + // loops included: that pass runs while the engine is healthy (OnDevicesChanged fires + // before the engine reinit), so a stopped source there was stopped by the app or has + // finished, and a deliberate Stop() must not be undone by the next device change. + // Without it, the pass only refreshes positions and adopts survivors — used right + // after a restore, when a Play() the engine rejected must not cost a source its + // slot. An app would consult its own audio state here instead of sweeping the scene. + void RememberAudibleSources(bool forgetStopped) + { + foreach (var source in UnityEngine.Object.FindObjectsByType(FindObjectsSortMode.None)) { - Debug.Log("[PlatformAudioController] Unpublishing microphone track."); - _room.LocalParticipant.UnpublishTrack(_track, stopOnUnpublish: false); + if (source.isPlaying) + _audibleSources[source] = source.time; + else if (forgetStopped) + _audibleSources.Remove(source); } - _track = null; + _sceneSwept = true; + } - if (_platformAudio != null) + // Puts the remembered audio back on the reopened engine. Note what this does NOT do: + // it never calls AudioSettings.Reset. Unity has already reopened its output by the + // time it raises the callback, so a reset adds nothing — and on Android it does real + // harm. Device-verified on a Pixel 8a (Android 16): reinitializing the engine makes + // Unity claim the headset's call link through the deprecated + // AudioManager.startBluetoothSco(), which evicts the SDK's setCommunicationDevice pin + // and leaves the platform's SCO state machine unable to connect — + // + // AS.AudioDeviceBroker: setCommunicationRouteForClient … type:bt_sco addr: + // … from API: startBluetoothSco()) from u/pid:… <- evicts our pinned device + // AS.BtHelper: requestScoState: failed to connect in state 1 <- every retry after + // + // after which call audio and game audio are both stuck on the loudspeaker for the + // rest of the session, however often the SDK re-pins the route. + void RestartAudibleSources() + { + // First-switch safety net: when no remember pass has ever swept the scene, a + // stopped looping source is taken to have been stopped by the engine reinit and + // is adopted rather than left silent. Once a sweep has run, an absent loop is + // one the app stopped (or never started), and adopting it would undo the app's + // intent. + if (_audibleSources.Count == 0 && !_sceneSwept) + foreach (var source in UnityEngine.Object.FindObjectsByType(FindObjectsSortMode.None)) + if (source.loop) + _audibleSources[source] = 0f; + + var restarted = 0; + // Copied because the loop drops destroyed sources from the dictionary. + foreach (var entry in new List>(_audibleSources)) { - try + var source = entry.Key; + if (source == null) { - _platformAudio.StopRecording(); + _audibleSources.Remove(source); + continue; } - catch (Exception e) + // A source the app deactivated is treated like one it stopped: forgotten, + // not retried. Play() on a disabled source only logs a warning on every + // callback, and force-playing it after a reactivation would undo the app's + // intent. + if (!source.isActiveAndEnabled) { - Debug.LogWarning($"[PlatformAudioController] Failed to stop recording: {e.Message}"); + _audibleSources.Remove(source); + continue; } + // Idempotent on purpose: a device switch raises several of these callbacks, + // and anything Unity left running must be left alone. + if (source.isPlaying) continue; + + if (source.clip != null) + source.time = Mathf.Clamp(entry.Value, 0f, Mathf.Max(0f, source.clip.length - 0.05f)); + source.Play(); + if (source.isPlaying) restarted++; } - _source?.Dispose(); - _source = null; + Debug.Log($"[PlatformAudioController] Restarted {restarted} of {_audibleSources.Count} remembered " + + $"source(s) on {AudioSettings.speakerMode} @ {AudioSettings.outputSampleRate} Hz."); + } + + static string FormatDeviceLists(IReadOnlyList playout, IReadOnlyList recording) + { + var sb = new StringBuilder("Playout devices:"); + foreach (var device in playout) + { + sb.Append($"\n [{device.Index}] {device.Name} (kind={device.Kind}"); + if (device.IsSelected) + sb.Append(", selected"); + sb.Append(')'); + } + sb.Append("\nRecording devices:"); + foreach (var device in recording) + sb.Append($"\n [{device.Index}] {device.Name}"); + return sb.ToString(); + } + + public void Dispose() + { + Unpublish(); + StopCapture(); - _platformAudio?.Dispose(); - _platformAudio = null; + if (_platformAudio != null) + { + AudioSettings.OnAudioConfigurationChanged -= OnUnityAudioConfigurationChanged; + _platformAudio.DevicesChanged -= OnDevicesChanged; + _platformAudio.Dispose(); + _platformAudio = null; + } _room = null; } diff --git a/Samples~/Meet/Assets/Plugins/Android/AndroidManifest.xml b/Samples~/Meet/Assets/Plugins/Android/AndroidManifest.xml index 94ea9440..e5cca6ac 100644 --- a/Samples~/Meet/Assets/Plugins/Android/AndroidManifest.xml +++ b/Samples~/Meet/Assets/Plugins/Android/AndroidManifest.xml @@ -7,6 +7,7 @@ + 0) - _platformAudio.SetRecordingDevice(0); - if (_platformAudio.PlayoutDeviceCount > 0) - _platformAudio.SetPlayoutDevice(0); + EchoCancellation = echoCancellation, + NoiseSuppression = noiseSuppression, + AutoGainControl = autoGainControl, + PreferHardware = preferHardwareProcessing + }; - Debug.Log($"PlatformAudio ready. AEC={echoCancellation}, NS={noiseSuppression}, AGC={autoGainControl}, HW={preferHardwareProcessing}"); - } - catch (System.Exception e) + _platformAudioController = new PlatformAudioController(LocalAudioTrackName, audioOptions); + if (!_platformAudioController.Initialize()) { - Debug.LogError($"Failed to initialize PlatformAudio, falling back to Unity audio: {e.Message}"); + Debug.LogError("Failed to initialize PlatformAudio, falling back to Unity audio"); usePlatformAudio = false; - _platformAudio = null; + _platformAudioController = null; + return; } + + Debug.Log($"PlatformAudio ready. AEC={echoCancellation}, NS={noiseSuppression}, AGC={autoGainControl}, HW={preferHardwareProcessing}"); } private void OnApplicationPause(bool pause) @@ -150,8 +139,7 @@ private void OnDestroy() } CleanUpAllTracks(); _webCamTexture?.Stop(); - _platformAudioSource?.Dispose(); - _platformAudio?.Dispose(); + _platformAudioController?.Dispose(); _room?.Disconnect(); } @@ -168,11 +156,25 @@ private void OnEndCall() { if (_room == null) return; + // Disable call audio while keeping the app-owned audio session active, so + // Unity audio (e.g. background music) survives the hang-up on iOS. + if (usePlatformAudio) + _platformAudioController?.SetSessionAudioEnabled(false); + _room.Disconnect(); + TeardownCall(); + } + + // Shared end-of-call teardown: stops the mic capture and the tracks + // (CleanUpAllTracks) and resets the call state and UI. Every part is safe to run + // twice — OnEndCall's own Disconnect can also raise OnDisconnected. + private void TeardownCall() + { CleanUpAllTracks(); _room = null; _localId = null; - buttonBar.SetConnected(false); + if (buttonBar != null) + buttonBar.SetConnected(false); } private void OnToggleCamera() @@ -249,6 +251,22 @@ private IEnumerator ConnectToRoom() _localId = _room.LocalParticipant.Identity; buttonBar.SetConnected(true); + // Enable call audio now that we're in a room. On iOS this turns on WebRTC's + // VPIO unit while the app keeps ownership of the audio session; leaving the + // room disables it again (see OnEndCall / OnDisconnected) so other Unity + // audio keeps playing. + if (usePlatformAudio) + _platformAudioController?.SetSessionAudioEnabled(true); + +#if UNITY_ANDROID && !UNITY_EDITOR + // Keep the mic capture running for the whole call, even while muted: without + // an active capture Android treats the communication-mode request as inactive + // and the SDK's output route pin is not honored after a Bluetooth episode — see + // PlatformAudioController.StartCapture. Publishing (unmuting) reuses the capture. + if (usePlatformAudio && _platformAudioController != null) + StartCoroutine(_platformAudioController.StartCapture()); +#endif + EnsureParticipantTile(_localId); foreach (var remote in _room.RemoteParticipants.Values) EnsureParticipantTile(remote.Identity); @@ -360,7 +378,7 @@ private void AddRemoteAudioTrack(RemoteAudioTrack audioTrack) { var sid = audioTrack.Sid; - if (usePlatformAudio && _platformAudio != null) + if (usePlatformAudio && _platformAudioController != null) { // PlatformAudio mode: ADM handles speaker playback automatically. // No AudioStream / GameObject needed. @@ -433,7 +451,19 @@ private void OnParticipantDisconnected(Participant participant, DisconnectReason } private void OnDisconnected(Room room) - => Debug.Log($"Disconnected from room: {room.DisconnectReason}"); + { + Debug.Log($"Disconnected from room: {room.DisconnectReason}"); + + // Covers server-initiated disconnects (kick, room deleted, token expiry) as + // well as OnEndCall. Stopping the capture here matters: it is deliberately + // kept running across mute cycles, so without the teardown a server-side + // disconnect would leave the microphone recording — indicator on — with no + // call to feed. The audio session itself stays active for Unity. + if (usePlatformAudio) + _platformAudioController?.SetSessionAudioEnabled(false); + + TeardownCall(); + } private void OnTrackMuted(TrackPublication publication, Participant participant) { @@ -549,7 +579,7 @@ private IEnumerator PublishLocalMicrophone() { if (_microphoneActive) yield break; - if (usePlatformAudio && _platformAudio != null) + if (usePlatformAudio && _platformAudioController != null) yield return PublishLocalMicrophonePlatform(); else yield return PublishLocalMicrophoneUnity(); @@ -562,45 +592,8 @@ private IEnumerator PublishLocalMicrophonePlatform() { Debug.Log("Publishing microphone using PlatformAudio (ADM)"); - // Start recording (in case it was stopped by a previous mute). - // This turns on the privacy indicator on macOS/iOS. On Android this also - // awaits the RECORD_AUDIO runtime permission dialog if not yet granted. - if (_platformAudio != null) - { - yield return _platformAudio.StartRecording(); - } - - var audioOptions = new AudioProcessingOptions - { - EchoCancellation = echoCancellation, - NoiseSuppression = noiseSuppression, - AutoGainControl = autoGainControl, - PreferHardware = preferHardwareProcessing - }; - - _platformAudioSource = new PlatformAudioSource(_platformAudio, audioOptions); - _localAudioTrack = LocalAudioTrack.CreateAudioTrack(LocalAudioTrackName, _platformAudioSource, _room); - - var options = new TrackPublishOptions - { - AudioEncoding = new AudioEncoding { MaxBitrate = 64000 }, - Source = TrackSource.SourceMicrophone - }; - - var publish = _room.LocalParticipant.PublishTrack(_localAudioTrack, options); - yield return publish; - - if (publish.IsError) - { - Debug.LogError("Failed to publish microphone track"); - _platformAudioSource?.Dispose(); - _platformAudioSource = null; - _localAudioTrack = null; - yield break; - } - - _microphoneActive = true; - Debug.Log("Microphone published via PlatformAudio (AEC enabled)"); + yield return _platformAudioController.Publish(_room); + _microphoneActive = _platformAudioController.IsPublished; } private IEnumerator PublishLocalMicrophoneUnity() @@ -643,19 +636,11 @@ private IEnumerator PublishLocalMicrophoneUnity() private void UnpublishLocalMicrophone() { - if (usePlatformAudio && _platformAudioSource != null) + if (usePlatformAudio && _platformAudioController != null) { - try - { - _platformAudio?.StopRecording(); - } - catch (System.Exception e) - { - Debug.LogWarning($"Failed to stop recording: {e.Message}"); - } - - _platformAudioSource.Dispose(); - _platformAudioSource = null; + // The controller owns the platform track: this stops recording and + // unpublishes while keeping the ADM alive for the next unmute. + _platformAudioController.Unpublish(); } else { @@ -670,10 +655,11 @@ private void UnpublishLocalMicrophone() } _audioObjects.Remove(LocalAudioTrackName); } + + _room.LocalParticipant.UnpublishTrack(_localAudioTrack, false); + _localAudioTrack = null; } - _room.LocalParticipant.UnpublishTrack(_localAudioTrack, false); - _localAudioTrack = null; if (_participantTiles.TryGetValue(_localId, out var tile)) tile.SetMicMuted(true); _microphoneActive = false; @@ -743,8 +729,11 @@ private void CleanUpAllTracks() DisposeSource(ref _localRtcAudioSource); DisposeSource(ref _localRtcVideoSource); - _platformAudioSource?.Dispose(); - _platformAudioSource = null; + // Keep the ADM itself alive so the next call can reuse it; only the mic + // capture and track go away here (ConnectToRoom restarts the capture on the + // next call). + _platformAudioController?.Unpublish(); + _platformAudioController?.StopCapture(); foreach (var obj in _audioObjects.Values) { diff --git a/Samples~/Meet/Assets/Runtime/PlatformAudioController.cs b/Samples~/Meet/Assets/Runtime/PlatformAudioController.cs new file mode 100644 index 00000000..ff7bb755 --- /dev/null +++ b/Samples~/Meet/Assets/Runtime/PlatformAudioController.cs @@ -0,0 +1,398 @@ +using System; +using System.Collections; +using System.Collections.Generic; +using System.Text; +using LiveKit; +using LiveKit.Proto; +using UnityEngine; + +// Drives the duplex platform audio (WebRTC ADM): captures the default microphone with +// the configured audio processing (AEC/NS/AGC) and publishes it as a LiveKit track, and +// selects the default playout device through which remote tracks are played back +// automatically. Publish/Unpublish can be cycled (e.g. a mute toggle) while the ADM stays +// alive; Dispose tears everything down in dependency order. +// +// Output routing is owned by the SDK: PlatformAudio routes to the best available output +// per its ranked OutputPreference (default: Bluetooth > wired headset > speaker > +// earpiece) and keeps the route pinned across device changes while a call is in +// progress. This controller only demonstrates the observability side by logging +// DevicesChanged. On Android an active mic capture is what keeps the SDK's route +// authoritative (since Android 13 the OS only honors an app's communication-mode request +// while it has active voice-communication capture), so start the capture with +// StartCapture when the call begins (even when joining muted) — it then stays open +// across mute cycles until StopCapture when the call ends. See StartCapture and +// Unpublish. +// +// The call audio session itself is gated by SetSessionAudioEnabled: the ADM is created +// once at app start and kept alive, but the platform's call session is only held for the +// duration of a call. See Initialize and SetSessionAudioEnabled. +public sealed class PlatformAudioController : IDisposable +{ + readonly string _trackName; + readonly AudioProcessingOptions _audioOptions; + + PlatformAudio _platformAudio; + PlatformAudioSource _source; + LocalAudioTrack _track; + Room _room; + bool _isRecording; + // What should be audible after an output device change, remembered from before it. + readonly Dictionary _audibleSources = new Dictionary(); + // Whether a remember pass has ever swept the scene; gates the adopt-loops fallback + // in RestartAudibleSources to the very first switch. + bool _sceneSwept; + + public bool IsInitialized => _platformAudio != null; + public bool IsPublished { get; private set; } + + public PlatformAudioController(string trackName, AudioProcessingOptions audioOptions) + { + _trackName = trackName; + _audioOptions = audioOptions; + } + + // Creates the WebRTC ADM. This MUST run before Room.Connect so the SDK wires automatic + // speaker playout for remote tracks to this ADM — otherwise remote audio is never + // routed to an output and stays silent. Returns false if the ADM could not be created. + public bool Initialize() + { + if (!InitializePlatformAudio()) + return false; + + // The SDK routes output automatically from here on; the default + // PlatformAudio.OutputPreference ranking is already what a call app wants. + // A custom ranking would be a one-liner: + // _platformAudio.OutputPreference = new[] { AudioOutputKind.WiredHeadset, AudioOutputKind.Speaker }; + _platformAudio.DevicesChanged += OnDevicesChanged; + AudioSettings.OnAudioConfigurationChanged += OnUnityAudioConfigurationChanged; + + // Session audio defaults to enabled, so declare "no call yet" right away: the + // platform's call audio session should only be held while a call is actually in + // progress — enabled means "in a call". On iOS this drops the session to its + // music-friendly idle state; on Android — where the session is only taken by + // the first action that needs it, never by construction — it keeps a later + // routing action from taking the call session outside a call. The caller must + // re-enable it when its call starts and disable it again when the call ends + // (MeetManager does so on join/leave, LiveKitAgentSession around + // Connect/EndSession). + _platformAudio.SetSessionAudioEnabled(false); + return true; + } + + // Starts recording and publishes the mic track into the room. Initialize() must have + // been called (before the room connected) first. On any failure it unpublishes whatever + // was constructed and leaves IsPublished false; the ADM stays alive so a later Publish + // can retry. + public IEnumerator Publish(Room room) + { + _room = room; + + if (_platformAudio == null) + { + Debug.LogError("[PlatformAudioController] Publish called before Initialize(); aborting."); + yield break; + } + if (IsPublished) + yield break; + + // No-op when StartCapture already ran at call start (the normal case on + // Android) or when the capture was kept running across a mute cycle (see + // Unpublish). + yield return StartCapture(); + + _source = new PlatformAudioSource(_platformAudio, _audioOptions); + _track = LocalAudioTrack.CreateAudioTrack(_trackName, _source, _room); + + Debug.Log($"[PlatformAudioController] Publishing mic track '{_trackName}'..."); + var options = new TrackPublishOptions + { + AudioEncoding = new AudioEncoding { MaxBitrate = 64000 }, + Source = TrackSource.SourceMicrophone + }; + var publish = _room.LocalParticipant.PublishTrack(_track, options); + yield return publish; + if (publish.IsError) + { + Debug.LogError("[PlatformAudioController] Failed to publish microphone track."); + Unpublish(); + yield break; + } + + IsPublished = true; + Debug.Log($"[PlatformAudioController] Microphone track '{_trackName}' published."); + } + + // Starts the microphone capture without publishing a track; Publish() reuses the + // running capture. On macOS this turns on the recording privacy indicator; on iOS + // and Android the coroutine first awaits the OS microphone-permission dialog and + // only then opens the capture. On Android call this as soon as the call starts, even when + // joining muted: since Android 13 the app's communication-mode request — and with + // it the SDK's output route pin — is only honored while the app has ACTIVE + // voice-communication capture or playback, and the ADM's playout stream does not + // register as active, only the recorder does. The SDK re-asserts its routing policy + // whenever the capture (re)starts. + public IEnumerator StartCapture() + { + if (_platformAudio == null) + { + Debug.LogError("[PlatformAudioController] StartCapture called before Initialize(); aborting."); + yield break; + } + if (_isRecording) + yield break; + + Debug.Log("[PlatformAudioController] Starting platform recording."); + yield return _platformAudio.StartRecording(); + _isRecording = true; + } + + // Tears down the mic capture and track but keeps the ADM alive: remote playout + // continues and a later Publish() reuses it (e.g. a mute/unmute toggle). + public void Unpublish() + { + IsPublished = false; + + if (_track != null && _room != null) + { + Debug.Log("[PlatformAudioController] Unpublishing microphone track."); + _room.LocalParticipant.UnpublishTrack(_track, stopOnUnpublish: false); + } + _track = null; + +#if UNITY_ANDROID && !UNITY_EDITOR + // Keep the capture stream open while muted. Since Android 13, AudioService only + // honors this app's communication-mode request — and with it the SDK's output + // route pin — while the app has ACTIVE voice-communication capture or playback: + // with the recorder stopped, the mode drops back to MODE_NORMAL and the platform + // re-asserts the earpiece route. The track is unpublished and its source + // disposed below, so no audio reaches the room, but the OS mic-in-use indicator + // stays on while muted — same as other conferencing apps. Recording stops in + // StopCapture (call end) or Dispose. +#else + StopCapture(); +#endif + + _source?.Dispose(); + _source = null; + } + + // Gates the platform's call audio session: enabled means a call is in progress. + // On iOS it switches WebRTC's VPIO unit on/off while the app keeps ownership of the + // audio session, on Android 12+ it takes and releases the communication mode plus + // the SDK's output route pin — both so other Unity audio (e.g. background music) + // keeps playing outside a call. Call with true after joining a room and false when + // leaving it; Initialize() already disabled it for the idle app. + public void SetSessionAudioEnabled(bool enabled) + { + _platformAudio?.SetSessionAudioEnabled(enabled); + } + + // Stops the microphone capture if it is running. Only call this once the call has + // ended (after Unpublish): on Android, stopping the capture while still in a call + // hands routing authority back to the platform — see StartCapture. The next + // StartCapture (or Publish) restarts it. + public void StopCapture() + { + if (_platformAudio == null || !_isRecording) + return; + try + { + _platformAudio.StopRecording(); + } + catch (Exception e) + { + Debug.LogWarning($"[PlatformAudioController] Failed to stop recording: {e.Message}"); + } + _isRecording = false; + } + + // Sets up PlatformAudio with the default recording/playout devices. + bool InitializePlatformAudio() + { + try + { + _platformAudio = new PlatformAudio(); + Debug.Log( + $"[PlatformAudioController] PlatformAudio initialized " + + $"({_platformAudio.RecordingDeviceCount} mic(s), {_platformAudio.PlayoutDeviceCount} speaker(s))."); + + var (recording, playout) = _platformAudio.GetDevices(); + Debug.Log(FormatDeviceLists(playout, recording)); + + if (_platformAudio.RecordingDeviceCount > 0) + _platformAudio.SetRecordingDevice(0); + if (_platformAudio.PlayoutDeviceCount > 0) + _platformAudio.SetPlayoutDevice(0); + + return true; + } + catch (Exception e) + { + Debug.LogError($"[PlatformAudioController] Failed to initialize PlatformAudio: {e.Message}"); + _platformAudio?.Dispose(); + _platformAudio = null; + return false; + } + } + + // Demonstrates the SDK's routing observability: the routing backend raises + // DevicesChanged (on the Unity main thread) whenever the available devices or the + // active route change — headset plugged/unplugged, Bluetooth connected, the route + // re-pinned after a device disappeared. An app would refresh its device picker here. + // This sample also uses it as the early warning for the Unity-audio recovery below. + void OnDevicesChanged(IReadOnlyList playout, IReadOnlyList recording) + { + Debug.Log("[PlatformAudioController] Audio devices changed.\n" + + FormatDeviceLists(playout, recording)); + + // Note what is audible while the engine is still healthy, but do not touch it: + // this event arrives early in a device switch (device-verified on a Pixel 8a / + // Android 16, where a Bluetooth headset's call profile appears ~650 ms before the + // media profile takes over), so reopening the engine here would reopen it onto the + // output the platform is about to leave. + RememberAudibleSources(forgetStopped: true); + } + + // Unity's audio engine opens an output device when the app starts. When that device + // goes away or another one takes over (Bluetooth connect or disconnect, wired + // plug/unplug), Unity reinitializes the engine, which stops every AudioSource, and + // raises this callback afterwards — device-verified on a Pixel 8a (Android 16): + // + // AudioTrack stop(11092): called with 92104 frames delivered <- sources stopped + // [PlatformAudioController] Unity audio configuration changed <- 25 ms later + // + // So the app has to restart its audio here, and it cannot learn what to restart from + // the scene at this point: everything is already stopped. What should be audible has + // to be remembered from before the switch (RememberAudibleSources) and put back now. + // Leaving that out is exactly how game audio ends up silent on the new device. + // + // deviceWasChanged is false even for a real device change on Android, so it cannot be + // used to filter these callbacks; the recovery reacts to all of them and stays safe + // through idempotence instead (a source already playing is left alone). + void OnUnityAudioConfigurationChanged(bool deviceWasChanged) + { + Debug.Log("[PlatformAudioController] Unity audio configuration changed " + + $"(deviceWasChanged={deviceWasChanged}, outputSampleRate={AudioSettings.outputSampleRate}, " + + $"speakerMode={AudioSettings.speakerMode})."); + + // Restore FIRST: the engine reinit has already stopped every source, so a + // remember pass at this point would see nothing playing and forget the very + // sources — one-shots above all — that it is supposed to put back. The refresh + // afterwards carries the restarted positions into the next switch without + // evicting anything, so a Play() the engine rejected mid-teardown keeps its + // slot for the next callback of the same switch. + RestartAudibleSources(); + RememberAudibleSources(forgetStopped: false); + } + + // Records what this sample intends to keep audible, so a device change can put it + // back. With forgetStopped, a source that is not playing is dropped from the set — + // loops included: that pass runs while the engine is healthy (OnDevicesChanged fires + // before the engine reinit), so a stopped source there was stopped by the app or has + // finished, and a deliberate Stop() must not be undone by the next device change. + // Without it, the pass only refreshes positions and adopts survivors — used right + // after a restore, when a Play() the engine rejected must not cost a source its + // slot. An app would consult its own audio state here instead of sweeping the scene. + void RememberAudibleSources(bool forgetStopped) + { + foreach (var source in UnityEngine.Object.FindObjectsByType(FindObjectsSortMode.None)) + { + if (source.isPlaying) + _audibleSources[source] = source.time; + else if (forgetStopped) + _audibleSources.Remove(source); + } + _sceneSwept = true; + } + + // Puts the remembered audio back on the reopened engine. Note what this does NOT do: + // it never calls AudioSettings.Reset. Unity has already reopened its output by the + // time it raises the callback, so a reset adds nothing — and on Android it does real + // harm. Device-verified on a Pixel 8a (Android 16): reinitializing the engine makes + // Unity claim the headset's call link through the deprecated + // AudioManager.startBluetoothSco(), which evicts the SDK's setCommunicationDevice pin + // and leaves the platform's SCO state machine unable to connect — + // + // AS.AudioDeviceBroker: setCommunicationRouteForClient … type:bt_sco addr: + // … from API: startBluetoothSco()) from u/pid:… <- evicts our pinned device + // AS.BtHelper: requestScoState: failed to connect in state 1 <- every retry after + // + // after which call audio and game audio are both stuck on the loudspeaker for the + // rest of the session, however often the SDK re-pins the route. + void RestartAudibleSources() + { + // First-switch safety net: when no remember pass has ever swept the scene, a + // stopped looping source is taken to have been stopped by the engine reinit and + // is adopted rather than left silent. Once a sweep has run, an absent loop is + // one the app stopped (or never started), and adopting it would undo the app's + // intent. + if (_audibleSources.Count == 0 && !_sceneSwept) + foreach (var source in UnityEngine.Object.FindObjectsByType(FindObjectsSortMode.None)) + if (source.loop) + _audibleSources[source] = 0f; + + var restarted = 0; + // Copied because the loop drops destroyed sources from the dictionary. + foreach (var entry in new List>(_audibleSources)) + { + var source = entry.Key; + if (source == null) + { + _audibleSources.Remove(source); + continue; + } + // A source the app deactivated is treated like one it stopped: forgotten, + // not retried. Play() on a disabled source only logs a warning on every + // callback, and force-playing it after a reactivation would undo the app's + // intent. + if (!source.isActiveAndEnabled) + { + _audibleSources.Remove(source); + continue; + } + // Idempotent on purpose: a device switch raises several of these callbacks, + // and anything Unity left running must be left alone. + if (source.isPlaying) continue; + + if (source.clip != null) + source.time = Mathf.Clamp(entry.Value, 0f, Mathf.Max(0f, source.clip.length - 0.05f)); + source.Play(); + if (source.isPlaying) restarted++; + } + + Debug.Log($"[PlatformAudioController] Restarted {restarted} of {_audibleSources.Count} remembered " + + $"source(s) on {AudioSettings.speakerMode} @ {AudioSettings.outputSampleRate} Hz."); + } + + static string FormatDeviceLists(IReadOnlyList playout, IReadOnlyList recording) + { + var sb = new StringBuilder("Playout devices:"); + foreach (var device in playout) + { + sb.Append($"\n [{device.Index}] {device.Name} (kind={device.Kind}"); + if (device.IsSelected) + sb.Append(", selected"); + sb.Append(')'); + } + sb.Append("\nRecording devices:"); + foreach (var device in recording) + sb.Append($"\n [{device.Index}] {device.Name}"); + return sb.ToString(); + } + + public void Dispose() + { + Unpublish(); + StopCapture(); + + if (_platformAudio != null) + { + AudioSettings.OnAudioConfigurationChanged -= OnUnityAudioConfigurationChanged; + _platformAudio.DevicesChanged -= OnDevicesChanged; + _platformAudio.Dispose(); + _platformAudio = null; + } + + _room = null; + } +} diff --git a/Samples~/Meet/Assets/Runtime/PlatformAudioController.cs.meta b/Samples~/Meet/Assets/Runtime/PlatformAudioController.cs.meta new file mode 100644 index 00000000..217b7816 --- /dev/null +++ b/Samples~/Meet/Assets/Runtime/PlatformAudioController.cs.meta @@ -0,0 +1,11 @@ +fileFormatVersion: 2 +guid: e60c87b3bbd504941ae86b78548a89d5 +MonoImporter: + externalObjects: {} + serializedVersion: 2 + defaultReferences: [] + executionOrder: 0 + icon: {instanceID: 0} + userData: + assetBundleName: + assetBundleVariant: diff --git a/Samples~/Meet/README.md b/Samples~/Meet/README.md index 864d185b..0a97bf63 100644 --- a/Samples~/Meet/README.md +++ b/Samples~/Meet/README.md @@ -18,6 +18,8 @@ In order to connect to your LiveKit server, configure the token source component The LiveKit Unity SDK offers two audio systems. The Unity audio path uses the Unity APIs for audio input and output. Platform Audio is the alternative, where the native LiveKit plugin manages audio input and output. You can select which path to use on the MeetManager component. +With Platform Audio, output routing (Bluetooth/wired headset/speaker/earpiece on mobile) is handled by the SDK's `PlatformAudio.OutputPreference` policy — the sample contains no routing code of its own and only logs the SDK's `DevicesChanged` events (see `PlatformAudioController`). The audio-routing section of the [SDK README](https://github.com/livekit/client-sdk-unity#audio-output-routing) documents the API and per-platform behavior. + ### Common sample package In order to get access to common sample functions like the on device scrolling log, make sure to import the [Common](https://github.com/livekit/client-sdk-unity/tree/main/Samples~/Common) sample from the LiveKit Unity Package in the package manager. \ No newline at end of file diff --git a/Tests/PlayMode/PlatformAudioTests.cs b/Tests/PlayMode/PlatformAudioTests.cs index 28b0ed7b..4a487a1c 100644 --- a/Tests/PlayMode/PlatformAudioTests.cs +++ b/Tests/PlayMode/PlatformAudioTests.cs @@ -102,6 +102,190 @@ public IEnumerator SetRecordingDeviceByIndex_OutOfRange_Throws() yield break; } + [UnityTest] + public IEnumerator OutputPreference_DefaultsAndRoundtrips() + { + using var platformAudio = PlatformAudioTestHelper.TryCreateOrIgnore(); + + // Documented default ranking. + CollectionAssert.AreEqual( + new[] + { + AudioOutputKind.Bluetooth, + AudioOutputKind.WiredHeadset, + AudioOutputKind.Speaker, + AudioOutputKind.Earpiece, + }, + platformAudio.OutputPreference); + + // Set/get roundtrip preserves order and content. + var ranked = new[] { AudioOutputKind.Usb, AudioOutputKind.Speaker, AudioOutputKind.Bluetooth }; + platformAudio.OutputPreference = ranked; + CollectionAssert.AreEqual(ranked, platformAudio.OutputPreference); + + yield break; + } + + [UnityTest] + public IEnumerator OutputPreference_RejectsInvalidLists() + { + using var platformAudio = PlatformAudioTestHelper.TryCreateOrIgnore(); + + Assert.Throws(() => platformAudio.OutputPreference = null); + Assert.Throws(() => + platformAudio.OutputPreference = new[] { AudioOutputKind.Unknown }); + Assert.Throws(() => + platformAudio.OutputPreference = new[] { AudioOutputKind.Speaker, AudioOutputKind.Speaker }); + + // A rejected assignment leaves the stored preference untouched. + CollectionAssert.AreEqual( + new[] + { + AudioOutputKind.Bluetooth, + AudioOutputKind.WiredHeadset, + AudioOutputKind.Speaker, + AudioOutputKind.Earpiece, + }, + platformAudio.OutputPreference); + + yield break; + } + + [UnityTest] + public IEnumerator SpeakerPreference_BoolAndListOrderAreOneState() + { + using var platformAudio = PlatformAudioTestHelper.TryCreateOrIgnore(); + + // Default ranking has Speaker ahead of Earpiece. + Assert.IsTrue(platformAudio.IsSpeakerOutputPreferred); + + // Setting the bool rewrites the Speaker/Earpiece order inside the list. + platformAudio.IsSpeakerOutputPreferred = false; + CollectionAssert.AreEqual( + new[] + { + AudioOutputKind.Bluetooth, + AudioOutputKind.WiredHeadset, + AudioOutputKind.Earpiece, + AudioOutputKind.Speaker, + }, + platformAudio.OutputPreference); + Assert.IsFalse(platformAudio.IsSpeakerOutputPreferred); + + // Setting the list order flips the bool back — the list is the source of truth. + platformAudio.OutputPreference = new[] + { + AudioOutputKind.Speaker, + AudioOutputKind.Earpiece, + AudioOutputKind.Bluetooth, + }; + Assert.IsTrue(platformAudio.IsSpeakerOutputPreferred); + + // A missing kind is inserted next to the present one so the value round-trips. + platformAudio.OutputPreference = new[] { AudioOutputKind.Bluetooth, AudioOutputKind.Speaker }; + platformAudio.IsSpeakerOutputPreferred = false; + CollectionAssert.AreEqual( + new[] { AudioOutputKind.Bluetooth, AudioOutputKind.Earpiece, AudioOutputKind.Speaker }, + platformAudio.OutputPreference); + Assert.IsFalse(platformAudio.IsSpeakerOutputPreferred); + + yield break; + } + + [UnityTest] + public IEnumerator SelectOutput_BogusDevice_Throws() + { + using var platformAudio = PlatformAudioTestHelper.TryCreateOrIgnore(); + + var bogus = new AudioDevice { Index = 9999, Name = "not-a-device", Guid = "no-such-guid" }; + Assert.Throws(() => platformAudio.SelectOutput(bogus)); + + // Clearing is always safe, whether or not an override exists. + Assert.DoesNotThrow(() => platformAudio.ClearOutputOverride()); + + yield break; + } + + [UnityTest] + public IEnumerator DevicesChanged_SubscribeUnsubscribe_SafeAcrossDispose() + { + var platformAudio = PlatformAudioTestHelper.TryCreateOrIgnore(); + + Action, IReadOnlyList> handler = (playout, recording) => { }; + platformAudio.DevicesChanged += handler; + platformAudio.Dispose(); + + Assert.DoesNotThrow(() => platformAudio.DevicesChanged -= handler); + Assert.DoesNotThrow(() => platformAudio.DevicesChanged += handler); + Assert.DoesNotThrow(() => platformAudio.Dispose()); + + yield break; + } + + [UnityTest] + public IEnumerator CreateDisposeCreate_OneSession_Works() + { + // The native ADM is ref-counted across PlatformAudio instances; after a full + // dispose the count must have returned to zero cleanly so a later instance in + // the same session comes up working (an app's second call after tearing the + // first one down). + var first = PlatformAudioTestHelper.TryCreateOrIgnore(); + first.OutputPreference = new[] { AudioOutputKind.Usb }; + first.Dispose(); + + using var second = new PlatformAudio(); + Assert.DoesNotThrow(() => second.GetDevices()); + + // Preference state is per instance: the first instance's mutation must not + // leak into the fresh one. + CollectionAssert.AreEqual( + new[] + { + AudioOutputKind.Bluetooth, + AudioOutputKind.WiredHeadset, + AudioOutputKind.Speaker, + AudioOutputKind.Earpiece, + }, + second.OutputPreference); + + yield break; + } + + [UnityTest] + public IEnumerator PublicMembers_AfterDispose_ThrowObjectDisposed() + { + var platformAudio = PlatformAudioTestHelper.TryCreateOrIgnore(); + platformAudio.Dispose(); + + Assert.Throws(() => _ = platformAudio.RecordingDeviceCount); + Assert.Throws(() => _ = platformAudio.PlayoutDeviceCount); + Assert.Throws(() => platformAudio.GetDevices()); + Assert.Throws(() => _ = platformAudio.OutputPreference); + Assert.Throws(() => + platformAudio.OutputPreference = new[] { AudioOutputKind.Speaker }); + Assert.Throws(() => _ = platformAudio.IsSpeakerOutputPreferred); + Assert.Throws(() => platformAudio.IsSpeakerOutputPreferred = true); + Assert.Throws(() => + platformAudio.SelectOutput(new AudioDevice { Index = 0, Name = "any" })); + Assert.Throws(() => platformAudio.ClearOutputOverride()); + Assert.Throws(() => platformAudio.SetRecordingDevice((uint)0)); + Assert.Throws(() => platformAudio.SetRecordingDevice("")); + Assert.Throws(() => platformAudio.SetPlayoutDevice((uint)0)); + Assert.Throws(() => platformAudio.SetPlayoutDevice("")); + Assert.Throws(() => platformAudio.StopRecording()); + Assert.Throws(() => platformAudio.SetSessionAudioEnabled(true)); + + // StartRecording is an iterator method: the guard throws on the first MoveNext. + var start = platformAudio.StartRecording(); + Assert.Throws(() => start.MoveNext()); + + // The guards must not break dispose idempotency or event safety + // (DevicesChanged_SubscribeUnsubscribe_SafeAcrossDispose covers the rest). + Assert.DoesNotThrow(() => platformAudio.Dispose()); + + yield break; + } + [UnityTest] public IEnumerator StartThenStopRecording_DoesNotThrow() {