Skip to content

feat: audio width follows the output route; report lost audio - #51

Merged
bilipp merged 4 commits into
mainfrom
feat/audio-route-channels
Oct 12, 2026
Merged

bilipp merged 4 commits into
mainfrom
feat/audio-route-channels

Conversation

@bilipp

@bilipp bilipp commented Oct 12, 2026

Copy link
Copy Markdown
Owner

Stacked on #46.

Summary

Audio width that follows the output route. Until now the audio decoder read the session's negotiated output width (AVAudioSession.outputNumberOfChannels) once, when the decoder was created, and kept it for the life of the session. The downmix was swresample's automatic matrix, picked implicitly by sample format. So:

  • HDMI receiver (host asks for the route's width, as Lume does): AC-3/E-AC-3/AAC 5.1, FLAC 7.1 and TrueHD went out as discrete LPCM with a channel bitmap. Unchanged.
  • AirPlay stereo, HomePods, TV speakers: 5.1/7.1 was folded to stereo by swresample's automatic matrix (centre and surrounds at −3 dB, LFE dropped, not normalized for float). That was already a real mix, not channel dropping. Mono went out as one Centre-tagged channel.
  • A route change mid-playback was not followed. Going from HDMI 7.1 to AirPlay kept feeding 6/8 channels to a stereo route. Earlier tvOS device work showed this fails the renderer (silence, wedged pipeline, now .renderFailed). Going the other way, from AirPlay to HDMI, stayed downmixed to stereo until the next session.

Now:

  • AudioOutputRoute observes the shared audio session's negotiated width through routeChangeNotification (every reason, including .routeConfigurationChange). On macOS it is fixed at stereo. Tests and hosts can drive it with update. The engine still only observes the session: category, activation and preferred channel count stay the host's job.
  • AudioLayoutPolicy is a pure value type that maps source layout × route width to the target layout:
    • discrete when the source fits the route,
    • otherwise the widest standard layout both fill (7.1 / 5.1(side) / stereo), never inventing an LFE,
    • mono goes to both speakers of any route wider than one (a stereo pair has no centre speaker),
    • layouts the renderer cannot name (wide, top, matrix-encoded stereo) are remixed instead of passed untagged.
  • DownmixMatrix: explicit ITU-R BS.775 levels (centre and surrounds −3 dB, LFE omitted), built by swr_build_matrix2 and set with swr_set_matrix, so the matrix no longer depends on the source's sample format. It is deliberately not normalized, for loudness parity with stereo channels on zap. The documented trade-off is clipping on peaks where every channel is near full scale.
  • AudioDecoder reads the route width per decoded frame and rebuilds swr when it changes. It emits the pending buffer first, so the switch lands on a frame boundary. Same-rate swr holds no samples back, so nothing is lost.
  • SystemRenderer: when the route narrows, audio already decoded for the wider route is kept out of the renderer. That covers queued frames, retained refill audio, and what the renderer itself still holds. The narrowing therefore costs the renderer's lead in silence, about what a route change costs anyway, instead of risking a failed renderer.

Does the system downmix by itself? On macOS it does: 8-channel LPCM with a channel bitmap, fed to AVSampleBufferAudioRenderer on built-in 2-channel speakers, renders (status .rendering, clock running). I checked this locally; it is not a committed test, because CI runners have no stable audio device. On tvOS the evidence goes the other way: the switch from maximumOutputNumberOfChannels to outputNumberOfChannels was made because a renderer fed more channels than the session outputs failed outright on device, with layout tags attached. So the engine keeps mixing on iOS/tvOS and does not pass multichannel through to narrower routes. The suggested maximumOutputNumberOfChannels is deliberately not used.

Audio lost. PlayerSession.audioHealth (.healthy / .absent / .lost(reason)) is also in Diagnostics and in its log line. Reasons:

  • decoderFailed: the decoder gave up. Its .decodeFailed error was already emitted, and the lane was silently excluded.
  • rendererFailed: the audio renderer failed. The session still fails with .renderFailed, as before.
  • starved: the clock runs and video is ahead of the playhead, but no audio has been enqueued beyond the playhead for 2 s of media.

The detector (AudioLossDetector) is a pure value type. Nothing counts while paused, buffering or seeking, when video is dry too (that is buffering), for audio-only sources, after the audio track's end, or across a seek. Gaps inside the audio don't count either, because the high-water mark is past them. starved is reported once per episode as .error(EngineError(code: .audioLost)) and clears when audio comes back.

API (additive only — Lume compiles unchanged)

  • New public types: AudioOutputRoute (+ Snapshot), AudioOutputLayout, AudioHealth, AudioLossReason.
  • PlayerConfiguration.audioOutputRoute: AudioOutputRoute?. Defaults to nil, which follows the system session.
  • PlayerSession.audioHealth and PlayerSession.audioOutput. Diagnostics gains audioHealth, audioOutput and audioRoute; its description gains aout=… route=… and, only when lost, audio=lost(…).
  • AudioDecoder.init(parameters:input:output:route:) and currentOutput. The existing maxOutputChannels: initializer is kept and pins a fixed route.
  • SystemRenderer.init(muted:audioOutputRoute:), with the new parameter defaulted.
  • EngineError.Code.audioLost. Lume only logs .error events (LumeEngineCoordinator.handle(event:)) and never switches over EngineError.Code. .error already carries non-fatal per-lane failures (a failed audio decoder is reported the same way without changing state), so this fits the existing semantics.
  • No new PlayerEvent cases.

Tests

  • AudioLayoutPolicy: speaker sets match FFmpeg's named layouts; passthrough when the source fits; fold-down ladder; no invented LFE; mono → stereo; unnameable channels remixed.
  • DownmixMatrix: 5.1→2.0 and 7.1→2.0 coefficients (centre −3 dB into both sides, surrounds into their own side, LFE 0, no cross-feed); 7.1→5.1; mono→stereo; the centre survives every stereo fold-down and no non-LFE channel is dropped.
  • AudioRoute (decoder, forced widths):
    • surround71.mkv at 8/6/2 and truehd.mkv at 8/2. Bitmaps are checked, and the centre-only tone arrives in both stereo channels at −3 dB.
    • basic.mp4 mono → stereo.
    • A route change from 8 to 2 mid-stream switches exactly once, at a frame boundary, with sample-accurate PTS abutment and the full 4 s duration intact.
  • AudioLossDetector: every branch, including legitimate silences, pause/seek, recovery and re-reporting, and permanent losses.
  • DecoderTests: the AAC decode test now expects its mono sine as 2 channels (mono → stereo on a stereo route — the one intended behaviour change for existing callers).
  • AudioRouteSession (new fixture surround51.ts: H.264 + AC-3 5.1, centre-only tone):
    • A playing session refits 6→2→6 channels on route changes and keeps playing; the renderer is not failed.
    • Stopping the audio renderer while video plays yields .lost(.starved) and exactly one .error(.audioLost).
    • A session without audio reports .absent.

Verification

  • swift test: full suite green (see the coordinator report for counts).
  • xcodebuild build for generic tvOS and iOS: no new warnings.

Not verified (needs an Apple TV)

  • AirPlay stereo:
    • AC-3/E-AC-3/AAC 5.1 channels report aout=6>2ch@2 route=2ch/2 AirPlay.
    • Dialogue is clear and audible on both sides.
    • No audible clipping on loud scenes (the matrix is not normalized).
    • A mono channel (aout=1>2ch) plays on both sides.
  • HomePod stereo pair:
    • Same checks as AirPlay.
    • Both HomePods play for stereo, downmixed-5.1 and mono sources. A known system issue on tvOS 27 makes the buffered-audio path play on one HomePod regardless of engine, so compare with Lume's AVPlayer engine.
    • Note the reported route= width; HomePods set as the Apple TV's default output may negotiate more than 2.
  • HDMI 5.1 receiver:
    • The route reports 6 (route=6ch/6 HDMIOutput, or 8 at max).
    • 5.1 sources play discrete (aout=6>6ch@6), with each channel on the right speaker (centre on centre, LFE on sub).
    • 7.1 sources fold to 5.1 (aout=8>6ch@6).
  • Route changes during playback (Control Center / AirPlay picker), HDMI ↔ AirPlay ↔ HomePod:
    • The heartbeat's aout= follows the route.
    • The renderer does not fail.
    • At most a short silence (~1–2 s) on narrowing.
    • No audio=lost and no .audioLost.
    • route=… changes count up.
  • Whether outputNumberOfChannels follows a route change on its own, or the host has to re-assert setPreferredOutputNumberOfChannels after one. If route=2ch/8 shows on HDMI after switching back, Lume should re-apply its preferred count on route change.
  • No false audio=lost(starved) over long live sessions (ad splices, encoder restarts, reconnects).

Base automatically changed from feat/live-playout-and-startup-events to main October 12, 2026 22:14
…utput on tvOS

PlayerSession.displayCriteria describes the display mode the selected video
wants, from what the engine outputs: the refresh rate after deinterlacing
(measured from enqueued frames once there are any, snapped to the film and
broadcast grid or no preference) and the dynamic range after Dolby Vision
conversion. DisplayCriteriaMatcher (tvOS, opt-in) applies it through the
window's AVDisplayManager and holds the session's start, bounded, while the
TV switches; PlayerSession.holdStart/releaseStartHold carry that hold.
LumePlayer.matchesDisplayCriteria wires it into the facade. TrackInfo.Video
gains isInterlaced and the Dolby Vision configuration record.
The engine has no Dolby Vision output path: it presents the base layer of
every profile it does not convert. Dolby Vision streams now ask for that
base layer's range (compatibility ID before the first frame, the frames'
transfer function after), and the synthesized dvh1 description is gone.
DolbyVisionDetection.Record now comes from the public
DolbyVisionConfiguration instead of parsing AV_PKT_DATA_DOVI_CONF a second
time, and DolbyVisionConfiguration.baseLayerIsIPT asks Record.claimsIPT, so
the profile 5/10.0/20 rule lives in one place. Its doc now says what it is
since conversion became a per-frame decision: the stream's claim.
The audio decoder read the negotiated output width once, at creation, so a
route change mid-playback (HDMI receiver to AirPlay, HomePods) kept feeding
the old width. AudioOutputRoute now follows the shared audio session's
negotiated width through route-change notifications (stereo on macOS); the
decoder refits per AudioLayoutPolicy at the next frame boundary, with an
explicit ITU-R BS.775 DownmixMatrix (centre and surrounds -3 dB, LFE
omitted) set into swresample, and mono goes out on both speakers. When the
route narrows, the renderer keeps audio decoded for the wider route away
from the renderer.

PlayerSession.audioHealth (also in Diagnostics) reports audio that died
while video plays on: decoder failed, renderer failed, or nothing reaching
the renderer at a running playhead (AudioLossDetector), the latter also as
.error with the new EngineError.Code.audioLost. Additive API only.
@bilipp
bilipp force-pushed the feat/audio-route-channels branch from 984532d to 4310850 Compare October 12, 2026 22:33
@bilipp
bilipp merged commit 89d6de6 into main Oct 12, 2026
5 checks passed
@bilipp
bilipp deleted the feat/audio-route-channels branch October 12, 2026 22:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant