diff --git a/AGENTS.md b/AGENTS.md index 17725e7..38f275d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -85,6 +85,9 @@ This split is the core invariant of the codebase: - **HDR metadata is written on every delivered buffer, from the frame.** `Decode/HDRMetadata` turns the frame's mastering display (ST 2086) and content light level side data into Apple's big-endian SEI payloads on the pixel buffer in `VideoDecoder.deliver`, and HDR10+ (ST 2094-40) into a T.35 message the `SampleBufferBuilder` attaches per sample (`kCMSampleAttachmentKey_HDR10PlusPerFrameData`). No path keeps it by itself: VideoToolbox, as FFmpeg drives it, leaves its surfaces without either key, and the engine's own buffers carry only what it writes. Read the frame, not the codec parameters — FFmpeg already folds the container's boxes into every frame and lets the bitstream's SEI override them. Keys are written *or removed* on every buffer (pools recycle), and the format description is rebuilt whenever they change, because `CMVideoFormatDescriptionMatchesImageBuffer` compares them; don't replace that match with a cheaper key. - **Multi-layer HEVC reaches the codec as its base layer only.** `Decode/HEVCBaseLayerFilter` drops every `nuh_layer_id > 0` NAL unit from packets and extradata before `avcodec_send_packet`/`avcodec_open2`. FFmpeg files an MV-HEVC stream's second-view SPS/PPS (DV profile 20, spatial video) alongside the base layer's, its VideoToolbox glue packs all of them into one `hvcC`, VideoToolbox rejects that session, and FFmpeg falls back to software *without an error* — 4K HEVC in software, which stutters on Apple TV. The engine presents only the base view, so stripping loses nothing. Relatedly, `VideoDecoder.isHardwareActive` is judged from the frames' pixel format, not from whether a VT device was attached, and such a silent fallback emits `.downgradedToSoftware` — keep it that way, or this class of bug goes invisible again. - **Dolby Vision with an IPT base layer is converted, not tagged.** Profiles 5, 10.0 and 20 (compatibility ID 0) carry IPTPQc2, which shows pink/green as YCbCr. No colour attachment fixes that — the RPU reshaping changes per scene and Apple's decoders don't apply it to decoded frames — so `Decode/DolbyVisionConverter` turns each frame into HDR10 on the GPU in `VideoDecoder.deliver`, the one point every frame (hardware, software, deinterlaced) passes with its side data. Whether a frame is IPT is decided *per frame* by `Decode/DolbyVisionDetection`, never by the configuration record alone — remuxes mislabel HDR10 as P5 and broadcasts drop RPUs without changing it, and converting HDR10 as IPT gives wrong colours. Order: an `AVCOL_SPC_IPT_C2` frame is IPT; a frame with its *own* RPU is what the RPU's `signal_color_space` says (2 = IPT); a frame without one that states a matrix (BT.2020 PQ) is that matrix; only an untagged frame without its own RPU falls back to the record (genuine P5 often leaves the VUI matrix unspecified). "Own" matters because FFmpeg's HEVC decoder attaches the last RPU's `AV_FRAME_DATA_DOVI_METADATA` to every later frame; only `AV_FRAME_DATA_DOVI_RPU_BUFFER` marks the frame that carried one (AV1 attaches metadata only to its own frame). HDR10-compatible profiles (8.1 etc.) must stay on the zero-copy path. GPU refusals are per-frame (iOS refuses a backgrounded app); only a converter that can't exist gives up for the session. +- **Display criteria describe the output, not the container.** `PlayerSession.displayCriteria` (pure resolution in `Media/ContentDisplayCriteria`) is what an Apple TV with Match Content should switch the TV to: the refresh rate *after* deinterlacing (field rate turns 1080i50 into 50 Hz) and the dynamic range of what is presented (IPT Dolby Vision profiles are converted, so HDR10; every other Dolby Vision profile is presented as its base layer, so HDR10, HLG or SDR by its compatibility ID and the frames' transfer function). Never ask for Dolby Vision mode while the engine has no Dolby Vision output path — that belongs to the PLAN.md §7 compressed-`dvh1` spike. Before the first frame it is predicted from the stream; once the renderer has been handed frames, their colour tags and measured cadence (`Render/FrameCadence`) decide. Rates snap to the film/broadcast grid or ask for none — never an odd rate. Applying it is tvOS-only and opt-in (`Media/DisplayCriteriaMatcher`, or `LumePlayerView(…, matchesDisplayCriteria: true)`): one matcher per player screen, kept across sessions so zapping between same-mode channels never switches the TV, reset only when the screen goes away. The mode-switch blank is sat out with `PlayerSession.holdStart(for:)`, which the session bounds itself and the stall watchdog ignores — keep both, or a host that never releases wedges playback, or a held start reads as a stall. +- **Dolby Vision with an IPT base layer is converted, not tagged.** Profiles 5, 10.0 and 20 (compatibility ID 0) carry IPTPQc2, which shows pink/green as YCbCr. No colour attachment fixes that — the RPU reshaping changes per scene and Apple's decoders don't apply it to decoded frames — so `Decode/DolbyVisionConverter` turns each frame into HDR10 on the GPU in `VideoDecoder.deliver`, the one point every frame (hardware, software, deinterlaced) passes with its side data. Detection is the stream's DV configuration record *or* an `AVCOL_SPC_IPT_C2` frame; many P5 encodes leave the VUI matrix unspecified, so don't reduce it to the tag. HDR10-compatible profiles (8.1 etc.) must stay on the zero-copy path. GPU refusals are per-frame (iOS refuses a backgrounded app); only a converter that can't exist gives up for the session. +- **Audio is fitted to the negotiated route width, never the route's capability.** `AudioDecoder` reads `AudioOutputRoute.outputChannels` (the session's `outputNumberOfChannels`, followed through route-change notifications) per frame and remixes per `AudioLayoutPolicy` with an explicit `DownmixMatrix` — a fold-down mixes centre and surrounds in, it never drops channels. Don't switch it to `maximumOutputNumberOfChannels`: a renderer fed more channels than the session outputs failed outright on tvOS. The engine only observes the audio session; setting its category, activation or preferred channel count is the host's job (Lume does it). When the route narrows, `SystemRenderer` keeps audio decoded for the wider route out of the renderer. - **Zero global mutable state:** all configuration is `PlayerConfiguration` passed at session creation. No `static var` knobs. - **No force unwraps / `as!` / `fatalError` in engine paths**; failures return typed `EngineError`. No `assertionFailure` in production channel/data paths — misuse degrades, never crashes. - **Never scrape FFmpeg log strings for control flow** (`av_log` is diagnostics only). Silence is never a failure mode: pipelines are supervised, stalls are detected from playhead ground truth, and failures surface as typed `PlayerEvent`s on the session's `AsyncStream`. diff --git a/PLAN.md b/PLAN.md index 6dc77a6..bb4f4d7 100644 --- a/PLAN.md +++ b/PLAN.md @@ -190,6 +190,7 @@ Licensing note: engine code is **MIT** (`LICENSE`); FFmpeg is LGPL 2.1+, and the ## 6. Feature implementation notes (selected) - **Now Playing (explicit requirement):** `NowPlayingCenter` module in `LumeEngineMedia` — `MPNowPlayingInfoCenter` (title/artwork/elapsed/duration/rate, live-stream flag) + `MPRemoteCommandCenter` (play/pause/seek/skip/next-episode hook), audio-session lifecycle, route-change and interruption handling. Works on the lock screen/Control Center including while audio/video routes to an Apple TV. Note: full-screen *video* AirPlay of FFmpeg-decoded content is not possible for third parties (system limitation); Lume's existing switch-to-AVPlayer-on-AirPlay stays, and the integration phase adds a shared app-level Now Playing service so lock-screen metadata is present in **both** engines (closing today's gap). +- **Audio width follows the route:** decoded audio leaves the engine in the layout `AudioLayoutPolicy` picks for the *negotiated* route width (`AVAudioSession.outputNumberOfChannels`, followed through route changes by `AudioOutputRoute`; stereo on macOS): discrete multichannel when the route carries it (HDMI receiver), an explicit ITU-R BS.775 fold-down (`DownmixMatrix`: centre and surrounds at −3 dB, LFE omitted, not normalized) when it does not, mono on both speakers of a stereo route. A route change refits at the next decoded frame; audio decoded for a wider route never reaches a narrower one. The engine observes the audio session only — category, activation and the preferred channel count are the host's. `PlayerSession.audioHealth` reports audio that died while video plays on (decoder gave up, renderer failed, or nothing reaching the renderer at a running playhead). - **PiP:** `AVPictureInPictureController(contentSource: .init(sampleBufferDisplayLayer:playbackDelegate:))` from the SystemRenderer; subtitle cues composited into the sample buffers during PiP so subs survive. - **Trickplay:** a `ThumbnailSession` — separate lightweight demux+decode (keyframes only, HW decode, downscale to ~320 px) filling a time-indexed cache; API `thumbnails(at:interval:)`. Works for VOD/catch-up; live uses the rewind buffer. - **Live rewind (time-shift DVR):** disk-backed packet spool behind a custom AVIO layer; seeking within spool = local demux, seeking to live edge = passthrough. Configurable size/duration caps. @@ -197,6 +198,7 @@ Licensing note: engine code is **MIT** (`LICENSE`); FFmpeg is LGPL 2.1+, and the - **Precache:** same AVIO disk-cache layer, `precache(url:leadingSeconds:)` background task with LRU budget. - **Upscaling:** MetalFX spatial scaler stage (D5), quality/perf switch, auto-disable on thermal pressure (`ProcessInfo.thermalState`). - **HDR:** HDR10/HLG passthrough via sample-buffer attachments. DV with an HDR10-compatible base layer (P8.1, P10.1, …) plays as HDR10. DV with an IPTPQc2 base layer (P5, P10.0, P20 — compatibility ID 0) is converted to HDR10 before enqueue by a Metal compute pass (`Decode/DolbyVisionConverter`) that applies each frame's RPU reshaping (polynomial + MMR) and the IPT→LMS→BT.2020 matrices FFmpeg exports as `AV_FRAME_DATA_DOVI_METADATA`; the renderer then tone-maps it like any HDR10 stream. Dynamic DV display management (L1/L2/L8 trims) is not carried over. HDR10 static metadata (mastering display, MaxCLL/MaxFALL — from the bitstream's SEI or the container, falling back to a DV RPU's level 6 for the light level) is attached to every delivered pixel buffer, since neither VideoToolbox nor the engine's own buffers carry it otherwise, and so reaches the format description; HDR10+ dynamic metadata travels per sample as `kCMSampleAttachmentKey_HDR10PlusPerFrameData` (`Decode/HDRMetadata`). Whether the renderer acts on HDR10+ for decoded (not compressed) samples is still to be confirmed on an HDR10+ display. EDR headroom handling on macOS/visionOS Metal path. +- **Match Content (tvOS):** a sample-buffer pipeline gets no automatic display mode switching, so the engine derives `ContentDisplayCriteria` from its own output — refresh rate after deinterlacing (measured from enqueued frame PTS once there are frames, snapped to 23.976/24/25/29.97/30/50/59.94/60 or no preference), dynamic range of what is presented (IPT Dolby Vision profiles → HDR10 after conversion; other Dolby Vision profiles → their base layer's range by compatibility ID, cross-checked with the frames' transfer function; otherwise the frames' or the stream's transfer function) — and `DisplayCriteriaMatcher` sets it as the window's `AVDisplayManager.preferredDisplayCriteria` via `AVDisplayCriteria(refreshRate:formatDescription:)` (tvOS 17+). Dolby Vision mode is never requested: the engine presents the base layer, and only a real DV output path (§7 spike) should ask for it. The start is held, bounded, until the TV finishes switching. Opt-in; mid-stream changes are not chased (each switch blanks the TV). - **360°/VR:** MetalRenderer projection (equirect sphere / fisheye), CoreMotion + gesture camera; visionOS immersive presentation later. - **Blu-ray ISO/DVD:** libbluray/libdvdnav behind the `AVIOProtocol` seam; menu-less title playback first (longest-title heuristic), chapters mapped to the chapter API. - **AI subtitles:** whisper.cpp (Metal) tapping the decoded-audio channel; streaming word-level segments feed the subtitle model as a synthetic track; on-device translation via Translation.framework. diff --git a/Sources/LumeEngine/LumePlayer.swift b/Sources/LumeEngine/LumePlayer.swift index a70a3c3..34d0f40 100644 --- a/Sources/LumeEngine/LumePlayer.swift +++ b/Sources/LumeEngine/LumePlayer.swift @@ -51,6 +51,24 @@ public final class LumePlayer { /// layer tree directly (UIKit/AppKit). public var displayLayer: LumeDisplayLayer? { session?.renderer.displayLayer } + /// tvOS: switch the TV to each loaded stream's refresh rate and dynamic + /// range (Match Content) while it plays in a `LumePlayerView`, which + /// supplies the window; `stop()` and the view going away hand the display + /// back. Set before `load`. Off by default; ignored on other platforms. + public var matchesDisplayCriteria = false { + didSet { + #if os(tvOS) + if !matchesDisplayCriteria { displayMatcher.reset() } + #endif + } + } + + #if os(tvOS) + /// One per player, so loading the next stream does not hand the display + /// back in between (see `DisplayCriteriaMatcher`). + let displayMatcher = DisplayCriteriaMatcher() + #endif + private var session: PlayerSession? private var eventTask: Task? private var tickTask: Task? @@ -86,6 +104,10 @@ public final class LumePlayer { do { let info = try await session.open(url: url) + #if os(tvOS) + // Before the app's play(): the start is held while the TV switches. + if matchesDisplayCriteria { await displayMatcher.match(session) } + #endif mediaInfo = info duration = info.duration.map(MediaTime.seconds) return info @@ -135,6 +157,9 @@ public final class LumePlayer { } public func stop() async { + #if os(tvOS) + displayMatcher.reset() + #endif await teardownSession() state = .idle position = 0 diff --git a/Sources/LumeEngine/LumePlayerView.swift b/Sources/LumeEngine/LumePlayerView.swift index 74a00e3..04e4956 100644 --- a/Sources/LumeEngine/LumePlayerView.swift +++ b/Sources/LumeEngine/LumePlayerView.swift @@ -18,12 +18,29 @@ public struct LumePlayerView: View { } public var body: some View { - if let layer = player.displayLayer { - LayerHostView(layer: layer, gravity: gravity) + ZStack { + if let layer = player.displayLayer { + #if os(tvOS) + LayerHostView( + layer: layer, + gravity: gravity, + displayMatcher: player.matchesDisplayCriteria ? player.displayMatcher : nil + ) .accessibilityLabel("Video") - } else { - Color.black + #else + LayerHostView(layer: layer, gravity: gravity) + .accessibilityLabel("Video") + #endif + } else { + Color.black + } } + #if os(tvOS) + // The whole view, not the layer host: that one is swapped out while + // the player moves to the next stream, which must not hand the + // display back in between. + .onDisappear { [player] in player.displayMatcher.reset() } + #endif } } @@ -33,21 +50,49 @@ import UIKit private struct LayerHostView: UIViewRepresentable { let layer: LumeDisplayLayer let gravity: AVLayerVideoGravity + #if os(tvOS) + /// Given this view's window, so it knows which display to match. + var displayMatcher: DisplayCriteriaMatcher? + #endif func makeUIView(context: Context) -> LayerHostUIView { let view = LayerHostUIView() view.install(layer: layer, gravity: gravity) + #if os(tvOS) + view.displayMatcher = displayMatcher + #endif return view } func updateUIView(_ view: LayerHostUIView, context: Context) { view.install(layer: layer, gravity: gravity) + #if os(tvOS) + view.displayMatcher = displayMatcher + #endif } } final class LayerHostUIView: UIView { private weak var hosted: LumeDisplayLayer? + #if os(tvOS) + weak var displayMatcher: DisplayCriteriaMatcher? { + didSet { lendWindow() } + } + + override func didMoveToWindow() { + super.didMoveToWindow() + lendWindow() + } + + /// Only ever sets a window: leaving one is not the end of playback (the + /// host is swapped while the next stream loads); the player view's + /// disappearance is. + private func lendWindow() { + if let window { displayMatcher?.window = window } + } + #endif + func install(layer: LumeDisplayLayer, gravity: AVLayerVideoGravity) { layer.videoGravity = gravity guard hosted !== layer else { return } diff --git a/Sources/LumeEngineCore/Decode/AudioDecoder.swift b/Sources/LumeEngineCore/Decode/AudioDecoder.swift index f8bebb0..61a51b6 100644 --- a/Sources/LumeEngineCore/Decode/AudioDecoder.swift +++ b/Sources/LumeEngineCore/Decode/AudioDecoder.swift @@ -7,13 +7,15 @@ import Foundation /// FFmpeg audio decode + resample stage: `Channel` in, /// `Channel` out. /// -/// Output is always interleaved Float32 at the source sample rate, downmixed -/// to at most `maxOutputChannels` — the single canonical format the renderer -/// consumes. Sources with more channels than the output route can carry are -/// downmixed here by swresample (normalized matrix); handing the sample-buffer -/// renderer more channels than the route supports produces audibly broken -/// output. Resampling context is rebuilt automatically when source parameters -/// change mid-stream (codec/parameter changes are routine on IPTV). +/// Output is always interleaved Float32 at the source sample rate, in the +/// speaker layout `AudioLayoutPolicy` picks for the output route — the single +/// canonical format the renderer consumes. Sources with more channels than +/// the route carries are folded down here by swresample with an explicit +/// matrix (`DownmixMatrix`); handing the sample-buffer renderer more channels +/// than the route supports produces audibly broken output. The resampling +/// context is rebuilt when source parameters change mid-stream (codec and +/// parameter changes are routine on IPTV) and when the route's width changes +/// (`AudioOutputRoute`), in both cases at a frame boundary. /// /// Decoded frames are coalesced to at least `minOutputDuration` before /// emission. Some codecs decode to frames far too small to schedule @@ -42,6 +44,7 @@ public final class AudioDecoder: @unchecked Sendable { private var swrSourceFormat: Int32 = -1 private var swrSourceRate: Int32 = -1 private var swrSourceChannels: Int32 = -1 + private var swrRouteChannels = -1 private var outputLayout = AVChannelLayout() private var currentSerial: UInt64? private var consecutiveErrors = 0 @@ -59,7 +62,17 @@ public final class AudioDecoder: @unchecked Sendable { private static let minOutputDuration = 0.020 private let maxConsecutiveErrors = 100 - private let maxOutputChannels: Int + private let route: AudioOutputRoute + /// What the resampler currently produces. Guarded by `lock`. + private var appliedOutput: AudioOutputLayout? + + /// The layout decoded audio currently leaves in, and the route width it + /// was chosen for; nil before the first frame. + public var currentOutput: AudioOutputLayout? { + lock.lock() + defer { lock.unlock() } + return appliedOutput + } /// Channel budget of the current output route. The renderer downstream /// cannot correctly render more channels than the route carries, so @@ -72,7 +85,9 @@ public final class AudioDecoder: @unchecked Sendable { /// Feeding the renderer more channels than the session actually outputs /// fails the renderer outright (silence, wedged pipeline) — configuring /// the session for surround is the app's job (PLAN.md §2.1), the engine - /// just honors whatever was negotiated at session-creation time. + /// just honors whatever was negotiated. This reads it once, for a decoder + /// of fixed width; `PlayerSession` follows route changes instead + /// (`AudioOutputRoute`). public static func defaultMaxOutputChannels() -> Int { #if canImport(UIKit) return max(2, AVAudioSession.sharedInstance().outputNumberOfChannels) @@ -84,16 +99,32 @@ public final class AudioDecoder: @unchecked Sendable { #endif } - public init( + /// A decoder whose output width is fixed at `maxOutputChannels`. + public convenience init( parameters: CodecParameters, input: Channel, output: Channel, maxOutputChannels: Int = AudioDecoder.defaultMaxOutputChannels() + ) { + self.init( + parameters: parameters, + input: input, + output: output, + route: AudioOutputRoute(outputChannels: maxOutputChannels) + ) + } + + /// A decoder whose output follows `route` as it changes. + public init( + parameters: CodecParameters, + input: Channel, + output: Channel, + route: AudioOutputRoute ) { self.parameters = parameters self.input = input self.output = output - self.maxOutputChannels = max(1, maxOutputChannels) + self.route = route var continuation: AsyncStream.Continuation! events = AsyncStream(bufferingPolicy: .unbounded) { continuation = $0 } eventSink = continuation @@ -329,12 +360,17 @@ public final class AudioDecoder: @unchecked Sendable { let format = frame.pointee.format let rate = frame.pointee.sample_rate let channels = frame.pointee.ch_layout.nb_channels + let routeChannels = route.outputChannels - if swrContext != nil, format == swrSourceFormat, rate == swrSourceRate, channels == swrSourceChannels { + if swrContext != nil, format == swrSourceFormat, rate == swrSourceRate, channels == swrSourceChannels, + routeChannels == swrRouteChannels { return true } - emitPending() // samples of the previous format cannot share a buffer + // Samples of the previous layout cannot share a buffer. Same-rate + // conversion holds nothing back inside swr, so emitting the pending + // buffer is all a switch needs to land on a frame boundary. + emitPending() swr_free(&swrContext) // swr needs concrete layouts on both sides to build a (down)mix // matrix; sources reporting an unspecified order (routine for PCM) @@ -346,12 +382,27 @@ public final class AudioDecoder: @unchecked Sendable { av_channel_layout_default(&inputLayout, channels) } defer { av_channel_layout_uninit(&inputLayout) } + av_channel_layout_uninit(&outputLayout) - if channels > Int32(maxOutputChannels) { - av_channel_layout_default(&outputLayout, Int32(maxOutputChannels)) + var matrix: [Double]? + if inputLayout.order == AV_CHANNEL_ORDER_NATIVE { + let source = SpeakerSet(rawValue: inputLayout.u.mask) + let plan = AudioLayoutPolicy.plan(source: source, routeChannels: routeChannels) + if plan.remixes { + av_channel_layout_from_mask(&outputLayout, plan.output.rawValue) + matrix = DownmixMatrix.flatCoefficients(from: source, to: plan.output) + } else { + av_channel_layout_copy(&outputLayout, &inputLayout) + } + } else if channels > Int32(routeChannels) { + // No positions to plan with (more channels than FFmpeg has a + // default order for): fold to the route's default layout and let + // swr do what it can. + av_channel_layout_default(&outputLayout, Int32(routeChannels)) } else { av_channel_layout_copy(&outputLayout, &inputLayout) } + var newContext: OpaquePointer? let result = swr_alloc_set_opts2( &newContext, @@ -359,7 +410,16 @@ public final class AudioDecoder: @unchecked Sendable { &inputLayout, AVSampleFormat(rawValue: format), rate, 0, nil ) + // A custom matrix goes in between allocation and init. Without one + // (no remix, or a layout pair the builder refused) swr builds its own + // from the two layouts. + if result >= 0, let created = newContext, let matrix { + _ = matrix.withUnsafeBufferPointer { + swr_set_matrix(created, $0.baseAddress, Int32(DownmixMatrix.stride)) + } + } guard result >= 0, let created = newContext, swr_init(created) >= 0 else { + swr_free(&newContext) handleDecodeError(nil, EngineError(code: .decodeFailed, message: "swresample init failed")) return false } @@ -367,6 +427,15 @@ public final class AudioDecoder: @unchecked Sendable { swrSourceFormat = format swrSourceRate = rate swrSourceChannels = channels + swrRouteChannels = routeChannels + let applied = AudioOutputLayout( + sourceChannels: Int(channels), + outputChannels: Int(outputLayout.nb_channels), + routeChannels: routeChannels + ) + lock.lock() + appliedOutput = applied + lock.unlock() return true } diff --git a/Sources/LumeEngineCore/Decode/AudioLayoutPolicy.swift b/Sources/LumeEngineCore/Decode/AudioLayoutPolicy.swift new file mode 100644 index 0000000..b435039 --- /dev/null +++ b/Sources/LumeEngineCore/Decode/AudioLayoutPolicy.swift @@ -0,0 +1,169 @@ +internal import CFFmpeg + +/// Speaker positions in FFmpeg's native channel-mask bit order (`AV_CH_*`), +/// which is also WAVE order and Core Audio's `AudioChannelBitmap` for bits +/// 0...17. Kept free of FFmpeg types so the layout policy is testable as plain +/// values. +struct SpeakerSet: OptionSet, Hashable, Sendable { + let rawValue: UInt64 + + static let frontLeft = SpeakerSet(rawValue: 1 << 0) + static let frontRight = SpeakerSet(rawValue: 1 << 1) + static let frontCenter = SpeakerSet(rawValue: 1 << 2) + static let lowFrequency = SpeakerSet(rawValue: 1 << 3) + static let backLeft = SpeakerSet(rawValue: 1 << 4) + static let backRight = SpeakerSet(rawValue: 1 << 5) + static let sideLeft = SpeakerSet(rawValue: 1 << 9) + static let sideRight = SpeakerSet(rawValue: 1 << 10) + + static let mono: SpeakerSet = [.frontCenter] + static let stereo: SpeakerSet = [.frontLeft, .frontRight] + /// 5.1 with the surrounds on the side pair (`AV_CH_LAYOUT_5POINT1`, + /// "5.1(side)"): the layout the AC-3 and E-AC-3 decoders — most surround + /// on IPTV — already hand the renderer. + static let surround51: SpeakerSet = [.frontLeft, .frontRight, .frontCenter, .lowFrequency, .sideLeft, .sideRight] + static let surround71: SpeakerSet = surround51.union([.backLeft, .backRight]) + + /// Positions a format description can name with a channel bitmap + /// (`SampleBufferBuilder`). Anything outside reaches the renderer without + /// a layout, and the renderer cannot place untagged multichannel audio. + static let bitmapExpressible = SpeakerSet(rawValue: 0x3FFFF) + + var count: Int { rawValue.nonzeroBitCount } +} + +/// What the audio lane hands the renderer, for diagnostics: the source's +/// channel count, the count after the engine's remix, and the route width the +/// remix was chosen for. +public struct AudioOutputLayout: Sendable, Equatable, CustomStringConvertible { + public let sourceChannels: Int + public let outputChannels: Int + public let routeChannels: Int + + /// The engine folds the source down to fit the route. + public var isDownmixed: Bool { outputChannels < sourceChannels } + + public var description: String { + "\(sourceChannels)>\(outputChannels)ch@\(routeChannels)" + } +} + +/// Which speaker layout the engine hands the audio renderer, given the +/// source's layout and how many channels the output route carries. +/// +/// The route width is the audio session's *negotiated* output +/// (`AVAudioSession.outputNumberOfChannels`), not what the hardware could do: +/// handing `AVSampleBufferAudioRenderer` more channels than the session +/// outputs failed it outright on tvOS — silence and a wedged pipeline — which +/// is why the engine mixes instead of leaving it to the system (see +/// `AudioOutputRoute`). +/// +/// Pure value logic: the decoder asks it once per source format and route +/// width, and swresample does what it says. +enum AudioLayoutPolicy { + struct Plan: Equatable, Sendable { + /// Speakers the renderer receives. + let output: SpeakerSet + /// The engine mixes the channels (down, or mono up to stereo) rather + /// than passing them through as decoded. + let remixes: Bool + } + + /// - Parameters: + /// - source: the decoded layout, resolved to native positions. + /// - routeChannels: channels the output route carries. + static func plan(source: SpeakerSet, routeChannels: Int) -> Plan { + let route = max(1, routeChannels) + let channels = source.count + + // A lone centre channel is placed on two speakers by the engine, not + // by whatever the route does with a centre it may not have: a stereo + // pair (two HomePods, a TV's speakers) has no centre speaker at all. + // Equal power on each side, the standard pan law. + if source == .mono, route >= 2 { + return Plan(output: .stereo, remixes: true) + } + + // Fits the route and the renderer can name every channel: hand it + // over untouched. A receiver on HDMI gets discrete surround. + if channels <= route, source.isSubset(of: .bitmapExpressible) { + return Plan(output: source, remixes: false) + } + + // Fold down to the widest standard layout both the route and the + // source fill. A layout the renderer cannot name (wide or top + // channels, a matrix-encoded stereo pair) is remixed to the standard + // one of its own width rather than passed through untagged. + let width = min(route, channels) + var target: SpeakerSet = width >= 8 ? .surround71 + : width >= 6 ? .surround51 + : width >= 2 ? .stereo + : .mono + // Never invent an LFE channel the source does not have. + if !source.contains(.lowFrequency) { + target.remove(.lowFrequency) + } + return Plan(output: target, remixes: true) + } +} + +/// The coefficients the engine's remix uses, built by swresample's own +/// matrix builder with the levels below. +/// +/// Levels follow ITU-R BS.775 fold-down: centre and surrounds at −3 dB into +/// their front side, LFE omitted (it carries nothing a stereo programme needs, +/// and full-range speakers rarely reproduce it). The dialogue channel thus +/// always reaches both front speakers — the failure this exists to prevent is +/// a downmix that drops channels instead of mixing them, which loses the +/// centre and with it the dialogue. +/// +/// The matrix is not normalized, matching what swresample does on its own for +/// float output: a 5.1 fold-down plays as loud as a stereo channel next to it +/// on the same IPTV list. Normalizing would make every surround channel +/// ~7.7 dB quieter than its stereo neighbours, which viewers hear as a volume +/// jump on every zap; the cost is that peaks where every channel is near full +/// scale at once can exceed 1.0 and clip at the output. +enum DownmixMatrix { + static let centerLevel = 0.707_106_781_186_547_6 + static let surroundLevel = 0.707_106_781_186_547_6 + static let lfeLevel = 0.0 + + /// swresample's matrix bound (`SWR_CH_MAX`); its builder walks the full + /// square, so the buffer must span it whatever the layouts are. + static let stride = 64 + + /// Row-major coefficients, `[output][input]`, in native channel order; + /// nil when swresample cannot build a matrix between the two layouts + /// (e.g. a source without any front channel). + static func coefficients(from source: SpeakerSet, to output: SpeakerSet) -> [[Double]]? { + guard let flat = flatCoefficients(from: source, to: output) else { return nil } + return (0.. [Double]? { + var inputLayout = AVChannelLayout() + var outputLayout = AVChannelLayout() + defer { + av_channel_layout_uninit(&inputLayout) + av_channel_layout_uninit(&outputLayout) + } + guard av_channel_layout_from_mask(&inputLayout, source.rawValue) >= 0, + av_channel_layout_from_mask(&outputLayout, output.rawValue) >= 0 + else { return nil } + + var matrix = [Double](repeating: 0, count: stride * stride) + let result = matrix.withUnsafeMutableBufferPointer { buffer in + swr_build_matrix2( + &inputLayout, &outputLayout, + centerLevel, surroundLevel, lfeLevel, + Double(Int32.max), // no normalization, see the type docs + 1.0, buffer.baseAddress, stride, + AV_MATRIX_ENCODING_NONE, nil + ) + } + return result >= 0 ? matrix : nil + } +} diff --git a/Sources/LumeEngineCore/Decode/DolbyVisionDetection.swift b/Sources/LumeEngineCore/Decode/DolbyVisionDetection.swift index 815a03f..125d1bf 100644 --- a/Sources/LumeEngineCore/Decode/DolbyVisionDetection.swift +++ b/Sources/LumeEngineCore/Decode/DolbyVisionDetection.swift @@ -92,15 +92,12 @@ extension DolbyVisionDetection.Record { /// The stream's configuration record (`dvcC`/`dvvC`, the Matroska block /// addition mapping, or a TS descriptor), if it has one. init?(_ parameters: UnsafePointer) { - guard let sideData = av_packet_side_data_get( - parameters.pointee.coded_side_data, - parameters.pointee.nb_coded_side_data, - AV_PKT_DATA_DOVI_CONF - ), let bytes = sideData.pointee.data, - sideData.pointee.size >= MemoryLayout.size - else { return nil } - let record = UnsafeRawPointer(bytes).loadUnaligned(as: AVDOVIDecoderConfigurationRecord.self) - self.init(profile: record.dv_profile, compatibilityID: record.dv_bl_signal_compatibility_id) + guard let configuration = DolbyVisionConfiguration(parameters: parameters) else { return nil } + self.init(configuration) + } + + init(_ configuration: DolbyVisionConfiguration) { + self.init(profile: configuration.profile, compatibilityID: configuration.baseLayerCompatibilityID) } } diff --git a/Sources/LumeEngineCore/Demux/MediaInfo+FFmpeg.swift b/Sources/LumeEngineCore/Demux/MediaInfo+FFmpeg.swift index 5fa29d9..99471ba 100644 --- a/Sources/LumeEngineCore/Demux/MediaInfo+FFmpeg.swift +++ b/Sources/LumeEngineCore/Demux/MediaInfo+FFmpeg.swift @@ -62,6 +62,33 @@ extension MediaInfo { } } +extension DolbyVisionConfiguration { + /// The configuration record FFmpeg filed with the stream + /// (`AV_PKT_DATA_DOVI_CONF`, from `dvcC`/`dvvC`/`dvwC` or the MPEG-TS + /// descriptor); nil when there is none. + init?(parameters: UnsafePointer) { + guard let sideData = av_packet_side_data_get( + parameters.pointee.coded_side_data, + parameters.pointee.nb_coded_side_data, + AV_PKT_DATA_DOVI_CONF + ), let bytes = sideData.pointee.data, + sideData.pointee.size >= MemoryLayout.size + else { return nil } + let record = UnsafeRawPointer(bytes).loadUnaligned(as: AVDOVIDecoderConfigurationRecord.self) + self.init( + versionMajor: record.dv_version_major, + versionMinor: record.dv_version_minor, + profile: record.dv_profile, + level: record.dv_level, + hasRPU: record.rpu_present_flag != 0, + hasEnhancementLayer: record.el_present_flag != 0, + hasBaseLayer: record.bl_present_flag != 0, + baseLayerCompatibilityID: record.dv_bl_signal_compatibility_id, + metadataCompression: record.dv_md_compression + ) + } +} + extension TrackInfo { init(stream: UnsafeMutablePointer, context: UnsafeMutablePointer) { let st = stream.pointee @@ -143,7 +170,9 @@ extension TrackInfo { colorPrimaries: par.color_primaries.rawValue, colorTransfer: par.color_trc.rawValue, colorSpace: par.color_space.rawValue, - colorRangeFull: par.color_range == AVCOL_RANGE_JPEG + colorRangeFull: par.color_range == AVCOL_RANGE_JPEG, + isInterlaced: Self.isInterlaced(par.field_order), + dolbyVision: DolbyVisionConfiguration(parameters: st.codecpar) ) } else { video = nil @@ -158,4 +187,12 @@ extension TrackInfo { audio = nil } } + + private static func isInterlaced(_ fieldOrder: AVFieldOrder) -> Bool? { + switch fieldOrder { + case AV_FIELD_UNKNOWN: nil + case AV_FIELD_PROGRESSIVE: false + default: true + } + } } diff --git a/Sources/LumeEngineCore/Demux/TrackInfo.swift b/Sources/LumeEngineCore/Demux/TrackInfo.swift index 0cb47c7..d469be0 100644 --- a/Sources/LumeEngineCore/Demux/TrackInfo.swift +++ b/Sources/LumeEngineCore/Demux/TrackInfo.swift @@ -21,6 +21,13 @@ public struct TrackInfo: Sendable, Identifiable, Equatable { public let colorTransfer: UInt32 public let colorSpace: UInt32 public let colorRangeFull: Bool + /// Whether the stream is coded interlaced, from the field order the + /// container or codec reports; nil when it reports none. A prediction + /// only: whether frames really come out interlaced is decided per + /// frame (`VideoDecoder.isDeinterlacing`). + public let isInterlaced: Bool? + /// The stream's Dolby Vision configuration record; nil when it has none. + public let dolbyVision: DolbyVisionConfiguration? } public struct Audio: Sendable, Equatable { @@ -57,6 +64,53 @@ public struct TrackInfo: Sendable, Identifiable, Equatable { public internal(set) var isDecodable = true } +/// A Dolby Vision decoder configuration record (`dvcC`/`dvvC`/`dvwC`), as the +/// container or the stream's descriptor carries it. +public struct DolbyVisionConfiguration: Sendable, Equatable { + public let versionMajor: UInt8 + public let versionMinor: UInt8 + public let profile: UInt8 + public let level: UInt8 + public let hasRPU: Bool + public let hasEnhancementLayer: Bool + public let hasBaseLayer: Bool + /// What the base layer is on its own: 0 none (IPTPQc2), 1 HDR10, 2 SDR, + /// 4 HLG, 6 BT.2020 HDR10. + public let baseLayerCompatibilityID: UInt8 + public let metadataCompression: UInt8 + + public init( + versionMajor: UInt8 = 1, + versionMinor: UInt8 = 0, + profile: UInt8, + level: UInt8, + hasRPU: Bool = true, + hasEnhancementLayer: Bool = false, + hasBaseLayer: Bool = true, + baseLayerCompatibilityID: UInt8, + metadataCompression: UInt8 = 0 + ) { + self.versionMajor = versionMajor + self.versionMinor = versionMinor + self.profile = profile + self.level = level + self.hasRPU = hasRPU + self.hasEnhancementLayer = hasEnhancementLayer + self.hasBaseLayer = hasBaseLayer + self.baseLayerCompatibilityID = baseLayerCompatibilityID + self.metadataCompression = metadataCompression + } + + /// The record claims an IPTPQc2 base layer rather than a displayable + /// fallback (HDR10, SDR, HLG): profiles 5, 10.0 and 20. Records lie + /// (remuxes label HDR10 as profile 5), so the engine decides per frame + /// whether to convert to HDR10 (`DolbyVisionDetection`); this is the + /// stream-level claim. + public var baseLayerIsIPT: Bool { + DolbyVisionDetection.Record(self).claimsIPT + } +} + /// A chapter marker, in engine microseconds. public struct ChapterInfo: Sendable, Equatable { public let title: String? diff --git a/Sources/LumeEngineCore/Errors/EngineError.swift b/Sources/LumeEngineCore/Errors/EngineError.swift index 6d645c1..c93a54d 100644 --- a/Sources/LumeEngineCore/Errors/EngineError.swift +++ b/Sources/LumeEngineCore/Errors/EngineError.swift @@ -25,6 +25,13 @@ public struct EngineError: Error, Sendable, CustomStringConvertible, Equatable { /// ends arrives as `.ioError` instead: FFmpeg cannot tell it from a /// dropped connection.) case sourceEnded + /// Audio stopped reaching the renderer while video plays on. Not + /// fatal: the session keeps playing, without sound, and recovers by + /// itself if audio comes back. `PlayerSession.audioHealth` says which + /// stage lost it. (An audio decoder that gives up reports its own + /// `.decodeFailed` instead, and a failed audio renderer fails the + /// session with `.renderFailed`.) + case audioLost } /// The HTTP status class a source answered with, when FFmpeg reported one. diff --git a/Sources/LumeEngineCore/Media/AudioOutputRoute.swift b/Sources/LumeEngineCore/Media/AudioOutputRoute.swift new file mode 100644 index 0000000..1d6e014 --- /dev/null +++ b/Sources/LumeEngineCore/Media/AudioOutputRoute.swift @@ -0,0 +1,140 @@ +#if os(iOS) || os(tvOS) || os(visionOS) +import AVFAudio +#endif +import Foundation +import Synchronization + +/// How many channels the audio output route carries, kept current while the +/// route changes under a playing session (HDMI receiver → AirPlay → HomePods). +/// +/// The audio decoder reads it for every decoded frame and folds its output to +/// fit (`AudioLayoutPolicy`), so a route change takes effect at the next frame +/// boundary; the renderer reads it to keep audio decoded for a wider, earlier +/// route away from a narrower one. +/// +/// The width is what the audio session *negotiated* +/// (`outputNumberOfChannels`), never what the hardware could carry +/// (`maximumOutputNumberOfChannels`): multichannel LPCM only flows once the +/// host asks for it with `setPreferredOutputNumberOfChannels`, and a renderer +/// fed more channels than the session outputs failed outright on tvOS. +/// Configuring and activating the session is the host's job — this type only +/// observes it. +/// +/// `system()` follows the shared `AVAudioSession`. On macOS there is no audio +/// session; it reports stereo, which is what the default output overwhelmingly +/// is. A host that knows better (a macOS multichannel interface), and tests, +/// create one with `init(outputChannels:)` and drive it with `update`. +public final class AudioOutputRoute: Sendable { + public struct Snapshot: Sendable, Equatable, CustomStringConvertible { + /// Channels the route carries for this app now. + public var outputChannels: Int + /// Channels the route could carry if the host asked for them; nil + /// where nothing reports it. More than `outputChannels` on a surround + /// route means the host did not ask for the route's full width. + public var maximumOutputChannels: Int? + /// Output port types (`HDMIOutput`, `AirPlay`, …). Types only, never + /// the user-visible port names. + public var ports: [String] + /// How often the width has changed since the route was created. + public var changes: Int + + public var description: String { + var line = "\(outputChannels)ch" + if let maximumOutputChannels { line += "/\(maximumOutputChannels)" } + if !ports.isEmpty { line += " " + ports.joined(separator: "+") } + return line + } + } + + private let state: Mutex + private let observers = Mutex<[any NSObjectProtocol]>([]) + + /// A route of fixed width that only `update` changes. + public init(outputChannels: Int, maximumOutputChannels: Int? = nil, ports: [String] = []) { + state = Mutex(Snapshot( + outputChannels: max(1, outputChannels), + maximumOutputChannels: maximumOutputChannels, + ports: ports, + changes: 0 + )) + } + + deinit { + for observer in observers.withLock({ $0 }) { + NotificationCenter.default.removeObserver(observer) + } + } + + /// The route of the shared audio session, followed through route changes. + public static func system() -> AudioOutputRoute { + #if os(iOS) || os(tvOS) || os(visionOS) + let current = sessionSnapshot() + let route = AudioOutputRoute( + outputChannels: current.outputChannels, + maximumOutputChannels: current.maximumOutputChannels, + ports: current.ports + ) + route.followSession() + return route + #else + return AudioOutputRoute(outputChannels: 2) + #endif + } + + /// Channels the route carries now. Cheap: read once per decoded frame. + public var outputChannels: Int { + state.withLock { $0.outputChannels } + } + + public var snapshot: Snapshot { + state.withLock { $0 } + } + + /// Reports a new width (and, when known, capability and ports). + public func update(outputChannels: Int, maximumOutputChannels: Int? = nil, ports: [String]? = nil) { + state.withLock { snapshot in + let width = max(1, outputChannels) + if width != snapshot.outputChannels { snapshot.changes += 1 } + snapshot.outputChannels = width + if let maximumOutputChannels { snapshot.maximumOutputChannels = maximumOutputChannels } + if let ports { snapshot.ports = ports } + } + } + + #if os(iOS) || os(tvOS) || os(visionOS) + /// The session's current output. Clamped to stereo at the bottom: an + /// inactive session reports 0 or 1, and a stereo fold-down is right for + /// every real route that is narrower. + private static func sessionSnapshot() -> (outputChannels: Int, maximumOutputChannels: Int, ports: [String]) { + let session = AVAudioSession.sharedInstance() + return ( + max(2, session.outputNumberOfChannels), + session.maximumOutputNumberOfChannels, + session.currentRoute.outputs.map(\.portType.rawValue) + ) + } + + /// Route changes are posted on a system thread; the width is re-read + /// there. A change of the negotiated width alone (the host asking for more + /// channels) arrives as `.routeConfigurationChange`, so every reason is + /// taken. + private func followSession() { + // The token is created inside the lock so it never leaves the + // mutex's region (it is not Sendable). + observers.withLock { observers in + observers.append(NotificationCenter.default.addObserver( + forName: AVAudioSession.routeChangeNotification, + object: nil, + queue: nil + ) { [weak self] _ in + let current = Self.sessionSnapshot() + self?.update( + outputChannels: current.outputChannels, + maximumOutputChannels: current.maximumOutputChannels, + ports: current.ports + ) + }) + } + } + #endif +} diff --git a/Sources/LumeEngineCore/Media/ContentDisplayCriteria.swift b/Sources/LumeEngineCore/Media/ContentDisplayCriteria.swift new file mode 100644 index 0000000..b2e0691 --- /dev/null +++ b/Sources/LumeEngineCore/Media/ContentDisplayCriteria.swift @@ -0,0 +1,369 @@ +import CoreMedia +import CoreVideo +import Foundation + +/// The display mode a session's video wants: refresh rate and dynamic range, +/// as an Apple TV with Match Content switches the TV for it. +/// +/// Describes what the engine *outputs*, which is not what the container says. +/// The deinterlacer at field rate turns 1080i50 into 50p, so the TV should +/// run at 50 Hz, not 25; Dolby Vision with an IPT base layer (profiles 5, +/// 10.0, 20) is converted to HDR10 before it is enqueued, and every other +/// Dolby Vision profile is presented as its base layer (HDR10, HLG or SDR) — +/// so the TV is asked for that, never for Dolby Vision. Frames the renderer has been handed decide when +/// there are any (`basis == .output`); before that the stream's own metadata, +/// with the deinterlacer's effect predicted, stands in (`.stream`). +/// +/// Cross-platform and pure: tvOS applies it with `DisplayCriteriaMatcher`, +/// anything else can log or show it. +public struct ContentDisplayCriteria: Sendable, Equatable, CustomStringConvertible { + public enum DynamicRange: String, Sendable { + case sdr + case hdr10 + case hlg + /// Never produced today: the engine presents Dolby Vision streams as + /// their base layer (or converts them to HDR10). Reserved for a real + /// Dolby Vision output path (PLAN.md §7, compressed `dvh1` enqueue) — + /// that is the point to ask the TV for Dolby Vision mode — so switches + /// written now stay exhaustive then. + case dolbyVision + } + + /// What the criteria were derived from. + public enum Basis: String, Sendable { + /// Stream metadata, with the deinterlacer's effect predicted. + case stream + /// Frames handed to the renderer: their colour tags and measured + /// cadence. + case output + } + + /// The refresh rate to ask for: one of the film and broadcast rates + /// (23.976, 24, 25, 29.97, 30, 50, 59.94, 60 — NTSC rates exact, e.g. + /// 24000/1001). Nil when the content's rate is unknown or off that grid + /// (variable, 12.5, 48, 100 fps): a TV asked for an odd rate either + /// refuses or picks something worse than the mode it is already in. + public let refreshRate: Double? + public let dynamicRange: DynamicRange + public let width: Int + public let height: Int + public let basis: Basis + + // Ingredients of `makeFormatDescription()`. + let codecType: CMVideoCodecType + let colorPrimaries: String? + let transferFunction: String? + let yCbCrMatrix: String? + let masteringDisplayColorVolume: Data? + let contentLightLevel: Data? + + /// Whether the two ask the TV for the same mode — the only thing that + /// decides whether applying one after the other switches the TV again. + public func asksForSameMode(as other: ContentDisplayCriteria) -> Bool { + refreshRate == other.refreshRate && dynamicRange == other.dynamicRange + } + + public var description: String { + let rate = refreshRate.map { String(format: "%.3gHz", $0) } ?? "anyHz" + return "\(rate)/\(dynamicRange.rawValue)(\(basis.rawValue))" + } + + /// A video format description carrying these criteria, for + /// `AVDisplayCriteria(refreshRate:formatDescription:)`. Colour tags come + /// from the enqueued frames when there are any, so the description says + /// what the display is actually handed. + public func makeFormatDescription() -> CMVideoFormatDescription? { + var extensions: [CFString: Any] = [:] + if let colorPrimaries { extensions[kCMFormatDescriptionExtension_ColorPrimaries] = colorPrimaries } + if let transferFunction { extensions[kCMFormatDescriptionExtension_TransferFunction] = transferFunction } + if let yCbCrMatrix { extensions[kCMFormatDescriptionExtension_YCbCrMatrix] = yCbCrMatrix } + if let masteringDisplayColorVolume { + extensions[kCMFormatDescriptionExtension_MasteringDisplayColorVolume] = masteringDisplayColorVolume + } + if let contentLightLevel { + extensions[kCMFormatDescriptionExtension_ContentLightLevelInfo] = contentLightLevel + } + var description: CMVideoFormatDescription? + let status = CMVideoFormatDescriptionCreate( + allocator: kCFAllocatorDefault, + codecType: codecType, + width: Int32(clamping: width), + height: Int32(clamping: height), + extensions: extensions as CFDictionary, + formatDescriptionOut: &description + ) + return status == noErr ? description : nil + } +} + +// MARK: - Resolution + +extension ContentDisplayCriteria { + /// What the stream says about itself (from `TrackInfo`). + struct Stream: Sendable, Equatable { + var codecName: String + var width: Int + var height: Int + /// Frames per second as the container guesses it; 0 when unknown. + var frameRate: Double + var isInterlaced: Bool? + /// Raw transfer characteristics (ITU-T H.273 code point). + var colorTransfer: UInt32 + var dolbyVision: DolbyVisionConfiguration? + + init( + codecName: String, + width: Int, + height: Int, + frameRate: Double, + isInterlaced: Bool? = nil, + colorTransfer: UInt32 = 2, + dolbyVision: DolbyVisionConfiguration? = nil + ) { + self.codecName = codecName + self.width = width + self.height = height + self.frameRate = frameRate + self.isInterlaced = isInterlaced + self.colorTransfer = colorTransfer + self.dolbyVision = dolbyVision + } + + init?(track: TrackInfo) { + guard let video = track.video else { return nil } + self.init( + codecName: track.codecName, + width: video.width, + height: video.height, + frameRate: video.fps, + isInterlaced: video.isInterlaced, + colorTransfer: video.colorTransfer, + dolbyVision: video.dolbyVision + ) + } + } + + /// What the renderer has actually been handed. + struct Output: Sendable, Equatable { + var width: Int + var height: Int + var colorPrimaries: String? + var transferFunction: String? + var yCbCrMatrix: String? + var masteringDisplayColorVolume: Data? + var contentLightLevel: Data? + /// Measured interval between frames, engine µs; nil until known. + var frameInterval: Double? + /// Frames are going through the deinterlacer. + var isDeinterlacing: Bool + + init( + width: Int, + height: Int, + colorPrimaries: String? = nil, + transferFunction: String? = nil, + yCbCrMatrix: String? = nil, + masteringDisplayColorVolume: Data? = nil, + contentLightLevel: Data? = nil, + frameInterval: Double? = nil, + isDeinterlacing: Bool = false + ) { + self.width = width + self.height = height + self.colorPrimaries = colorPrimaries + self.transferFunction = transferFunction + self.yCbCrMatrix = yCbCrMatrix + self.masteringDisplayColorVolume = masteringDisplayColorVolume + self.contentLightLevel = contentLightLevel + self.frameInterval = frameInterval + self.isDeinterlacing = isDeinterlacing + } + + /// Reads the tags the enqueued frames carried into their format + /// description (pixel buffer attachments propagate into it). + init(formatDescription: CMFormatDescription, frameInterval: Double?, isDeinterlacing: Bool) { + func string(_ key: CFString) -> String? { + CMFormatDescriptionGetExtension(formatDescription, extensionKey: key) as? String + } + func data(_ key: CFString) -> Data? { + CMFormatDescriptionGetExtension(formatDescription, extensionKey: key) as? Data + } + let dimensions = CMVideoFormatDescriptionGetDimensions(formatDescription) + self.init( + width: Int(dimensions.width), + height: Int(dimensions.height), + colorPrimaries: string(kCMFormatDescriptionExtension_ColorPrimaries), + transferFunction: string(kCMFormatDescriptionExtension_TransferFunction), + yCbCrMatrix: string(kCMFormatDescriptionExtension_YCbCrMatrix), + masteringDisplayColorVolume: data(kCMFormatDescriptionExtension_MasteringDisplayColorVolume), + contentLightLevel: data(kCMFormatDescriptionExtension_ContentLightLevelInfo), + frameInterval: frameInterval, + isDeinterlacing: isDeinterlacing + ) + } + } + + /// The criteria for a stream, refined by its output once there is any. + static func resolve( + stream: Stream, + deinterlacing: VideoDecoder.Deinterlacing, + output: Output? + ) -> ContentDisplayCriteria { + let frameRate: Double + let basis: Basis + if let interval = output?.frameInterval, interval > 0 { + // The cadence of what is enqueued already includes whatever the + // deinterlacer did to it. + frameRate = Double(MediaTime.timeBase) / interval + basis = .output + } else { + let deinterlaced = output?.isDeinterlacing ?? willDeinterlace(stream, deinterlacing) + let fieldRate = deinterlaced && deinterlacing.rate == .field + frameRate = stream.frameRate * (fieldRate ? 2 : 1) + basis = .stream + } + + let range = dynamicRange(stream: stream, output: output) + let tags = output.map { + ColorTags(primaries: $0.colorPrimaries, transfer: $0.transferFunction, matrix: $0.yCbCrMatrix) + } ?? ColorTags.canonical(for: range) + + return ContentDisplayCriteria( + refreshRate: nominalRefreshRate(for: frameRate), + dynamicRange: range, + width: output?.width ?? stream.width, + height: output?.height ?? stream.height, + basis: basis, + codecType: codecType(codecName: stream.codecName), + colorPrimaries: tags.primaries, + transferFunction: tags.transfer, + yCbCrMatrix: tags.matrix, + masteringDisplayColorVolume: output?.masteringDisplayColorVolume, + contentLightLevel: output?.contentLightLevel + ) + } + + /// Before the first frame: whether the deinterlacer is going to engage. + /// `.auto` engages on the first frame the decoder flags interlaced, which + /// the stream's field order predicts; an unknown field order predicts + /// progressive, the common case. + private static func willDeinterlace(_ stream: Stream, _ policy: VideoDecoder.Deinterlacing) -> Bool { + switch policy.mode { + case .off: false + case .always: true + case .auto: stream.isInterlaced == true + } + } + + /// The range of what is presented. Dolby Vision with an IPT base layer is + /// converted to HDR10, so it is HDR10 whatever its frames were tagged + /// before conversion. Any other Dolby Vision stream is presented as its + /// base layer — the engine has no Dolby Vision output, the RPU is not + /// applied — so it is treated like the HDR10, HLG or SDR stream it then + /// is: the transfer function of the enqueued frames decides, and before + /// there are any, the base layer's compatibility ID (more reliable than + /// the VUI, which DV encodes often leave unspecified), else the stream's + /// transfer characteristics. Asking for Dolby Vision mode belongs to a + /// real Dolby Vision output path (PLAN.md §7), not to this one. + static func dynamicRange(stream: Stream, output: Output?) -> DynamicRange { + if stream.dolbyVision?.baseLayerIsIPT == true { + return .hdr10 + } + if let output { + // An untagged frame is shown as SDR, so it is SDR. + let transfer = output.transferFunction + if transfer == kCVImageBufferTransferFunction_SMPTE_ST_2084_PQ as String { return .hdr10 } + if transfer == kCVImageBufferTransferFunction_ITU_R_2100_HLG as String { return .hlg } + return .sdr + } + if let presented = stream.dolbyVision.flatMap(baseLayerRange) { + return presented + } + // H.273 transfer characteristics, which FFmpeg's enum mirrors: + // 16 is SMPTE ST 2084 (PQ), 18 is ARIB STD-B67 (HLG). + switch stream.colorTransfer { + case 16: return .hdr10 + case 18: return .hlg + default: return .sdr + } + } + + /// What a Dolby Vision base layer is on its own, by its signal + /// compatibility ID; nil for IDs that say nothing displayable. + private static func baseLayerRange(_ configuration: DolbyVisionConfiguration) -> DynamicRange? { + switch configuration.baseLayerCompatibilityID { + case 1, 6: .hdr10 // HDR10 (6: BT.2020 HDR10, profile 7) + case 4: .hlg + case 2: .sdr + default: nil + } + } + + // MARK: Refresh rates + + /// The rates an Apple TV offers a TV: film, PAL and NTSC, at frame and + /// field rate. + static let nominalRefreshRates: [Double] = [ + 24000.0 / 1001, 24, 25, 30000.0 / 1001, 30, 50, 60000.0 / 1001, 60, + ] + + /// Relative distance within which a measured rate counts as nominal. + /// Wide enough for timestamp jitter, narrow enough that nothing between + /// two grid rates (25.5, 48) is pulled onto one. 23.976 and 24 are 0.1 % + /// apart; nearest-wins keeps them apart. + static let refreshRateTolerance = 0.005 + + static func nominalRefreshRate(for frameRate: Double) -> Double? { + guard frameRate.isFinite, frameRate > 0 else { return nil } + guard let nearest = nominalRefreshRates.min(by: { abs($0 - frameRate) < abs($1 - frameRate) }), + abs(nearest - frameRate) / nearest <= refreshRateTolerance + else { return nil } + return nearest + } + + // MARK: Format description ingredients + + private struct ColorTags { + var primaries: String? + var transfer: String? + var matrix: String? + + /// What a stream of that range is tagged as, for the criteria built + /// before any frame exists. + static func canonical(for range: DynamicRange) -> ColorTags { + switch range { + case .sdr: + ColorTags( + primaries: kCVImageBufferColorPrimaries_ITU_R_709_2 as String, + transfer: kCVImageBufferTransferFunction_ITU_R_709_2 as String, + matrix: kCVImageBufferYCbCrMatrix_ITU_R_709_2 as String + ) + case .hdr10, .dolbyVision: + ColorTags( + primaries: kCVImageBufferColorPrimaries_ITU_R_2020 as String, + transfer: kCVImageBufferTransferFunction_SMPTE_ST_2084_PQ as String, + matrix: kCVImageBufferYCbCrMatrix_ITU_R_2020 as String + ) + case .hlg: + ColorTags( + primaries: kCVImageBufferColorPrimaries_ITU_R_2020 as String, + transfer: kCVImageBufferTransferFunction_ITU_R_2100_HLG as String, + matrix: kCVImageBufferYCbCrMatrix_ITU_R_2020 as String + ) + } + } + } + + /// The sample entry type of the stream's codec. The display mode does not + /// depend on it; a familiar type keeps the description ordinary, so an + /// unknown codec is described as H.264. + static func codecType(codecName: String) -> CMVideoCodecType { + switch codecName { + case "hevc": return kCMVideoCodecType_HEVC + case "av1": return kCMVideoCodecType_AV1 + case "vp9": return kCMVideoCodecType_VP9 + case "mpeg2video": return kCMVideoCodecType_MPEG2Video + default: return kCMVideoCodecType_H264 + } + } +} diff --git a/Sources/LumeEngineCore/Media/DisplayCriteriaMatcher.swift b/Sources/LumeEngineCore/Media/DisplayCriteriaMatcher.swift new file mode 100644 index 0000000..24f203b --- /dev/null +++ b/Sources/LumeEngineCore/Media/DisplayCriteriaMatcher.swift @@ -0,0 +1,203 @@ +#if os(tvOS) +@preconcurrency import AVKit +import Foundation +import UIKit + +/// Match Content for the sample-buffer pipeline: asks the TV for the refresh +/// rate and dynamic range a session's video wants, through the window's +/// `AVDisplayManager`. +/// +/// AVPlayer does this by itself; a sample-buffer renderer cannot, so without +/// it an Apple TV set to Match Frame Rate / Match Dynamic Range leaves a +/// 1080i50 channel judder-converted to the UI's 60 Hz and HDR in the SDR +/// mode. The criteria are the engine's output (`PlayerSession.displayCriteria`), +/// not the container's: field-rate deinterlacing makes 50i a 50 Hz request, +/// and an IPT Dolby Vision stream converted to HDR10 asks for HDR10. +/// +/// Opt-in: nothing changes the display mode unless a host creates a matcher. +/// One per player screen, kept across sessions: zapping between two channels +/// that want the same mode must not send the TV through the UI's mode and +/// back, so `match` only asks when the mode differs, and only `reset` (the +/// player screen going away) hands the display back. An unreset preference +/// keeps the TV in the content's mode after playback — call `reset`. +/// +/// A mode switch blanks the TV for one to three seconds. While the matcher is +/// waiting for one, it holds the session's start (`PlayerSession.holdStart`), +/// so the opening plays to a lit screen; the hold is bounded by +/// `Options.maximumHold`, in the session as well as here. +@MainActor +public final class DisplayCriteriaMatcher { + public struct Options: Sendable { + /// Hold the session's start while the TV switches mode. + public var holdsStartDuringModeSwitch = true + /// Upper bound on that hold, counted from `match(_:)`. The session + /// starts by itself once it passes. + public var maximumHold: Double = 6 + /// How long to wait for the first frames to confirm the criteria + /// before asking for what the stream predicts. Frames usually arrive + /// well inside it; a prediction that turns out wrong costs a second + /// mode switch. + public var outputWait: Double = 2 + /// After asking for a mode, how long to wait for the switch to begin + /// before concluding the TV is already in it. + public var switchStartWindow: Double = 0.75 + + public init() {} + } + + /// The window whose display is matched — the one showing the video. + /// Without one nothing is applied. + public weak var window: UIWindow? + public let options: Options + /// What the display was last asked for; nil when nothing is, or after + /// `reset`. + public private(set) var appliedCriteria: ContentDisplayCriteria? + + private var trackingTask: Task? + private weak var heldSession: PlayerSession? + /// Sessions that show no frames are given up on after this long. + private static let trackingLimit: Double = 15 + + public init(window: UIWindow? = nil, options: Options = Options()) { + self.window = window + self.options = options + } + + /// True while the user's settings let apps switch the display mode + /// (Settings → Video and Audio → Match Content). + public var isMatchingEnabled: Bool { + window?.avDisplayManager.isDisplayCriteriaMatchingEnabled ?? false + } + + /// True while the display is switching modes and may be blank. + public var isModeSwitchInProgress: Bool { + window?.avDisplayManager.isDisplayModeSwitchInProgress ?? false + } + + /// Matches the display to `session`'s video. Call after `open` succeeds + /// and before `play()`: the start hold is in place when this returns, the + /// rest (waiting for the first frames, the switch) continues in the + /// background. A later call for another session supersedes this one. + public func match(_ session: PlayerSession) async { + trackingTask?.cancel() + if let previous = heldSession, previous !== session { + await previous.releaseStartHold() + } + heldSession = nil + + let holding = options.holdsStartDuringModeSwitch && isMatchingEnabled + if holding { + heldSession = session + await session.holdStart(for: options.maximumHold) + } + let started = ProcessInfo.processInfo.systemUptime + trackingTask = Task { [weak self] in + await self?.track(session, holding: holding, since: started) + } + } + + /// Hands the display back to the system's default mode and stops + /// matching. Call when the player screen goes away — not between + /// sessions on it. + public func reset() { + trackingTask?.cancel() + trackingTask = nil + if let session = heldSession { + Task { await session.releaseStartHold() } + } + heldSession = nil + if appliedCriteria != nil { + window?.avDisplayManager.preferredDisplayCriteria = nil + } + appliedCriteria = nil + } + + // MARK: Tracking + + /// Applies the criteria once the first frames confirm them, or the + /// stream's prediction when frames are slow, correcting a prediction once + /// if the frames disagree. Mid-stream changes are not chased: every + /// switch blanks the TV, and a live channel's ad break must not. + private func track(_ session: PlayerSession, holding: Bool, since started: TimeInterval) async { + var holding = holding + var applied: ContentDisplayCriteria? + while !Task.isCancelled { + let elapsed = ProcessInfo.processInfo.systemUptime - started + if let criteria = await session.displayCriteria { + let confirmed = criteria.basis == .output + if applied == nil, confirmed || elapsed >= options.outputWait { + await apply(criteria, to: session, holding: holding, since: started) + applied = criteria + // The TV is in its mode (or the hold ran out): start. + if holding { + holding = false + await release(session) + } + } else if let previous = applied, previous.basis == .stream, confirmed { + if !previous.asksForSameMode(as: criteria) { + await apply(criteria, to: session, holding: false, since: started) + } + applied = criteria + } + if applied?.basis == .output { break } + } else if await session.mediaInfo != nil, await session.selectedVideoTrackIndex == nil { + break // nothing to match: no video + } + guard elapsed < Self.trackingLimit else { break } + try? await Task.sleep(for: .milliseconds(100)) + } + // Gave up without asking (no video, no frames): don't keep the + // session waiting for a switch that is not coming. A cancelled track + // was superseded, and whoever cancelled it released the hold. + if holding, !Task.isCancelled { + await release(session) + } + } + + private func release(_ session: PlayerSession) async { + await session.releaseStartHold() + if heldSession === session { heldSession = nil } + } + + private func apply( + _ criteria: ContentDisplayCriteria, + to session: PlayerSession, + holding: Bool, + since started: TimeInterval + ) async { + guard let manager = window?.avDisplayManager, + manager.isDisplayCriteriaMatchingEnabled, + let format = criteria.makeFormatDescription() + else { return } + if let current = appliedCriteria, current.asksForSameMode(as: criteria) { + appliedCriteria = criteria + return // already in it: asking again would at best do nothing + } + // A refresh rate of 0 leaves the rate to the system: content whose + // rate is unknown or off the grid still gets its dynamic range. + manager.preferredDisplayCriteria = AVDisplayCriteria( + refreshRate: Float(criteria.refreshRate ?? 0), + formatDescription: format + ) + appliedCriteria = criteria + guard holding else { return } + await waitForModeSwitch(manager, deadline: started + options.maximumHold) + } + + /// Waits for the switch the last request started, if it started one. + /// Polled: the flag is the documented signal, and a switch lasts seconds. + private func waitForModeSwitch(_ manager: AVDisplayManager, deadline: TimeInterval) async { + func now() -> TimeInterval { ProcessInfo.processInfo.systemUptime } + let asked = now() + var switching = manager.isDisplayModeSwitchInProgress + while !switching, now() - asked < options.switchStartWindow, now() < deadline, !Task.isCancelled { + try? await Task.sleep(for: .milliseconds(50)) + switching = manager.isDisplayModeSwitchInProgress + } + while switching, now() < deadline, !Task.isCancelled { + try? await Task.sleep(for: .milliseconds(50)) + switching = manager.isDisplayModeSwitchInProgress + } + } +} +#endif diff --git a/Sources/LumeEngineCore/Render/FrameCadence.swift b/Sources/LumeEngineCore/Render/FrameCadence.swift new file mode 100644 index 0000000..e455a92 --- /dev/null +++ b/Sources/LumeEngineCore/Render/FrameCadence.swift @@ -0,0 +1,55 @@ +/// The interval between the video frames the renderer is actually handed — +/// measured, because the container's frame rate is a guess about the input, +/// not a statement about the output. The deinterlacer at field rate doubles +/// it (1080i50 comes out as 50p), a field-coded stream can report its field +/// rate as its frame rate, and live sources carry no reliable rate at all. +/// +/// Pure value type, confined to whoever feeds it (the renderer keeps it under +/// its lock): one subtraction per frame, the statistics only when asked. +struct FrameCadence: Sendable, Equatable { + /// Intervals kept: enough to see through the odd dropped or repeated frame. + static let capacity = 32 + /// Intervals needed before a cadence is reported. Fewer is a guess. + static let minimumSamples = 8 + /// Longer gaps are dropouts or seams, not frame intervals (5 fps floor). + static let maximumInterval: Int64 = 200_000 + + private var lastPTS: Int64? + private var intervals: [Int64] = [] + private var nextSlot = 0 + + /// Forgets the previous frame, keeping what was learned: after a seek the + /// stream is the same, only its timeline jumped. + mutating func restart() { + lastPTS = nil + } + + mutating func observe(pts: Int64) { + guard MediaTime.isValid(pts) else { return } + defer { lastPTS = pts } + guard let lastPTS else { return } + let interval = pts - lastPTS + // Backwards or too long: a seam or a dropout, not a frame interval. + guard interval > 0, interval <= Self.maximumInterval else { return } + if intervals.count < Self.capacity { + intervals.append(interval) + } else { + intervals[nextSlot] = interval + nextSlot = (nextSlot + 1) % Self.capacity + } + } + + /// The typical frame interval in engine µs, nil until enough frames were + /// seen. The mean of the middle half: timestamps quantized by a 90 kHz + /// clock and rounded to µs alternate between two neighbouring values + /// (59.94 fps is 16 678 / 16 689 µs), so a single median sample would be + /// off by more than the gap between 59.94 and 60; the outer quartiles + /// drop the odd duplicated or skipped frame. + var interval: Double? { + guard intervals.count >= Self.minimumSamples else { return nil } + let sorted = intervals.sorted() + let quarter = sorted.count / 4 + let middle = sorted[quarter..<(sorted.count - quarter)] + return Double(middle.reduce(0, +)) / Double(middle.count) + } +} diff --git a/Sources/LumeEngineCore/Render/SystemRenderer.swift b/Sources/LumeEngineCore/Render/SystemRenderer.swift index 5513a50..f2a30fb 100644 --- a/Sources/LumeEngineCore/Render/SystemRenderer.swift +++ b/Sources/LumeEngineCore/Render/SystemRenderer.swift @@ -37,6 +37,11 @@ public final class SystemRenderer: @unchecked Sendable { /// by a flush): startup diagnostics. private var firstVideoEnqueue: TimeInterval? private var firstAudioEnqueue: TimeInterval? + /// What the display is actually being handed — format and cadence of the + /// enqueued frames, after deinterlacing and Dolby Vision conversion. Read + /// for the display criteria (`PlayerSession.displayCriteria`). + private var videoCadence = FrameCadence() + private var lastVideoFormat: CMVideoFormatDescription? private var videoPumpScheduled = false private var audioPumpScheduled = false private var stopped = false @@ -46,6 +51,8 @@ public final class SystemRenderer: @unchecked Sendable { private var audioFormatCache: (description: CMAudioFormatDescription, sampleRate: Int, channels: Int, channelBitmap: UInt64)? // Audio-queue-confined; serial changes re-anchor it after a seek flush. private var audioTimeline = AudioTimeline() + /// Route width at the previous audio pump pass. Audio-queue-confined. + private var lastAudioRouteChannels = Int.max /// Audio enqueued but not yet played, kept so it can be enqueued again /// after the renderer drops it on its own (see `refillAfterAutomaticFlush`). @@ -55,6 +62,7 @@ public final class SystemRenderer: @unchecked Sendable { let serial: UInt64 /// End of the sample on the engine timeline (µs). let end: Int64 + let channels: Int } private var retainedAudio: [RetainedAudio] = [] /// Backstop only: the renderer stops asking for audio once it holds a @@ -83,7 +91,13 @@ public final class SystemRenderer: @unchecked Sendable { private var audioLaneAttached = false private var reportedFailure = false - public init(muted: Bool = false) { + /// The route the audio lane's decoder fits its output to. Audio decoded + /// for a wider route before it narrowed is held back from the renderer + /// (see `pumpAudio`); nil feeds every frame. + private let audioOutputRoute: AudioOutputRoute? + + public init(muted: Bool = false, audioOutputRoute: AudioOutputRoute? = nil) { + self.audioOutputRoute = audioOutputRoute displayLayer = AVSampleBufferDisplayLayer() displayLayer.videoGravity = .resizeAspect videoRenderer = displayLayer.sampleBufferRenderer @@ -166,7 +180,11 @@ public final class SystemRenderer: @unchecked Sendable { // answer; that audio is retained too, so drop it and re-enqueue // everything in order. audioRenderer.flush() - for entry in retainedAudio where entry.serial == serial && (!MediaTime.isValid(flushTime) || entry.end > flushTime) { + // A route change is the usual cause of the flush; audio retained for + // a wider route than the new one stays out (see `pumpAudio`). + let routeChannels = audioOutputRoute?.outputChannels ?? .max + for entry in retainedAudio where entry.serial == serial && entry.channels <= routeChannels + && (!MediaTime.isValid(flushTime) || entry.end > flushTime) { audioRenderer.enqueue(entry.sample) } // A seek that flushed while this ran may have been overtaken by the @@ -312,6 +330,7 @@ public final class SystemRenderer: @unchecked Sendable { lastEnqueuedAudioPTS = MediaTime.noTimestamp firstEnqueuedVideoPTS = MediaTime.noTimestamp firstEnqueuedAudioPTS = MediaTime.noTimestamp + videoCadence.restart() lock.unlock() videoRenderer.flush() @@ -345,6 +364,15 @@ public final class SystemRenderer: @unchecked Sendable { return (firstVideoEnqueue, firstAudioEnqueue) } + /// Format of the most recently enqueued video frame and the measured + /// interval between frames (engine µs; nil until enough were seen). Nil + /// before the first frame. + var videoOutput: (format: CMVideoFormatDescription, frameInterval: Double?)? { + lock.lock() + defer { lock.unlock() } + return lastVideoFormat.map { ($0, videoCadence.interval) } + } + /// Raw renderer health for diagnostics (status is /// `AVQueuedSampleBufferRenderingStatus`: 0 unknown, 1 rendering, 2 failed). public var rendererHealth: (videoStatus: Int, audioStatus: Int, videoError: String?, audioError: String?) { @@ -356,6 +384,10 @@ public final class SystemRenderer: @unchecked Sendable { ) } + var audioRendererFailed: Bool { + audioRenderer.status == .failed + } + public func shutdown() { lock.lock() stopped = true @@ -402,6 +434,8 @@ public final class SystemRenderer: @unchecked Sendable { lastEnqueuedVideoPTS = frame.pts if !MediaTime.isValid(firstEnqueuedVideoPTS) { firstEnqueuedVideoPTS = frame.pts } if firstVideoEnqueue == nil { firstVideoEnqueue = ProcessInfo.processInfo.systemUptime } + videoCadence.observe(pts: frame.pts) + lastVideoFormat = videoFormatCache lock.unlock() } catch { continue // malformed frame: drop, keep the pump alive @@ -417,12 +451,25 @@ public final class SystemRenderer: @unchecked Sendable { lock.unlock() guard let input, !isStopped else { return } + // The route narrowed (HDMI surround → AirPlay stereo): audio already + // decoded for the wider route must not reach the renderer, which + // fails on more channels than the route carries. The decoder refits + // at its next frame; until then what is queued for the old width is + // dropped, and whatever the renderer still holds of it is replaced by + // the retained audio that fits — usually nothing, so the switch costs + // the renderer's lead in silence, as a route change does anyway. + let routeChannels = audioOutputRoute?.outputChannels ?? .max + if routeChannels < lastAudioRouteChannels { + refillAfterAutomaticFlush(from: currentTime) + } + lastAudioRouteChannels = routeChannels + while audioRenderer.isReadyForMoreMediaData { guard let frame = input.tryReceive() else { scheduleAudioRetry() return } - guard frame.serial == serial else { continue } + guard frame.serial == serial, frame.channels <= routeChannels else { continue } do { let sample = try SampleBufferBuilder.audio( from: frame, @@ -435,7 +482,7 @@ public final class SystemRenderer: @unchecked Sendable { if !MediaTime.isValid(firstEnqueuedAudioPTS) { firstEnqueuedAudioPTS = frame.pts } if firstAudioEnqueue == nil { firstAudioEnqueue = ProcessInfo.processInfo.systemUptime } lock.unlock() - retainAudio(sample, serial: serial, end: frame.pts + frame.duration) + retainAudio(sample, serial: serial, end: frame.pts + frame.duration, channels: frame.channels) } catch { continue } @@ -443,13 +490,13 @@ public final class SystemRenderer: @unchecked Sendable { } /// Keeps `sample` until the clock has played past it. Audio queue only. - private func retainAudio(_ sample: CMSampleBuffer, serial: UInt64, end: Int64) { + private func retainAudio(_ sample: CMSampleBuffer, serial: UInt64, end: Int64, channels: Int) { let now = currentTime retainedAudio.removeAll { entry in entry.serial != serial || (MediaTime.isValid(now) && entry.end <= now) } if retainedAudio.count >= Self.maxRetainedAudio { retainedAudio.removeFirst() } - retainedAudio.append(RetainedAudio(sample: sample, serial: serial, end: end)) + retainedAudio.append(RetainedAudio(sample: sample, serial: serial, end: end, channels: channels)) } private func scheduleVideoRetry() { diff --git a/Sources/LumeEngineCore/Session/AudioHealth.swift b/Sources/LumeEngineCore/Session/AudioHealth.swift new file mode 100644 index 0000000..5c69f4f --- /dev/null +++ b/Sources/LumeEngineCore/Session/AudioHealth.swift @@ -0,0 +1,153 @@ +import Foundation + +/// Whether the session's audio is reaching the viewer, as far as the engine +/// can tell. Read `PlayerSession.audioHealth`; it is also in `Diagnostics`. +/// +/// Exists because audio can die while video plays on: a decoder that gives up +/// leaves its lane at end of stream, and the session — correctly — keeps +/// showing the picture. Without this the only trace was one `.error` event, +/// and the viewer sat in front of a silent channel. +public enum AudioHealth: Sendable, Equatable, CustomStringConvertible { + /// Audio is playing, or nothing says otherwise yet. + case healthy + /// The session has no audio lane: the source carries no audio, or the + /// host opened it without (`enableAudio`) or turned it off. + case absent + /// The audio lane has stopped while the session plays on. + case lost(AudioLossReason) + + public var description: String { + switch self { + case .healthy: "ok" + case .absent: "none" + case .lost(let reason): "lost(\(reason.rawValue))" + } + } +} + +/// Which stage the audio was lost in. +public enum AudioLossReason: String, Sendable, Equatable { + /// The audio decoder gave up on the stream (its error went out as an + /// `.error` event). Permanent for the track: switching tracks, or audio + /// off and on, builds a new decoder. + case decoderFailed + /// The system audio renderer failed. The session fails with + /// `.renderFailed` as well; this records that it was the audio lane. + case rendererFailed + /// The clock runs and video plays, but for a while now the audio renderer + /// has had nothing to play at the playhead: decoded audio stopped reaching + /// it, or reached it too late to play. Reported once per episode as an + /// `.error` with `EngineError.Code.audioLost`; clears by itself when audio + /// comes back. + case starved +} + +/// Decides `AudioHealth` from what the session observes on its monitor tick. +/// +/// Pure value logic, so every branch is testable without a pipeline that has +/// to be made to lose its audio. +/// +/// The starvation judgement is deliberately narrow, so that moments without +/// audio that are part of normal playback never count: it needs a running +/// clock (not paused, buffering or seeking), a video lane that is itself ahead +/// of the playhead (a whole pipeline running dry is buffering, which the +/// session handles), an audio lane still expecting data (not at its end), and +/// the playhead to have moved `threshold` seconds of media with no audio +/// enqueued beyond it. A gap *inside* the audio does not count either: the +/// renderer's high-water mark is past the gap as soon as audio after it was +/// enqueued. +struct AudioLossDetector { + /// Seconds of media the playhead must cover without audio ahead of it. + static let threshold = 2.0 + + struct Observation: Equatable { + /// An audio lane is attached. + var hasAudioLane: Bool + /// The audio decoder reached the end of its stream: nothing more is + /// coming, and silence after it is the source's end, not a loss. + var audioEnded: Bool + /// The audio renderer reports `.failed`. + var audioRendererFailed: Bool + /// The session plays and the clock rate is non-zero. + var clockRunning: Bool + /// Playhead, seconds; nil while the clock has no valid time. + var playhead: Double? + /// Audio is enqueued beyond the playhead. + var audioAhead: Bool + /// Video is enqueued beyond the playhead; nil without a video lane. + var videoAhead: Bool? + /// The session's seek serial: a seek rebuilds every queue, so any + /// episode in progress is void. + var serial: UInt64 + } + + private(set) var health: AudioHealth = .absent + /// A loss only a new decoder clears. + private var permanent: AudioLossReason? + /// Playhead when audio was first seen missing; nil while it is not. + private var dryFrom: Double? + private var serial: UInt64 = 0 + + /// The audio decoder failed for good. + mutating func decoderFailed() { + permanent = .decoderFailed + health = .lost(.decoderFailed) + } + + /// The audio lane was rebuilt or removed (track switch, audio off/on): + /// whatever was wrong with the old one is gone with it. + mutating func laneReplaced() { + permanent = nil + dryFrom = nil + health = .healthy + } + + /// Updates `health`; returns the reason when a starvation episode begins, + /// which is the moment to tell the host. + mutating func observe(_ observation: Observation) -> AudioLossReason? { + guard observation.hasAudioLane else { + permanent = nil + dryFrom = nil + health = .absent + return nil + } + if health == .absent { + health = .healthy + } + if observation.audioRendererFailed { + permanent = .rendererFailed + } + if let permanent { + dryFrom = nil + health = .lost(permanent) + return nil + } + if observation.serial != serial { + serial = observation.serial + dryFrom = nil + health = .healthy + } + if observation.audioEnded || observation.audioAhead { + dryFrom = nil + health = .healthy + return nil + } + // Not judged: paused, buffering or seeking (the clock does not run), + // no time yet, or no healthy video to compare against. A standing + // verdict stays until audio is seen ahead again. + guard observation.clockRunning, + let playhead = observation.playhead, + observation.videoAhead == true + else { + dryFrom = nil + return nil + } + guard let start = dryFrom, playhead >= start else { + dryFrom = playhead + return nil + } + guard playhead - start >= Self.threshold, health != .lost(.starved) else { return nil } + health = .lost(.starved) + return .starved + } +} diff --git a/Sources/LumeEngineCore/Session/PlayerSession+DisplayCriteria.swift b/Sources/LumeEngineCore/Session/PlayerSession+DisplayCriteria.swift new file mode 100644 index 0000000..62e8f63 --- /dev/null +++ b/Sources/LumeEngineCore/Session/PlayerSession+DisplayCriteria.swift @@ -0,0 +1,30 @@ +extension PlayerSession { + /// The display mode the selected video wants — refresh rate and dynamic + /// range of what the engine outputs, after deinterlacing and Dolby Vision + /// conversion. Predicted from the stream once it is open + /// (`basis == .stream`), confirmed from the enqueued frames once enough + /// have gone to the renderer (`.output`). Nil without a video lane. + /// + /// Cheap (a lock and a sort of a few dozen intervals); poll it at a low + /// rate. tvOS applies it with `DisplayCriteriaMatcher`. + public var displayCriteria: ContentDisplayCriteria? { + guard let info = mediaInfo, + let index = selectedVideoTrackIndex, + let track = info.tracks.first(where: { $0.index == index }), + let stream = ContentDisplayCriteria.Stream(track: track) + else { return nil } + let isDeinterlacing = activeVideoDecoder?.isDeinterlacing ?? false + let output = renderer.videoOutput.map { + ContentDisplayCriteria.Output( + formatDescription: $0.format, + frameInterval: $0.frameInterval, + isDeinterlacing: isDeinterlacing + ) + } + return ContentDisplayCriteria.resolve( + stream: stream, + deinterlacing: configuration.deinterlace, + output: output + ) + } +} diff --git a/Sources/LumeEngineCore/Session/PlayerSession.swift b/Sources/LumeEngineCore/Session/PlayerSession.swift index 40d1593..d115d7d 100644 --- a/Sources/LumeEngineCore/Session/PlayerSession.swift +++ b/Sources/LumeEngineCore/Session/PlayerSession.swift @@ -70,6 +70,11 @@ public struct PlayerConfiguration: Sendable { public var autoEnableForcedSubtitlesForForeignAudio = false /// Seconds without clock progress (while expected) before `.stalled` fires. public var stallThreshold: Double = 8 + /// The output route the audio is fitted to. nil follows the shared audio + /// session through route changes (iOS/tvOS/visionOS; stereo on macOS). + /// Pass one to pin the width or drive it yourself, e.g. for a macOS + /// multichannel interface. + public var audioOutputRoute: AudioOutputRoute? public init() {} } @@ -96,7 +101,7 @@ public actor PlayerSession { public nonisolated let events: AsyncStream private let eventSink: AsyncStream.Continuation - private let configuration: PlayerConfiguration + let configuration: PlayerConfiguration /// The presentation backend; expose `renderer.displayLayer` in a view. public nonisolated let renderer: SystemRenderer /// Active subtitle cues (embedded track or external file). Query with @@ -151,6 +156,9 @@ public actor PlayerSession { /// The stage that stopped at the most recent `.stalled`; nil before any. public private(set) var lastStallReason: StallReason? + /// `systemUptime` until which the clock may not start (see `holdStart`). + private var startHoldDeadline: TimeInterval? + // Startup timeline, `systemUptime` stamps (see `startup`). private var openStartedAt: TimeInterval? private var openedAt: TimeInterval? @@ -164,9 +172,20 @@ public actor PlayerSession { public private(set) var selectedSubtitleTrackIndex: Int32? public private(set) var usingExternalSubtitles = false + /// Width the audio lane is fitted to; shared with its decoder and the renderer. + private let audioRoute: AudioOutputRoute + private var audioLoss = AudioLossDetector() + + /// Whether audio is reaching the viewer (see `AudioHealth`). + public var audioHealth: AudioHealth { audioLoss.health } + + /// The layout the audio lane hands the renderer; nil without audio. + public var audioOutput: AudioOutputLayout? { audioDecoder?.currentOutput } + public init(configuration: PlayerConfiguration = PlayerConfiguration()) { self.configuration = configuration - renderer = SystemRenderer(muted: configuration.muted) + audioRoute = configuration.audioOutputRoute ?? AudioOutputRoute.system() + renderer = SystemRenderer(muted: configuration.muted, audioOutputRoute: audioRoute) var continuation: AsyncStream.Continuation! events = AsyncStream(bufferingPolicy: .unbounded) { continuation = $0 } eventSink = continuation @@ -268,7 +287,7 @@ public actor PlayerSession { let packets = makeAudioPacketChannel() let frames = Channel(capacity: configuration.audioQueueDepth, measure: { $0.duration }) demuxer.attach(channel: packets, toStream: track.index) - let decoder = AudioDecoder(parameters: parameters, input: packets, output: frames) + let decoder = AudioDecoder(parameters: parameters, input: packets, output: frames, route: audioRoute) audioPackets = packets audioFrames = frames audioDecoder = decoder @@ -423,6 +442,14 @@ public actor PlayerSession { public let replayedPackets: Int public let startup: StartupTimeline public let lastStallReason: StallReason? + /// The display mode the video wants (see `displayCriteria`). + public let displayCriteria: ContentDisplayCriteria? + /// Whether audio is reaching the viewer. + public let audioHealth: AudioHealth + /// Source → output channels of the audio lane; nil without one. + public let audioOutput: AudioOutputLayout? + /// The output route the audio is fitted to. + public let audioRoute: AudioOutputRoute.Snapshot public var description: String { func lane(_ queue: (count: Int, seconds: Double)?, lead: Double, first: Double) -> String { @@ -448,6 +475,9 @@ public actor PlayerSession { timelineSeams, replayedPackets ) if let lastStallReason { line += " stall=\(lastStallReason.rawValue)" } + if let displayCriteria { line += " display=\(displayCriteria)" } + line += " aout=\(audioOutput.map(String.init(describing:)) ?? "off") route=\(audioRoute)" + if case .lost = audioHealth { line += " audio=\(audioHealth)" } line += " startup=[\(startup)]" return line } @@ -483,10 +513,42 @@ public actor PlayerSession { timelineSeams: demuxer?.timelineSeams ?? 0, replayedPackets: demuxer?.replayedPackets ?? 0, startup: startup, - lastStallReason: lastStallReason + lastStallReason: lastStallReason, + displayCriteria: displayCriteria, + audioHealth: audioHealth, + audioOutput: audioOutput, + audioRoute: audioRoute.snapshot ) } + // MARK: Start hold + + /// Keeps the clock from starting — the session buffers and stays in + /// `.buffering` — until `releaseStartHold()` or until `seconds` have + /// passed, whichever comes first. For a host that has to wait for + /// something outside the engine before the first frame is worth showing: + /// a TV switching display mode blanks for seconds, and playback started + /// into that blank loses its opening (see `DisplayCriteriaMatcher`). + /// Bounded here, not by the caller, so a host that never releases cannot + /// wedge the session. Ignored once playing: a running clock is not + /// stopped, and a later rebuffer must not wait on a stale hold. + public func holdStart(for seconds: Double) { + guard seconds > 0, state != .playing else { return } + startHoldDeadline = ProcessInfo.processInfo.systemUptime + min(seconds, 30) + } + + /// Lets a held start go ahead (see `holdStart`). Idempotent. + public func releaseStartHold() { + guard startHoldDeadline != nil else { return } + startHoldDeadline = nil + evaluatePlayback() + } + + public var isStartHeld: Bool { + guard let deadline = startHoldDeadline else { return false } + return ProcessInfo.processInfo.systemUptime < deadline + } + // MARK: Seek /// Seeks to `position` seconds (media-relative). Flush order matters: @@ -565,6 +627,7 @@ public actor PlayerSession { // MARK: Track-lane internals (used by PlayerSession+Tracks.swift) var activeDemuxer: Demuxer? { demuxer } + var activeVideoDecoder: VideoDecoder? { videoDecoder } /// Packet channels are budgeted in media time (`packetReadAhead`); the /// element counts are memory backstops for streams that report no packet @@ -629,6 +692,7 @@ public actor PlayerSession { audioFrames = nil selectedAudioTrackIndex = nil audioAtEOF = false + audioLoss.laneReplaced() renderer.attach(video: videoFrames, audio: nil) } @@ -642,11 +706,12 @@ public actor PlayerSession { let packets = makeAudioPacketChannel() let frames = Channel(capacity: configuration.audioQueueDepth, measure: { $0.duration }) demuxer.attach(channel: packets, toStream: trackIndex) - let decoder = AudioDecoder(parameters: parameters, input: packets, output: frames) + let decoder = AudioDecoder(parameters: parameters, input: packets, output: frames, route: audioRoute) audioPackets = packets audioFrames = frames audioDecoder = decoder audioAtEOF = false + audioLoss.laneReplaced() selectedAudioTrackIndex = trackIndex let pump = Task { [events = decoder.events] in for await event in events { @@ -804,7 +869,12 @@ public actor PlayerSession { // One lane dying is not necessarily fatal (e.g. broken audio track, // healthy video) — surface it; the monitor decides on full failure. eventSink.yield(.error(error)) - if isVideo { videoAtEOF = true } else { audioAtEOF = true } + if isVideo { + videoAtEOF = true + } else { + audioAtEOF = true + audioLoss.decoderFailed() + } lastError = error } } @@ -819,7 +889,34 @@ public actor PlayerSession { evaluatePlayback() runStallWatchdog() noteFirstFrameShown() + evaluateAudioHealth() + } + } + + /// Audio that stops while the picture plays on is a failure the viewer + /// notices and nothing else would report (see `AudioLossDetector` for + /// what does not count). + private func evaluateAudioHealth() { + let playhead = renderer.currentTime + let highWater = renderer.enqueuedHighWaterMark + func ahead(_ mark: Int64) -> Bool { + MediaTime.isValid(mark) && MediaTime.isValid(playhead) && mark > playhead } + let observation = AudioLossDetector.Observation( + hasAudioLane: audioDecoder != nil, + audioEnded: audioAtEOF, + audioRendererFailed: renderer.audioRendererFailed, + clockRunning: state == .playing && renderer.rate > 0, + playhead: MediaTime.isValid(playhead) ? MediaTime.seconds(playhead) : nil, + audioAhead: ahead(highWater.audio), + videoAhead: videoDecoder != nil ? ahead(highWater.video) : nil, + serial: currentSerial + ) + guard audioLoss.observe(observation) == .starved else { return } + eventSink.yield(.error(EngineError( + code: .audioLost, + message: "audio stopped reaching the renderer while video plays" + ))) } /// A container seek can "succeed" while landing somewhere else entirely — @@ -878,6 +975,7 @@ public actor PlayerSession { && (state == .playing || state == .buffering) && pendingSeekTarget == nil && !demuxAtEOF + && !isStartHeld // buffers fill and stop growing while held guard expectingProgress else { lastProgressDate = .distantFuture @@ -1010,6 +1108,11 @@ public actor PlayerSession { switch state { case .ready, .paused, .buffering: + if isStartHeld { + if state != .buffering { state = .buffering } + return + } + startHoldDeadline = nil // Start when every lane is as buffered as it can usefully get, // or when EOF means no more is coming. if allLanesReady || allEOF || demuxAtEOF { diff --git a/Tests/LumeEngineCoreTests/AudioRouteTests.swift b/Tests/LumeEngineCoreTests/AudioRouteTests.swift new file mode 100644 index 0000000..f792888 --- /dev/null +++ b/Tests/LumeEngineCoreTests/AudioRouteTests.swift @@ -0,0 +1,548 @@ +import CFFmpeg +import Foundation +import Synchronization +import Testing +@testable import LumeEngineCore + +/// Output width that follows the route: the layout policy and the downmix +/// matrix as plain values, then the decoder at forced route widths and across +/// a route change mid-stream. +@Suite("AudioLayoutPolicy") +struct AudioLayoutPolicyTests { + private static let surround51Back: SpeakerSet = [.frontLeft, .frontRight, .frontCenter, .lowFrequency, .backLeft, .backRight] + private static let surround70: SpeakerSet = SpeakerSet.surround71.subtracting(.lowFrequency) + /// A matrix-encoded stereo pair (`AV_CH_STEREO_LEFT | AV_CH_STEREO_RIGHT`): + /// two channels the renderer cannot name. + private static let stereoDownmix = SpeakerSet(rawValue: 1 << 29 | 1 << 30) + + private func plan(_ source: SpeakerSet, route: Int) -> AudioLayoutPolicy.Plan { + AudioLayoutPolicy.plan(source: source, routeChannels: route) + } + + @Test("speaker sets are FFmpeg's native layouts") + func layoutsMatchFFmpeg() { + #expect(SpeakerSet.mono.rawValue == layoutMask("mono")) + #expect(SpeakerSet.stereo.rawValue == layoutMask("stereo")) + #expect(SpeakerSet.surround51.rawValue == layoutMask("5.1(side)")) + #expect(SpeakerSet.surround71.rawValue == layoutMask("7.1")) + } + + @Test("multichannel goes out discrete when the route carries it") + func passthroughWhenItFits() { + for route in [6, 8] { + #expect(plan(.surround51, route: route) == .init(output: .surround51, remixes: false)) + #expect(plan(Self.surround51Back, route: route) == .init(output: Self.surround51Back, remixes: false)) + } + #expect(plan(.surround71, route: 8) == .init(output: .surround71, remixes: false)) + for route in [2, 6, 8] { + #expect(plan(.stereo, route: route) == .init(output: .stereo, remixes: false)) + } + } + + @Test("a narrower route gets the widest standard layout both sides fill") + func foldsDownToTheRoute() { + #expect(plan(.surround51, route: 2) == .init(output: .stereo, remixes: true)) + #expect(plan(Self.surround51Back, route: 2) == .init(output: .stereo, remixes: true)) + #expect(plan(.surround71, route: 2) == .init(output: .stereo, remixes: true)) + #expect(plan(.surround71, route: 6) == .init(output: .surround51, remixes: true)) + #expect(plan(.surround71, route: 7) == .init(output: .surround51, remixes: true)) + // Odd widths fall to the standard layout below them. + #expect(plan(.surround51, route: 4) == .init(output: .stereo, remixes: true)) + } + + @Test("no LFE channel is invented") + func noInventedLFE() { + #expect(plan(Self.surround70, route: 6) == .init(output: SpeakerSet.surround51.subtracting(.lowFrequency), remixes: true)) + } + + @Test("mono reaches both speakers of any route wider than one") + func monoBecomesStereo() { + for route in [2, 6, 8] { + #expect(plan(.mono, route: route) == .init(output: .stereo, remixes: true)) + } + #expect(plan(.mono, route: 1) == .init(output: .mono, remixes: false)) + #expect(plan(.mono, route: 0) == .init(output: .mono, remixes: false)) + #expect(plan(.stereo, route: 1) == .init(output: .mono, remixes: true)) + } + + @Test("channels the renderer cannot name are remixed, not passed untagged") + func unnameableChannelsAreRemixed() { + #expect(plan(Self.stereoDownmix, route: 8) == .init(output: .stereo, remixes: true)) + } +} + +@Suite("DownmixMatrix") +struct DownmixMatrixTests { + private let minus3dB = 0.707_106_781_186_547_6 + + /// The coefficient of input speaker `input` in output speaker `output`. + private func weight( + _ matrix: [[Double]], from source: SpeakerSet, to target: SpeakerSet, + _ input: SpeakerSet, into output: SpeakerSet + ) -> Double { + func index(_ speaker: SpeakerSet, in set: SpeakerSet) -> Int { + // Native order is ascending bit order. + (set.rawValue & (speaker.rawValue - 1)).nonzeroBitCount + } + return matrix[index(output, in: target)][index(input, in: source)] + } + + @Test("5.1 to stereo: ITU fold-down, centre in both, LFE omitted, no cross-feed") + func surround51ToStereo() throws { + let source = SpeakerSet.surround51 + let matrix = try #require(DownmixMatrix.coefficients(from: source, to: .stereo)) + #expect(matrix.count == 2 && matrix.allSatisfy { $0.count == 6 }) + func w(_ input: SpeakerSet, _ output: SpeakerSet) -> Double { + weight(matrix, from: source, to: .stereo, input, into: output) + } + #expect(w(.frontLeft, .frontLeft) == 1) + #expect(w(.frontRight, .frontRight) == 1) + #expect(w(.frontLeft, .frontRight) == 0) + #expect(w(.frontRight, .frontLeft) == 0) + #expect(abs(w(.frontCenter, .frontLeft) - minus3dB) < 1e-9) + #expect(abs(w(.frontCenter, .frontRight) - minus3dB) < 1e-9) + #expect(abs(w(.sideLeft, .frontLeft) - minus3dB) < 1e-9) + #expect(abs(w(.sideRight, .frontRight) - minus3dB) < 1e-9) + #expect(w(.sideLeft, .frontRight) == 0) + #expect(w(.lowFrequency, .frontLeft) == 0) + #expect(w(.lowFrequency, .frontRight) == 0) + } + + @Test("7.1 to stereo: every surround folds into its own side") + func surround71ToStereo() throws { + let source = SpeakerSet.surround71 + let matrix = try #require(DownmixMatrix.coefficients(from: source, to: .stereo)) + func w(_ input: SpeakerSet, _ output: SpeakerSet) -> Double { + weight(matrix, from: source, to: .stereo, input, into: output) + } + for (input, side) in [(SpeakerSet.backLeft, SpeakerSet.frontLeft), (.sideLeft, .frontLeft), + (.backRight, .frontRight), (.sideRight, .frontRight)] { + #expect(abs(w(input, side) - minus3dB) < 1e-9) + } + #expect(w(.backLeft, .frontRight) == 0) + #expect(abs(w(.frontCenter, .frontLeft) - minus3dB) < 1e-9) + #expect(abs(w(.frontCenter, .frontRight) - minus3dB) < 1e-9) + } + + @Test("7.1 to 5.1: fronts, centre and LFE pass straight; backs join the sides") + func surround71To51() throws { + let source = SpeakerSet.surround71 + let target = SpeakerSet.surround51 + let matrix = try #require(DownmixMatrix.coefficients(from: source, to: target)) + func w(_ input: SpeakerSet, _ output: SpeakerSet) -> Double { + weight(matrix, from: source, to: target, input, into: output) + } + for speaker in [SpeakerSet.frontLeft, .frontRight, .frontCenter, .lowFrequency] { + #expect(w(speaker, speaker) == 1) + } + #expect(w(.backLeft, .sideLeft) > 0.5) + #expect(w(.backRight, .sideRight) > 0.5) + #expect(w(.frontCenter, .frontLeft) == 0, "a route with a centre speaker keeps dialogue on it") + } + + @Test("mono to stereo: equal power on both sides") + func monoToStereo() throws { + let matrix = try #require(DownmixMatrix.coefficients(from: .mono, to: .stereo)) + #expect(matrix.count == 2) + #expect(abs(matrix[0][0] - minus3dB) < 1e-9) + #expect(abs(matrix[1][0] - minus3dB) < 1e-9) + } + + @Test("dialogue survives every stereo fold-down") + func dialogueNeverDropped() throws { + let sources: [SpeakerSet] = [ + [.frontLeft, .frontRight, .frontCenter], + .surround51, + [.frontLeft, .frontRight, .frontCenter, .lowFrequency, .backLeft, .backRight], + .surround71, + SpeakerSet.surround51.union(SpeakerSet(rawValue: 1 << 8)), // 6.1, back centre + ] + for source in sources { + let plan = AudioLayoutPolicy.plan(source: source, routeChannels: 2) + let matrix = try #require(DownmixMatrix.coefficients(from: source, to: plan.output)) + let center = (source.rawValue & (SpeakerSet.frontCenter.rawValue - 1)).nonzeroBitCount + #expect(matrix[0][center] > 0.5 && matrix[1][center] > 0.5, "centre lost folding \(source.rawValue)") + // Every input channel except the LFE reaches the output. + for input in 0.. 0, "channel \(input) of \(source.rawValue) dropped") + } + } + } +} + +@Suite("AudioRoute", .serialized) +struct AudioRouteDecodeTests { + /// Decodes an audio-only fixture with the decoder following `route`. + /// `onFrame` runs on the drain thread for each frame, in order — the place + /// to change the route mid-stream. + private func decode( + fixture: String, + route: AudioOutputRoute, + frameQueue: Int = 256, + onFrame: @escaping @Sendable (Int) -> Void = { _ in } + ) async throws -> [AudioFrame] { + let demuxer = Demuxer(url: try Fixtures.path(fixture)) + var events = demuxer.events.makeAsyncIterator() + demuxer.start() + guard case .opened(let info)? = await events.next() else { + demuxer.shutdown() + throw EngineError(code: .openFailed, message: "\(fixture) failed to open") + } + defer { demuxer.shutdown() } + + let track = try #require(info.audioTracks.first) + let parameters = try #require(demuxer.codecParameters(forStream: track.index)) + let packets = Channel(capacity: 4096) + let frames = Channel(capacity: frameQueue, measure: { $0.duration }) + demuxer.attach(channel: packets, toStream: track.index) + + let decoder = AudioDecoder(parameters: parameters, input: packets, output: frames, route: route) + var decodeEvents = decoder.events.makeAsyncIterator() + decoder.start() + demuxer.resume() + defer { decoder.shutdown() } + + let collector = ChannelDrain(frames, into: [AudioFrame]()) { collected, frame in + collected.append(frame) + onFrame(collected.count) + } + while let event = await events.next() { + if case .endOfStream = event { + decoder.signalEndOfStream() + break + } + } + guard case .endOfStream? = await decodeEvents.next() else { + throw EngineError(code: .decodeFailed, message: "\(fixture): decoder never reached EOF") + } + decoder.shutdown() + return await collector.value + } + + /// Per-channel peak amplitude over all frames. + private func peaks(_ frames: [AudioFrame]) -> [Float] { + guard let channels = frames.first?.channels else { return [] } + var peaks = [Float](repeating: 0, count: channels) + for frame in frames { + let samples = frame.samples + for index in 0.. 0.05) + let stereoPeaks = peaks(stereo) + #expect(abs(stereoPeaks[0] - stereoPeaks[1]) < 1e-4) + #expect(abs(stereoPeaks[0] / center - 0.7071) < 0.01, "centre at \(stereoPeaks[0] / center) of its level") + // On a 5.1 route the centre stays on the centre speaker. + #expect(abs(peaks(fiveOne)[2] - center) < 1e-4) + } + + @Test("TrueHD 5.1 at route widths 8 and 2", .timeLimit(.minutes(1))) + func truehdAtRouteWidths() async throws { + let wide = try await decode(fixture: "truehd.mkv", route: AudioOutputRoute(outputChannels: 8)) + #expect(wide.allSatisfy { $0.channels == 6 && $0.channelBitmap == SpeakerSet.surround51.rawValue }) + + let stereo = try await decode(fixture: "truehd.mkv", route: AudioOutputRoute(outputChannels: 2)) + #expect(!stereo.isEmpty) + #expect(stereo.allSatisfy { $0.channels == 2 }) + let center = peaks(wide)[2] + let stereoPeaks = peaks(stereo) + #expect(center > 0.05) + #expect(abs(stereoPeaks[0] / center - 0.7071) < 0.01 && abs(stereoPeaks[1] / center - 0.7071) < 0.01) + let seconds = Double(stereo.reduce(0) { $0 + $1.sampleCount }) / Double(stereo[0].sampleRate) + #expect(abs(seconds - 4.0) < 0.1) + } + + @Test("mono goes out on both speakers", .timeLimit(.minutes(1))) + func monoToStereo() async throws { + // basic.mp4's AAC track is a mono sine. + let mono = try await decode(fixture: "basic.mp4", route: AudioOutputRoute(outputChannels: 1)) + #expect(mono.first?.channels == 1) + let stereo = try await decode(fixture: "basic.mp4", route: AudioOutputRoute(outputChannels: 2)) + #expect(stereo.allSatisfy { $0.channels == 2 && $0.channelBitmap == SpeakerSet.stereo.rawValue }) + let level = peaks(mono)[0] + let sides = peaks(stereo) + #expect(level > 0.05) + #expect(abs(sides[0] - sides[1]) < 1e-4) + #expect(abs(sides[0] / level - 0.7071) < 0.01) + } + + @Test("a route change mid-stream switches layout at a frame boundary, losing nothing", .timeLimit(.minutes(1))) + func routeChangeMidStream() async throws { + let route = AudioOutputRoute(outputChannels: 8) + // A short frame queue keeps the decoder from finishing the file + // before the route changes. + let frames = try await decode(fixture: "surround71.mkv", route: route, frameQueue: 4) { count in + if count == 40 { route.update(outputChannels: 2) } + } + let widths = frames.map(\.channels) + let switchIndex = try #require(widths.firstIndex(of: 2), "the layout must follow the route") + #expect(switchIndex >= 40, "frames decoded for the old route are not rewritten") + #expect(widths[.. AudioLossDetector.Observation { + AudioLossDetector.Observation( + hasAudioLane: true, audioEnded: false, audioRendererFailed: false, + clockRunning: true, playhead: playhead, + audioAhead: audioAhead, videoAhead: videoAhead, serial: serial + ) + } + + /// Feeds `observation` with the playhead advanced in 0.2 s ticks over + /// `seconds`; returns every reason the detector reported. + private func run( + _ detector: inout AudioLossDetector, from start: Double, seconds: Double, + _ observation: (Double) -> AudioLossDetector.Observation + ) -> [AudioLossReason] { + var reported: [AudioLossReason] = [] + var playhead = start + while playhead <= start + seconds + 1e-9 { + if let reason = detector.observe(observation(playhead)) { reported.append(reason) } + playhead += 0.2 + } + return reported + } + + @Test("no lane is absent; a lane starts healthy") + func absentAndHealthy() { + var detector = AudioLossDetector() + var observation = playing(at: 0) + observation.hasAudioLane = false + #expect(detector.observe(observation) == nil) + #expect(detector.health == .absent) + #expect(detector.observe(playing(at: 0)) == nil) + #expect(detector.health == .healthy) + } + + @Test("audio missing at a running playhead while video plays is reported once, after the threshold") + func starvationReportedOnce() { + var detector = AudioLossDetector() + let early = run(&detector, from: 0, seconds: 1.6) { playing(at: $0, audioAhead: false) } + #expect(early.isEmpty, "not before \(AudioLossDetector.threshold) s") + #expect(detector.health == .healthy) + let later = run(&detector, from: 1.8, seconds: 3) { playing(at: $0, audioAhead: false) } + #expect(later == [.starved]) + #expect(detector.health == .lost(.starved)) + } + + @Test("audio coming back clears the verdict, and a second episode is reported again") + func recovery() { + var detector = AudioLossDetector() + _ = run(&detector, from: 0, seconds: 3) { playing(at: $0, audioAhead: false) } + #expect(detector.health == .lost(.starved)) + #expect(detector.observe(playing(at: 3.2)) == nil) + #expect(detector.health == .healthy) + let again = run(&detector, from: 3.4, seconds: 3) { playing(at: $0, audioAhead: false) } + #expect(again == [.starved]) + } + + @Test("moments without audio that belong to playback are not a loss") + func legitimateSilence() { + // Video dry too: the whole pipeline is starving, which is buffering. + var detector = AudioLossDetector() + #expect(run(&detector, from: 0, seconds: 5) { playing(at: $0, audioAhead: false, videoAhead: false) }.isEmpty) + // Audio-only source: nothing to compare against. + #expect(run(&detector, from: 0, seconds: 5) { playing(at: $0, audioAhead: false, videoAhead: nil) }.isEmpty) + // The audio track ended (VOD whose audio is shorter than its video). + #expect(run(&detector, from: 0, seconds: 5) { + var observation = playing(at: $0, audioAhead: false) + observation.audioEnded = true + return observation + }.isEmpty) + // Paused or buffering: the clock does not run. + #expect(run(&detector, from: 0, seconds: 5) { + var observation = playing(at: $0, audioAhead: false) + observation.clockRunning = false + return observation + }.isEmpty) + #expect(detector.health == .healthy) + } + + @Test("a pause restarts the count; a seek voids the episode") + func pauseAndSeek() { + var detector = AudioLossDetector() + _ = run(&detector, from: 0, seconds: 1.6) { playing(at: $0, audioAhead: false) } + var paused = playing(at: 1.6, audioAhead: false) + paused.clockRunning = false + #expect(detector.observe(paused) == nil) + #expect(run(&detector, from: 1.6, seconds: 1.6) { playing(at: $0, audioAhead: false) }.isEmpty) + + _ = run(&detector, from: 3.4, seconds: 3) { playing(at: $0, audioAhead: false) } + #expect(detector.health == .lost(.starved)) + // Seek back: new serial, playhead earlier — a fresh start. + #expect(detector.observe(playing(at: 0.5, audioAhead: false, serial: 1)) == nil) + #expect(detector.health == .healthy) + } + + @Test("decoder and renderer failures are permanent until the lane is rebuilt") + func permanentLosses() { + var detector = AudioLossDetector() + _ = detector.observe(playing(at: 0)) + detector.decoderFailed() + #expect(detector.health == .lost(.decoderFailed)) + // Audio enqueued before the failure still plays out; the verdict stands. + #expect(detector.observe(playing(at: 0.2)) == nil) + #expect(detector.health == .lost(.decoderFailed)) + detector.laneReplaced() + #expect(detector.health == .healthy) + + var failed = playing(at: 1) + failed.audioRendererFailed = true + #expect(detector.observe(failed) == nil) + #expect(detector.health == .lost(.rendererFailed)) + #expect(detector.observe(playing(at: 1.2, serial: 4)) == nil) + #expect(detector.health == .lost(.rendererFailed)) + } +} + +@Suite("AudioRouteSession", .serialized) +struct AudioRouteSessionTests { + private func eventually( + timeout: TimeInterval = 10, + _ condition: () async -> Bool + ) async -> Bool { + let deadline = Date(timeIntervalSinceNow: timeout) + while Date() < deadline { + if await condition() { return true } + try? await Task.sleep(for: .milliseconds(100)) + } + return false + } + + private func makeSession(route: AudioOutputRoute, enableAudio: Bool = true) -> PlayerSession { + var configuration = PlayerConfiguration() + configuration.muted = true + configuration.bufferTarget = 0.5 + configuration.enableAudio = enableAudio + configuration.audioOutputRoute = route + return PlayerSession(configuration: configuration) + } + + @Test("a playing session refits its audio when the route narrows and widens", .timeLimit(.minutes(1))) + func followsRouteChanges() async throws { + let route = AudioOutputRoute(outputChannels: 8) + let session = makeSession(route: route) + _ = try await session.open(url: try Fixtures.path("surround51.ts")) + await session.play() + #expect(await eventually { await session.position > 1 }) + #expect(await session.audioOutput == AudioOutputLayout(sourceChannels: 6, outputChannels: 6, routeChannels: 8)) + + // HDMI receiver → AirPlay: the next decoded frame folds down to stereo. + route.update(outputChannels: 2) + let narrowed = await eventually { await session.audioOutput?.outputChannels == 2 } + #expect(narrowed, "audio must refit to the stereo route") + let before = await session.position + let continued = await eventually { await session.position > before + 1.5 } + #expect(continued, "playback must go on through the route change") + #expect(!session.renderer.audioRendererFailed) + #expect(await session.state == .playing) + #expect(await session.audioHealth == .healthy) + + // And back to surround. + route.update(outputChannels: 6) + #expect(await eventually { await session.audioOutput?.outputChannels == 6 }) + let resumed = await session.position + #expect(await eventually { await session.position > resumed + 1 }) + #expect(!session.renderer.audioRendererFailed) + + let diagnostics = await session.diagnostics + #expect(diagnostics.audioOutput?.routeChannels == 6) + #expect(diagnostics.audioRoute.changes == 2) + #expect(diagnostics.description.contains("aout=6>6ch@6")) + await session.shutdown() + } + + @Test("audio that stops reaching the renderer while video plays is reported", .timeLimit(.minutes(1))) + func reportsLostAudio() async throws { + let session = makeSession(route: AudioOutputRoute(outputChannels: 2)) + let errors = Mutex<[EngineError.Code]>([]) + let listener = Task { [events = session.events] in + for await event in events { + if case .error(let error) = event { errors.withLock { $0.append(error.code) } } + } + } + _ = try await session.open(url: try Fixtures.path("surround51.ts")) + await session.play() + #expect(await eventually { await session.position > 1 }) + #expect(await session.audioHealth == .healthy) + + // The renderer stops taking audio; the clock and the picture go on. + session.renderer.audioRenderer.stopRequestingMediaData() + let lost = await eventually(timeout: 15) { await session.audioHealth == .lost(.starved) } + let health = await session.audioHealth + #expect(lost, "lost audio must be detected, health is \(health)") + // The event travels through the stream after the verdict is set. + let reported = await eventually(timeout: 2) { errors.withLock { $0.contains(.audioLost) } } + let codes = errors.withLock { $0 } + #expect(reported && codes.filter { $0 == .audioLost }.count == 1, "errors: \(codes)") + #expect(await session.state == .playing, "video keeps playing") + #expect(await session.diagnostics.description.contains("audio=lost(starved)")) + + await session.shutdown() + listener.cancel() + } + + @Test("a session without audio reports none", .timeLimit(.minutes(1))) + func noAudioLane() async throws { + let session = makeSession(route: AudioOutputRoute(outputChannels: 2), enableAudio: false) + _ = try await session.open(url: try Fixtures.path("basic.mp4")) + await session.play() + #expect(await eventually { await session.position > 0.5 }) + #expect(await session.audioHealth == .absent) + #expect(await session.audioOutput == nil) + await session.shutdown() + } +} + +private extension Array { + subscript(safe index: Int) -> Element? { + indices.contains(index) ? self[index] : nil + } +} + +/// FFmpeg's mask for a named layout — the reference the speaker-set +/// constants are checked against. +private func layoutMask(_ name: String) -> UInt64 { + var layout = AVChannelLayout() + defer { av_channel_layout_uninit(&layout) } + guard av_channel_layout_from_string(&layout, name) >= 0 else { return 0 } + return layout.u.mask +} diff --git a/Tests/LumeEngineCoreTests/DecoderTests.swift b/Tests/LumeEngineCoreTests/DecoderTests.swift index 33b5cb2..a69e7e1 100644 --- a/Tests/LumeEngineCoreTests/DecoderTests.swift +++ b/Tests/LumeEngineCoreTests/DecoderTests.swift @@ -333,7 +333,8 @@ struct DecoderTests { let collected = await collector.value let rate = try #require(collected.rates.first) #expect(collected.rates.count == 1) - #expect(collected.channels == [1], "sine fixture is mono") + // The sine fixture is mono; on a stereo route it goes out on both sides. + #expect(collected.channels == [2], "mono source on a stereo route") let seconds = Double(collected.totalSamples) / Double(rate) #expect(abs(seconds - 10.0) < 0.3, "expected ~10 s of audio, got \(seconds)") let ptsValues = collected.ptsValues diff --git a/Tests/LumeEngineCoreTests/DisplayCriteriaTests.swift b/Tests/LumeEngineCoreTests/DisplayCriteriaTests.swift new file mode 100644 index 0000000..d9db566 --- /dev/null +++ b/Tests/LumeEngineCoreTests/DisplayCriteriaTests.swift @@ -0,0 +1,408 @@ +import CoreMedia +import CoreVideo +import Foundation +import Testing +@testable import LumeEngineCore + +/// The display mode asked of the TV is what the engine outputs, not what the +/// container says: deinterlacing doubles the rate, Dolby Vision with an IPT +/// base layer leaves as HDR10, any other Dolby Vision stream as its base +/// layer. Getting it wrong is visible on every frame — a 50p football match +/// judders at 25 Hz; a base layer shown in the wrong range is mis-toned. +@Suite("ContentDisplayCriteria") +struct ContentDisplayCriteriaTests { + private typealias Criteria = ContentDisplayCriteria + + private func resolve( + _ stream: Criteria.Stream, + deinterlacing: VideoDecoder.Deinterlacing = VideoDecoder.Deinterlacing(), + output: Criteria.Output? = nil + ) -> Criteria { + Criteria.resolve(stream: stream, deinterlacing: deinterlacing, output: output) + } + + private func broadcast(fps: Double = 25, interlaced: Bool? = true) -> Criteria.Stream { + Criteria.Stream(codecName: "h264", width: 1920, height: 1080, frameRate: fps, isInterlaced: interlaced) + } + + /// Transfer 2 (unspecified) by default: DV encodes often leave the VUI + /// empty, so the configuration record has to decide. + private func dolbyVision(profile: UInt8, compatibility: UInt8, transfer: UInt32 = 2) -> Criteria.Stream { + Criteria.Stream( + codecName: "hevc", width: 3840, height: 2160, frameRate: 24000.0 / 1001, + isInterlaced: false, colorTransfer: transfer, + dolbyVision: DolbyVisionConfiguration(profile: profile, level: 6, baseLayerCompatibilityID: compatibility) + ) + } + + // MARK: Refresh rate + + @Test("1080i50 deinterlaced at field rate asks for 50 Hz, before the first frame") + func interlacedFieldRateDoubles() { + let criteria = resolve(broadcast()) + #expect(criteria.refreshRate == 50) + #expect(criteria.basis == .stream) + } + + @Test("frame-rate deinterlacing, or none, keeps the frame rate") + func interlacedFrameRateKeeps() { + #expect(resolve(broadcast(), deinterlacing: .init(mode: .auto, rate: .frame)).refreshRate == 25) + #expect(resolve(broadcast(), deinterlacing: .off).refreshRate == 25) + } + + @Test("a progressive or unflagged stream is not predicted to deinterlace; .always is") + func deinterlacePrediction() { + #expect(resolve(broadcast(interlaced: false)).refreshRate == 25) + #expect(resolve(broadcast(interlaced: nil)).refreshRate == 25) + #expect(resolve(broadcast(interlaced: false), deinterlacing: .init(mode: .always)).refreshRate == 50) + } + + @Test("NTSC and film rates snap to their exact values, apart from their integer neighbours") + func ntscRates() { + let cases: [(Double, Double)] = [ + (23.976, 24000.0 / 1001), (24, 24), (25, 25), + (29.97, 30000.0 / 1001), (30, 30), (50, 50), + (59.94, 60000.0 / 1001), (60, 60), + // Timestamp jitter around a grid rate. + (25.04, 25), (59.95, 60000.0 / 1001), (23.98, 24000.0 / 1001), (23.995, 24), + ] + for (fps, expected) in cases { + let rate = resolve(broadcast(fps: fps, interlaced: false)).refreshRate + #expect(rate == expected, "\(fps) fps must ask for \(expected), got \(String(describing: rate))") + } + // 29.97i deinterlaced at field rate is 59.94p. + #expect(resolve(broadcast(fps: 30000.0 / 1001)).refreshRate == 60000.0 / 1001) + } + + @Test("unknown or off-grid rates ask for no rate at all") + func unknownRates() { + for fps in [0, -1, .nan, .infinity, 12.5, 15, 48, 25.5, 100, 120] { + let rate = resolve(broadcast(fps: fps, interlaced: false)).refreshRate + #expect(rate == nil, "\(fps) fps must not ask for a rate, got \(String(describing: rate))") + } + } + + @Test("the measured cadence of the output beats the container's guess") + func outputCadenceWins() { + // Field-coded H.264 can report its field rate; what comes out decides. + let reportsFieldRate = broadcast(fps: 50, interlaced: true) + let output = Criteria.Output(width: 1920, height: 1080, frameInterval: 20_000, isDeinterlacing: true) + let criteria = resolve(reportsFieldRate, output: output) + #expect(criteria.refreshRate == 50) + #expect(criteria.basis == .output) + + // Interlaced in the container, progressive frames out: 25. + let progressive = Criteria.Output(width: 1920, height: 1080, frameInterval: 40_000, isDeinterlacing: false) + #expect(resolve(broadcast(), output: progressive).refreshRate == 25) + } + + @Test("frames without a cadence yet: the decoder's deinterlacing state replaces the prediction") + func outputWithoutCadence() { + let flaggedLate = Criteria.Output(width: 1920, height: 1080, isDeinterlacing: true) + let criteria = resolve(broadcast(interlaced: nil), output: flaggedLate) + #expect(criteria.refreshRate == 50) + #expect(criteria.basis == .stream, "a rate not yet measured is still a prediction") + } + + // MARK: Dynamic range + + @Test("SDR, HDR10 and HLG from the stream's transfer characteristics") + func streamRanges() { + var stream = broadcast(interlaced: false) + #expect(resolve(stream).dynamicRange == .sdr) + stream.colorTransfer = 16 + #expect(resolve(stream).dynamicRange == .hdr10) + stream.colorTransfer = 18 + #expect(resolve(stream).dynamicRange == .hlg) + } + + @Test("the frames' transfer function beats the stream's") + func outputRange() { + let untagged = broadcast(interlaced: false) + let pq = Criteria.Output( + width: 3840, height: 2160, + transferFunction: kCVImageBufferTransferFunction_SMPTE_ST_2084_PQ as String + ) + #expect(resolve(untagged, output: pq).dynamicRange == .hdr10) + let hlg = Criteria.Output( + width: 3840, height: 2160, + transferFunction: kCVImageBufferTransferFunction_ITU_R_2100_HLG as String + ) + #expect(resolve(untagged, output: hlg).dynamicRange == .hlg) + + var taggedPQ = untagged + taggedPQ.colorTransfer = 16 + let sdrFrames = Criteria.Output( + width: 1920, height: 1080, + transferFunction: kCVImageBufferTransferFunction_ITU_R_709_2 as String + ) + #expect(resolve(taggedPQ, output: sdrFrames).dynamicRange == .sdr) + } + + @Test("Dolby Vision profile 5, 10.0 and 20 are converted, so they ask for HDR10") + func iptProfilesAskForHDR10() { + for profile: UInt8 in [5, 10, 20] { + let criteria = resolve(dolbyVision(profile: profile, compatibility: 0)) + #expect(criteria.dynamicRange == .hdr10, "profile \(profile)") + #expect(criteria.refreshRate == 24000.0 / 1001) + } + } + + @Test("other Dolby Vision profiles ask for the range of the base layer that is presented") + func compatibleProfilesAskForBaseLayer() { + let cases: [(profile: UInt8, compatibility: UInt8, range: Criteria.DynamicRange)] = [ + (8, 1, .hdr10), (10, 1, .hdr10), (7, 6, .hdr10), (8, 4, .hlg), (8, 2, .sdr), (9, 2, .sdr), + ] + for (profile, compatibility, range) in cases { + let criteria = resolve(dolbyVision(profile: profile, compatibility: compatibility)) + #expect(criteria.dynamicRange == range, "\(profile).\(compatibility) must ask for \(range)") + } + } + + @Test("Dolby Vision is never asked for: there is no Dolby Vision output path") + func neverDolbyVision() { + for profile: UInt8 in [4, 5, 7, 8, 9, 10, 20] { + for compatibility: UInt8 in [0, 1, 2, 4, 6, 9] { + let criteria = resolve(dolbyVision(profile: profile, compatibility: compatibility)) + #expect(criteria.dynamicRange != .dolbyVision, "\(profile).\(compatibility)") + } + } + } + + @Test("once frames exist, their transfer function cross-checks the compatibility ID") + func baseLayerFramesDecide() { + let pq = Criteria.Output( + width: 3840, height: 2160, + transferFunction: kCVImageBufferTransferFunction_SMPTE_ST_2084_PQ as String + ) + let hlg = Criteria.Output( + width: 3840, height: 2160, + transferFunction: kCVImageBufferTransferFunction_ITU_R_2100_HLG as String + ) + #expect(resolve(dolbyVision(profile: 8, compatibility: 1), output: pq).dynamicRange == .hdr10) + #expect(resolve(dolbyVision(profile: 8, compatibility: 4), output: hlg).dynamicRange == .hlg) + // A record that disagrees with what is actually presented loses. + #expect(resolve(dolbyVision(profile: 8, compatibility: 1), output: hlg).dynamicRange == .hlg) + // An unknown compatibility ID falls back to the stream's transfer. + #expect(resolve(dolbyVision(profile: 8, compatibility: 9, transfer: 18)).dynamicRange == .hlg) + // The converted IPT profiles are HDR10 whatever their frames were tagged. + let untagged = Criteria.Output(width: 3840, height: 2160) + #expect(resolve(dolbyVision(profile: 5, compatibility: 0), output: untagged).dynamicRange == .hdr10) + } + + // MARK: Format description + + @Test("HDR10 converted from profile 5 is described as plain HEVC HDR10") + func convertedFormatDescription() throws { + let format = try #require(resolve(dolbyVision(profile: 5, compatibility: 0)).makeFormatDescription()) + #expect(CMFormatDescriptionGetMediaSubType(format) == kCMVideoCodecType_HEVC) + #expect(extensionString(format, kCMFormatDescriptionExtension_TransferFunction) + == kCVImageBufferTransferFunction_SMPTE_ST_2084_PQ as String) + #expect(extensionString(format, kCMFormatDescriptionExtension_ColorPrimaries) + == kCVImageBufferColorPrimaries_ITU_R_2020 as String) + #expect(CMFormatDescriptionGetExtension(format, extensionKey: kCMFormatDescriptionExtension_SampleDescriptionExtensionAtoms) == nil) + let dimensions = CMVideoFormatDescriptionGetDimensions(format) + #expect(dimensions.width == 3840 && dimensions.height == 2160) + } + + @Test("profile 8.1 is described as the HDR10 HEVC it is presented as, with no Dolby Vision signalling") + func baseLayerFormatDescription() throws { + let format = try #require(resolve(dolbyVision(profile: 8, compatibility: 1)).makeFormatDescription()) + #expect(CMFormatDescriptionGetMediaSubType(format) == kCMVideoCodecType_HEVC) + #expect(extensionString(format, kCMFormatDescriptionExtension_TransferFunction) + == kCVImageBufferTransferFunction_SMPTE_ST_2084_PQ as String) + #expect(CMFormatDescriptionGetExtension(format, extensionKey: kCMFormatDescriptionExtension_SampleDescriptionExtensionAtoms) == nil) + } + + @Test("SDR before the first frame is described as BT.709") + func sdrFormatDescription() throws { + let format = try #require(resolve(broadcast()).makeFormatDescription()) + #expect(CMFormatDescriptionGetMediaSubType(format) == kCMVideoCodecType_H264) + #expect(extensionString(format, kCMFormatDescriptionExtension_TransferFunction) + == kCVImageBufferTransferFunction_ITU_R_709_2 as String) + } + + @Test("asking twice for the same mode is recognised, whatever the basis") + func sameMode() { + let predicted = resolve(broadcast()) + let confirmed = resolve(broadcast(), output: .init(width: 1920, height: 1080, frameInterval: 20_000)) + #expect(predicted.asksForSameMode(as: confirmed)) + #expect(!predicted.asksForSameMode(as: resolve(broadcast(), deinterlacing: .off))) + #expect(confirmed.description == "50Hz/sdr(output)") + } + + private func extensionString(_ format: CMFormatDescription, _ key: CFString) -> String? { + CMFormatDescriptionGetExtension(format, extensionKey: key) as? String + } +} + +@Suite("FrameCadence") +struct FrameCadenceTests { + @Test("needs a handful of intervals before it reports one") + func needsSamples() { + var cadence = FrameCadence() + for frame in 0.. PlayerSession { + var configuration = PlayerConfiguration() + configuration.muted = true + configuration.bufferTarget = 0.5 + configuration.deinterlace = deinterlacing + return PlayerSession(configuration: configuration) + } + + /// Polls until the session's criteria are confirmed by its output. + private func confirmedCriteria(_ session: PlayerSession, timeout: TimeInterval = 10) async -> ContentDisplayCriteria? { + let deadline = Date(timeIntervalSinceNow: timeout) + while Date() < deadline { + if let criteria = await session.displayCriteria, criteria.basis == .output { return criteria } + try? await Task.sleep(for: .milliseconds(50)) + } + return nil + } + + @Test("the interlaced fixture asks for 50 Hz SDR, predicted at open and confirmed by its frames", .timeLimit(.minutes(1))) + func interlacedFieldRate() async throws { + let session = session() + let info = try await session.open(url: try Fixtures.path("interlaced.ts")) + #expect(info.videoTracks.first?.video?.isInterlaced == true) + let predicted = await session.displayCriteria + #expect(predicted?.refreshRate == 50, "predicted \(String(describing: predicted))") + + await session.play() + let confirmed = await confirmedCriteria(session) + #expect(confirmed?.refreshRate == 50, "confirmed \(String(describing: confirmed))") + #expect(confirmed?.dynamicRange == .sdr) + #expect(await session.diagnostics.description.contains("display=50Hz/sdr(output)")) + await session.shutdown() + } + + @Test("deinterlacing at frame rate keeps the fixture at 25 Hz", .timeLimit(.minutes(1))) + func interlacedFrameRate() async throws { + let session = session(deinterlacing: .init(mode: .auto, rate: .frame)) + _ = try await session.open(url: try Fixtures.path("interlaced.ts")) + await session.play() + let confirmed = await confirmedCriteria(session) + #expect(confirmed?.refreshRate == 25, "confirmed \(String(describing: confirmed))") + await session.shutdown() + } + + @Test("a progressive 30 fps file asks for 30 Hz", .timeLimit(.minutes(1))) + func progressive() async throws { + let session = session() + let info = try await session.open(url: try Fixtures.path("basic.mp4")) + #expect(info.videoTracks.first?.video?.isInterlaced != true) + await session.play() + let confirmed = await confirmedCriteria(session) + #expect(confirmed?.refreshRate == 30, "confirmed \(String(describing: confirmed))") + await session.shutdown() + } + + @Test("an audio-only source has no display criteria", .timeLimit(.minutes(1))) + func audioOnly() async throws { + let session = session() + _ = try await session.open(url: try Fixtures.path("surround71.mkv")) + #expect(await session.displayCriteria == nil) + await session.shutdown() + } +} + +/// The start hold a host uses to sit out a display mode switch: the clock +/// must not start into a blank TV, and must start by itself once the bound +/// passes, whatever the host does. +@Suite("Start hold", .serialized) +struct StartHoldTests { + private func makeSession(stallThreshold: Double = 8) -> PlayerSession { + var configuration = PlayerConfiguration() + configuration.muted = true + configuration.bufferTarget = 0.5 + configuration.stallThreshold = stallThreshold + return PlayerSession(configuration: configuration) + } + + private func eventually(timeout: TimeInterval = 10, _ condition: () async -> Bool) async -> Bool { + let deadline = Date(timeIntervalSinceNow: timeout) + while Date() < deadline { + if await condition() { return true } + try? await Task.sleep(for: .milliseconds(50)) + } + return false + } + + @Test("a held session buffers and waits; releasing starts it", .timeLimit(.minutes(1))) + func holdThenRelease() async throws { + // A short stall threshold: waiting while held must not read as a stall. + let session = makeSession(stallThreshold: 1) + var events = session.events.makeAsyncIterator() + _ = try await session.open(url: try Fixtures.path("basic.mp4")) + await session.holdStart(for: 20) + await session.play() + + try await Task.sleep(for: .seconds(2)) + #expect(await session.state == .buffering) + #expect(await session.isStartHeld) + #expect(await session.position == 0) + + await session.releaseStartHold() + #expect(await eventually { await session.state == .playing }) + await session.shutdown() + + var stalled = false + while let event = await events.next() { + if case .stalled = event { stalled = true } + } + #expect(!stalled, "a held start is not a stall") + } + + @Test("the hold runs out by itself", .timeLimit(.minutes(1))) + func holdExpires() async throws { + let session = makeSession() + _ = try await session.open(url: try Fixtures.path("basic.mp4")) + await session.holdStart(for: 1) + await session.play() + #expect(await session.state != .playing) + #expect(await eventually { await session.state == .playing }) + #expect(await !session.isStartHeld) + await session.shutdown() + } +} diff --git a/Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh b/Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh index befe268..d5105e6 100644 --- a/Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh +++ b/Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh @@ -18,6 +18,7 @@ # replay.ts — 8 s TS whose bytes jump back ~3 s at ~6 s, as a reconnect replays them # ac4.ts — TS whose first audio (ger) is declared AC-4, which no decoder here plays, then eng AAC # hdr10.mp4 — HEVC Main 10, BT.2020 PQ, mastering display + content light level SEI, HDR10+ per frame +# surround51.ts — 20 s H.264 + AC-3 5.1 MPEG-TS, tone in the centre channel only set -euo pipefail FFMPEG="${FFMPEG:-ffmpeg}" @@ -329,4 +330,15 @@ JSON echo "generated: $OUT/hdr10.mp4" fi +# Broadcast surround the way IPTV carries it: H.264 + AC-3 5.1 in MPEG-TS, +# long enough to change the output route mid-playback. The tone sits in the +# centre channel alone, so a fold-down that drops channels instead of mixing +# them goes silent. +gen surround51.ts \ + -f lavfi -i "testsrc2=duration=20:size=320x180:rate=25" \ + -f lavfi -i "sine=frequency=440:duration=20" \ + -filter_complex "[1:a]pan=5.1(side)|FC=c0[a]" -map 0:v -map "[a]" \ + -c:v libx264 -preset ultrafast -pix_fmt yuv420p -g 25 \ + -c:a ac3 -b:a 384k -shortest -f mpegts + echo "fixtures ready in $OUT"