diff --git a/AGENTS.md b/AGENTS.md index 17725e7..ccece3b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -51,7 +51,7 @@ Prerequisites: build/scripts/build-ffmpeg.sh macos-arm64 # one slice is enough for local dev build/scripts/make-xcframework.sh ``` -- Tests need a host `ffmpeg` CLI (Homebrew) — fixtures are generated lazily on first test run by `Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh` into `TestStreams/generated/` (gitignored, idempotent). `Fixtures.swift` finds Homebrew ffmpeg automatically; override with the `FFMPEG` env var. +- Tests need a host `ffmpeg` CLI (Homebrew) — fixtures are generated lazily on first test run by `Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh` into `TestStreams/generated/` (gitignored, idempotent). `Fixtures.swift` finds Homebrew ffmpeg automatically; override with the `FFMPEG` env var. The closed-caption fixture (`captions.ts`) also needs `python3` (the one Xcode's command-line tools provide is enough), because no ffmpeg CLI option writes captions into the video; without it that fixture is skipped and its tests fail as missing. CI (`.github/workflows/ci.yml`) builds the macOS FFmpeg slice (cached on `build/versions.json` + scripts + patches hashes) and runs `swift test` with a strict one-thread cooperative pool (see above), checks the Markdown docs offline (`.github/scripts/check-docs.py`: relative links and in-repo anchors resolve, `CLAUDE.md` is still the `AGENTS.md` symlink — run it after editing docs), then cross-compiles the library for iOS and tvOS via `xcodebuild` (compile-only, one cached slice each — visionOS is left to local verification because that matrix entry was never confirmed on a runner — the xrOS SDK ships with Xcode, but a visionOS *destination* also needs Xcode's visionOS platform component); tagged releases build all 10 platform slices and attach the xcframework. @@ -85,6 +85,11 @@ This split is the core invariant of the codebase: - **HDR metadata is written on every delivered buffer, from the frame.** `Decode/HDRMetadata` turns the frame's mastering display (ST 2086) and content light level side data into Apple's big-endian SEI payloads on the pixel buffer in `VideoDecoder.deliver`, and HDR10+ (ST 2094-40) into a T.35 message the `SampleBufferBuilder` attaches per sample (`kCMSampleAttachmentKey_HDR10PlusPerFrameData`). No path keeps it by itself: VideoToolbox, as FFmpeg drives it, leaves its surfaces without either key, and the engine's own buffers carry only what it writes. Read the frame, not the codec parameters — FFmpeg already folds the container's boxes into every frame and lets the bitstream's SEI override them. Keys are written *or removed* on every buffer (pools recycle), and the format description is rebuilt whenever they change, because `CMVideoFormatDescriptionMatchesImageBuffer` compares them; don't replace that match with a cheaper key. - **Multi-layer HEVC reaches the codec as its base layer only.** `Decode/HEVCBaseLayerFilter` drops every `nuh_layer_id > 0` NAL unit from packets and extradata before `avcodec_send_packet`/`avcodec_open2`. FFmpeg files an MV-HEVC stream's second-view SPS/PPS (DV profile 20, spatial video) alongside the base layer's, its VideoToolbox glue packs all of them into one `hvcC`, VideoToolbox rejects that session, and FFmpeg falls back to software *without an error* — 4K HEVC in software, which stutters on Apple TV. The engine presents only the base view, so stripping loses nothing. Relatedly, `VideoDecoder.isHardwareActive` is judged from the frames' pixel format, not from whether a VT device was attached, and such a silent fallback emits `.downgradedToSoftware` — keep it that way, or this class of bug goes invisible again. - **Dolby Vision with an IPT base layer is converted, not tagged.** Profiles 5, 10.0 and 20 (compatibility ID 0) carry IPTPQc2, which shows pink/green as YCbCr. No colour attachment fixes that — the RPU reshaping changes per scene and Apple's decoders don't apply it to decoded frames — so `Decode/DolbyVisionConverter` turns each frame into HDR10 on the GPU in `VideoDecoder.deliver`, the one point every frame (hardware, software, deinterlaced) passes with its side data. Whether a frame is IPT is decided *per frame* by `Decode/DolbyVisionDetection`, never by the configuration record alone — remuxes mislabel HDR10 as P5 and broadcasts drop RPUs without changing it, and converting HDR10 as IPT gives wrong colours. Order: an `AVCOL_SPC_IPT_C2` frame is IPT; a frame with its *own* RPU is what the RPU's `signal_color_space` says (2 = IPT); a frame without one that states a matrix (BT.2020 PQ) is that matrix; only an untagged frame without its own RPU falls back to the record (genuine P5 often leaves the VUI matrix unspecified). "Own" matters because FFmpeg's HEVC decoder attaches the last RPU's `AV_FRAME_DATA_DOVI_METADATA` to every later frame; only `AV_FRAME_DATA_DOVI_RPU_BUFFER` marks the frame that carried one (AV1 attaches metadata only to its own frame). HDR10-compatible profiles (8.1 etc.) must stay on the zero-copy path. GPU refusals are per-frame (iOS refuses a backgrounded app); only a converter that can't exist gives up for the session. +- **Display criteria describe the output, not the container.** `PlayerSession.displayCriteria` (pure resolution in `Media/ContentDisplayCriteria`) is what an Apple TV with Match Content should switch the TV to: the refresh rate *after* deinterlacing (field rate turns 1080i50 into 50 Hz) and the dynamic range of what is presented (IPT Dolby Vision profiles are converted, so HDR10; every other Dolby Vision profile is presented as its base layer, so HDR10, HLG or SDR by its compatibility ID and the frames' transfer function). Never ask for Dolby Vision mode while the engine has no Dolby Vision output path — that belongs to the PLAN.md §7 compressed-`dvh1` spike. Before the first frame it is predicted from the stream; once the renderer has been handed frames, their colour tags and measured cadence (`Render/FrameCadence`) decide. Rates snap to the film/broadcast grid or ask for none — never an odd rate. Applying it is tvOS-only and opt-in (`Media/DisplayCriteriaMatcher`, or `LumePlayerView(…, matchesDisplayCriteria: true)`): one matcher per player screen, kept across sessions so zapping between same-mode channels never switches the TV, reset only when the screen goes away. The mode-switch blank is sat out with `PlayerSession.holdStart(for:)`, which the session bounds itself and the stall watchdog ignores — keep both, or a host that never releases wedges playback, or a held start reads as a stall. +- **Dolby Vision with an IPT base layer is converted, not tagged.** Profiles 5, 10.0 and 20 (compatibility ID 0) carry IPTPQc2, which shows pink/green as YCbCr. No colour attachment fixes that — the RPU reshaping changes per scene and Apple's decoders don't apply it to decoded frames — so `Decode/DolbyVisionConverter` turns each frame into HDR10 on the GPU in `VideoDecoder.deliver`, the one point every frame (hardware, software, deinterlaced) passes with its side data. Detection is the stream's DV configuration record *or* an `AVCOL_SPC_IPT_C2` frame; many P5 encodes leave the VUI matrix unspecified, so don't reduce it to the tag. HDR10-compatible profiles (8.1 etc.) must stay on the zero-copy path. GPU refusals are per-frame (iOS refuses a backgrounded app); only a converter that can't exist gives up for the session. +- **Audio is fitted to the negotiated route width, never the route's capability.** `AudioDecoder` reads `AudioOutputRoute.outputChannels` (the session's `outputNumberOfChannels`, followed through route-change notifications) per frame and remixes per `AudioLayoutPolicy` with an explicit `DownmixMatrix` — a fold-down mixes centre and surrounds in, it never drops channels. Don't switch it to `maximumOutputNumberOfChannels`: a renderer fed more channels than the session outputs failed outright on tvOS. The engine only observes the audio session; setting its category, activation or preferred channel count is the host's job (Lume does it). When the route narrows, `SystemRenderer` keeps audio decoded for the wider route out of the renderer. +- **How long a subtitle shows is the timeline's decision, not the decoder's.** Decoders hand `Subtitle/SubtitleStore` a `SubtitleEvent` (start, end *or none*, text, whether the track replaces the whole screen with each update); `Subtitle/SubtitleTimeline` decides the rest: an open cue ends at the next update of its track (capped at 16 s), an empty update on a closed-caption/DVB/PGS/DVD track is a clear that ends what is showing, a seek ends open cues, re-read ranges are deduplicated, and the store is bounded. Don't turn `end_display_time == UINT32_MAX` into a number in a decoder or drop empty updates there — both put stuck or stacked cues on screen. +- **Closed captions leave the video lane through a tap, never a wait.** CEA-608 in A/53 side data (`AV_FRAME_DATA_A53_CC`) is lifted off each decoded frame in `VideoDecoder.receiveFrames` (before deinterlacing) by `Subtitle/ClosedCaptionTap`, stamped with the frame's PTS, and `trySend`-ed to an ordinary `SubtitleDecoder` running `ccaption` in real-time mode — a slow caption lane drops captions, never frames. The track is synthetic (`TrackInfo.embeddedClosedCaptionsIndex`) and appears in `PlayerSession.mediaInfo` once caption content is seen; `.opened` keeps the open-time track list, because no new `PlayerEvent` case may be added for it. Only CEA-608 field 1 (CC1/CC2) is decoded: FFmpeg has no CEA-708 (DTVCC) decoder, and teletext needs libzvbi, which the build does not include. - **Zero global mutable state:** all configuration is `PlayerConfiguration` passed at session creation. No `static var` knobs. - **No force unwraps / `as!` / `fatalError` in engine paths**; failures return typed `EngineError`. No `assertionFailure` in production channel/data paths — misuse degrades, never crashes. - **Never scrape FFmpeg log strings for control flow** (`av_log` is diagnostics only). Silence is never a failure mode: pipelines are supervised, stalls are detected from playhead ground truth, and failures surface as typed `PlayerEvent`s on the session's `AsyncStream`. diff --git a/PLAN.md b/PLAN.md index 6dc77a6..e8a5ccc 100644 --- a/PLAN.md +++ b/PLAN.md @@ -142,6 +142,7 @@ Brightness/contrast/saturation and MetalFX spatial upscaling operate CVPixelBuff **D6 — One subtitle engine, composited as overlay.** libass renders ASS/SSA (full effects, embedded fonts via attachment streams, word-by-word \k tags); PGS/DVB/VobSub decoded by FFmpeg to bitmaps; SRT/VTT parsed to styled cues rendered through the same layout box. Output = time-tagged `SubtitleImage` cues composited on a CALayer above the video surface (and into the PiP content when active; into Metal for HDR subtitle mode). Dual subtitle slots (main + secondary). External files attach as additional demux sessions. +**Cue lifecycle and closed captions (as built).** Every subtitle source — demuxed stream, sidecar file, or captions lifted from the video — reaches one time-indexed store as display updates; the store's `SubtitleTimeline` alone decides how long each shows (open cues end at the next update or a clear, capped; seeks end open cues; bounded size). CEA-608 captions carried as A/53 side data are tapped off decoded video frames without ever blocking the video lane and surface as a synthetic "CC1" subtitle track once seen, additively through `PlayerSession.mediaInfo` (no new event). **System caption respect:** map `MACaptionAppearance` (user's Style settings) onto text-cue styling by default. **D7 — Buffering is PTS-math, not heuristics.** @@ -190,6 +191,7 @@ Licensing note: engine code is **MIT** (`LICENSE`); FFmpeg is LGPL 2.1+, and the ## 6. Feature implementation notes (selected) - **Now Playing (explicit requirement):** `NowPlayingCenter` module in `LumeEngineMedia` — `MPNowPlayingInfoCenter` (title/artwork/elapsed/duration/rate, live-stream flag) + `MPRemoteCommandCenter` (play/pause/seek/skip/next-episode hook), audio-session lifecycle, route-change and interruption handling. Works on the lock screen/Control Center including while audio/video routes to an Apple TV. Note: full-screen *video* AirPlay of FFmpeg-decoded content is not possible for third parties (system limitation); Lume's existing switch-to-AVPlayer-on-AirPlay stays, and the integration phase adds a shared app-level Now Playing service so lock-screen metadata is present in **both** engines (closing today's gap). +- **Audio width follows the route:** decoded audio leaves the engine in the layout `AudioLayoutPolicy` picks for the *negotiated* route width (`AVAudioSession.outputNumberOfChannels`, followed through route changes by `AudioOutputRoute`; stereo on macOS): discrete multichannel when the route carries it (HDMI receiver), an explicit ITU-R BS.775 fold-down (`DownmixMatrix`: centre and surrounds at −3 dB, LFE omitted, not normalized) when it does not, mono on both speakers of a stereo route. A route change refits at the next decoded frame; audio decoded for a wider route never reaches a narrower one. The engine observes the audio session only — category, activation and the preferred channel count are the host's. `PlayerSession.audioHealth` reports audio that died while video plays on (decoder gave up, renderer failed, or nothing reaching the renderer at a running playhead). - **PiP:** `AVPictureInPictureController(contentSource: .init(sampleBufferDisplayLayer:playbackDelegate:))` from the SystemRenderer; subtitle cues composited into the sample buffers during PiP so subs survive. - **Trickplay:** a `ThumbnailSession` — separate lightweight demux+decode (keyframes only, HW decode, downscale to ~320 px) filling a time-indexed cache; API `thumbnails(at:interval:)`. Works for VOD/catch-up; live uses the rewind buffer. - **Live rewind (time-shift DVR):** disk-backed packet spool behind a custom AVIO layer; seeking within spool = local demux, seeking to live edge = passthrough. Configurable size/duration caps. @@ -197,6 +199,7 @@ Licensing note: engine code is **MIT** (`LICENSE`); FFmpeg is LGPL 2.1+, and the - **Precache:** same AVIO disk-cache layer, `precache(url:leadingSeconds:)` background task with LRU budget. - **Upscaling:** MetalFX spatial scaler stage (D5), quality/perf switch, auto-disable on thermal pressure (`ProcessInfo.thermalState`). - **HDR:** HDR10/HLG passthrough via sample-buffer attachments. DV with an HDR10-compatible base layer (P8.1, P10.1, …) plays as HDR10. DV with an IPTPQc2 base layer (P5, P10.0, P20 — compatibility ID 0) is converted to HDR10 before enqueue by a Metal compute pass (`Decode/DolbyVisionConverter`) that applies each frame's RPU reshaping (polynomial + MMR) and the IPT→LMS→BT.2020 matrices FFmpeg exports as `AV_FRAME_DATA_DOVI_METADATA`; the renderer then tone-maps it like any HDR10 stream. Dynamic DV display management (L1/L2/L8 trims) is not carried over. HDR10 static metadata (mastering display, MaxCLL/MaxFALL — from the bitstream's SEI or the container, falling back to a DV RPU's level 6 for the light level) is attached to every delivered pixel buffer, since neither VideoToolbox nor the engine's own buffers carry it otherwise, and so reaches the format description; HDR10+ dynamic metadata travels per sample as `kCMSampleAttachmentKey_HDR10PlusPerFrameData` (`Decode/HDRMetadata`). Whether the renderer acts on HDR10+ for decoded (not compressed) samples is still to be confirmed on an HDR10+ display. EDR headroom handling on macOS/visionOS Metal path. +- **Match Content (tvOS):** a sample-buffer pipeline gets no automatic display mode switching, so the engine derives `ContentDisplayCriteria` from its own output — refresh rate after deinterlacing (measured from enqueued frame PTS once there are frames, snapped to 23.976/24/25/29.97/30/50/59.94/60 or no preference), dynamic range of what is presented (IPT Dolby Vision profiles → HDR10 after conversion; other Dolby Vision profiles → their base layer's range by compatibility ID, cross-checked with the frames' transfer function; otherwise the frames' or the stream's transfer function) — and `DisplayCriteriaMatcher` sets it as the window's `AVDisplayManager.preferredDisplayCriteria` via `AVDisplayCriteria(refreshRate:formatDescription:)` (tvOS 17+). Dolby Vision mode is never requested: the engine presents the base layer, and only a real DV output path (§7 spike) should ask for it. The start is held, bounded, until the TV finishes switching. Opt-in; mid-stream changes are not chased (each switch blanks the TV). - **360°/VR:** MetalRenderer projection (equirect sphere / fisheye), CoreMotion + gesture camera; visionOS immersive presentation later. - **Blu-ray ISO/DVD:** libbluray/libdvdnav behind the `AVIOProtocol` seam; menu-less title playback first (longest-title heuristic), chapters mapped to the chapter API. - **AI subtitles:** whisper.cpp (Metal) tapping the decoded-audio channel; streaming word-level segments feed the subtitle model as a synthetic track; on-device translation via Translation.framework. diff --git a/Sources/LumeEngine/LumePlayer.swift b/Sources/LumeEngine/LumePlayer.swift index a70a3c3..d88ebb1 100644 --- a/Sources/LumeEngine/LumePlayer.swift +++ b/Sources/LumeEngine/LumePlayer.swift @@ -51,6 +51,24 @@ public final class LumePlayer { /// layer tree directly (UIKit/AppKit). public var displayLayer: LumeDisplayLayer? { session?.renderer.displayLayer } + /// tvOS: switch the TV to each loaded stream's refresh rate and dynamic + /// range (Match Content) while it plays in a `LumePlayerView`, which + /// supplies the window; `stop()` and the view going away hand the display + /// back. Set before `load`. Off by default; ignored on other platforms. + public var matchesDisplayCriteria = false { + didSet { + #if os(tvOS) + if !matchesDisplayCriteria { displayMatcher.reset() } + #endif + } + } + + #if os(tvOS) + /// One per player, so loading the next stream does not hand the display + /// back in between (see `DisplayCriteriaMatcher`). + let displayMatcher = DisplayCriteriaMatcher() + #endif + private var session: PlayerSession? private var eventTask: Task? private var tickTask: Task? @@ -86,6 +104,10 @@ public final class LumePlayer { do { let info = try await session.open(url: url) + #if os(tvOS) + // Before the app's play(): the start is held while the TV switches. + if matchesDisplayCriteria { await displayMatcher.match(session) } + #endif mediaInfo = info duration = info.duration.map(MediaTime.seconds) return info @@ -135,6 +157,9 @@ public final class LumePlayer { } public func stop() async { + #if os(tvOS) + displayMatcher.reset() + #endif await teardownSession() state = .idle position = 0 @@ -176,6 +201,11 @@ public final class LumePlayer { private func tick() async { guard let session else { return } position = await session.position + // Tracks can turn up after open (closed captions in the video). + if let mediaInfo, let current = await session.mediaInfo, + current.tracks.count != mediaInfo.tracks.count { + self.mediaInfo = current + } let now = session.renderer.currentTime if MediaTime.isValid(now) { diff --git a/Sources/LumeEngine/LumePlayerView.swift b/Sources/LumeEngine/LumePlayerView.swift index 74a00e3..04e4956 100644 --- a/Sources/LumeEngine/LumePlayerView.swift +++ b/Sources/LumeEngine/LumePlayerView.swift @@ -18,12 +18,29 @@ public struct LumePlayerView: View { } public var body: some View { - if let layer = player.displayLayer { - LayerHostView(layer: layer, gravity: gravity) + ZStack { + if let layer = player.displayLayer { + #if os(tvOS) + LayerHostView( + layer: layer, + gravity: gravity, + displayMatcher: player.matchesDisplayCriteria ? player.displayMatcher : nil + ) .accessibilityLabel("Video") - } else { - Color.black + #else + LayerHostView(layer: layer, gravity: gravity) + .accessibilityLabel("Video") + #endif + } else { + Color.black + } } + #if os(tvOS) + // The whole view, not the layer host: that one is swapped out while + // the player moves to the next stream, which must not hand the + // display back in between. + .onDisappear { [player] in player.displayMatcher.reset() } + #endif } } @@ -33,21 +50,49 @@ import UIKit private struct LayerHostView: UIViewRepresentable { let layer: LumeDisplayLayer let gravity: AVLayerVideoGravity + #if os(tvOS) + /// Given this view's window, so it knows which display to match. + var displayMatcher: DisplayCriteriaMatcher? + #endif func makeUIView(context: Context) -> LayerHostUIView { let view = LayerHostUIView() view.install(layer: layer, gravity: gravity) + #if os(tvOS) + view.displayMatcher = displayMatcher + #endif return view } func updateUIView(_ view: LayerHostUIView, context: Context) { view.install(layer: layer, gravity: gravity) + #if os(tvOS) + view.displayMatcher = displayMatcher + #endif } } final class LayerHostUIView: UIView { private weak var hosted: LumeDisplayLayer? + #if os(tvOS) + weak var displayMatcher: DisplayCriteriaMatcher? { + didSet { lendWindow() } + } + + override func didMoveToWindow() { + super.didMoveToWindow() + lendWindow() + } + + /// Only ever sets a window: leaving one is not the end of playback (the + /// host is swapped while the next stream loads); the player view's + /// disappearance is. + private func lendWindow() { + if let window { displayMatcher?.window = window } + } + #endif + func install(layer: LumeDisplayLayer, gravity: AVLayerVideoGravity) { layer.videoGravity = gravity guard hosted !== layer else { return } diff --git a/Sources/LumeEngineCore/Decode/AudioDecoder.swift b/Sources/LumeEngineCore/Decode/AudioDecoder.swift index f8bebb0..61a51b6 100644 --- a/Sources/LumeEngineCore/Decode/AudioDecoder.swift +++ b/Sources/LumeEngineCore/Decode/AudioDecoder.swift @@ -7,13 +7,15 @@ import Foundation /// FFmpeg audio decode + resample stage: `Channel` in, /// `Channel` out. /// -/// Output is always interleaved Float32 at the source sample rate, downmixed -/// to at most `maxOutputChannels` — the single canonical format the renderer -/// consumes. Sources with more channels than the output route can carry are -/// downmixed here by swresample (normalized matrix); handing the sample-buffer -/// renderer more channels than the route supports produces audibly broken -/// output. Resampling context is rebuilt automatically when source parameters -/// change mid-stream (codec/parameter changes are routine on IPTV). +/// Output is always interleaved Float32 at the source sample rate, in the +/// speaker layout `AudioLayoutPolicy` picks for the output route — the single +/// canonical format the renderer consumes. Sources with more channels than +/// the route carries are folded down here by swresample with an explicit +/// matrix (`DownmixMatrix`); handing the sample-buffer renderer more channels +/// than the route supports produces audibly broken output. The resampling +/// context is rebuilt when source parameters change mid-stream (codec and +/// parameter changes are routine on IPTV) and when the route's width changes +/// (`AudioOutputRoute`), in both cases at a frame boundary. /// /// Decoded frames are coalesced to at least `minOutputDuration` before /// emission. Some codecs decode to frames far too small to schedule @@ -42,6 +44,7 @@ public final class AudioDecoder: @unchecked Sendable { private var swrSourceFormat: Int32 = -1 private var swrSourceRate: Int32 = -1 private var swrSourceChannels: Int32 = -1 + private var swrRouteChannels = -1 private var outputLayout = AVChannelLayout() private var currentSerial: UInt64? private var consecutiveErrors = 0 @@ -59,7 +62,17 @@ public final class AudioDecoder: @unchecked Sendable { private static let minOutputDuration = 0.020 private let maxConsecutiveErrors = 100 - private let maxOutputChannels: Int + private let route: AudioOutputRoute + /// What the resampler currently produces. Guarded by `lock`. + private var appliedOutput: AudioOutputLayout? + + /// The layout decoded audio currently leaves in, and the route width it + /// was chosen for; nil before the first frame. + public var currentOutput: AudioOutputLayout? { + lock.lock() + defer { lock.unlock() } + return appliedOutput + } /// Channel budget of the current output route. The renderer downstream /// cannot correctly render more channels than the route carries, so @@ -72,7 +85,9 @@ public final class AudioDecoder: @unchecked Sendable { /// Feeding the renderer more channels than the session actually outputs /// fails the renderer outright (silence, wedged pipeline) — configuring /// the session for surround is the app's job (PLAN.md §2.1), the engine - /// just honors whatever was negotiated at session-creation time. + /// just honors whatever was negotiated. This reads it once, for a decoder + /// of fixed width; `PlayerSession` follows route changes instead + /// (`AudioOutputRoute`). public static func defaultMaxOutputChannels() -> Int { #if canImport(UIKit) return max(2, AVAudioSession.sharedInstance().outputNumberOfChannels) @@ -84,16 +99,32 @@ public final class AudioDecoder: @unchecked Sendable { #endif } - public init( + /// A decoder whose output width is fixed at `maxOutputChannels`. + public convenience init( parameters: CodecParameters, input: Channel, output: Channel, maxOutputChannels: Int = AudioDecoder.defaultMaxOutputChannels() + ) { + self.init( + parameters: parameters, + input: input, + output: output, + route: AudioOutputRoute(outputChannels: maxOutputChannels) + ) + } + + /// A decoder whose output follows `route` as it changes. + public init( + parameters: CodecParameters, + input: Channel, + output: Channel, + route: AudioOutputRoute ) { self.parameters = parameters self.input = input self.output = output - self.maxOutputChannels = max(1, maxOutputChannels) + self.route = route var continuation: AsyncStream.Continuation! events = AsyncStream(bufferingPolicy: .unbounded) { continuation = $0 } eventSink = continuation @@ -329,12 +360,17 @@ public final class AudioDecoder: @unchecked Sendable { let format = frame.pointee.format let rate = frame.pointee.sample_rate let channels = frame.pointee.ch_layout.nb_channels + let routeChannels = route.outputChannels - if swrContext != nil, format == swrSourceFormat, rate == swrSourceRate, channels == swrSourceChannels { + if swrContext != nil, format == swrSourceFormat, rate == swrSourceRate, channels == swrSourceChannels, + routeChannels == swrRouteChannels { return true } - emitPending() // samples of the previous format cannot share a buffer + // Samples of the previous layout cannot share a buffer. Same-rate + // conversion holds nothing back inside swr, so emitting the pending + // buffer is all a switch needs to land on a frame boundary. + emitPending() swr_free(&swrContext) // swr needs concrete layouts on both sides to build a (down)mix // matrix; sources reporting an unspecified order (routine for PCM) @@ -346,12 +382,27 @@ public final class AudioDecoder: @unchecked Sendable { av_channel_layout_default(&inputLayout, channels) } defer { av_channel_layout_uninit(&inputLayout) } + av_channel_layout_uninit(&outputLayout) - if channels > Int32(maxOutputChannels) { - av_channel_layout_default(&outputLayout, Int32(maxOutputChannels)) + var matrix: [Double]? + if inputLayout.order == AV_CHANNEL_ORDER_NATIVE { + let source = SpeakerSet(rawValue: inputLayout.u.mask) + let plan = AudioLayoutPolicy.plan(source: source, routeChannels: routeChannels) + if plan.remixes { + av_channel_layout_from_mask(&outputLayout, plan.output.rawValue) + matrix = DownmixMatrix.flatCoefficients(from: source, to: plan.output) + } else { + av_channel_layout_copy(&outputLayout, &inputLayout) + } + } else if channels > Int32(routeChannels) { + // No positions to plan with (more channels than FFmpeg has a + // default order for): fold to the route's default layout and let + // swr do what it can. + av_channel_layout_default(&outputLayout, Int32(routeChannels)) } else { av_channel_layout_copy(&outputLayout, &inputLayout) } + var newContext: OpaquePointer? let result = swr_alloc_set_opts2( &newContext, @@ -359,7 +410,16 @@ public final class AudioDecoder: @unchecked Sendable { &inputLayout, AVSampleFormat(rawValue: format), rate, 0, nil ) + // A custom matrix goes in between allocation and init. Without one + // (no remix, or a layout pair the builder refused) swr builds its own + // from the two layouts. + if result >= 0, let created = newContext, let matrix { + _ = matrix.withUnsafeBufferPointer { + swr_set_matrix(created, $0.baseAddress, Int32(DownmixMatrix.stride)) + } + } guard result >= 0, let created = newContext, swr_init(created) >= 0 else { + swr_free(&newContext) handleDecodeError(nil, EngineError(code: .decodeFailed, message: "swresample init failed")) return false } @@ -367,6 +427,15 @@ public final class AudioDecoder: @unchecked Sendable { swrSourceFormat = format swrSourceRate = rate swrSourceChannels = channels + swrRouteChannels = routeChannels + let applied = AudioOutputLayout( + sourceChannels: Int(channels), + outputChannels: Int(outputLayout.nb_channels), + routeChannels: routeChannels + ) + lock.lock() + appliedOutput = applied + lock.unlock() return true } diff --git a/Sources/LumeEngineCore/Decode/AudioLayoutPolicy.swift b/Sources/LumeEngineCore/Decode/AudioLayoutPolicy.swift new file mode 100644 index 0000000..b435039 --- /dev/null +++ b/Sources/LumeEngineCore/Decode/AudioLayoutPolicy.swift @@ -0,0 +1,169 @@ +internal import CFFmpeg + +/// Speaker positions in FFmpeg's native channel-mask bit order (`AV_CH_*`), +/// which is also WAVE order and Core Audio's `AudioChannelBitmap` for bits +/// 0...17. Kept free of FFmpeg types so the layout policy is testable as plain +/// values. +struct SpeakerSet: OptionSet, Hashable, Sendable { + let rawValue: UInt64 + + static let frontLeft = SpeakerSet(rawValue: 1 << 0) + static let frontRight = SpeakerSet(rawValue: 1 << 1) + static let frontCenter = SpeakerSet(rawValue: 1 << 2) + static let lowFrequency = SpeakerSet(rawValue: 1 << 3) + static let backLeft = SpeakerSet(rawValue: 1 << 4) + static let backRight = SpeakerSet(rawValue: 1 << 5) + static let sideLeft = SpeakerSet(rawValue: 1 << 9) + static let sideRight = SpeakerSet(rawValue: 1 << 10) + + static let mono: SpeakerSet = [.frontCenter] + static let stereo: SpeakerSet = [.frontLeft, .frontRight] + /// 5.1 with the surrounds on the side pair (`AV_CH_LAYOUT_5POINT1`, + /// "5.1(side)"): the layout the AC-3 and E-AC-3 decoders — most surround + /// on IPTV — already hand the renderer. + static let surround51: SpeakerSet = [.frontLeft, .frontRight, .frontCenter, .lowFrequency, .sideLeft, .sideRight] + static let surround71: SpeakerSet = surround51.union([.backLeft, .backRight]) + + /// Positions a format description can name with a channel bitmap + /// (`SampleBufferBuilder`). Anything outside reaches the renderer without + /// a layout, and the renderer cannot place untagged multichannel audio. + static let bitmapExpressible = SpeakerSet(rawValue: 0x3FFFF) + + var count: Int { rawValue.nonzeroBitCount } +} + +/// What the audio lane hands the renderer, for diagnostics: the source's +/// channel count, the count after the engine's remix, and the route width the +/// remix was chosen for. +public struct AudioOutputLayout: Sendable, Equatable, CustomStringConvertible { + public let sourceChannels: Int + public let outputChannels: Int + public let routeChannels: Int + + /// The engine folds the source down to fit the route. + public var isDownmixed: Bool { outputChannels < sourceChannels } + + public var description: String { + "\(sourceChannels)>\(outputChannels)ch@\(routeChannels)" + } +} + +/// Which speaker layout the engine hands the audio renderer, given the +/// source's layout and how many channels the output route carries. +/// +/// The route width is the audio session's *negotiated* output +/// (`AVAudioSession.outputNumberOfChannels`), not what the hardware could do: +/// handing `AVSampleBufferAudioRenderer` more channels than the session +/// outputs failed it outright on tvOS — silence and a wedged pipeline — which +/// is why the engine mixes instead of leaving it to the system (see +/// `AudioOutputRoute`). +/// +/// Pure value logic: the decoder asks it once per source format and route +/// width, and swresample does what it says. +enum AudioLayoutPolicy { + struct Plan: Equatable, Sendable { + /// Speakers the renderer receives. + let output: SpeakerSet + /// The engine mixes the channels (down, or mono up to stereo) rather + /// than passing them through as decoded. + let remixes: Bool + } + + /// - Parameters: + /// - source: the decoded layout, resolved to native positions. + /// - routeChannels: channels the output route carries. + static func plan(source: SpeakerSet, routeChannels: Int) -> Plan { + let route = max(1, routeChannels) + let channels = source.count + + // A lone centre channel is placed on two speakers by the engine, not + // by whatever the route does with a centre it may not have: a stereo + // pair (two HomePods, a TV's speakers) has no centre speaker at all. + // Equal power on each side, the standard pan law. + if source == .mono, route >= 2 { + return Plan(output: .stereo, remixes: true) + } + + // Fits the route and the renderer can name every channel: hand it + // over untouched. A receiver on HDMI gets discrete surround. + if channels <= route, source.isSubset(of: .bitmapExpressible) { + return Plan(output: source, remixes: false) + } + + // Fold down to the widest standard layout both the route and the + // source fill. A layout the renderer cannot name (wide or top + // channels, a matrix-encoded stereo pair) is remixed to the standard + // one of its own width rather than passed through untagged. + let width = min(route, channels) + var target: SpeakerSet = width >= 8 ? .surround71 + : width >= 6 ? .surround51 + : width >= 2 ? .stereo + : .mono + // Never invent an LFE channel the source does not have. + if !source.contains(.lowFrequency) { + target.remove(.lowFrequency) + } + return Plan(output: target, remixes: true) + } +} + +/// The coefficients the engine's remix uses, built by swresample's own +/// matrix builder with the levels below. +/// +/// Levels follow ITU-R BS.775 fold-down: centre and surrounds at −3 dB into +/// their front side, LFE omitted (it carries nothing a stereo programme needs, +/// and full-range speakers rarely reproduce it). The dialogue channel thus +/// always reaches both front speakers — the failure this exists to prevent is +/// a downmix that drops channels instead of mixing them, which loses the +/// centre and with it the dialogue. +/// +/// The matrix is not normalized, matching what swresample does on its own for +/// float output: a 5.1 fold-down plays as loud as a stereo channel next to it +/// on the same IPTV list. Normalizing would make every surround channel +/// ~7.7 dB quieter than its stereo neighbours, which viewers hear as a volume +/// jump on every zap; the cost is that peaks where every channel is near full +/// scale at once can exceed 1.0 and clip at the output. +enum DownmixMatrix { + static let centerLevel = 0.707_106_781_186_547_6 + static let surroundLevel = 0.707_106_781_186_547_6 + static let lfeLevel = 0.0 + + /// swresample's matrix bound (`SWR_CH_MAX`); its builder walks the full + /// square, so the buffer must span it whatever the layouts are. + static let stride = 64 + + /// Row-major coefficients, `[output][input]`, in native channel order; + /// nil when swresample cannot build a matrix between the two layouts + /// (e.g. a source without any front channel). + static func coefficients(from source: SpeakerSet, to output: SpeakerSet) -> [[Double]]? { + guard let flat = flatCoefficients(from: source, to: output) else { return nil } + return (0.. [Double]? { + var inputLayout = AVChannelLayout() + var outputLayout = AVChannelLayout() + defer { + av_channel_layout_uninit(&inputLayout) + av_channel_layout_uninit(&outputLayout) + } + guard av_channel_layout_from_mask(&inputLayout, source.rawValue) >= 0, + av_channel_layout_from_mask(&outputLayout, output.rawValue) >= 0 + else { return nil } + + var matrix = [Double](repeating: 0, count: stride * stride) + let result = matrix.withUnsafeMutableBufferPointer { buffer in + swr_build_matrix2( + &inputLayout, &outputLayout, + centerLevel, surroundLevel, lfeLevel, + Double(Int32.max), // no normalization, see the type docs + 1.0, buffer.baseAddress, stride, + AV_MATRIX_ENCODING_NONE, nil + ) + } + return result >= 0 ? matrix : nil + } +} diff --git a/Sources/LumeEngineCore/Decode/DolbyVisionDetection.swift b/Sources/LumeEngineCore/Decode/DolbyVisionDetection.swift index 815a03f..125d1bf 100644 --- a/Sources/LumeEngineCore/Decode/DolbyVisionDetection.swift +++ b/Sources/LumeEngineCore/Decode/DolbyVisionDetection.swift @@ -92,15 +92,12 @@ extension DolbyVisionDetection.Record { /// The stream's configuration record (`dvcC`/`dvvC`, the Matroska block /// addition mapping, or a TS descriptor), if it has one. init?(_ parameters: UnsafePointer) { - guard let sideData = av_packet_side_data_get( - parameters.pointee.coded_side_data, - parameters.pointee.nb_coded_side_data, - AV_PKT_DATA_DOVI_CONF - ), let bytes = sideData.pointee.data, - sideData.pointee.size >= MemoryLayout.size - else { return nil } - let record = UnsafeRawPointer(bytes).loadUnaligned(as: AVDOVIDecoderConfigurationRecord.self) - self.init(profile: record.dv_profile, compatibilityID: record.dv_bl_signal_compatibility_id) + guard let configuration = DolbyVisionConfiguration(parameters: parameters) else { return nil } + self.init(configuration) + } + + init(_ configuration: DolbyVisionConfiguration) { + self.init(profile: configuration.profile, compatibilityID: configuration.baseLayerCompatibilityID) } } diff --git a/Sources/LumeEngineCore/Decode/VideoDecoder.swift b/Sources/LumeEngineCore/Decode/VideoDecoder.swift index 9dbc9d5..5cd00a8 100644 --- a/Sources/LumeEngineCore/Decode/VideoDecoder.swift +++ b/Sources/LumeEngineCore/Decode/VideoDecoder.swift @@ -140,6 +140,10 @@ public final class VideoDecoder: @unchecked Sendable { private let maxConsecutiveErrors = 100 + /// Closed captions riding in the video (A/53 side data) leave here, ahead + /// of deinterlacing and conversion, for the session's caption lane. + let captionTap = ClosedCaptionTap() + public init( parameters: CodecParameters, input: Channel, @@ -365,6 +369,7 @@ public final class VideoDecoder: @unchecked Sendable { consecutiveErrors = 0 if usingHardware { hardwareFailures.recordFrame() } noteFrameSource(frame) + captionTap.offer(frame, serial: currentSerial ?? 0) emit(frame: frame) av_frame_unref(frame) } diff --git a/Sources/LumeEngineCore/Demux/MediaInfo+FFmpeg.swift b/Sources/LumeEngineCore/Demux/MediaInfo+FFmpeg.swift index 5fa29d9..99471ba 100644 --- a/Sources/LumeEngineCore/Demux/MediaInfo+FFmpeg.swift +++ b/Sources/LumeEngineCore/Demux/MediaInfo+FFmpeg.swift @@ -62,6 +62,33 @@ extension MediaInfo { } } +extension DolbyVisionConfiguration { + /// The configuration record FFmpeg filed with the stream + /// (`AV_PKT_DATA_DOVI_CONF`, from `dvcC`/`dvvC`/`dvwC` or the MPEG-TS + /// descriptor); nil when there is none. + init?(parameters: UnsafePointer) { + guard let sideData = av_packet_side_data_get( + parameters.pointee.coded_side_data, + parameters.pointee.nb_coded_side_data, + AV_PKT_DATA_DOVI_CONF + ), let bytes = sideData.pointee.data, + sideData.pointee.size >= MemoryLayout.size + else { return nil } + let record = UnsafeRawPointer(bytes).loadUnaligned(as: AVDOVIDecoderConfigurationRecord.self) + self.init( + versionMajor: record.dv_version_major, + versionMinor: record.dv_version_minor, + profile: record.dv_profile, + level: record.dv_level, + hasRPU: record.rpu_present_flag != 0, + hasEnhancementLayer: record.el_present_flag != 0, + hasBaseLayer: record.bl_present_flag != 0, + baseLayerCompatibilityID: record.dv_bl_signal_compatibility_id, + metadataCompression: record.dv_md_compression + ) + } +} + extension TrackInfo { init(stream: UnsafeMutablePointer, context: UnsafeMutablePointer) { let st = stream.pointee @@ -143,7 +170,9 @@ extension TrackInfo { colorPrimaries: par.color_primaries.rawValue, colorTransfer: par.color_trc.rawValue, colorSpace: par.color_space.rawValue, - colorRangeFull: par.color_range == AVCOL_RANGE_JPEG + colorRangeFull: par.color_range == AVCOL_RANGE_JPEG, + isInterlaced: Self.isInterlaced(par.field_order), + dolbyVision: DolbyVisionConfiguration(parameters: st.codecpar) ) } else { video = nil @@ -158,4 +187,12 @@ extension TrackInfo { audio = nil } } + + private static func isInterlaced(_ fieldOrder: AVFieldOrder) -> Bool? { + switch fieldOrder { + case AV_FIELD_UNKNOWN: nil + case AV_FIELD_PROGRESSIVE: false + default: true + } + } } diff --git a/Sources/LumeEngineCore/Demux/TrackInfo.swift b/Sources/LumeEngineCore/Demux/TrackInfo.swift index 0cb47c7..0fdf75b 100644 --- a/Sources/LumeEngineCore/Demux/TrackInfo.swift +++ b/Sources/LumeEngineCore/Demux/TrackInfo.swift @@ -21,6 +21,13 @@ public struct TrackInfo: Sendable, Identifiable, Equatable { public let colorTransfer: UInt32 public let colorSpace: UInt32 public let colorRangeFull: Bool + /// Whether the stream is coded interlaced, from the field order the + /// container or codec reports; nil when it reports none. A prediction + /// only: whether frames really come out interlaced is decided per + /// frame (`VideoDecoder.isDeinterlacing`). + public let isInterlaced: Bool? + /// The stream's Dolby Vision configuration record; nil when it has none. + public let dolbyVision: DolbyVisionConfiguration? } public struct Audio: Sendable, Equatable { @@ -57,6 +64,68 @@ public struct TrackInfo: Sendable, Identifiable, Equatable { public internal(set) var isDecodable = true } +/// A Dolby Vision decoder configuration record (`dvcC`/`dvvC`/`dvwC`), as the +/// container or the stream's descriptor carries it. +public struct DolbyVisionConfiguration: Sendable, Equatable { + public let versionMajor: UInt8 + public let versionMinor: UInt8 + public let profile: UInt8 + public let level: UInt8 + public let hasRPU: Bool + public let hasEnhancementLayer: Bool + public let hasBaseLayer: Bool + /// What the base layer is on its own: 0 none (IPTPQc2), 1 HDR10, 2 SDR, + /// 4 HLG, 6 BT.2020 HDR10. + public let baseLayerCompatibilityID: UInt8 + public let metadataCompression: UInt8 + + public init( + versionMajor: UInt8 = 1, + versionMinor: UInt8 = 0, + profile: UInt8, + level: UInt8, + hasRPU: Bool = true, + hasEnhancementLayer: Bool = false, + hasBaseLayer: Bool = true, + baseLayerCompatibilityID: UInt8, + metadataCompression: UInt8 = 0 + ) { + self.versionMajor = versionMajor + self.versionMinor = versionMinor + self.profile = profile + self.level = level + self.hasRPU = hasRPU + self.hasEnhancementLayer = hasEnhancementLayer + self.hasBaseLayer = hasBaseLayer + self.baseLayerCompatibilityID = baseLayerCompatibilityID + self.metadataCompression = metadataCompression + } + + /// The record claims an IPTPQc2 base layer rather than a displayable + /// fallback (HDR10, SDR, HLG): profiles 5, 10.0 and 20. Records lie + /// (remuxes label HDR10 as profile 5), so the engine decides per frame + /// whether to convert to HDR10 (`DolbyVisionDetection`); this is the + /// stream-level claim. + public var baseLayerIsIPT: Bool { + DolbyVisionDetection.Record(self).claimsIPT + } +} + +extension TrackInfo { + /// Index of the closed-caption track carried inside the video stream + /// (CEA-608 in A/53 side data — how US broadcast and much IPTV ship + /// captions). It is no container stream, so it gets an index no container + /// reaches ("CC" in ASCII, channel 1); pass it to `selectSubtitleTrack` + /// like any other subtitle index. + public static let embeddedClosedCaptionsIndex: Int32 = 0x4343_0001 + + /// True for the synthetic closed-caption track (`embeddedClosedCaptionsIndex`), + /// which `PlayerSession.mediaInfo` lists once captions turn up in the video. + public var isEmbeddedClosedCaptions: Bool { + index == Self.embeddedClosedCaptionsIndex + } +} + /// A chapter marker, in engine microseconds. public struct ChapterInfo: Sendable, Equatable { public let title: String? @@ -81,7 +150,9 @@ public struct MediaInfo: Sendable { public let isSeekable: Bool public let bitrate: Int64 public let metadata: [String: String] - public let tracks: [TrackInfo] + /// Streams found at open, plus tracks discovered while playing (the + /// closed-caption track — see `PlayerSession.mediaInfo`). + public internal(set) var tracks: [TrackInfo] public let chapters: [ChapterInfo] public var videoTracks: [TrackInfo] { tracks.filter { $0.kind == .video } } @@ -90,4 +161,13 @@ public struct MediaInfo: Sendable { /// Live heuristic: no known duration. public var isLive: Bool { duration == nil } + + /// This info with `track` appended, unless a track with its index is + /// already listed. + func adding(_ track: TrackInfo) -> MediaInfo { + guard !tracks.contains(where: { $0.index == track.index }) else { return self } + var copy = self + copy.tracks.append(track) + return copy + } } diff --git a/Sources/LumeEngineCore/Errors/EngineError.swift b/Sources/LumeEngineCore/Errors/EngineError.swift index 6d645c1..c93a54d 100644 --- a/Sources/LumeEngineCore/Errors/EngineError.swift +++ b/Sources/LumeEngineCore/Errors/EngineError.swift @@ -25,6 +25,13 @@ public struct EngineError: Error, Sendable, CustomStringConvertible, Equatable { /// ends arrives as `.ioError` instead: FFmpeg cannot tell it from a /// dropped connection.) case sourceEnded + /// Audio stopped reaching the renderer while video plays on. Not + /// fatal: the session keeps playing, without sound, and recovers by + /// itself if audio comes back. `PlayerSession.audioHealth` says which + /// stage lost it. (An audio decoder that gives up reports its own + /// `.decodeFailed` instead, and a failed audio renderer fails the + /// session with `.renderFailed`.) + case audioLost } /// The HTTP status class a source answered with, when FFmpeg reported one. diff --git a/Sources/LumeEngineCore/Media/AudioOutputRoute.swift b/Sources/LumeEngineCore/Media/AudioOutputRoute.swift new file mode 100644 index 0000000..1d6e014 --- /dev/null +++ b/Sources/LumeEngineCore/Media/AudioOutputRoute.swift @@ -0,0 +1,140 @@ +#if os(iOS) || os(tvOS) || os(visionOS) +import AVFAudio +#endif +import Foundation +import Synchronization + +/// How many channels the audio output route carries, kept current while the +/// route changes under a playing session (HDMI receiver → AirPlay → HomePods). +/// +/// The audio decoder reads it for every decoded frame and folds its output to +/// fit (`AudioLayoutPolicy`), so a route change takes effect at the next frame +/// boundary; the renderer reads it to keep audio decoded for a wider, earlier +/// route away from a narrower one. +/// +/// The width is what the audio session *negotiated* +/// (`outputNumberOfChannels`), never what the hardware could carry +/// (`maximumOutputNumberOfChannels`): multichannel LPCM only flows once the +/// host asks for it with `setPreferredOutputNumberOfChannels`, and a renderer +/// fed more channels than the session outputs failed outright on tvOS. +/// Configuring and activating the session is the host's job — this type only +/// observes it. +/// +/// `system()` follows the shared `AVAudioSession`. On macOS there is no audio +/// session; it reports stereo, which is what the default output overwhelmingly +/// is. A host that knows better (a macOS multichannel interface), and tests, +/// create one with `init(outputChannels:)` and drive it with `update`. +public final class AudioOutputRoute: Sendable { + public struct Snapshot: Sendable, Equatable, CustomStringConvertible { + /// Channels the route carries for this app now. + public var outputChannels: Int + /// Channels the route could carry if the host asked for them; nil + /// where nothing reports it. More than `outputChannels` on a surround + /// route means the host did not ask for the route's full width. + public var maximumOutputChannels: Int? + /// Output port types (`HDMIOutput`, `AirPlay`, …). Types only, never + /// the user-visible port names. + public var ports: [String] + /// How often the width has changed since the route was created. + public var changes: Int + + public var description: String { + var line = "\(outputChannels)ch" + if let maximumOutputChannels { line += "/\(maximumOutputChannels)" } + if !ports.isEmpty { line += " " + ports.joined(separator: "+") } + return line + } + } + + private let state: Mutex + private let observers = Mutex<[any NSObjectProtocol]>([]) + + /// A route of fixed width that only `update` changes. + public init(outputChannels: Int, maximumOutputChannels: Int? = nil, ports: [String] = []) { + state = Mutex(Snapshot( + outputChannels: max(1, outputChannels), + maximumOutputChannels: maximumOutputChannels, + ports: ports, + changes: 0 + )) + } + + deinit { + for observer in observers.withLock({ $0 }) { + NotificationCenter.default.removeObserver(observer) + } + } + + /// The route of the shared audio session, followed through route changes. + public static func system() -> AudioOutputRoute { + #if os(iOS) || os(tvOS) || os(visionOS) + let current = sessionSnapshot() + let route = AudioOutputRoute( + outputChannels: current.outputChannels, + maximumOutputChannels: current.maximumOutputChannels, + ports: current.ports + ) + route.followSession() + return route + #else + return AudioOutputRoute(outputChannels: 2) + #endif + } + + /// Channels the route carries now. Cheap: read once per decoded frame. + public var outputChannels: Int { + state.withLock { $0.outputChannels } + } + + public var snapshot: Snapshot { + state.withLock { $0 } + } + + /// Reports a new width (and, when known, capability and ports). + public func update(outputChannels: Int, maximumOutputChannels: Int? = nil, ports: [String]? = nil) { + state.withLock { snapshot in + let width = max(1, outputChannels) + if width != snapshot.outputChannels { snapshot.changes += 1 } + snapshot.outputChannels = width + if let maximumOutputChannels { snapshot.maximumOutputChannels = maximumOutputChannels } + if let ports { snapshot.ports = ports } + } + } + + #if os(iOS) || os(tvOS) || os(visionOS) + /// The session's current output. Clamped to stereo at the bottom: an + /// inactive session reports 0 or 1, and a stereo fold-down is right for + /// every real route that is narrower. + private static func sessionSnapshot() -> (outputChannels: Int, maximumOutputChannels: Int, ports: [String]) { + let session = AVAudioSession.sharedInstance() + return ( + max(2, session.outputNumberOfChannels), + session.maximumOutputNumberOfChannels, + session.currentRoute.outputs.map(\.portType.rawValue) + ) + } + + /// Route changes are posted on a system thread; the width is re-read + /// there. A change of the negotiated width alone (the host asking for more + /// channels) arrives as `.routeConfigurationChange`, so every reason is + /// taken. + private func followSession() { + // The token is created inside the lock so it never leaves the + // mutex's region (it is not Sendable). + observers.withLock { observers in + observers.append(NotificationCenter.default.addObserver( + forName: AVAudioSession.routeChangeNotification, + object: nil, + queue: nil + ) { [weak self] _ in + let current = Self.sessionSnapshot() + self?.update( + outputChannels: current.outputChannels, + maximumOutputChannels: current.maximumOutputChannels, + ports: current.ports + ) + }) + } + } + #endif +} diff --git a/Sources/LumeEngineCore/Media/ContentDisplayCriteria.swift b/Sources/LumeEngineCore/Media/ContentDisplayCriteria.swift new file mode 100644 index 0000000..b2e0691 --- /dev/null +++ b/Sources/LumeEngineCore/Media/ContentDisplayCriteria.swift @@ -0,0 +1,369 @@ +import CoreMedia +import CoreVideo +import Foundation + +/// The display mode a session's video wants: refresh rate and dynamic range, +/// as an Apple TV with Match Content switches the TV for it. +/// +/// Describes what the engine *outputs*, which is not what the container says. +/// The deinterlacer at field rate turns 1080i50 into 50p, so the TV should +/// run at 50 Hz, not 25; Dolby Vision with an IPT base layer (profiles 5, +/// 10.0, 20) is converted to HDR10 before it is enqueued, and every other +/// Dolby Vision profile is presented as its base layer (HDR10, HLG or SDR) — +/// so the TV is asked for that, never for Dolby Vision. Frames the renderer has been handed decide when +/// there are any (`basis == .output`); before that the stream's own metadata, +/// with the deinterlacer's effect predicted, stands in (`.stream`). +/// +/// Cross-platform and pure: tvOS applies it with `DisplayCriteriaMatcher`, +/// anything else can log or show it. +public struct ContentDisplayCriteria: Sendable, Equatable, CustomStringConvertible { + public enum DynamicRange: String, Sendable { + case sdr + case hdr10 + case hlg + /// Never produced today: the engine presents Dolby Vision streams as + /// their base layer (or converts them to HDR10). Reserved for a real + /// Dolby Vision output path (PLAN.md §7, compressed `dvh1` enqueue) — + /// that is the point to ask the TV for Dolby Vision mode — so switches + /// written now stay exhaustive then. + case dolbyVision + } + + /// What the criteria were derived from. + public enum Basis: String, Sendable { + /// Stream metadata, with the deinterlacer's effect predicted. + case stream + /// Frames handed to the renderer: their colour tags and measured + /// cadence. + case output + } + + /// The refresh rate to ask for: one of the film and broadcast rates + /// (23.976, 24, 25, 29.97, 30, 50, 59.94, 60 — NTSC rates exact, e.g. + /// 24000/1001). Nil when the content's rate is unknown or off that grid + /// (variable, 12.5, 48, 100 fps): a TV asked for an odd rate either + /// refuses or picks something worse than the mode it is already in. + public let refreshRate: Double? + public let dynamicRange: DynamicRange + public let width: Int + public let height: Int + public let basis: Basis + + // Ingredients of `makeFormatDescription()`. + let codecType: CMVideoCodecType + let colorPrimaries: String? + let transferFunction: String? + let yCbCrMatrix: String? + let masteringDisplayColorVolume: Data? + let contentLightLevel: Data? + + /// Whether the two ask the TV for the same mode — the only thing that + /// decides whether applying one after the other switches the TV again. + public func asksForSameMode(as other: ContentDisplayCriteria) -> Bool { + refreshRate == other.refreshRate && dynamicRange == other.dynamicRange + } + + public var description: String { + let rate = refreshRate.map { String(format: "%.3gHz", $0) } ?? "anyHz" + return "\(rate)/\(dynamicRange.rawValue)(\(basis.rawValue))" + } + + /// A video format description carrying these criteria, for + /// `AVDisplayCriteria(refreshRate:formatDescription:)`. Colour tags come + /// from the enqueued frames when there are any, so the description says + /// what the display is actually handed. + public func makeFormatDescription() -> CMVideoFormatDescription? { + var extensions: [CFString: Any] = [:] + if let colorPrimaries { extensions[kCMFormatDescriptionExtension_ColorPrimaries] = colorPrimaries } + if let transferFunction { extensions[kCMFormatDescriptionExtension_TransferFunction] = transferFunction } + if let yCbCrMatrix { extensions[kCMFormatDescriptionExtension_YCbCrMatrix] = yCbCrMatrix } + if let masteringDisplayColorVolume { + extensions[kCMFormatDescriptionExtension_MasteringDisplayColorVolume] = masteringDisplayColorVolume + } + if let contentLightLevel { + extensions[kCMFormatDescriptionExtension_ContentLightLevelInfo] = contentLightLevel + } + var description: CMVideoFormatDescription? + let status = CMVideoFormatDescriptionCreate( + allocator: kCFAllocatorDefault, + codecType: codecType, + width: Int32(clamping: width), + height: Int32(clamping: height), + extensions: extensions as CFDictionary, + formatDescriptionOut: &description + ) + return status == noErr ? description : nil + } +} + +// MARK: - Resolution + +extension ContentDisplayCriteria { + /// What the stream says about itself (from `TrackInfo`). + struct Stream: Sendable, Equatable { + var codecName: String + var width: Int + var height: Int + /// Frames per second as the container guesses it; 0 when unknown. + var frameRate: Double + var isInterlaced: Bool? + /// Raw transfer characteristics (ITU-T H.273 code point). + var colorTransfer: UInt32 + var dolbyVision: DolbyVisionConfiguration? + + init( + codecName: String, + width: Int, + height: Int, + frameRate: Double, + isInterlaced: Bool? = nil, + colorTransfer: UInt32 = 2, + dolbyVision: DolbyVisionConfiguration? = nil + ) { + self.codecName = codecName + self.width = width + self.height = height + self.frameRate = frameRate + self.isInterlaced = isInterlaced + self.colorTransfer = colorTransfer + self.dolbyVision = dolbyVision + } + + init?(track: TrackInfo) { + guard let video = track.video else { return nil } + self.init( + codecName: track.codecName, + width: video.width, + height: video.height, + frameRate: video.fps, + isInterlaced: video.isInterlaced, + colorTransfer: video.colorTransfer, + dolbyVision: video.dolbyVision + ) + } + } + + /// What the renderer has actually been handed. + struct Output: Sendable, Equatable { + var width: Int + var height: Int + var colorPrimaries: String? + var transferFunction: String? + var yCbCrMatrix: String? + var masteringDisplayColorVolume: Data? + var contentLightLevel: Data? + /// Measured interval between frames, engine µs; nil until known. + var frameInterval: Double? + /// Frames are going through the deinterlacer. + var isDeinterlacing: Bool + + init( + width: Int, + height: Int, + colorPrimaries: String? = nil, + transferFunction: String? = nil, + yCbCrMatrix: String? = nil, + masteringDisplayColorVolume: Data? = nil, + contentLightLevel: Data? = nil, + frameInterval: Double? = nil, + isDeinterlacing: Bool = false + ) { + self.width = width + self.height = height + self.colorPrimaries = colorPrimaries + self.transferFunction = transferFunction + self.yCbCrMatrix = yCbCrMatrix + self.masteringDisplayColorVolume = masteringDisplayColorVolume + self.contentLightLevel = contentLightLevel + self.frameInterval = frameInterval + self.isDeinterlacing = isDeinterlacing + } + + /// Reads the tags the enqueued frames carried into their format + /// description (pixel buffer attachments propagate into it). + init(formatDescription: CMFormatDescription, frameInterval: Double?, isDeinterlacing: Bool) { + func string(_ key: CFString) -> String? { + CMFormatDescriptionGetExtension(formatDescription, extensionKey: key) as? String + } + func data(_ key: CFString) -> Data? { + CMFormatDescriptionGetExtension(formatDescription, extensionKey: key) as? Data + } + let dimensions = CMVideoFormatDescriptionGetDimensions(formatDescription) + self.init( + width: Int(dimensions.width), + height: Int(dimensions.height), + colorPrimaries: string(kCMFormatDescriptionExtension_ColorPrimaries), + transferFunction: string(kCMFormatDescriptionExtension_TransferFunction), + yCbCrMatrix: string(kCMFormatDescriptionExtension_YCbCrMatrix), + masteringDisplayColorVolume: data(kCMFormatDescriptionExtension_MasteringDisplayColorVolume), + contentLightLevel: data(kCMFormatDescriptionExtension_ContentLightLevelInfo), + frameInterval: frameInterval, + isDeinterlacing: isDeinterlacing + ) + } + } + + /// The criteria for a stream, refined by its output once there is any. + static func resolve( + stream: Stream, + deinterlacing: VideoDecoder.Deinterlacing, + output: Output? + ) -> ContentDisplayCriteria { + let frameRate: Double + let basis: Basis + if let interval = output?.frameInterval, interval > 0 { + // The cadence of what is enqueued already includes whatever the + // deinterlacer did to it. + frameRate = Double(MediaTime.timeBase) / interval + basis = .output + } else { + let deinterlaced = output?.isDeinterlacing ?? willDeinterlace(stream, deinterlacing) + let fieldRate = deinterlaced && deinterlacing.rate == .field + frameRate = stream.frameRate * (fieldRate ? 2 : 1) + basis = .stream + } + + let range = dynamicRange(stream: stream, output: output) + let tags = output.map { + ColorTags(primaries: $0.colorPrimaries, transfer: $0.transferFunction, matrix: $0.yCbCrMatrix) + } ?? ColorTags.canonical(for: range) + + return ContentDisplayCriteria( + refreshRate: nominalRefreshRate(for: frameRate), + dynamicRange: range, + width: output?.width ?? stream.width, + height: output?.height ?? stream.height, + basis: basis, + codecType: codecType(codecName: stream.codecName), + colorPrimaries: tags.primaries, + transferFunction: tags.transfer, + yCbCrMatrix: tags.matrix, + masteringDisplayColorVolume: output?.masteringDisplayColorVolume, + contentLightLevel: output?.contentLightLevel + ) + } + + /// Before the first frame: whether the deinterlacer is going to engage. + /// `.auto` engages on the first frame the decoder flags interlaced, which + /// the stream's field order predicts; an unknown field order predicts + /// progressive, the common case. + private static func willDeinterlace(_ stream: Stream, _ policy: VideoDecoder.Deinterlacing) -> Bool { + switch policy.mode { + case .off: false + case .always: true + case .auto: stream.isInterlaced == true + } + } + + /// The range of what is presented. Dolby Vision with an IPT base layer is + /// converted to HDR10, so it is HDR10 whatever its frames were tagged + /// before conversion. Any other Dolby Vision stream is presented as its + /// base layer — the engine has no Dolby Vision output, the RPU is not + /// applied — so it is treated like the HDR10, HLG or SDR stream it then + /// is: the transfer function of the enqueued frames decides, and before + /// there are any, the base layer's compatibility ID (more reliable than + /// the VUI, which DV encodes often leave unspecified), else the stream's + /// transfer characteristics. Asking for Dolby Vision mode belongs to a + /// real Dolby Vision output path (PLAN.md §7), not to this one. + static func dynamicRange(stream: Stream, output: Output?) -> DynamicRange { + if stream.dolbyVision?.baseLayerIsIPT == true { + return .hdr10 + } + if let output { + // An untagged frame is shown as SDR, so it is SDR. + let transfer = output.transferFunction + if transfer == kCVImageBufferTransferFunction_SMPTE_ST_2084_PQ as String { return .hdr10 } + if transfer == kCVImageBufferTransferFunction_ITU_R_2100_HLG as String { return .hlg } + return .sdr + } + if let presented = stream.dolbyVision.flatMap(baseLayerRange) { + return presented + } + // H.273 transfer characteristics, which FFmpeg's enum mirrors: + // 16 is SMPTE ST 2084 (PQ), 18 is ARIB STD-B67 (HLG). + switch stream.colorTransfer { + case 16: return .hdr10 + case 18: return .hlg + default: return .sdr + } + } + + /// What a Dolby Vision base layer is on its own, by its signal + /// compatibility ID; nil for IDs that say nothing displayable. + private static func baseLayerRange(_ configuration: DolbyVisionConfiguration) -> DynamicRange? { + switch configuration.baseLayerCompatibilityID { + case 1, 6: .hdr10 // HDR10 (6: BT.2020 HDR10, profile 7) + case 4: .hlg + case 2: .sdr + default: nil + } + } + + // MARK: Refresh rates + + /// The rates an Apple TV offers a TV: film, PAL and NTSC, at frame and + /// field rate. + static let nominalRefreshRates: [Double] = [ + 24000.0 / 1001, 24, 25, 30000.0 / 1001, 30, 50, 60000.0 / 1001, 60, + ] + + /// Relative distance within which a measured rate counts as nominal. + /// Wide enough for timestamp jitter, narrow enough that nothing between + /// two grid rates (25.5, 48) is pulled onto one. 23.976 and 24 are 0.1 % + /// apart; nearest-wins keeps them apart. + static let refreshRateTolerance = 0.005 + + static func nominalRefreshRate(for frameRate: Double) -> Double? { + guard frameRate.isFinite, frameRate > 0 else { return nil } + guard let nearest = nominalRefreshRates.min(by: { abs($0 - frameRate) < abs($1 - frameRate) }), + abs(nearest - frameRate) / nearest <= refreshRateTolerance + else { return nil } + return nearest + } + + // MARK: Format description ingredients + + private struct ColorTags { + var primaries: String? + var transfer: String? + var matrix: String? + + /// What a stream of that range is tagged as, for the criteria built + /// before any frame exists. + static func canonical(for range: DynamicRange) -> ColorTags { + switch range { + case .sdr: + ColorTags( + primaries: kCVImageBufferColorPrimaries_ITU_R_709_2 as String, + transfer: kCVImageBufferTransferFunction_ITU_R_709_2 as String, + matrix: kCVImageBufferYCbCrMatrix_ITU_R_709_2 as String + ) + case .hdr10, .dolbyVision: + ColorTags( + primaries: kCVImageBufferColorPrimaries_ITU_R_2020 as String, + transfer: kCVImageBufferTransferFunction_SMPTE_ST_2084_PQ as String, + matrix: kCVImageBufferYCbCrMatrix_ITU_R_2020 as String + ) + case .hlg: + ColorTags( + primaries: kCVImageBufferColorPrimaries_ITU_R_2020 as String, + transfer: kCVImageBufferTransferFunction_ITU_R_2100_HLG as String, + matrix: kCVImageBufferYCbCrMatrix_ITU_R_2020 as String + ) + } + } + } + + /// The sample entry type of the stream's codec. The display mode does not + /// depend on it; a familiar type keeps the description ordinary, so an + /// unknown codec is described as H.264. + static func codecType(codecName: String) -> CMVideoCodecType { + switch codecName { + case "hevc": return kCMVideoCodecType_HEVC + case "av1": return kCMVideoCodecType_AV1 + case "vp9": return kCMVideoCodecType_VP9 + case "mpeg2video": return kCMVideoCodecType_MPEG2Video + default: return kCMVideoCodecType_H264 + } + } +} diff --git a/Sources/LumeEngineCore/Media/DisplayCriteriaMatcher.swift b/Sources/LumeEngineCore/Media/DisplayCriteriaMatcher.swift new file mode 100644 index 0000000..24f203b --- /dev/null +++ b/Sources/LumeEngineCore/Media/DisplayCriteriaMatcher.swift @@ -0,0 +1,203 @@ +#if os(tvOS) +@preconcurrency import AVKit +import Foundation +import UIKit + +/// Match Content for the sample-buffer pipeline: asks the TV for the refresh +/// rate and dynamic range a session's video wants, through the window's +/// `AVDisplayManager`. +/// +/// AVPlayer does this by itself; a sample-buffer renderer cannot, so without +/// it an Apple TV set to Match Frame Rate / Match Dynamic Range leaves a +/// 1080i50 channel judder-converted to the UI's 60 Hz and HDR in the SDR +/// mode. The criteria are the engine's output (`PlayerSession.displayCriteria`), +/// not the container's: field-rate deinterlacing makes 50i a 50 Hz request, +/// and an IPT Dolby Vision stream converted to HDR10 asks for HDR10. +/// +/// Opt-in: nothing changes the display mode unless a host creates a matcher. +/// One per player screen, kept across sessions: zapping between two channels +/// that want the same mode must not send the TV through the UI's mode and +/// back, so `match` only asks when the mode differs, and only `reset` (the +/// player screen going away) hands the display back. An unreset preference +/// keeps the TV in the content's mode after playback — call `reset`. +/// +/// A mode switch blanks the TV for one to three seconds. While the matcher is +/// waiting for one, it holds the session's start (`PlayerSession.holdStart`), +/// so the opening plays to a lit screen; the hold is bounded by +/// `Options.maximumHold`, in the session as well as here. +@MainActor +public final class DisplayCriteriaMatcher { + public struct Options: Sendable { + /// Hold the session's start while the TV switches mode. + public var holdsStartDuringModeSwitch = true + /// Upper bound on that hold, counted from `match(_:)`. The session + /// starts by itself once it passes. + public var maximumHold: Double = 6 + /// How long to wait for the first frames to confirm the criteria + /// before asking for what the stream predicts. Frames usually arrive + /// well inside it; a prediction that turns out wrong costs a second + /// mode switch. + public var outputWait: Double = 2 + /// After asking for a mode, how long to wait for the switch to begin + /// before concluding the TV is already in it. + public var switchStartWindow: Double = 0.75 + + public init() {} + } + + /// The window whose display is matched — the one showing the video. + /// Without one nothing is applied. + public weak var window: UIWindow? + public let options: Options + /// What the display was last asked for; nil when nothing is, or after + /// `reset`. + public private(set) var appliedCriteria: ContentDisplayCriteria? + + private var trackingTask: Task? + private weak var heldSession: PlayerSession? + /// Sessions that show no frames are given up on after this long. + private static let trackingLimit: Double = 15 + + public init(window: UIWindow? = nil, options: Options = Options()) { + self.window = window + self.options = options + } + + /// True while the user's settings let apps switch the display mode + /// (Settings → Video and Audio → Match Content). + public var isMatchingEnabled: Bool { + window?.avDisplayManager.isDisplayCriteriaMatchingEnabled ?? false + } + + /// True while the display is switching modes and may be blank. + public var isModeSwitchInProgress: Bool { + window?.avDisplayManager.isDisplayModeSwitchInProgress ?? false + } + + /// Matches the display to `session`'s video. Call after `open` succeeds + /// and before `play()`: the start hold is in place when this returns, the + /// rest (waiting for the first frames, the switch) continues in the + /// background. A later call for another session supersedes this one. + public func match(_ session: PlayerSession) async { + trackingTask?.cancel() + if let previous = heldSession, previous !== session { + await previous.releaseStartHold() + } + heldSession = nil + + let holding = options.holdsStartDuringModeSwitch && isMatchingEnabled + if holding { + heldSession = session + await session.holdStart(for: options.maximumHold) + } + let started = ProcessInfo.processInfo.systemUptime + trackingTask = Task { [weak self] in + await self?.track(session, holding: holding, since: started) + } + } + + /// Hands the display back to the system's default mode and stops + /// matching. Call when the player screen goes away — not between + /// sessions on it. + public func reset() { + trackingTask?.cancel() + trackingTask = nil + if let session = heldSession { + Task { await session.releaseStartHold() } + } + heldSession = nil + if appliedCriteria != nil { + window?.avDisplayManager.preferredDisplayCriteria = nil + } + appliedCriteria = nil + } + + // MARK: Tracking + + /// Applies the criteria once the first frames confirm them, or the + /// stream's prediction when frames are slow, correcting a prediction once + /// if the frames disagree. Mid-stream changes are not chased: every + /// switch blanks the TV, and a live channel's ad break must not. + private func track(_ session: PlayerSession, holding: Bool, since started: TimeInterval) async { + var holding = holding + var applied: ContentDisplayCriteria? + while !Task.isCancelled { + let elapsed = ProcessInfo.processInfo.systemUptime - started + if let criteria = await session.displayCriteria { + let confirmed = criteria.basis == .output + if applied == nil, confirmed || elapsed >= options.outputWait { + await apply(criteria, to: session, holding: holding, since: started) + applied = criteria + // The TV is in its mode (or the hold ran out): start. + if holding { + holding = false + await release(session) + } + } else if let previous = applied, previous.basis == .stream, confirmed { + if !previous.asksForSameMode(as: criteria) { + await apply(criteria, to: session, holding: false, since: started) + } + applied = criteria + } + if applied?.basis == .output { break } + } else if await session.mediaInfo != nil, await session.selectedVideoTrackIndex == nil { + break // nothing to match: no video + } + guard elapsed < Self.trackingLimit else { break } + try? await Task.sleep(for: .milliseconds(100)) + } + // Gave up without asking (no video, no frames): don't keep the + // session waiting for a switch that is not coming. A cancelled track + // was superseded, and whoever cancelled it released the hold. + if holding, !Task.isCancelled { + await release(session) + } + } + + private func release(_ session: PlayerSession) async { + await session.releaseStartHold() + if heldSession === session { heldSession = nil } + } + + private func apply( + _ criteria: ContentDisplayCriteria, + to session: PlayerSession, + holding: Bool, + since started: TimeInterval + ) async { + guard let manager = window?.avDisplayManager, + manager.isDisplayCriteriaMatchingEnabled, + let format = criteria.makeFormatDescription() + else { return } + if let current = appliedCriteria, current.asksForSameMode(as: criteria) { + appliedCriteria = criteria + return // already in it: asking again would at best do nothing + } + // A refresh rate of 0 leaves the rate to the system: content whose + // rate is unknown or off the grid still gets its dynamic range. + manager.preferredDisplayCriteria = AVDisplayCriteria( + refreshRate: Float(criteria.refreshRate ?? 0), + formatDescription: format + ) + appliedCriteria = criteria + guard holding else { return } + await waitForModeSwitch(manager, deadline: started + options.maximumHold) + } + + /// Waits for the switch the last request started, if it started one. + /// Polled: the flag is the documented signal, and a switch lasts seconds. + private func waitForModeSwitch(_ manager: AVDisplayManager, deadline: TimeInterval) async { + func now() -> TimeInterval { ProcessInfo.processInfo.systemUptime } + let asked = now() + var switching = manager.isDisplayModeSwitchInProgress + while !switching, now() - asked < options.switchStartWindow, now() < deadline, !Task.isCancelled { + try? await Task.sleep(for: .milliseconds(50)) + switching = manager.isDisplayModeSwitchInProgress + } + while switching, now() < deadline, !Task.isCancelled { + try? await Task.sleep(for: .milliseconds(50)) + switching = manager.isDisplayModeSwitchInProgress + } + } +} +#endif diff --git a/Sources/LumeEngineCore/Render/FrameCadence.swift b/Sources/LumeEngineCore/Render/FrameCadence.swift new file mode 100644 index 0000000..e455a92 --- /dev/null +++ b/Sources/LumeEngineCore/Render/FrameCadence.swift @@ -0,0 +1,55 @@ +/// The interval between the video frames the renderer is actually handed — +/// measured, because the container's frame rate is a guess about the input, +/// not a statement about the output. The deinterlacer at field rate doubles +/// it (1080i50 comes out as 50p), a field-coded stream can report its field +/// rate as its frame rate, and live sources carry no reliable rate at all. +/// +/// Pure value type, confined to whoever feeds it (the renderer keeps it under +/// its lock): one subtraction per frame, the statistics only when asked. +struct FrameCadence: Sendable, Equatable { + /// Intervals kept: enough to see through the odd dropped or repeated frame. + static let capacity = 32 + /// Intervals needed before a cadence is reported. Fewer is a guess. + static let minimumSamples = 8 + /// Longer gaps are dropouts or seams, not frame intervals (5 fps floor). + static let maximumInterval: Int64 = 200_000 + + private var lastPTS: Int64? + private var intervals: [Int64] = [] + private var nextSlot = 0 + + /// Forgets the previous frame, keeping what was learned: after a seek the + /// stream is the same, only its timeline jumped. + mutating func restart() { + lastPTS = nil + } + + mutating func observe(pts: Int64) { + guard MediaTime.isValid(pts) else { return } + defer { lastPTS = pts } + guard let lastPTS else { return } + let interval = pts - lastPTS + // Backwards or too long: a seam or a dropout, not a frame interval. + guard interval > 0, interval <= Self.maximumInterval else { return } + if intervals.count < Self.capacity { + intervals.append(interval) + } else { + intervals[nextSlot] = interval + nextSlot = (nextSlot + 1) % Self.capacity + } + } + + /// The typical frame interval in engine µs, nil until enough frames were + /// seen. The mean of the middle half: timestamps quantized by a 90 kHz + /// clock and rounded to µs alternate between two neighbouring values + /// (59.94 fps is 16 678 / 16 689 µs), so a single median sample would be + /// off by more than the gap between 59.94 and 60; the outer quartiles + /// drop the odd duplicated or skipped frame. + var interval: Double? { + guard intervals.count >= Self.minimumSamples else { return nil } + let sorted = intervals.sorted() + let quarter = sorted.count / 4 + let middle = sorted[quarter..<(sorted.count - quarter)] + return Double(middle.reduce(0, +)) / Double(middle.count) + } +} diff --git a/Sources/LumeEngineCore/Render/SystemRenderer.swift b/Sources/LumeEngineCore/Render/SystemRenderer.swift index 5513a50..f2a30fb 100644 --- a/Sources/LumeEngineCore/Render/SystemRenderer.swift +++ b/Sources/LumeEngineCore/Render/SystemRenderer.swift @@ -37,6 +37,11 @@ public final class SystemRenderer: @unchecked Sendable { /// by a flush): startup diagnostics. private var firstVideoEnqueue: TimeInterval? private var firstAudioEnqueue: TimeInterval? + /// What the display is actually being handed — format and cadence of the + /// enqueued frames, after deinterlacing and Dolby Vision conversion. Read + /// for the display criteria (`PlayerSession.displayCriteria`). + private var videoCadence = FrameCadence() + private var lastVideoFormat: CMVideoFormatDescription? private var videoPumpScheduled = false private var audioPumpScheduled = false private var stopped = false @@ -46,6 +51,8 @@ public final class SystemRenderer: @unchecked Sendable { private var audioFormatCache: (description: CMAudioFormatDescription, sampleRate: Int, channels: Int, channelBitmap: UInt64)? // Audio-queue-confined; serial changes re-anchor it after a seek flush. private var audioTimeline = AudioTimeline() + /// Route width at the previous audio pump pass. Audio-queue-confined. + private var lastAudioRouteChannels = Int.max /// Audio enqueued but not yet played, kept so it can be enqueued again /// after the renderer drops it on its own (see `refillAfterAutomaticFlush`). @@ -55,6 +62,7 @@ public final class SystemRenderer: @unchecked Sendable { let serial: UInt64 /// End of the sample on the engine timeline (µs). let end: Int64 + let channels: Int } private var retainedAudio: [RetainedAudio] = [] /// Backstop only: the renderer stops asking for audio once it holds a @@ -83,7 +91,13 @@ public final class SystemRenderer: @unchecked Sendable { private var audioLaneAttached = false private var reportedFailure = false - public init(muted: Bool = false) { + /// The route the audio lane's decoder fits its output to. Audio decoded + /// for a wider route before it narrowed is held back from the renderer + /// (see `pumpAudio`); nil feeds every frame. + private let audioOutputRoute: AudioOutputRoute? + + public init(muted: Bool = false, audioOutputRoute: AudioOutputRoute? = nil) { + self.audioOutputRoute = audioOutputRoute displayLayer = AVSampleBufferDisplayLayer() displayLayer.videoGravity = .resizeAspect videoRenderer = displayLayer.sampleBufferRenderer @@ -166,7 +180,11 @@ public final class SystemRenderer: @unchecked Sendable { // answer; that audio is retained too, so drop it and re-enqueue // everything in order. audioRenderer.flush() - for entry in retainedAudio where entry.serial == serial && (!MediaTime.isValid(flushTime) || entry.end > flushTime) { + // A route change is the usual cause of the flush; audio retained for + // a wider route than the new one stays out (see `pumpAudio`). + let routeChannels = audioOutputRoute?.outputChannels ?? .max + for entry in retainedAudio where entry.serial == serial && entry.channels <= routeChannels + && (!MediaTime.isValid(flushTime) || entry.end > flushTime) { audioRenderer.enqueue(entry.sample) } // A seek that flushed while this ran may have been overtaken by the @@ -312,6 +330,7 @@ public final class SystemRenderer: @unchecked Sendable { lastEnqueuedAudioPTS = MediaTime.noTimestamp firstEnqueuedVideoPTS = MediaTime.noTimestamp firstEnqueuedAudioPTS = MediaTime.noTimestamp + videoCadence.restart() lock.unlock() videoRenderer.flush() @@ -345,6 +364,15 @@ public final class SystemRenderer: @unchecked Sendable { return (firstVideoEnqueue, firstAudioEnqueue) } + /// Format of the most recently enqueued video frame and the measured + /// interval between frames (engine µs; nil until enough were seen). Nil + /// before the first frame. + var videoOutput: (format: CMVideoFormatDescription, frameInterval: Double?)? { + lock.lock() + defer { lock.unlock() } + return lastVideoFormat.map { ($0, videoCadence.interval) } + } + /// Raw renderer health for diagnostics (status is /// `AVQueuedSampleBufferRenderingStatus`: 0 unknown, 1 rendering, 2 failed). public var rendererHealth: (videoStatus: Int, audioStatus: Int, videoError: String?, audioError: String?) { @@ -356,6 +384,10 @@ public final class SystemRenderer: @unchecked Sendable { ) } + var audioRendererFailed: Bool { + audioRenderer.status == .failed + } + public func shutdown() { lock.lock() stopped = true @@ -402,6 +434,8 @@ public final class SystemRenderer: @unchecked Sendable { lastEnqueuedVideoPTS = frame.pts if !MediaTime.isValid(firstEnqueuedVideoPTS) { firstEnqueuedVideoPTS = frame.pts } if firstVideoEnqueue == nil { firstVideoEnqueue = ProcessInfo.processInfo.systemUptime } + videoCadence.observe(pts: frame.pts) + lastVideoFormat = videoFormatCache lock.unlock() } catch { continue // malformed frame: drop, keep the pump alive @@ -417,12 +451,25 @@ public final class SystemRenderer: @unchecked Sendable { lock.unlock() guard let input, !isStopped else { return } + // The route narrowed (HDMI surround → AirPlay stereo): audio already + // decoded for the wider route must not reach the renderer, which + // fails on more channels than the route carries. The decoder refits + // at its next frame; until then what is queued for the old width is + // dropped, and whatever the renderer still holds of it is replaced by + // the retained audio that fits — usually nothing, so the switch costs + // the renderer's lead in silence, as a route change does anyway. + let routeChannels = audioOutputRoute?.outputChannels ?? .max + if routeChannels < lastAudioRouteChannels { + refillAfterAutomaticFlush(from: currentTime) + } + lastAudioRouteChannels = routeChannels + while audioRenderer.isReadyForMoreMediaData { guard let frame = input.tryReceive() else { scheduleAudioRetry() return } - guard frame.serial == serial else { continue } + guard frame.serial == serial, frame.channels <= routeChannels else { continue } do { let sample = try SampleBufferBuilder.audio( from: frame, @@ -435,7 +482,7 @@ public final class SystemRenderer: @unchecked Sendable { if !MediaTime.isValid(firstEnqueuedAudioPTS) { firstEnqueuedAudioPTS = frame.pts } if firstAudioEnqueue == nil { firstAudioEnqueue = ProcessInfo.processInfo.systemUptime } lock.unlock() - retainAudio(sample, serial: serial, end: frame.pts + frame.duration) + retainAudio(sample, serial: serial, end: frame.pts + frame.duration, channels: frame.channels) } catch { continue } @@ -443,13 +490,13 @@ public final class SystemRenderer: @unchecked Sendable { } /// Keeps `sample` until the clock has played past it. Audio queue only. - private func retainAudio(_ sample: CMSampleBuffer, serial: UInt64, end: Int64) { + private func retainAudio(_ sample: CMSampleBuffer, serial: UInt64, end: Int64, channels: Int) { let now = currentTime retainedAudio.removeAll { entry in entry.serial != serial || (MediaTime.isValid(now) && entry.end <= now) } if retainedAudio.count >= Self.maxRetainedAudio { retainedAudio.removeFirst() } - retainedAudio.append(RetainedAudio(sample: sample, serial: serial, end: end)) + retainedAudio.append(RetainedAudio(sample: sample, serial: serial, end: end, channels: channels)) } private func scheduleVideoRetry() { diff --git a/Sources/LumeEngineCore/Session/AudioHealth.swift b/Sources/LumeEngineCore/Session/AudioHealth.swift new file mode 100644 index 0000000..5c69f4f --- /dev/null +++ b/Sources/LumeEngineCore/Session/AudioHealth.swift @@ -0,0 +1,153 @@ +import Foundation + +/// Whether the session's audio is reaching the viewer, as far as the engine +/// can tell. Read `PlayerSession.audioHealth`; it is also in `Diagnostics`. +/// +/// Exists because audio can die while video plays on: a decoder that gives up +/// leaves its lane at end of stream, and the session — correctly — keeps +/// showing the picture. Without this the only trace was one `.error` event, +/// and the viewer sat in front of a silent channel. +public enum AudioHealth: Sendable, Equatable, CustomStringConvertible { + /// Audio is playing, or nothing says otherwise yet. + case healthy + /// The session has no audio lane: the source carries no audio, or the + /// host opened it without (`enableAudio`) or turned it off. + case absent + /// The audio lane has stopped while the session plays on. + case lost(AudioLossReason) + + public var description: String { + switch self { + case .healthy: "ok" + case .absent: "none" + case .lost(let reason): "lost(\(reason.rawValue))" + } + } +} + +/// Which stage the audio was lost in. +public enum AudioLossReason: String, Sendable, Equatable { + /// The audio decoder gave up on the stream (its error went out as an + /// `.error` event). Permanent for the track: switching tracks, or audio + /// off and on, builds a new decoder. + case decoderFailed + /// The system audio renderer failed. The session fails with + /// `.renderFailed` as well; this records that it was the audio lane. + case rendererFailed + /// The clock runs and video plays, but for a while now the audio renderer + /// has had nothing to play at the playhead: decoded audio stopped reaching + /// it, or reached it too late to play. Reported once per episode as an + /// `.error` with `EngineError.Code.audioLost`; clears by itself when audio + /// comes back. + case starved +} + +/// Decides `AudioHealth` from what the session observes on its monitor tick. +/// +/// Pure value logic, so every branch is testable without a pipeline that has +/// to be made to lose its audio. +/// +/// The starvation judgement is deliberately narrow, so that moments without +/// audio that are part of normal playback never count: it needs a running +/// clock (not paused, buffering or seeking), a video lane that is itself ahead +/// of the playhead (a whole pipeline running dry is buffering, which the +/// session handles), an audio lane still expecting data (not at its end), and +/// the playhead to have moved `threshold` seconds of media with no audio +/// enqueued beyond it. A gap *inside* the audio does not count either: the +/// renderer's high-water mark is past the gap as soon as audio after it was +/// enqueued. +struct AudioLossDetector { + /// Seconds of media the playhead must cover without audio ahead of it. + static let threshold = 2.0 + + struct Observation: Equatable { + /// An audio lane is attached. + var hasAudioLane: Bool + /// The audio decoder reached the end of its stream: nothing more is + /// coming, and silence after it is the source's end, not a loss. + var audioEnded: Bool + /// The audio renderer reports `.failed`. + var audioRendererFailed: Bool + /// The session plays and the clock rate is non-zero. + var clockRunning: Bool + /// Playhead, seconds; nil while the clock has no valid time. + var playhead: Double? + /// Audio is enqueued beyond the playhead. + var audioAhead: Bool + /// Video is enqueued beyond the playhead; nil without a video lane. + var videoAhead: Bool? + /// The session's seek serial: a seek rebuilds every queue, so any + /// episode in progress is void. + var serial: UInt64 + } + + private(set) var health: AudioHealth = .absent + /// A loss only a new decoder clears. + private var permanent: AudioLossReason? + /// Playhead when audio was first seen missing; nil while it is not. + private var dryFrom: Double? + private var serial: UInt64 = 0 + + /// The audio decoder failed for good. + mutating func decoderFailed() { + permanent = .decoderFailed + health = .lost(.decoderFailed) + } + + /// The audio lane was rebuilt or removed (track switch, audio off/on): + /// whatever was wrong with the old one is gone with it. + mutating func laneReplaced() { + permanent = nil + dryFrom = nil + health = .healthy + } + + /// Updates `health`; returns the reason when a starvation episode begins, + /// which is the moment to tell the host. + mutating func observe(_ observation: Observation) -> AudioLossReason? { + guard observation.hasAudioLane else { + permanent = nil + dryFrom = nil + health = .absent + return nil + } + if health == .absent { + health = .healthy + } + if observation.audioRendererFailed { + permanent = .rendererFailed + } + if let permanent { + dryFrom = nil + health = .lost(permanent) + return nil + } + if observation.serial != serial { + serial = observation.serial + dryFrom = nil + health = .healthy + } + if observation.audioEnded || observation.audioAhead { + dryFrom = nil + health = .healthy + return nil + } + // Not judged: paused, buffering or seeking (the clock does not run), + // no time yet, or no healthy video to compare against. A standing + // verdict stays until audio is seen ahead again. + guard observation.clockRunning, + let playhead = observation.playhead, + observation.videoAhead == true + else { + dryFrom = nil + return nil + } + guard let start = dryFrom, playhead >= start else { + dryFrom = playhead + return nil + } + guard playhead - start >= Self.threshold, health != .lost(.starved) else { return nil } + health = .lost(.starved) + return .starved + } +} diff --git a/Sources/LumeEngineCore/Session/PlayerSession+DisplayCriteria.swift b/Sources/LumeEngineCore/Session/PlayerSession+DisplayCriteria.swift new file mode 100644 index 0000000..62e8f63 --- /dev/null +++ b/Sources/LumeEngineCore/Session/PlayerSession+DisplayCriteria.swift @@ -0,0 +1,30 @@ +extension PlayerSession { + /// The display mode the selected video wants — refresh rate and dynamic + /// range of what the engine outputs, after deinterlacing and Dolby Vision + /// conversion. Predicted from the stream once it is open + /// (`basis == .stream`), confirmed from the enqueued frames once enough + /// have gone to the renderer (`.output`). Nil without a video lane. + /// + /// Cheap (a lock and a sort of a few dozen intervals); poll it at a low + /// rate. tvOS applies it with `DisplayCriteriaMatcher`. + public var displayCriteria: ContentDisplayCriteria? { + guard let info = mediaInfo, + let index = selectedVideoTrackIndex, + let track = info.tracks.first(where: { $0.index == index }), + let stream = ContentDisplayCriteria.Stream(track: track) + else { return nil } + let isDeinterlacing = activeVideoDecoder?.isDeinterlacing ?? false + let output = renderer.videoOutput.map { + ContentDisplayCriteria.Output( + formatDescription: $0.format, + frameInterval: $0.frameInterval, + isDeinterlacing: isDeinterlacing + ) + } + return ContentDisplayCriteria.resolve( + stream: stream, + deinterlacing: configuration.deinterlace, + output: output + ) + } +} diff --git a/Sources/LumeEngineCore/Session/PlayerSession+Tracks.swift b/Sources/LumeEngineCore/Session/PlayerSession+Tracks.swift index 8ea401f..7961e93 100644 --- a/Sources/LumeEngineCore/Session/PlayerSession+Tracks.swift +++ b/Sources/LumeEngineCore/Session/PlayerSession+Tracks.swift @@ -52,8 +52,15 @@ extension PlayerSession { } /// Selects an embedded subtitle track, or `nil` to disable subtitles. + /// `TrackInfo.embeddedClosedCaptionsIndex` selects the captions carried in + /// the video stream — also before they have turned up in `mediaInfo`, so + /// a host can keep captions on across channels. public func selectSubtitleTrack(_ index: Int32?) { teardownSubtitleLane() + if index == TrackInfo.embeddedClosedCaptionsIndex { + attachClosedCaptionLane() + return + } guard let index, let demuxer = activeDemuxer, let parameters = demuxer.codecParameters(forStream: index) @@ -83,6 +90,33 @@ extension PlayerSession { } } + /// Feeds the captions the video decoder finds (see `ClosedCaptionTap`) + /// to a caption decoder writing into the cue store. + /// + /// No backfill seek, unlike a subtitle stream: the captions ride in video + /// frames that are decoded a fraction of a second ahead of the playhead, + /// so nothing worth recovering has been read past — and live sources, + /// where captions mostly are, do not survive a seek. + private func attachClosedCaptionLane() { + guard let video = activeVideoDecoder, + let parameters = CodecParameters.closedCaptions() + else { return } + // One payload per frame: a few seconds of slack for a decoder that + // falls behind, beyond which captions are dropped, never frames. + let packets = Channel(capacity: 256) + let decoder = SubtitleDecoder( + parameters: parameters, + input: packets, + store: subtitles, + // CC1/CC2. FFmpeg would otherwise lock onto whichever field's + // data comes first, which can be field 2's metadata. + codecOptions: ["data_field": "first"] + ) + installSubtitleLane(trackIndex: TrackInfo.embeddedClosedCaptionsIndex, packets: packets, decoder: decoder) + decoder.start() + video.captionTap.attach(packets) + } + /// Loads a sidecar subtitle file (SRT/VTT/ASS/…) into the session's cue /// store, replacing any embedded selection. Resolves once the file is /// fully parsed. diff --git a/Sources/LumeEngineCore/Session/PlayerSession.swift b/Sources/LumeEngineCore/Session/PlayerSession.swift index 40d1593..7c688c5 100644 --- a/Sources/LumeEngineCore/Session/PlayerSession.swift +++ b/Sources/LumeEngineCore/Session/PlayerSession.swift @@ -70,6 +70,11 @@ public struct PlayerConfiguration: Sendable { public var autoEnableForcedSubtitlesForForeignAudio = false /// Seconds without clock progress (while expected) before `.stalled` fires. public var stallThreshold: Double = 8 + /// The output route the audio is fitted to. nil follows the shared audio + /// session through route changes (iOS/tvOS/visionOS; stereo on macOS). + /// Pass one to pin the width or drive it yourself, e.g. for a macOS + /// multichannel interface. + public var audioOutputRoute: AudioOutputRoute? public init() {} } @@ -96,7 +101,7 @@ public actor PlayerSession { public nonisolated let events: AsyncStream private let eventSink: AsyncStream.Continuation - private let configuration: PlayerConfiguration + let configuration: PlayerConfiguration /// The presentation backend; expose `renderer.displayLayer` in a view. public nonisolated let renderer: SystemRenderer /// Active subtitle cues (embedded track or external file). Query with @@ -151,6 +156,9 @@ public actor PlayerSession { /// The stage that stopped at the most recent `.stalled`; nil before any. public private(set) var lastStallReason: StallReason? + /// `systemUptime` until which the clock may not start (see `holdStart`). + private var startHoldDeadline: TimeInterval? + // Startup timeline, `systemUptime` stamps (see `startup`). private var openStartedAt: TimeInterval? private var openedAt: TimeInterval? @@ -164,9 +172,20 @@ public actor PlayerSession { public private(set) var selectedSubtitleTrackIndex: Int32? public private(set) var usingExternalSubtitles = false + /// Width the audio lane is fitted to; shared with its decoder and the renderer. + private let audioRoute: AudioOutputRoute + private var audioLoss = AudioLossDetector() + + /// Whether audio is reaching the viewer (see `AudioHealth`). + public var audioHealth: AudioHealth { audioLoss.health } + + /// The layout the audio lane hands the renderer; nil without audio. + public var audioOutput: AudioOutputLayout? { audioDecoder?.currentOutput } + public init(configuration: PlayerConfiguration = PlayerConfiguration()) { self.configuration = configuration - renderer = SystemRenderer(muted: configuration.muted) + audioRoute = configuration.audioOutputRoute ?? AudioOutputRoute.system() + renderer = SystemRenderer(muted: configuration.muted, audioOutputRoute: audioRoute) var continuation: AsyncStream.Continuation! events = AsyncStream(bufferingPolicy: .unbounded) { continuation = $0 } eventSink = continuation @@ -268,7 +287,7 @@ public actor PlayerSession { let packets = makeAudioPacketChannel() let frames = Channel(capacity: configuration.audioQueueDepth, measure: { $0.duration }) demuxer.attach(channel: packets, toStream: track.index) - let decoder = AudioDecoder(parameters: parameters, input: packets, output: frames) + let decoder = AudioDecoder(parameters: parameters, input: packets, output: frames, route: audioRoute) audioPackets = packets audioFrames = frames audioDecoder = decoder @@ -376,7 +395,15 @@ public actor PlayerSession { info?.duration.map(MediaTime.seconds) } - public var mediaInfo: MediaInfo? { info } + /// What `open` returned, plus tracks found since: the closed-caption track + /// (`TrackInfo.embeddedClosedCaptionsIndex`) appears once captions turn up + /// in the video, which no open-time probe can tell. Re-read it to offer + /// that track; `.opened` carries the open-time value only. + public var mediaInfo: MediaInfo? { + guard let info else { return nil } + guard videoDecoder?.captionTap.hasDetectedCaptions == true else { return info } + return info.adding(.embeddedClosedCaptions) + } /// How far startup has got and how long each step took. public var startup: StartupTimeline { @@ -423,6 +450,14 @@ public actor PlayerSession { public let replayedPackets: Int public let startup: StartupTimeline public let lastStallReason: StallReason? + /// The display mode the video wants (see `displayCriteria`). + public let displayCriteria: ContentDisplayCriteria? + /// Whether audio is reaching the viewer. + public let audioHealth: AudioHealth + /// Source → output channels of the audio lane; nil without one. + public let audioOutput: AudioOutputLayout? + /// The output route the audio is fitted to. + public let audioRoute: AudioOutputRoute.Snapshot public var description: String { func lane(_ queue: (count: Int, seconds: Double)?, lead: Double, first: Double) -> String { @@ -448,6 +483,9 @@ public actor PlayerSession { timelineSeams, replayedPackets ) if let lastStallReason { line += " stall=\(lastStallReason.rawValue)" } + if let displayCriteria { line += " display=\(displayCriteria)" } + line += " aout=\(audioOutput.map(String.init(describing:)) ?? "off") route=\(audioRoute)" + if case .lost = audioHealth { line += " audio=\(audioHealth)" } line += " startup=[\(startup)]" return line } @@ -483,10 +521,42 @@ public actor PlayerSession { timelineSeams: demuxer?.timelineSeams ?? 0, replayedPackets: demuxer?.replayedPackets ?? 0, startup: startup, - lastStallReason: lastStallReason + lastStallReason: lastStallReason, + displayCriteria: displayCriteria, + audioHealth: audioHealth, + audioOutput: audioOutput, + audioRoute: audioRoute.snapshot ) } + // MARK: Start hold + + /// Keeps the clock from starting — the session buffers and stays in + /// `.buffering` — until `releaseStartHold()` or until `seconds` have + /// passed, whichever comes first. For a host that has to wait for + /// something outside the engine before the first frame is worth showing: + /// a TV switching display mode blanks for seconds, and playback started + /// into that blank loses its opening (see `DisplayCriteriaMatcher`). + /// Bounded here, not by the caller, so a host that never releases cannot + /// wedge the session. Ignored once playing: a running clock is not + /// stopped, and a later rebuffer must not wait on a stale hold. + public func holdStart(for seconds: Double) { + guard seconds > 0, state != .playing else { return } + startHoldDeadline = ProcessInfo.processInfo.systemUptime + min(seconds, 30) + } + + /// Lets a held start go ahead (see `holdStart`). Idempotent. + public func releaseStartHold() { + guard startHoldDeadline != nil else { return } + startHoldDeadline = nil + evaluatePlayback() + } + + public var isStartHeld: Bool { + guard let deadline = startHoldDeadline else { return false } + return ProcessInfo.processInfo.systemUptime < deadline + } + // MARK: Seek /// Seeks to `position` seconds (media-relative). Flush order matters: @@ -517,6 +587,7 @@ public actor PlayerSession { demuxAtEOF = false videoAtEOF = false audioAtEOF = false + subtitles.closeOpenCues(at: target) demuxer.seek(to: target) // Completion continues in handleDemux(.didSeek). } @@ -565,6 +636,7 @@ public actor PlayerSession { // MARK: Track-lane internals (used by PlayerSession+Tracks.swift) var activeDemuxer: Demuxer? { demuxer } + var activeVideoDecoder: VideoDecoder? { videoDecoder } /// Packet channels are budgeted in media time (`packetReadAhead`); the /// element counts are memory backstops for streams that report no packet @@ -629,6 +701,7 @@ public actor PlayerSession { audioFrames = nil selectedAudioTrackIndex = nil audioAtEOF = false + audioLoss.laneReplaced() renderer.attach(video: videoFrames, audio: nil) } @@ -642,11 +715,12 @@ public actor PlayerSession { let packets = makeAudioPacketChannel() let frames = Channel(capacity: configuration.audioQueueDepth, measure: { $0.duration }) demuxer.attach(channel: packets, toStream: trackIndex) - let decoder = AudioDecoder(parameters: parameters, input: packets, output: frames) + let decoder = AudioDecoder(parameters: parameters, input: packets, output: frames, route: audioRoute) audioPackets = packets audioFrames = frames audioDecoder = decoder audioAtEOF = false + audioLoss.laneReplaced() selectedAudioTrackIndex = trackIndex let pump = Task { [events = decoder.events] in for await event in events { @@ -662,6 +736,7 @@ public actor PlayerSession { if let old = selectedSubtitleTrackIndex { demuxer?.detach(streamIndex: old) } + videoDecoder?.captionTap.attach(nil) if let retired = subtitleDecoder { retired.requestStop() // no cue of the old track lands after `removeAll` below retire { retired.shutdown(deadline: 2) } @@ -804,7 +879,12 @@ public actor PlayerSession { // One lane dying is not necessarily fatal (e.g. broken audio track, // healthy video) — surface it; the monitor decides on full failure. eventSink.yield(.error(error)) - if isVideo { videoAtEOF = true } else { audioAtEOF = true } + if isVideo { + videoAtEOF = true + } else { + audioAtEOF = true + audioLoss.decoderFailed() + } lastError = error } } @@ -819,9 +899,36 @@ public actor PlayerSession { evaluatePlayback() runStallWatchdog() noteFirstFrameShown() + evaluateAudioHealth() } } + /// Audio that stops while the picture plays on is a failure the viewer + /// notices and nothing else would report (see `AudioLossDetector` for + /// what does not count). + private func evaluateAudioHealth() { + let playhead = renderer.currentTime + let highWater = renderer.enqueuedHighWaterMark + func ahead(_ mark: Int64) -> Bool { + MediaTime.isValid(mark) && MediaTime.isValid(playhead) && mark > playhead + } + let observation = AudioLossDetector.Observation( + hasAudioLane: audioDecoder != nil, + audioEnded: audioAtEOF, + audioRendererFailed: renderer.audioRendererFailed, + clockRunning: state == .playing && renderer.rate > 0, + playhead: MediaTime.isValid(playhead) ? MediaTime.seconds(playhead) : nil, + audioAhead: ahead(highWater.audio), + videoAhead: videoDecoder != nil ? ahead(highWater.video) : nil, + serial: currentSerial + ) + guard audioLoss.observe(observation) == .starved else { return } + eventSink.yield(.error(EngineError( + code: .audioLost, + message: "audio stopped reaching the renderer while video plays" + ))) + } + /// A container seek can "succeed" while landing somewhere else entirely — /// matroska falls back to the earliest cluster when its cues are /// unreachable (end-of-file cues over non-range HTTP is the classic IPTV @@ -878,6 +985,7 @@ public actor PlayerSession { && (state == .playing || state == .buffering) && pendingSeekTarget == nil && !demuxAtEOF + && !isStartHeld // buffers fill and stop growing while held guard expectingProgress else { lastProgressDate = .distantFuture @@ -1010,6 +1118,11 @@ public actor PlayerSession { switch state { case .ready, .paused, .buffering: + if isStartHeld { + if state != .buffering { state = .buffering } + return + } + startHoldDeadline = nil // Start when every lane is as buffered as it can usefully get, // or when EOF means no more is coming. if allLanesReady || allEOF || demuxAtEOF { diff --git a/Sources/LumeEngineCore/Subtitle/ClosedCaptions.swift b/Sources/LumeEngineCore/Subtitle/ClosedCaptions.swift new file mode 100644 index 0000000..d35af1c --- /dev/null +++ b/Sources/LumeEngineCore/Subtitle/ClosedCaptions.swift @@ -0,0 +1,130 @@ +internal import CFFmpeg +import Foundation +import Synchronization + +/// Closed captions carried inside the video stream. +/// +/// US broadcast (ATSC) and much IPTV ship captions not as a subtitle stream +/// but as A/53 user data in the video itself — SEI messages in H.264/HEVC, +/// picture user data in MPEG-2 — which FFmpeg's video decoders hand out as +/// `AV_FRAME_DATA_A53_CC` side data on the decoded frame: a run of 3-byte +/// `cc_data` triples. The tap lifts those payloads off the video lane and +/// feeds them, as packets stamped with the frame's PTS, to an ordinary +/// `SubtitleDecoder` running FFmpeg's `ccaption` decoder. That keeps captions +/// on the video's own timestamps (engine µs, display order), and keeps +/// caption decoding — and its failures — off the video thread. +/// +/// The video thread never waits on it: payloads go out with `trySend`, and a +/// caption lane that falls behind loses captions, never frames. +final class ClosedCaptionTap: Sendable { + private struct State { + var detected = false + var sink: Channel? + } + + private let state = Mutex(State()) + + /// True once the video carried caption data with content (null padding + /// alone does not count — broadcast encoders send it on every frame). + var hasDetectedCaptions: Bool { + state.withLock { $0.detected } + } + + /// Starts (or with `nil` stops) forwarding caption payloads. + func attach(_ sink: Channel?) { + state.withLock { $0.sink = sink } + } + + /// Video decode thread: called with every decoded frame, before any + /// filtering. A frame without caption data costs one side-data lookup. + func offer(_ frame: UnsafeMutablePointer, serial: UInt64) { + guard let sideData = av_frame_get_side_data(frame, AV_FRAME_DATA_A53_CC), + let data = sideData.pointee.data, + sideData.pointee.size > 0 + else { return } + let payload = UnsafeRawBufferPointer(start: data, count: Int(sideData.pointee.size)) + + let (detected, sink) = state.withLock { ($0.detected, $0.sink) } + if !detected, ClosedCaptionData.carriesCaptions(payload) { + state.withLock { $0.detected = true } + } + + let pts = frame.pointee.best_effort_timestamp + guard let sink, MediaTime.isValid(pts), + let packet = Packet(copying: payload, pts: pts, serial: serial) + else { return } + _ = sink.trySend(packet) + } +} + +/// Reading A/53 `cc_data` (CEA-708 §4.4: `cc_valid`, `cc_type`, two data +/// bytes per triple). +enum ClosedCaptionData { + /// Whether the payload holds caption data for CC1/CC2 (CEA-608 field 1) + /// beyond null padding — what decides that a stream has captions to + /// offer. Field 2 (CC3/CC4, mostly XDS metadata) and DTVCC (708) packets + /// do not count: the caption decoder shows field 1 only. + static func carriesCaptions(_ payload: UnsafeRawBufferPointer) -> Bool { + var offset = 0 + while offset + 3 <= payload.count { + let header = payload[offset] + let valid = header & 0x04 != 0 + let fieldOne = header & 0x03 == 0 + if valid, fieldOne, payload[offset + 1] & 0x7F != 0 || payload[offset + 2] & 0x7F != 0 { + return true + } + offset += 3 + } + return false + } +} + +extension TrackInfo { + /// The synthetic track for captions found in the video stream. + static let embeddedClosedCaptions = TrackInfo( + index: embeddedClosedCaptionsIndex, + kind: .subtitle, + codecName: String(cString: avcodec_get_name(AV_CODEC_ID_EIA_608)), + codecID: AV_CODEC_ID_EIA_608.rawValue, + language: nil, + title: "CC1", + isDefault: false, + isForced: false, + bitrate: 0, + video: nil, + audio: nil, + wrapBits: 64 + ) +} + +extension CodecParameters { + /// Parameters for the caption lane: no container stream describes it. + static func closedCaptions() -> CodecParameters? { + guard let scratch = avcodec_parameters_alloc() else { return nil } + defer { + var pointer: UnsafeMutablePointer? = scratch + avcodec_parameters_free(&pointer) + } + scratch.pointee.codec_type = AVMEDIA_TYPE_SUBTITLE + scratch.pointee.codec_id = AV_CODEC_ID_EIA_608 + return CodecParameters(copying: scratch) + } +} + +extension Packet { + /// A packet owning a copy of `bytes`, on the engine timeline already. + convenience init?(copying bytes: UnsafeRawBufferPointer, pts: Int64, serial: UInt64) { + guard let base = bytes.baseAddress, !bytes.isEmpty, + let raw = av_packet_alloc() + else { return nil } + guard av_new_packet(raw, Int32(bytes.count)) >= 0, let destination = raw.pointee.data else { + var pointer: UnsafeMutablePointer? = raw + av_packet_free(&pointer) + return nil + } + destination.update(from: base.assumingMemoryBound(to: UInt8.self), count: bytes.count) + raw.pointee.pts = pts + raw.pointee.dts = pts + self.init(adopting: raw, pts: pts, dts: pts, duration: 0, serial: serial) + } +} diff --git a/Sources/LumeEngineCore/Subtitle/SubtitleCue.swift b/Sources/LumeEngineCore/Subtitle/SubtitleCue.swift index a262a93..9e4f274 100644 --- a/Sources/LumeEngineCore/Subtitle/SubtitleCue.swift +++ b/Sources/LumeEngineCore/Subtitle/SubtitleCue.swift @@ -1,4 +1,5 @@ import Foundation +import Synchronization /// One subtitle cue on the engine timeline. Text-only for now; bitmap cues /// (PGS/DVB) and full libass styling land with the subtitle pack (PLAN.md P5+). @@ -19,75 +20,47 @@ public struct SubtitleCue: Sendable, Equatable, Identifiable { } /// Time-indexed cue container, safe for concurrent insert (decoder thread) and -/// query (UI tick). Cues are kept sorted by start time. +/// query (UI tick). The lifecycle policy — how long a cue shows, what ends it, +/// how many are kept — is `SubtitleTimeline`'s. /// /// Cues outlive seeks on purpose — the store is a timeline, not a queue — so /// the same packet reaches it again whenever the demuxer re-reads a range it /// already delivered: a backward seek, or the seek that backfills a subtitle /// track selected after the demuxer read past its packets. An identical cue -/// (same start, end, and text) is therefore dropped, or every re-read would -/// show its lines doubled. -public final class SubtitleStore: @unchecked Sendable { - private let lock = NSLock() - private var cues: [SubtitleCue] = [] - private var nextID: UInt64 = 0 +/// is therefore dropped, or every re-read would show its lines doubled. +public final class SubtitleStore: Sendable { + private let timeline = Mutex(SubtitleTimeline()) public init() {} + /// A text cue with a known end (tests, and the shape every text format + /// decodes to). func insert(start: Int64, end: Int64, text: String) { let trimmed = text.trimmingCharacters(in: .whitespacesAndNewlines) guard !trimmed.isEmpty, end > start else { return } - lock.lock() - defer { lock.unlock() } - // Insertion point by start time (cues arrive nearly sorted). - var index = cues.count - while index > 0 && cues[index - 1].start > start { index -= 1 } - // Cues with the same start sit directly before the insertion point. - var existing = index - 1 - while existing >= 0 && cues[existing].start == start { - if cues[existing].end == end && cues[existing].text == trimmed { return } - existing -= 1 - } - cues.insert(SubtitleCue(id: nextID, start: start, end: end, text: trimmed), at: index) - nextID += 1 + apply(SubtitleEvent(start: start, end: end, text: trimmed, replacesPrevious: false)) + } + + func apply(_ event: SubtitleEvent) { + timeline.withLock { $0.apply(event) } + } + + /// Ends cues still open at `time`: the session seeked there, and what the + /// source shows after it is unknown until its next update. + func closeOpenCues(at time: Int64) { + timeline.withLock { $0.closeOpenCues(at: time) } } /// All cues covering `time` (engine µs). public func activeCues(at time: Int64) -> [SubtitleCue] { - lock.lock() - defer { lock.unlock() } - // Binary search for the first cue starting after `time`, then walk back. - // Overlapping cues are rare and short-lived; a bounded backward scan is - // simpler than an interval tree and plenty fast. - var low = 0 - var high = cues.count - while low < high { - let mid = (low + high) / 2 - if cues[mid].start <= time { low = mid + 1 } else { high = mid } - } - var active: [SubtitleCue] = [] - var index = low - 1 - var scanned = 0 - while index >= 0 && scanned < 64 { - let cue = cues[index] - if cue.start <= time && time < cue.end { - active.append(cue) - } - index -= 1 - scanned += 1 - } - return active.sorted { $0.start < $1.start } + timeline.withLock { $0.activeCues(at: time) } } public var count: Int { - lock.lock() - defer { lock.unlock() } - return cues.count + timeline.withLock { $0.count } } public func removeAll() { - lock.lock() - defer { lock.unlock() } - cues.removeAll() + timeline.withLock { $0.removeAll() } } } diff --git a/Sources/LumeEngineCore/Subtitle/SubtitleDecoder.swift b/Sources/LumeEngineCore/Subtitle/SubtitleDecoder.swift index c8ab793..4878e26 100644 --- a/Sources/LumeEngineCore/Subtitle/SubtitleDecoder.swift +++ b/Sources/LumeEngineCore/Subtitle/SubtitleDecoder.swift @@ -3,13 +3,24 @@ import Foundation /// Text subtitle decode stage: `Channel` in, `SubtitleStore` out. /// Runs `avcodec_decode_subtitle2`, so every FFmpeg text subtitle codec -/// (subrip, ass, webvtt, mov_text, eia_608 CC) feeds one cue model. +/// (subrip, ass, webvtt, mov_text, eia_608 CC) feeds one cue model — and +/// so does the closed-caption lane, whose packets come from the video decoder +/// (`ClosedCaptionTap`) rather than the demuxer. Each decoded update goes to +/// the store as a `SubtitleEvent`; how long it shows is the store's policy. /// Bitmap codecs (PGS/DVB/VobSub) are recognized but skipped for now. public final class SubtitleDecoder: @unchecked Sendable { public let store: SubtitleStore private let parameters: CodecParameters private let input: Channel + private let codecOptions: [String: String] + /// Each update is the whole screen (closed captions, bitmap formats) — + /// see `SubtitleEvent.replacesPrevious`. + private let replacesPrevious: Bool + /// CEA-608 captions run in FFmpeg's real-time mode (below), which stamps + /// an update with the time of the *previous* one; the packet that caused + /// it carries the time it actually shows. + private let isClosedCaptions: Bool private let lock = NSCondition() private var started = false @@ -17,11 +28,39 @@ public final class SubtitleDecoder: @unchecked Sendable { private var stopRequested = false private var codecContext: UnsafeMutablePointer? + /// Decode-thread-only: seek generation of the last packet. + private var currentSerial: UInt64? - public init(parameters: CodecParameters, input: Channel, store: SubtitleStore = SubtitleStore()) { + public convenience init(parameters: CodecParameters, input: Channel, store: SubtitleStore = SubtitleStore()) { + self.init(parameters: parameters, input: input, store: store, codecOptions: [:]) + } + + /// `codecOptions` go to `avcodec_open2` (the caption lane picks its field). + init(parameters: CodecParameters, input: Channel, store: SubtitleStore, codecOptions: [String: String]) { self.parameters = parameters self.input = input self.store = store + let codecID = parameters.raw.pointee.codec_id + isClosedCaptions = codecID == AV_CODEC_ID_EIA_608 + replacesPrevious = Self.replacesPrevious(codecID) + var options = codecOptions + if isClosedCaptions, options["real_time"] == nil { + // Real-time mode emits each screen change as it is decoded, open + // ended, and a clear as an empty update. The default buffered mode + // emits a caption only once the next change has ended it — after + // its own duration, by which time playback (a fraction of a second + // behind decode) has already passed most of it. + options["real_time"] = "1" + } + self.codecOptions = options + } + + /// Formats where every update describes the whole screen and stays until + /// the next one: closed captions, teletext, and the bitmap formats. + static func replacesPrevious(_ codecID: AVCodecID) -> Bool { + if codecID == AV_CODEC_ID_EIA_608 || codecID == AV_CODEC_ID_DVB_TELETEXT { return true } + guard let descriptor = avcodec_descriptor_get(codecID) else { return false } + return descriptor.pointee.props & AV_CODEC_PROP_BITMAP_SUB != 0 } public func start() { @@ -122,19 +161,35 @@ public final class SubtitleDecoder: @unchecked Sendable { private func decode(packet: Packet) { guard let context = codecContext else { return } + // Seek boundary: decoders with screen state (caption roll-up rows, + // PGS/DVB object and palette sets) must not carry it across. + if let serial = currentSerial, serial != packet.serial { + avcodec_flush_buffers(context) + } + currentSerial = packet.serial + var subtitle = AVSubtitle() var gotSubtitle: Int32 = 0 let result = avcodec_decode_subtitle2(context, &subtitle, &gotSubtitle, packet.raw) guard result >= 0, gotSubtitle != 0 else { return } defer { avsubtitle_free(&subtitle) } - // Packet timestamps are engine µs; display offsets are milliseconds. - let basePTS = MediaTime.isValid(packet.pts) ? packet.pts : 0 - let start = basePTS + Int64(subtitle.start_display_time) * 1_000 - var end = basePTS + Int64(subtitle.end_display_time) * 1_000 - if subtitle.end_display_time == 0 || end <= start { - end = packet.duration > 0 ? basePTS + packet.duration : start + 3_000_000 + // Packet timestamps are engine µs, and so is `subtitle.pts` + // (`pkt_timebase` is µs; PGS takes it from its presentation segment); + // display offsets are milliseconds. + let base: Int64 + if !isClosedCaptions, MediaTime.isValid(subtitle.pts) { + base = subtitle.pts + } else { + base = MediaTime.isValid(packet.pts) ? packet.pts : 0 } + let span = SubtitleEvent.span( + base: base, + startDisplay: isClosedCaptions ? 0 : subtitle.start_display_time, + endDisplay: subtitle.end_display_time, + packetDuration: packet.duration, + replacesPrevious: replacesPrevious + ) var lines: [String] = [] for i in 0..= 0 else { + var options: OpaquePointer? + for (key, value) in codecOptions { + av_dict_set(&options, key, value, 0) + } + defer { av_dict_free(&options) } + guard avcodec_open2(context, codec, &options) >= 0 else { var pointer: UnsafeMutablePointer? = context avcodec_free_context(&pointer) return false diff --git a/Sources/LumeEngineCore/Subtitle/SubtitleTimeline.swift b/Sources/LumeEngineCore/Subtitle/SubtitleTimeline.swift new file mode 100644 index 0000000..d3f8381 --- /dev/null +++ b/Sources/LumeEngineCore/Subtitle/SubtitleTimeline.swift @@ -0,0 +1,226 @@ +import Foundation + +/// One display update from a subtitle decoder on the engine timeline (µs), +/// before the timeline decides how long it stays up. +struct SubtitleEvent: Sendable, Equatable { + var start: Int64 + /// `nil` when the source gives no end. DVB, PGS and closed captions say + /// exactly that — "until the next update" — and a decoder that turned it + /// into a number would leave the text up for 49 days (`UINT32_MAX` ms). + var end: Int64? + /// Display text, trimmed. Empty on a replacing track means "clear the + /// screen", which is how those formats end what is showing. + var text: String + /// The track describes the whole screen with every update (closed + /// captions, DVB, PGS, DVD): an update ends everything shown before it. + /// Text formats (SRT, ASS, WebVTT, mov_text) overlap freely instead, each + /// cue carrying its own end. + var replacesPrevious: Bool + + /// Where an `AVSubtitle` sits on the engine timeline. `base` is the + /// subtitle's PTS in engine µs, display times are milliseconds after it. + /// `nil` end means open: shows until replaced (see `SubtitleTimeline`). + static func span( + base: Int64, + startDisplay: UInt32, + endDisplay: UInt32, + packetDuration: Int64, + replacesPrevious: Bool + ) -> (start: Int64, end: Int64?) { + let start = base + Int64(startDisplay) * 1_000 + // UINT32_MAX is FFmpeg's "no end" (PGS always, closed captions in + // real-time mode, DVB without a computed end). + guard endDisplay != .max else { return (start, nil) } + let end = base + Int64(endDisplay) * 1_000 + if endDisplay != 0, end > start { return (start, end) } + // No usable end. A replacing track's update lasts until the next one; + // a text cue without one gets its packet's duration, or a reading-time + // guess, as it always has. + if replacesPrevious { return (start, nil) } + return (start, packetDuration > 0 ? base + packetDuration : start + 3_000_000) + } +} + +/// The cue-lifecycle policy behind `SubtitleStore`: a timeline of cues sorted +/// by start, deciding how long each one shows. Pure value type — the store +/// adds the lock. +/// +/// The failure modes it exists for, all of which used to reach the screen: +/// - Cues without an end (PGS, closed captions, DVB "until the next page") +/// stayed up for good, piling onto every later cue. +/// - The empty update that DVB, PGS and closed captions send to clear the +/// screen was dropped as "no text", so the previous cue stayed for its full +/// timeout (DVB pages time out after tens of seconds). +/// - Every cue a live session ever decoded was kept: a day of roll-up +/// captions is hundreds of thousands of them. +/// +/// The rules: +/// - An open cue ends where the next update of its track starts, or after +/// `openEndedLimit` when none comes. +/// - On a replacing track every update — cue or clear — ends everything +/// shown before it, and when one arrives out of order (a range re-read +/// after a seek) the update already stored after it bounds it in turn. +/// - Text tracks keep their overlaps; only their open cues are ended by the +/// next cue. +/// - An update already held is dropped, so re-reading a range (backward +/// seek, backfill seek) does not double its lines. On a replacing track a +/// different update at the same start replaces the stored one: the screen +/// at one instant has one state, and the latest decode of it wins. +/// - At most `capacity` cues are kept; the ones farthest from where cues are +/// being added go first. +struct SubtitleTimeline: Sendable { + /// How long an open cue stays without a successor. Caption decoders on + /// televisions erase a caption left standing by about this long without + /// caption data; a longer limit leaves stale text over the next scene + /// when a clear is lost, a shorter one cuts captions that are still + /// being spoken. + static let openEndedLimit: Int64 = 16_000_000 + + /// Cues kept. Far above any film's subtitle file (a few thousand cues); + /// for a live track it is half an hour or more of roll-up captions. + static let capacity = 8_192 + + /// How far back from an insertion point active cues are looked for. + /// Overlap is rare and short-lived; a replacing track has at most one. + private static let scanLimit = 64 + + private struct Entry: Sendable { + var cue: SubtitleCue + /// The end is the limit, not the source's: the next update or a seek + /// may still bound it. + var provisional: Bool + } + + private var entries: [Entry] = [] + private var nextID: UInt64 = 0 + + var count: Int { entries.count } + + mutating func apply(_ event: SubtitleEvent) { + if event.text.isEmpty { + // A clear. Text tracks have no such thing; an empty text cue is + // just nothing to show. + guard event.replacesPrevious else { return } + end(at: event.start, onlyProvisional: false) + return + } + + if isDuplicate(event) { return } + if event.replacesPrevious { removeEntries(startingAt: event.start) } + // What this update takes over from: everything on screen for a + // replacing track, only the cues with no end of their own otherwise. + end(at: event.start, onlyProvisional: !event.replacesPrevious) + + let index = insertionIndex(for: event.start) + var end = event.end ?? event.start + Self.openEndedLimit + if (event.replacesPrevious || event.end == nil), index < entries.count { + // Arrived out of order: the next update already stored ends it. + end = min(end, entries[index].cue.start) + } + guard end > event.start else { return } + + entries.insert( + Entry( + cue: SubtitleCue(id: nextID, start: event.start, end: end, text: event.text), + provisional: event.end == nil + ), + at: index + ) + nextID += 1 + trim(keeping: index) + } + + /// Ends open cues at a discontinuity (a seek to `time`): the screen state + /// after it is unknown until the source sends its next update, and + /// showing the stale one meanwhile is worse than showing nothing. Cues + /// with an end from the source are facts about the timeline and stay. + mutating func closeOpenCues(at time: Int64) { + end(at: time, onlyProvisional: true) + } + + mutating func removeAll() { + entries.removeAll() + } + + /// All cues covering `time` (engine µs), ordered by start. + func activeCues(at time: Int64) -> [SubtitleCue] { + var active: [SubtitleCue] = [] + var index = insertionIndex(for: time) - 1 + var scanned = 0 + while index >= 0 && scanned < Self.scanLimit { + let cue = entries[index].cue + if cue.start <= time && time < cue.end { + active.append(cue) + } + index -= 1 + scanned += 1 + } + return active.reversed() + } + + // MARK: Internals + + /// First entry starting after `time`. + private func insertionIndex(for time: Int64) -> Int { + var low = 0 + var high = entries.count + while low < high { + let mid = (low + high) / 2 + if entries[mid].cue.start <= time { low = mid + 1 } else { high = mid } + } + return low + } + + private func isDuplicate(_ event: SubtitleEvent) -> Bool { + var index = insertionIndex(for: event.start) - 1 + while index >= 0 && entries[index].cue.start == event.start { + let entry = entries[index] + if entry.cue.text == event.text { + // A stored end may have been bounded by a successor since, so + // it only has to match where both ends came from the source. + if event.replacesPrevious || event.end == nil || entry.provisional { return true } + if entry.cue.end == event.end { return true } + } + index -= 1 + } + return false + } + + private mutating func removeEntries(startingAt start: Int64) { + var index = insertionIndex(for: start) - 1 + while index >= 0 && entries[index].cue.start == start { + entries.remove(at: index) + index -= 1 + } + } + + /// Ends the cues that started before `time` and still show at it. + private mutating func end(at time: Int64, onlyProvisional: Bool) { + var index = insertionIndex(for: time) - 1 + var scanned = 0 + while index >= 0 && scanned < Self.scanLimit { + let entry = entries[index] + if entry.cue.start < time, entry.cue.end > time, !onlyProvisional || entry.provisional { + entries[index].cue = SubtitleCue( + id: entry.cue.id, start: entry.cue.start, end: time, text: entry.cue.text + ) + } + index -= 1 + scanned += 1 + } + } + + /// Drops cues beyond `capacity` from the end farther from `index`, where + /// cues are being added — the front on a live timeline, the back after a + /// seek far into the past of a long file. Drops an eighth more than + /// needed so the shift is not paid on every insert. + private mutating func trim(keeping index: Int) { + guard entries.count > Self.capacity else { return } + let excess = entries.count - Self.capacity + Self.capacity / 8 + if index < entries.count / 2 { + entries.removeLast(excess) + } else { + entries.removeFirst(excess) + } + } +} diff --git a/Tests/LumeEngineCoreTests/AudioRouteTests.swift b/Tests/LumeEngineCoreTests/AudioRouteTests.swift new file mode 100644 index 0000000..f792888 --- /dev/null +++ b/Tests/LumeEngineCoreTests/AudioRouteTests.swift @@ -0,0 +1,548 @@ +import CFFmpeg +import Foundation +import Synchronization +import Testing +@testable import LumeEngineCore + +/// Output width that follows the route: the layout policy and the downmix +/// matrix as plain values, then the decoder at forced route widths and across +/// a route change mid-stream. +@Suite("AudioLayoutPolicy") +struct AudioLayoutPolicyTests { + private static let surround51Back: SpeakerSet = [.frontLeft, .frontRight, .frontCenter, .lowFrequency, .backLeft, .backRight] + private static let surround70: SpeakerSet = SpeakerSet.surround71.subtracting(.lowFrequency) + /// A matrix-encoded stereo pair (`AV_CH_STEREO_LEFT | AV_CH_STEREO_RIGHT`): + /// two channels the renderer cannot name. + private static let stereoDownmix = SpeakerSet(rawValue: 1 << 29 | 1 << 30) + + private func plan(_ source: SpeakerSet, route: Int) -> AudioLayoutPolicy.Plan { + AudioLayoutPolicy.plan(source: source, routeChannels: route) + } + + @Test("speaker sets are FFmpeg's native layouts") + func layoutsMatchFFmpeg() { + #expect(SpeakerSet.mono.rawValue == layoutMask("mono")) + #expect(SpeakerSet.stereo.rawValue == layoutMask("stereo")) + #expect(SpeakerSet.surround51.rawValue == layoutMask("5.1(side)")) + #expect(SpeakerSet.surround71.rawValue == layoutMask("7.1")) + } + + @Test("multichannel goes out discrete when the route carries it") + func passthroughWhenItFits() { + for route in [6, 8] { + #expect(plan(.surround51, route: route) == .init(output: .surround51, remixes: false)) + #expect(plan(Self.surround51Back, route: route) == .init(output: Self.surround51Back, remixes: false)) + } + #expect(plan(.surround71, route: 8) == .init(output: .surround71, remixes: false)) + for route in [2, 6, 8] { + #expect(plan(.stereo, route: route) == .init(output: .stereo, remixes: false)) + } + } + + @Test("a narrower route gets the widest standard layout both sides fill") + func foldsDownToTheRoute() { + #expect(plan(.surround51, route: 2) == .init(output: .stereo, remixes: true)) + #expect(plan(Self.surround51Back, route: 2) == .init(output: .stereo, remixes: true)) + #expect(plan(.surround71, route: 2) == .init(output: .stereo, remixes: true)) + #expect(plan(.surround71, route: 6) == .init(output: .surround51, remixes: true)) + #expect(plan(.surround71, route: 7) == .init(output: .surround51, remixes: true)) + // Odd widths fall to the standard layout below them. + #expect(plan(.surround51, route: 4) == .init(output: .stereo, remixes: true)) + } + + @Test("no LFE channel is invented") + func noInventedLFE() { + #expect(plan(Self.surround70, route: 6) == .init(output: SpeakerSet.surround51.subtracting(.lowFrequency), remixes: true)) + } + + @Test("mono reaches both speakers of any route wider than one") + func monoBecomesStereo() { + for route in [2, 6, 8] { + #expect(plan(.mono, route: route) == .init(output: .stereo, remixes: true)) + } + #expect(plan(.mono, route: 1) == .init(output: .mono, remixes: false)) + #expect(plan(.mono, route: 0) == .init(output: .mono, remixes: false)) + #expect(plan(.stereo, route: 1) == .init(output: .mono, remixes: true)) + } + + @Test("channels the renderer cannot name are remixed, not passed untagged") + func unnameableChannelsAreRemixed() { + #expect(plan(Self.stereoDownmix, route: 8) == .init(output: .stereo, remixes: true)) + } +} + +@Suite("DownmixMatrix") +struct DownmixMatrixTests { + private let minus3dB = 0.707_106_781_186_547_6 + + /// The coefficient of input speaker `input` in output speaker `output`. + private func weight( + _ matrix: [[Double]], from source: SpeakerSet, to target: SpeakerSet, + _ input: SpeakerSet, into output: SpeakerSet + ) -> Double { + func index(_ speaker: SpeakerSet, in set: SpeakerSet) -> Int { + // Native order is ascending bit order. + (set.rawValue & (speaker.rawValue - 1)).nonzeroBitCount + } + return matrix[index(output, in: target)][index(input, in: source)] + } + + @Test("5.1 to stereo: ITU fold-down, centre in both, LFE omitted, no cross-feed") + func surround51ToStereo() throws { + let source = SpeakerSet.surround51 + let matrix = try #require(DownmixMatrix.coefficients(from: source, to: .stereo)) + #expect(matrix.count == 2 && matrix.allSatisfy { $0.count == 6 }) + func w(_ input: SpeakerSet, _ output: SpeakerSet) -> Double { + weight(matrix, from: source, to: .stereo, input, into: output) + } + #expect(w(.frontLeft, .frontLeft) == 1) + #expect(w(.frontRight, .frontRight) == 1) + #expect(w(.frontLeft, .frontRight) == 0) + #expect(w(.frontRight, .frontLeft) == 0) + #expect(abs(w(.frontCenter, .frontLeft) - minus3dB) < 1e-9) + #expect(abs(w(.frontCenter, .frontRight) - minus3dB) < 1e-9) + #expect(abs(w(.sideLeft, .frontLeft) - minus3dB) < 1e-9) + #expect(abs(w(.sideRight, .frontRight) - minus3dB) < 1e-9) + #expect(w(.sideLeft, .frontRight) == 0) + #expect(w(.lowFrequency, .frontLeft) == 0) + #expect(w(.lowFrequency, .frontRight) == 0) + } + + @Test("7.1 to stereo: every surround folds into its own side") + func surround71ToStereo() throws { + let source = SpeakerSet.surround71 + let matrix = try #require(DownmixMatrix.coefficients(from: source, to: .stereo)) + func w(_ input: SpeakerSet, _ output: SpeakerSet) -> Double { + weight(matrix, from: source, to: .stereo, input, into: output) + } + for (input, side) in [(SpeakerSet.backLeft, SpeakerSet.frontLeft), (.sideLeft, .frontLeft), + (.backRight, .frontRight), (.sideRight, .frontRight)] { + #expect(abs(w(input, side) - minus3dB) < 1e-9) + } + #expect(w(.backLeft, .frontRight) == 0) + #expect(abs(w(.frontCenter, .frontLeft) - minus3dB) < 1e-9) + #expect(abs(w(.frontCenter, .frontRight) - minus3dB) < 1e-9) + } + + @Test("7.1 to 5.1: fronts, centre and LFE pass straight; backs join the sides") + func surround71To51() throws { + let source = SpeakerSet.surround71 + let target = SpeakerSet.surround51 + let matrix = try #require(DownmixMatrix.coefficients(from: source, to: target)) + func w(_ input: SpeakerSet, _ output: SpeakerSet) -> Double { + weight(matrix, from: source, to: target, input, into: output) + } + for speaker in [SpeakerSet.frontLeft, .frontRight, .frontCenter, .lowFrequency] { + #expect(w(speaker, speaker) == 1) + } + #expect(w(.backLeft, .sideLeft) > 0.5) + #expect(w(.backRight, .sideRight) > 0.5) + #expect(w(.frontCenter, .frontLeft) == 0, "a route with a centre speaker keeps dialogue on it") + } + + @Test("mono to stereo: equal power on both sides") + func monoToStereo() throws { + let matrix = try #require(DownmixMatrix.coefficients(from: .mono, to: .stereo)) + #expect(matrix.count == 2) + #expect(abs(matrix[0][0] - minus3dB) < 1e-9) + #expect(abs(matrix[1][0] - minus3dB) < 1e-9) + } + + @Test("dialogue survives every stereo fold-down") + func dialogueNeverDropped() throws { + let sources: [SpeakerSet] = [ + [.frontLeft, .frontRight, .frontCenter], + .surround51, + [.frontLeft, .frontRight, .frontCenter, .lowFrequency, .backLeft, .backRight], + .surround71, + SpeakerSet.surround51.union(SpeakerSet(rawValue: 1 << 8)), // 6.1, back centre + ] + for source in sources { + let plan = AudioLayoutPolicy.plan(source: source, routeChannels: 2) + let matrix = try #require(DownmixMatrix.coefficients(from: source, to: plan.output)) + let center = (source.rawValue & (SpeakerSet.frontCenter.rawValue - 1)).nonzeroBitCount + #expect(matrix[0][center] > 0.5 && matrix[1][center] > 0.5, "centre lost folding \(source.rawValue)") + // Every input channel except the LFE reaches the output. + for input in 0.. 0, "channel \(input) of \(source.rawValue) dropped") + } + } + } +} + +@Suite("AudioRoute", .serialized) +struct AudioRouteDecodeTests { + /// Decodes an audio-only fixture with the decoder following `route`. + /// `onFrame` runs on the drain thread for each frame, in order — the place + /// to change the route mid-stream. + private func decode( + fixture: String, + route: AudioOutputRoute, + frameQueue: Int = 256, + onFrame: @escaping @Sendable (Int) -> Void = { _ in } + ) async throws -> [AudioFrame] { + let demuxer = Demuxer(url: try Fixtures.path(fixture)) + var events = demuxer.events.makeAsyncIterator() + demuxer.start() + guard case .opened(let info)? = await events.next() else { + demuxer.shutdown() + throw EngineError(code: .openFailed, message: "\(fixture) failed to open") + } + defer { demuxer.shutdown() } + + let track = try #require(info.audioTracks.first) + let parameters = try #require(demuxer.codecParameters(forStream: track.index)) + let packets = Channel(capacity: 4096) + let frames = Channel(capacity: frameQueue, measure: { $0.duration }) + demuxer.attach(channel: packets, toStream: track.index) + + let decoder = AudioDecoder(parameters: parameters, input: packets, output: frames, route: route) + var decodeEvents = decoder.events.makeAsyncIterator() + decoder.start() + demuxer.resume() + defer { decoder.shutdown() } + + let collector = ChannelDrain(frames, into: [AudioFrame]()) { collected, frame in + collected.append(frame) + onFrame(collected.count) + } + while let event = await events.next() { + if case .endOfStream = event { + decoder.signalEndOfStream() + break + } + } + guard case .endOfStream? = await decodeEvents.next() else { + throw EngineError(code: .decodeFailed, message: "\(fixture): decoder never reached EOF") + } + decoder.shutdown() + return await collector.value + } + + /// Per-channel peak amplitude over all frames. + private func peaks(_ frames: [AudioFrame]) -> [Float] { + guard let channels = frames.first?.channels else { return [] } + var peaks = [Float](repeating: 0, count: channels) + for frame in frames { + let samples = frame.samples + for index in 0.. 0.05) + let stereoPeaks = peaks(stereo) + #expect(abs(stereoPeaks[0] - stereoPeaks[1]) < 1e-4) + #expect(abs(stereoPeaks[0] / center - 0.7071) < 0.01, "centre at \(stereoPeaks[0] / center) of its level") + // On a 5.1 route the centre stays on the centre speaker. + #expect(abs(peaks(fiveOne)[2] - center) < 1e-4) + } + + @Test("TrueHD 5.1 at route widths 8 and 2", .timeLimit(.minutes(1))) + func truehdAtRouteWidths() async throws { + let wide = try await decode(fixture: "truehd.mkv", route: AudioOutputRoute(outputChannels: 8)) + #expect(wide.allSatisfy { $0.channels == 6 && $0.channelBitmap == SpeakerSet.surround51.rawValue }) + + let stereo = try await decode(fixture: "truehd.mkv", route: AudioOutputRoute(outputChannels: 2)) + #expect(!stereo.isEmpty) + #expect(stereo.allSatisfy { $0.channels == 2 }) + let center = peaks(wide)[2] + let stereoPeaks = peaks(stereo) + #expect(center > 0.05) + #expect(abs(stereoPeaks[0] / center - 0.7071) < 0.01 && abs(stereoPeaks[1] / center - 0.7071) < 0.01) + let seconds = Double(stereo.reduce(0) { $0 + $1.sampleCount }) / Double(stereo[0].sampleRate) + #expect(abs(seconds - 4.0) < 0.1) + } + + @Test("mono goes out on both speakers", .timeLimit(.minutes(1))) + func monoToStereo() async throws { + // basic.mp4's AAC track is a mono sine. + let mono = try await decode(fixture: "basic.mp4", route: AudioOutputRoute(outputChannels: 1)) + #expect(mono.first?.channels == 1) + let stereo = try await decode(fixture: "basic.mp4", route: AudioOutputRoute(outputChannels: 2)) + #expect(stereo.allSatisfy { $0.channels == 2 && $0.channelBitmap == SpeakerSet.stereo.rawValue }) + let level = peaks(mono)[0] + let sides = peaks(stereo) + #expect(level > 0.05) + #expect(abs(sides[0] - sides[1]) < 1e-4) + #expect(abs(sides[0] / level - 0.7071) < 0.01) + } + + @Test("a route change mid-stream switches layout at a frame boundary, losing nothing", .timeLimit(.minutes(1))) + func routeChangeMidStream() async throws { + let route = AudioOutputRoute(outputChannels: 8) + // A short frame queue keeps the decoder from finishing the file + // before the route changes. + let frames = try await decode(fixture: "surround71.mkv", route: route, frameQueue: 4) { count in + if count == 40 { route.update(outputChannels: 2) } + } + let widths = frames.map(\.channels) + let switchIndex = try #require(widths.firstIndex(of: 2), "the layout must follow the route") + #expect(switchIndex >= 40, "frames decoded for the old route are not rewritten") + #expect(widths[.. AudioLossDetector.Observation { + AudioLossDetector.Observation( + hasAudioLane: true, audioEnded: false, audioRendererFailed: false, + clockRunning: true, playhead: playhead, + audioAhead: audioAhead, videoAhead: videoAhead, serial: serial + ) + } + + /// Feeds `observation` with the playhead advanced in 0.2 s ticks over + /// `seconds`; returns every reason the detector reported. + private func run( + _ detector: inout AudioLossDetector, from start: Double, seconds: Double, + _ observation: (Double) -> AudioLossDetector.Observation + ) -> [AudioLossReason] { + var reported: [AudioLossReason] = [] + var playhead = start + while playhead <= start + seconds + 1e-9 { + if let reason = detector.observe(observation(playhead)) { reported.append(reason) } + playhead += 0.2 + } + return reported + } + + @Test("no lane is absent; a lane starts healthy") + func absentAndHealthy() { + var detector = AudioLossDetector() + var observation = playing(at: 0) + observation.hasAudioLane = false + #expect(detector.observe(observation) == nil) + #expect(detector.health == .absent) + #expect(detector.observe(playing(at: 0)) == nil) + #expect(detector.health == .healthy) + } + + @Test("audio missing at a running playhead while video plays is reported once, after the threshold") + func starvationReportedOnce() { + var detector = AudioLossDetector() + let early = run(&detector, from: 0, seconds: 1.6) { playing(at: $0, audioAhead: false) } + #expect(early.isEmpty, "not before \(AudioLossDetector.threshold) s") + #expect(detector.health == .healthy) + let later = run(&detector, from: 1.8, seconds: 3) { playing(at: $0, audioAhead: false) } + #expect(later == [.starved]) + #expect(detector.health == .lost(.starved)) + } + + @Test("audio coming back clears the verdict, and a second episode is reported again") + func recovery() { + var detector = AudioLossDetector() + _ = run(&detector, from: 0, seconds: 3) { playing(at: $0, audioAhead: false) } + #expect(detector.health == .lost(.starved)) + #expect(detector.observe(playing(at: 3.2)) == nil) + #expect(detector.health == .healthy) + let again = run(&detector, from: 3.4, seconds: 3) { playing(at: $0, audioAhead: false) } + #expect(again == [.starved]) + } + + @Test("moments without audio that belong to playback are not a loss") + func legitimateSilence() { + // Video dry too: the whole pipeline is starving, which is buffering. + var detector = AudioLossDetector() + #expect(run(&detector, from: 0, seconds: 5) { playing(at: $0, audioAhead: false, videoAhead: false) }.isEmpty) + // Audio-only source: nothing to compare against. + #expect(run(&detector, from: 0, seconds: 5) { playing(at: $0, audioAhead: false, videoAhead: nil) }.isEmpty) + // The audio track ended (VOD whose audio is shorter than its video). + #expect(run(&detector, from: 0, seconds: 5) { + var observation = playing(at: $0, audioAhead: false) + observation.audioEnded = true + return observation + }.isEmpty) + // Paused or buffering: the clock does not run. + #expect(run(&detector, from: 0, seconds: 5) { + var observation = playing(at: $0, audioAhead: false) + observation.clockRunning = false + return observation + }.isEmpty) + #expect(detector.health == .healthy) + } + + @Test("a pause restarts the count; a seek voids the episode") + func pauseAndSeek() { + var detector = AudioLossDetector() + _ = run(&detector, from: 0, seconds: 1.6) { playing(at: $0, audioAhead: false) } + var paused = playing(at: 1.6, audioAhead: false) + paused.clockRunning = false + #expect(detector.observe(paused) == nil) + #expect(run(&detector, from: 1.6, seconds: 1.6) { playing(at: $0, audioAhead: false) }.isEmpty) + + _ = run(&detector, from: 3.4, seconds: 3) { playing(at: $0, audioAhead: false) } + #expect(detector.health == .lost(.starved)) + // Seek back: new serial, playhead earlier — a fresh start. + #expect(detector.observe(playing(at: 0.5, audioAhead: false, serial: 1)) == nil) + #expect(detector.health == .healthy) + } + + @Test("decoder and renderer failures are permanent until the lane is rebuilt") + func permanentLosses() { + var detector = AudioLossDetector() + _ = detector.observe(playing(at: 0)) + detector.decoderFailed() + #expect(detector.health == .lost(.decoderFailed)) + // Audio enqueued before the failure still plays out; the verdict stands. + #expect(detector.observe(playing(at: 0.2)) == nil) + #expect(detector.health == .lost(.decoderFailed)) + detector.laneReplaced() + #expect(detector.health == .healthy) + + var failed = playing(at: 1) + failed.audioRendererFailed = true + #expect(detector.observe(failed) == nil) + #expect(detector.health == .lost(.rendererFailed)) + #expect(detector.observe(playing(at: 1.2, serial: 4)) == nil) + #expect(detector.health == .lost(.rendererFailed)) + } +} + +@Suite("AudioRouteSession", .serialized) +struct AudioRouteSessionTests { + private func eventually( + timeout: TimeInterval = 10, + _ condition: () async -> Bool + ) async -> Bool { + let deadline = Date(timeIntervalSinceNow: timeout) + while Date() < deadline { + if await condition() { return true } + try? await Task.sleep(for: .milliseconds(100)) + } + return false + } + + private func makeSession(route: AudioOutputRoute, enableAudio: Bool = true) -> PlayerSession { + var configuration = PlayerConfiguration() + configuration.muted = true + configuration.bufferTarget = 0.5 + configuration.enableAudio = enableAudio + configuration.audioOutputRoute = route + return PlayerSession(configuration: configuration) + } + + @Test("a playing session refits its audio when the route narrows and widens", .timeLimit(.minutes(1))) + func followsRouteChanges() async throws { + let route = AudioOutputRoute(outputChannels: 8) + let session = makeSession(route: route) + _ = try await session.open(url: try Fixtures.path("surround51.ts")) + await session.play() + #expect(await eventually { await session.position > 1 }) + #expect(await session.audioOutput == AudioOutputLayout(sourceChannels: 6, outputChannels: 6, routeChannels: 8)) + + // HDMI receiver → AirPlay: the next decoded frame folds down to stereo. + route.update(outputChannels: 2) + let narrowed = await eventually { await session.audioOutput?.outputChannels == 2 } + #expect(narrowed, "audio must refit to the stereo route") + let before = await session.position + let continued = await eventually { await session.position > before + 1.5 } + #expect(continued, "playback must go on through the route change") + #expect(!session.renderer.audioRendererFailed) + #expect(await session.state == .playing) + #expect(await session.audioHealth == .healthy) + + // And back to surround. + route.update(outputChannels: 6) + #expect(await eventually { await session.audioOutput?.outputChannels == 6 }) + let resumed = await session.position + #expect(await eventually { await session.position > resumed + 1 }) + #expect(!session.renderer.audioRendererFailed) + + let diagnostics = await session.diagnostics + #expect(diagnostics.audioOutput?.routeChannels == 6) + #expect(diagnostics.audioRoute.changes == 2) + #expect(diagnostics.description.contains("aout=6>6ch@6")) + await session.shutdown() + } + + @Test("audio that stops reaching the renderer while video plays is reported", .timeLimit(.minutes(1))) + func reportsLostAudio() async throws { + let session = makeSession(route: AudioOutputRoute(outputChannels: 2)) + let errors = Mutex<[EngineError.Code]>([]) + let listener = Task { [events = session.events] in + for await event in events { + if case .error(let error) = event { errors.withLock { $0.append(error.code) } } + } + } + _ = try await session.open(url: try Fixtures.path("surround51.ts")) + await session.play() + #expect(await eventually { await session.position > 1 }) + #expect(await session.audioHealth == .healthy) + + // The renderer stops taking audio; the clock and the picture go on. + session.renderer.audioRenderer.stopRequestingMediaData() + let lost = await eventually(timeout: 15) { await session.audioHealth == .lost(.starved) } + let health = await session.audioHealth + #expect(lost, "lost audio must be detected, health is \(health)") + // The event travels through the stream after the verdict is set. + let reported = await eventually(timeout: 2) { errors.withLock { $0.contains(.audioLost) } } + let codes = errors.withLock { $0 } + #expect(reported && codes.filter { $0 == .audioLost }.count == 1, "errors: \(codes)") + #expect(await session.state == .playing, "video keeps playing") + #expect(await session.diagnostics.description.contains("audio=lost(starved)")) + + await session.shutdown() + listener.cancel() + } + + @Test("a session without audio reports none", .timeLimit(.minutes(1))) + func noAudioLane() async throws { + let session = makeSession(route: AudioOutputRoute(outputChannels: 2), enableAudio: false) + _ = try await session.open(url: try Fixtures.path("basic.mp4")) + await session.play() + #expect(await eventually { await session.position > 0.5 }) + #expect(await session.audioHealth == .absent) + #expect(await session.audioOutput == nil) + await session.shutdown() + } +} + +private extension Array { + subscript(safe index: Int) -> Element? { + indices.contains(index) ? self[index] : nil + } +} + +/// FFmpeg's mask for a named layout — the reference the speaker-set +/// constants are checked against. +private func layoutMask(_ name: String) -> UInt64 { + var layout = AVChannelLayout() + defer { av_channel_layout_uninit(&layout) } + guard av_channel_layout_from_string(&layout, name) >= 0 else { return 0 } + return layout.u.mask +} diff --git a/Tests/LumeEngineCoreTests/ClosedCaptionTests.swift b/Tests/LumeEngineCoreTests/ClosedCaptionTests.swift new file mode 100644 index 0000000..35774a6 --- /dev/null +++ b/Tests/LumeEngineCoreTests/ClosedCaptionTests.swift @@ -0,0 +1,222 @@ +import CFFmpeg +import Foundation +import Testing +@testable import LumeEngineCore + +/// CEA-608 caption data as a broadcast encoder writes it: one byte pair for +/// CC1 per frame, null padding on every frame without one. Mirrors the +/// caption script in `generate-fixtures.sh`. +private enum CaptionScript { + static let frameDuration: Int64 = 1_000_000 / 30 + + static func oddParity(_ value: UInt8) -> UInt8 { + let bits = value & 0x7F + return bits.nonzeroBitCount % 2 == 0 ? bits | 0x80 : bits + } + + /// Byte pairs per frame index for two pop-on captions (shown at frames + /// 30 and 120, erased at 90 and 180 — 1–3 s and 4–6 s at 30 fps). + static let pairs: [Int: (UInt8, UInt8)] = { + var script: [Int: (UInt8, UInt8)] = [:] + func popOn(load: Int, _ text: String, show: Int, clear: Int) { + var pairs: [(UInt8, UInt8)] = [(0x14, 0x20), (0x14, 0x70)] // load, row 15 + let bytes = Array(text.utf8) + for index in stride(from: 0, to: bytes.count, by: 2) { + pairs.append((bytes[index], index + 1 < bytes.count ? bytes[index + 1] : 0)) + } + for (offset, pair) in pairs.enumerated() { script[load + offset] = pair } + script[show] = (0x14, 0x2F) // end of caption + script[clear] = (0x14, 0x2C) // erase displayed memory + } + popOn(load: 5, "HELLO CAPTIONS", show: 30, clear: 90) + popOn(load: 100, "SECOND CAPTION", show: 120, clear: 180) + return script + }() + + /// The A/53 `cc_data` triples of one frame: CC1 data, field-2 nulls. + static func ccData(frame: Int) -> [UInt8] { + let (first, second) = pairs[frame] ?? (0, 0) + return [0xFC, oddParity(first), oddParity(second), 0xFD, 0x80, 0x80] + } +} + +@Suite("Closed captions") +struct ClosedCaptionTests { + @Test("caption detection ignores null padding, field 2 and invalid triples") + func detection() { + func carries(_ bytes: [UInt8]) -> Bool { + bytes.withUnsafeBytes { ClosedCaptionData.carriesCaptions($0) } + } + #expect(!carries([0xFC, 0x80, 0x80, 0xFD, 0x80, 0x80]), "null padding on both fields") + #expect(!carries([0xFD, 0x14, 0x20]), "field 2 (CC3/XDS)") + #expect(!carries([0xF8, 0x14, 0x20]), "cc_valid not set") + #expect(!carries([0xFE, 0x41, 0x42]), "DTVCC packet data") + #expect(carries([0xFD, 0x80, 0x80, 0xFC, 0x94, 0x20]), "CC1 control code") + #expect(!carries([0xFC, 0x94]), "truncated triple") + } + + /// The decoder runs FFmpeg's caption decoder in real-time mode, which + /// stamps an update with the time of the one before; the cue has to start + /// on the frame that put the caption on screen and end on the frame that + /// erased it, or captions drift from the speech they transcribe. + @Test("caption packets become cues on their frames' timestamps", .timeLimit(.minutes(1))) + func decodeCaptionPackets() throws { + let parameters = try #require(CodecParameters.closedCaptions()) + let packets = Channel(capacity: 512) + let store = SubtitleStore() + let decoder = SubtitleDecoder( + parameters: parameters, input: packets, store: store, codecOptions: ["data_field": "first"] + ) + decoder.start() + + let origin: Int64 = 50_000_000 + for frame in 0..<240 { + let bytes = CaptionScript.ccData(frame: frame) + let packet = try #require(bytes.withUnsafeBytes { + Packet(copying: $0, pts: origin + Int64(frame) * CaptionScript.frameDuration, serial: 0) + }) + try packets.send(packet) + } + #expect(decoder.drainAndShutdown()) + + func time(frame: Int) -> Int64 { origin + Int64(frame) * CaptionScript.frameDuration } + #expect(store.count == 2) + let first = store.activeCues(at: time(frame: 30)) + #expect(first.map(\.text) == ["HELLO CAPTIONS"]) + #expect(first.first?.start == time(frame: 30)) + #expect(first.first?.end == time(frame: 90)) + #expect(store.activeCues(at: time(frame: 29)).isEmpty) + #expect(store.activeCues(at: time(frame: 90)).isEmpty) + #expect(store.activeCues(at: time(frame: 150)).map(\.text) == ["SECOND CAPTION"]) + #expect(store.activeCues(at: time(frame: 200)).isEmpty) + } + + @Test("the tap detects captions and forwards them stamped with the frame's PTS") + func tapForwardsSideData() throws { + let tap = ClosedCaptionTap() + var frame: UnsafeMutablePointer? = av_frame_alloc() + defer { av_frame_free(&frame) } + let raw = try #require(frame) + + // No side data: nothing detected, nothing forwarded. + let sink = Channel(capacity: 4) + tap.attach(sink) + tap.offer(raw, serial: 3) + #expect(!tap.hasDetectedCaptions) + #expect(sink.tryReceive() == nil) + + func attach(_ bytes: [UInt8]) throws { + av_frame_remove_side_data(raw, AV_FRAME_DATA_A53_CC) + let sideData = try #require(av_frame_new_side_data(raw, AV_FRAME_DATA_A53_CC, bytes.count)) + sideData.pointee.data.update(from: bytes, count: bytes.count) + } + + // Padding only: forwarded (the decoder needs the frame cadence), but + // no caption track to offer yet. + raw.pointee.best_effort_timestamp = 7_000_000 + try attach(CaptionScript.ccData(frame: 0)) + tap.offer(raw, serial: 3) + #expect(!tap.hasDetectedCaptions) + let padding = try #require(sink.tryReceive()) + #expect(padding.pts == 7_000_000) + #expect(padding.serial == 3) + #expect(padding.byteCount == 6) + + raw.pointee.best_effort_timestamp = 7_033_333 + try attach(CaptionScript.ccData(frame: 5)) + tap.offer(raw, serial: 3) + #expect(tap.hasDetectedCaptions) + #expect(sink.tryReceive()?.pts == 7_033_333) + + // Detached: detection stays, forwarding stops. + tap.attach(nil) + tap.offer(raw, serial: 3) + #expect(tap.hasDetectedCaptions) + #expect(sink.tryReceive() == nil) + } +} + +@Suite("Closed captions in playback", .serialized) +struct ClosedCaptionPlaybackTests { + private func eventually( + timeout: TimeInterval = 10, + _ condition: () async -> Bool + ) async -> Bool { + let deadline = Date(timeIntervalSinceNow: timeout) + while Date() < deadline { + if await condition() { return true } + try? await Task.sleep(for: .milliseconds(100)) + } + return false + } + + private func makeSession(hardware: Bool) -> PlayerSession { + var configuration = PlayerConfiguration() + configuration.muted = true + configuration.bufferTarget = 0.3 + configuration.hardwareDecode = hardware ? .videoToolbox : .software + return PlayerSession(configuration: configuration) + } + + /// End to end: SEI in an H.264 TS → frame side data (VideoToolbox frames + /// carry it too: FFmpeg parses SEI itself) → caption lane → cues on the + /// video's timestamps. Seeks back to 0 after selecting so the first + /// caption is decoded again rather than raced against the decode lead. + @Test( + "captions in the video surface as a CC track and decode in sync", + .timeLimit(.minutes(1)), + arguments: [true, false] + ) + func captionsFromVideo(hardware: Bool) async throws { + let session = makeSession(hardware: hardware) + let info = try await session.open(url: try Fixtures.path("captions.ts")) + let listedAtOpen = info.tracks.contains(where: \.isEmbeddedClosedCaptions) + #expect(!listedAtOpen, "unknown at open") + + let offered = await eventually { + await session.mediaInfo?.subtitleTracks.contains(where: \.isEmbeddedClosedCaptions) == true + } + #expect(offered, "the CC track should appear once captions are decoded") + + await session.selectSubtitleTrack(TrackInfo.embeddedClosedCaptionsIndex) + #expect(await session.selectedSubtitleTrackIndex == TrackInfo.embeddedClosedCaptionsIndex) + await session.seek(to: 0) + await session.play() + + // Done once the second caption has been erased again. Not a cue + // count: what was decoded before the seek joined the first caption + // halfway through loading and may have left a fragment of its own. + let start = info.startTime + let frame: Int64 = 1_000_000 / 30 + let sawBoth = await eventually(timeout: 30) { + let second = session.subtitles.activeCues(at: start + 5_000_000) + return second.map(\.text) == ["SECOND CAPTION"] && second.allSatisfy { $0.end <= start + 6_000_000 + frame } + } + #expect(sawBoth, "both captions should decode, store has \(session.subtitles.count)") + + let first = session.subtitles.activeCues(at: start + 2_000_000) + #expect(first.map(\.text) == ["HELLO CAPTIONS"]) + if let cue = first.first { + #expect(abs(cue.start - (start + 1_000_000)) <= frame, "shown at 1 s, got \(cue.start - start) µs") + #expect(abs(cue.end - (start + 3_000_000)) <= frame, "erased at 3 s, got \(cue.end - start) µs") + } + #expect(session.subtitles.activeCues(at: start + 3_500_000).isEmpty) + #expect(session.subtitles.activeCues(at: start + 5_000_000).map(\.text) == ["SECOND CAPTION"]) + + await session.selectSubtitleTrack(nil) + #expect(session.subtitles.count == 0) + await session.shutdown() + } + + @Test("a source without captions offers no CC track", .timeLimit(.minutes(1))) + func noCaptionsNoTrack() async throws { + let session = makeSession(hardware: true) + _ = try await session.open(url: try Fixtures.path("basic.mp4")) + await session.play() + _ = await eventually { await session.position > 1 } + let tracks = await session.mediaInfo?.subtitleTracks ?? [] + let listed = tracks.contains(where: \.isEmbeddedClosedCaptions) + #expect(!listed) + await session.shutdown() + } +} diff --git a/Tests/LumeEngineCoreTests/DecoderTests.swift b/Tests/LumeEngineCoreTests/DecoderTests.swift index 33b5cb2..a69e7e1 100644 --- a/Tests/LumeEngineCoreTests/DecoderTests.swift +++ b/Tests/LumeEngineCoreTests/DecoderTests.swift @@ -333,7 +333,8 @@ struct DecoderTests { let collected = await collector.value let rate = try #require(collected.rates.first) #expect(collected.rates.count == 1) - #expect(collected.channels == [1], "sine fixture is mono") + // The sine fixture is mono; on a stereo route it goes out on both sides. + #expect(collected.channels == [2], "mono source on a stereo route") let seconds = Double(collected.totalSamples) / Double(rate) #expect(abs(seconds - 10.0) < 0.3, "expected ~10 s of audio, got \(seconds)") let ptsValues = collected.ptsValues diff --git a/Tests/LumeEngineCoreTests/DisplayCriteriaTests.swift b/Tests/LumeEngineCoreTests/DisplayCriteriaTests.swift new file mode 100644 index 0000000..d9db566 --- /dev/null +++ b/Tests/LumeEngineCoreTests/DisplayCriteriaTests.swift @@ -0,0 +1,408 @@ +import CoreMedia +import CoreVideo +import Foundation +import Testing +@testable import LumeEngineCore + +/// The display mode asked of the TV is what the engine outputs, not what the +/// container says: deinterlacing doubles the rate, Dolby Vision with an IPT +/// base layer leaves as HDR10, any other Dolby Vision stream as its base +/// layer. Getting it wrong is visible on every frame — a 50p football match +/// judders at 25 Hz; a base layer shown in the wrong range is mis-toned. +@Suite("ContentDisplayCriteria") +struct ContentDisplayCriteriaTests { + private typealias Criteria = ContentDisplayCriteria + + private func resolve( + _ stream: Criteria.Stream, + deinterlacing: VideoDecoder.Deinterlacing = VideoDecoder.Deinterlacing(), + output: Criteria.Output? = nil + ) -> Criteria { + Criteria.resolve(stream: stream, deinterlacing: deinterlacing, output: output) + } + + private func broadcast(fps: Double = 25, interlaced: Bool? = true) -> Criteria.Stream { + Criteria.Stream(codecName: "h264", width: 1920, height: 1080, frameRate: fps, isInterlaced: interlaced) + } + + /// Transfer 2 (unspecified) by default: DV encodes often leave the VUI + /// empty, so the configuration record has to decide. + private func dolbyVision(profile: UInt8, compatibility: UInt8, transfer: UInt32 = 2) -> Criteria.Stream { + Criteria.Stream( + codecName: "hevc", width: 3840, height: 2160, frameRate: 24000.0 / 1001, + isInterlaced: false, colorTransfer: transfer, + dolbyVision: DolbyVisionConfiguration(profile: profile, level: 6, baseLayerCompatibilityID: compatibility) + ) + } + + // MARK: Refresh rate + + @Test("1080i50 deinterlaced at field rate asks for 50 Hz, before the first frame") + func interlacedFieldRateDoubles() { + let criteria = resolve(broadcast()) + #expect(criteria.refreshRate == 50) + #expect(criteria.basis == .stream) + } + + @Test("frame-rate deinterlacing, or none, keeps the frame rate") + func interlacedFrameRateKeeps() { + #expect(resolve(broadcast(), deinterlacing: .init(mode: .auto, rate: .frame)).refreshRate == 25) + #expect(resolve(broadcast(), deinterlacing: .off).refreshRate == 25) + } + + @Test("a progressive or unflagged stream is not predicted to deinterlace; .always is") + func deinterlacePrediction() { + #expect(resolve(broadcast(interlaced: false)).refreshRate == 25) + #expect(resolve(broadcast(interlaced: nil)).refreshRate == 25) + #expect(resolve(broadcast(interlaced: false), deinterlacing: .init(mode: .always)).refreshRate == 50) + } + + @Test("NTSC and film rates snap to their exact values, apart from their integer neighbours") + func ntscRates() { + let cases: [(Double, Double)] = [ + (23.976, 24000.0 / 1001), (24, 24), (25, 25), + (29.97, 30000.0 / 1001), (30, 30), (50, 50), + (59.94, 60000.0 / 1001), (60, 60), + // Timestamp jitter around a grid rate. + (25.04, 25), (59.95, 60000.0 / 1001), (23.98, 24000.0 / 1001), (23.995, 24), + ] + for (fps, expected) in cases { + let rate = resolve(broadcast(fps: fps, interlaced: false)).refreshRate + #expect(rate == expected, "\(fps) fps must ask for \(expected), got \(String(describing: rate))") + } + // 29.97i deinterlaced at field rate is 59.94p. + #expect(resolve(broadcast(fps: 30000.0 / 1001)).refreshRate == 60000.0 / 1001) + } + + @Test("unknown or off-grid rates ask for no rate at all") + func unknownRates() { + for fps in [0, -1, .nan, .infinity, 12.5, 15, 48, 25.5, 100, 120] { + let rate = resolve(broadcast(fps: fps, interlaced: false)).refreshRate + #expect(rate == nil, "\(fps) fps must not ask for a rate, got \(String(describing: rate))") + } + } + + @Test("the measured cadence of the output beats the container's guess") + func outputCadenceWins() { + // Field-coded H.264 can report its field rate; what comes out decides. + let reportsFieldRate = broadcast(fps: 50, interlaced: true) + let output = Criteria.Output(width: 1920, height: 1080, frameInterval: 20_000, isDeinterlacing: true) + let criteria = resolve(reportsFieldRate, output: output) + #expect(criteria.refreshRate == 50) + #expect(criteria.basis == .output) + + // Interlaced in the container, progressive frames out: 25. + let progressive = Criteria.Output(width: 1920, height: 1080, frameInterval: 40_000, isDeinterlacing: false) + #expect(resolve(broadcast(), output: progressive).refreshRate == 25) + } + + @Test("frames without a cadence yet: the decoder's deinterlacing state replaces the prediction") + func outputWithoutCadence() { + let flaggedLate = Criteria.Output(width: 1920, height: 1080, isDeinterlacing: true) + let criteria = resolve(broadcast(interlaced: nil), output: flaggedLate) + #expect(criteria.refreshRate == 50) + #expect(criteria.basis == .stream, "a rate not yet measured is still a prediction") + } + + // MARK: Dynamic range + + @Test("SDR, HDR10 and HLG from the stream's transfer characteristics") + func streamRanges() { + var stream = broadcast(interlaced: false) + #expect(resolve(stream).dynamicRange == .sdr) + stream.colorTransfer = 16 + #expect(resolve(stream).dynamicRange == .hdr10) + stream.colorTransfer = 18 + #expect(resolve(stream).dynamicRange == .hlg) + } + + @Test("the frames' transfer function beats the stream's") + func outputRange() { + let untagged = broadcast(interlaced: false) + let pq = Criteria.Output( + width: 3840, height: 2160, + transferFunction: kCVImageBufferTransferFunction_SMPTE_ST_2084_PQ as String + ) + #expect(resolve(untagged, output: pq).dynamicRange == .hdr10) + let hlg = Criteria.Output( + width: 3840, height: 2160, + transferFunction: kCVImageBufferTransferFunction_ITU_R_2100_HLG as String + ) + #expect(resolve(untagged, output: hlg).dynamicRange == .hlg) + + var taggedPQ = untagged + taggedPQ.colorTransfer = 16 + let sdrFrames = Criteria.Output( + width: 1920, height: 1080, + transferFunction: kCVImageBufferTransferFunction_ITU_R_709_2 as String + ) + #expect(resolve(taggedPQ, output: sdrFrames).dynamicRange == .sdr) + } + + @Test("Dolby Vision profile 5, 10.0 and 20 are converted, so they ask for HDR10") + func iptProfilesAskForHDR10() { + for profile: UInt8 in [5, 10, 20] { + let criteria = resolve(dolbyVision(profile: profile, compatibility: 0)) + #expect(criteria.dynamicRange == .hdr10, "profile \(profile)") + #expect(criteria.refreshRate == 24000.0 / 1001) + } + } + + @Test("other Dolby Vision profiles ask for the range of the base layer that is presented") + func compatibleProfilesAskForBaseLayer() { + let cases: [(profile: UInt8, compatibility: UInt8, range: Criteria.DynamicRange)] = [ + (8, 1, .hdr10), (10, 1, .hdr10), (7, 6, .hdr10), (8, 4, .hlg), (8, 2, .sdr), (9, 2, .sdr), + ] + for (profile, compatibility, range) in cases { + let criteria = resolve(dolbyVision(profile: profile, compatibility: compatibility)) + #expect(criteria.dynamicRange == range, "\(profile).\(compatibility) must ask for \(range)") + } + } + + @Test("Dolby Vision is never asked for: there is no Dolby Vision output path") + func neverDolbyVision() { + for profile: UInt8 in [4, 5, 7, 8, 9, 10, 20] { + for compatibility: UInt8 in [0, 1, 2, 4, 6, 9] { + let criteria = resolve(dolbyVision(profile: profile, compatibility: compatibility)) + #expect(criteria.dynamicRange != .dolbyVision, "\(profile).\(compatibility)") + } + } + } + + @Test("once frames exist, their transfer function cross-checks the compatibility ID") + func baseLayerFramesDecide() { + let pq = Criteria.Output( + width: 3840, height: 2160, + transferFunction: kCVImageBufferTransferFunction_SMPTE_ST_2084_PQ as String + ) + let hlg = Criteria.Output( + width: 3840, height: 2160, + transferFunction: kCVImageBufferTransferFunction_ITU_R_2100_HLG as String + ) + #expect(resolve(dolbyVision(profile: 8, compatibility: 1), output: pq).dynamicRange == .hdr10) + #expect(resolve(dolbyVision(profile: 8, compatibility: 4), output: hlg).dynamicRange == .hlg) + // A record that disagrees with what is actually presented loses. + #expect(resolve(dolbyVision(profile: 8, compatibility: 1), output: hlg).dynamicRange == .hlg) + // An unknown compatibility ID falls back to the stream's transfer. + #expect(resolve(dolbyVision(profile: 8, compatibility: 9, transfer: 18)).dynamicRange == .hlg) + // The converted IPT profiles are HDR10 whatever their frames were tagged. + let untagged = Criteria.Output(width: 3840, height: 2160) + #expect(resolve(dolbyVision(profile: 5, compatibility: 0), output: untagged).dynamicRange == .hdr10) + } + + // MARK: Format description + + @Test("HDR10 converted from profile 5 is described as plain HEVC HDR10") + func convertedFormatDescription() throws { + let format = try #require(resolve(dolbyVision(profile: 5, compatibility: 0)).makeFormatDescription()) + #expect(CMFormatDescriptionGetMediaSubType(format) == kCMVideoCodecType_HEVC) + #expect(extensionString(format, kCMFormatDescriptionExtension_TransferFunction) + == kCVImageBufferTransferFunction_SMPTE_ST_2084_PQ as String) + #expect(extensionString(format, kCMFormatDescriptionExtension_ColorPrimaries) + == kCVImageBufferColorPrimaries_ITU_R_2020 as String) + #expect(CMFormatDescriptionGetExtension(format, extensionKey: kCMFormatDescriptionExtension_SampleDescriptionExtensionAtoms) == nil) + let dimensions = CMVideoFormatDescriptionGetDimensions(format) + #expect(dimensions.width == 3840 && dimensions.height == 2160) + } + + @Test("profile 8.1 is described as the HDR10 HEVC it is presented as, with no Dolby Vision signalling") + func baseLayerFormatDescription() throws { + let format = try #require(resolve(dolbyVision(profile: 8, compatibility: 1)).makeFormatDescription()) + #expect(CMFormatDescriptionGetMediaSubType(format) == kCMVideoCodecType_HEVC) + #expect(extensionString(format, kCMFormatDescriptionExtension_TransferFunction) + == kCVImageBufferTransferFunction_SMPTE_ST_2084_PQ as String) + #expect(CMFormatDescriptionGetExtension(format, extensionKey: kCMFormatDescriptionExtension_SampleDescriptionExtensionAtoms) == nil) + } + + @Test("SDR before the first frame is described as BT.709") + func sdrFormatDescription() throws { + let format = try #require(resolve(broadcast()).makeFormatDescription()) + #expect(CMFormatDescriptionGetMediaSubType(format) == kCMVideoCodecType_H264) + #expect(extensionString(format, kCMFormatDescriptionExtension_TransferFunction) + == kCVImageBufferTransferFunction_ITU_R_709_2 as String) + } + + @Test("asking twice for the same mode is recognised, whatever the basis") + func sameMode() { + let predicted = resolve(broadcast()) + let confirmed = resolve(broadcast(), output: .init(width: 1920, height: 1080, frameInterval: 20_000)) + #expect(predicted.asksForSameMode(as: confirmed)) + #expect(!predicted.asksForSameMode(as: resolve(broadcast(), deinterlacing: .off))) + #expect(confirmed.description == "50Hz/sdr(output)") + } + + private func extensionString(_ format: CMFormatDescription, _ key: CFString) -> String? { + CMFormatDescriptionGetExtension(format, extensionKey: key) as? String + } +} + +@Suite("FrameCadence") +struct FrameCadenceTests { + @Test("needs a handful of intervals before it reports one") + func needsSamples() { + var cadence = FrameCadence() + for frame in 0.. PlayerSession { + var configuration = PlayerConfiguration() + configuration.muted = true + configuration.bufferTarget = 0.5 + configuration.deinterlace = deinterlacing + return PlayerSession(configuration: configuration) + } + + /// Polls until the session's criteria are confirmed by its output. + private func confirmedCriteria(_ session: PlayerSession, timeout: TimeInterval = 10) async -> ContentDisplayCriteria? { + let deadline = Date(timeIntervalSinceNow: timeout) + while Date() < deadline { + if let criteria = await session.displayCriteria, criteria.basis == .output { return criteria } + try? await Task.sleep(for: .milliseconds(50)) + } + return nil + } + + @Test("the interlaced fixture asks for 50 Hz SDR, predicted at open and confirmed by its frames", .timeLimit(.minutes(1))) + func interlacedFieldRate() async throws { + let session = session() + let info = try await session.open(url: try Fixtures.path("interlaced.ts")) + #expect(info.videoTracks.first?.video?.isInterlaced == true) + let predicted = await session.displayCriteria + #expect(predicted?.refreshRate == 50, "predicted \(String(describing: predicted))") + + await session.play() + let confirmed = await confirmedCriteria(session) + #expect(confirmed?.refreshRate == 50, "confirmed \(String(describing: confirmed))") + #expect(confirmed?.dynamicRange == .sdr) + #expect(await session.diagnostics.description.contains("display=50Hz/sdr(output)")) + await session.shutdown() + } + + @Test("deinterlacing at frame rate keeps the fixture at 25 Hz", .timeLimit(.minutes(1))) + func interlacedFrameRate() async throws { + let session = session(deinterlacing: .init(mode: .auto, rate: .frame)) + _ = try await session.open(url: try Fixtures.path("interlaced.ts")) + await session.play() + let confirmed = await confirmedCriteria(session) + #expect(confirmed?.refreshRate == 25, "confirmed \(String(describing: confirmed))") + await session.shutdown() + } + + @Test("a progressive 30 fps file asks for 30 Hz", .timeLimit(.minutes(1))) + func progressive() async throws { + let session = session() + let info = try await session.open(url: try Fixtures.path("basic.mp4")) + #expect(info.videoTracks.first?.video?.isInterlaced != true) + await session.play() + let confirmed = await confirmedCriteria(session) + #expect(confirmed?.refreshRate == 30, "confirmed \(String(describing: confirmed))") + await session.shutdown() + } + + @Test("an audio-only source has no display criteria", .timeLimit(.minutes(1))) + func audioOnly() async throws { + let session = session() + _ = try await session.open(url: try Fixtures.path("surround71.mkv")) + #expect(await session.displayCriteria == nil) + await session.shutdown() + } +} + +/// The start hold a host uses to sit out a display mode switch: the clock +/// must not start into a blank TV, and must start by itself once the bound +/// passes, whatever the host does. +@Suite("Start hold", .serialized) +struct StartHoldTests { + private func makeSession(stallThreshold: Double = 8) -> PlayerSession { + var configuration = PlayerConfiguration() + configuration.muted = true + configuration.bufferTarget = 0.5 + configuration.stallThreshold = stallThreshold + return PlayerSession(configuration: configuration) + } + + private func eventually(timeout: TimeInterval = 10, _ condition: () async -> Bool) async -> Bool { + let deadline = Date(timeIntervalSinceNow: timeout) + while Date() < deadline { + if await condition() { return true } + try? await Task.sleep(for: .milliseconds(50)) + } + return false + } + + @Test("a held session buffers and waits; releasing starts it", .timeLimit(.minutes(1))) + func holdThenRelease() async throws { + // A short stall threshold: waiting while held must not read as a stall. + let session = makeSession(stallThreshold: 1) + var events = session.events.makeAsyncIterator() + _ = try await session.open(url: try Fixtures.path("basic.mp4")) + await session.holdStart(for: 20) + await session.play() + + try await Task.sleep(for: .seconds(2)) + #expect(await session.state == .buffering) + #expect(await session.isStartHeld) + #expect(await session.position == 0) + + await session.releaseStartHold() + #expect(await eventually { await session.state == .playing }) + await session.shutdown() + + var stalled = false + while let event = await events.next() { + if case .stalled = event { stalled = true } + } + #expect(!stalled, "a held start is not a stall") + } + + @Test("the hold runs out by itself", .timeLimit(.minutes(1))) + func holdExpires() async throws { + let session = makeSession() + _ = try await session.open(url: try Fixtures.path("basic.mp4")) + await session.holdStart(for: 1) + await session.play() + #expect(await session.state != .playing) + #expect(await eventually { await session.state == .playing }) + #expect(await !session.isStartHeld) + await session.shutdown() + } +} diff --git a/Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh b/Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh index befe268..9edace4 100644 --- a/Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh +++ b/Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh @@ -18,6 +18,8 @@ # replay.ts — 8 s TS whose bytes jump back ~3 s at ~6 s, as a reconnect replays them # ac4.ts — TS whose first audio (ger) is declared AC-4, which no decoder here plays, then eng AAC # hdr10.mp4 — HEVC Main 10, BT.2020 PQ, mastering display + content light level SEI, HDR10+ per frame +# surround51.ts — 20 s H.264 + AC-3 5.1 MPEG-TS, tone in the centre channel only +# captions.ts — H.264 TS with CEA-608 captions in A/53 SEI (1–3 s, 4–6 s; needs python3) set -euo pipefail FFMPEG="${FFMPEG:-ffmpeg}" @@ -329,4 +331,101 @@ JSON echo "generated: $OUT/hdr10.mp4" fi +# Broadcast surround the way IPTV carries it: H.264 + AC-3 5.1 in MPEG-TS, +# long enough to change the output route mid-playback. The tone sits in the +# centre channel alone, so a fold-down that drops channels instead of mixing +# them goes silent. +gen surround51.ts \ + -f lavfi -i "testsrc2=duration=20:size=320x180:rate=25" \ + -f lavfi -i "sine=frequency=440:duration=20" \ + -filter_complex "[1:a]pan=5.1(side)|FC=c0[a]" -map 0:v -map "[a]" \ + -c:v libx264 -preset ultrafast -pix_fmt yuv420p -g 25 \ + -c:a ac3 -b:a 384k -shortest -f mpegts + +# Closed captions the way US broadcast and much IPTV carry them: CEA-608 +# byte pairs inside the video, as A/53 user data in an SEI message on every +# frame — not as a subtitle stream. No ffmpeg CLI option writes captions from +# a caption file into the video, so a short script puts the SEI in front of +# each picture of a B-frame-free encode, so the n-th picture in the file is +# the n-th on screen. Two pop-on captions, each loaded off screen first: +# "HELLO CAPTIONS" shown at 1 s and erased at 3 s, "SECOND CAPTION" at 4–6 s. +if [ ! -f "$OUT/captions.ts" ]; then + if command -v python3 >/dev/null 2>&1; then + "$FFMPEG" -hide_banner -loglevel error -y \ + -f lavfi -i "testsrc2=duration=8:size=320x180:rate=30" \ + -c:v libx264 -preset ultrafast -pix_fmt yuv420p -bf 0 -g 30 \ + -f h264 "$OUT/captions-source.h264" + python3 -I - "$OUT/captions-source.h264" "$OUT/captions.h264" <<'PY' +import sys + +source, destination = sys.argv[1], sys.argv[2] +stream = open(source, 'rb').read() + +def nal_units(data): + starts = [] + index = 0 + while True: + found = data.find(b'\x00\x00\x01', index) + if found < 0: + break + starts.append(found + 3) + index = found + 3 + for position, start in enumerate(starts): + end = starts[position + 1] - 3 if position + 1 < len(starts) else len(data) + yield data[start:end].rstrip(b'\x00') + +def odd_parity(value): + value &= 0x7F + return value | (0x80 if bin(value).count('1') % 2 == 0 else 0) + +# frame -> CEA-608 byte pair for CC1 (field 1); every other frame carries nulls. +script = {} +def pop_on(load, text, show, clear): + pairs = [(0x14, 0x20), (0x14, 0x70)] # resume caption loading, row 15 + for i in range(0, len(text), 2): + pairs.append((ord(text[i]), ord(text[i + 1]) if i + 1 < len(text) else 0)) + for offset, pair in enumerate(pairs): + script[load + offset] = pair + script[show] = (0x14, 0x2F) # end of caption: swap it on screen + script[clear] = (0x14, 0x2C) # erase displayed memory + +pop_on(5, 'HELLO CAPTIONS', 30, 90) +pop_on(100, 'SECOND CAPTION', 120, 180) + +def sei(frame): + first, second = script.get(frame, (0, 0)) + cc_data = bytes([0xFC, odd_parity(first), odd_parity(second), 0xFD, 0x80, 0x80]) + # ITU-T T.35: United States, ATSC, 'GA94', cc_data with two triples. + payload = bytes([0xB5, 0x00, 0x31]) + b'GA94' + bytes([0x03, 0x40 | 2, 0xFF]) + cc_data + bytes([0xFF]) + rbsp = bytes([4, len(payload)]) + payload + bytes([0x80]) + nal = bytearray([0x06]) + zeros = 0 + for byte in rbsp: + if zeros >= 2 and byte <= 3: + nal.append(3) # emulation prevention + zeros = 0 + nal.append(byte) + zeros = zeros + 1 if byte == 0 else 0 + return bytes(nal) + +out = bytearray() +frame = 0 +for nal in nal_units(stream): + # First slice of a picture: a slice NAL whose first_mb_in_slice is 0. + if nal and nal[0] & 0x1F in (1, 5) and len(nal) > 1 and nal[1] & 0x80: + out += b'\x00\x00\x00\x01' + sei(frame) + frame += 1 + out += b'\x00\x00\x00\x01' + nal +open(destination, 'wb').write(out) +PY + "$FFMPEG" -hide_banner -loglevel error -y \ + -framerate 30 -i "$OUT/captions.h264" -c:v copy -f mpegts \ + "$OUT/captions.ts" + rm -f "$OUT/captions-source.h264" "$OUT/captions.h264" + echo "generated: $OUT/captions.ts" + else + echo "skipped: $OUT/captions.ts (needs python3)" >&2 + fi +fi + echo "fixtures ready in $OUT" diff --git a/Tests/LumeEngineCoreTests/SubtitleTimelineTests.swift b/Tests/LumeEngineCoreTests/SubtitleTimelineTests.swift new file mode 100644 index 0000000..ee38af7 --- /dev/null +++ b/Tests/LumeEngineCoreTests/SubtitleTimelineTests.swift @@ -0,0 +1,216 @@ +import Foundation +import Testing +@testable import LumeEngineCore + +@Suite("Subtitle cue lifecycle") +struct SubtitleTimelineTests { + private let second: Int64 = 1_000_000 + + private func screen(_ start: Int64, _ text: String, end: Int64? = nil) -> SubtitleEvent { + SubtitleEvent(start: start, end: end, text: text, replacesPrevious: true) + } + + private func text(_ start: Int64, _ text: String, end: Int64?) -> SubtitleEvent { + SubtitleEvent(start: start, end: end, text: text, replacesPrevious: false) + } + + private func shown(_ timeline: SubtitleTimeline, at time: Int64) -> [String] { + timeline.activeCues(at: time).map(\.text) + } + + // MARK: Timing of one AVSubtitle + + @Test("UINT32_MAX end means open, on any track") + func endlessIsOpen() { + let span = SubtitleEvent.span( + base: 10 * second, startDisplay: 0, endDisplay: .max, packetDuration: 0, replacesPrevious: false + ) + #expect(span.start == 10 * second) + #expect(span.end == nil) + } + + @Test("a missing end is open on a replacing track, a guess on a text track") + func missingEnd() { + let replacing = SubtitleEvent.span( + base: 10 * second, startDisplay: 0, endDisplay: 0, packetDuration: 2 * second, replacesPrevious: true + ) + #expect(replacing.end == nil) + + let withDuration = SubtitleEvent.span( + base: 10 * second, startDisplay: 0, endDisplay: 0, packetDuration: 2 * second, replacesPrevious: false + ) + #expect(withDuration.end == 12 * second) + + let withoutDuration = SubtitleEvent.span( + base: 10 * second, startDisplay: 500, endDisplay: 0, packetDuration: 0, replacesPrevious: false + ) + #expect(withoutDuration.start == 10_500_000) + #expect(withoutDuration.end == 13_500_000) + } + + @Test("display times are milliseconds after the base") + func explicitSpan() { + let span = SubtitleEvent.span( + base: 10 * second, startDisplay: 250, endDisplay: 4_000, packetDuration: 0, replacesPrevious: true + ) + #expect(span.start == 10_250_000) + #expect(span.end == 14 * second) + } + + // MARK: Replacing tracks (captions, DVB, PGS) + + /// Before the timeline, a cue without an end (`UINT32_MAX` ms) stayed up + /// for 49 days and every later caption stacked under it. + @Test("an open cue ends where the next update starts") + func openCueEndsAtNext() { + var timeline = SubtitleTimeline() + timeline.apply(screen(1 * second, "one")) + timeline.apply(screen(4 * second, "two")) + + #expect(shown(timeline, at: 3 * second) == ["one"]) + #expect(shown(timeline, at: 4 * second) == ["two"]) + #expect(shown(timeline, at: 5 * second) == ["two"]) + } + + /// The empty update DVB/PGS/captions send to clear the screen used to be + /// dropped as "no text", leaving the previous cue up. + @Test("a clear ends what is showing and shows nothing itself") + func clearEndsCue() { + var timeline = SubtitleTimeline() + timeline.apply(screen(1 * second, "one")) + timeline.apply(screen(3 * second, "")) + + #expect(shown(timeline, at: 2 * second) == ["one"]) + #expect(shown(timeline, at: 3 * second).isEmpty) + #expect(timeline.count == 1) + } + + @Test("a cue with an end of its own is still ended by the next update") + func explicitEndCutByNext() { + var timeline = SubtitleTimeline() + // DVB page timeouts are long; the next page replaces it well before. + timeline.apply(screen(1 * second, "page one", end: 31 * second)) + timeline.apply(screen(5 * second, "page two", end: 35 * second)) + #expect(shown(timeline, at: 4 * second) == ["page one"]) + #expect(shown(timeline, at: 6 * second) == ["page two"]) + #expect(shown(timeline, at: 36 * second).isEmpty) + } + + @Test("an open cue without a successor stops after the limit") + func openCueIsCapped() { + var timeline = SubtitleTimeline() + timeline.apply(screen(1 * second, "stuck")) + let limit = SubtitleTimeline.openEndedLimit + #expect(shown(timeline, at: 1 * second + limit - 1) == ["stuck"]) + #expect(shown(timeline, at: 1 * second + limit).isEmpty) + } + + @Test("an update arriving before one already stored is bounded by it") + func outOfOrderIsBounded() { + var timeline = SubtitleTimeline() + timeline.apply(screen(5 * second, "later")) + timeline.apply(screen(2 * second, "earlier")) + #expect(shown(timeline, at: 4 * second) == ["earlier"]) + #expect(shown(timeline, at: 5 * second) == ["later"]) + } + + /// A re-read range (backward seek) sends the same updates again; the + /// stored ones were bounded by their successors meanwhile, so the ends + /// differ and only start and text can identify them. + @Test("re-reading a range does not double it") + func rereadIsDeduplicated() { + var timeline = SubtitleTimeline() + for _ in 0..<2 { + timeline.apply(screen(1 * second, "one")) + timeline.apply(screen(3 * second, "")) + timeline.apply(screen(4 * second, "two")) + } + #expect(timeline.count == 2) + #expect(shown(timeline, at: 2 * second) == ["one"]) + #expect(shown(timeline, at: 3_500_000).isEmpty) + #expect(shown(timeline, at: 4_500_000) == ["two"]) + } + + @Test("on a replacing track, a different update at the same instant wins") + func sameStartReplaces() { + var timeline = SubtitleTimeline() + timeline.apply(screen(1 * second, "ROW ONE\nROW TWO")) + // After a seek the caption decoder starts from an empty screen and + // decodes the same instant differently. + timeline.apply(screen(1 * second, "ROW TWO")) + #expect(timeline.count == 1) + #expect(shown(timeline, at: 2 * second) == ["ROW TWO"]) + } + + // MARK: Text tracks (SRT, ASS, WebVTT) + + @Test("text cues keep their overlaps") + func textOverlapKept() { + var timeline = SubtitleTimeline() + timeline.apply(text(1 * second, "one", end: 5 * second)) + timeline.apply(text(2 * second, "two", end: 4 * second)) + #expect(shown(timeline, at: 3 * second) == ["one", "two"]) + #expect(shown(timeline, at: 4_500_000) == ["one"]) + } + + @Test("an open text cue ends at the next cue, a closed one does not") + func openTextCueEndsAtNext() { + var timeline = SubtitleTimeline() + timeline.apply(text(1 * second, "open", end: nil)) + timeline.apply(text(3 * second, "next", end: 6 * second)) + #expect(shown(timeline, at: 2 * second) == ["open"]) + #expect(shown(timeline, at: 4 * second) == ["next"]) + } + + @Test("an empty text cue is nothing, not a clear") + func emptyTextCueIgnored() { + var timeline = SubtitleTimeline() + timeline.apply(text(1 * second, "one", end: 5 * second)) + timeline.apply(text(2 * second, "", end: 3 * second)) + #expect(shown(timeline, at: 4 * second) == ["one"]) + } + + // MARK: Seeks and growth + + @Test("a seek ends open cues, not the ones the source gave an end") + func seekClosesOpenCues() { + var timeline = SubtitleTimeline() + timeline.apply(screen(1 * second, "open")) + timeline.apply(text(1 * second, "closed", end: 9 * second)) + timeline.closeOpenCues(at: 4 * second) + #expect(shown(timeline, at: 3 * second) == ["open", "closed"]) + #expect(shown(timeline, at: 5 * second) == ["closed"]) + + // Cues after the seek target are the future of the new position. + var later = SubtitleTimeline() + later.apply(screen(10 * second, "ahead")) + later.closeOpenCues(at: 4 * second) + #expect(shown(later, at: 11 * second) == ["ahead"]) + } + + /// A day of live roll-up captions is hundreds of thousands of updates; + /// every one of them used to be kept. + @Test("the timeline keeps a bounded number of cues, dropping the oldest") + func boundedGrowth() { + var timeline = SubtitleTimeline() + let total = SubtitleTimeline.capacity * 2 + for index in 0..