diff --git a/AGENTS.md b/AGENTS.md index d04eaa5..e986b78 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -51,7 +51,7 @@ Prerequisites: build/scripts/build-ffmpeg.sh macos-arm64 # one slice is enough for local dev build/scripts/make-xcframework.sh ``` -- Tests need a host `ffmpeg` CLI (Homebrew) — fixtures are generated lazily on first test run by `Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh` into `TestStreams/generated/` (gitignored, idempotent). `Fixtures.swift` finds Homebrew ffmpeg automatically; override with the `FFMPEG` env var. +- Tests need a host `ffmpeg` CLI (Homebrew) — fixtures are generated lazily on first test run by `Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh` into `TestStreams/generated/` (gitignored, idempotent). `Fixtures.swift` finds Homebrew ffmpeg automatically; override with the `FFMPEG` env var. The closed-caption and bitmap-subtitle fixtures (`captions.ts`, `pgs.mkv`, `dvbsub.ts`, `dvdsub.mkv`) also need `python3` (the one Xcode's command-line tools provide is enough), because no ffmpeg CLI option writes captions into the video or renders subtitle bitmaps; without it those fixtures are skipped and their tests fail as missing. CI (`.github/workflows/ci.yml`) builds the macOS FFmpeg slice (cached on `build/versions.json` + scripts + patches hashes) and runs `swift test` with a strict one-thread cooperative pool (see above), checks the Markdown docs offline (`.github/scripts/check-docs.py`: relative links and in-repo anchors resolve, `CLAUDE.md` is still the `AGENTS.md` symlink — run it after editing docs), then cross-compiles the library for iOS and tvOS via `xcodebuild` (compile-only, one cached slice each — visionOS is left to local verification because that matrix entry was never confirmed on a runner — the xrOS SDK ships with Xcode, but a visionOS *destination* also needs Xcode's visionOS platform component); tagged releases build all 10 platform slices and attach the xcframework. @@ -86,6 +86,11 @@ This split is the core invariant of the codebase: - **Multi-layer HEVC reaches the codec as its base layer only.** `Decode/HEVCBaseLayerFilter` drops every `nuh_layer_id > 0` NAL unit from packets and extradata before `avcodec_send_packet`/`avcodec_open2`. FFmpeg files an MV-HEVC stream's second-view SPS/PPS (DV profile 20, spatial video) alongside the base layer's, its VideoToolbox glue packs all of them into one `hvcC`, VideoToolbox rejects that session, and FFmpeg falls back to software *without an error* — 4K HEVC in software, which stutters on Apple TV. The engine presents only the base view, so stripping loses nothing. Relatedly, `VideoDecoder.isHardwareActive` is judged from the frames' pixel format, not from whether a VT device was attached, and such a silent fallback emits `.downgradedToSoftware` — keep it that way, or this class of bug goes invisible again. - **Dolby Vision with an IPT base layer is converted, not tagged.** Profiles 5, 10.0 and 20 (compatibility ID 0) carry IPTPQc2, which shows pink/green as YCbCr. No colour attachment fixes that — the RPU reshaping changes per scene and Apple's decoders don't apply it to decoded frames — so `Decode/DolbyVisionConverter` turns each frame into HDR10 on the GPU in `VideoDecoder.deliver`, the one point every frame (hardware, software, deinterlaced) passes with its side data. Whether a frame is IPT is decided *per frame* by `Decode/DolbyVisionDetection`, never by the configuration record alone — remuxes mislabel HDR10 as P5 and broadcasts drop RPUs without changing it, and converting HDR10 as IPT gives wrong colours. Order: an `AVCOL_SPC_IPT_C2` frame is IPT; a frame with its *own* RPU is what the RPU's `signal_color_space` says (2 = IPT); a frame without one that states a matrix (BT.2020 PQ) is that matrix; only an untagged frame without its own RPU falls back to the record (genuine P5 often leaves the VUI matrix unspecified). "Own" matters because FFmpeg's HEVC decoder attaches the last RPU's `AV_FRAME_DATA_DOVI_METADATA` to every later frame; only `AV_FRAME_DATA_DOVI_RPU_BUFFER` marks the frame that carried one (AV1 attaches metadata only to its own frame). HDR10-compatible profiles (8.1 etc.) must stay on the zero-copy path. GPU refusals are per-frame (iOS refuses a backgrounded app); only a converter that can't exist gives up for the session. - **Display criteria describe the output, not the container.** `PlayerSession.displayCriteria` (pure resolution in `Media/ContentDisplayCriteria`) is what an Apple TV with Match Content should switch the TV to: the refresh rate *after* deinterlacing (field rate turns 1080i50 into 50 Hz) and the dynamic range of what is presented (IPT Dolby Vision profiles are converted, so HDR10; every other Dolby Vision profile is presented as its base layer, so HDR10, HLG or SDR by its compatibility ID and the frames' transfer function). Never ask for Dolby Vision mode while the engine has no Dolby Vision output path — that belongs to the PLAN.md §7 compressed-`dvh1` spike. Before the first frame it is predicted from the stream; once the renderer has been handed frames, their colour tags and measured cadence (`Render/FrameCadence`) decide. Rates snap to the film/broadcast grid or ask for none — never an odd rate. Applying it is tvOS-only and opt-in (`Media/DisplayCriteriaMatcher`, or `LumePlayerView(…, matchesDisplayCriteria: true)`): one matcher per player screen, kept across sessions so zapping between same-mode channels never switches the TV, reset only when the screen goes away. The mode-switch blank is sat out with `PlayerSession.holdStart(for:)`, which the session bounds itself and the stall watchdog ignores — keep both, or a host that never releases wedges playback, or a held start reads as a stall. +- **Dolby Vision with an IPT base layer is converted, not tagged.** Profiles 5, 10.0 and 20 (compatibility ID 0) carry IPTPQc2, which shows pink/green as YCbCr. No colour attachment fixes that — the RPU reshaping changes per scene and Apple's decoders don't apply it to decoded frames — so `Decode/DolbyVisionConverter` turns each frame into HDR10 on the GPU in `VideoDecoder.deliver`, the one point every frame (hardware, software, deinterlaced) passes with its side data. Detection is the stream's DV configuration record *or* an `AVCOL_SPC_IPT_C2` frame; many P5 encodes leave the VUI matrix unspecified, so don't reduce it to the tag. HDR10-compatible profiles (8.1 etc.) must stay on the zero-copy path. GPU refusals are per-frame (iOS refuses a backgrounded app); only a converter that can't exist gives up for the session. +- **Audio is fitted to the negotiated route width, never the route's capability.** `AudioDecoder` reads `AudioOutputRoute.outputChannels` (the session's `outputNumberOfChannels`, followed through route-change notifications) per frame and remixes per `AudioLayoutPolicy` with an explicit `DownmixMatrix` — a fold-down mixes centre and surrounds in, it never drops channels. Don't switch it to `maximumOutputNumberOfChannels`: a renderer fed more channels than the session outputs failed outright on tvOS. The engine only observes the audio session; setting its category, activation or preferred channel count is the host's job (Lume does it). When the route narrows, `SystemRenderer` keeps audio decoded for the wider route out of the renderer. +- **How long a subtitle shows is the timeline's decision, not the decoder's.** Decoders hand `Subtitle/SubtitleStore` a `SubtitleEvent` (start, end *or none*, text, whether the track replaces the whole screen with each update); `Subtitle/SubtitleTimeline` decides the rest: an open cue ends at the next update of its track (capped at 16 s), an empty update on a closed-caption/DVB/PGS/DVD track is a clear that ends what is showing, a seek ends open cues, re-read ranges are deduplicated, and the store is bounded. Don't turn `end_display_time == UINT32_MAX` into a number in a decoder or drop empty updates there — both put stuck or stacked cues on screen. +- **Bitmap subtitles (DVB, PGS, DVD) are images on the cue, positioned by their own canvas.** `Subtitle/SubtitleBitmap` turns `SUBTITLE_BITMAP` rects into premultiplied sRGB `CGImage`s with a rect normalized to the canvas the decoder reports *after* decoding (DVB display definition, PGS presentation size, DVD `size:`), never to the video's pixel size — an HD DVB page over SD video, or PGS authored at 1080p over a 720p encode, would land off screen. The DVB context's size is zeroed before opening so the stream's display definition sets it. Bitmap cues carry empty `text` (text consumers like Lume show nothing rather than garbage); drawing them is the host's job, since `LumePlayerView` is bare video. The timeline bounds bitmap memory separately (`imageByteBudget`). +- **Closed captions leave the video lane through a tap, never a wait.** CEA-608 in A/53 side data (`AV_FRAME_DATA_A53_CC`) is lifted off each decoded frame in `VideoDecoder.receiveFrames` (before deinterlacing) by `Subtitle/ClosedCaptionTap`, stamped with the frame's PTS, and `trySend`-ed to an ordinary `SubtitleDecoder` running `ccaption` in real-time mode — a slow caption lane drops captions, never frames. The track is synthetic (`TrackInfo.embeddedClosedCaptionsIndex`) and appears in `PlayerSession.mediaInfo` once caption content is seen; `.opened` keeps the open-time track list, because no new `PlayerEvent` case may be added for it. Only CEA-608 field 1 (CC1/CC2) is decoded: FFmpeg has no CEA-708 (DTVCC) decoder, and teletext needs libzvbi, which the build does not include. - **Zero global mutable state:** all configuration is `PlayerConfiguration` passed at session creation. No `static var` knobs. - **No force unwraps / `as!` / `fatalError` in engine paths**; failures return typed `EngineError`. No `assertionFailure` in production channel/data paths — misuse degrades, never crashes. - **Never scrape FFmpeg log strings for control flow** (`av_log` is diagnostics only). Silence is never a failure mode: pipelines are supervised, stalls are detected from playhead ground truth, and failures surface as typed `PlayerEvent`s on the session's `AsyncStream`. diff --git a/PLAN.md b/PLAN.md index 782e84e..1acf292 100644 --- a/PLAN.md +++ b/PLAN.md @@ -142,6 +142,7 @@ Brightness/contrast/saturation and MetalFX spatial upscaling operate CVPixelBuff **D6 — One subtitle engine, composited as overlay.** libass renders ASS/SSA (full effects, embedded fonts via attachment streams, word-by-word \k tags); PGS/DVB/VobSub decoded by FFmpeg to bitmaps; SRT/VTT parsed to styled cues rendered through the same layout box. Output = time-tagged `SubtitleImage` cues composited on a CALayer above the video surface (and into the PiP content when active; into Metal for HDR subtitle mode). Dual subtitle slots (main + secondary). External files attach as additional demux sessions. +**Cue lifecycle and closed captions (as built).** Every subtitle source — demuxed stream, sidecar file, or captions lifted from the video — reaches one time-indexed store as display updates; the store's `SubtitleTimeline` alone decides how long each shows (open cues end at the next update or a clear, capped; seeks end open cues; bounded size). CEA-608 captions carried as A/53 side data are tapped off decoded video frames without ever blocking the video lane and surface as a synthetic "CC1" subtitle track once seen, additively through `PlayerSession.mediaInfo` (no new event). PGS/DVB/DVD bitmaps reach the same store as `SubtitleImage`s (premultiplied `CGImage` + rect normalized to the subtitle's authoring canvas) on cues with empty text; compositing them is still the host's job until the overlay layer described above exists. **System caption respect:** map `MACaptionAppearance` (user's Style settings) onto text-cue styling by default. **D7 — Buffering is PTS-math, not heuristics.** @@ -190,6 +191,7 @@ Licensing note: engine code is **MIT** (`LICENSE`); FFmpeg is LGPL 2.1+, and the ## 6. Feature implementation notes (selected) - **Now Playing (explicit requirement):** `NowPlayingCenter` module in `LumeEngineMedia` — `MPNowPlayingInfoCenter` (title/artwork/elapsed/duration/rate, live-stream flag) + `MPRemoteCommandCenter` (play/pause/seek/skip/next-episode hook), audio-session lifecycle, route-change and interruption handling. Works on the lock screen/Control Center including while audio/video routes to an Apple TV. Note: full-screen *video* AirPlay of FFmpeg-decoded content is not possible for third parties (system limitation); Lume's existing switch-to-AVPlayer-on-AirPlay stays, and the integration phase adds a shared app-level Now Playing service so lock-screen metadata is present in **both** engines (closing today's gap). +- **Audio width follows the route:** decoded audio leaves the engine in the layout `AudioLayoutPolicy` picks for the *negotiated* route width (`AVAudioSession.outputNumberOfChannels`, followed through route changes by `AudioOutputRoute`; stereo on macOS): discrete multichannel when the route carries it (HDMI receiver), an explicit ITU-R BS.775 fold-down (`DownmixMatrix`: centre and surrounds at −3 dB, LFE omitted, not normalized) when it does not, mono on both speakers of a stereo route. A route change refits at the next decoded frame; audio decoded for a wider route never reaches a narrower one. The engine observes the audio session only — category, activation and the preferred channel count are the host's. `PlayerSession.audioHealth` reports audio that died while video plays on (decoder gave up, renderer failed, or nothing reaching the renderer at a running playhead). - **PiP:** `AVPictureInPictureController(contentSource: .init(sampleBufferDisplayLayer:playbackDelegate:))` from the SystemRenderer; subtitle cues composited into the sample buffers during PiP so subs survive. - **Trickplay:** a `ThumbnailSession` — separate lightweight demux+decode (keyframes only, HW decode, downscale to ~320 px) filling a time-indexed cache; API `thumbnails(at:interval:)`. Works for VOD/catch-up; live uses the rewind buffer. - **Live rewind (time-shift DVR):** disk-backed packet spool behind a custom AVIO layer; seeking within spool = local demux, seeking to live edge = passthrough. Configurable size/duration caps. diff --git a/Sources/LumeEngine/LumePlayer.swift b/Sources/LumeEngine/LumePlayer.swift index 34d0f40..51b4c76 100644 --- a/Sources/LumeEngine/LumePlayer.swift +++ b/Sources/LumeEngine/LumePlayer.swift @@ -28,6 +28,10 @@ public final class LumePlayer { public private(set) var isHardwareDecoding = false /// Text of the currently visible subtitle cue(s), `nil` when none. public private(set) var subtitleText: String? + /// Bitmaps of the currently visible subtitle cue(s) — DVB, PGS and DVD + /// subtitles, which have no text. `LumePlayerView` does not draw them: + /// overlay each image at its `rect` scaled to the video's display area. + public private(set) var subtitleImages: [SubtitleImage] = [] public var audioTracks: [TrackInfo] { mediaInfo?.audioTracks ?? [] } public var subtitleTracks: [TrackInfo] { mediaInfo?.subtitleTracks ?? [] } @@ -146,7 +150,10 @@ public final class LumePlayer { public func selectSubtitleTrack(_ index: Int32?) { let session = session Task { await session?.selectSubtitleTrack(index) } - if index == nil { subtitleText = nil } + if index == nil { + subtitleText = nil + subtitleImages = [] + } } public func loadExternalSubtitles(url: String) async throws { @@ -165,6 +172,8 @@ public final class LumePlayer { position = 0 mediaInfo = nil duration = nil + subtitleText = nil + subtitleImages = [] } // MARK: Internals @@ -201,14 +210,22 @@ public final class LumePlayer { private func tick() async { guard let session else { return } position = await session.position + // Tracks can turn up after open (closed captions in the video). + if let mediaInfo, let current = await session.mediaInfo, + current.tracks.count != mediaInfo.tracks.count { + self.mediaInfo = current + } let now = session.renderer.currentTime if MediaTime.isValid(now) { let cues = session.subtitles.activeCues(at: now) - let text = cues.map(\.text).joined(separator: "\n") + let text = cues.map(\.text).filter { !$0.isEmpty }.joined(separator: "\n") subtitleText = text.isEmpty ? nil : text + let images = cues.flatMap(\.images) + if images != subtitleImages { subtitleImages = images } } else { subtitleText = nil + if !subtitleImages.isEmpty { subtitleImages = [] } } } diff --git a/Sources/LumeEngineCore/Decode/AudioDecoder.swift b/Sources/LumeEngineCore/Decode/AudioDecoder.swift index f8bebb0..61a51b6 100644 --- a/Sources/LumeEngineCore/Decode/AudioDecoder.swift +++ b/Sources/LumeEngineCore/Decode/AudioDecoder.swift @@ -7,13 +7,15 @@ import Foundation /// FFmpeg audio decode + resample stage: `Channel` in, /// `Channel` out. /// -/// Output is always interleaved Float32 at the source sample rate, downmixed -/// to at most `maxOutputChannels` — the single canonical format the renderer -/// consumes. Sources with more channels than the output route can carry are -/// downmixed here by swresample (normalized matrix); handing the sample-buffer -/// renderer more channels than the route supports produces audibly broken -/// output. Resampling context is rebuilt automatically when source parameters -/// change mid-stream (codec/parameter changes are routine on IPTV). +/// Output is always interleaved Float32 at the source sample rate, in the +/// speaker layout `AudioLayoutPolicy` picks for the output route — the single +/// canonical format the renderer consumes. Sources with more channels than +/// the route carries are folded down here by swresample with an explicit +/// matrix (`DownmixMatrix`); handing the sample-buffer renderer more channels +/// than the route supports produces audibly broken output. The resampling +/// context is rebuilt when source parameters change mid-stream (codec and +/// parameter changes are routine on IPTV) and when the route's width changes +/// (`AudioOutputRoute`), in both cases at a frame boundary. /// /// Decoded frames are coalesced to at least `minOutputDuration` before /// emission. Some codecs decode to frames far too small to schedule @@ -42,6 +44,7 @@ public final class AudioDecoder: @unchecked Sendable { private var swrSourceFormat: Int32 = -1 private var swrSourceRate: Int32 = -1 private var swrSourceChannels: Int32 = -1 + private var swrRouteChannels = -1 private var outputLayout = AVChannelLayout() private var currentSerial: UInt64? private var consecutiveErrors = 0 @@ -59,7 +62,17 @@ public final class AudioDecoder: @unchecked Sendable { private static let minOutputDuration = 0.020 private let maxConsecutiveErrors = 100 - private let maxOutputChannels: Int + private let route: AudioOutputRoute + /// What the resampler currently produces. Guarded by `lock`. + private var appliedOutput: AudioOutputLayout? + + /// The layout decoded audio currently leaves in, and the route width it + /// was chosen for; nil before the first frame. + public var currentOutput: AudioOutputLayout? { + lock.lock() + defer { lock.unlock() } + return appliedOutput + } /// Channel budget of the current output route. The renderer downstream /// cannot correctly render more channels than the route carries, so @@ -72,7 +85,9 @@ public final class AudioDecoder: @unchecked Sendable { /// Feeding the renderer more channels than the session actually outputs /// fails the renderer outright (silence, wedged pipeline) — configuring /// the session for surround is the app's job (PLAN.md §2.1), the engine - /// just honors whatever was negotiated at session-creation time. + /// just honors whatever was negotiated. This reads it once, for a decoder + /// of fixed width; `PlayerSession` follows route changes instead + /// (`AudioOutputRoute`). public static func defaultMaxOutputChannels() -> Int { #if canImport(UIKit) return max(2, AVAudioSession.sharedInstance().outputNumberOfChannels) @@ -84,16 +99,32 @@ public final class AudioDecoder: @unchecked Sendable { #endif } - public init( + /// A decoder whose output width is fixed at `maxOutputChannels`. + public convenience init( parameters: CodecParameters, input: Channel, output: Channel, maxOutputChannels: Int = AudioDecoder.defaultMaxOutputChannels() + ) { + self.init( + parameters: parameters, + input: input, + output: output, + route: AudioOutputRoute(outputChannels: maxOutputChannels) + ) + } + + /// A decoder whose output follows `route` as it changes. + public init( + parameters: CodecParameters, + input: Channel, + output: Channel, + route: AudioOutputRoute ) { self.parameters = parameters self.input = input self.output = output - self.maxOutputChannels = max(1, maxOutputChannels) + self.route = route var continuation: AsyncStream.Continuation! events = AsyncStream(bufferingPolicy: .unbounded) { continuation = $0 } eventSink = continuation @@ -329,12 +360,17 @@ public final class AudioDecoder: @unchecked Sendable { let format = frame.pointee.format let rate = frame.pointee.sample_rate let channels = frame.pointee.ch_layout.nb_channels + let routeChannels = route.outputChannels - if swrContext != nil, format == swrSourceFormat, rate == swrSourceRate, channels == swrSourceChannels { + if swrContext != nil, format == swrSourceFormat, rate == swrSourceRate, channels == swrSourceChannels, + routeChannels == swrRouteChannels { return true } - emitPending() // samples of the previous format cannot share a buffer + // Samples of the previous layout cannot share a buffer. Same-rate + // conversion holds nothing back inside swr, so emitting the pending + // buffer is all a switch needs to land on a frame boundary. + emitPending() swr_free(&swrContext) // swr needs concrete layouts on both sides to build a (down)mix // matrix; sources reporting an unspecified order (routine for PCM) @@ -346,12 +382,27 @@ public final class AudioDecoder: @unchecked Sendable { av_channel_layout_default(&inputLayout, channels) } defer { av_channel_layout_uninit(&inputLayout) } + av_channel_layout_uninit(&outputLayout) - if channels > Int32(maxOutputChannels) { - av_channel_layout_default(&outputLayout, Int32(maxOutputChannels)) + var matrix: [Double]? + if inputLayout.order == AV_CHANNEL_ORDER_NATIVE { + let source = SpeakerSet(rawValue: inputLayout.u.mask) + let plan = AudioLayoutPolicy.plan(source: source, routeChannels: routeChannels) + if plan.remixes { + av_channel_layout_from_mask(&outputLayout, plan.output.rawValue) + matrix = DownmixMatrix.flatCoefficients(from: source, to: plan.output) + } else { + av_channel_layout_copy(&outputLayout, &inputLayout) + } + } else if channels > Int32(routeChannels) { + // No positions to plan with (more channels than FFmpeg has a + // default order for): fold to the route's default layout and let + // swr do what it can. + av_channel_layout_default(&outputLayout, Int32(routeChannels)) } else { av_channel_layout_copy(&outputLayout, &inputLayout) } + var newContext: OpaquePointer? let result = swr_alloc_set_opts2( &newContext, @@ -359,7 +410,16 @@ public final class AudioDecoder: @unchecked Sendable { &inputLayout, AVSampleFormat(rawValue: format), rate, 0, nil ) + // A custom matrix goes in between allocation and init. Without one + // (no remix, or a layout pair the builder refused) swr builds its own + // from the two layouts. + if result >= 0, let created = newContext, let matrix { + _ = matrix.withUnsafeBufferPointer { + swr_set_matrix(created, $0.baseAddress, Int32(DownmixMatrix.stride)) + } + } guard result >= 0, let created = newContext, swr_init(created) >= 0 else { + swr_free(&newContext) handleDecodeError(nil, EngineError(code: .decodeFailed, message: "swresample init failed")) return false } @@ -367,6 +427,15 @@ public final class AudioDecoder: @unchecked Sendable { swrSourceFormat = format swrSourceRate = rate swrSourceChannels = channels + swrRouteChannels = routeChannels + let applied = AudioOutputLayout( + sourceChannels: Int(channels), + outputChannels: Int(outputLayout.nb_channels), + routeChannels: routeChannels + ) + lock.lock() + appliedOutput = applied + lock.unlock() return true } diff --git a/Sources/LumeEngineCore/Decode/AudioLayoutPolicy.swift b/Sources/LumeEngineCore/Decode/AudioLayoutPolicy.swift new file mode 100644 index 0000000..b435039 --- /dev/null +++ b/Sources/LumeEngineCore/Decode/AudioLayoutPolicy.swift @@ -0,0 +1,169 @@ +internal import CFFmpeg + +/// Speaker positions in FFmpeg's native channel-mask bit order (`AV_CH_*`), +/// which is also WAVE order and Core Audio's `AudioChannelBitmap` for bits +/// 0...17. Kept free of FFmpeg types so the layout policy is testable as plain +/// values. +struct SpeakerSet: OptionSet, Hashable, Sendable { + let rawValue: UInt64 + + static let frontLeft = SpeakerSet(rawValue: 1 << 0) + static let frontRight = SpeakerSet(rawValue: 1 << 1) + static let frontCenter = SpeakerSet(rawValue: 1 << 2) + static let lowFrequency = SpeakerSet(rawValue: 1 << 3) + static let backLeft = SpeakerSet(rawValue: 1 << 4) + static let backRight = SpeakerSet(rawValue: 1 << 5) + static let sideLeft = SpeakerSet(rawValue: 1 << 9) + static let sideRight = SpeakerSet(rawValue: 1 << 10) + + static let mono: SpeakerSet = [.frontCenter] + static let stereo: SpeakerSet = [.frontLeft, .frontRight] + /// 5.1 with the surrounds on the side pair (`AV_CH_LAYOUT_5POINT1`, + /// "5.1(side)"): the layout the AC-3 and E-AC-3 decoders — most surround + /// on IPTV — already hand the renderer. + static let surround51: SpeakerSet = [.frontLeft, .frontRight, .frontCenter, .lowFrequency, .sideLeft, .sideRight] + static let surround71: SpeakerSet = surround51.union([.backLeft, .backRight]) + + /// Positions a format description can name with a channel bitmap + /// (`SampleBufferBuilder`). Anything outside reaches the renderer without + /// a layout, and the renderer cannot place untagged multichannel audio. + static let bitmapExpressible = SpeakerSet(rawValue: 0x3FFFF) + + var count: Int { rawValue.nonzeroBitCount } +} + +/// What the audio lane hands the renderer, for diagnostics: the source's +/// channel count, the count after the engine's remix, and the route width the +/// remix was chosen for. +public struct AudioOutputLayout: Sendable, Equatable, CustomStringConvertible { + public let sourceChannels: Int + public let outputChannels: Int + public let routeChannels: Int + + /// The engine folds the source down to fit the route. + public var isDownmixed: Bool { outputChannels < sourceChannels } + + public var description: String { + "\(sourceChannels)>\(outputChannels)ch@\(routeChannels)" + } +} + +/// Which speaker layout the engine hands the audio renderer, given the +/// source's layout and how many channels the output route carries. +/// +/// The route width is the audio session's *negotiated* output +/// (`AVAudioSession.outputNumberOfChannels`), not what the hardware could do: +/// handing `AVSampleBufferAudioRenderer` more channels than the session +/// outputs failed it outright on tvOS — silence and a wedged pipeline — which +/// is why the engine mixes instead of leaving it to the system (see +/// `AudioOutputRoute`). +/// +/// Pure value logic: the decoder asks it once per source format and route +/// width, and swresample does what it says. +enum AudioLayoutPolicy { + struct Plan: Equatable, Sendable { + /// Speakers the renderer receives. + let output: SpeakerSet + /// The engine mixes the channels (down, or mono up to stereo) rather + /// than passing them through as decoded. + let remixes: Bool + } + + /// - Parameters: + /// - source: the decoded layout, resolved to native positions. + /// - routeChannels: channels the output route carries. + static func plan(source: SpeakerSet, routeChannels: Int) -> Plan { + let route = max(1, routeChannels) + let channels = source.count + + // A lone centre channel is placed on two speakers by the engine, not + // by whatever the route does with a centre it may not have: a stereo + // pair (two HomePods, a TV's speakers) has no centre speaker at all. + // Equal power on each side, the standard pan law. + if source == .mono, route >= 2 { + return Plan(output: .stereo, remixes: true) + } + + // Fits the route and the renderer can name every channel: hand it + // over untouched. A receiver on HDMI gets discrete surround. + if channels <= route, source.isSubset(of: .bitmapExpressible) { + return Plan(output: source, remixes: false) + } + + // Fold down to the widest standard layout both the route and the + // source fill. A layout the renderer cannot name (wide or top + // channels, a matrix-encoded stereo pair) is remixed to the standard + // one of its own width rather than passed through untagged. + let width = min(route, channels) + var target: SpeakerSet = width >= 8 ? .surround71 + : width >= 6 ? .surround51 + : width >= 2 ? .stereo + : .mono + // Never invent an LFE channel the source does not have. + if !source.contains(.lowFrequency) { + target.remove(.lowFrequency) + } + return Plan(output: target, remixes: true) + } +} + +/// The coefficients the engine's remix uses, built by swresample's own +/// matrix builder with the levels below. +/// +/// Levels follow ITU-R BS.775 fold-down: centre and surrounds at −3 dB into +/// their front side, LFE omitted (it carries nothing a stereo programme needs, +/// and full-range speakers rarely reproduce it). The dialogue channel thus +/// always reaches both front speakers — the failure this exists to prevent is +/// a downmix that drops channels instead of mixing them, which loses the +/// centre and with it the dialogue. +/// +/// The matrix is not normalized, matching what swresample does on its own for +/// float output: a 5.1 fold-down plays as loud as a stereo channel next to it +/// on the same IPTV list. Normalizing would make every surround channel +/// ~7.7 dB quieter than its stereo neighbours, which viewers hear as a volume +/// jump on every zap; the cost is that peaks where every channel is near full +/// scale at once can exceed 1.0 and clip at the output. +enum DownmixMatrix { + static let centerLevel = 0.707_106_781_186_547_6 + static let surroundLevel = 0.707_106_781_186_547_6 + static let lfeLevel = 0.0 + + /// swresample's matrix bound (`SWR_CH_MAX`); its builder walks the full + /// square, so the buffer must span it whatever the layouts are. + static let stride = 64 + + /// Row-major coefficients, `[output][input]`, in native channel order; + /// nil when swresample cannot build a matrix between the two layouts + /// (e.g. a source without any front channel). + static func coefficients(from source: SpeakerSet, to output: SpeakerSet) -> [[Double]]? { + guard let flat = flatCoefficients(from: source, to: output) else { return nil } + return (0.. [Double]? { + var inputLayout = AVChannelLayout() + var outputLayout = AVChannelLayout() + defer { + av_channel_layout_uninit(&inputLayout) + av_channel_layout_uninit(&outputLayout) + } + guard av_channel_layout_from_mask(&inputLayout, source.rawValue) >= 0, + av_channel_layout_from_mask(&outputLayout, output.rawValue) >= 0 + else { return nil } + + var matrix = [Double](repeating: 0, count: stride * stride) + let result = matrix.withUnsafeMutableBufferPointer { buffer in + swr_build_matrix2( + &inputLayout, &outputLayout, + centerLevel, surroundLevel, lfeLevel, + Double(Int32.max), // no normalization, see the type docs + 1.0, buffer.baseAddress, stride, + AV_MATRIX_ENCODING_NONE, nil + ) + } + return result >= 0 ? matrix : nil + } +} diff --git a/Sources/LumeEngineCore/Decode/VideoDecoder.swift b/Sources/LumeEngineCore/Decode/VideoDecoder.swift index 9dbc9d5..5cd00a8 100644 --- a/Sources/LumeEngineCore/Decode/VideoDecoder.swift +++ b/Sources/LumeEngineCore/Decode/VideoDecoder.swift @@ -140,6 +140,10 @@ public final class VideoDecoder: @unchecked Sendable { private let maxConsecutiveErrors = 100 + /// Closed captions riding in the video (A/53 side data) leave here, ahead + /// of deinterlacing and conversion, for the session's caption lane. + let captionTap = ClosedCaptionTap() + public init( parameters: CodecParameters, input: Channel, @@ -365,6 +369,7 @@ public final class VideoDecoder: @unchecked Sendable { consecutiveErrors = 0 if usingHardware { hardwareFailures.recordFrame() } noteFrameSource(frame) + captionTap.offer(frame, serial: currentSerial ?? 0) emit(frame: frame) av_frame_unref(frame) } diff --git a/Sources/LumeEngineCore/Demux/TrackInfo.swift b/Sources/LumeEngineCore/Demux/TrackInfo.swift index d469be0..0fdf75b 100644 --- a/Sources/LumeEngineCore/Demux/TrackInfo.swift +++ b/Sources/LumeEngineCore/Demux/TrackInfo.swift @@ -111,6 +111,21 @@ public struct DolbyVisionConfiguration: Sendable, Equatable { } } +extension TrackInfo { + /// Index of the closed-caption track carried inside the video stream + /// (CEA-608 in A/53 side data — how US broadcast and much IPTV ship + /// captions). It is no container stream, so it gets an index no container + /// reaches ("CC" in ASCII, channel 1); pass it to `selectSubtitleTrack` + /// like any other subtitle index. + public static let embeddedClosedCaptionsIndex: Int32 = 0x4343_0001 + + /// True for the synthetic closed-caption track (`embeddedClosedCaptionsIndex`), + /// which `PlayerSession.mediaInfo` lists once captions turn up in the video. + public var isEmbeddedClosedCaptions: Bool { + index == Self.embeddedClosedCaptionsIndex + } +} + /// A chapter marker, in engine microseconds. public struct ChapterInfo: Sendable, Equatable { public let title: String? @@ -135,7 +150,9 @@ public struct MediaInfo: Sendable { public let isSeekable: Bool public let bitrate: Int64 public let metadata: [String: String] - public let tracks: [TrackInfo] + /// Streams found at open, plus tracks discovered while playing (the + /// closed-caption track — see `PlayerSession.mediaInfo`). + public internal(set) var tracks: [TrackInfo] public let chapters: [ChapterInfo] public var videoTracks: [TrackInfo] { tracks.filter { $0.kind == .video } } @@ -144,4 +161,13 @@ public struct MediaInfo: Sendable { /// Live heuristic: no known duration. public var isLive: Bool { duration == nil } + + /// This info with `track` appended, unless a track with its index is + /// already listed. + func adding(_ track: TrackInfo) -> MediaInfo { + guard !tracks.contains(where: { $0.index == track.index }) else { return self } + var copy = self + copy.tracks.append(track) + return copy + } } diff --git a/Sources/LumeEngineCore/Errors/EngineError.swift b/Sources/LumeEngineCore/Errors/EngineError.swift index 6d645c1..c93a54d 100644 --- a/Sources/LumeEngineCore/Errors/EngineError.swift +++ b/Sources/LumeEngineCore/Errors/EngineError.swift @@ -25,6 +25,13 @@ public struct EngineError: Error, Sendable, CustomStringConvertible, Equatable { /// ends arrives as `.ioError` instead: FFmpeg cannot tell it from a /// dropped connection.) case sourceEnded + /// Audio stopped reaching the renderer while video plays on. Not + /// fatal: the session keeps playing, without sound, and recovers by + /// itself if audio comes back. `PlayerSession.audioHealth` says which + /// stage lost it. (An audio decoder that gives up reports its own + /// `.decodeFailed` instead, and a failed audio renderer fails the + /// session with `.renderFailed`.) + case audioLost } /// The HTTP status class a source answered with, when FFmpeg reported one. diff --git a/Sources/LumeEngineCore/Media/AudioOutputRoute.swift b/Sources/LumeEngineCore/Media/AudioOutputRoute.swift new file mode 100644 index 0000000..1d6e014 --- /dev/null +++ b/Sources/LumeEngineCore/Media/AudioOutputRoute.swift @@ -0,0 +1,140 @@ +#if os(iOS) || os(tvOS) || os(visionOS) +import AVFAudio +#endif +import Foundation +import Synchronization + +/// How many channels the audio output route carries, kept current while the +/// route changes under a playing session (HDMI receiver → AirPlay → HomePods). +/// +/// The audio decoder reads it for every decoded frame and folds its output to +/// fit (`AudioLayoutPolicy`), so a route change takes effect at the next frame +/// boundary; the renderer reads it to keep audio decoded for a wider, earlier +/// route away from a narrower one. +/// +/// The width is what the audio session *negotiated* +/// (`outputNumberOfChannels`), never what the hardware could carry +/// (`maximumOutputNumberOfChannels`): multichannel LPCM only flows once the +/// host asks for it with `setPreferredOutputNumberOfChannels`, and a renderer +/// fed more channels than the session outputs failed outright on tvOS. +/// Configuring and activating the session is the host's job — this type only +/// observes it. +/// +/// `system()` follows the shared `AVAudioSession`. On macOS there is no audio +/// session; it reports stereo, which is what the default output overwhelmingly +/// is. A host that knows better (a macOS multichannel interface), and tests, +/// create one with `init(outputChannels:)` and drive it with `update`. +public final class AudioOutputRoute: Sendable { + public struct Snapshot: Sendable, Equatable, CustomStringConvertible { + /// Channels the route carries for this app now. + public var outputChannels: Int + /// Channels the route could carry if the host asked for them; nil + /// where nothing reports it. More than `outputChannels` on a surround + /// route means the host did not ask for the route's full width. + public var maximumOutputChannels: Int? + /// Output port types (`HDMIOutput`, `AirPlay`, …). Types only, never + /// the user-visible port names. + public var ports: [String] + /// How often the width has changed since the route was created. + public var changes: Int + + public var description: String { + var line = "\(outputChannels)ch" + if let maximumOutputChannels { line += "/\(maximumOutputChannels)" } + if !ports.isEmpty { line += " " + ports.joined(separator: "+") } + return line + } + } + + private let state: Mutex + private let observers = Mutex<[any NSObjectProtocol]>([]) + + /// A route of fixed width that only `update` changes. + public init(outputChannels: Int, maximumOutputChannels: Int? = nil, ports: [String] = []) { + state = Mutex(Snapshot( + outputChannels: max(1, outputChannels), + maximumOutputChannels: maximumOutputChannels, + ports: ports, + changes: 0 + )) + } + + deinit { + for observer in observers.withLock({ $0 }) { + NotificationCenter.default.removeObserver(observer) + } + } + + /// The route of the shared audio session, followed through route changes. + public static func system() -> AudioOutputRoute { + #if os(iOS) || os(tvOS) || os(visionOS) + let current = sessionSnapshot() + let route = AudioOutputRoute( + outputChannels: current.outputChannels, + maximumOutputChannels: current.maximumOutputChannels, + ports: current.ports + ) + route.followSession() + return route + #else + return AudioOutputRoute(outputChannels: 2) + #endif + } + + /// Channels the route carries now. Cheap: read once per decoded frame. + public var outputChannels: Int { + state.withLock { $0.outputChannels } + } + + public var snapshot: Snapshot { + state.withLock { $0 } + } + + /// Reports a new width (and, when known, capability and ports). + public func update(outputChannels: Int, maximumOutputChannels: Int? = nil, ports: [String]? = nil) { + state.withLock { snapshot in + let width = max(1, outputChannels) + if width != snapshot.outputChannels { snapshot.changes += 1 } + snapshot.outputChannels = width + if let maximumOutputChannels { snapshot.maximumOutputChannels = maximumOutputChannels } + if let ports { snapshot.ports = ports } + } + } + + #if os(iOS) || os(tvOS) || os(visionOS) + /// The session's current output. Clamped to stereo at the bottom: an + /// inactive session reports 0 or 1, and a stereo fold-down is right for + /// every real route that is narrower. + private static func sessionSnapshot() -> (outputChannels: Int, maximumOutputChannels: Int, ports: [String]) { + let session = AVAudioSession.sharedInstance() + return ( + max(2, session.outputNumberOfChannels), + session.maximumOutputNumberOfChannels, + session.currentRoute.outputs.map(\.portType.rawValue) + ) + } + + /// Route changes are posted on a system thread; the width is re-read + /// there. A change of the negotiated width alone (the host asking for more + /// channels) arrives as `.routeConfigurationChange`, so every reason is + /// taken. + private func followSession() { + // The token is created inside the lock so it never leaves the + // mutex's region (it is not Sendable). + observers.withLock { observers in + observers.append(NotificationCenter.default.addObserver( + forName: AVAudioSession.routeChangeNotification, + object: nil, + queue: nil + ) { [weak self] _ in + let current = Self.sessionSnapshot() + self?.update( + outputChannels: current.outputChannels, + maximumOutputChannels: current.maximumOutputChannels, + ports: current.ports + ) + }) + } + } + #endif +} diff --git a/Sources/LumeEngineCore/Render/SystemRenderer.swift b/Sources/LumeEngineCore/Render/SystemRenderer.swift index 3abb97a..f2a30fb 100644 --- a/Sources/LumeEngineCore/Render/SystemRenderer.swift +++ b/Sources/LumeEngineCore/Render/SystemRenderer.swift @@ -51,6 +51,8 @@ public final class SystemRenderer: @unchecked Sendable { private var audioFormatCache: (description: CMAudioFormatDescription, sampleRate: Int, channels: Int, channelBitmap: UInt64)? // Audio-queue-confined; serial changes re-anchor it after a seek flush. private var audioTimeline = AudioTimeline() + /// Route width at the previous audio pump pass. Audio-queue-confined. + private var lastAudioRouteChannels = Int.max /// Audio enqueued but not yet played, kept so it can be enqueued again /// after the renderer drops it on its own (see `refillAfterAutomaticFlush`). @@ -60,6 +62,7 @@ public final class SystemRenderer: @unchecked Sendable { let serial: UInt64 /// End of the sample on the engine timeline (µs). let end: Int64 + let channels: Int } private var retainedAudio: [RetainedAudio] = [] /// Backstop only: the renderer stops asking for audio once it holds a @@ -88,7 +91,13 @@ public final class SystemRenderer: @unchecked Sendable { private var audioLaneAttached = false private var reportedFailure = false - public init(muted: Bool = false) { + /// The route the audio lane's decoder fits its output to. Audio decoded + /// for a wider route before it narrowed is held back from the renderer + /// (see `pumpAudio`); nil feeds every frame. + private let audioOutputRoute: AudioOutputRoute? + + public init(muted: Bool = false, audioOutputRoute: AudioOutputRoute? = nil) { + self.audioOutputRoute = audioOutputRoute displayLayer = AVSampleBufferDisplayLayer() displayLayer.videoGravity = .resizeAspect videoRenderer = displayLayer.sampleBufferRenderer @@ -171,7 +180,11 @@ public final class SystemRenderer: @unchecked Sendable { // answer; that audio is retained too, so drop it and re-enqueue // everything in order. audioRenderer.flush() - for entry in retainedAudio where entry.serial == serial && (!MediaTime.isValid(flushTime) || entry.end > flushTime) { + // A route change is the usual cause of the flush; audio retained for + // a wider route than the new one stays out (see `pumpAudio`). + let routeChannels = audioOutputRoute?.outputChannels ?? .max + for entry in retainedAudio where entry.serial == serial && entry.channels <= routeChannels + && (!MediaTime.isValid(flushTime) || entry.end > flushTime) { audioRenderer.enqueue(entry.sample) } // A seek that flushed while this ran may have been overtaken by the @@ -371,6 +384,10 @@ public final class SystemRenderer: @unchecked Sendable { ) } + var audioRendererFailed: Bool { + audioRenderer.status == .failed + } + public func shutdown() { lock.lock() stopped = true @@ -434,12 +451,25 @@ public final class SystemRenderer: @unchecked Sendable { lock.unlock() guard let input, !isStopped else { return } + // The route narrowed (HDMI surround → AirPlay stereo): audio already + // decoded for the wider route must not reach the renderer, which + // fails on more channels than the route carries. The decoder refits + // at its next frame; until then what is queued for the old width is + // dropped, and whatever the renderer still holds of it is replaced by + // the retained audio that fits — usually nothing, so the switch costs + // the renderer's lead in silence, as a route change does anyway. + let routeChannels = audioOutputRoute?.outputChannels ?? .max + if routeChannels < lastAudioRouteChannels { + refillAfterAutomaticFlush(from: currentTime) + } + lastAudioRouteChannels = routeChannels + while audioRenderer.isReadyForMoreMediaData { guard let frame = input.tryReceive() else { scheduleAudioRetry() return } - guard frame.serial == serial else { continue } + guard frame.serial == serial, frame.channels <= routeChannels else { continue } do { let sample = try SampleBufferBuilder.audio( from: frame, @@ -452,7 +482,7 @@ public final class SystemRenderer: @unchecked Sendable { if !MediaTime.isValid(firstEnqueuedAudioPTS) { firstEnqueuedAudioPTS = frame.pts } if firstAudioEnqueue == nil { firstAudioEnqueue = ProcessInfo.processInfo.systemUptime } lock.unlock() - retainAudio(sample, serial: serial, end: frame.pts + frame.duration) + retainAudio(sample, serial: serial, end: frame.pts + frame.duration, channels: frame.channels) } catch { continue } @@ -460,13 +490,13 @@ public final class SystemRenderer: @unchecked Sendable { } /// Keeps `sample` until the clock has played past it. Audio queue only. - private func retainAudio(_ sample: CMSampleBuffer, serial: UInt64, end: Int64) { + private func retainAudio(_ sample: CMSampleBuffer, serial: UInt64, end: Int64, channels: Int) { let now = currentTime retainedAudio.removeAll { entry in entry.serial != serial || (MediaTime.isValid(now) && entry.end <= now) } if retainedAudio.count >= Self.maxRetainedAudio { retainedAudio.removeFirst() } - retainedAudio.append(RetainedAudio(sample: sample, serial: serial, end: end)) + retainedAudio.append(RetainedAudio(sample: sample, serial: serial, end: end, channels: channels)) } private func scheduleVideoRetry() { diff --git a/Sources/LumeEngineCore/Session/AudioHealth.swift b/Sources/LumeEngineCore/Session/AudioHealth.swift new file mode 100644 index 0000000..5c69f4f --- /dev/null +++ b/Sources/LumeEngineCore/Session/AudioHealth.swift @@ -0,0 +1,153 @@ +import Foundation + +/// Whether the session's audio is reaching the viewer, as far as the engine +/// can tell. Read `PlayerSession.audioHealth`; it is also in `Diagnostics`. +/// +/// Exists because audio can die while video plays on: a decoder that gives up +/// leaves its lane at end of stream, and the session — correctly — keeps +/// showing the picture. Without this the only trace was one `.error` event, +/// and the viewer sat in front of a silent channel. +public enum AudioHealth: Sendable, Equatable, CustomStringConvertible { + /// Audio is playing, or nothing says otherwise yet. + case healthy + /// The session has no audio lane: the source carries no audio, or the + /// host opened it without (`enableAudio`) or turned it off. + case absent + /// The audio lane has stopped while the session plays on. + case lost(AudioLossReason) + + public var description: String { + switch self { + case .healthy: "ok" + case .absent: "none" + case .lost(let reason): "lost(\(reason.rawValue))" + } + } +} + +/// Which stage the audio was lost in. +public enum AudioLossReason: String, Sendable, Equatable { + /// The audio decoder gave up on the stream (its error went out as an + /// `.error` event). Permanent for the track: switching tracks, or audio + /// off and on, builds a new decoder. + case decoderFailed + /// The system audio renderer failed. The session fails with + /// `.renderFailed` as well; this records that it was the audio lane. + case rendererFailed + /// The clock runs and video plays, but for a while now the audio renderer + /// has had nothing to play at the playhead: decoded audio stopped reaching + /// it, or reached it too late to play. Reported once per episode as an + /// `.error` with `EngineError.Code.audioLost`; clears by itself when audio + /// comes back. + case starved +} + +/// Decides `AudioHealth` from what the session observes on its monitor tick. +/// +/// Pure value logic, so every branch is testable without a pipeline that has +/// to be made to lose its audio. +/// +/// The starvation judgement is deliberately narrow, so that moments without +/// audio that are part of normal playback never count: it needs a running +/// clock (not paused, buffering or seeking), a video lane that is itself ahead +/// of the playhead (a whole pipeline running dry is buffering, which the +/// session handles), an audio lane still expecting data (not at its end), and +/// the playhead to have moved `threshold` seconds of media with no audio +/// enqueued beyond it. A gap *inside* the audio does not count either: the +/// renderer's high-water mark is past the gap as soon as audio after it was +/// enqueued. +struct AudioLossDetector { + /// Seconds of media the playhead must cover without audio ahead of it. + static let threshold = 2.0 + + struct Observation: Equatable { + /// An audio lane is attached. + var hasAudioLane: Bool + /// The audio decoder reached the end of its stream: nothing more is + /// coming, and silence after it is the source's end, not a loss. + var audioEnded: Bool + /// The audio renderer reports `.failed`. + var audioRendererFailed: Bool + /// The session plays and the clock rate is non-zero. + var clockRunning: Bool + /// Playhead, seconds; nil while the clock has no valid time. + var playhead: Double? + /// Audio is enqueued beyond the playhead. + var audioAhead: Bool + /// Video is enqueued beyond the playhead; nil without a video lane. + var videoAhead: Bool? + /// The session's seek serial: a seek rebuilds every queue, so any + /// episode in progress is void. + var serial: UInt64 + } + + private(set) var health: AudioHealth = .absent + /// A loss only a new decoder clears. + private var permanent: AudioLossReason? + /// Playhead when audio was first seen missing; nil while it is not. + private var dryFrom: Double? + private var serial: UInt64 = 0 + + /// The audio decoder failed for good. + mutating func decoderFailed() { + permanent = .decoderFailed + health = .lost(.decoderFailed) + } + + /// The audio lane was rebuilt or removed (track switch, audio off/on): + /// whatever was wrong with the old one is gone with it. + mutating func laneReplaced() { + permanent = nil + dryFrom = nil + health = .healthy + } + + /// Updates `health`; returns the reason when a starvation episode begins, + /// which is the moment to tell the host. + mutating func observe(_ observation: Observation) -> AudioLossReason? { + guard observation.hasAudioLane else { + permanent = nil + dryFrom = nil + health = .absent + return nil + } + if health == .absent { + health = .healthy + } + if observation.audioRendererFailed { + permanent = .rendererFailed + } + if let permanent { + dryFrom = nil + health = .lost(permanent) + return nil + } + if observation.serial != serial { + serial = observation.serial + dryFrom = nil + health = .healthy + } + if observation.audioEnded || observation.audioAhead { + dryFrom = nil + health = .healthy + return nil + } + // Not judged: paused, buffering or seeking (the clock does not run), + // no time yet, or no healthy video to compare against. A standing + // verdict stays until audio is seen ahead again. + guard observation.clockRunning, + let playhead = observation.playhead, + observation.videoAhead == true + else { + dryFrom = nil + return nil + } + guard let start = dryFrom, playhead >= start else { + dryFrom = playhead + return nil + } + guard playhead - start >= Self.threshold, health != .lost(.starved) else { return nil } + health = .lost(.starved) + return .starved + } +} diff --git a/Sources/LumeEngineCore/Session/PlayerSession+Tracks.swift b/Sources/LumeEngineCore/Session/PlayerSession+Tracks.swift index 8ea401f..7961e93 100644 --- a/Sources/LumeEngineCore/Session/PlayerSession+Tracks.swift +++ b/Sources/LumeEngineCore/Session/PlayerSession+Tracks.swift @@ -52,8 +52,15 @@ extension PlayerSession { } /// Selects an embedded subtitle track, or `nil` to disable subtitles. + /// `TrackInfo.embeddedClosedCaptionsIndex` selects the captions carried in + /// the video stream — also before they have turned up in `mediaInfo`, so + /// a host can keep captions on across channels. public func selectSubtitleTrack(_ index: Int32?) { teardownSubtitleLane() + if index == TrackInfo.embeddedClosedCaptionsIndex { + attachClosedCaptionLane() + return + } guard let index, let demuxer = activeDemuxer, let parameters = demuxer.codecParameters(forStream: index) @@ -83,6 +90,33 @@ extension PlayerSession { } } + /// Feeds the captions the video decoder finds (see `ClosedCaptionTap`) + /// to a caption decoder writing into the cue store. + /// + /// No backfill seek, unlike a subtitle stream: the captions ride in video + /// frames that are decoded a fraction of a second ahead of the playhead, + /// so nothing worth recovering has been read past — and live sources, + /// where captions mostly are, do not survive a seek. + private func attachClosedCaptionLane() { + guard let video = activeVideoDecoder, + let parameters = CodecParameters.closedCaptions() + else { return } + // One payload per frame: a few seconds of slack for a decoder that + // falls behind, beyond which captions are dropped, never frames. + let packets = Channel(capacity: 256) + let decoder = SubtitleDecoder( + parameters: parameters, + input: packets, + store: subtitles, + // CC1/CC2. FFmpeg would otherwise lock onto whichever field's + // data comes first, which can be field 2's metadata. + codecOptions: ["data_field": "first"] + ) + installSubtitleLane(trackIndex: TrackInfo.embeddedClosedCaptionsIndex, packets: packets, decoder: decoder) + decoder.start() + video.captionTap.attach(packets) + } + /// Loads a sidecar subtitle file (SRT/VTT/ASS/…) into the session's cue /// store, replacing any embedded selection. Resolves once the file is /// fully parsed. diff --git a/Sources/LumeEngineCore/Session/PlayerSession.swift b/Sources/LumeEngineCore/Session/PlayerSession.swift index 3d91295..7c688c5 100644 --- a/Sources/LumeEngineCore/Session/PlayerSession.swift +++ b/Sources/LumeEngineCore/Session/PlayerSession.swift @@ -70,6 +70,11 @@ public struct PlayerConfiguration: Sendable { public var autoEnableForcedSubtitlesForForeignAudio = false /// Seconds without clock progress (while expected) before `.stalled` fires. public var stallThreshold: Double = 8 + /// The output route the audio is fitted to. nil follows the shared audio + /// session through route changes (iOS/tvOS/visionOS; stereo on macOS). + /// Pass one to pin the width or drive it yourself, e.g. for a macOS + /// multichannel interface. + public var audioOutputRoute: AudioOutputRoute? public init() {} } @@ -167,9 +172,20 @@ public actor PlayerSession { public private(set) var selectedSubtitleTrackIndex: Int32? public private(set) var usingExternalSubtitles = false + /// Width the audio lane is fitted to; shared with its decoder and the renderer. + private let audioRoute: AudioOutputRoute + private var audioLoss = AudioLossDetector() + + /// Whether audio is reaching the viewer (see `AudioHealth`). + public var audioHealth: AudioHealth { audioLoss.health } + + /// The layout the audio lane hands the renderer; nil without audio. + public var audioOutput: AudioOutputLayout? { audioDecoder?.currentOutput } + public init(configuration: PlayerConfiguration = PlayerConfiguration()) { self.configuration = configuration - renderer = SystemRenderer(muted: configuration.muted) + audioRoute = configuration.audioOutputRoute ?? AudioOutputRoute.system() + renderer = SystemRenderer(muted: configuration.muted, audioOutputRoute: audioRoute) var continuation: AsyncStream.Continuation! events = AsyncStream(bufferingPolicy: .unbounded) { continuation = $0 } eventSink = continuation @@ -271,7 +287,7 @@ public actor PlayerSession { let packets = makeAudioPacketChannel() let frames = Channel(capacity: configuration.audioQueueDepth, measure: { $0.duration }) demuxer.attach(channel: packets, toStream: track.index) - let decoder = AudioDecoder(parameters: parameters, input: packets, output: frames) + let decoder = AudioDecoder(parameters: parameters, input: packets, output: frames, route: audioRoute) audioPackets = packets audioFrames = frames audioDecoder = decoder @@ -379,7 +395,15 @@ public actor PlayerSession { info?.duration.map(MediaTime.seconds) } - public var mediaInfo: MediaInfo? { info } + /// What `open` returned, plus tracks found since: the closed-caption track + /// (`TrackInfo.embeddedClosedCaptionsIndex`) appears once captions turn up + /// in the video, which no open-time probe can tell. Re-read it to offer + /// that track; `.opened` carries the open-time value only. + public var mediaInfo: MediaInfo? { + guard let info else { return nil } + guard videoDecoder?.captionTap.hasDetectedCaptions == true else { return info } + return info.adding(.embeddedClosedCaptions) + } /// How far startup has got and how long each step took. public var startup: StartupTimeline { @@ -428,6 +452,12 @@ public actor PlayerSession { public let lastStallReason: StallReason? /// The display mode the video wants (see `displayCriteria`). public let displayCriteria: ContentDisplayCriteria? + /// Whether audio is reaching the viewer. + public let audioHealth: AudioHealth + /// Source → output channels of the audio lane; nil without one. + public let audioOutput: AudioOutputLayout? + /// The output route the audio is fitted to. + public let audioRoute: AudioOutputRoute.Snapshot public var description: String { func lane(_ queue: (count: Int, seconds: Double)?, lead: Double, first: Double) -> String { @@ -454,6 +484,8 @@ public actor PlayerSession { ) if let lastStallReason { line += " stall=\(lastStallReason.rawValue)" } if let displayCriteria { line += " display=\(displayCriteria)" } + line += " aout=\(audioOutput.map(String.init(describing:)) ?? "off") route=\(audioRoute)" + if case .lost = audioHealth { line += " audio=\(audioHealth)" } line += " startup=[\(startup)]" return line } @@ -490,7 +522,10 @@ public actor PlayerSession { replayedPackets: demuxer?.replayedPackets ?? 0, startup: startup, lastStallReason: lastStallReason, - displayCriteria: displayCriteria + displayCriteria: displayCriteria, + audioHealth: audioHealth, + audioOutput: audioOutput, + audioRoute: audioRoute.snapshot ) } @@ -552,6 +587,7 @@ public actor PlayerSession { demuxAtEOF = false videoAtEOF = false audioAtEOF = false + subtitles.closeOpenCues(at: target) demuxer.seek(to: target) // Completion continues in handleDemux(.didSeek). } @@ -665,6 +701,7 @@ public actor PlayerSession { audioFrames = nil selectedAudioTrackIndex = nil audioAtEOF = false + audioLoss.laneReplaced() renderer.attach(video: videoFrames, audio: nil) } @@ -678,11 +715,12 @@ public actor PlayerSession { let packets = makeAudioPacketChannel() let frames = Channel(capacity: configuration.audioQueueDepth, measure: { $0.duration }) demuxer.attach(channel: packets, toStream: trackIndex) - let decoder = AudioDecoder(parameters: parameters, input: packets, output: frames) + let decoder = AudioDecoder(parameters: parameters, input: packets, output: frames, route: audioRoute) audioPackets = packets audioFrames = frames audioDecoder = decoder audioAtEOF = false + audioLoss.laneReplaced() selectedAudioTrackIndex = trackIndex let pump = Task { [events = decoder.events] in for await event in events { @@ -698,6 +736,7 @@ public actor PlayerSession { if let old = selectedSubtitleTrackIndex { demuxer?.detach(streamIndex: old) } + videoDecoder?.captionTap.attach(nil) if let retired = subtitleDecoder { retired.requestStop() // no cue of the old track lands after `removeAll` below retire { retired.shutdown(deadline: 2) } @@ -840,7 +879,12 @@ public actor PlayerSession { // One lane dying is not necessarily fatal (e.g. broken audio track, // healthy video) — surface it; the monitor decides on full failure. eventSink.yield(.error(error)) - if isVideo { videoAtEOF = true } else { audioAtEOF = true } + if isVideo { + videoAtEOF = true + } else { + audioAtEOF = true + audioLoss.decoderFailed() + } lastError = error } } @@ -855,9 +899,36 @@ public actor PlayerSession { evaluatePlayback() runStallWatchdog() noteFirstFrameShown() + evaluateAudioHealth() } } + /// Audio that stops while the picture plays on is a failure the viewer + /// notices and nothing else would report (see `AudioLossDetector` for + /// what does not count). + private func evaluateAudioHealth() { + let playhead = renderer.currentTime + let highWater = renderer.enqueuedHighWaterMark + func ahead(_ mark: Int64) -> Bool { + MediaTime.isValid(mark) && MediaTime.isValid(playhead) && mark > playhead + } + let observation = AudioLossDetector.Observation( + hasAudioLane: audioDecoder != nil, + audioEnded: audioAtEOF, + audioRendererFailed: renderer.audioRendererFailed, + clockRunning: state == .playing && renderer.rate > 0, + playhead: MediaTime.isValid(playhead) ? MediaTime.seconds(playhead) : nil, + audioAhead: ahead(highWater.audio), + videoAhead: videoDecoder != nil ? ahead(highWater.video) : nil, + serial: currentSerial + ) + guard audioLoss.observe(observation) == .starved else { return } + eventSink.yield(.error(EngineError( + code: .audioLost, + message: "audio stopped reaching the renderer while video plays" + ))) + } + /// A container seek can "succeed" while landing somewhere else entirely — /// matroska falls back to the earliest cluster when its cues are /// unreachable (end-of-file cues over non-range HTTP is the classic IPTV diff --git a/Sources/LumeEngineCore/Subtitle/ClosedCaptions.swift b/Sources/LumeEngineCore/Subtitle/ClosedCaptions.swift new file mode 100644 index 0000000..d35af1c --- /dev/null +++ b/Sources/LumeEngineCore/Subtitle/ClosedCaptions.swift @@ -0,0 +1,130 @@ +internal import CFFmpeg +import Foundation +import Synchronization + +/// Closed captions carried inside the video stream. +/// +/// US broadcast (ATSC) and much IPTV ship captions not as a subtitle stream +/// but as A/53 user data in the video itself — SEI messages in H.264/HEVC, +/// picture user data in MPEG-2 — which FFmpeg's video decoders hand out as +/// `AV_FRAME_DATA_A53_CC` side data on the decoded frame: a run of 3-byte +/// `cc_data` triples. The tap lifts those payloads off the video lane and +/// feeds them, as packets stamped with the frame's PTS, to an ordinary +/// `SubtitleDecoder` running FFmpeg's `ccaption` decoder. That keeps captions +/// on the video's own timestamps (engine µs, display order), and keeps +/// caption decoding — and its failures — off the video thread. +/// +/// The video thread never waits on it: payloads go out with `trySend`, and a +/// caption lane that falls behind loses captions, never frames. +final class ClosedCaptionTap: Sendable { + private struct State { + var detected = false + var sink: Channel? + } + + private let state = Mutex(State()) + + /// True once the video carried caption data with content (null padding + /// alone does not count — broadcast encoders send it on every frame). + var hasDetectedCaptions: Bool { + state.withLock { $0.detected } + } + + /// Starts (or with `nil` stops) forwarding caption payloads. + func attach(_ sink: Channel?) { + state.withLock { $0.sink = sink } + } + + /// Video decode thread: called with every decoded frame, before any + /// filtering. A frame without caption data costs one side-data lookup. + func offer(_ frame: UnsafeMutablePointer, serial: UInt64) { + guard let sideData = av_frame_get_side_data(frame, AV_FRAME_DATA_A53_CC), + let data = sideData.pointee.data, + sideData.pointee.size > 0 + else { return } + let payload = UnsafeRawBufferPointer(start: data, count: Int(sideData.pointee.size)) + + let (detected, sink) = state.withLock { ($0.detected, $0.sink) } + if !detected, ClosedCaptionData.carriesCaptions(payload) { + state.withLock { $0.detected = true } + } + + let pts = frame.pointee.best_effort_timestamp + guard let sink, MediaTime.isValid(pts), + let packet = Packet(copying: payload, pts: pts, serial: serial) + else { return } + _ = sink.trySend(packet) + } +} + +/// Reading A/53 `cc_data` (CEA-708 §4.4: `cc_valid`, `cc_type`, two data +/// bytes per triple). +enum ClosedCaptionData { + /// Whether the payload holds caption data for CC1/CC2 (CEA-608 field 1) + /// beyond null padding — what decides that a stream has captions to + /// offer. Field 2 (CC3/CC4, mostly XDS metadata) and DTVCC (708) packets + /// do not count: the caption decoder shows field 1 only. + static func carriesCaptions(_ payload: UnsafeRawBufferPointer) -> Bool { + var offset = 0 + while offset + 3 <= payload.count { + let header = payload[offset] + let valid = header & 0x04 != 0 + let fieldOne = header & 0x03 == 0 + if valid, fieldOne, payload[offset + 1] & 0x7F != 0 || payload[offset + 2] & 0x7F != 0 { + return true + } + offset += 3 + } + return false + } +} + +extension TrackInfo { + /// The synthetic track for captions found in the video stream. + static let embeddedClosedCaptions = TrackInfo( + index: embeddedClosedCaptionsIndex, + kind: .subtitle, + codecName: String(cString: avcodec_get_name(AV_CODEC_ID_EIA_608)), + codecID: AV_CODEC_ID_EIA_608.rawValue, + language: nil, + title: "CC1", + isDefault: false, + isForced: false, + bitrate: 0, + video: nil, + audio: nil, + wrapBits: 64 + ) +} + +extension CodecParameters { + /// Parameters for the caption lane: no container stream describes it. + static func closedCaptions() -> CodecParameters? { + guard let scratch = avcodec_parameters_alloc() else { return nil } + defer { + var pointer: UnsafeMutablePointer? = scratch + avcodec_parameters_free(&pointer) + } + scratch.pointee.codec_type = AVMEDIA_TYPE_SUBTITLE + scratch.pointee.codec_id = AV_CODEC_ID_EIA_608 + return CodecParameters(copying: scratch) + } +} + +extension Packet { + /// A packet owning a copy of `bytes`, on the engine timeline already. + convenience init?(copying bytes: UnsafeRawBufferPointer, pts: Int64, serial: UInt64) { + guard let base = bytes.baseAddress, !bytes.isEmpty, + let raw = av_packet_alloc() + else { return nil } + guard av_new_packet(raw, Int32(bytes.count)) >= 0, let destination = raw.pointee.data else { + var pointer: UnsafeMutablePointer? = raw + av_packet_free(&pointer) + return nil + } + destination.update(from: base.assumingMemoryBound(to: UInt8.self), count: bytes.count) + raw.pointee.pts = pts + raw.pointee.dts = pts + self.init(adopting: raw, pts: pts, dts: pts, duration: 0, serial: serial) + } +} diff --git a/Sources/LumeEngineCore/Subtitle/SubtitleBitmap.swift b/Sources/LumeEngineCore/Subtitle/SubtitleBitmap.swift new file mode 100644 index 0000000..9998156 --- /dev/null +++ b/Sources/LumeEngineCore/Subtitle/SubtitleBitmap.swift @@ -0,0 +1,163 @@ +internal import CFFmpeg +import CoreGraphics +import Foundation + +/// One bitmap of a DVB, PGS or DVD subtitle, ready to draw over the video. +/// +/// `rect` is normalized to the picture the subtitle was authored for: (0, 0) +/// is the top-left corner of the video, (1, 1) the bottom-right, so a host +/// draws the image at `rect` scaled by the size of the video's display area +/// (after aspect fitting). The authoring canvas — a DVB display definition +/// (720×576 without one), a PGS presentation's video size, a DVD's `size:` — +/// need not match the stream's pixel size; normalizing makes that moot. +public struct SubtitleImage: Sendable, Equatable { + /// Premultiplied RGBA in sRGB. + public let image: CGImage + /// Position and size as fractions of the video picture. + public let rect: CGRect + + public init(image: CGImage, rect: CGRect) { + self.image = image + self.rect = rect + } + + /// Memory the bitmap holds, for the store's budget. + var byteCount: Int { image.bytesPerRow * image.height } +} + +/// Palette-indexed `SUBTITLE_BITMAP` rects to `SubtitleImage`s. +enum SubtitleBitmap { + /// The canvas when a decoder reports none — DVB's own default (ETSI EN + /// 300 743), and the PAL DVD frame; NTSC DVDs carry their size. + static let fallbackCanvas = (width: 720, height: 576) + + /// The images of one decoded subtitle. `canvasWidth`/`canvasHeight` are + /// the codec context's dimensions after decoding, which the bitmap + /// decoders set to their authoring canvas. + static func images(of subtitle: AVSubtitle, canvasWidth: Int32, canvasHeight: Int32) -> [SubtitleImage] { + var rects: [UnsafeMutablePointer] = [] + for index in 0.. SubtitleImage? { + let width = Int(rect.w) + let height = Int(rect.h) + let stride = Int(rect.linesize.0) + guard width > 0, height > 0, stride >= width, + let indices = rect.data.0, let paletteData = rect.data.1, + rect.nb_colors > 0 + else { return nil } + + let palette = UnsafeRawPointer(paletteData).withMemoryRebound( + to: UInt32.self, capacity: Int(rect.nb_colors) + ) { Array(UnsafeBufferPointer(start: $0, count: Int(rect.nb_colors))) } + let pixels = premultipliedPixels( + indices: UnsafeBufferPointer(start: indices, count: stride * height), + width: width, height: height, stride: stride, palette: palette + ) + guard pixels.contains(where: { $0 >> 24 != 0 }), + let image = makeImage(pixels: pixels, width: width, height: height) + else { return nil } // fully transparent: nothing to draw + + return SubtitleImage( + image: image, + rect: normalizedRect( + x: Int(rect.x), y: Int(rect.y), width: width, height: height, + canvasWidth: canvasWidth, canvasHeight: canvasHeight + ) + ) + } + + /// Palette-indexed pixels to premultiplied ARGB words. FFmpeg's palettes + /// are straight-alpha `0xAARRGGBB`; Core Animation and Core Graphics + /// composite premultiplied, and straight alpha drawn as premultiplied + /// shows a bright fringe around every glyph's anti-aliased edge. An index + /// beyond the palette is transparent. + static func premultipliedPixels( + indices: UnsafeBufferPointer, + width: Int, + height: Int, + stride: Int, + palette: [UInt32] + ) -> [UInt32] { + let premultiplied = palette.map(premultiply) + var pixels = [UInt32](repeating: 0, count: width * height) + for row in 0.. UInt32 { + let alpha = argb >> 24 + func channel(_ shift: UInt32) -> UInt32 { + ((argb >> shift & 0xFF) * alpha + 127) / 255 + } + return alpha << 24 | channel(16) << 16 | channel(8) << 8 | channel(0) + } + + /// ARGB words in native (little-endian) order are BGRA bytes in memory: + /// `byteOrder32Little` + `premultipliedFirst`, Core Graphics' native layout. + static func makeImage(pixels: [UInt32], width: Int, height: Int) -> CGImage? { + guard pixels.count == width * height, + let space = CGColorSpace(name: CGColorSpace.sRGB) + else { return nil } + let data = pixels.withUnsafeBytes { Data($0) } + guard let provider = CGDataProvider(data: data as CFData) else { return nil } + return CGImage( + width: width, + height: height, + bitsPerComponent: 8, + bitsPerPixel: 32, + bytesPerRow: width * 4, + space: space, + bitmapInfo: CGBitmapInfo(rawValue: CGBitmapInfo.byteOrder32Little.rawValue + | CGImageAlphaInfo.premultipliedFirst.rawValue), + provider: provider, + decode: nil, + shouldInterpolate: true, + intent: .defaultIntent + ) + } + + static func normalizedRect( + x: Int, y: Int, width: Int, height: Int, + canvasWidth: Int, canvasHeight: Int + ) -> CGRect { + guard canvasWidth > 0, canvasHeight > 0 else { return .zero } + let canvasW = Double(canvasWidth) + let canvasH = Double(canvasHeight) + return CGRect( + x: Double(x) / canvasW, + y: Double(y) / canvasH, + width: Double(width) / canvasW, + height: Double(height) / canvasH + ) + } +} diff --git a/Sources/LumeEngineCore/Subtitle/SubtitleCue.swift b/Sources/LumeEngineCore/Subtitle/SubtitleCue.swift index a262a93..3e53b56 100644 --- a/Sources/LumeEngineCore/Subtitle/SubtitleCue.swift +++ b/Sources/LumeEngineCore/Subtitle/SubtitleCue.swift @@ -1,7 +1,14 @@ import Foundation +import Synchronization -/// One subtitle cue on the engine timeline. Text-only for now; bitmap cues -/// (PGS/DVB) and full libass styling land with the subtitle pack (PLAN.md P5+). +/// One subtitle cue on the engine timeline: text, bitmaps, or (rarely) both. +/// Full libass styling lands with the subtitle pack (PLAN.md P5+). +/// +/// Bitmap formats (DVB, PGS, DVD) arrive as `images` with empty `text`, so a +/// consumer that only reads `text` shows nothing for them rather than +/// something wrong. Drawing them is the host's job: `LumePlayerView` is bare +/// video. Draw each image at its `rect` scaled to the video's display area, +/// above the video. public struct SubtitleCue: Sendable, Equatable, Identifiable { public let id: UInt64 /// Engine µs. @@ -9,85 +16,64 @@ public struct SubtitleCue: Sendable, Equatable, Identifiable { public let end: Int64 /// Plain display text (override tags stripped, line breaks preserved). public let text: String + /// Bitmaps to draw over the video, positioned relative to it. + public let images: [SubtitleImage] - public init(id: UInt64, start: Int64, end: Int64, text: String) { + public init(id: UInt64, start: Int64, end: Int64, text: String, images: [SubtitleImage] = []) { self.id = id self.start = start self.end = end self.text = text + self.images = images + } + + func ending(at time: Int64) -> SubtitleCue { + SubtitleCue(id: id, start: start, end: time, text: text, images: images) } } /// Time-indexed cue container, safe for concurrent insert (decoder thread) and -/// query (UI tick). Cues are kept sorted by start time. +/// query (UI tick). The lifecycle policy — how long a cue shows, what ends it, +/// how many are kept — is `SubtitleTimeline`'s. /// /// Cues outlive seeks on purpose — the store is a timeline, not a queue — so /// the same packet reaches it again whenever the demuxer re-reads a range it /// already delivered: a backward seek, or the seek that backfills a subtitle /// track selected after the demuxer read past its packets. An identical cue -/// (same start, end, and text) is therefore dropped, or every re-read would -/// show its lines doubled. -public final class SubtitleStore: @unchecked Sendable { - private let lock = NSLock() - private var cues: [SubtitleCue] = [] - private var nextID: UInt64 = 0 +/// is therefore dropped, or every re-read would show its lines doubled. +public final class SubtitleStore: Sendable { + private let timeline = Mutex(SubtitleTimeline()) public init() {} + /// A text cue with a known end (tests, and the shape every text format + /// decodes to). func insert(start: Int64, end: Int64, text: String) { let trimmed = text.trimmingCharacters(in: .whitespacesAndNewlines) guard !trimmed.isEmpty, end > start else { return } - lock.lock() - defer { lock.unlock() } - // Insertion point by start time (cues arrive nearly sorted). - var index = cues.count - while index > 0 && cues[index - 1].start > start { index -= 1 } - // Cues with the same start sit directly before the insertion point. - var existing = index - 1 - while existing >= 0 && cues[existing].start == start { - if cues[existing].end == end && cues[existing].text == trimmed { return } - existing -= 1 - } - cues.insert(SubtitleCue(id: nextID, start: start, end: end, text: trimmed), at: index) - nextID += 1 + apply(SubtitleEvent(start: start, end: end, text: trimmed, replacesPrevious: false)) + } + + func apply(_ event: SubtitleEvent) { + timeline.withLock { $0.apply(event) } + } + + /// Ends cues still open at `time`: the session seeked there, and what the + /// source shows after it is unknown until its next update. + func closeOpenCues(at time: Int64) { + timeline.withLock { $0.closeOpenCues(at: time) } } /// All cues covering `time` (engine µs). public func activeCues(at time: Int64) -> [SubtitleCue] { - lock.lock() - defer { lock.unlock() } - // Binary search for the first cue starting after `time`, then walk back. - // Overlapping cues are rare and short-lived; a bounded backward scan is - // simpler than an interval tree and plenty fast. - var low = 0 - var high = cues.count - while low < high { - let mid = (low + high) / 2 - if cues[mid].start <= time { low = mid + 1 } else { high = mid } - } - var active: [SubtitleCue] = [] - var index = low - 1 - var scanned = 0 - while index >= 0 && scanned < 64 { - let cue = cues[index] - if cue.start <= time && time < cue.end { - active.append(cue) - } - index -= 1 - scanned += 1 - } - return active.sorted { $0.start < $1.start } + timeline.withLock { $0.activeCues(at: time) } } public var count: Int { - lock.lock() - defer { lock.unlock() } - return cues.count + timeline.withLock { $0.count } } public func removeAll() { - lock.lock() - defer { lock.unlock() } - cues.removeAll() + timeline.withLock { $0.removeAll() } } } diff --git a/Sources/LumeEngineCore/Subtitle/SubtitleDecoder.swift b/Sources/LumeEngineCore/Subtitle/SubtitleDecoder.swift index c8ab793..cda9a33 100644 --- a/Sources/LumeEngineCore/Subtitle/SubtitleDecoder.swift +++ b/Sources/LumeEngineCore/Subtitle/SubtitleDecoder.swift @@ -3,13 +3,25 @@ import Foundation /// Text subtitle decode stage: `Channel` in, `SubtitleStore` out. /// Runs `avcodec_decode_subtitle2`, so every FFmpeg text subtitle codec -/// (subrip, ass, webvtt, mov_text, eia_608 CC) feeds one cue model. -/// Bitmap codecs (PGS/DVB/VobSub) are recognized but skipped for now. +/// (subrip, ass, webvtt, mov_text, eia_608 CC) feeds one cue model — and +/// so does the closed-caption lane, whose packets come from the video decoder +/// (`ClosedCaptionTap`) rather than the demuxer. Each decoded update goes to +/// the store as a `SubtitleEvent`; how long it shows is the store's policy. +/// Bitmap codecs (DVB, PGS, VobSub) produce cues with `images` instead of +/// text (`SubtitleBitmap`). public final class SubtitleDecoder: @unchecked Sendable { public let store: SubtitleStore private let parameters: CodecParameters private let input: Channel + private let codecOptions: [String: String] + /// Each update is the whole screen (closed captions, bitmap formats) — + /// see `SubtitleEvent.replacesPrevious`. + private let replacesPrevious: Bool + /// CEA-608 captions run in FFmpeg's real-time mode (below), which stamps + /// an update with the time of the *previous* one; the packet that caused + /// it carries the time it actually shows. + private let isClosedCaptions: Bool private let lock = NSCondition() private var started = false @@ -17,11 +29,39 @@ public final class SubtitleDecoder: @unchecked Sendable { private var stopRequested = false private var codecContext: UnsafeMutablePointer? + /// Decode-thread-only: seek generation of the last packet. + private var currentSerial: UInt64? - public init(parameters: CodecParameters, input: Channel, store: SubtitleStore = SubtitleStore()) { + public convenience init(parameters: CodecParameters, input: Channel, store: SubtitleStore = SubtitleStore()) { + self.init(parameters: parameters, input: input, store: store, codecOptions: [:]) + } + + /// `codecOptions` go to `avcodec_open2` (the caption lane picks its field). + init(parameters: CodecParameters, input: Channel, store: SubtitleStore, codecOptions: [String: String]) { self.parameters = parameters self.input = input self.store = store + let codecID = parameters.raw.pointee.codec_id + isClosedCaptions = codecID == AV_CODEC_ID_EIA_608 + replacesPrevious = Self.replacesPrevious(codecID) + var options = codecOptions + if isClosedCaptions, options["real_time"] == nil { + // Real-time mode emits each screen change as it is decoded, open + // ended, and a clear as an empty update. The default buffered mode + // emits a caption only once the next change has ended it — after + // its own duration, by which time playback (a fraction of a second + // behind decode) has already passed most of it. + options["real_time"] = "1" + } + self.codecOptions = options + } + + /// Formats where every update describes the whole screen and stays until + /// the next one: closed captions, teletext, and the bitmap formats. + static func replacesPrevious(_ codecID: AVCodecID) -> Bool { + if codecID == AV_CODEC_ID_EIA_608 || codecID == AV_CODEC_ID_DVB_TELETEXT { return true } + guard let descriptor = avcodec_descriptor_get(codecID) else { return false } + return descriptor.pointee.props & AV_CODEC_PROP_BITMAP_SUB != 0 } public func start() { @@ -122,19 +162,35 @@ public final class SubtitleDecoder: @unchecked Sendable { private func decode(packet: Packet) { guard let context = codecContext else { return } + // Seek boundary: decoders with screen state (caption roll-up rows, + // PGS/DVB object and palette sets) must not carry it across. + if let serial = currentSerial, serial != packet.serial { + avcodec_flush_buffers(context) + } + currentSerial = packet.serial + var subtitle = AVSubtitle() var gotSubtitle: Int32 = 0 let result = avcodec_decode_subtitle2(context, &subtitle, &gotSubtitle, packet.raw) guard result >= 0, gotSubtitle != 0 else { return } defer { avsubtitle_free(&subtitle) } - // Packet timestamps are engine µs; display offsets are milliseconds. - let basePTS = MediaTime.isValid(packet.pts) ? packet.pts : 0 - let start = basePTS + Int64(subtitle.start_display_time) * 1_000 - var end = basePTS + Int64(subtitle.end_display_time) * 1_000 - if subtitle.end_display_time == 0 || end <= start { - end = packet.duration > 0 ? basePTS + packet.duration : start + 3_000_000 + // Packet timestamps are engine µs, and so is `subtitle.pts` + // (`pkt_timebase` is µs; PGS takes it from its presentation segment); + // display offsets are milliseconds. + let base: Int64 + if !isClosedCaptions, MediaTime.isValid(subtitle.pts) { + base = subtitle.pts + } else { + base = MediaTime.isValid(packet.pts) ? packet.pts : 0 } + let span = SubtitleEvent.span( + base: base, + startDisplay: isClosedCaptions ? 0 : subtitle.start_display_time, + endDisplay: subtitle.end_display_time, + packetDuration: packet.duration, + replacesPrevious: replacesPrevious + ) var lines: [String] = [] for i in 0..= 0 else { + var options: OpaquePointer? + for (key, value) in codecOptions { + av_dict_set(&options, key, value, 0) + } + defer { av_dict_free(&options) } + guard avcodec_open2(context, codec, &options) >= 0 else { var pointer: UnsafeMutablePointer? = context avcodec_free_context(&pointer) return false diff --git a/Sources/LumeEngineCore/Subtitle/SubtitleTimeline.swift b/Sources/LumeEngineCore/Subtitle/SubtitleTimeline.swift new file mode 100644 index 0000000..cbcee35 --- /dev/null +++ b/Sources/LumeEngineCore/Subtitle/SubtitleTimeline.swift @@ -0,0 +1,259 @@ +import Foundation + +/// One display update from a subtitle decoder on the engine timeline (µs), +/// before the timeline decides how long it stays up. +struct SubtitleEvent: Sendable, Equatable { + var start: Int64 + /// `nil` when the source gives no end. DVB, PGS and closed captions say + /// exactly that — "until the next update" — and a decoder that turned it + /// into a number would leave the text up for 49 days (`UINT32_MAX` ms). + var end: Int64? + /// Display text, trimmed. Empty — with no images either — on a replacing + /// track means "clear the screen", which is how those formats end what is + /// showing. + var text: String + /// The track describes the whole screen with every update (closed + /// captions, DVB, PGS, DVD): an update ends everything shown before it. + /// Text formats (SRT, ASS, WebVTT, mov_text) overlap freely instead, each + /// cue carrying its own end. + var replacesPrevious: Bool + /// Bitmaps (DVB, PGS, DVD). + var images: [SubtitleImage] = [] + + var isEmpty: Bool { text.isEmpty && images.isEmpty } + + /// Where an `AVSubtitle` sits on the engine timeline. `base` is the + /// subtitle's PTS in engine µs, display times are milliseconds after it. + /// `nil` end means open: shows until replaced (see `SubtitleTimeline`). + static func span( + base: Int64, + startDisplay: UInt32, + endDisplay: UInt32, + packetDuration: Int64, + replacesPrevious: Bool + ) -> (start: Int64, end: Int64?) { + let start = base + Int64(startDisplay) * 1_000 + // UINT32_MAX is FFmpeg's "no end" (PGS always, closed captions in + // real-time mode, DVB without a computed end). + guard endDisplay != .max else { return (start, nil) } + let end = base + Int64(endDisplay) * 1_000 + if endDisplay != 0, end > start { return (start, end) } + // No usable end. A replacing track's update lasts until the next one; + // a text cue without one gets its packet's duration, or a reading-time + // guess, as it always has. + if replacesPrevious { return (start, nil) } + return (start, packetDuration > 0 ? base + packetDuration : start + 3_000_000) + } +} + +/// The cue-lifecycle policy behind `SubtitleStore`: a timeline of cues sorted +/// by start, deciding how long each one shows. Pure value type — the store +/// adds the lock. +/// +/// The failure modes it exists for, all of which used to reach the screen: +/// - Cues without an end (PGS, closed captions, DVB "until the next page") +/// stayed up for good, piling onto every later cue. +/// - The empty update that DVB, PGS and closed captions send to clear the +/// screen was dropped as "no text", so the previous cue stayed for its full +/// timeout (DVB pages time out after tens of seconds). +/// - Every cue a live session ever decoded was kept: a day of roll-up +/// captions is hundreds of thousands of them. +/// +/// The rules: +/// - An open cue ends where the next update of its track starts, or after +/// `openEndedLimit` when none comes. +/// - On a replacing track every update — cue or clear — ends everything +/// shown before it, and when one arrives out of order (a range re-read +/// after a seek) the update already stored after it bounds it in turn. +/// - Text tracks keep their overlaps; only their open cues are ended by the +/// next cue. +/// - An update already held is dropped, so re-reading a range (backward +/// seek, backfill seek) does not double its lines. On a replacing track a +/// different update at the same start replaces the stored one: the screen +/// at one instant has one state, and the latest decode of it wins. +/// - At most `capacity` cues, holding at most `imageByteBudget` of bitmaps, +/// are kept; the ones farthest from where cues are being added go first. +struct SubtitleTimeline: Sendable { + /// How long an open cue stays without a successor. Caption decoders on + /// televisions erase a caption left standing by about this long without + /// caption data; a longer limit leaves stale text over the next scene + /// when a clear is lost, a shorter one cuts captions that are still + /// being spoken. + static let openEndedLimit: Int64 = 16_000_000 + + /// Cues kept. Far above any film's subtitle file (a few thousand cues); + /// for a live track it is half an hour or more of roll-up captions. + static let capacity = 8_192 + + /// Bitmap memory kept. A full-HD PGS line is the better part of a + /// megabyte decoded, so a cue count alone would let a film's bitmap track + /// grow to gigabytes; this keeps the last minute or more of a live DVB + /// page, and a backward seek re-decodes whatever was dropped. + static let imageByteBudget = 64 << 20 + + /// How far back from an insertion point active cues are looked for. + /// Overlap is rare and short-lived; a replacing track has at most one. + private static let scanLimit = 64 + + private struct Entry: Sendable { + var cue: SubtitleCue + /// The end is the limit, not the source's: the next update or a seek + /// may still bound it. + var provisional: Bool + + var imageBytes: Int { cue.images.reduce(0) { $0 + $1.byteCount } } + } + + private var entries: [Entry] = [] + private var nextID: UInt64 = 0 + /// Sum of `imageBytes` over `entries`. + private(set) var imageBytes = 0 + + var count: Int { entries.count } + + mutating func apply(_ event: SubtitleEvent) { + if event.isEmpty { + // A clear. Text tracks have no such thing; an empty text cue is + // just nothing to show. + guard event.replacesPrevious else { return } + end(at: event.start, onlyProvisional: false) + return + } + + if isDuplicate(event) { return } + if event.replacesPrevious { removeEntries(startingAt: event.start) } + // What this update takes over from: everything on screen for a + // replacing track, only the cues with no end of their own otherwise. + end(at: event.start, onlyProvisional: !event.replacesPrevious) + + let index = insertionIndex(for: event.start) + var end = event.end ?? event.start + Self.openEndedLimit + if (event.replacesPrevious || event.end == nil), index < entries.count { + // Arrived out of order: the next update already stored ends it. + end = min(end, entries[index].cue.start) + } + guard end > event.start else { return } + + let entry = Entry( + cue: SubtitleCue(id: nextID, start: event.start, end: end, text: event.text, images: event.images), + provisional: event.end == nil + ) + entries.insert(entry, at: index) + imageBytes += entry.imageBytes + nextID += 1 + trim(keeping: index) + } + + /// Ends open cues at a discontinuity (a seek to `time`): the screen state + /// after it is unknown until the source sends its next update, and + /// showing the stale one meanwhile is worse than showing nothing. Cues + /// with an end from the source are facts about the timeline and stay. + mutating func closeOpenCues(at time: Int64) { + end(at: time, onlyProvisional: true) + } + + mutating func removeAll() { + entries.removeAll() + imageBytes = 0 + } + + /// All cues covering `time` (engine µs), ordered by start. + func activeCues(at time: Int64) -> [SubtitleCue] { + var active: [SubtitleCue] = [] + var index = insertionIndex(for: time) - 1 + var scanned = 0 + while index >= 0 && scanned < Self.scanLimit { + let cue = entries[index].cue + if cue.start <= time && time < cue.end { + active.append(cue) + } + index -= 1 + scanned += 1 + } + return active.reversed() + } + + // MARK: Internals + + /// First entry starting after `time`. + private func insertionIndex(for time: Int64) -> Int { + var low = 0 + var high = entries.count + while low < high { + let mid = (low + high) / 2 + if entries[mid].cue.start <= time { low = mid + 1 } else { high = mid } + } + return low + } + + private func isDuplicate(_ event: SubtitleEvent) -> Bool { + var index = insertionIndex(for: event.start) - 1 + while index >= 0 && entries[index].cue.start == event.start { + let entry = entries[index] + // Bitmaps decoded twice are different objects; where and how big + // they are identifies them. + if entry.cue.text == event.text, + entry.cue.images.map(\.rect) == event.images.map(\.rect), + entry.cue.images.map(\.image.width) == event.images.map(\.image.width) { + // A stored end may have been bounded by a successor since, so + // it only has to match where both ends came from the source. + if event.replacesPrevious || event.end == nil || entry.provisional { return true } + if entry.cue.end == event.end { return true } + } + index -= 1 + } + return false + } + + private mutating func removeEntries(startingAt start: Int64) { + var index = insertionIndex(for: start) - 1 + while index >= 0 && entries[index].cue.start == start { + imageBytes -= entries.remove(at: index).imageBytes + index -= 1 + } + } + + /// Ends the cues that started before `time` and still show at it. + private mutating func end(at time: Int64, onlyProvisional: Bool) { + var index = insertionIndex(for: time) - 1 + var scanned = 0 + while index >= 0 && scanned < Self.scanLimit { + let entry = entries[index] + if entry.cue.start < time, entry.cue.end > time, !onlyProvisional || entry.provisional { + entries[index].cue = entry.cue.ending(at: time) + } + index -= 1 + scanned += 1 + } + } + + /// Drops cues beyond `capacity` or `imageByteBudget` from the end farther + /// from `index`, where cues are being added — the front on a live + /// timeline, the back after a seek far into the past of a long file. Over + /// capacity it drops an eighth more than needed, so the shift is not paid + /// on every insert. The cue just added always stays. + private mutating func trim(keeping index: Int) { + var kept = index + if entries.count > Self.capacity { + drop(entries.count - Self.capacity + Self.capacity / 8, keeping: &kept) + } + while imageBytes > Self.imageByteBudget, entries.count > 1 { + drop(1, keeping: &kept) + } + } + + private mutating func drop(_ count: Int, keeping kept: inout Int) { + let count = min(count, entries.count - 1) + guard count > 0 else { return } + let removed: ArraySlice + if kept < entries.count / 2 { + removed = entries[(entries.count - count)...] + entries.removeLast(count) + } else { + removed = entries[.. AudioLayoutPolicy.Plan { + AudioLayoutPolicy.plan(source: source, routeChannels: route) + } + + @Test("speaker sets are FFmpeg's native layouts") + func layoutsMatchFFmpeg() { + #expect(SpeakerSet.mono.rawValue == layoutMask("mono")) + #expect(SpeakerSet.stereo.rawValue == layoutMask("stereo")) + #expect(SpeakerSet.surround51.rawValue == layoutMask("5.1(side)")) + #expect(SpeakerSet.surround71.rawValue == layoutMask("7.1")) + } + + @Test("multichannel goes out discrete when the route carries it") + func passthroughWhenItFits() { + for route in [6, 8] { + #expect(plan(.surround51, route: route) == .init(output: .surround51, remixes: false)) + #expect(plan(Self.surround51Back, route: route) == .init(output: Self.surround51Back, remixes: false)) + } + #expect(plan(.surround71, route: 8) == .init(output: .surround71, remixes: false)) + for route in [2, 6, 8] { + #expect(plan(.stereo, route: route) == .init(output: .stereo, remixes: false)) + } + } + + @Test("a narrower route gets the widest standard layout both sides fill") + func foldsDownToTheRoute() { + #expect(plan(.surround51, route: 2) == .init(output: .stereo, remixes: true)) + #expect(plan(Self.surround51Back, route: 2) == .init(output: .stereo, remixes: true)) + #expect(plan(.surround71, route: 2) == .init(output: .stereo, remixes: true)) + #expect(plan(.surround71, route: 6) == .init(output: .surround51, remixes: true)) + #expect(plan(.surround71, route: 7) == .init(output: .surround51, remixes: true)) + // Odd widths fall to the standard layout below them. + #expect(plan(.surround51, route: 4) == .init(output: .stereo, remixes: true)) + } + + @Test("no LFE channel is invented") + func noInventedLFE() { + #expect(plan(Self.surround70, route: 6) == .init(output: SpeakerSet.surround51.subtracting(.lowFrequency), remixes: true)) + } + + @Test("mono reaches both speakers of any route wider than one") + func monoBecomesStereo() { + for route in [2, 6, 8] { + #expect(plan(.mono, route: route) == .init(output: .stereo, remixes: true)) + } + #expect(plan(.mono, route: 1) == .init(output: .mono, remixes: false)) + #expect(plan(.mono, route: 0) == .init(output: .mono, remixes: false)) + #expect(plan(.stereo, route: 1) == .init(output: .mono, remixes: true)) + } + + @Test("channels the renderer cannot name are remixed, not passed untagged") + func unnameableChannelsAreRemixed() { + #expect(plan(Self.stereoDownmix, route: 8) == .init(output: .stereo, remixes: true)) + } +} + +@Suite("DownmixMatrix") +struct DownmixMatrixTests { + private let minus3dB = 0.707_106_781_186_547_6 + + /// The coefficient of input speaker `input` in output speaker `output`. + private func weight( + _ matrix: [[Double]], from source: SpeakerSet, to target: SpeakerSet, + _ input: SpeakerSet, into output: SpeakerSet + ) -> Double { + func index(_ speaker: SpeakerSet, in set: SpeakerSet) -> Int { + // Native order is ascending bit order. + (set.rawValue & (speaker.rawValue - 1)).nonzeroBitCount + } + return matrix[index(output, in: target)][index(input, in: source)] + } + + @Test("5.1 to stereo: ITU fold-down, centre in both, LFE omitted, no cross-feed") + func surround51ToStereo() throws { + let source = SpeakerSet.surround51 + let matrix = try #require(DownmixMatrix.coefficients(from: source, to: .stereo)) + #expect(matrix.count == 2 && matrix.allSatisfy { $0.count == 6 }) + func w(_ input: SpeakerSet, _ output: SpeakerSet) -> Double { + weight(matrix, from: source, to: .stereo, input, into: output) + } + #expect(w(.frontLeft, .frontLeft) == 1) + #expect(w(.frontRight, .frontRight) == 1) + #expect(w(.frontLeft, .frontRight) == 0) + #expect(w(.frontRight, .frontLeft) == 0) + #expect(abs(w(.frontCenter, .frontLeft) - minus3dB) < 1e-9) + #expect(abs(w(.frontCenter, .frontRight) - minus3dB) < 1e-9) + #expect(abs(w(.sideLeft, .frontLeft) - minus3dB) < 1e-9) + #expect(abs(w(.sideRight, .frontRight) - minus3dB) < 1e-9) + #expect(w(.sideLeft, .frontRight) == 0) + #expect(w(.lowFrequency, .frontLeft) == 0) + #expect(w(.lowFrequency, .frontRight) == 0) + } + + @Test("7.1 to stereo: every surround folds into its own side") + func surround71ToStereo() throws { + let source = SpeakerSet.surround71 + let matrix = try #require(DownmixMatrix.coefficients(from: source, to: .stereo)) + func w(_ input: SpeakerSet, _ output: SpeakerSet) -> Double { + weight(matrix, from: source, to: .stereo, input, into: output) + } + for (input, side) in [(SpeakerSet.backLeft, SpeakerSet.frontLeft), (.sideLeft, .frontLeft), + (.backRight, .frontRight), (.sideRight, .frontRight)] { + #expect(abs(w(input, side) - minus3dB) < 1e-9) + } + #expect(w(.backLeft, .frontRight) == 0) + #expect(abs(w(.frontCenter, .frontLeft) - minus3dB) < 1e-9) + #expect(abs(w(.frontCenter, .frontRight) - minus3dB) < 1e-9) + } + + @Test("7.1 to 5.1: fronts, centre and LFE pass straight; backs join the sides") + func surround71To51() throws { + let source = SpeakerSet.surround71 + let target = SpeakerSet.surround51 + let matrix = try #require(DownmixMatrix.coefficients(from: source, to: target)) + func w(_ input: SpeakerSet, _ output: SpeakerSet) -> Double { + weight(matrix, from: source, to: target, input, into: output) + } + for speaker in [SpeakerSet.frontLeft, .frontRight, .frontCenter, .lowFrequency] { + #expect(w(speaker, speaker) == 1) + } + #expect(w(.backLeft, .sideLeft) > 0.5) + #expect(w(.backRight, .sideRight) > 0.5) + #expect(w(.frontCenter, .frontLeft) == 0, "a route with a centre speaker keeps dialogue on it") + } + + @Test("mono to stereo: equal power on both sides") + func monoToStereo() throws { + let matrix = try #require(DownmixMatrix.coefficients(from: .mono, to: .stereo)) + #expect(matrix.count == 2) + #expect(abs(matrix[0][0] - minus3dB) < 1e-9) + #expect(abs(matrix[1][0] - minus3dB) < 1e-9) + } + + @Test("dialogue survives every stereo fold-down") + func dialogueNeverDropped() throws { + let sources: [SpeakerSet] = [ + [.frontLeft, .frontRight, .frontCenter], + .surround51, + [.frontLeft, .frontRight, .frontCenter, .lowFrequency, .backLeft, .backRight], + .surround71, + SpeakerSet.surround51.union(SpeakerSet(rawValue: 1 << 8)), // 6.1, back centre + ] + for source in sources { + let plan = AudioLayoutPolicy.plan(source: source, routeChannels: 2) + let matrix = try #require(DownmixMatrix.coefficients(from: source, to: plan.output)) + let center = (source.rawValue & (SpeakerSet.frontCenter.rawValue - 1)).nonzeroBitCount + #expect(matrix[0][center] > 0.5 && matrix[1][center] > 0.5, "centre lost folding \(source.rawValue)") + // Every input channel except the LFE reaches the output. + for input in 0.. 0, "channel \(input) of \(source.rawValue) dropped") + } + } + } +} + +@Suite("AudioRoute", .serialized) +struct AudioRouteDecodeTests { + /// Decodes an audio-only fixture with the decoder following `route`. + /// `onFrame` runs on the drain thread for each frame, in order — the place + /// to change the route mid-stream. + private func decode( + fixture: String, + route: AudioOutputRoute, + frameQueue: Int = 256, + onFrame: @escaping @Sendable (Int) -> Void = { _ in } + ) async throws -> [AudioFrame] { + let demuxer = Demuxer(url: try Fixtures.path(fixture)) + var events = demuxer.events.makeAsyncIterator() + demuxer.start() + guard case .opened(let info)? = await events.next() else { + demuxer.shutdown() + throw EngineError(code: .openFailed, message: "\(fixture) failed to open") + } + defer { demuxer.shutdown() } + + let track = try #require(info.audioTracks.first) + let parameters = try #require(demuxer.codecParameters(forStream: track.index)) + let packets = Channel(capacity: 4096) + let frames = Channel(capacity: frameQueue, measure: { $0.duration }) + demuxer.attach(channel: packets, toStream: track.index) + + let decoder = AudioDecoder(parameters: parameters, input: packets, output: frames, route: route) + var decodeEvents = decoder.events.makeAsyncIterator() + decoder.start() + demuxer.resume() + defer { decoder.shutdown() } + + let collector = ChannelDrain(frames, into: [AudioFrame]()) { collected, frame in + collected.append(frame) + onFrame(collected.count) + } + while let event = await events.next() { + if case .endOfStream = event { + decoder.signalEndOfStream() + break + } + } + guard case .endOfStream? = await decodeEvents.next() else { + throw EngineError(code: .decodeFailed, message: "\(fixture): decoder never reached EOF") + } + decoder.shutdown() + return await collector.value + } + + /// Per-channel peak amplitude over all frames. + private func peaks(_ frames: [AudioFrame]) -> [Float] { + guard let channels = frames.first?.channels else { return [] } + var peaks = [Float](repeating: 0, count: channels) + for frame in frames { + let samples = frame.samples + for index in 0.. 0.05) + let stereoPeaks = peaks(stereo) + #expect(abs(stereoPeaks[0] - stereoPeaks[1]) < 1e-4) + #expect(abs(stereoPeaks[0] / center - 0.7071) < 0.01, "centre at \(stereoPeaks[0] / center) of its level") + // On a 5.1 route the centre stays on the centre speaker. + #expect(abs(peaks(fiveOne)[2] - center) < 1e-4) + } + + @Test("TrueHD 5.1 at route widths 8 and 2", .timeLimit(.minutes(1))) + func truehdAtRouteWidths() async throws { + let wide = try await decode(fixture: "truehd.mkv", route: AudioOutputRoute(outputChannels: 8)) + #expect(wide.allSatisfy { $0.channels == 6 && $0.channelBitmap == SpeakerSet.surround51.rawValue }) + + let stereo = try await decode(fixture: "truehd.mkv", route: AudioOutputRoute(outputChannels: 2)) + #expect(!stereo.isEmpty) + #expect(stereo.allSatisfy { $0.channels == 2 }) + let center = peaks(wide)[2] + let stereoPeaks = peaks(stereo) + #expect(center > 0.05) + #expect(abs(stereoPeaks[0] / center - 0.7071) < 0.01 && abs(stereoPeaks[1] / center - 0.7071) < 0.01) + let seconds = Double(stereo.reduce(0) { $0 + $1.sampleCount }) / Double(stereo[0].sampleRate) + #expect(abs(seconds - 4.0) < 0.1) + } + + @Test("mono goes out on both speakers", .timeLimit(.minutes(1))) + func monoToStereo() async throws { + // basic.mp4's AAC track is a mono sine. + let mono = try await decode(fixture: "basic.mp4", route: AudioOutputRoute(outputChannels: 1)) + #expect(mono.first?.channels == 1) + let stereo = try await decode(fixture: "basic.mp4", route: AudioOutputRoute(outputChannels: 2)) + #expect(stereo.allSatisfy { $0.channels == 2 && $0.channelBitmap == SpeakerSet.stereo.rawValue }) + let level = peaks(mono)[0] + let sides = peaks(stereo) + #expect(level > 0.05) + #expect(abs(sides[0] - sides[1]) < 1e-4) + #expect(abs(sides[0] / level - 0.7071) < 0.01) + } + + @Test("a route change mid-stream switches layout at a frame boundary, losing nothing", .timeLimit(.minutes(1))) + func routeChangeMidStream() async throws { + let route = AudioOutputRoute(outputChannels: 8) + // A short frame queue keeps the decoder from finishing the file + // before the route changes. + let frames = try await decode(fixture: "surround71.mkv", route: route, frameQueue: 4) { count in + if count == 40 { route.update(outputChannels: 2) } + } + let widths = frames.map(\.channels) + let switchIndex = try #require(widths.firstIndex(of: 2), "the layout must follow the route") + #expect(switchIndex >= 40, "frames decoded for the old route are not rewritten") + #expect(widths[.. AudioLossDetector.Observation { + AudioLossDetector.Observation( + hasAudioLane: true, audioEnded: false, audioRendererFailed: false, + clockRunning: true, playhead: playhead, + audioAhead: audioAhead, videoAhead: videoAhead, serial: serial + ) + } + + /// Feeds `observation` with the playhead advanced in 0.2 s ticks over + /// `seconds`; returns every reason the detector reported. + private func run( + _ detector: inout AudioLossDetector, from start: Double, seconds: Double, + _ observation: (Double) -> AudioLossDetector.Observation + ) -> [AudioLossReason] { + var reported: [AudioLossReason] = [] + var playhead = start + while playhead <= start + seconds + 1e-9 { + if let reason = detector.observe(observation(playhead)) { reported.append(reason) } + playhead += 0.2 + } + return reported + } + + @Test("no lane is absent; a lane starts healthy") + func absentAndHealthy() { + var detector = AudioLossDetector() + var observation = playing(at: 0) + observation.hasAudioLane = false + #expect(detector.observe(observation) == nil) + #expect(detector.health == .absent) + #expect(detector.observe(playing(at: 0)) == nil) + #expect(detector.health == .healthy) + } + + @Test("audio missing at a running playhead while video plays is reported once, after the threshold") + func starvationReportedOnce() { + var detector = AudioLossDetector() + let early = run(&detector, from: 0, seconds: 1.6) { playing(at: $0, audioAhead: false) } + #expect(early.isEmpty, "not before \(AudioLossDetector.threshold) s") + #expect(detector.health == .healthy) + let later = run(&detector, from: 1.8, seconds: 3) { playing(at: $0, audioAhead: false) } + #expect(later == [.starved]) + #expect(detector.health == .lost(.starved)) + } + + @Test("audio coming back clears the verdict, and a second episode is reported again") + func recovery() { + var detector = AudioLossDetector() + _ = run(&detector, from: 0, seconds: 3) { playing(at: $0, audioAhead: false) } + #expect(detector.health == .lost(.starved)) + #expect(detector.observe(playing(at: 3.2)) == nil) + #expect(detector.health == .healthy) + let again = run(&detector, from: 3.4, seconds: 3) { playing(at: $0, audioAhead: false) } + #expect(again == [.starved]) + } + + @Test("moments without audio that belong to playback are not a loss") + func legitimateSilence() { + // Video dry too: the whole pipeline is starving, which is buffering. + var detector = AudioLossDetector() + #expect(run(&detector, from: 0, seconds: 5) { playing(at: $0, audioAhead: false, videoAhead: false) }.isEmpty) + // Audio-only source: nothing to compare against. + #expect(run(&detector, from: 0, seconds: 5) { playing(at: $0, audioAhead: false, videoAhead: nil) }.isEmpty) + // The audio track ended (VOD whose audio is shorter than its video). + #expect(run(&detector, from: 0, seconds: 5) { + var observation = playing(at: $0, audioAhead: false) + observation.audioEnded = true + return observation + }.isEmpty) + // Paused or buffering: the clock does not run. + #expect(run(&detector, from: 0, seconds: 5) { + var observation = playing(at: $0, audioAhead: false) + observation.clockRunning = false + return observation + }.isEmpty) + #expect(detector.health == .healthy) + } + + @Test("a pause restarts the count; a seek voids the episode") + func pauseAndSeek() { + var detector = AudioLossDetector() + _ = run(&detector, from: 0, seconds: 1.6) { playing(at: $0, audioAhead: false) } + var paused = playing(at: 1.6, audioAhead: false) + paused.clockRunning = false + #expect(detector.observe(paused) == nil) + #expect(run(&detector, from: 1.6, seconds: 1.6) { playing(at: $0, audioAhead: false) }.isEmpty) + + _ = run(&detector, from: 3.4, seconds: 3) { playing(at: $0, audioAhead: false) } + #expect(detector.health == .lost(.starved)) + // Seek back: new serial, playhead earlier — a fresh start. + #expect(detector.observe(playing(at: 0.5, audioAhead: false, serial: 1)) == nil) + #expect(detector.health == .healthy) + } + + @Test("decoder and renderer failures are permanent until the lane is rebuilt") + func permanentLosses() { + var detector = AudioLossDetector() + _ = detector.observe(playing(at: 0)) + detector.decoderFailed() + #expect(detector.health == .lost(.decoderFailed)) + // Audio enqueued before the failure still plays out; the verdict stands. + #expect(detector.observe(playing(at: 0.2)) == nil) + #expect(detector.health == .lost(.decoderFailed)) + detector.laneReplaced() + #expect(detector.health == .healthy) + + var failed = playing(at: 1) + failed.audioRendererFailed = true + #expect(detector.observe(failed) == nil) + #expect(detector.health == .lost(.rendererFailed)) + #expect(detector.observe(playing(at: 1.2, serial: 4)) == nil) + #expect(detector.health == .lost(.rendererFailed)) + } +} + +@Suite("AudioRouteSession", .serialized) +struct AudioRouteSessionTests { + private func eventually( + timeout: TimeInterval = 10, + _ condition: () async -> Bool + ) async -> Bool { + let deadline = Date(timeIntervalSinceNow: timeout) + while Date() < deadline { + if await condition() { return true } + try? await Task.sleep(for: .milliseconds(100)) + } + return false + } + + private func makeSession(route: AudioOutputRoute, enableAudio: Bool = true) -> PlayerSession { + var configuration = PlayerConfiguration() + configuration.muted = true + configuration.bufferTarget = 0.5 + configuration.enableAudio = enableAudio + configuration.audioOutputRoute = route + return PlayerSession(configuration: configuration) + } + + @Test("a playing session refits its audio when the route narrows and widens", .timeLimit(.minutes(1))) + func followsRouteChanges() async throws { + let route = AudioOutputRoute(outputChannels: 8) + let session = makeSession(route: route) + _ = try await session.open(url: try Fixtures.path("surround51.ts")) + await session.play() + #expect(await eventually { await session.position > 1 }) + #expect(await session.audioOutput == AudioOutputLayout(sourceChannels: 6, outputChannels: 6, routeChannels: 8)) + + // HDMI receiver → AirPlay: the next decoded frame folds down to stereo. + route.update(outputChannels: 2) + let narrowed = await eventually { await session.audioOutput?.outputChannels == 2 } + #expect(narrowed, "audio must refit to the stereo route") + let before = await session.position + let continued = await eventually { await session.position > before + 1.5 } + #expect(continued, "playback must go on through the route change") + #expect(!session.renderer.audioRendererFailed) + #expect(await session.state == .playing) + #expect(await session.audioHealth == .healthy) + + // And back to surround. + route.update(outputChannels: 6) + #expect(await eventually { await session.audioOutput?.outputChannels == 6 }) + let resumed = await session.position + #expect(await eventually { await session.position > resumed + 1 }) + #expect(!session.renderer.audioRendererFailed) + + let diagnostics = await session.diagnostics + #expect(diagnostics.audioOutput?.routeChannels == 6) + #expect(diagnostics.audioRoute.changes == 2) + #expect(diagnostics.description.contains("aout=6>6ch@6")) + await session.shutdown() + } + + @Test("audio that stops reaching the renderer while video plays is reported", .timeLimit(.minutes(1))) + func reportsLostAudio() async throws { + let session = makeSession(route: AudioOutputRoute(outputChannels: 2)) + let errors = Mutex<[EngineError.Code]>([]) + let listener = Task { [events = session.events] in + for await event in events { + if case .error(let error) = event { errors.withLock { $0.append(error.code) } } + } + } + _ = try await session.open(url: try Fixtures.path("surround51.ts")) + await session.play() + #expect(await eventually { await session.position > 1 }) + #expect(await session.audioHealth == .healthy) + + // The renderer stops taking audio; the clock and the picture go on. + session.renderer.audioRenderer.stopRequestingMediaData() + let lost = await eventually(timeout: 15) { await session.audioHealth == .lost(.starved) } + let health = await session.audioHealth + #expect(lost, "lost audio must be detected, health is \(health)") + // The event travels through the stream after the verdict is set. + let reported = await eventually(timeout: 2) { errors.withLock { $0.contains(.audioLost) } } + let codes = errors.withLock { $0 } + #expect(reported && codes.filter { $0 == .audioLost }.count == 1, "errors: \(codes)") + #expect(await session.state == .playing, "video keeps playing") + #expect(await session.diagnostics.description.contains("audio=lost(starved)")) + + await session.shutdown() + listener.cancel() + } + + @Test("a session without audio reports none", .timeLimit(.minutes(1))) + func noAudioLane() async throws { + let session = makeSession(route: AudioOutputRoute(outputChannels: 2), enableAudio: false) + _ = try await session.open(url: try Fixtures.path("basic.mp4")) + await session.play() + #expect(await eventually { await session.position > 0.5 }) + #expect(await session.audioHealth == .absent) + #expect(await session.audioOutput == nil) + await session.shutdown() + } +} + +private extension Array { + subscript(safe index: Int) -> Element? { + indices.contains(index) ? self[index] : nil + } +} + +/// FFmpeg's mask for a named layout — the reference the speaker-set +/// constants are checked against. +private func layoutMask(_ name: String) -> UInt64 { + var layout = AVChannelLayout() + defer { av_channel_layout_uninit(&layout) } + guard av_channel_layout_from_string(&layout, name) >= 0 else { return 0 } + return layout.u.mask +} diff --git a/Tests/LumeEngineCoreTests/BitmapSubtitleTests.swift b/Tests/LumeEngineCoreTests/BitmapSubtitleTests.swift new file mode 100644 index 0000000..6ea7db1 --- /dev/null +++ b/Tests/LumeEngineCoreTests/BitmapSubtitleTests.swift @@ -0,0 +1,178 @@ +import CoreGraphics +import Foundation +import Testing +@testable import LumeEngineCore + +/// Reads one pixel of a `SubtitleBitmap` image as (r, g, b, a), premultiplied. +private func pixel(of image: CGImage, x: Int, y: Int) -> (r: Int, g: Int, b: Int, a: Int)? { + guard let data = image.dataProvider?.data as Data? else { return nil } + let offset = y * image.bytesPerRow + x * 4 + guard offset + 4 <= data.count else { return nil } + // byteOrder32Little + premultipliedFirst: B, G, R, A in memory. + return (Int(data[offset + 2]), Int(data[offset + 1]), Int(data[offset]), Int(data[offset + 3])) +} + +@Suite("Bitmap subtitle conversion") +struct BitmapSubtitleConversionTests { + @Test("palette colours are premultiplied") + func premultiply() { + #expect(SubtitleBitmap.premultiply(0xFF_FF_FF_FF) == 0xFF_FF_FF_FF) + #expect(SubtitleBitmap.premultiply(0x00_FF_FF_FF) == 0) + // Half-transparent red: colour scaled by alpha, alpha kept. + #expect(SubtitleBitmap.premultiply(0x80_FF_00_00) == 0x80_80_00_00) + #expect(SubtitleBitmap.premultiply(0x40_80_40_20) == 0x40_20_10_08) + } + + @Test("indexed rows honour the stride; indices past the palette are transparent") + func indexedPixels() { + // Two rows of three pixels, stride four (one byte of padding each). + let indices: [UInt8] = [0, 1, 2, 9, 2, 1, 7, 9] + let palette: [UInt32] = [0x00_00_00_00, 0xFF_FF_FF_FF, 0x80_FF_00_00] + let pixels = indices.withUnsafeBufferPointer { + SubtitleBitmap.premultipliedPixels(indices: $0, width: 3, height: 2, stride: 4, palette: palette) + } + #expect(pixels == [0, 0xFF_FF_FF_FF, 0x80_80_00_00, 0x80_80_00_00, 0xFF_FF_FF_FF, 0]) + } + + @Test("an image round-trips as premultiplied BGRA") + func imageLayout() throws { + let image = try #require(SubtitleBitmap.makeImage(pixels: [0xFF_FF_FF_FF, 0x80_80_00_00], width: 2, height: 1)) + #expect(image.width == 2 && image.height == 1) + #expect(image.alphaInfo == .premultipliedFirst) + let white = try #require(pixel(of: image, x: 0, y: 0)) + #expect(white == (255, 255, 255, 255)) + let red = try #require(pixel(of: image, x: 1, y: 0)) + #expect(red == (128, 0, 0, 128)) + } + + @Test("rects are normalized to the authoring canvas") + func normalizedRect() { + let rect = SubtitleBitmap.normalizedRect(x: 480, y: 900, width: 960, height: 108, canvasWidth: 1920, canvasHeight: 1080) + #expect(rect == CGRect(x: 0.25, y: 0.8333333333333334, width: 0.5, height: 0.1)) + #expect(SubtitleBitmap.normalizedRect(x: 0, y: 0, width: 1, height: 1, canvasWidth: 0, canvasHeight: 0) == .zero) + } + + private func image(width: Int, height: Int) throws -> SubtitleImage { + let image = try #require(SubtitleBitmap.makeImage( + pixels: [UInt32](repeating: 0xFF_FF_FF_FF, count: width * height), width: width, height: height + )) + return SubtitleImage(image: image, rect: CGRect(x: 0.4, y: 0.9, width: 0.1, height: 0.05)) + } + + @Test("a bitmap update is a cue, not a clear, and a re-read does not double it") + func bitmapCuesInTimeline() throws { + var timeline = SubtitleTimeline() + let bitmap = try image(width: 8, height: 4) + timeline.apply(SubtitleEvent(start: 1_000_000, end: nil, text: "", replacesPrevious: true, images: [bitmap])) + timeline.apply(SubtitleEvent(start: 3_000_000, end: nil, text: "", replacesPrevious: true)) + let shown = timeline.activeCues(at: 2_000_000) + #expect(shown.count == 1) + #expect(shown.first?.images.map(\.rect) == [bitmap.rect]) + #expect(shown.first?.text == "") + #expect(timeline.activeCues(at: 3_000_000).isEmpty) + + // The same update decoded again makes a new CGImage of the same size. + let again = try image(width: 8, height: 4) + timeline.apply(SubtitleEvent(start: 1_000_000, end: nil, text: "", replacesPrevious: true, images: [again])) + #expect(timeline.count == 1) + } + + @Test("bitmap memory is bounded, dropping the oldest cues") + func imageBudget() throws { + var timeline = SubtitleTimeline() + // 1 MB per cue: the budget holds 64 of them. + let bitmap = try image(width: 512, height: 512) + for index in 0..<100 { + timeline.apply(SubtitleEvent( + start: Int64(index) * 1_000_000, end: nil, text: "", replacesPrevious: true, images: [bitmap] + )) + } + #expect(timeline.imageBytes <= SubtitleTimeline.imageByteBudget) + #expect(timeline.count == SubtitleTimeline.imageByteBudget / bitmap.byteCount) + #expect(timeline.activeCues(at: 99_500_000).count == 1) + #expect(timeline.activeCues(at: 500_000).isEmpty) + timeline.removeAll() + #expect(timeline.imageBytes == 0) + } +} + +@Suite("Bitmap subtitles in playback", .serialized) +struct BitmapSubtitlePlaybackTests { + private func eventually( + timeout: TimeInterval = 10, + _ condition: () async -> Bool + ) async -> Bool { + let deadline = Date(timeIntervalSinceNow: timeout) + while Date() < deadline { + if await condition() { return true } + try? await Task.sleep(for: .milliseconds(100)) + } + return false + } + + /// The fixtures carry the same two bitmaps (see `generate-fixtures.sh`): + /// 64x16 at (928, 1000) for 1–3 s, 48x8 at (936, 960) for 4–6 s, on a + /// 1920x1080 canvas over a 320x180 video — so a canvas taken from the + /// video instead of the subtitle would put them far off screen. + @Test( + "bitmap subtitles become positioned image cues", + .timeLimit(.minutes(1)), + arguments: ["pgs.mkv", "dvbsub.ts", "dvdsub.mkv"] + ) + func bitmapCues(fixture: String) async throws { + var configuration = PlayerConfiguration() + configuration.muted = true + configuration.bufferTarget = 0.3 + let session = PlayerSession(configuration: configuration) + let info = try await session.open(url: try Fixtures.path(fixture)) + let track = try #require(info.subtitleTracks.first) + + await session.selectSubtitleTrack(track.index) + let start = info.startTime + let decoded = await eventually { + session.subtitles.activeCues(at: start + 5_000_000).first?.images.isEmpty == false + } + #expect(decoded, "\(fixture): cues should decode, store has \(session.subtitles.count)") + + let first = try #require(session.subtitles.activeCues(at: start + 2_000_000).first, "\(fixture): first cue") + #expect(first.text.isEmpty, "bitmap cues carry no text, so text consumers show nothing") + let image = try #require(first.images.first) + #expect(first.images.count == 1) + #expect(abs(first.start - (start + 1_000_000)) <= 40_000, "\(fixture): starts at \(first.start - start)") + #expect(abs(first.end - (start + 3_000_000)) <= 40_000, "\(fixture): ends at \(first.end - start)") + + // Decoders may crop fully transparent borders (FFmpeg's DVD decoder + // drops the 8 transparent columns); the visible part must still land + // within a few canvas pixels of the original. + let tolerance = 4.0 / 1080 + #expect(abs(image.rect.maxX - 992.0 / 1920) <= tolerance, "\(fixture): \(image.rect)") + #expect(abs(image.rect.maxY - 1016.0 / 1080) <= tolerance, "\(fixture): \(image.rect)") + #expect(image.rect.minX >= 928.0 / 1920 - tolerance, "\(fixture): \(image.rect)") + #expect(image.rect.minX <= 936.0 / 1920 + tolerance, "\(fixture): \(image.rect)") + #expect(abs(image.rect.minY - 1000.0 / 1080) <= tolerance, "\(fixture): \(image.rect)") + #expect(image.image.width == Int((image.rect.width * 1920).rounded()), "\(fixture): one image pixel per canvas pixel") + + // Column 2 is transparent, column 16 opaque white, column 48 + // half-transparent red (premultiplied). The DVB and DVD encoders + // requantize the palette, hence the tolerances. + let offsetX = Int(((image.rect.minX - 928.0 / 1920) * 1920).rounded()) + let offsetY = Int(((image.rect.minY - 1000.0 / 1080) * 1080).rounded()) + let row = 8 - offsetY + if offsetX <= 2 { + let clear = try #require(pixel(of: image.image, x: 2 - offsetX, y: row)) + #expect(clear.a <= 16, "\(fixture): \(clear)") + } + let white = try #require(pixel(of: image.image, x: 16 - offsetX, y: row)) + #expect(white.a >= 240 && white.r >= 220 && white.g >= 220 && white.b >= 220, "\(fixture): \(white)") + let red = try #require(pixel(of: image.image, x: 48 - offsetX, y: row)) + #expect((100...160).contains(red.a) && red.r >= red.a - 30 && red.g <= 30 && red.b <= 30, "\(fixture): \(red)") + #expect(red.r <= red.a, "\(fixture): premultiplied colour never exceeds alpha") + + #expect(session.subtitles.activeCues(at: start + 3_500_000).isEmpty, "\(fixture): cleared at 3 s") + let second = try #require(session.subtitles.activeCues(at: start + 5_000_000).first?.images.first) + #expect(abs(second.rect.width - 48.0 / 1920) <= tolerance, "\(fixture): \(second.rect)") + #expect(session.subtitles.activeCues(at: start + 6_500_000).isEmpty, "\(fixture): cleared at 6 s") + + await session.shutdown() + } +} diff --git a/Tests/LumeEngineCoreTests/ClosedCaptionTests.swift b/Tests/LumeEngineCoreTests/ClosedCaptionTests.swift new file mode 100644 index 0000000..35774a6 --- /dev/null +++ b/Tests/LumeEngineCoreTests/ClosedCaptionTests.swift @@ -0,0 +1,222 @@ +import CFFmpeg +import Foundation +import Testing +@testable import LumeEngineCore + +/// CEA-608 caption data as a broadcast encoder writes it: one byte pair for +/// CC1 per frame, null padding on every frame without one. Mirrors the +/// caption script in `generate-fixtures.sh`. +private enum CaptionScript { + static let frameDuration: Int64 = 1_000_000 / 30 + + static func oddParity(_ value: UInt8) -> UInt8 { + let bits = value & 0x7F + return bits.nonzeroBitCount % 2 == 0 ? bits | 0x80 : bits + } + + /// Byte pairs per frame index for two pop-on captions (shown at frames + /// 30 and 120, erased at 90 and 180 — 1–3 s and 4–6 s at 30 fps). + static let pairs: [Int: (UInt8, UInt8)] = { + var script: [Int: (UInt8, UInt8)] = [:] + func popOn(load: Int, _ text: String, show: Int, clear: Int) { + var pairs: [(UInt8, UInt8)] = [(0x14, 0x20), (0x14, 0x70)] // load, row 15 + let bytes = Array(text.utf8) + for index in stride(from: 0, to: bytes.count, by: 2) { + pairs.append((bytes[index], index + 1 < bytes.count ? bytes[index + 1] : 0)) + } + for (offset, pair) in pairs.enumerated() { script[load + offset] = pair } + script[show] = (0x14, 0x2F) // end of caption + script[clear] = (0x14, 0x2C) // erase displayed memory + } + popOn(load: 5, "HELLO CAPTIONS", show: 30, clear: 90) + popOn(load: 100, "SECOND CAPTION", show: 120, clear: 180) + return script + }() + + /// The A/53 `cc_data` triples of one frame: CC1 data, field-2 nulls. + static func ccData(frame: Int) -> [UInt8] { + let (first, second) = pairs[frame] ?? (0, 0) + return [0xFC, oddParity(first), oddParity(second), 0xFD, 0x80, 0x80] + } +} + +@Suite("Closed captions") +struct ClosedCaptionTests { + @Test("caption detection ignores null padding, field 2 and invalid triples") + func detection() { + func carries(_ bytes: [UInt8]) -> Bool { + bytes.withUnsafeBytes { ClosedCaptionData.carriesCaptions($0) } + } + #expect(!carries([0xFC, 0x80, 0x80, 0xFD, 0x80, 0x80]), "null padding on both fields") + #expect(!carries([0xFD, 0x14, 0x20]), "field 2 (CC3/XDS)") + #expect(!carries([0xF8, 0x14, 0x20]), "cc_valid not set") + #expect(!carries([0xFE, 0x41, 0x42]), "DTVCC packet data") + #expect(carries([0xFD, 0x80, 0x80, 0xFC, 0x94, 0x20]), "CC1 control code") + #expect(!carries([0xFC, 0x94]), "truncated triple") + } + + /// The decoder runs FFmpeg's caption decoder in real-time mode, which + /// stamps an update with the time of the one before; the cue has to start + /// on the frame that put the caption on screen and end on the frame that + /// erased it, or captions drift from the speech they transcribe. + @Test("caption packets become cues on their frames' timestamps", .timeLimit(.minutes(1))) + func decodeCaptionPackets() throws { + let parameters = try #require(CodecParameters.closedCaptions()) + let packets = Channel(capacity: 512) + let store = SubtitleStore() + let decoder = SubtitleDecoder( + parameters: parameters, input: packets, store: store, codecOptions: ["data_field": "first"] + ) + decoder.start() + + let origin: Int64 = 50_000_000 + for frame in 0..<240 { + let bytes = CaptionScript.ccData(frame: frame) + let packet = try #require(bytes.withUnsafeBytes { + Packet(copying: $0, pts: origin + Int64(frame) * CaptionScript.frameDuration, serial: 0) + }) + try packets.send(packet) + } + #expect(decoder.drainAndShutdown()) + + func time(frame: Int) -> Int64 { origin + Int64(frame) * CaptionScript.frameDuration } + #expect(store.count == 2) + let first = store.activeCues(at: time(frame: 30)) + #expect(first.map(\.text) == ["HELLO CAPTIONS"]) + #expect(first.first?.start == time(frame: 30)) + #expect(first.first?.end == time(frame: 90)) + #expect(store.activeCues(at: time(frame: 29)).isEmpty) + #expect(store.activeCues(at: time(frame: 90)).isEmpty) + #expect(store.activeCues(at: time(frame: 150)).map(\.text) == ["SECOND CAPTION"]) + #expect(store.activeCues(at: time(frame: 200)).isEmpty) + } + + @Test("the tap detects captions and forwards them stamped with the frame's PTS") + func tapForwardsSideData() throws { + let tap = ClosedCaptionTap() + var frame: UnsafeMutablePointer? = av_frame_alloc() + defer { av_frame_free(&frame) } + let raw = try #require(frame) + + // No side data: nothing detected, nothing forwarded. + let sink = Channel(capacity: 4) + tap.attach(sink) + tap.offer(raw, serial: 3) + #expect(!tap.hasDetectedCaptions) + #expect(sink.tryReceive() == nil) + + func attach(_ bytes: [UInt8]) throws { + av_frame_remove_side_data(raw, AV_FRAME_DATA_A53_CC) + let sideData = try #require(av_frame_new_side_data(raw, AV_FRAME_DATA_A53_CC, bytes.count)) + sideData.pointee.data.update(from: bytes, count: bytes.count) + } + + // Padding only: forwarded (the decoder needs the frame cadence), but + // no caption track to offer yet. + raw.pointee.best_effort_timestamp = 7_000_000 + try attach(CaptionScript.ccData(frame: 0)) + tap.offer(raw, serial: 3) + #expect(!tap.hasDetectedCaptions) + let padding = try #require(sink.tryReceive()) + #expect(padding.pts == 7_000_000) + #expect(padding.serial == 3) + #expect(padding.byteCount == 6) + + raw.pointee.best_effort_timestamp = 7_033_333 + try attach(CaptionScript.ccData(frame: 5)) + tap.offer(raw, serial: 3) + #expect(tap.hasDetectedCaptions) + #expect(sink.tryReceive()?.pts == 7_033_333) + + // Detached: detection stays, forwarding stops. + tap.attach(nil) + tap.offer(raw, serial: 3) + #expect(tap.hasDetectedCaptions) + #expect(sink.tryReceive() == nil) + } +} + +@Suite("Closed captions in playback", .serialized) +struct ClosedCaptionPlaybackTests { + private func eventually( + timeout: TimeInterval = 10, + _ condition: () async -> Bool + ) async -> Bool { + let deadline = Date(timeIntervalSinceNow: timeout) + while Date() < deadline { + if await condition() { return true } + try? await Task.sleep(for: .milliseconds(100)) + } + return false + } + + private func makeSession(hardware: Bool) -> PlayerSession { + var configuration = PlayerConfiguration() + configuration.muted = true + configuration.bufferTarget = 0.3 + configuration.hardwareDecode = hardware ? .videoToolbox : .software + return PlayerSession(configuration: configuration) + } + + /// End to end: SEI in an H.264 TS → frame side data (VideoToolbox frames + /// carry it too: FFmpeg parses SEI itself) → caption lane → cues on the + /// video's timestamps. Seeks back to 0 after selecting so the first + /// caption is decoded again rather than raced against the decode lead. + @Test( + "captions in the video surface as a CC track and decode in sync", + .timeLimit(.minutes(1)), + arguments: [true, false] + ) + func captionsFromVideo(hardware: Bool) async throws { + let session = makeSession(hardware: hardware) + let info = try await session.open(url: try Fixtures.path("captions.ts")) + let listedAtOpen = info.tracks.contains(where: \.isEmbeddedClosedCaptions) + #expect(!listedAtOpen, "unknown at open") + + let offered = await eventually { + await session.mediaInfo?.subtitleTracks.contains(where: \.isEmbeddedClosedCaptions) == true + } + #expect(offered, "the CC track should appear once captions are decoded") + + await session.selectSubtitleTrack(TrackInfo.embeddedClosedCaptionsIndex) + #expect(await session.selectedSubtitleTrackIndex == TrackInfo.embeddedClosedCaptionsIndex) + await session.seek(to: 0) + await session.play() + + // Done once the second caption has been erased again. Not a cue + // count: what was decoded before the seek joined the first caption + // halfway through loading and may have left a fragment of its own. + let start = info.startTime + let frame: Int64 = 1_000_000 / 30 + let sawBoth = await eventually(timeout: 30) { + let second = session.subtitles.activeCues(at: start + 5_000_000) + return second.map(\.text) == ["SECOND CAPTION"] && second.allSatisfy { $0.end <= start + 6_000_000 + frame } + } + #expect(sawBoth, "both captions should decode, store has \(session.subtitles.count)") + + let first = session.subtitles.activeCues(at: start + 2_000_000) + #expect(first.map(\.text) == ["HELLO CAPTIONS"]) + if let cue = first.first { + #expect(abs(cue.start - (start + 1_000_000)) <= frame, "shown at 1 s, got \(cue.start - start) µs") + #expect(abs(cue.end - (start + 3_000_000)) <= frame, "erased at 3 s, got \(cue.end - start) µs") + } + #expect(session.subtitles.activeCues(at: start + 3_500_000).isEmpty) + #expect(session.subtitles.activeCues(at: start + 5_000_000).map(\.text) == ["SECOND CAPTION"]) + + await session.selectSubtitleTrack(nil) + #expect(session.subtitles.count == 0) + await session.shutdown() + } + + @Test("a source without captions offers no CC track", .timeLimit(.minutes(1))) + func noCaptionsNoTrack() async throws { + let session = makeSession(hardware: true) + _ = try await session.open(url: try Fixtures.path("basic.mp4")) + await session.play() + _ = await eventually { await session.position > 1 } + let tracks = await session.mediaInfo?.subtitleTracks ?? [] + let listed = tracks.contains(where: \.isEmbeddedClosedCaptions) + #expect(!listed) + await session.shutdown() + } +} diff --git a/Tests/LumeEngineCoreTests/DecoderTests.swift b/Tests/LumeEngineCoreTests/DecoderTests.swift index 33b5cb2..a69e7e1 100644 --- a/Tests/LumeEngineCoreTests/DecoderTests.swift +++ b/Tests/LumeEngineCoreTests/DecoderTests.swift @@ -333,7 +333,8 @@ struct DecoderTests { let collected = await collector.value let rate = try #require(collected.rates.first) #expect(collected.rates.count == 1) - #expect(collected.channels == [1], "sine fixture is mono") + // The sine fixture is mono; on a stereo route it goes out on both sides. + #expect(collected.channels == [2], "mono source on a stereo route") let seconds = Double(collected.totalSamples) / Double(rate) #expect(abs(seconds - 10.0) < 0.3, "expected ~10 s of audio, got \(seconds)") let ptsValues = collected.ptsValues diff --git a/Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh b/Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh index befe268..30a27fe 100644 --- a/Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh +++ b/Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh @@ -18,6 +18,10 @@ # replay.ts — 8 s TS whose bytes jump back ~3 s at ~6 s, as a reconnect replays them # ac4.ts — TS whose first audio (ger) is declared AC-4, which no decoder here plays, then eng AAC # hdr10.mp4 — HEVC Main 10, BT.2020 PQ, mastering display + content light level SEI, HDR10+ per frame +# surround51.ts — 20 s H.264 + AC-3 5.1 MPEG-TS, tone in the centre channel only +# captions.ts — H.264 TS with CEA-608 captions in A/53 SEI (1–3 s, 4–6 s; needs python3) +# pgs.mkv, dvbsub.ts, dvdsub.mkv — the same two bitmap subtitles (1–3 s, 4–6 s) +# as PGS, DVB and DVD subtitles on a 1920x1080 canvas (needs python3) set -euo pipefail FFMPEG="${FFMPEG:-ffmpeg}" @@ -329,4 +333,174 @@ JSON echo "generated: $OUT/hdr10.mp4" fi +# Broadcast surround the way IPTV carries it: H.264 + AC-3 5.1 in MPEG-TS, +# long enough to change the output route mid-playback. The tone sits in the +# centre channel alone, so a fold-down that drops channels instead of mixing +# them goes silent. +gen surround51.ts \ + -f lavfi -i "testsrc2=duration=20:size=320x180:rate=25" \ + -f lavfi -i "sine=frequency=440:duration=20" \ + -filter_complex "[1:a]pan=5.1(side)|FC=c0[a]" -map 0:v -map "[a]" \ + -c:v libx264 -preset ultrafast -pix_fmt yuv420p -g 25 \ + -c:a ac3 -b:a 384k -shortest -f mpegts + +# Closed captions the way US broadcast and much IPTV carry them: CEA-608 +# byte pairs inside the video, as A/53 user data in an SEI message on every +# frame — not as a subtitle stream. No ffmpeg CLI option writes captions from +# a caption file into the video, so a short script puts the SEI in front of +# each picture of a B-frame-free encode, so the n-th picture in the file is +# the n-th on screen. Two pop-on captions, each loaded off screen first: +# "HELLO CAPTIONS" shown at 1 s and erased at 3 s, "SECOND CAPTION" at 4–6 s. +if [ ! -f "$OUT/captions.ts" ]; then + if command -v python3 >/dev/null 2>&1; then + "$FFMPEG" -hide_banner -loglevel error -y \ + -f lavfi -i "testsrc2=duration=8:size=320x180:rate=30" \ + -c:v libx264 -preset ultrafast -pix_fmt yuv420p -bf 0 -g 30 \ + -f h264 "$OUT/captions-source.h264" + python3 -I - "$OUT/captions-source.h264" "$OUT/captions.h264" <<'PY' +import sys + +source, destination = sys.argv[1], sys.argv[2] +stream = open(source, 'rb').read() + +def nal_units(data): + starts = [] + index = 0 + while True: + found = data.find(b'\x00\x00\x01', index) + if found < 0: + break + starts.append(found + 3) + index = found + 3 + for position, start in enumerate(starts): + end = starts[position + 1] - 3 if position + 1 < len(starts) else len(data) + yield data[start:end].rstrip(b'\x00') + +def odd_parity(value): + value &= 0x7F + return value | (0x80 if bin(value).count('1') % 2 == 0 else 0) + +# frame -> CEA-608 byte pair for CC1 (field 1); every other frame carries nulls. +script = {} +def pop_on(load, text, show, clear): + pairs = [(0x14, 0x20), (0x14, 0x70)] # resume caption loading, row 15 + for i in range(0, len(text), 2): + pairs.append((ord(text[i]), ord(text[i + 1]) if i + 1 < len(text) else 0)) + for offset, pair in enumerate(pairs): + script[load + offset] = pair + script[show] = (0x14, 0x2F) # end of caption: swap it on screen + script[clear] = (0x14, 0x2C) # erase displayed memory + +pop_on(5, 'HELLO CAPTIONS', 30, 90) +pop_on(100, 'SECOND CAPTION', 120, 180) + +def sei(frame): + first, second = script.get(frame, (0, 0)) + cc_data = bytes([0xFC, odd_parity(first), odd_parity(second), 0xFD, 0x80, 0x80]) + # ITU-T T.35: United States, ATSC, 'GA94', cc_data with two triples. + payload = bytes([0xB5, 0x00, 0x31]) + b'GA94' + bytes([0x03, 0x40 | 2, 0xFF]) + cc_data + bytes([0xFF]) + rbsp = bytes([4, len(payload)]) + payload + bytes([0x80]) + nal = bytearray([0x06]) + zeros = 0 + for byte in rbsp: + if zeros >= 2 and byte <= 3: + nal.append(3) # emulation prevention + zeros = 0 + nal.append(byte) + zeros = zeros + 1 if byte == 0 else 0 + return bytes(nal) + +out = bytearray() +frame = 0 +for nal in nal_units(stream): + # First slice of a picture: a slice NAL whose first_mb_in_slice is 0. + if nal and nal[0] & 0x1F in (1, 5) and len(nal) > 1 and nal[1] & 0x80: + out += b'\x00\x00\x00\x01' + sei(frame) + frame += 1 + out += b'\x00\x00\x00\x01' + nal +open(destination, 'wb').write(out) +PY + "$FFMPEG" -hide_banner -loglevel error -y \ + -framerate 30 -i "$OUT/captions.h264" -c:v copy -f mpegts \ + "$OUT/captions.ts" + rm -f "$OUT/captions-source.h264" "$OUT/captions.h264" + echo "generated: $OUT/captions.ts" + else + echo "skipped: $OUT/captions.ts (needs python3)" >&2 + fi +fi + +# Bitmap subtitles in the three formats broadcast and disc rips carry: PGS +# (Blu-ray), DVB (European broadcast) and DVD. ffmpeg cannot render text into +# subtitle bitmaps, but it converts between bitmap formats, so a short script +# writes the PGS original by hand: a 64x16 bitmap (8 transparent, 24 opaque +# white and 32 half-transparent red pixels per row) at (928, 1000) of a +# 1920x1080 canvas shown 1–3 s, then a 48x8 white one at (936, 960) shown +# 4–6 s, each ended by an empty presentation. A leading empty presentation at +# 0 keeps ffmpeg from shifting the subtitles to start at 0, and +# -fix_sub_duration turns PGS's "until the next one" into the durations the +# DVB and DVD encoders need. The DVB conversion keeps the canvas as a display +# definition segment. +if [ ! -f "$OUT/pgs.mkv" ] || [ ! -f "$OUT/dvbsub.ts" ] || [ ! -f "$OUT/dvdsub.mkv" ]; then + if command -v python3 >/dev/null 2>&1; then + python3 -I - "$OUT/bitmap-subs.sup" <<'PY' +import struct +import sys + +def segment(kind, pts, payload): + return b'PG' + struct.pack('>IIBH', pts, 0, kind, len(payload)) + payload + +def display_set(pts, composition, position=None, palette=None, bitmap=None): + objects = 1 if bitmap else 0 + pcs = struct.pack('>HHBHBBBB', 1920, 1080, 0x10, composition, 0x80 if bitmap else 0x00, 0, 0, objects) + if bitmap: + x, y = position + width, height, rle = bitmap + pcs += struct.pack('>HBBHH', 0, 0, 0, x, y) + out = segment(0x16, pts, pcs) + out += segment(0x17, pts, struct.pack('>BBHHHH', 1, 0, x, y, width, height)) + pds = struct.pack('>BB', 0, 0) + b''.join(struct.pack('>BBBBB', *entry) for entry in palette) + out += segment(0x14, pts, pds) + ods = struct.pack('>HBB', 0, 0, 0xC0) + struct.pack('>I', len(rle) + 4)[1:] + out += segment(0x15, pts, ods + struct.pack('>HH', width, height) + rle) + else: + out = segment(0x16, pts, pcs) + out += segment(0x17, pts, struct.pack('>BBHHHH', 1, 0, 0, 0, 1, 1)) + return out + segment(0x80, pts, b'') + +# (index, Y, Cr, Cb, alpha): transparent, opaque white, half-transparent red. +palette = [(0, 16, 128, 128, 0), (1, 235, 128, 128, 255), (2, 63, 240, 102, 128)] +# PGS run-length lines: 00 0n = n of colour 0, 00 8n c = n of colour c, 00 00 = end. +first = (64, 16, bytes([0, 0x08, 0, 0x80 | 24, 1, 0, 0x80 | 32, 2, 0, 0]) * 16) +second = (48, 8, bytes([0, 0x80 | 48, 1, 0, 0]) * 8) + +stream = display_set(0, 0) +stream += display_set(90_000 * 1, 1, (928, 1000), palette, first) +stream += display_set(90_000 * 3, 2) +stream += display_set(90_000 * 4, 3, (936, 960), palette, second) +stream += display_set(90_000 * 6, 4) +open(sys.argv[1], 'wb').write(stream) +PY + "$FFMPEG" -hide_banner -loglevel error -y \ + -f lavfi -i "testsrc2=duration=8:size=320x180:rate=25" \ + -i "$OUT/bitmap-subs.sup" -map 0:v -map 1:s \ + -c:v libx264 -preset ultrafast -pix_fmt yuv420p -c:s copy \ + "$OUT/pgs.mkv" + "$FFMPEG" -hide_banner -loglevel error -y \ + -f lavfi -i "testsrc2=duration=8:size=320x180:rate=25" \ + -fix_sub_duration -i "$OUT/bitmap-subs.sup" -map 0:v -map 1:s \ + -c:v libx264 -preset ultrafast -pix_fmt yuv420p -c:s dvbsub \ + -muxdelay 0 -f mpegts "$OUT/dvbsub.ts" + "$FFMPEG" -hide_banner -loglevel error -y \ + -f lavfi -i "testsrc2=duration=8:size=320x180:rate=25" \ + -fix_sub_duration -i "$OUT/bitmap-subs.sup" -map 0:v -map 1:s \ + -c:v libx264 -preset ultrafast -pix_fmt yuv420p -c:s dvdsub \ + "$OUT/dvdsub.mkv" + rm -f "$OUT/bitmap-subs.sup" + echo "generated: $OUT/pgs.mkv $OUT/dvbsub.ts $OUT/dvdsub.mkv" + else + echo "skipped: bitmap subtitle fixtures (needs python3)" >&2 + fi +fi + echo "fixtures ready in $OUT" diff --git a/Tests/LumeEngineCoreTests/SubtitleTimelineTests.swift b/Tests/LumeEngineCoreTests/SubtitleTimelineTests.swift new file mode 100644 index 0000000..ee38af7 --- /dev/null +++ b/Tests/LumeEngineCoreTests/SubtitleTimelineTests.swift @@ -0,0 +1,216 @@ +import Foundation +import Testing +@testable import LumeEngineCore + +@Suite("Subtitle cue lifecycle") +struct SubtitleTimelineTests { + private let second: Int64 = 1_000_000 + + private func screen(_ start: Int64, _ text: String, end: Int64? = nil) -> SubtitleEvent { + SubtitleEvent(start: start, end: end, text: text, replacesPrevious: true) + } + + private func text(_ start: Int64, _ text: String, end: Int64?) -> SubtitleEvent { + SubtitleEvent(start: start, end: end, text: text, replacesPrevious: false) + } + + private func shown(_ timeline: SubtitleTimeline, at time: Int64) -> [String] { + timeline.activeCues(at: time).map(\.text) + } + + // MARK: Timing of one AVSubtitle + + @Test("UINT32_MAX end means open, on any track") + func endlessIsOpen() { + let span = SubtitleEvent.span( + base: 10 * second, startDisplay: 0, endDisplay: .max, packetDuration: 0, replacesPrevious: false + ) + #expect(span.start == 10 * second) + #expect(span.end == nil) + } + + @Test("a missing end is open on a replacing track, a guess on a text track") + func missingEnd() { + let replacing = SubtitleEvent.span( + base: 10 * second, startDisplay: 0, endDisplay: 0, packetDuration: 2 * second, replacesPrevious: true + ) + #expect(replacing.end == nil) + + let withDuration = SubtitleEvent.span( + base: 10 * second, startDisplay: 0, endDisplay: 0, packetDuration: 2 * second, replacesPrevious: false + ) + #expect(withDuration.end == 12 * second) + + let withoutDuration = SubtitleEvent.span( + base: 10 * second, startDisplay: 500, endDisplay: 0, packetDuration: 0, replacesPrevious: false + ) + #expect(withoutDuration.start == 10_500_000) + #expect(withoutDuration.end == 13_500_000) + } + + @Test("display times are milliseconds after the base") + func explicitSpan() { + let span = SubtitleEvent.span( + base: 10 * second, startDisplay: 250, endDisplay: 4_000, packetDuration: 0, replacesPrevious: true + ) + #expect(span.start == 10_250_000) + #expect(span.end == 14 * second) + } + + // MARK: Replacing tracks (captions, DVB, PGS) + + /// Before the timeline, a cue without an end (`UINT32_MAX` ms) stayed up + /// for 49 days and every later caption stacked under it. + @Test("an open cue ends where the next update starts") + func openCueEndsAtNext() { + var timeline = SubtitleTimeline() + timeline.apply(screen(1 * second, "one")) + timeline.apply(screen(4 * second, "two")) + + #expect(shown(timeline, at: 3 * second) == ["one"]) + #expect(shown(timeline, at: 4 * second) == ["two"]) + #expect(shown(timeline, at: 5 * second) == ["two"]) + } + + /// The empty update DVB/PGS/captions send to clear the screen used to be + /// dropped as "no text", leaving the previous cue up. + @Test("a clear ends what is showing and shows nothing itself") + func clearEndsCue() { + var timeline = SubtitleTimeline() + timeline.apply(screen(1 * second, "one")) + timeline.apply(screen(3 * second, "")) + + #expect(shown(timeline, at: 2 * second) == ["one"]) + #expect(shown(timeline, at: 3 * second).isEmpty) + #expect(timeline.count == 1) + } + + @Test("a cue with an end of its own is still ended by the next update") + func explicitEndCutByNext() { + var timeline = SubtitleTimeline() + // DVB page timeouts are long; the next page replaces it well before. + timeline.apply(screen(1 * second, "page one", end: 31 * second)) + timeline.apply(screen(5 * second, "page two", end: 35 * second)) + #expect(shown(timeline, at: 4 * second) == ["page one"]) + #expect(shown(timeline, at: 6 * second) == ["page two"]) + #expect(shown(timeline, at: 36 * second).isEmpty) + } + + @Test("an open cue without a successor stops after the limit") + func openCueIsCapped() { + var timeline = SubtitleTimeline() + timeline.apply(screen(1 * second, "stuck")) + let limit = SubtitleTimeline.openEndedLimit + #expect(shown(timeline, at: 1 * second + limit - 1) == ["stuck"]) + #expect(shown(timeline, at: 1 * second + limit).isEmpty) + } + + @Test("an update arriving before one already stored is bounded by it") + func outOfOrderIsBounded() { + var timeline = SubtitleTimeline() + timeline.apply(screen(5 * second, "later")) + timeline.apply(screen(2 * second, "earlier")) + #expect(shown(timeline, at: 4 * second) == ["earlier"]) + #expect(shown(timeline, at: 5 * second) == ["later"]) + } + + /// A re-read range (backward seek) sends the same updates again; the + /// stored ones were bounded by their successors meanwhile, so the ends + /// differ and only start and text can identify them. + @Test("re-reading a range does not double it") + func rereadIsDeduplicated() { + var timeline = SubtitleTimeline() + for _ in 0..<2 { + timeline.apply(screen(1 * second, "one")) + timeline.apply(screen(3 * second, "")) + timeline.apply(screen(4 * second, "two")) + } + #expect(timeline.count == 2) + #expect(shown(timeline, at: 2 * second) == ["one"]) + #expect(shown(timeline, at: 3_500_000).isEmpty) + #expect(shown(timeline, at: 4_500_000) == ["two"]) + } + + @Test("on a replacing track, a different update at the same instant wins") + func sameStartReplaces() { + var timeline = SubtitleTimeline() + timeline.apply(screen(1 * second, "ROW ONE\nROW TWO")) + // After a seek the caption decoder starts from an empty screen and + // decodes the same instant differently. + timeline.apply(screen(1 * second, "ROW TWO")) + #expect(timeline.count == 1) + #expect(shown(timeline, at: 2 * second) == ["ROW TWO"]) + } + + // MARK: Text tracks (SRT, ASS, WebVTT) + + @Test("text cues keep their overlaps") + func textOverlapKept() { + var timeline = SubtitleTimeline() + timeline.apply(text(1 * second, "one", end: 5 * second)) + timeline.apply(text(2 * second, "two", end: 4 * second)) + #expect(shown(timeline, at: 3 * second) == ["one", "two"]) + #expect(shown(timeline, at: 4_500_000) == ["one"]) + } + + @Test("an open text cue ends at the next cue, a closed one does not") + func openTextCueEndsAtNext() { + var timeline = SubtitleTimeline() + timeline.apply(text(1 * second, "open", end: nil)) + timeline.apply(text(3 * second, "next", end: 6 * second)) + #expect(shown(timeline, at: 2 * second) == ["open"]) + #expect(shown(timeline, at: 4 * second) == ["next"]) + } + + @Test("an empty text cue is nothing, not a clear") + func emptyTextCueIgnored() { + var timeline = SubtitleTimeline() + timeline.apply(text(1 * second, "one", end: 5 * second)) + timeline.apply(text(2 * second, "", end: 3 * second)) + #expect(shown(timeline, at: 4 * second) == ["one"]) + } + + // MARK: Seeks and growth + + @Test("a seek ends open cues, not the ones the source gave an end") + func seekClosesOpenCues() { + var timeline = SubtitleTimeline() + timeline.apply(screen(1 * second, "open")) + timeline.apply(text(1 * second, "closed", end: 9 * second)) + timeline.closeOpenCues(at: 4 * second) + #expect(shown(timeline, at: 3 * second) == ["open", "closed"]) + #expect(shown(timeline, at: 5 * second) == ["closed"]) + + // Cues after the seek target are the future of the new position. + var later = SubtitleTimeline() + later.apply(screen(10 * second, "ahead")) + later.closeOpenCues(at: 4 * second) + #expect(shown(later, at: 11 * second) == ["ahead"]) + } + + /// A day of live roll-up captions is hundreds of thousands of updates; + /// every one of them used to be kept. + @Test("the timeline keeps a bounded number of cues, dropping the oldest") + func boundedGrowth() { + var timeline = SubtitleTimeline() + let total = SubtitleTimeline.capacity * 2 + for index in 0..