From 43108503a8cffd91584ea3a6d3ec0e7a61db0896 Mon Sep 17 00:00:00 2001
From: Philipp Bischoff
Date: Mon, 12 Oct 2026 22:42:50 +0200
Subject: [PATCH 1/4] feat: fit audio to the output route as it changes, and
report lost audio
The audio decoder read the negotiated output width once, at creation, so a
route change mid-playback (HDMI receiver to AirPlay, HomePods) kept feeding
the old width. AudioOutputRoute now follows the shared audio session's
negotiated width through route-change notifications (stereo on macOS); the
decoder refits per AudioLayoutPolicy at the next frame boundary, with an
explicit ITU-R BS.775 DownmixMatrix (centre and surrounds -3 dB, LFE
omitted) set into swresample, and mono goes out on both speakers. When the
route narrows, the renderer keeps audio decoded for the wider route away
from the renderer.
PlayerSession.audioHealth (also in Diagnostics) reports audio that died
while video plays on: decoder failed, renderer failed, or nothing reaching
the renderer at a running playhead (AudioLossDetector), the latter also as
.error with the new EngineError.Code.audioLost. Additive API only.
---
AGENTS.md | 2 +
PLAN.md | 1 +
.../LumeEngineCore/Decode/AudioDecoder.swift | 99 +++-
.../Decode/AudioLayoutPolicy.swift | 169 ++++++
.../LumeEngineCore/Errors/EngineError.swift | 7 +
.../Media/AudioOutputRoute.swift | 140 +++++
.../Render/SystemRenderer.swift | 42 +-
.../LumeEngineCore/Session/AudioHealth.swift | 153 +++++
.../Session/PlayerSession.swift | 71 ++-
.../LumeEngineCoreTests/AudioRouteTests.swift | 548 ++++++++++++++++++
Tests/LumeEngineCoreTests/DecoderTests.swift | 3 +-
.../Fixtures/generate-fixtures.sh | 12 +
12 files changed, 1220 insertions(+), 27 deletions(-)
create mode 100644 Sources/LumeEngineCore/Decode/AudioLayoutPolicy.swift
create mode 100644 Sources/LumeEngineCore/Media/AudioOutputRoute.swift
create mode 100644 Sources/LumeEngineCore/Session/AudioHealth.swift
create mode 100644 Tests/LumeEngineCoreTests/AudioRouteTests.swift
diff --git a/AGENTS.md b/AGENTS.md
index d04eaa5..38f275d 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -86,6 +86,8 @@ This split is the core invariant of the codebase:
- **Multi-layer HEVC reaches the codec as its base layer only.** `Decode/HEVCBaseLayerFilter` drops every `nuh_layer_id > 0` NAL unit from packets and extradata before `avcodec_send_packet`/`avcodec_open2`. FFmpeg files an MV-HEVC stream's second-view SPS/PPS (DV profile 20, spatial video) alongside the base layer's, its VideoToolbox glue packs all of them into one `hvcC`, VideoToolbox rejects that session, and FFmpeg falls back to software *without an error* — 4K HEVC in software, which stutters on Apple TV. The engine presents only the base view, so stripping loses nothing. Relatedly, `VideoDecoder.isHardwareActive` is judged from the frames' pixel format, not from whether a VT device was attached, and such a silent fallback emits `.downgradedToSoftware` — keep it that way, or this class of bug goes invisible again.
- **Dolby Vision with an IPT base layer is converted, not tagged.** Profiles 5, 10.0 and 20 (compatibility ID 0) carry IPTPQc2, which shows pink/green as YCbCr. No colour attachment fixes that — the RPU reshaping changes per scene and Apple's decoders don't apply it to decoded frames — so `Decode/DolbyVisionConverter` turns each frame into HDR10 on the GPU in `VideoDecoder.deliver`, the one point every frame (hardware, software, deinterlaced) passes with its side data. Whether a frame is IPT is decided *per frame* by `Decode/DolbyVisionDetection`, never by the configuration record alone — remuxes mislabel HDR10 as P5 and broadcasts drop RPUs without changing it, and converting HDR10 as IPT gives wrong colours. Order: an `AVCOL_SPC_IPT_C2` frame is IPT; a frame with its *own* RPU is what the RPU's `signal_color_space` says (2 = IPT); a frame without one that states a matrix (BT.2020 PQ) is that matrix; only an untagged frame without its own RPU falls back to the record (genuine P5 often leaves the VUI matrix unspecified). "Own" matters because FFmpeg's HEVC decoder attaches the last RPU's `AV_FRAME_DATA_DOVI_METADATA` to every later frame; only `AV_FRAME_DATA_DOVI_RPU_BUFFER` marks the frame that carried one (AV1 attaches metadata only to its own frame). HDR10-compatible profiles (8.1 etc.) must stay on the zero-copy path. GPU refusals are per-frame (iOS refuses a backgrounded app); only a converter that can't exist gives up for the session.
- **Display criteria describe the output, not the container.** `PlayerSession.displayCriteria` (pure resolution in `Media/ContentDisplayCriteria`) is what an Apple TV with Match Content should switch the TV to: the refresh rate *after* deinterlacing (field rate turns 1080i50 into 50 Hz) and the dynamic range of what is presented (IPT Dolby Vision profiles are converted, so HDR10; every other Dolby Vision profile is presented as its base layer, so HDR10, HLG or SDR by its compatibility ID and the frames' transfer function). Never ask for Dolby Vision mode while the engine has no Dolby Vision output path — that belongs to the PLAN.md §7 compressed-`dvh1` spike. Before the first frame it is predicted from the stream; once the renderer has been handed frames, their colour tags and measured cadence (`Render/FrameCadence`) decide. Rates snap to the film/broadcast grid or ask for none — never an odd rate. Applying it is tvOS-only and opt-in (`Media/DisplayCriteriaMatcher`, or `LumePlayerView(…, matchesDisplayCriteria: true)`): one matcher per player screen, kept across sessions so zapping between same-mode channels never switches the TV, reset only when the screen goes away. The mode-switch blank is sat out with `PlayerSession.holdStart(for:)`, which the session bounds itself and the stall watchdog ignores — keep both, or a host that never releases wedges playback, or a held start reads as a stall.
+- **Dolby Vision with an IPT base layer is converted, not tagged.** Profiles 5, 10.0 and 20 (compatibility ID 0) carry IPTPQc2, which shows pink/green as YCbCr. No colour attachment fixes that — the RPU reshaping changes per scene and Apple's decoders don't apply it to decoded frames — so `Decode/DolbyVisionConverter` turns each frame into HDR10 on the GPU in `VideoDecoder.deliver`, the one point every frame (hardware, software, deinterlaced) passes with its side data. Detection is the stream's DV configuration record *or* an `AVCOL_SPC_IPT_C2` frame; many P5 encodes leave the VUI matrix unspecified, so don't reduce it to the tag. HDR10-compatible profiles (8.1 etc.) must stay on the zero-copy path. GPU refusals are per-frame (iOS refuses a backgrounded app); only a converter that can't exist gives up for the session.
+- **Audio is fitted to the negotiated route width, never the route's capability.** `AudioDecoder` reads `AudioOutputRoute.outputChannels` (the session's `outputNumberOfChannels`, followed through route-change notifications) per frame and remixes per `AudioLayoutPolicy` with an explicit `DownmixMatrix` — a fold-down mixes centre and surrounds in, it never drops channels. Don't switch it to `maximumOutputNumberOfChannels`: a renderer fed more channels than the session outputs failed outright on tvOS. The engine only observes the audio session; setting its category, activation or preferred channel count is the host's job (Lume does it). When the route narrows, `SystemRenderer` keeps audio decoded for the wider route out of the renderer.
- **Zero global mutable state:** all configuration is `PlayerConfiguration` passed at session creation. No `static var` knobs.
- **No force unwraps / `as!` / `fatalError` in engine paths**; failures return typed `EngineError`. No `assertionFailure` in production channel/data paths — misuse degrades, never crashes.
- **Never scrape FFmpeg log strings for control flow** (`av_log` is diagnostics only). Silence is never a failure mode: pipelines are supervised, stalls are detected from playhead ground truth, and failures surface as typed `PlayerEvent`s on the session's `AsyncStream`.
diff --git a/PLAN.md b/PLAN.md
index 782e84e..bb4f4d7 100644
--- a/PLAN.md
+++ b/PLAN.md
@@ -190,6 +190,7 @@ Licensing note: engine code is **MIT** (`LICENSE`); FFmpeg is LGPL 2.1+, and the
## 6. Feature implementation notes (selected)
- **Now Playing (explicit requirement):** `NowPlayingCenter` module in `LumeEngineMedia` — `MPNowPlayingInfoCenter` (title/artwork/elapsed/duration/rate, live-stream flag) + `MPRemoteCommandCenter` (play/pause/seek/skip/next-episode hook), audio-session lifecycle, route-change and interruption handling. Works on the lock screen/Control Center including while audio/video routes to an Apple TV. Note: full-screen *video* AirPlay of FFmpeg-decoded content is not possible for third parties (system limitation); Lume's existing switch-to-AVPlayer-on-AirPlay stays, and the integration phase adds a shared app-level Now Playing service so lock-screen metadata is present in **both** engines (closing today's gap).
+- **Audio width follows the route:** decoded audio leaves the engine in the layout `AudioLayoutPolicy` picks for the *negotiated* route width (`AVAudioSession.outputNumberOfChannels`, followed through route changes by `AudioOutputRoute`; stereo on macOS): discrete multichannel when the route carries it (HDMI receiver), an explicit ITU-R BS.775 fold-down (`DownmixMatrix`: centre and surrounds at −3 dB, LFE omitted, not normalized) when it does not, mono on both speakers of a stereo route. A route change refits at the next decoded frame; audio decoded for a wider route never reaches a narrower one. The engine observes the audio session only — category, activation and the preferred channel count are the host's. `PlayerSession.audioHealth` reports audio that died while video plays on (decoder gave up, renderer failed, or nothing reaching the renderer at a running playhead).
- **PiP:** `AVPictureInPictureController(contentSource: .init(sampleBufferDisplayLayer:playbackDelegate:))` from the SystemRenderer; subtitle cues composited into the sample buffers during PiP so subs survive.
- **Trickplay:** a `ThumbnailSession` — separate lightweight demux+decode (keyframes only, HW decode, downscale to ~320 px) filling a time-indexed cache; API `thumbnails(at:interval:)`. Works for VOD/catch-up; live uses the rewind buffer.
- **Live rewind (time-shift DVR):** disk-backed packet spool behind a custom AVIO layer; seeking within spool = local demux, seeking to live edge = passthrough. Configurable size/duration caps.
diff --git a/Sources/LumeEngineCore/Decode/AudioDecoder.swift b/Sources/LumeEngineCore/Decode/AudioDecoder.swift
index f8bebb0..61a51b6 100644
--- a/Sources/LumeEngineCore/Decode/AudioDecoder.swift
+++ b/Sources/LumeEngineCore/Decode/AudioDecoder.swift
@@ -7,13 +7,15 @@ import Foundation
/// FFmpeg audio decode + resample stage: `Channel` in,
/// `Channel` out.
///
-/// Output is always interleaved Float32 at the source sample rate, downmixed
-/// to at most `maxOutputChannels` — the single canonical format the renderer
-/// consumes. Sources with more channels than the output route can carry are
-/// downmixed here by swresample (normalized matrix); handing the sample-buffer
-/// renderer more channels than the route supports produces audibly broken
-/// output. Resampling context is rebuilt automatically when source parameters
-/// change mid-stream (codec/parameter changes are routine on IPTV).
+/// Output is always interleaved Float32 at the source sample rate, in the
+/// speaker layout `AudioLayoutPolicy` picks for the output route — the single
+/// canonical format the renderer consumes. Sources with more channels than
+/// the route carries are folded down here by swresample with an explicit
+/// matrix (`DownmixMatrix`); handing the sample-buffer renderer more channels
+/// than the route supports produces audibly broken output. The resampling
+/// context is rebuilt when source parameters change mid-stream (codec and
+/// parameter changes are routine on IPTV) and when the route's width changes
+/// (`AudioOutputRoute`), in both cases at a frame boundary.
///
/// Decoded frames are coalesced to at least `minOutputDuration` before
/// emission. Some codecs decode to frames far too small to schedule
@@ -42,6 +44,7 @@ public final class AudioDecoder: @unchecked Sendable {
private var swrSourceFormat: Int32 = -1
private var swrSourceRate: Int32 = -1
private var swrSourceChannels: Int32 = -1
+ private var swrRouteChannels = -1
private var outputLayout = AVChannelLayout()
private var currentSerial: UInt64?
private var consecutiveErrors = 0
@@ -59,7 +62,17 @@ public final class AudioDecoder: @unchecked Sendable {
private static let minOutputDuration = 0.020
private let maxConsecutiveErrors = 100
- private let maxOutputChannels: Int
+ private let route: AudioOutputRoute
+ /// What the resampler currently produces. Guarded by `lock`.
+ private var appliedOutput: AudioOutputLayout?
+
+ /// The layout decoded audio currently leaves in, and the route width it
+ /// was chosen for; nil before the first frame.
+ public var currentOutput: AudioOutputLayout? {
+ lock.lock()
+ defer { lock.unlock() }
+ return appliedOutput
+ }
/// Channel budget of the current output route. The renderer downstream
/// cannot correctly render more channels than the route carries, so
@@ -72,7 +85,9 @@ public final class AudioDecoder: @unchecked Sendable {
/// Feeding the renderer more channels than the session actually outputs
/// fails the renderer outright (silence, wedged pipeline) — configuring
/// the session for surround is the app's job (PLAN.md §2.1), the engine
- /// just honors whatever was negotiated at session-creation time.
+ /// just honors whatever was negotiated. This reads it once, for a decoder
+ /// of fixed width; `PlayerSession` follows route changes instead
+ /// (`AudioOutputRoute`).
public static func defaultMaxOutputChannels() -> Int {
#if canImport(UIKit)
return max(2, AVAudioSession.sharedInstance().outputNumberOfChannels)
@@ -84,16 +99,32 @@ public final class AudioDecoder: @unchecked Sendable {
#endif
}
- public init(
+ /// A decoder whose output width is fixed at `maxOutputChannels`.
+ public convenience init(
parameters: CodecParameters,
input: Channel,
output: Channel,
maxOutputChannels: Int = AudioDecoder.defaultMaxOutputChannels()
+ ) {
+ self.init(
+ parameters: parameters,
+ input: input,
+ output: output,
+ route: AudioOutputRoute(outputChannels: maxOutputChannels)
+ )
+ }
+
+ /// A decoder whose output follows `route` as it changes.
+ public init(
+ parameters: CodecParameters,
+ input: Channel,
+ output: Channel,
+ route: AudioOutputRoute
) {
self.parameters = parameters
self.input = input
self.output = output
- self.maxOutputChannels = max(1, maxOutputChannels)
+ self.route = route
var continuation: AsyncStream.Continuation!
events = AsyncStream(bufferingPolicy: .unbounded) { continuation = $0 }
eventSink = continuation
@@ -329,12 +360,17 @@ public final class AudioDecoder: @unchecked Sendable {
let format = frame.pointee.format
let rate = frame.pointee.sample_rate
let channels = frame.pointee.ch_layout.nb_channels
+ let routeChannels = route.outputChannels
- if swrContext != nil, format == swrSourceFormat, rate == swrSourceRate, channels == swrSourceChannels {
+ if swrContext != nil, format == swrSourceFormat, rate == swrSourceRate, channels == swrSourceChannels,
+ routeChannels == swrRouteChannels {
return true
}
- emitPending() // samples of the previous format cannot share a buffer
+ // Samples of the previous layout cannot share a buffer. Same-rate
+ // conversion holds nothing back inside swr, so emitting the pending
+ // buffer is all a switch needs to land on a frame boundary.
+ emitPending()
swr_free(&swrContext)
// swr needs concrete layouts on both sides to build a (down)mix
// matrix; sources reporting an unspecified order (routine for PCM)
@@ -346,12 +382,27 @@ public final class AudioDecoder: @unchecked Sendable {
av_channel_layout_default(&inputLayout, channels)
}
defer { av_channel_layout_uninit(&inputLayout) }
+
av_channel_layout_uninit(&outputLayout)
- if channels > Int32(maxOutputChannels) {
- av_channel_layout_default(&outputLayout, Int32(maxOutputChannels))
+ var matrix: [Double]?
+ if inputLayout.order == AV_CHANNEL_ORDER_NATIVE {
+ let source = SpeakerSet(rawValue: inputLayout.u.mask)
+ let plan = AudioLayoutPolicy.plan(source: source, routeChannels: routeChannels)
+ if plan.remixes {
+ av_channel_layout_from_mask(&outputLayout, plan.output.rawValue)
+ matrix = DownmixMatrix.flatCoefficients(from: source, to: plan.output)
+ } else {
+ av_channel_layout_copy(&outputLayout, &inputLayout)
+ }
+ } else if channels > Int32(routeChannels) {
+ // No positions to plan with (more channels than FFmpeg has a
+ // default order for): fold to the route's default layout and let
+ // swr do what it can.
+ av_channel_layout_default(&outputLayout, Int32(routeChannels))
} else {
av_channel_layout_copy(&outputLayout, &inputLayout)
}
+
var newContext: OpaquePointer?
let result = swr_alloc_set_opts2(
&newContext,
@@ -359,7 +410,16 @@ public final class AudioDecoder: @unchecked Sendable {
&inputLayout, AVSampleFormat(rawValue: format), rate,
0, nil
)
+ // A custom matrix goes in between allocation and init. Without one
+ // (no remix, or a layout pair the builder refused) swr builds its own
+ // from the two layouts.
+ if result >= 0, let created = newContext, let matrix {
+ _ = matrix.withUnsafeBufferPointer {
+ swr_set_matrix(created, $0.baseAddress, Int32(DownmixMatrix.stride))
+ }
+ }
guard result >= 0, let created = newContext, swr_init(created) >= 0 else {
+ swr_free(&newContext)
handleDecodeError(nil, EngineError(code: .decodeFailed, message: "swresample init failed"))
return false
}
@@ -367,6 +427,15 @@ public final class AudioDecoder: @unchecked Sendable {
swrSourceFormat = format
swrSourceRate = rate
swrSourceChannels = channels
+ swrRouteChannels = routeChannels
+ let applied = AudioOutputLayout(
+ sourceChannels: Int(channels),
+ outputChannels: Int(outputLayout.nb_channels),
+ routeChannels: routeChannels
+ )
+ lock.lock()
+ appliedOutput = applied
+ lock.unlock()
return true
}
diff --git a/Sources/LumeEngineCore/Decode/AudioLayoutPolicy.swift b/Sources/LumeEngineCore/Decode/AudioLayoutPolicy.swift
new file mode 100644
index 0000000..b435039
--- /dev/null
+++ b/Sources/LumeEngineCore/Decode/AudioLayoutPolicy.swift
@@ -0,0 +1,169 @@
+internal import CFFmpeg
+
+/// Speaker positions in FFmpeg's native channel-mask bit order (`AV_CH_*`),
+/// which is also WAVE order and Core Audio's `AudioChannelBitmap` for bits
+/// 0...17. Kept free of FFmpeg types so the layout policy is testable as plain
+/// values.
+struct SpeakerSet: OptionSet, Hashable, Sendable {
+ let rawValue: UInt64
+
+ static let frontLeft = SpeakerSet(rawValue: 1 << 0)
+ static let frontRight = SpeakerSet(rawValue: 1 << 1)
+ static let frontCenter = SpeakerSet(rawValue: 1 << 2)
+ static let lowFrequency = SpeakerSet(rawValue: 1 << 3)
+ static let backLeft = SpeakerSet(rawValue: 1 << 4)
+ static let backRight = SpeakerSet(rawValue: 1 << 5)
+ static let sideLeft = SpeakerSet(rawValue: 1 << 9)
+ static let sideRight = SpeakerSet(rawValue: 1 << 10)
+
+ static let mono: SpeakerSet = [.frontCenter]
+ static let stereo: SpeakerSet = [.frontLeft, .frontRight]
+ /// 5.1 with the surrounds on the side pair (`AV_CH_LAYOUT_5POINT1`,
+ /// "5.1(side)"): the layout the AC-3 and E-AC-3 decoders — most surround
+ /// on IPTV — already hand the renderer.
+ static let surround51: SpeakerSet = [.frontLeft, .frontRight, .frontCenter, .lowFrequency, .sideLeft, .sideRight]
+ static let surround71: SpeakerSet = surround51.union([.backLeft, .backRight])
+
+ /// Positions a format description can name with a channel bitmap
+ /// (`SampleBufferBuilder`). Anything outside reaches the renderer without
+ /// a layout, and the renderer cannot place untagged multichannel audio.
+ static let bitmapExpressible = SpeakerSet(rawValue: 0x3FFFF)
+
+ var count: Int { rawValue.nonzeroBitCount }
+}
+
+/// What the audio lane hands the renderer, for diagnostics: the source's
+/// channel count, the count after the engine's remix, and the route width the
+/// remix was chosen for.
+public struct AudioOutputLayout: Sendable, Equatable, CustomStringConvertible {
+ public let sourceChannels: Int
+ public let outputChannels: Int
+ public let routeChannels: Int
+
+ /// The engine folds the source down to fit the route.
+ public var isDownmixed: Bool { outputChannels < sourceChannels }
+
+ public var description: String {
+ "\(sourceChannels)>\(outputChannels)ch@\(routeChannels)"
+ }
+}
+
+/// Which speaker layout the engine hands the audio renderer, given the
+/// source's layout and how many channels the output route carries.
+///
+/// The route width is the audio session's *negotiated* output
+/// (`AVAudioSession.outputNumberOfChannels`), not what the hardware could do:
+/// handing `AVSampleBufferAudioRenderer` more channels than the session
+/// outputs failed it outright on tvOS — silence and a wedged pipeline — which
+/// is why the engine mixes instead of leaving it to the system (see
+/// `AudioOutputRoute`).
+///
+/// Pure value logic: the decoder asks it once per source format and route
+/// width, and swresample does what it says.
+enum AudioLayoutPolicy {
+ struct Plan: Equatable, Sendable {
+ /// Speakers the renderer receives.
+ let output: SpeakerSet
+ /// The engine mixes the channels (down, or mono up to stereo) rather
+ /// than passing them through as decoded.
+ let remixes: Bool
+ }
+
+ /// - Parameters:
+ /// - source: the decoded layout, resolved to native positions.
+ /// - routeChannels: channels the output route carries.
+ static func plan(source: SpeakerSet, routeChannels: Int) -> Plan {
+ let route = max(1, routeChannels)
+ let channels = source.count
+
+ // A lone centre channel is placed on two speakers by the engine, not
+ // by whatever the route does with a centre it may not have: a stereo
+ // pair (two HomePods, a TV's speakers) has no centre speaker at all.
+ // Equal power on each side, the standard pan law.
+ if source == .mono, route >= 2 {
+ return Plan(output: .stereo, remixes: true)
+ }
+
+ // Fits the route and the renderer can name every channel: hand it
+ // over untouched. A receiver on HDMI gets discrete surround.
+ if channels <= route, source.isSubset(of: .bitmapExpressible) {
+ return Plan(output: source, remixes: false)
+ }
+
+ // Fold down to the widest standard layout both the route and the
+ // source fill. A layout the renderer cannot name (wide or top
+ // channels, a matrix-encoded stereo pair) is remixed to the standard
+ // one of its own width rather than passed through untagged.
+ let width = min(route, channels)
+ var target: SpeakerSet = width >= 8 ? .surround71
+ : width >= 6 ? .surround51
+ : width >= 2 ? .stereo
+ : .mono
+ // Never invent an LFE channel the source does not have.
+ if !source.contains(.lowFrequency) {
+ target.remove(.lowFrequency)
+ }
+ return Plan(output: target, remixes: true)
+ }
+}
+
+/// The coefficients the engine's remix uses, built by swresample's own
+/// matrix builder with the levels below.
+///
+/// Levels follow ITU-R BS.775 fold-down: centre and surrounds at −3 dB into
+/// their front side, LFE omitted (it carries nothing a stereo programme needs,
+/// and full-range speakers rarely reproduce it). The dialogue channel thus
+/// always reaches both front speakers — the failure this exists to prevent is
+/// a downmix that drops channels instead of mixing them, which loses the
+/// centre and with it the dialogue.
+///
+/// The matrix is not normalized, matching what swresample does on its own for
+/// float output: a 5.1 fold-down plays as loud as a stereo channel next to it
+/// on the same IPTV list. Normalizing would make every surround channel
+/// ~7.7 dB quieter than its stereo neighbours, which viewers hear as a volume
+/// jump on every zap; the cost is that peaks where every channel is near full
+/// scale at once can exceed 1.0 and clip at the output.
+enum DownmixMatrix {
+ static let centerLevel = 0.707_106_781_186_547_6
+ static let surroundLevel = 0.707_106_781_186_547_6
+ static let lfeLevel = 0.0
+
+ /// swresample's matrix bound (`SWR_CH_MAX`); its builder walks the full
+ /// square, so the buffer must span it whatever the layouts are.
+ static let stride = 64
+
+ /// Row-major coefficients, `[output][input]`, in native channel order;
+ /// nil when swresample cannot build a matrix between the two layouts
+ /// (e.g. a source without any front channel).
+ static func coefficients(from source: SpeakerSet, to output: SpeakerSet) -> [[Double]]? {
+ guard let flat = flatCoefficients(from: source, to: output) else { return nil }
+ return (0.. [Double]? {
+ var inputLayout = AVChannelLayout()
+ var outputLayout = AVChannelLayout()
+ defer {
+ av_channel_layout_uninit(&inputLayout)
+ av_channel_layout_uninit(&outputLayout)
+ }
+ guard av_channel_layout_from_mask(&inputLayout, source.rawValue) >= 0,
+ av_channel_layout_from_mask(&outputLayout, output.rawValue) >= 0
+ else { return nil }
+
+ var matrix = [Double](repeating: 0, count: stride * stride)
+ let result = matrix.withUnsafeMutableBufferPointer { buffer in
+ swr_build_matrix2(
+ &inputLayout, &outputLayout,
+ centerLevel, surroundLevel, lfeLevel,
+ Double(Int32.max), // no normalization, see the type docs
+ 1.0, buffer.baseAddress, stride,
+ AV_MATRIX_ENCODING_NONE, nil
+ )
+ }
+ return result >= 0 ? matrix : nil
+ }
+}
diff --git a/Sources/LumeEngineCore/Errors/EngineError.swift b/Sources/LumeEngineCore/Errors/EngineError.swift
index 6d645c1..c93a54d 100644
--- a/Sources/LumeEngineCore/Errors/EngineError.swift
+++ b/Sources/LumeEngineCore/Errors/EngineError.swift
@@ -25,6 +25,13 @@ public struct EngineError: Error, Sendable, CustomStringConvertible, Equatable {
/// ends arrives as `.ioError` instead: FFmpeg cannot tell it from a
/// dropped connection.)
case sourceEnded
+ /// Audio stopped reaching the renderer while video plays on. Not
+ /// fatal: the session keeps playing, without sound, and recovers by
+ /// itself if audio comes back. `PlayerSession.audioHealth` says which
+ /// stage lost it. (An audio decoder that gives up reports its own
+ /// `.decodeFailed` instead, and a failed audio renderer fails the
+ /// session with `.renderFailed`.)
+ case audioLost
}
/// The HTTP status class a source answered with, when FFmpeg reported one.
diff --git a/Sources/LumeEngineCore/Media/AudioOutputRoute.swift b/Sources/LumeEngineCore/Media/AudioOutputRoute.swift
new file mode 100644
index 0000000..1d6e014
--- /dev/null
+++ b/Sources/LumeEngineCore/Media/AudioOutputRoute.swift
@@ -0,0 +1,140 @@
+#if os(iOS) || os(tvOS) || os(visionOS)
+import AVFAudio
+#endif
+import Foundation
+import Synchronization
+
+/// How many channels the audio output route carries, kept current while the
+/// route changes under a playing session (HDMI receiver → AirPlay → HomePods).
+///
+/// The audio decoder reads it for every decoded frame and folds its output to
+/// fit (`AudioLayoutPolicy`), so a route change takes effect at the next frame
+/// boundary; the renderer reads it to keep audio decoded for a wider, earlier
+/// route away from a narrower one.
+///
+/// The width is what the audio session *negotiated*
+/// (`outputNumberOfChannels`), never what the hardware could carry
+/// (`maximumOutputNumberOfChannels`): multichannel LPCM only flows once the
+/// host asks for it with `setPreferredOutputNumberOfChannels`, and a renderer
+/// fed more channels than the session outputs failed outright on tvOS.
+/// Configuring and activating the session is the host's job — this type only
+/// observes it.
+///
+/// `system()` follows the shared `AVAudioSession`. On macOS there is no audio
+/// session; it reports stereo, which is what the default output overwhelmingly
+/// is. A host that knows better (a macOS multichannel interface), and tests,
+/// create one with `init(outputChannels:)` and drive it with `update`.
+public final class AudioOutputRoute: Sendable {
+ public struct Snapshot: Sendable, Equatable, CustomStringConvertible {
+ /// Channels the route carries for this app now.
+ public var outputChannels: Int
+ /// Channels the route could carry if the host asked for them; nil
+ /// where nothing reports it. More than `outputChannels` on a surround
+ /// route means the host did not ask for the route's full width.
+ public var maximumOutputChannels: Int?
+ /// Output port types (`HDMIOutput`, `AirPlay`, …). Types only, never
+ /// the user-visible port names.
+ public var ports: [String]
+ /// How often the width has changed since the route was created.
+ public var changes: Int
+
+ public var description: String {
+ var line = "\(outputChannels)ch"
+ if let maximumOutputChannels { line += "/\(maximumOutputChannels)" }
+ if !ports.isEmpty { line += " " + ports.joined(separator: "+") }
+ return line
+ }
+ }
+
+ private let state: Mutex
+ private let observers = Mutex<[any NSObjectProtocol]>([])
+
+ /// A route of fixed width that only `update` changes.
+ public init(outputChannels: Int, maximumOutputChannels: Int? = nil, ports: [String] = []) {
+ state = Mutex(Snapshot(
+ outputChannels: max(1, outputChannels),
+ maximumOutputChannels: maximumOutputChannels,
+ ports: ports,
+ changes: 0
+ ))
+ }
+
+ deinit {
+ for observer in observers.withLock({ $0 }) {
+ NotificationCenter.default.removeObserver(observer)
+ }
+ }
+
+ /// The route of the shared audio session, followed through route changes.
+ public static func system() -> AudioOutputRoute {
+ #if os(iOS) || os(tvOS) || os(visionOS)
+ let current = sessionSnapshot()
+ let route = AudioOutputRoute(
+ outputChannels: current.outputChannels,
+ maximumOutputChannels: current.maximumOutputChannels,
+ ports: current.ports
+ )
+ route.followSession()
+ return route
+ #else
+ return AudioOutputRoute(outputChannels: 2)
+ #endif
+ }
+
+ /// Channels the route carries now. Cheap: read once per decoded frame.
+ public var outputChannels: Int {
+ state.withLock { $0.outputChannels }
+ }
+
+ public var snapshot: Snapshot {
+ state.withLock { $0 }
+ }
+
+ /// Reports a new width (and, when known, capability and ports).
+ public func update(outputChannels: Int, maximumOutputChannels: Int? = nil, ports: [String]? = nil) {
+ state.withLock { snapshot in
+ let width = max(1, outputChannels)
+ if width != snapshot.outputChannels { snapshot.changes += 1 }
+ snapshot.outputChannels = width
+ if let maximumOutputChannels { snapshot.maximumOutputChannels = maximumOutputChannels }
+ if let ports { snapshot.ports = ports }
+ }
+ }
+
+ #if os(iOS) || os(tvOS) || os(visionOS)
+ /// The session's current output. Clamped to stereo at the bottom: an
+ /// inactive session reports 0 or 1, and a stereo fold-down is right for
+ /// every real route that is narrower.
+ private static func sessionSnapshot() -> (outputChannels: Int, maximumOutputChannels: Int, ports: [String]) {
+ let session = AVAudioSession.sharedInstance()
+ return (
+ max(2, session.outputNumberOfChannels),
+ session.maximumOutputNumberOfChannels,
+ session.currentRoute.outputs.map(\.portType.rawValue)
+ )
+ }
+
+ /// Route changes are posted on a system thread; the width is re-read
+ /// there. A change of the negotiated width alone (the host asking for more
+ /// channels) arrives as `.routeConfigurationChange`, so every reason is
+ /// taken.
+ private func followSession() {
+ // The token is created inside the lock so it never leaves the
+ // mutex's region (it is not Sendable).
+ observers.withLock { observers in
+ observers.append(NotificationCenter.default.addObserver(
+ forName: AVAudioSession.routeChangeNotification,
+ object: nil,
+ queue: nil
+ ) { [weak self] _ in
+ let current = Self.sessionSnapshot()
+ self?.update(
+ outputChannels: current.outputChannels,
+ maximumOutputChannels: current.maximumOutputChannels,
+ ports: current.ports
+ )
+ })
+ }
+ }
+ #endif
+}
diff --git a/Sources/LumeEngineCore/Render/SystemRenderer.swift b/Sources/LumeEngineCore/Render/SystemRenderer.swift
index 3abb97a..f2a30fb 100644
--- a/Sources/LumeEngineCore/Render/SystemRenderer.swift
+++ b/Sources/LumeEngineCore/Render/SystemRenderer.swift
@@ -51,6 +51,8 @@ public final class SystemRenderer: @unchecked Sendable {
private var audioFormatCache: (description: CMAudioFormatDescription, sampleRate: Int, channels: Int, channelBitmap: UInt64)?
// Audio-queue-confined; serial changes re-anchor it after a seek flush.
private var audioTimeline = AudioTimeline()
+ /// Route width at the previous audio pump pass. Audio-queue-confined.
+ private var lastAudioRouteChannels = Int.max
/// Audio enqueued but not yet played, kept so it can be enqueued again
/// after the renderer drops it on its own (see `refillAfterAutomaticFlush`).
@@ -60,6 +62,7 @@ public final class SystemRenderer: @unchecked Sendable {
let serial: UInt64
/// End of the sample on the engine timeline (µs).
let end: Int64
+ let channels: Int
}
private var retainedAudio: [RetainedAudio] = []
/// Backstop only: the renderer stops asking for audio once it holds a
@@ -88,7 +91,13 @@ public final class SystemRenderer: @unchecked Sendable {
private var audioLaneAttached = false
private var reportedFailure = false
- public init(muted: Bool = false) {
+ /// The route the audio lane's decoder fits its output to. Audio decoded
+ /// for a wider route before it narrowed is held back from the renderer
+ /// (see `pumpAudio`); nil feeds every frame.
+ private let audioOutputRoute: AudioOutputRoute?
+
+ public init(muted: Bool = false, audioOutputRoute: AudioOutputRoute? = nil) {
+ self.audioOutputRoute = audioOutputRoute
displayLayer = AVSampleBufferDisplayLayer()
displayLayer.videoGravity = .resizeAspect
videoRenderer = displayLayer.sampleBufferRenderer
@@ -171,7 +180,11 @@ public final class SystemRenderer: @unchecked Sendable {
// answer; that audio is retained too, so drop it and re-enqueue
// everything in order.
audioRenderer.flush()
- for entry in retainedAudio where entry.serial == serial && (!MediaTime.isValid(flushTime) || entry.end > flushTime) {
+ // A route change is the usual cause of the flush; audio retained for
+ // a wider route than the new one stays out (see `pumpAudio`).
+ let routeChannels = audioOutputRoute?.outputChannels ?? .max
+ for entry in retainedAudio where entry.serial == serial && entry.channels <= routeChannels
+ && (!MediaTime.isValid(flushTime) || entry.end > flushTime) {
audioRenderer.enqueue(entry.sample)
}
// A seek that flushed while this ran may have been overtaken by the
@@ -371,6 +384,10 @@ public final class SystemRenderer: @unchecked Sendable {
)
}
+ var audioRendererFailed: Bool {
+ audioRenderer.status == .failed
+ }
+
public func shutdown() {
lock.lock()
stopped = true
@@ -434,12 +451,25 @@ public final class SystemRenderer: @unchecked Sendable {
lock.unlock()
guard let input, !isStopped else { return }
+ // The route narrowed (HDMI surround → AirPlay stereo): audio already
+ // decoded for the wider route must not reach the renderer, which
+ // fails on more channels than the route carries. The decoder refits
+ // at its next frame; until then what is queued for the old width is
+ // dropped, and whatever the renderer still holds of it is replaced by
+ // the retained audio that fits — usually nothing, so the switch costs
+ // the renderer's lead in silence, as a route change does anyway.
+ let routeChannels = audioOutputRoute?.outputChannels ?? .max
+ if routeChannels < lastAudioRouteChannels {
+ refillAfterAutomaticFlush(from: currentTime)
+ }
+ lastAudioRouteChannels = routeChannels
+
while audioRenderer.isReadyForMoreMediaData {
guard let frame = input.tryReceive() else {
scheduleAudioRetry()
return
}
- guard frame.serial == serial else { continue }
+ guard frame.serial == serial, frame.channels <= routeChannels else { continue }
do {
let sample = try SampleBufferBuilder.audio(
from: frame,
@@ -452,7 +482,7 @@ public final class SystemRenderer: @unchecked Sendable {
if !MediaTime.isValid(firstEnqueuedAudioPTS) { firstEnqueuedAudioPTS = frame.pts }
if firstAudioEnqueue == nil { firstAudioEnqueue = ProcessInfo.processInfo.systemUptime }
lock.unlock()
- retainAudio(sample, serial: serial, end: frame.pts + frame.duration)
+ retainAudio(sample, serial: serial, end: frame.pts + frame.duration, channels: frame.channels)
} catch {
continue
}
@@ -460,13 +490,13 @@ public final class SystemRenderer: @unchecked Sendable {
}
/// Keeps `sample` until the clock has played past it. Audio queue only.
- private func retainAudio(_ sample: CMSampleBuffer, serial: UInt64, end: Int64) {
+ private func retainAudio(_ sample: CMSampleBuffer, serial: UInt64, end: Int64, channels: Int) {
let now = currentTime
retainedAudio.removeAll { entry in
entry.serial != serial || (MediaTime.isValid(now) && entry.end <= now)
}
if retainedAudio.count >= Self.maxRetainedAudio { retainedAudio.removeFirst() }
- retainedAudio.append(RetainedAudio(sample: sample, serial: serial, end: end))
+ retainedAudio.append(RetainedAudio(sample: sample, serial: serial, end: end, channels: channels))
}
private func scheduleVideoRetry() {
diff --git a/Sources/LumeEngineCore/Session/AudioHealth.swift b/Sources/LumeEngineCore/Session/AudioHealth.swift
new file mode 100644
index 0000000..5c69f4f
--- /dev/null
+++ b/Sources/LumeEngineCore/Session/AudioHealth.swift
@@ -0,0 +1,153 @@
+import Foundation
+
+/// Whether the session's audio is reaching the viewer, as far as the engine
+/// can tell. Read `PlayerSession.audioHealth`; it is also in `Diagnostics`.
+///
+/// Exists because audio can die while video plays on: a decoder that gives up
+/// leaves its lane at end of stream, and the session — correctly — keeps
+/// showing the picture. Without this the only trace was one `.error` event,
+/// and the viewer sat in front of a silent channel.
+public enum AudioHealth: Sendable, Equatable, CustomStringConvertible {
+ /// Audio is playing, or nothing says otherwise yet.
+ case healthy
+ /// The session has no audio lane: the source carries no audio, or the
+ /// host opened it without (`enableAudio`) or turned it off.
+ case absent
+ /// The audio lane has stopped while the session plays on.
+ case lost(AudioLossReason)
+
+ public var description: String {
+ switch self {
+ case .healthy: "ok"
+ case .absent: "none"
+ case .lost(let reason): "lost(\(reason.rawValue))"
+ }
+ }
+}
+
+/// Which stage the audio was lost in.
+public enum AudioLossReason: String, Sendable, Equatable {
+ /// The audio decoder gave up on the stream (its error went out as an
+ /// `.error` event). Permanent for the track: switching tracks, or audio
+ /// off and on, builds a new decoder.
+ case decoderFailed
+ /// The system audio renderer failed. The session fails with
+ /// `.renderFailed` as well; this records that it was the audio lane.
+ case rendererFailed
+ /// The clock runs and video plays, but for a while now the audio renderer
+ /// has had nothing to play at the playhead: decoded audio stopped reaching
+ /// it, or reached it too late to play. Reported once per episode as an
+ /// `.error` with `EngineError.Code.audioLost`; clears by itself when audio
+ /// comes back.
+ case starved
+}
+
+/// Decides `AudioHealth` from what the session observes on its monitor tick.
+///
+/// Pure value logic, so every branch is testable without a pipeline that has
+/// to be made to lose its audio.
+///
+/// The starvation judgement is deliberately narrow, so that moments without
+/// audio that are part of normal playback never count: it needs a running
+/// clock (not paused, buffering or seeking), a video lane that is itself ahead
+/// of the playhead (a whole pipeline running dry is buffering, which the
+/// session handles), an audio lane still expecting data (not at its end), and
+/// the playhead to have moved `threshold` seconds of media with no audio
+/// enqueued beyond it. A gap *inside* the audio does not count either: the
+/// renderer's high-water mark is past the gap as soon as audio after it was
+/// enqueued.
+struct AudioLossDetector {
+ /// Seconds of media the playhead must cover without audio ahead of it.
+ static let threshold = 2.0
+
+ struct Observation: Equatable {
+ /// An audio lane is attached.
+ var hasAudioLane: Bool
+ /// The audio decoder reached the end of its stream: nothing more is
+ /// coming, and silence after it is the source's end, not a loss.
+ var audioEnded: Bool
+ /// The audio renderer reports `.failed`.
+ var audioRendererFailed: Bool
+ /// The session plays and the clock rate is non-zero.
+ var clockRunning: Bool
+ /// Playhead, seconds; nil while the clock has no valid time.
+ var playhead: Double?
+ /// Audio is enqueued beyond the playhead.
+ var audioAhead: Bool
+ /// Video is enqueued beyond the playhead; nil without a video lane.
+ var videoAhead: Bool?
+ /// The session's seek serial: a seek rebuilds every queue, so any
+ /// episode in progress is void.
+ var serial: UInt64
+ }
+
+ private(set) var health: AudioHealth = .absent
+ /// A loss only a new decoder clears.
+ private var permanent: AudioLossReason?
+ /// Playhead when audio was first seen missing; nil while it is not.
+ private var dryFrom: Double?
+ private var serial: UInt64 = 0
+
+ /// The audio decoder failed for good.
+ mutating func decoderFailed() {
+ permanent = .decoderFailed
+ health = .lost(.decoderFailed)
+ }
+
+ /// The audio lane was rebuilt or removed (track switch, audio off/on):
+ /// whatever was wrong with the old one is gone with it.
+ mutating func laneReplaced() {
+ permanent = nil
+ dryFrom = nil
+ health = .healthy
+ }
+
+ /// Updates `health`; returns the reason when a starvation episode begins,
+ /// which is the moment to tell the host.
+ mutating func observe(_ observation: Observation) -> AudioLossReason? {
+ guard observation.hasAudioLane else {
+ permanent = nil
+ dryFrom = nil
+ health = .absent
+ return nil
+ }
+ if health == .absent {
+ health = .healthy
+ }
+ if observation.audioRendererFailed {
+ permanent = .rendererFailed
+ }
+ if let permanent {
+ dryFrom = nil
+ health = .lost(permanent)
+ return nil
+ }
+ if observation.serial != serial {
+ serial = observation.serial
+ dryFrom = nil
+ health = .healthy
+ }
+ if observation.audioEnded || observation.audioAhead {
+ dryFrom = nil
+ health = .healthy
+ return nil
+ }
+ // Not judged: paused, buffering or seeking (the clock does not run),
+ // no time yet, or no healthy video to compare against. A standing
+ // verdict stays until audio is seen ahead again.
+ guard observation.clockRunning,
+ let playhead = observation.playhead,
+ observation.videoAhead == true
+ else {
+ dryFrom = nil
+ return nil
+ }
+ guard let start = dryFrom, playhead >= start else {
+ dryFrom = playhead
+ return nil
+ }
+ guard playhead - start >= Self.threshold, health != .lost(.starved) else { return nil }
+ health = .lost(.starved)
+ return .starved
+ }
+}
diff --git a/Sources/LumeEngineCore/Session/PlayerSession.swift b/Sources/LumeEngineCore/Session/PlayerSession.swift
index 3d91295..d115d7d 100644
--- a/Sources/LumeEngineCore/Session/PlayerSession.swift
+++ b/Sources/LumeEngineCore/Session/PlayerSession.swift
@@ -70,6 +70,11 @@ public struct PlayerConfiguration: Sendable {
public var autoEnableForcedSubtitlesForForeignAudio = false
/// Seconds without clock progress (while expected) before `.stalled` fires.
public var stallThreshold: Double = 8
+ /// The output route the audio is fitted to. nil follows the shared audio
+ /// session through route changes (iOS/tvOS/visionOS; stereo on macOS).
+ /// Pass one to pin the width or drive it yourself, e.g. for a macOS
+ /// multichannel interface.
+ public var audioOutputRoute: AudioOutputRoute?
public init() {}
}
@@ -167,9 +172,20 @@ public actor PlayerSession {
public private(set) var selectedSubtitleTrackIndex: Int32?
public private(set) var usingExternalSubtitles = false
+ /// Width the audio lane is fitted to; shared with its decoder and the renderer.
+ private let audioRoute: AudioOutputRoute
+ private var audioLoss = AudioLossDetector()
+
+ /// Whether audio is reaching the viewer (see `AudioHealth`).
+ public var audioHealth: AudioHealth { audioLoss.health }
+
+ /// The layout the audio lane hands the renderer; nil without audio.
+ public var audioOutput: AudioOutputLayout? { audioDecoder?.currentOutput }
+
public init(configuration: PlayerConfiguration = PlayerConfiguration()) {
self.configuration = configuration
- renderer = SystemRenderer(muted: configuration.muted)
+ audioRoute = configuration.audioOutputRoute ?? AudioOutputRoute.system()
+ renderer = SystemRenderer(muted: configuration.muted, audioOutputRoute: audioRoute)
var continuation: AsyncStream.Continuation!
events = AsyncStream(bufferingPolicy: .unbounded) { continuation = $0 }
eventSink = continuation
@@ -271,7 +287,7 @@ public actor PlayerSession {
let packets = makeAudioPacketChannel()
let frames = Channel(capacity: configuration.audioQueueDepth, measure: { $0.duration })
demuxer.attach(channel: packets, toStream: track.index)
- let decoder = AudioDecoder(parameters: parameters, input: packets, output: frames)
+ let decoder = AudioDecoder(parameters: parameters, input: packets, output: frames, route: audioRoute)
audioPackets = packets
audioFrames = frames
audioDecoder = decoder
@@ -428,6 +444,12 @@ public actor PlayerSession {
public let lastStallReason: StallReason?
/// The display mode the video wants (see `displayCriteria`).
public let displayCriteria: ContentDisplayCriteria?
+ /// Whether audio is reaching the viewer.
+ public let audioHealth: AudioHealth
+ /// Source → output channels of the audio lane; nil without one.
+ public let audioOutput: AudioOutputLayout?
+ /// The output route the audio is fitted to.
+ public let audioRoute: AudioOutputRoute.Snapshot
public var description: String {
func lane(_ queue: (count: Int, seconds: Double)?, lead: Double, first: Double) -> String {
@@ -454,6 +476,8 @@ public actor PlayerSession {
)
if let lastStallReason { line += " stall=\(lastStallReason.rawValue)" }
if let displayCriteria { line += " display=\(displayCriteria)" }
+ line += " aout=\(audioOutput.map(String.init(describing:)) ?? "off") route=\(audioRoute)"
+ if case .lost = audioHealth { line += " audio=\(audioHealth)" }
line += " startup=[\(startup)]"
return line
}
@@ -490,7 +514,10 @@ public actor PlayerSession {
replayedPackets: demuxer?.replayedPackets ?? 0,
startup: startup,
lastStallReason: lastStallReason,
- displayCriteria: displayCriteria
+ displayCriteria: displayCriteria,
+ audioHealth: audioHealth,
+ audioOutput: audioOutput,
+ audioRoute: audioRoute.snapshot
)
}
@@ -665,6 +692,7 @@ public actor PlayerSession {
audioFrames = nil
selectedAudioTrackIndex = nil
audioAtEOF = false
+ audioLoss.laneReplaced()
renderer.attach(video: videoFrames, audio: nil)
}
@@ -678,11 +706,12 @@ public actor PlayerSession {
let packets = makeAudioPacketChannel()
let frames = Channel(capacity: configuration.audioQueueDepth, measure: { $0.duration })
demuxer.attach(channel: packets, toStream: trackIndex)
- let decoder = AudioDecoder(parameters: parameters, input: packets, output: frames)
+ let decoder = AudioDecoder(parameters: parameters, input: packets, output: frames, route: audioRoute)
audioPackets = packets
audioFrames = frames
audioDecoder = decoder
audioAtEOF = false
+ audioLoss.laneReplaced()
selectedAudioTrackIndex = trackIndex
let pump = Task { [events = decoder.events] in
for await event in events {
@@ -840,7 +869,12 @@ public actor PlayerSession {
// One lane dying is not necessarily fatal (e.g. broken audio track,
// healthy video) — surface it; the monitor decides on full failure.
eventSink.yield(.error(error))
- if isVideo { videoAtEOF = true } else { audioAtEOF = true }
+ if isVideo {
+ videoAtEOF = true
+ } else {
+ audioAtEOF = true
+ audioLoss.decoderFailed()
+ }
lastError = error
}
}
@@ -855,9 +889,36 @@ public actor PlayerSession {
evaluatePlayback()
runStallWatchdog()
noteFirstFrameShown()
+ evaluateAudioHealth()
}
}
+ /// Audio that stops while the picture plays on is a failure the viewer
+ /// notices and nothing else would report (see `AudioLossDetector` for
+ /// what does not count).
+ private func evaluateAudioHealth() {
+ let playhead = renderer.currentTime
+ let highWater = renderer.enqueuedHighWaterMark
+ func ahead(_ mark: Int64) -> Bool {
+ MediaTime.isValid(mark) && MediaTime.isValid(playhead) && mark > playhead
+ }
+ let observation = AudioLossDetector.Observation(
+ hasAudioLane: audioDecoder != nil,
+ audioEnded: audioAtEOF,
+ audioRendererFailed: renderer.audioRendererFailed,
+ clockRunning: state == .playing && renderer.rate > 0,
+ playhead: MediaTime.isValid(playhead) ? MediaTime.seconds(playhead) : nil,
+ audioAhead: ahead(highWater.audio),
+ videoAhead: videoDecoder != nil ? ahead(highWater.video) : nil,
+ serial: currentSerial
+ )
+ guard audioLoss.observe(observation) == .starved else { return }
+ eventSink.yield(.error(EngineError(
+ code: .audioLost,
+ message: "audio stopped reaching the renderer while video plays"
+ )))
+ }
+
/// A container seek can "succeed" while landing somewhere else entirely —
/// matroska falls back to the earliest cluster when its cues are
/// unreachable (end-of-file cues over non-range HTTP is the classic IPTV
diff --git a/Tests/LumeEngineCoreTests/AudioRouteTests.swift b/Tests/LumeEngineCoreTests/AudioRouteTests.swift
new file mode 100644
index 0000000..f792888
--- /dev/null
+++ b/Tests/LumeEngineCoreTests/AudioRouteTests.swift
@@ -0,0 +1,548 @@
+import CFFmpeg
+import Foundation
+import Synchronization
+import Testing
+@testable import LumeEngineCore
+
+/// Output width that follows the route: the layout policy and the downmix
+/// matrix as plain values, then the decoder at forced route widths and across
+/// a route change mid-stream.
+@Suite("AudioLayoutPolicy")
+struct AudioLayoutPolicyTests {
+ private static let surround51Back: SpeakerSet = [.frontLeft, .frontRight, .frontCenter, .lowFrequency, .backLeft, .backRight]
+ private static let surround70: SpeakerSet = SpeakerSet.surround71.subtracting(.lowFrequency)
+ /// A matrix-encoded stereo pair (`AV_CH_STEREO_LEFT | AV_CH_STEREO_RIGHT`):
+ /// two channels the renderer cannot name.
+ private static let stereoDownmix = SpeakerSet(rawValue: 1 << 29 | 1 << 30)
+
+ private func plan(_ source: SpeakerSet, route: Int) -> AudioLayoutPolicy.Plan {
+ AudioLayoutPolicy.plan(source: source, routeChannels: route)
+ }
+
+ @Test("speaker sets are FFmpeg's native layouts")
+ func layoutsMatchFFmpeg() {
+ #expect(SpeakerSet.mono.rawValue == layoutMask("mono"))
+ #expect(SpeakerSet.stereo.rawValue == layoutMask("stereo"))
+ #expect(SpeakerSet.surround51.rawValue == layoutMask("5.1(side)"))
+ #expect(SpeakerSet.surround71.rawValue == layoutMask("7.1"))
+ }
+
+ @Test("multichannel goes out discrete when the route carries it")
+ func passthroughWhenItFits() {
+ for route in [6, 8] {
+ #expect(plan(.surround51, route: route) == .init(output: .surround51, remixes: false))
+ #expect(plan(Self.surround51Back, route: route) == .init(output: Self.surround51Back, remixes: false))
+ }
+ #expect(plan(.surround71, route: 8) == .init(output: .surround71, remixes: false))
+ for route in [2, 6, 8] {
+ #expect(plan(.stereo, route: route) == .init(output: .stereo, remixes: false))
+ }
+ }
+
+ @Test("a narrower route gets the widest standard layout both sides fill")
+ func foldsDownToTheRoute() {
+ #expect(plan(.surround51, route: 2) == .init(output: .stereo, remixes: true))
+ #expect(plan(Self.surround51Back, route: 2) == .init(output: .stereo, remixes: true))
+ #expect(plan(.surround71, route: 2) == .init(output: .stereo, remixes: true))
+ #expect(plan(.surround71, route: 6) == .init(output: .surround51, remixes: true))
+ #expect(plan(.surround71, route: 7) == .init(output: .surround51, remixes: true))
+ // Odd widths fall to the standard layout below them.
+ #expect(plan(.surround51, route: 4) == .init(output: .stereo, remixes: true))
+ }
+
+ @Test("no LFE channel is invented")
+ func noInventedLFE() {
+ #expect(plan(Self.surround70, route: 6) == .init(output: SpeakerSet.surround51.subtracting(.lowFrequency), remixes: true))
+ }
+
+ @Test("mono reaches both speakers of any route wider than one")
+ func monoBecomesStereo() {
+ for route in [2, 6, 8] {
+ #expect(plan(.mono, route: route) == .init(output: .stereo, remixes: true))
+ }
+ #expect(plan(.mono, route: 1) == .init(output: .mono, remixes: false))
+ #expect(plan(.mono, route: 0) == .init(output: .mono, remixes: false))
+ #expect(plan(.stereo, route: 1) == .init(output: .mono, remixes: true))
+ }
+
+ @Test("channels the renderer cannot name are remixed, not passed untagged")
+ func unnameableChannelsAreRemixed() {
+ #expect(plan(Self.stereoDownmix, route: 8) == .init(output: .stereo, remixes: true))
+ }
+}
+
+@Suite("DownmixMatrix")
+struct DownmixMatrixTests {
+ private let minus3dB = 0.707_106_781_186_547_6
+
+ /// The coefficient of input speaker `input` in output speaker `output`.
+ private func weight(
+ _ matrix: [[Double]], from source: SpeakerSet, to target: SpeakerSet,
+ _ input: SpeakerSet, into output: SpeakerSet
+ ) -> Double {
+ func index(_ speaker: SpeakerSet, in set: SpeakerSet) -> Int {
+ // Native order is ascending bit order.
+ (set.rawValue & (speaker.rawValue - 1)).nonzeroBitCount
+ }
+ return matrix[index(output, in: target)][index(input, in: source)]
+ }
+
+ @Test("5.1 to stereo: ITU fold-down, centre in both, LFE omitted, no cross-feed")
+ func surround51ToStereo() throws {
+ let source = SpeakerSet.surround51
+ let matrix = try #require(DownmixMatrix.coefficients(from: source, to: .stereo))
+ #expect(matrix.count == 2 && matrix.allSatisfy { $0.count == 6 })
+ func w(_ input: SpeakerSet, _ output: SpeakerSet) -> Double {
+ weight(matrix, from: source, to: .stereo, input, into: output)
+ }
+ #expect(w(.frontLeft, .frontLeft) == 1)
+ #expect(w(.frontRight, .frontRight) == 1)
+ #expect(w(.frontLeft, .frontRight) == 0)
+ #expect(w(.frontRight, .frontLeft) == 0)
+ #expect(abs(w(.frontCenter, .frontLeft) - minus3dB) < 1e-9)
+ #expect(abs(w(.frontCenter, .frontRight) - minus3dB) < 1e-9)
+ #expect(abs(w(.sideLeft, .frontLeft) - minus3dB) < 1e-9)
+ #expect(abs(w(.sideRight, .frontRight) - minus3dB) < 1e-9)
+ #expect(w(.sideLeft, .frontRight) == 0)
+ #expect(w(.lowFrequency, .frontLeft) == 0)
+ #expect(w(.lowFrequency, .frontRight) == 0)
+ }
+
+ @Test("7.1 to stereo: every surround folds into its own side")
+ func surround71ToStereo() throws {
+ let source = SpeakerSet.surround71
+ let matrix = try #require(DownmixMatrix.coefficients(from: source, to: .stereo))
+ func w(_ input: SpeakerSet, _ output: SpeakerSet) -> Double {
+ weight(matrix, from: source, to: .stereo, input, into: output)
+ }
+ for (input, side) in [(SpeakerSet.backLeft, SpeakerSet.frontLeft), (.sideLeft, .frontLeft),
+ (.backRight, .frontRight), (.sideRight, .frontRight)] {
+ #expect(abs(w(input, side) - minus3dB) < 1e-9)
+ }
+ #expect(w(.backLeft, .frontRight) == 0)
+ #expect(abs(w(.frontCenter, .frontLeft) - minus3dB) < 1e-9)
+ #expect(abs(w(.frontCenter, .frontRight) - minus3dB) < 1e-9)
+ }
+
+ @Test("7.1 to 5.1: fronts, centre and LFE pass straight; backs join the sides")
+ func surround71To51() throws {
+ let source = SpeakerSet.surround71
+ let target = SpeakerSet.surround51
+ let matrix = try #require(DownmixMatrix.coefficients(from: source, to: target))
+ func w(_ input: SpeakerSet, _ output: SpeakerSet) -> Double {
+ weight(matrix, from: source, to: target, input, into: output)
+ }
+ for speaker in [SpeakerSet.frontLeft, .frontRight, .frontCenter, .lowFrequency] {
+ #expect(w(speaker, speaker) == 1)
+ }
+ #expect(w(.backLeft, .sideLeft) > 0.5)
+ #expect(w(.backRight, .sideRight) > 0.5)
+ #expect(w(.frontCenter, .frontLeft) == 0, "a route with a centre speaker keeps dialogue on it")
+ }
+
+ @Test("mono to stereo: equal power on both sides")
+ func monoToStereo() throws {
+ let matrix = try #require(DownmixMatrix.coefficients(from: .mono, to: .stereo))
+ #expect(matrix.count == 2)
+ #expect(abs(matrix[0][0] - minus3dB) < 1e-9)
+ #expect(abs(matrix[1][0] - minus3dB) < 1e-9)
+ }
+
+ @Test("dialogue survives every stereo fold-down")
+ func dialogueNeverDropped() throws {
+ let sources: [SpeakerSet] = [
+ [.frontLeft, .frontRight, .frontCenter],
+ .surround51,
+ [.frontLeft, .frontRight, .frontCenter, .lowFrequency, .backLeft, .backRight],
+ .surround71,
+ SpeakerSet.surround51.union(SpeakerSet(rawValue: 1 << 8)), // 6.1, back centre
+ ]
+ for source in sources {
+ let plan = AudioLayoutPolicy.plan(source: source, routeChannels: 2)
+ let matrix = try #require(DownmixMatrix.coefficients(from: source, to: plan.output))
+ let center = (source.rawValue & (SpeakerSet.frontCenter.rawValue - 1)).nonzeroBitCount
+ #expect(matrix[0][center] > 0.5 && matrix[1][center] > 0.5, "centre lost folding \(source.rawValue)")
+ // Every input channel except the LFE reaches the output.
+ for input in 0.. 0, "channel \(input) of \(source.rawValue) dropped")
+ }
+ }
+ }
+}
+
+@Suite("AudioRoute", .serialized)
+struct AudioRouteDecodeTests {
+ /// Decodes an audio-only fixture with the decoder following `route`.
+ /// `onFrame` runs on the drain thread for each frame, in order — the place
+ /// to change the route mid-stream.
+ private func decode(
+ fixture: String,
+ route: AudioOutputRoute,
+ frameQueue: Int = 256,
+ onFrame: @escaping @Sendable (Int) -> Void = { _ in }
+ ) async throws -> [AudioFrame] {
+ let demuxer = Demuxer(url: try Fixtures.path(fixture))
+ var events = demuxer.events.makeAsyncIterator()
+ demuxer.start()
+ guard case .opened(let info)? = await events.next() else {
+ demuxer.shutdown()
+ throw EngineError(code: .openFailed, message: "\(fixture) failed to open")
+ }
+ defer { demuxer.shutdown() }
+
+ let track = try #require(info.audioTracks.first)
+ let parameters = try #require(demuxer.codecParameters(forStream: track.index))
+ let packets = Channel(capacity: 4096)
+ let frames = Channel(capacity: frameQueue, measure: { $0.duration })
+ demuxer.attach(channel: packets, toStream: track.index)
+
+ let decoder = AudioDecoder(parameters: parameters, input: packets, output: frames, route: route)
+ var decodeEvents = decoder.events.makeAsyncIterator()
+ decoder.start()
+ demuxer.resume()
+ defer { decoder.shutdown() }
+
+ let collector = ChannelDrain(frames, into: [AudioFrame]()) { collected, frame in
+ collected.append(frame)
+ onFrame(collected.count)
+ }
+ while let event = await events.next() {
+ if case .endOfStream = event {
+ decoder.signalEndOfStream()
+ break
+ }
+ }
+ guard case .endOfStream? = await decodeEvents.next() else {
+ throw EngineError(code: .decodeFailed, message: "\(fixture): decoder never reached EOF")
+ }
+ decoder.shutdown()
+ return await collector.value
+ }
+
+ /// Per-channel peak amplitude over all frames.
+ private func peaks(_ frames: [AudioFrame]) -> [Float] {
+ guard let channels = frames.first?.channels else { return [] }
+ var peaks = [Float](repeating: 0, count: channels)
+ for frame in frames {
+ let samples = frame.samples
+ for index in 0.. 0.05)
+ let stereoPeaks = peaks(stereo)
+ #expect(abs(stereoPeaks[0] - stereoPeaks[1]) < 1e-4)
+ #expect(abs(stereoPeaks[0] / center - 0.7071) < 0.01, "centre at \(stereoPeaks[0] / center) of its level")
+ // On a 5.1 route the centre stays on the centre speaker.
+ #expect(abs(peaks(fiveOne)[2] - center) < 1e-4)
+ }
+
+ @Test("TrueHD 5.1 at route widths 8 and 2", .timeLimit(.minutes(1)))
+ func truehdAtRouteWidths() async throws {
+ let wide = try await decode(fixture: "truehd.mkv", route: AudioOutputRoute(outputChannels: 8))
+ #expect(wide.allSatisfy { $0.channels == 6 && $0.channelBitmap == SpeakerSet.surround51.rawValue })
+
+ let stereo = try await decode(fixture: "truehd.mkv", route: AudioOutputRoute(outputChannels: 2))
+ #expect(!stereo.isEmpty)
+ #expect(stereo.allSatisfy { $0.channels == 2 })
+ let center = peaks(wide)[2]
+ let stereoPeaks = peaks(stereo)
+ #expect(center > 0.05)
+ #expect(abs(stereoPeaks[0] / center - 0.7071) < 0.01 && abs(stereoPeaks[1] / center - 0.7071) < 0.01)
+ let seconds = Double(stereo.reduce(0) { $0 + $1.sampleCount }) / Double(stereo[0].sampleRate)
+ #expect(abs(seconds - 4.0) < 0.1)
+ }
+
+ @Test("mono goes out on both speakers", .timeLimit(.minutes(1)))
+ func monoToStereo() async throws {
+ // basic.mp4's AAC track is a mono sine.
+ let mono = try await decode(fixture: "basic.mp4", route: AudioOutputRoute(outputChannels: 1))
+ #expect(mono.first?.channels == 1)
+ let stereo = try await decode(fixture: "basic.mp4", route: AudioOutputRoute(outputChannels: 2))
+ #expect(stereo.allSatisfy { $0.channels == 2 && $0.channelBitmap == SpeakerSet.stereo.rawValue })
+ let level = peaks(mono)[0]
+ let sides = peaks(stereo)
+ #expect(level > 0.05)
+ #expect(abs(sides[0] - sides[1]) < 1e-4)
+ #expect(abs(sides[0] / level - 0.7071) < 0.01)
+ }
+
+ @Test("a route change mid-stream switches layout at a frame boundary, losing nothing", .timeLimit(.minutes(1)))
+ func routeChangeMidStream() async throws {
+ let route = AudioOutputRoute(outputChannels: 8)
+ // A short frame queue keeps the decoder from finishing the file
+ // before the route changes.
+ let frames = try await decode(fixture: "surround71.mkv", route: route, frameQueue: 4) { count in
+ if count == 40 { route.update(outputChannels: 2) }
+ }
+ let widths = frames.map(\.channels)
+ let switchIndex = try #require(widths.firstIndex(of: 2), "the layout must follow the route")
+ #expect(switchIndex >= 40, "frames decoded for the old route are not rewritten")
+ #expect(widths[.. AudioLossDetector.Observation {
+ AudioLossDetector.Observation(
+ hasAudioLane: true, audioEnded: false, audioRendererFailed: false,
+ clockRunning: true, playhead: playhead,
+ audioAhead: audioAhead, videoAhead: videoAhead, serial: serial
+ )
+ }
+
+ /// Feeds `observation` with the playhead advanced in 0.2 s ticks over
+ /// `seconds`; returns every reason the detector reported.
+ private func run(
+ _ detector: inout AudioLossDetector, from start: Double, seconds: Double,
+ _ observation: (Double) -> AudioLossDetector.Observation
+ ) -> [AudioLossReason] {
+ var reported: [AudioLossReason] = []
+ var playhead = start
+ while playhead <= start + seconds + 1e-9 {
+ if let reason = detector.observe(observation(playhead)) { reported.append(reason) }
+ playhead += 0.2
+ }
+ return reported
+ }
+
+ @Test("no lane is absent; a lane starts healthy")
+ func absentAndHealthy() {
+ var detector = AudioLossDetector()
+ var observation = playing(at: 0)
+ observation.hasAudioLane = false
+ #expect(detector.observe(observation) == nil)
+ #expect(detector.health == .absent)
+ #expect(detector.observe(playing(at: 0)) == nil)
+ #expect(detector.health == .healthy)
+ }
+
+ @Test("audio missing at a running playhead while video plays is reported once, after the threshold")
+ func starvationReportedOnce() {
+ var detector = AudioLossDetector()
+ let early = run(&detector, from: 0, seconds: 1.6) { playing(at: $0, audioAhead: false) }
+ #expect(early.isEmpty, "not before \(AudioLossDetector.threshold) s")
+ #expect(detector.health == .healthy)
+ let later = run(&detector, from: 1.8, seconds: 3) { playing(at: $0, audioAhead: false) }
+ #expect(later == [.starved])
+ #expect(detector.health == .lost(.starved))
+ }
+
+ @Test("audio coming back clears the verdict, and a second episode is reported again")
+ func recovery() {
+ var detector = AudioLossDetector()
+ _ = run(&detector, from: 0, seconds: 3) { playing(at: $0, audioAhead: false) }
+ #expect(detector.health == .lost(.starved))
+ #expect(detector.observe(playing(at: 3.2)) == nil)
+ #expect(detector.health == .healthy)
+ let again = run(&detector, from: 3.4, seconds: 3) { playing(at: $0, audioAhead: false) }
+ #expect(again == [.starved])
+ }
+
+ @Test("moments without audio that belong to playback are not a loss")
+ func legitimateSilence() {
+ // Video dry too: the whole pipeline is starving, which is buffering.
+ var detector = AudioLossDetector()
+ #expect(run(&detector, from: 0, seconds: 5) { playing(at: $0, audioAhead: false, videoAhead: false) }.isEmpty)
+ // Audio-only source: nothing to compare against.
+ #expect(run(&detector, from: 0, seconds: 5) { playing(at: $0, audioAhead: false, videoAhead: nil) }.isEmpty)
+ // The audio track ended (VOD whose audio is shorter than its video).
+ #expect(run(&detector, from: 0, seconds: 5) {
+ var observation = playing(at: $0, audioAhead: false)
+ observation.audioEnded = true
+ return observation
+ }.isEmpty)
+ // Paused or buffering: the clock does not run.
+ #expect(run(&detector, from: 0, seconds: 5) {
+ var observation = playing(at: $0, audioAhead: false)
+ observation.clockRunning = false
+ return observation
+ }.isEmpty)
+ #expect(detector.health == .healthy)
+ }
+
+ @Test("a pause restarts the count; a seek voids the episode")
+ func pauseAndSeek() {
+ var detector = AudioLossDetector()
+ _ = run(&detector, from: 0, seconds: 1.6) { playing(at: $0, audioAhead: false) }
+ var paused = playing(at: 1.6, audioAhead: false)
+ paused.clockRunning = false
+ #expect(detector.observe(paused) == nil)
+ #expect(run(&detector, from: 1.6, seconds: 1.6) { playing(at: $0, audioAhead: false) }.isEmpty)
+
+ _ = run(&detector, from: 3.4, seconds: 3) { playing(at: $0, audioAhead: false) }
+ #expect(detector.health == .lost(.starved))
+ // Seek back: new serial, playhead earlier — a fresh start.
+ #expect(detector.observe(playing(at: 0.5, audioAhead: false, serial: 1)) == nil)
+ #expect(detector.health == .healthy)
+ }
+
+ @Test("decoder and renderer failures are permanent until the lane is rebuilt")
+ func permanentLosses() {
+ var detector = AudioLossDetector()
+ _ = detector.observe(playing(at: 0))
+ detector.decoderFailed()
+ #expect(detector.health == .lost(.decoderFailed))
+ // Audio enqueued before the failure still plays out; the verdict stands.
+ #expect(detector.observe(playing(at: 0.2)) == nil)
+ #expect(detector.health == .lost(.decoderFailed))
+ detector.laneReplaced()
+ #expect(detector.health == .healthy)
+
+ var failed = playing(at: 1)
+ failed.audioRendererFailed = true
+ #expect(detector.observe(failed) == nil)
+ #expect(detector.health == .lost(.rendererFailed))
+ #expect(detector.observe(playing(at: 1.2, serial: 4)) == nil)
+ #expect(detector.health == .lost(.rendererFailed))
+ }
+}
+
+@Suite("AudioRouteSession", .serialized)
+struct AudioRouteSessionTests {
+ private func eventually(
+ timeout: TimeInterval = 10,
+ _ condition: () async -> Bool
+ ) async -> Bool {
+ let deadline = Date(timeIntervalSinceNow: timeout)
+ while Date() < deadline {
+ if await condition() { return true }
+ try? await Task.sleep(for: .milliseconds(100))
+ }
+ return false
+ }
+
+ private func makeSession(route: AudioOutputRoute, enableAudio: Bool = true) -> PlayerSession {
+ var configuration = PlayerConfiguration()
+ configuration.muted = true
+ configuration.bufferTarget = 0.5
+ configuration.enableAudio = enableAudio
+ configuration.audioOutputRoute = route
+ return PlayerSession(configuration: configuration)
+ }
+
+ @Test("a playing session refits its audio when the route narrows and widens", .timeLimit(.minutes(1)))
+ func followsRouteChanges() async throws {
+ let route = AudioOutputRoute(outputChannels: 8)
+ let session = makeSession(route: route)
+ _ = try await session.open(url: try Fixtures.path("surround51.ts"))
+ await session.play()
+ #expect(await eventually { await session.position > 1 })
+ #expect(await session.audioOutput == AudioOutputLayout(sourceChannels: 6, outputChannels: 6, routeChannels: 8))
+
+ // HDMI receiver → AirPlay: the next decoded frame folds down to stereo.
+ route.update(outputChannels: 2)
+ let narrowed = await eventually { await session.audioOutput?.outputChannels == 2 }
+ #expect(narrowed, "audio must refit to the stereo route")
+ let before = await session.position
+ let continued = await eventually { await session.position > before + 1.5 }
+ #expect(continued, "playback must go on through the route change")
+ #expect(!session.renderer.audioRendererFailed)
+ #expect(await session.state == .playing)
+ #expect(await session.audioHealth == .healthy)
+
+ // And back to surround.
+ route.update(outputChannels: 6)
+ #expect(await eventually { await session.audioOutput?.outputChannels == 6 })
+ let resumed = await session.position
+ #expect(await eventually { await session.position > resumed + 1 })
+ #expect(!session.renderer.audioRendererFailed)
+
+ let diagnostics = await session.diagnostics
+ #expect(diagnostics.audioOutput?.routeChannels == 6)
+ #expect(diagnostics.audioRoute.changes == 2)
+ #expect(diagnostics.description.contains("aout=6>6ch@6"))
+ await session.shutdown()
+ }
+
+ @Test("audio that stops reaching the renderer while video plays is reported", .timeLimit(.minutes(1)))
+ func reportsLostAudio() async throws {
+ let session = makeSession(route: AudioOutputRoute(outputChannels: 2))
+ let errors = Mutex<[EngineError.Code]>([])
+ let listener = Task { [events = session.events] in
+ for await event in events {
+ if case .error(let error) = event { errors.withLock { $0.append(error.code) } }
+ }
+ }
+ _ = try await session.open(url: try Fixtures.path("surround51.ts"))
+ await session.play()
+ #expect(await eventually { await session.position > 1 })
+ #expect(await session.audioHealth == .healthy)
+
+ // The renderer stops taking audio; the clock and the picture go on.
+ session.renderer.audioRenderer.stopRequestingMediaData()
+ let lost = await eventually(timeout: 15) { await session.audioHealth == .lost(.starved) }
+ let health = await session.audioHealth
+ #expect(lost, "lost audio must be detected, health is \(health)")
+ // The event travels through the stream after the verdict is set.
+ let reported = await eventually(timeout: 2) { errors.withLock { $0.contains(.audioLost) } }
+ let codes = errors.withLock { $0 }
+ #expect(reported && codes.filter { $0 == .audioLost }.count == 1, "errors: \(codes)")
+ #expect(await session.state == .playing, "video keeps playing")
+ #expect(await session.diagnostics.description.contains("audio=lost(starved)"))
+
+ await session.shutdown()
+ listener.cancel()
+ }
+
+ @Test("a session without audio reports none", .timeLimit(.minutes(1)))
+ func noAudioLane() async throws {
+ let session = makeSession(route: AudioOutputRoute(outputChannels: 2), enableAudio: false)
+ _ = try await session.open(url: try Fixtures.path("basic.mp4"))
+ await session.play()
+ #expect(await eventually { await session.position > 0.5 })
+ #expect(await session.audioHealth == .absent)
+ #expect(await session.audioOutput == nil)
+ await session.shutdown()
+ }
+}
+
+private extension Array {
+ subscript(safe index: Int) -> Element? {
+ indices.contains(index) ? self[index] : nil
+ }
+}
+
+/// FFmpeg's mask for a named layout — the reference the speaker-set
+/// constants are checked against.
+private func layoutMask(_ name: String) -> UInt64 {
+ var layout = AVChannelLayout()
+ defer { av_channel_layout_uninit(&layout) }
+ guard av_channel_layout_from_string(&layout, name) >= 0 else { return 0 }
+ return layout.u.mask
+}
diff --git a/Tests/LumeEngineCoreTests/DecoderTests.swift b/Tests/LumeEngineCoreTests/DecoderTests.swift
index 33b5cb2..a69e7e1 100644
--- a/Tests/LumeEngineCoreTests/DecoderTests.swift
+++ b/Tests/LumeEngineCoreTests/DecoderTests.swift
@@ -333,7 +333,8 @@ struct DecoderTests {
let collected = await collector.value
let rate = try #require(collected.rates.first)
#expect(collected.rates.count == 1)
- #expect(collected.channels == [1], "sine fixture is mono")
+ // The sine fixture is mono; on a stereo route it goes out on both sides.
+ #expect(collected.channels == [2], "mono source on a stereo route")
let seconds = Double(collected.totalSamples) / Double(rate)
#expect(abs(seconds - 10.0) < 0.3, "expected ~10 s of audio, got \(seconds)")
let ptsValues = collected.ptsValues
diff --git a/Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh b/Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh
index befe268..d5105e6 100644
--- a/Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh
+++ b/Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh
@@ -18,6 +18,7 @@
# replay.ts — 8 s TS whose bytes jump back ~3 s at ~6 s, as a reconnect replays them
# ac4.ts — TS whose first audio (ger) is declared AC-4, which no decoder here plays, then eng AAC
# hdr10.mp4 — HEVC Main 10, BT.2020 PQ, mastering display + content light level SEI, HDR10+ per frame
+# surround51.ts — 20 s H.264 + AC-3 5.1 MPEG-TS, tone in the centre channel only
set -euo pipefail
FFMPEG="${FFMPEG:-ffmpeg}"
@@ -329,4 +330,15 @@ JSON
echo "generated: $OUT/hdr10.mp4"
fi
+# Broadcast surround the way IPTV carries it: H.264 + AC-3 5.1 in MPEG-TS,
+# long enough to change the output route mid-playback. The tone sits in the
+# centre channel alone, so a fold-down that drops channels instead of mixing
+# them goes silent.
+gen surround51.ts \
+ -f lavfi -i "testsrc2=duration=20:size=320x180:rate=25" \
+ -f lavfi -i "sine=frequency=440:duration=20" \
+ -filter_complex "[1:a]pan=5.1(side)|FC=c0[a]" -map 0:v -map "[a]" \
+ -c:v libx264 -preset ultrafast -pix_fmt yuv420p -g 25 \
+ -c:a ac3 -b:a 384k -shortest -f mpegts
+
echo "fixtures ready in $OUT"
From 6375c5382b0750da5d55298bb9e99b5ffd8ae7e8 Mon Sep 17 00:00:00 2001
From: Philipp Bischoff
Date: Mon, 12 Oct 2026 22:42:28 +0200
Subject: [PATCH 2/4] feat: subtitle cue lifecycle and CEA-608 closed captions
from the video
Cue lifecycle: decoders now hand the store display updates (start, end or
none, text, whether the track replaces the whole screen), and a pure
SubtitleTimeline decides how long each shows. Open cues (UINT32_MAX end:
PGS, DVB, captions) end at the next update of their track instead of
staying up for 49 days; the empty update replacing tracks send to clear
the screen now ends what is showing instead of being dropped; open cues
are capped at 16 s and ended by a seek; re-read ranges stay deduplicated;
the store is bounded at 8192 cues so a live session no longer grows
without limit. Subtitle decoders flush their state across seek serials.
SubtitleStore is now checked Sendable (Mutex) instead of @unchecked.
Closed captions: CEA-608 carried as A/53 side data on decoded video frames
is lifted off in VideoDecoder.receiveFrames by a ClosedCaptionTap, stamped
with the frame's PTS and trySend-ed to a SubtitleDecoder running ccaption
in real-time mode, anchored at the packet time. The video lane never
waits on it. A synthetic "CC1" subtitle track
(TrackInfo.embeddedClosedCaptionsIndex) appears in PlayerSession.mediaInfo
once caption content is seen, and selectSubtitleTrack accepts it.
Stacked on #46.
---
AGENTS.md | 4 +-
PLAN.md | 1 +
Sources/LumeEngine/LumePlayer.swift | 5 +
.../LumeEngineCore/Decode/VideoDecoder.swift | 5 +
Sources/LumeEngineCore/Demux/TrackInfo.swift | 28 ++-
.../Session/PlayerSession+Tracks.swift | 34 +++
.../Session/PlayerSession.swift | 12 +-
.../Subtitle/ClosedCaptions.swift | 130 ++++++++++
.../LumeEngineCore/Subtitle/SubtitleCue.swift | 71 ++----
.../Subtitle/SubtitleDecoder.swift | 96 +++++++-
.../Subtitle/SubtitleTimeline.swift | 226 ++++++++++++++++++
.../ClosedCaptionTests.swift | 222 +++++++++++++++++
.../Fixtures/generate-fixtures.sh | 87 +++++++
.../SubtitleTimelineTests.swift | 216 +++++++++++++++++
14 files changed, 1074 insertions(+), 63 deletions(-)
create mode 100644 Sources/LumeEngineCore/Subtitle/ClosedCaptions.swift
create mode 100644 Sources/LumeEngineCore/Subtitle/SubtitleTimeline.swift
create mode 100644 Tests/LumeEngineCoreTests/ClosedCaptionTests.swift
create mode 100644 Tests/LumeEngineCoreTests/SubtitleTimelineTests.swift
diff --git a/AGENTS.md b/AGENTS.md
index 38f275d..ccece3b 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -51,7 +51,7 @@ Prerequisites:
build/scripts/build-ffmpeg.sh macos-arm64 # one slice is enough for local dev
build/scripts/make-xcframework.sh
```
-- Tests need a host `ffmpeg` CLI (Homebrew) — fixtures are generated lazily on first test run by `Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh` into `TestStreams/generated/` (gitignored, idempotent). `Fixtures.swift` finds Homebrew ffmpeg automatically; override with the `FFMPEG` env var.
+- Tests need a host `ffmpeg` CLI (Homebrew) — fixtures are generated lazily on first test run by `Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh` into `TestStreams/generated/` (gitignored, idempotent). `Fixtures.swift` finds Homebrew ffmpeg automatically; override with the `FFMPEG` env var. The closed-caption fixture (`captions.ts`) also needs `python3` (the one Xcode's command-line tools provide is enough), because no ffmpeg CLI option writes captions into the video; without it that fixture is skipped and its tests fail as missing.
CI (`.github/workflows/ci.yml`) builds the macOS FFmpeg slice (cached on `build/versions.json` + scripts + patches hashes) and runs `swift test` with a strict one-thread cooperative pool (see above), checks the Markdown docs offline (`.github/scripts/check-docs.py`: relative links and in-repo anchors resolve, `CLAUDE.md` is still the `AGENTS.md` symlink — run it after editing docs), then cross-compiles the library for iOS and tvOS via `xcodebuild` (compile-only, one cached slice each — visionOS is left to local verification because that matrix entry was never confirmed on a runner — the xrOS SDK ships with Xcode, but a visionOS *destination* also needs Xcode's visionOS platform component); tagged releases build all 10 platform slices and attach the xcframework.
@@ -88,6 +88,8 @@ This split is the core invariant of the codebase:
- **Display criteria describe the output, not the container.** `PlayerSession.displayCriteria` (pure resolution in `Media/ContentDisplayCriteria`) is what an Apple TV with Match Content should switch the TV to: the refresh rate *after* deinterlacing (field rate turns 1080i50 into 50 Hz) and the dynamic range of what is presented (IPT Dolby Vision profiles are converted, so HDR10; every other Dolby Vision profile is presented as its base layer, so HDR10, HLG or SDR by its compatibility ID and the frames' transfer function). Never ask for Dolby Vision mode while the engine has no Dolby Vision output path — that belongs to the PLAN.md §7 compressed-`dvh1` spike. Before the first frame it is predicted from the stream; once the renderer has been handed frames, their colour tags and measured cadence (`Render/FrameCadence`) decide. Rates snap to the film/broadcast grid or ask for none — never an odd rate. Applying it is tvOS-only and opt-in (`Media/DisplayCriteriaMatcher`, or `LumePlayerView(…, matchesDisplayCriteria: true)`): one matcher per player screen, kept across sessions so zapping between same-mode channels never switches the TV, reset only when the screen goes away. The mode-switch blank is sat out with `PlayerSession.holdStart(for:)`, which the session bounds itself and the stall watchdog ignores — keep both, or a host that never releases wedges playback, or a held start reads as a stall.
- **Dolby Vision with an IPT base layer is converted, not tagged.** Profiles 5, 10.0 and 20 (compatibility ID 0) carry IPTPQc2, which shows pink/green as YCbCr. No colour attachment fixes that — the RPU reshaping changes per scene and Apple's decoders don't apply it to decoded frames — so `Decode/DolbyVisionConverter` turns each frame into HDR10 on the GPU in `VideoDecoder.deliver`, the one point every frame (hardware, software, deinterlaced) passes with its side data. Detection is the stream's DV configuration record *or* an `AVCOL_SPC_IPT_C2` frame; many P5 encodes leave the VUI matrix unspecified, so don't reduce it to the tag. HDR10-compatible profiles (8.1 etc.) must stay on the zero-copy path. GPU refusals are per-frame (iOS refuses a backgrounded app); only a converter that can't exist gives up for the session.
- **Audio is fitted to the negotiated route width, never the route's capability.** `AudioDecoder` reads `AudioOutputRoute.outputChannels` (the session's `outputNumberOfChannels`, followed through route-change notifications) per frame and remixes per `AudioLayoutPolicy` with an explicit `DownmixMatrix` — a fold-down mixes centre and surrounds in, it never drops channels. Don't switch it to `maximumOutputNumberOfChannels`: a renderer fed more channels than the session outputs failed outright on tvOS. The engine only observes the audio session; setting its category, activation or preferred channel count is the host's job (Lume does it). When the route narrows, `SystemRenderer` keeps audio decoded for the wider route out of the renderer.
+- **How long a subtitle shows is the timeline's decision, not the decoder's.** Decoders hand `Subtitle/SubtitleStore` a `SubtitleEvent` (start, end *or none*, text, whether the track replaces the whole screen with each update); `Subtitle/SubtitleTimeline` decides the rest: an open cue ends at the next update of its track (capped at 16 s), an empty update on a closed-caption/DVB/PGS/DVD track is a clear that ends what is showing, a seek ends open cues, re-read ranges are deduplicated, and the store is bounded. Don't turn `end_display_time == UINT32_MAX` into a number in a decoder or drop empty updates there — both put stuck or stacked cues on screen.
+- **Closed captions leave the video lane through a tap, never a wait.** CEA-608 in A/53 side data (`AV_FRAME_DATA_A53_CC`) is lifted off each decoded frame in `VideoDecoder.receiveFrames` (before deinterlacing) by `Subtitle/ClosedCaptionTap`, stamped with the frame's PTS, and `trySend`-ed to an ordinary `SubtitleDecoder` running `ccaption` in real-time mode — a slow caption lane drops captions, never frames. The track is synthetic (`TrackInfo.embeddedClosedCaptionsIndex`) and appears in `PlayerSession.mediaInfo` once caption content is seen; `.opened` keeps the open-time track list, because no new `PlayerEvent` case may be added for it. Only CEA-608 field 1 (CC1/CC2) is decoded: FFmpeg has no CEA-708 (DTVCC) decoder, and teletext needs libzvbi, which the build does not include.
- **Zero global mutable state:** all configuration is `PlayerConfiguration` passed at session creation. No `static var` knobs.
- **No force unwraps / `as!` / `fatalError` in engine paths**; failures return typed `EngineError`. No `assertionFailure` in production channel/data paths — misuse degrades, never crashes.
- **Never scrape FFmpeg log strings for control flow** (`av_log` is diagnostics only). Silence is never a failure mode: pipelines are supervised, stalls are detected from playhead ground truth, and failures surface as typed `PlayerEvent`s on the session's `AsyncStream`.
diff --git a/PLAN.md b/PLAN.md
index bb4f4d7..e8a5ccc 100644
--- a/PLAN.md
+++ b/PLAN.md
@@ -142,6 +142,7 @@ Brightness/contrast/saturation and MetalFX spatial upscaling operate CVPixelBuff
**D6 — One subtitle engine, composited as overlay.**
libass renders ASS/SSA (full effects, embedded fonts via attachment streams, word-by-word \k tags); PGS/DVB/VobSub decoded by FFmpeg to bitmaps; SRT/VTT parsed to styled cues rendered through the same layout box. Output = time-tagged `SubtitleImage` cues composited on a CALayer above the video surface (and into the PiP content when active; into Metal for HDR subtitle mode). Dual subtitle slots (main + secondary). External files attach as additional demux sessions.
+**Cue lifecycle and closed captions (as built).** Every subtitle source — demuxed stream, sidecar file, or captions lifted from the video — reaches one time-indexed store as display updates; the store's `SubtitleTimeline` alone decides how long each shows (open cues end at the next update or a clear, capped; seeks end open cues; bounded size). CEA-608 captions carried as A/53 side data are tapped off decoded video frames without ever blocking the video lane and surface as a synthetic "CC1" subtitle track once seen, additively through `PlayerSession.mediaInfo` (no new event).
**System caption respect:** map `MACaptionAppearance` (user's Style settings) onto text-cue styling by default.
**D7 — Buffering is PTS-math, not heuristics.**
diff --git a/Sources/LumeEngine/LumePlayer.swift b/Sources/LumeEngine/LumePlayer.swift
index 34d0f40..d88ebb1 100644
--- a/Sources/LumeEngine/LumePlayer.swift
+++ b/Sources/LumeEngine/LumePlayer.swift
@@ -201,6 +201,11 @@ public final class LumePlayer {
private func tick() async {
guard let session else { return }
position = await session.position
+ // Tracks can turn up after open (closed captions in the video).
+ if let mediaInfo, let current = await session.mediaInfo,
+ current.tracks.count != mediaInfo.tracks.count {
+ self.mediaInfo = current
+ }
let now = session.renderer.currentTime
if MediaTime.isValid(now) {
diff --git a/Sources/LumeEngineCore/Decode/VideoDecoder.swift b/Sources/LumeEngineCore/Decode/VideoDecoder.swift
index 9dbc9d5..5cd00a8 100644
--- a/Sources/LumeEngineCore/Decode/VideoDecoder.swift
+++ b/Sources/LumeEngineCore/Decode/VideoDecoder.swift
@@ -140,6 +140,10 @@ public final class VideoDecoder: @unchecked Sendable {
private let maxConsecutiveErrors = 100
+ /// Closed captions riding in the video (A/53 side data) leave here, ahead
+ /// of deinterlacing and conversion, for the session's caption lane.
+ let captionTap = ClosedCaptionTap()
+
public init(
parameters: CodecParameters,
input: Channel,
@@ -365,6 +369,7 @@ public final class VideoDecoder: @unchecked Sendable {
consecutiveErrors = 0
if usingHardware { hardwareFailures.recordFrame() }
noteFrameSource(frame)
+ captionTap.offer(frame, serial: currentSerial ?? 0)
emit(frame: frame)
av_frame_unref(frame)
}
diff --git a/Sources/LumeEngineCore/Demux/TrackInfo.swift b/Sources/LumeEngineCore/Demux/TrackInfo.swift
index d469be0..0fdf75b 100644
--- a/Sources/LumeEngineCore/Demux/TrackInfo.swift
+++ b/Sources/LumeEngineCore/Demux/TrackInfo.swift
@@ -111,6 +111,21 @@ public struct DolbyVisionConfiguration: Sendable, Equatable {
}
}
+extension TrackInfo {
+ /// Index of the closed-caption track carried inside the video stream
+ /// (CEA-608 in A/53 side data — how US broadcast and much IPTV ship
+ /// captions). It is no container stream, so it gets an index no container
+ /// reaches ("CC" in ASCII, channel 1); pass it to `selectSubtitleTrack`
+ /// like any other subtitle index.
+ public static let embeddedClosedCaptionsIndex: Int32 = 0x4343_0001
+
+ /// True for the synthetic closed-caption track (`embeddedClosedCaptionsIndex`),
+ /// which `PlayerSession.mediaInfo` lists once captions turn up in the video.
+ public var isEmbeddedClosedCaptions: Bool {
+ index == Self.embeddedClosedCaptionsIndex
+ }
+}
+
/// A chapter marker, in engine microseconds.
public struct ChapterInfo: Sendable, Equatable {
public let title: String?
@@ -135,7 +150,9 @@ public struct MediaInfo: Sendable {
public let isSeekable: Bool
public let bitrate: Int64
public let metadata: [String: String]
- public let tracks: [TrackInfo]
+ /// Streams found at open, plus tracks discovered while playing (the
+ /// closed-caption track — see `PlayerSession.mediaInfo`).
+ public internal(set) var tracks: [TrackInfo]
public let chapters: [ChapterInfo]
public var videoTracks: [TrackInfo] { tracks.filter { $0.kind == .video } }
@@ -144,4 +161,13 @@ public struct MediaInfo: Sendable {
/// Live heuristic: no known duration.
public var isLive: Bool { duration == nil }
+
+ /// This info with `track` appended, unless a track with its index is
+ /// already listed.
+ func adding(_ track: TrackInfo) -> MediaInfo {
+ guard !tracks.contains(where: { $0.index == track.index }) else { return self }
+ var copy = self
+ copy.tracks.append(track)
+ return copy
+ }
}
diff --git a/Sources/LumeEngineCore/Session/PlayerSession+Tracks.swift b/Sources/LumeEngineCore/Session/PlayerSession+Tracks.swift
index 8ea401f..7961e93 100644
--- a/Sources/LumeEngineCore/Session/PlayerSession+Tracks.swift
+++ b/Sources/LumeEngineCore/Session/PlayerSession+Tracks.swift
@@ -52,8 +52,15 @@ extension PlayerSession {
}
/// Selects an embedded subtitle track, or `nil` to disable subtitles.
+ /// `TrackInfo.embeddedClosedCaptionsIndex` selects the captions carried in
+ /// the video stream — also before they have turned up in `mediaInfo`, so
+ /// a host can keep captions on across channels.
public func selectSubtitleTrack(_ index: Int32?) {
teardownSubtitleLane()
+ if index == TrackInfo.embeddedClosedCaptionsIndex {
+ attachClosedCaptionLane()
+ return
+ }
guard let index,
let demuxer = activeDemuxer,
let parameters = demuxer.codecParameters(forStream: index)
@@ -83,6 +90,33 @@ extension PlayerSession {
}
}
+ /// Feeds the captions the video decoder finds (see `ClosedCaptionTap`)
+ /// to a caption decoder writing into the cue store.
+ ///
+ /// No backfill seek, unlike a subtitle stream: the captions ride in video
+ /// frames that are decoded a fraction of a second ahead of the playhead,
+ /// so nothing worth recovering has been read past — and live sources,
+ /// where captions mostly are, do not survive a seek.
+ private func attachClosedCaptionLane() {
+ guard let video = activeVideoDecoder,
+ let parameters = CodecParameters.closedCaptions()
+ else { return }
+ // One payload per frame: a few seconds of slack for a decoder that
+ // falls behind, beyond which captions are dropped, never frames.
+ let packets = Channel(capacity: 256)
+ let decoder = SubtitleDecoder(
+ parameters: parameters,
+ input: packets,
+ store: subtitles,
+ // CC1/CC2. FFmpeg would otherwise lock onto whichever field's
+ // data comes first, which can be field 2's metadata.
+ codecOptions: ["data_field": "first"]
+ )
+ installSubtitleLane(trackIndex: TrackInfo.embeddedClosedCaptionsIndex, packets: packets, decoder: decoder)
+ decoder.start()
+ video.captionTap.attach(packets)
+ }
+
/// Loads a sidecar subtitle file (SRT/VTT/ASS/…) into the session's cue
/// store, replacing any embedded selection. Resolves once the file is
/// fully parsed.
diff --git a/Sources/LumeEngineCore/Session/PlayerSession.swift b/Sources/LumeEngineCore/Session/PlayerSession.swift
index d115d7d..7c688c5 100644
--- a/Sources/LumeEngineCore/Session/PlayerSession.swift
+++ b/Sources/LumeEngineCore/Session/PlayerSession.swift
@@ -395,7 +395,15 @@ public actor PlayerSession {
info?.duration.map(MediaTime.seconds)
}
- public var mediaInfo: MediaInfo? { info }
+ /// What `open` returned, plus tracks found since: the closed-caption track
+ /// (`TrackInfo.embeddedClosedCaptionsIndex`) appears once captions turn up
+ /// in the video, which no open-time probe can tell. Re-read it to offer
+ /// that track; `.opened` carries the open-time value only.
+ public var mediaInfo: MediaInfo? {
+ guard let info else { return nil }
+ guard videoDecoder?.captionTap.hasDetectedCaptions == true else { return info }
+ return info.adding(.embeddedClosedCaptions)
+ }
/// How far startup has got and how long each step took.
public var startup: StartupTimeline {
@@ -579,6 +587,7 @@ public actor PlayerSession {
demuxAtEOF = false
videoAtEOF = false
audioAtEOF = false
+ subtitles.closeOpenCues(at: target)
demuxer.seek(to: target)
// Completion continues in handleDemux(.didSeek).
}
@@ -727,6 +736,7 @@ public actor PlayerSession {
if let old = selectedSubtitleTrackIndex {
demuxer?.detach(streamIndex: old)
}
+ videoDecoder?.captionTap.attach(nil)
if let retired = subtitleDecoder {
retired.requestStop() // no cue of the old track lands after `removeAll` below
retire { retired.shutdown(deadline: 2) }
diff --git a/Sources/LumeEngineCore/Subtitle/ClosedCaptions.swift b/Sources/LumeEngineCore/Subtitle/ClosedCaptions.swift
new file mode 100644
index 0000000..d35af1c
--- /dev/null
+++ b/Sources/LumeEngineCore/Subtitle/ClosedCaptions.swift
@@ -0,0 +1,130 @@
+internal import CFFmpeg
+import Foundation
+import Synchronization
+
+/// Closed captions carried inside the video stream.
+///
+/// US broadcast (ATSC) and much IPTV ship captions not as a subtitle stream
+/// but as A/53 user data in the video itself — SEI messages in H.264/HEVC,
+/// picture user data in MPEG-2 — which FFmpeg's video decoders hand out as
+/// `AV_FRAME_DATA_A53_CC` side data on the decoded frame: a run of 3-byte
+/// `cc_data` triples. The tap lifts those payloads off the video lane and
+/// feeds them, as packets stamped with the frame's PTS, to an ordinary
+/// `SubtitleDecoder` running FFmpeg's `ccaption` decoder. That keeps captions
+/// on the video's own timestamps (engine µs, display order), and keeps
+/// caption decoding — and its failures — off the video thread.
+///
+/// The video thread never waits on it: payloads go out with `trySend`, and a
+/// caption lane that falls behind loses captions, never frames.
+final class ClosedCaptionTap: Sendable {
+ private struct State {
+ var detected = false
+ var sink: Channel?
+ }
+
+ private let state = Mutex(State())
+
+ /// True once the video carried caption data with content (null padding
+ /// alone does not count — broadcast encoders send it on every frame).
+ var hasDetectedCaptions: Bool {
+ state.withLock { $0.detected }
+ }
+
+ /// Starts (or with `nil` stops) forwarding caption payloads.
+ func attach(_ sink: Channel?) {
+ state.withLock { $0.sink = sink }
+ }
+
+ /// Video decode thread: called with every decoded frame, before any
+ /// filtering. A frame without caption data costs one side-data lookup.
+ func offer(_ frame: UnsafeMutablePointer, serial: UInt64) {
+ guard let sideData = av_frame_get_side_data(frame, AV_FRAME_DATA_A53_CC),
+ let data = sideData.pointee.data,
+ sideData.pointee.size > 0
+ else { return }
+ let payload = UnsafeRawBufferPointer(start: data, count: Int(sideData.pointee.size))
+
+ let (detected, sink) = state.withLock { ($0.detected, $0.sink) }
+ if !detected, ClosedCaptionData.carriesCaptions(payload) {
+ state.withLock { $0.detected = true }
+ }
+
+ let pts = frame.pointee.best_effort_timestamp
+ guard let sink, MediaTime.isValid(pts),
+ let packet = Packet(copying: payload, pts: pts, serial: serial)
+ else { return }
+ _ = sink.trySend(packet)
+ }
+}
+
+/// Reading A/53 `cc_data` (CEA-708 §4.4: `cc_valid`, `cc_type`, two data
+/// bytes per triple).
+enum ClosedCaptionData {
+ /// Whether the payload holds caption data for CC1/CC2 (CEA-608 field 1)
+ /// beyond null padding — what decides that a stream has captions to
+ /// offer. Field 2 (CC3/CC4, mostly XDS metadata) and DTVCC (708) packets
+ /// do not count: the caption decoder shows field 1 only.
+ static func carriesCaptions(_ payload: UnsafeRawBufferPointer) -> Bool {
+ var offset = 0
+ while offset + 3 <= payload.count {
+ let header = payload[offset]
+ let valid = header & 0x04 != 0
+ let fieldOne = header & 0x03 == 0
+ if valid, fieldOne, payload[offset + 1] & 0x7F != 0 || payload[offset + 2] & 0x7F != 0 {
+ return true
+ }
+ offset += 3
+ }
+ return false
+ }
+}
+
+extension TrackInfo {
+ /// The synthetic track for captions found in the video stream.
+ static let embeddedClosedCaptions = TrackInfo(
+ index: embeddedClosedCaptionsIndex,
+ kind: .subtitle,
+ codecName: String(cString: avcodec_get_name(AV_CODEC_ID_EIA_608)),
+ codecID: AV_CODEC_ID_EIA_608.rawValue,
+ language: nil,
+ title: "CC1",
+ isDefault: false,
+ isForced: false,
+ bitrate: 0,
+ video: nil,
+ audio: nil,
+ wrapBits: 64
+ )
+}
+
+extension CodecParameters {
+ /// Parameters for the caption lane: no container stream describes it.
+ static func closedCaptions() -> CodecParameters? {
+ guard let scratch = avcodec_parameters_alloc() else { return nil }
+ defer {
+ var pointer: UnsafeMutablePointer? = scratch
+ avcodec_parameters_free(&pointer)
+ }
+ scratch.pointee.codec_type = AVMEDIA_TYPE_SUBTITLE
+ scratch.pointee.codec_id = AV_CODEC_ID_EIA_608
+ return CodecParameters(copying: scratch)
+ }
+}
+
+extension Packet {
+ /// A packet owning a copy of `bytes`, on the engine timeline already.
+ convenience init?(copying bytes: UnsafeRawBufferPointer, pts: Int64, serial: UInt64) {
+ guard let base = bytes.baseAddress, !bytes.isEmpty,
+ let raw = av_packet_alloc()
+ else { return nil }
+ guard av_new_packet(raw, Int32(bytes.count)) >= 0, let destination = raw.pointee.data else {
+ var pointer: UnsafeMutablePointer? = raw
+ av_packet_free(&pointer)
+ return nil
+ }
+ destination.update(from: base.assumingMemoryBound(to: UInt8.self), count: bytes.count)
+ raw.pointee.pts = pts
+ raw.pointee.dts = pts
+ self.init(adopting: raw, pts: pts, dts: pts, duration: 0, serial: serial)
+ }
+}
diff --git a/Sources/LumeEngineCore/Subtitle/SubtitleCue.swift b/Sources/LumeEngineCore/Subtitle/SubtitleCue.swift
index a262a93..9e4f274 100644
--- a/Sources/LumeEngineCore/Subtitle/SubtitleCue.swift
+++ b/Sources/LumeEngineCore/Subtitle/SubtitleCue.swift
@@ -1,4 +1,5 @@
import Foundation
+import Synchronization
/// One subtitle cue on the engine timeline. Text-only for now; bitmap cues
/// (PGS/DVB) and full libass styling land with the subtitle pack (PLAN.md P5+).
@@ -19,75 +20,47 @@ public struct SubtitleCue: Sendable, Equatable, Identifiable {
}
/// Time-indexed cue container, safe for concurrent insert (decoder thread) and
-/// query (UI tick). Cues are kept sorted by start time.
+/// query (UI tick). The lifecycle policy — how long a cue shows, what ends it,
+/// how many are kept — is `SubtitleTimeline`'s.
///
/// Cues outlive seeks on purpose — the store is a timeline, not a queue — so
/// the same packet reaches it again whenever the demuxer re-reads a range it
/// already delivered: a backward seek, or the seek that backfills a subtitle
/// track selected after the demuxer read past its packets. An identical cue
-/// (same start, end, and text) is therefore dropped, or every re-read would
-/// show its lines doubled.
-public final class SubtitleStore: @unchecked Sendable {
- private let lock = NSLock()
- private var cues: [SubtitleCue] = []
- private var nextID: UInt64 = 0
+/// is therefore dropped, or every re-read would show its lines doubled.
+public final class SubtitleStore: Sendable {
+ private let timeline = Mutex(SubtitleTimeline())
public init() {}
+ /// A text cue with a known end (tests, and the shape every text format
+ /// decodes to).
func insert(start: Int64, end: Int64, text: String) {
let trimmed = text.trimmingCharacters(in: .whitespacesAndNewlines)
guard !trimmed.isEmpty, end > start else { return }
- lock.lock()
- defer { lock.unlock() }
- // Insertion point by start time (cues arrive nearly sorted).
- var index = cues.count
- while index > 0 && cues[index - 1].start > start { index -= 1 }
- // Cues with the same start sit directly before the insertion point.
- var existing = index - 1
- while existing >= 0 && cues[existing].start == start {
- if cues[existing].end == end && cues[existing].text == trimmed { return }
- existing -= 1
- }
- cues.insert(SubtitleCue(id: nextID, start: start, end: end, text: trimmed), at: index)
- nextID += 1
+ apply(SubtitleEvent(start: start, end: end, text: trimmed, replacesPrevious: false))
+ }
+
+ func apply(_ event: SubtitleEvent) {
+ timeline.withLock { $0.apply(event) }
+ }
+
+ /// Ends cues still open at `time`: the session seeked there, and what the
+ /// source shows after it is unknown until its next update.
+ func closeOpenCues(at time: Int64) {
+ timeline.withLock { $0.closeOpenCues(at: time) }
}
/// All cues covering `time` (engine µs).
public func activeCues(at time: Int64) -> [SubtitleCue] {
- lock.lock()
- defer { lock.unlock() }
- // Binary search for the first cue starting after `time`, then walk back.
- // Overlapping cues are rare and short-lived; a bounded backward scan is
- // simpler than an interval tree and plenty fast.
- var low = 0
- var high = cues.count
- while low < high {
- let mid = (low + high) / 2
- if cues[mid].start <= time { low = mid + 1 } else { high = mid }
- }
- var active: [SubtitleCue] = []
- var index = low - 1
- var scanned = 0
- while index >= 0 && scanned < 64 {
- let cue = cues[index]
- if cue.start <= time && time < cue.end {
- active.append(cue)
- }
- index -= 1
- scanned += 1
- }
- return active.sorted { $0.start < $1.start }
+ timeline.withLock { $0.activeCues(at: time) }
}
public var count: Int {
- lock.lock()
- defer { lock.unlock() }
- return cues.count
+ timeline.withLock { $0.count }
}
public func removeAll() {
- lock.lock()
- defer { lock.unlock() }
- cues.removeAll()
+ timeline.withLock { $0.removeAll() }
}
}
diff --git a/Sources/LumeEngineCore/Subtitle/SubtitleDecoder.swift b/Sources/LumeEngineCore/Subtitle/SubtitleDecoder.swift
index c8ab793..4878e26 100644
--- a/Sources/LumeEngineCore/Subtitle/SubtitleDecoder.swift
+++ b/Sources/LumeEngineCore/Subtitle/SubtitleDecoder.swift
@@ -3,13 +3,24 @@ import Foundation
/// Text subtitle decode stage: `Channel` in, `SubtitleStore` out.
/// Runs `avcodec_decode_subtitle2`, so every FFmpeg text subtitle codec
-/// (subrip, ass, webvtt, mov_text, eia_608 CC) feeds one cue model.
+/// (subrip, ass, webvtt, mov_text, eia_608 CC) feeds one cue model — and
+/// so does the closed-caption lane, whose packets come from the video decoder
+/// (`ClosedCaptionTap`) rather than the demuxer. Each decoded update goes to
+/// the store as a `SubtitleEvent`; how long it shows is the store's policy.
/// Bitmap codecs (PGS/DVB/VobSub) are recognized but skipped for now.
public final class SubtitleDecoder: @unchecked Sendable {
public let store: SubtitleStore
private let parameters: CodecParameters
private let input: Channel
+ private let codecOptions: [String: String]
+ /// Each update is the whole screen (closed captions, bitmap formats) —
+ /// see `SubtitleEvent.replacesPrevious`.
+ private let replacesPrevious: Bool
+ /// CEA-608 captions run in FFmpeg's real-time mode (below), which stamps
+ /// an update with the time of the *previous* one; the packet that caused
+ /// it carries the time it actually shows.
+ private let isClosedCaptions: Bool
private let lock = NSCondition()
private var started = false
@@ -17,11 +28,39 @@ public final class SubtitleDecoder: @unchecked Sendable {
private var stopRequested = false
private var codecContext: UnsafeMutablePointer?
+ /// Decode-thread-only: seek generation of the last packet.
+ private var currentSerial: UInt64?
- public init(parameters: CodecParameters, input: Channel, store: SubtitleStore = SubtitleStore()) {
+ public convenience init(parameters: CodecParameters, input: Channel, store: SubtitleStore = SubtitleStore()) {
+ self.init(parameters: parameters, input: input, store: store, codecOptions: [:])
+ }
+
+ /// `codecOptions` go to `avcodec_open2` (the caption lane picks its field).
+ init(parameters: CodecParameters, input: Channel, store: SubtitleStore, codecOptions: [String: String]) {
self.parameters = parameters
self.input = input
self.store = store
+ let codecID = parameters.raw.pointee.codec_id
+ isClosedCaptions = codecID == AV_CODEC_ID_EIA_608
+ replacesPrevious = Self.replacesPrevious(codecID)
+ var options = codecOptions
+ if isClosedCaptions, options["real_time"] == nil {
+ // Real-time mode emits each screen change as it is decoded, open
+ // ended, and a clear as an empty update. The default buffered mode
+ // emits a caption only once the next change has ended it — after
+ // its own duration, by which time playback (a fraction of a second
+ // behind decode) has already passed most of it.
+ options["real_time"] = "1"
+ }
+ self.codecOptions = options
+ }
+
+ /// Formats where every update describes the whole screen and stays until
+ /// the next one: closed captions, teletext, and the bitmap formats.
+ static func replacesPrevious(_ codecID: AVCodecID) -> Bool {
+ if codecID == AV_CODEC_ID_EIA_608 || codecID == AV_CODEC_ID_DVB_TELETEXT { return true }
+ guard let descriptor = avcodec_descriptor_get(codecID) else { return false }
+ return descriptor.pointee.props & AV_CODEC_PROP_BITMAP_SUB != 0
}
public func start() {
@@ -122,19 +161,35 @@ public final class SubtitleDecoder: @unchecked Sendable {
private func decode(packet: Packet) {
guard let context = codecContext else { return }
+ // Seek boundary: decoders with screen state (caption roll-up rows,
+ // PGS/DVB object and palette sets) must not carry it across.
+ if let serial = currentSerial, serial != packet.serial {
+ avcodec_flush_buffers(context)
+ }
+ currentSerial = packet.serial
+
var subtitle = AVSubtitle()
var gotSubtitle: Int32 = 0
let result = avcodec_decode_subtitle2(context, &subtitle, &gotSubtitle, packet.raw)
guard result >= 0, gotSubtitle != 0 else { return }
defer { avsubtitle_free(&subtitle) }
- // Packet timestamps are engine µs; display offsets are milliseconds.
- let basePTS = MediaTime.isValid(packet.pts) ? packet.pts : 0
- let start = basePTS + Int64(subtitle.start_display_time) * 1_000
- var end = basePTS + Int64(subtitle.end_display_time) * 1_000
- if subtitle.end_display_time == 0 || end <= start {
- end = packet.duration > 0 ? basePTS + packet.duration : start + 3_000_000
+ // Packet timestamps are engine µs, and so is `subtitle.pts`
+ // (`pkt_timebase` is µs; PGS takes it from its presentation segment);
+ // display offsets are milliseconds.
+ let base: Int64
+ if !isClosedCaptions, MediaTime.isValid(subtitle.pts) {
+ base = subtitle.pts
+ } else {
+ base = MediaTime.isValid(packet.pts) ? packet.pts : 0
}
+ let span = SubtitleEvent.span(
+ base: base,
+ startDisplay: isClosedCaptions ? 0 : subtitle.start_display_time,
+ endDisplay: subtitle.end_display_time,
+ packetDuration: packet.duration,
+ replacesPrevious: replacesPrevious
+ )
var lines: [String] = []
for i in 0..= 0 else {
+ var options: OpaquePointer?
+ for (key, value) in codecOptions {
+ av_dict_set(&options, key, value, 0)
+ }
+ defer { av_dict_free(&options) }
+ guard avcodec_open2(context, codec, &options) >= 0 else {
var pointer: UnsafeMutablePointer? = context
avcodec_free_context(&pointer)
return false
diff --git a/Sources/LumeEngineCore/Subtitle/SubtitleTimeline.swift b/Sources/LumeEngineCore/Subtitle/SubtitleTimeline.swift
new file mode 100644
index 0000000..d3f8381
--- /dev/null
+++ b/Sources/LumeEngineCore/Subtitle/SubtitleTimeline.swift
@@ -0,0 +1,226 @@
+import Foundation
+
+/// One display update from a subtitle decoder on the engine timeline (µs),
+/// before the timeline decides how long it stays up.
+struct SubtitleEvent: Sendable, Equatable {
+ var start: Int64
+ /// `nil` when the source gives no end. DVB, PGS and closed captions say
+ /// exactly that — "until the next update" — and a decoder that turned it
+ /// into a number would leave the text up for 49 days (`UINT32_MAX` ms).
+ var end: Int64?
+ /// Display text, trimmed. Empty on a replacing track means "clear the
+ /// screen", which is how those formats end what is showing.
+ var text: String
+ /// The track describes the whole screen with every update (closed
+ /// captions, DVB, PGS, DVD): an update ends everything shown before it.
+ /// Text formats (SRT, ASS, WebVTT, mov_text) overlap freely instead, each
+ /// cue carrying its own end.
+ var replacesPrevious: Bool
+
+ /// Where an `AVSubtitle` sits on the engine timeline. `base` is the
+ /// subtitle's PTS in engine µs, display times are milliseconds after it.
+ /// `nil` end means open: shows until replaced (see `SubtitleTimeline`).
+ static func span(
+ base: Int64,
+ startDisplay: UInt32,
+ endDisplay: UInt32,
+ packetDuration: Int64,
+ replacesPrevious: Bool
+ ) -> (start: Int64, end: Int64?) {
+ let start = base + Int64(startDisplay) * 1_000
+ // UINT32_MAX is FFmpeg's "no end" (PGS always, closed captions in
+ // real-time mode, DVB without a computed end).
+ guard endDisplay != .max else { return (start, nil) }
+ let end = base + Int64(endDisplay) * 1_000
+ if endDisplay != 0, end > start { return (start, end) }
+ // No usable end. A replacing track's update lasts until the next one;
+ // a text cue without one gets its packet's duration, or a reading-time
+ // guess, as it always has.
+ if replacesPrevious { return (start, nil) }
+ return (start, packetDuration > 0 ? base + packetDuration : start + 3_000_000)
+ }
+}
+
+/// The cue-lifecycle policy behind `SubtitleStore`: a timeline of cues sorted
+/// by start, deciding how long each one shows. Pure value type — the store
+/// adds the lock.
+///
+/// The failure modes it exists for, all of which used to reach the screen:
+/// - Cues without an end (PGS, closed captions, DVB "until the next page")
+/// stayed up for good, piling onto every later cue.
+/// - The empty update that DVB, PGS and closed captions send to clear the
+/// screen was dropped as "no text", so the previous cue stayed for its full
+/// timeout (DVB pages time out after tens of seconds).
+/// - Every cue a live session ever decoded was kept: a day of roll-up
+/// captions is hundreds of thousands of them.
+///
+/// The rules:
+/// - An open cue ends where the next update of its track starts, or after
+/// `openEndedLimit` when none comes.
+/// - On a replacing track every update — cue or clear — ends everything
+/// shown before it, and when one arrives out of order (a range re-read
+/// after a seek) the update already stored after it bounds it in turn.
+/// - Text tracks keep their overlaps; only their open cues are ended by the
+/// next cue.
+/// - An update already held is dropped, so re-reading a range (backward
+/// seek, backfill seek) does not double its lines. On a replacing track a
+/// different update at the same start replaces the stored one: the screen
+/// at one instant has one state, and the latest decode of it wins.
+/// - At most `capacity` cues are kept; the ones farthest from where cues are
+/// being added go first.
+struct SubtitleTimeline: Sendable {
+ /// How long an open cue stays without a successor. Caption decoders on
+ /// televisions erase a caption left standing by about this long without
+ /// caption data; a longer limit leaves stale text over the next scene
+ /// when a clear is lost, a shorter one cuts captions that are still
+ /// being spoken.
+ static let openEndedLimit: Int64 = 16_000_000
+
+ /// Cues kept. Far above any film's subtitle file (a few thousand cues);
+ /// for a live track it is half an hour or more of roll-up captions.
+ static let capacity = 8_192
+
+ /// How far back from an insertion point active cues are looked for.
+ /// Overlap is rare and short-lived; a replacing track has at most one.
+ private static let scanLimit = 64
+
+ private struct Entry: Sendable {
+ var cue: SubtitleCue
+ /// The end is the limit, not the source's: the next update or a seek
+ /// may still bound it.
+ var provisional: Bool
+ }
+
+ private var entries: [Entry] = []
+ private var nextID: UInt64 = 0
+
+ var count: Int { entries.count }
+
+ mutating func apply(_ event: SubtitleEvent) {
+ if event.text.isEmpty {
+ // A clear. Text tracks have no such thing; an empty text cue is
+ // just nothing to show.
+ guard event.replacesPrevious else { return }
+ end(at: event.start, onlyProvisional: false)
+ return
+ }
+
+ if isDuplicate(event) { return }
+ if event.replacesPrevious { removeEntries(startingAt: event.start) }
+ // What this update takes over from: everything on screen for a
+ // replacing track, only the cues with no end of their own otherwise.
+ end(at: event.start, onlyProvisional: !event.replacesPrevious)
+
+ let index = insertionIndex(for: event.start)
+ var end = event.end ?? event.start + Self.openEndedLimit
+ if (event.replacesPrevious || event.end == nil), index < entries.count {
+ // Arrived out of order: the next update already stored ends it.
+ end = min(end, entries[index].cue.start)
+ }
+ guard end > event.start else { return }
+
+ entries.insert(
+ Entry(
+ cue: SubtitleCue(id: nextID, start: event.start, end: end, text: event.text),
+ provisional: event.end == nil
+ ),
+ at: index
+ )
+ nextID += 1
+ trim(keeping: index)
+ }
+
+ /// Ends open cues at a discontinuity (a seek to `time`): the screen state
+ /// after it is unknown until the source sends its next update, and
+ /// showing the stale one meanwhile is worse than showing nothing. Cues
+ /// with an end from the source are facts about the timeline and stay.
+ mutating func closeOpenCues(at time: Int64) {
+ end(at: time, onlyProvisional: true)
+ }
+
+ mutating func removeAll() {
+ entries.removeAll()
+ }
+
+ /// All cues covering `time` (engine µs), ordered by start.
+ func activeCues(at time: Int64) -> [SubtitleCue] {
+ var active: [SubtitleCue] = []
+ var index = insertionIndex(for: time) - 1
+ var scanned = 0
+ while index >= 0 && scanned < Self.scanLimit {
+ let cue = entries[index].cue
+ if cue.start <= time && time < cue.end {
+ active.append(cue)
+ }
+ index -= 1
+ scanned += 1
+ }
+ return active.reversed()
+ }
+
+ // MARK: Internals
+
+ /// First entry starting after `time`.
+ private func insertionIndex(for time: Int64) -> Int {
+ var low = 0
+ var high = entries.count
+ while low < high {
+ let mid = (low + high) / 2
+ if entries[mid].cue.start <= time { low = mid + 1 } else { high = mid }
+ }
+ return low
+ }
+
+ private func isDuplicate(_ event: SubtitleEvent) -> Bool {
+ var index = insertionIndex(for: event.start) - 1
+ while index >= 0 && entries[index].cue.start == event.start {
+ let entry = entries[index]
+ if entry.cue.text == event.text {
+ // A stored end may have been bounded by a successor since, so
+ // it only has to match where both ends came from the source.
+ if event.replacesPrevious || event.end == nil || entry.provisional { return true }
+ if entry.cue.end == event.end { return true }
+ }
+ index -= 1
+ }
+ return false
+ }
+
+ private mutating func removeEntries(startingAt start: Int64) {
+ var index = insertionIndex(for: start) - 1
+ while index >= 0 && entries[index].cue.start == start {
+ entries.remove(at: index)
+ index -= 1
+ }
+ }
+
+ /// Ends the cues that started before `time` and still show at it.
+ private mutating func end(at time: Int64, onlyProvisional: Bool) {
+ var index = insertionIndex(for: time) - 1
+ var scanned = 0
+ while index >= 0 && scanned < Self.scanLimit {
+ let entry = entries[index]
+ if entry.cue.start < time, entry.cue.end > time, !onlyProvisional || entry.provisional {
+ entries[index].cue = SubtitleCue(
+ id: entry.cue.id, start: entry.cue.start, end: time, text: entry.cue.text
+ )
+ }
+ index -= 1
+ scanned += 1
+ }
+ }
+
+ /// Drops cues beyond `capacity` from the end farther from `index`, where
+ /// cues are being added — the front on a live timeline, the back after a
+ /// seek far into the past of a long file. Drops an eighth more than
+ /// needed so the shift is not paid on every insert.
+ private mutating func trim(keeping index: Int) {
+ guard entries.count > Self.capacity else { return }
+ let excess = entries.count - Self.capacity + Self.capacity / 8
+ if index < entries.count / 2 {
+ entries.removeLast(excess)
+ } else {
+ entries.removeFirst(excess)
+ }
+ }
+}
diff --git a/Tests/LumeEngineCoreTests/ClosedCaptionTests.swift b/Tests/LumeEngineCoreTests/ClosedCaptionTests.swift
new file mode 100644
index 0000000..35774a6
--- /dev/null
+++ b/Tests/LumeEngineCoreTests/ClosedCaptionTests.swift
@@ -0,0 +1,222 @@
+import CFFmpeg
+import Foundation
+import Testing
+@testable import LumeEngineCore
+
+/// CEA-608 caption data as a broadcast encoder writes it: one byte pair for
+/// CC1 per frame, null padding on every frame without one. Mirrors the
+/// caption script in `generate-fixtures.sh`.
+private enum CaptionScript {
+ static let frameDuration: Int64 = 1_000_000 / 30
+
+ static func oddParity(_ value: UInt8) -> UInt8 {
+ let bits = value & 0x7F
+ return bits.nonzeroBitCount % 2 == 0 ? bits | 0x80 : bits
+ }
+
+ /// Byte pairs per frame index for two pop-on captions (shown at frames
+ /// 30 and 120, erased at 90 and 180 — 1–3 s and 4–6 s at 30 fps).
+ static let pairs: [Int: (UInt8, UInt8)] = {
+ var script: [Int: (UInt8, UInt8)] = [:]
+ func popOn(load: Int, _ text: String, show: Int, clear: Int) {
+ var pairs: [(UInt8, UInt8)] = [(0x14, 0x20), (0x14, 0x70)] // load, row 15
+ let bytes = Array(text.utf8)
+ for index in stride(from: 0, to: bytes.count, by: 2) {
+ pairs.append((bytes[index], index + 1 < bytes.count ? bytes[index + 1] : 0))
+ }
+ for (offset, pair) in pairs.enumerated() { script[load + offset] = pair }
+ script[show] = (0x14, 0x2F) // end of caption
+ script[clear] = (0x14, 0x2C) // erase displayed memory
+ }
+ popOn(load: 5, "HELLO CAPTIONS", show: 30, clear: 90)
+ popOn(load: 100, "SECOND CAPTION", show: 120, clear: 180)
+ return script
+ }()
+
+ /// The A/53 `cc_data` triples of one frame: CC1 data, field-2 nulls.
+ static func ccData(frame: Int) -> [UInt8] {
+ let (first, second) = pairs[frame] ?? (0, 0)
+ return [0xFC, oddParity(first), oddParity(second), 0xFD, 0x80, 0x80]
+ }
+}
+
+@Suite("Closed captions")
+struct ClosedCaptionTests {
+ @Test("caption detection ignores null padding, field 2 and invalid triples")
+ func detection() {
+ func carries(_ bytes: [UInt8]) -> Bool {
+ bytes.withUnsafeBytes { ClosedCaptionData.carriesCaptions($0) }
+ }
+ #expect(!carries([0xFC, 0x80, 0x80, 0xFD, 0x80, 0x80]), "null padding on both fields")
+ #expect(!carries([0xFD, 0x14, 0x20]), "field 2 (CC3/XDS)")
+ #expect(!carries([0xF8, 0x14, 0x20]), "cc_valid not set")
+ #expect(!carries([0xFE, 0x41, 0x42]), "DTVCC packet data")
+ #expect(carries([0xFD, 0x80, 0x80, 0xFC, 0x94, 0x20]), "CC1 control code")
+ #expect(!carries([0xFC, 0x94]), "truncated triple")
+ }
+
+ /// The decoder runs FFmpeg's caption decoder in real-time mode, which
+ /// stamps an update with the time of the one before; the cue has to start
+ /// on the frame that put the caption on screen and end on the frame that
+ /// erased it, or captions drift from the speech they transcribe.
+ @Test("caption packets become cues on their frames' timestamps", .timeLimit(.minutes(1)))
+ func decodeCaptionPackets() throws {
+ let parameters = try #require(CodecParameters.closedCaptions())
+ let packets = Channel(capacity: 512)
+ let store = SubtitleStore()
+ let decoder = SubtitleDecoder(
+ parameters: parameters, input: packets, store: store, codecOptions: ["data_field": "first"]
+ )
+ decoder.start()
+
+ let origin: Int64 = 50_000_000
+ for frame in 0..<240 {
+ let bytes = CaptionScript.ccData(frame: frame)
+ let packet = try #require(bytes.withUnsafeBytes {
+ Packet(copying: $0, pts: origin + Int64(frame) * CaptionScript.frameDuration, serial: 0)
+ })
+ try packets.send(packet)
+ }
+ #expect(decoder.drainAndShutdown())
+
+ func time(frame: Int) -> Int64 { origin + Int64(frame) * CaptionScript.frameDuration }
+ #expect(store.count == 2)
+ let first = store.activeCues(at: time(frame: 30))
+ #expect(first.map(\.text) == ["HELLO CAPTIONS"])
+ #expect(first.first?.start == time(frame: 30))
+ #expect(first.first?.end == time(frame: 90))
+ #expect(store.activeCues(at: time(frame: 29)).isEmpty)
+ #expect(store.activeCues(at: time(frame: 90)).isEmpty)
+ #expect(store.activeCues(at: time(frame: 150)).map(\.text) == ["SECOND CAPTION"])
+ #expect(store.activeCues(at: time(frame: 200)).isEmpty)
+ }
+
+ @Test("the tap detects captions and forwards them stamped with the frame's PTS")
+ func tapForwardsSideData() throws {
+ let tap = ClosedCaptionTap()
+ var frame: UnsafeMutablePointer? = av_frame_alloc()
+ defer { av_frame_free(&frame) }
+ let raw = try #require(frame)
+
+ // No side data: nothing detected, nothing forwarded.
+ let sink = Channel(capacity: 4)
+ tap.attach(sink)
+ tap.offer(raw, serial: 3)
+ #expect(!tap.hasDetectedCaptions)
+ #expect(sink.tryReceive() == nil)
+
+ func attach(_ bytes: [UInt8]) throws {
+ av_frame_remove_side_data(raw, AV_FRAME_DATA_A53_CC)
+ let sideData = try #require(av_frame_new_side_data(raw, AV_FRAME_DATA_A53_CC, bytes.count))
+ sideData.pointee.data.update(from: bytes, count: bytes.count)
+ }
+
+ // Padding only: forwarded (the decoder needs the frame cadence), but
+ // no caption track to offer yet.
+ raw.pointee.best_effort_timestamp = 7_000_000
+ try attach(CaptionScript.ccData(frame: 0))
+ tap.offer(raw, serial: 3)
+ #expect(!tap.hasDetectedCaptions)
+ let padding = try #require(sink.tryReceive())
+ #expect(padding.pts == 7_000_000)
+ #expect(padding.serial == 3)
+ #expect(padding.byteCount == 6)
+
+ raw.pointee.best_effort_timestamp = 7_033_333
+ try attach(CaptionScript.ccData(frame: 5))
+ tap.offer(raw, serial: 3)
+ #expect(tap.hasDetectedCaptions)
+ #expect(sink.tryReceive()?.pts == 7_033_333)
+
+ // Detached: detection stays, forwarding stops.
+ tap.attach(nil)
+ tap.offer(raw, serial: 3)
+ #expect(tap.hasDetectedCaptions)
+ #expect(sink.tryReceive() == nil)
+ }
+}
+
+@Suite("Closed captions in playback", .serialized)
+struct ClosedCaptionPlaybackTests {
+ private func eventually(
+ timeout: TimeInterval = 10,
+ _ condition: () async -> Bool
+ ) async -> Bool {
+ let deadline = Date(timeIntervalSinceNow: timeout)
+ while Date() < deadline {
+ if await condition() { return true }
+ try? await Task.sleep(for: .milliseconds(100))
+ }
+ return false
+ }
+
+ private func makeSession(hardware: Bool) -> PlayerSession {
+ var configuration = PlayerConfiguration()
+ configuration.muted = true
+ configuration.bufferTarget = 0.3
+ configuration.hardwareDecode = hardware ? .videoToolbox : .software
+ return PlayerSession(configuration: configuration)
+ }
+
+ /// End to end: SEI in an H.264 TS → frame side data (VideoToolbox frames
+ /// carry it too: FFmpeg parses SEI itself) → caption lane → cues on the
+ /// video's timestamps. Seeks back to 0 after selecting so the first
+ /// caption is decoded again rather than raced against the decode lead.
+ @Test(
+ "captions in the video surface as a CC track and decode in sync",
+ .timeLimit(.minutes(1)),
+ arguments: [true, false]
+ )
+ func captionsFromVideo(hardware: Bool) async throws {
+ let session = makeSession(hardware: hardware)
+ let info = try await session.open(url: try Fixtures.path("captions.ts"))
+ let listedAtOpen = info.tracks.contains(where: \.isEmbeddedClosedCaptions)
+ #expect(!listedAtOpen, "unknown at open")
+
+ let offered = await eventually {
+ await session.mediaInfo?.subtitleTracks.contains(where: \.isEmbeddedClosedCaptions) == true
+ }
+ #expect(offered, "the CC track should appear once captions are decoded")
+
+ await session.selectSubtitleTrack(TrackInfo.embeddedClosedCaptionsIndex)
+ #expect(await session.selectedSubtitleTrackIndex == TrackInfo.embeddedClosedCaptionsIndex)
+ await session.seek(to: 0)
+ await session.play()
+
+ // Done once the second caption has been erased again. Not a cue
+ // count: what was decoded before the seek joined the first caption
+ // halfway through loading and may have left a fragment of its own.
+ let start = info.startTime
+ let frame: Int64 = 1_000_000 / 30
+ let sawBoth = await eventually(timeout: 30) {
+ let second = session.subtitles.activeCues(at: start + 5_000_000)
+ return second.map(\.text) == ["SECOND CAPTION"] && second.allSatisfy { $0.end <= start + 6_000_000 + frame }
+ }
+ #expect(sawBoth, "both captions should decode, store has \(session.subtitles.count)")
+
+ let first = session.subtitles.activeCues(at: start + 2_000_000)
+ #expect(first.map(\.text) == ["HELLO CAPTIONS"])
+ if let cue = first.first {
+ #expect(abs(cue.start - (start + 1_000_000)) <= frame, "shown at 1 s, got \(cue.start - start) µs")
+ #expect(abs(cue.end - (start + 3_000_000)) <= frame, "erased at 3 s, got \(cue.end - start) µs")
+ }
+ #expect(session.subtitles.activeCues(at: start + 3_500_000).isEmpty)
+ #expect(session.subtitles.activeCues(at: start + 5_000_000).map(\.text) == ["SECOND CAPTION"])
+
+ await session.selectSubtitleTrack(nil)
+ #expect(session.subtitles.count == 0)
+ await session.shutdown()
+ }
+
+ @Test("a source without captions offers no CC track", .timeLimit(.minutes(1)))
+ func noCaptionsNoTrack() async throws {
+ let session = makeSession(hardware: true)
+ _ = try await session.open(url: try Fixtures.path("basic.mp4"))
+ await session.play()
+ _ = await eventually { await session.position > 1 }
+ let tracks = await session.mediaInfo?.subtitleTracks ?? []
+ let listed = tracks.contains(where: \.isEmbeddedClosedCaptions)
+ #expect(!listed)
+ await session.shutdown()
+ }
+}
diff --git a/Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh b/Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh
index d5105e6..9edace4 100644
--- a/Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh
+++ b/Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh
@@ -19,6 +19,7 @@
# ac4.ts — TS whose first audio (ger) is declared AC-4, which no decoder here plays, then eng AAC
# hdr10.mp4 — HEVC Main 10, BT.2020 PQ, mastering display + content light level SEI, HDR10+ per frame
# surround51.ts — 20 s H.264 + AC-3 5.1 MPEG-TS, tone in the centre channel only
+# captions.ts — H.264 TS with CEA-608 captions in A/53 SEI (1–3 s, 4–6 s; needs python3)
set -euo pipefail
FFMPEG="${FFMPEG:-ffmpeg}"
@@ -341,4 +342,90 @@ gen surround51.ts \
-c:v libx264 -preset ultrafast -pix_fmt yuv420p -g 25 \
-c:a ac3 -b:a 384k -shortest -f mpegts
+# Closed captions the way US broadcast and much IPTV carry them: CEA-608
+# byte pairs inside the video, as A/53 user data in an SEI message on every
+# frame — not as a subtitle stream. No ffmpeg CLI option writes captions from
+# a caption file into the video, so a short script puts the SEI in front of
+# each picture of a B-frame-free encode, so the n-th picture in the file is
+# the n-th on screen. Two pop-on captions, each loaded off screen first:
+# "HELLO CAPTIONS" shown at 1 s and erased at 3 s, "SECOND CAPTION" at 4–6 s.
+if [ ! -f "$OUT/captions.ts" ]; then
+ if command -v python3 >/dev/null 2>&1; then
+ "$FFMPEG" -hide_banner -loglevel error -y \
+ -f lavfi -i "testsrc2=duration=8:size=320x180:rate=30" \
+ -c:v libx264 -preset ultrafast -pix_fmt yuv420p -bf 0 -g 30 \
+ -f h264 "$OUT/captions-source.h264"
+ python3 -I - "$OUT/captions-source.h264" "$OUT/captions.h264" <<'PY'
+import sys
+
+source, destination = sys.argv[1], sys.argv[2]
+stream = open(source, 'rb').read()
+
+def nal_units(data):
+ starts = []
+ index = 0
+ while True:
+ found = data.find(b'\x00\x00\x01', index)
+ if found < 0:
+ break
+ starts.append(found + 3)
+ index = found + 3
+ for position, start in enumerate(starts):
+ end = starts[position + 1] - 3 if position + 1 < len(starts) else len(data)
+ yield data[start:end].rstrip(b'\x00')
+
+def odd_parity(value):
+ value &= 0x7F
+ return value | (0x80 if bin(value).count('1') % 2 == 0 else 0)
+
+# frame -> CEA-608 byte pair for CC1 (field 1); every other frame carries nulls.
+script = {}
+def pop_on(load, text, show, clear):
+ pairs = [(0x14, 0x20), (0x14, 0x70)] # resume caption loading, row 15
+ for i in range(0, len(text), 2):
+ pairs.append((ord(text[i]), ord(text[i + 1]) if i + 1 < len(text) else 0))
+ for offset, pair in enumerate(pairs):
+ script[load + offset] = pair
+ script[show] = (0x14, 0x2F) # end of caption: swap it on screen
+ script[clear] = (0x14, 0x2C) # erase displayed memory
+
+pop_on(5, 'HELLO CAPTIONS', 30, 90)
+pop_on(100, 'SECOND CAPTION', 120, 180)
+
+def sei(frame):
+ first, second = script.get(frame, (0, 0))
+ cc_data = bytes([0xFC, odd_parity(first), odd_parity(second), 0xFD, 0x80, 0x80])
+ # ITU-T T.35: United States, ATSC, 'GA94', cc_data with two triples.
+ payload = bytes([0xB5, 0x00, 0x31]) + b'GA94' + bytes([0x03, 0x40 | 2, 0xFF]) + cc_data + bytes([0xFF])
+ rbsp = bytes([4, len(payload)]) + payload + bytes([0x80])
+ nal = bytearray([0x06])
+ zeros = 0
+ for byte in rbsp:
+ if zeros >= 2 and byte <= 3:
+ nal.append(3) # emulation prevention
+ zeros = 0
+ nal.append(byte)
+ zeros = zeros + 1 if byte == 0 else 0
+ return bytes(nal)
+
+out = bytearray()
+frame = 0
+for nal in nal_units(stream):
+ # First slice of a picture: a slice NAL whose first_mb_in_slice is 0.
+ if nal and nal[0] & 0x1F in (1, 5) and len(nal) > 1 and nal[1] & 0x80:
+ out += b'\x00\x00\x00\x01' + sei(frame)
+ frame += 1
+ out += b'\x00\x00\x00\x01' + nal
+open(destination, 'wb').write(out)
+PY
+ "$FFMPEG" -hide_banner -loglevel error -y \
+ -framerate 30 -i "$OUT/captions.h264" -c:v copy -f mpegts \
+ "$OUT/captions.ts"
+ rm -f "$OUT/captions-source.h264" "$OUT/captions.h264"
+ echo "generated: $OUT/captions.ts"
+ else
+ echo "skipped: $OUT/captions.ts (needs python3)" >&2
+ fi
+fi
+
echo "fixtures ready in $OUT"
diff --git a/Tests/LumeEngineCoreTests/SubtitleTimelineTests.swift b/Tests/LumeEngineCoreTests/SubtitleTimelineTests.swift
new file mode 100644
index 0000000..ee38af7
--- /dev/null
+++ b/Tests/LumeEngineCoreTests/SubtitleTimelineTests.swift
@@ -0,0 +1,216 @@
+import Foundation
+import Testing
+@testable import LumeEngineCore
+
+@Suite("Subtitle cue lifecycle")
+struct SubtitleTimelineTests {
+ private let second: Int64 = 1_000_000
+
+ private func screen(_ start: Int64, _ text: String, end: Int64? = nil) -> SubtitleEvent {
+ SubtitleEvent(start: start, end: end, text: text, replacesPrevious: true)
+ }
+
+ private func text(_ start: Int64, _ text: String, end: Int64?) -> SubtitleEvent {
+ SubtitleEvent(start: start, end: end, text: text, replacesPrevious: false)
+ }
+
+ private func shown(_ timeline: SubtitleTimeline, at time: Int64) -> [String] {
+ timeline.activeCues(at: time).map(\.text)
+ }
+
+ // MARK: Timing of one AVSubtitle
+
+ @Test("UINT32_MAX end means open, on any track")
+ func endlessIsOpen() {
+ let span = SubtitleEvent.span(
+ base: 10 * second, startDisplay: 0, endDisplay: .max, packetDuration: 0, replacesPrevious: false
+ )
+ #expect(span.start == 10 * second)
+ #expect(span.end == nil)
+ }
+
+ @Test("a missing end is open on a replacing track, a guess on a text track")
+ func missingEnd() {
+ let replacing = SubtitleEvent.span(
+ base: 10 * second, startDisplay: 0, endDisplay: 0, packetDuration: 2 * second, replacesPrevious: true
+ )
+ #expect(replacing.end == nil)
+
+ let withDuration = SubtitleEvent.span(
+ base: 10 * second, startDisplay: 0, endDisplay: 0, packetDuration: 2 * second, replacesPrevious: false
+ )
+ #expect(withDuration.end == 12 * second)
+
+ let withoutDuration = SubtitleEvent.span(
+ base: 10 * second, startDisplay: 500, endDisplay: 0, packetDuration: 0, replacesPrevious: false
+ )
+ #expect(withoutDuration.start == 10_500_000)
+ #expect(withoutDuration.end == 13_500_000)
+ }
+
+ @Test("display times are milliseconds after the base")
+ func explicitSpan() {
+ let span = SubtitleEvent.span(
+ base: 10 * second, startDisplay: 250, endDisplay: 4_000, packetDuration: 0, replacesPrevious: true
+ )
+ #expect(span.start == 10_250_000)
+ #expect(span.end == 14 * second)
+ }
+
+ // MARK: Replacing tracks (captions, DVB, PGS)
+
+ /// Before the timeline, a cue without an end (`UINT32_MAX` ms) stayed up
+ /// for 49 days and every later caption stacked under it.
+ @Test("an open cue ends where the next update starts")
+ func openCueEndsAtNext() {
+ var timeline = SubtitleTimeline()
+ timeline.apply(screen(1 * second, "one"))
+ timeline.apply(screen(4 * second, "two"))
+
+ #expect(shown(timeline, at: 3 * second) == ["one"])
+ #expect(shown(timeline, at: 4 * second) == ["two"])
+ #expect(shown(timeline, at: 5 * second) == ["two"])
+ }
+
+ /// The empty update DVB/PGS/captions send to clear the screen used to be
+ /// dropped as "no text", leaving the previous cue up.
+ @Test("a clear ends what is showing and shows nothing itself")
+ func clearEndsCue() {
+ var timeline = SubtitleTimeline()
+ timeline.apply(screen(1 * second, "one"))
+ timeline.apply(screen(3 * second, ""))
+
+ #expect(shown(timeline, at: 2 * second) == ["one"])
+ #expect(shown(timeline, at: 3 * second).isEmpty)
+ #expect(timeline.count == 1)
+ }
+
+ @Test("a cue with an end of its own is still ended by the next update")
+ func explicitEndCutByNext() {
+ var timeline = SubtitleTimeline()
+ // DVB page timeouts are long; the next page replaces it well before.
+ timeline.apply(screen(1 * second, "page one", end: 31 * second))
+ timeline.apply(screen(5 * second, "page two", end: 35 * second))
+ #expect(shown(timeline, at: 4 * second) == ["page one"])
+ #expect(shown(timeline, at: 6 * second) == ["page two"])
+ #expect(shown(timeline, at: 36 * second).isEmpty)
+ }
+
+ @Test("an open cue without a successor stops after the limit")
+ func openCueIsCapped() {
+ var timeline = SubtitleTimeline()
+ timeline.apply(screen(1 * second, "stuck"))
+ let limit = SubtitleTimeline.openEndedLimit
+ #expect(shown(timeline, at: 1 * second + limit - 1) == ["stuck"])
+ #expect(shown(timeline, at: 1 * second + limit).isEmpty)
+ }
+
+ @Test("an update arriving before one already stored is bounded by it")
+ func outOfOrderIsBounded() {
+ var timeline = SubtitleTimeline()
+ timeline.apply(screen(5 * second, "later"))
+ timeline.apply(screen(2 * second, "earlier"))
+ #expect(shown(timeline, at: 4 * second) == ["earlier"])
+ #expect(shown(timeline, at: 5 * second) == ["later"])
+ }
+
+ /// A re-read range (backward seek) sends the same updates again; the
+ /// stored ones were bounded by their successors meanwhile, so the ends
+ /// differ and only start and text can identify them.
+ @Test("re-reading a range does not double it")
+ func rereadIsDeduplicated() {
+ var timeline = SubtitleTimeline()
+ for _ in 0..<2 {
+ timeline.apply(screen(1 * second, "one"))
+ timeline.apply(screen(3 * second, ""))
+ timeline.apply(screen(4 * second, "two"))
+ }
+ #expect(timeline.count == 2)
+ #expect(shown(timeline, at: 2 * second) == ["one"])
+ #expect(shown(timeline, at: 3_500_000).isEmpty)
+ #expect(shown(timeline, at: 4_500_000) == ["two"])
+ }
+
+ @Test("on a replacing track, a different update at the same instant wins")
+ func sameStartReplaces() {
+ var timeline = SubtitleTimeline()
+ timeline.apply(screen(1 * second, "ROW ONE\nROW TWO"))
+ // After a seek the caption decoder starts from an empty screen and
+ // decodes the same instant differently.
+ timeline.apply(screen(1 * second, "ROW TWO"))
+ #expect(timeline.count == 1)
+ #expect(shown(timeline, at: 2 * second) == ["ROW TWO"])
+ }
+
+ // MARK: Text tracks (SRT, ASS, WebVTT)
+
+ @Test("text cues keep their overlaps")
+ func textOverlapKept() {
+ var timeline = SubtitleTimeline()
+ timeline.apply(text(1 * second, "one", end: 5 * second))
+ timeline.apply(text(2 * second, "two", end: 4 * second))
+ #expect(shown(timeline, at: 3 * second) == ["one", "two"])
+ #expect(shown(timeline, at: 4_500_000) == ["one"])
+ }
+
+ @Test("an open text cue ends at the next cue, a closed one does not")
+ func openTextCueEndsAtNext() {
+ var timeline = SubtitleTimeline()
+ timeline.apply(text(1 * second, "open", end: nil))
+ timeline.apply(text(3 * second, "next", end: 6 * second))
+ #expect(shown(timeline, at: 2 * second) == ["open"])
+ #expect(shown(timeline, at: 4 * second) == ["next"])
+ }
+
+ @Test("an empty text cue is nothing, not a clear")
+ func emptyTextCueIgnored() {
+ var timeline = SubtitleTimeline()
+ timeline.apply(text(1 * second, "one", end: 5 * second))
+ timeline.apply(text(2 * second, "", end: 3 * second))
+ #expect(shown(timeline, at: 4 * second) == ["one"])
+ }
+
+ // MARK: Seeks and growth
+
+ @Test("a seek ends open cues, not the ones the source gave an end")
+ func seekClosesOpenCues() {
+ var timeline = SubtitleTimeline()
+ timeline.apply(screen(1 * second, "open"))
+ timeline.apply(text(1 * second, "closed", end: 9 * second))
+ timeline.closeOpenCues(at: 4 * second)
+ #expect(shown(timeline, at: 3 * second) == ["open", "closed"])
+ #expect(shown(timeline, at: 5 * second) == ["closed"])
+
+ // Cues after the seek target are the future of the new position.
+ var later = SubtitleTimeline()
+ later.apply(screen(10 * second, "ahead"))
+ later.closeOpenCues(at: 4 * second)
+ #expect(shown(later, at: 11 * second) == ["ahead"])
+ }
+
+ /// A day of live roll-up captions is hundreds of thousands of updates;
+ /// every one of them used to be kept.
+ @Test("the timeline keeps a bounded number of cues, dropping the oldest")
+ func boundedGrowth() {
+ var timeline = SubtitleTimeline()
+ let total = SubtitleTimeline.capacity * 2
+ for index in 0..
Date: Mon, 12 Oct 2026 22:52:11 +0200
Subject: [PATCH 3/4] feat: DVB, PGS and DVD bitmap subtitles as positioned
image cues
SUBTITLE_BITMAP rects from avcodec_decode_subtitle2 were dropped. They
now become SubtitleImages on the cue: a premultiplied sRGB CGImage
(palette ARGB -> premultiplied BGRA) and a rect normalized to the
subtitle's authoring canvas as the decoder reports it after decoding
(DVB display definition, PGS presentation size, DVD size:), with a 720x576
fallback grown to fit the rects. The DVB context's size is zeroed before
opening so the stream's display definition decides it.
SubtitleCue gains images (defaulted, empty for text cues); bitmap cues
carry empty text, so text-only consumers keep working and show nothing.
The timeline treats an update with images as content rather than a clear,
deduplicates re-decoded bitmaps by position and size, and bounds bitmap
memory at 64 MB on top of the cue count. LumePlayer exposes the active
cues' images as subtitleImages; LumePlayerView has no subtitle overlay, so
drawing stays with the host.
Stacked on #46.
---
AGENTS.md | 3 +-
PLAN.md | 2 +-
Sources/LumeEngine/LumePlayer.swift | 16 +-
.../Subtitle/SubtitleBitmap.swift | 163 ++++++++++++++++
.../LumeEngineCore/Subtitle/SubtitleCue.swift | 19 +-
.../Subtitle/SubtitleDecoder.swift | 22 ++-
.../Subtitle/SubtitleTimeline.swift | 83 +++++---
.../BitmapSubtitleTests.swift | 178 ++++++++++++++++++
.../Fixtures/generate-fixtures.sh | 75 ++++++++
9 files changed, 526 insertions(+), 35 deletions(-)
create mode 100644 Sources/LumeEngineCore/Subtitle/SubtitleBitmap.swift
create mode 100644 Tests/LumeEngineCoreTests/BitmapSubtitleTests.swift
diff --git a/AGENTS.md b/AGENTS.md
index ccece3b..e986b78 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -51,7 +51,7 @@ Prerequisites:
build/scripts/build-ffmpeg.sh macos-arm64 # one slice is enough for local dev
build/scripts/make-xcframework.sh
```
-- Tests need a host `ffmpeg` CLI (Homebrew) — fixtures are generated lazily on first test run by `Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh` into `TestStreams/generated/` (gitignored, idempotent). `Fixtures.swift` finds Homebrew ffmpeg automatically; override with the `FFMPEG` env var. The closed-caption fixture (`captions.ts`) also needs `python3` (the one Xcode's command-line tools provide is enough), because no ffmpeg CLI option writes captions into the video; without it that fixture is skipped and its tests fail as missing.
+- Tests need a host `ffmpeg` CLI (Homebrew) — fixtures are generated lazily on first test run by `Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh` into `TestStreams/generated/` (gitignored, idempotent). `Fixtures.swift` finds Homebrew ffmpeg automatically; override with the `FFMPEG` env var. The closed-caption and bitmap-subtitle fixtures (`captions.ts`, `pgs.mkv`, `dvbsub.ts`, `dvdsub.mkv`) also need `python3` (the one Xcode's command-line tools provide is enough), because no ffmpeg CLI option writes captions into the video or renders subtitle bitmaps; without it those fixtures are skipped and their tests fail as missing.
CI (`.github/workflows/ci.yml`) builds the macOS FFmpeg slice (cached on `build/versions.json` + scripts + patches hashes) and runs `swift test` with a strict one-thread cooperative pool (see above), checks the Markdown docs offline (`.github/scripts/check-docs.py`: relative links and in-repo anchors resolve, `CLAUDE.md` is still the `AGENTS.md` symlink — run it after editing docs), then cross-compiles the library for iOS and tvOS via `xcodebuild` (compile-only, one cached slice each — visionOS is left to local verification because that matrix entry was never confirmed on a runner — the xrOS SDK ships with Xcode, but a visionOS *destination* also needs Xcode's visionOS platform component); tagged releases build all 10 platform slices and attach the xcframework.
@@ -89,6 +89,7 @@ This split is the core invariant of the codebase:
- **Dolby Vision with an IPT base layer is converted, not tagged.** Profiles 5, 10.0 and 20 (compatibility ID 0) carry IPTPQc2, which shows pink/green as YCbCr. No colour attachment fixes that — the RPU reshaping changes per scene and Apple's decoders don't apply it to decoded frames — so `Decode/DolbyVisionConverter` turns each frame into HDR10 on the GPU in `VideoDecoder.deliver`, the one point every frame (hardware, software, deinterlaced) passes with its side data. Detection is the stream's DV configuration record *or* an `AVCOL_SPC_IPT_C2` frame; many P5 encodes leave the VUI matrix unspecified, so don't reduce it to the tag. HDR10-compatible profiles (8.1 etc.) must stay on the zero-copy path. GPU refusals are per-frame (iOS refuses a backgrounded app); only a converter that can't exist gives up for the session.
- **Audio is fitted to the negotiated route width, never the route's capability.** `AudioDecoder` reads `AudioOutputRoute.outputChannels` (the session's `outputNumberOfChannels`, followed through route-change notifications) per frame and remixes per `AudioLayoutPolicy` with an explicit `DownmixMatrix` — a fold-down mixes centre and surrounds in, it never drops channels. Don't switch it to `maximumOutputNumberOfChannels`: a renderer fed more channels than the session outputs failed outright on tvOS. The engine only observes the audio session; setting its category, activation or preferred channel count is the host's job (Lume does it). When the route narrows, `SystemRenderer` keeps audio decoded for the wider route out of the renderer.
- **How long a subtitle shows is the timeline's decision, not the decoder's.** Decoders hand `Subtitle/SubtitleStore` a `SubtitleEvent` (start, end *or none*, text, whether the track replaces the whole screen with each update); `Subtitle/SubtitleTimeline` decides the rest: an open cue ends at the next update of its track (capped at 16 s), an empty update on a closed-caption/DVB/PGS/DVD track is a clear that ends what is showing, a seek ends open cues, re-read ranges are deduplicated, and the store is bounded. Don't turn `end_display_time == UINT32_MAX` into a number in a decoder or drop empty updates there — both put stuck or stacked cues on screen.
+- **Bitmap subtitles (DVB, PGS, DVD) are images on the cue, positioned by their own canvas.** `Subtitle/SubtitleBitmap` turns `SUBTITLE_BITMAP` rects into premultiplied sRGB `CGImage`s with a rect normalized to the canvas the decoder reports *after* decoding (DVB display definition, PGS presentation size, DVD `size:`), never to the video's pixel size — an HD DVB page over SD video, or PGS authored at 1080p over a 720p encode, would land off screen. The DVB context's size is zeroed before opening so the stream's display definition sets it. Bitmap cues carry empty `text` (text consumers like Lume show nothing rather than garbage); drawing them is the host's job, since `LumePlayerView` is bare video. The timeline bounds bitmap memory separately (`imageByteBudget`).
- **Closed captions leave the video lane through a tap, never a wait.** CEA-608 in A/53 side data (`AV_FRAME_DATA_A53_CC`) is lifted off each decoded frame in `VideoDecoder.receiveFrames` (before deinterlacing) by `Subtitle/ClosedCaptionTap`, stamped with the frame's PTS, and `trySend`-ed to an ordinary `SubtitleDecoder` running `ccaption` in real-time mode — a slow caption lane drops captions, never frames. The track is synthetic (`TrackInfo.embeddedClosedCaptionsIndex`) and appears in `PlayerSession.mediaInfo` once caption content is seen; `.opened` keeps the open-time track list, because no new `PlayerEvent` case may be added for it. Only CEA-608 field 1 (CC1/CC2) is decoded: FFmpeg has no CEA-708 (DTVCC) decoder, and teletext needs libzvbi, which the build does not include.
- **Zero global mutable state:** all configuration is `PlayerConfiguration` passed at session creation. No `static var` knobs.
- **No force unwraps / `as!` / `fatalError` in engine paths**; failures return typed `EngineError`. No `assertionFailure` in production channel/data paths — misuse degrades, never crashes.
diff --git a/PLAN.md b/PLAN.md
index e8a5ccc..1acf292 100644
--- a/PLAN.md
+++ b/PLAN.md
@@ -142,7 +142,7 @@ Brightness/contrast/saturation and MetalFX spatial upscaling operate CVPixelBuff
**D6 — One subtitle engine, composited as overlay.**
libass renders ASS/SSA (full effects, embedded fonts via attachment streams, word-by-word \k tags); PGS/DVB/VobSub decoded by FFmpeg to bitmaps; SRT/VTT parsed to styled cues rendered through the same layout box. Output = time-tagged `SubtitleImage` cues composited on a CALayer above the video surface (and into the PiP content when active; into Metal for HDR subtitle mode). Dual subtitle slots (main + secondary). External files attach as additional demux sessions.
-**Cue lifecycle and closed captions (as built).** Every subtitle source — demuxed stream, sidecar file, or captions lifted from the video — reaches one time-indexed store as display updates; the store's `SubtitleTimeline` alone decides how long each shows (open cues end at the next update or a clear, capped; seeks end open cues; bounded size). CEA-608 captions carried as A/53 side data are tapped off decoded video frames without ever blocking the video lane and surface as a synthetic "CC1" subtitle track once seen, additively through `PlayerSession.mediaInfo` (no new event).
+**Cue lifecycle and closed captions (as built).** Every subtitle source — demuxed stream, sidecar file, or captions lifted from the video — reaches one time-indexed store as display updates; the store's `SubtitleTimeline` alone decides how long each shows (open cues end at the next update or a clear, capped; seeks end open cues; bounded size). CEA-608 captions carried as A/53 side data are tapped off decoded video frames without ever blocking the video lane and surface as a synthetic "CC1" subtitle track once seen, additively through `PlayerSession.mediaInfo` (no new event). PGS/DVB/DVD bitmaps reach the same store as `SubtitleImage`s (premultiplied `CGImage` + rect normalized to the subtitle's authoring canvas) on cues with empty text; compositing them is still the host's job until the overlay layer described above exists.
**System caption respect:** map `MACaptionAppearance` (user's Style settings) onto text-cue styling by default.
**D7 — Buffering is PTS-math, not heuristics.**
diff --git a/Sources/LumeEngine/LumePlayer.swift b/Sources/LumeEngine/LumePlayer.swift
index d88ebb1..51b4c76 100644
--- a/Sources/LumeEngine/LumePlayer.swift
+++ b/Sources/LumeEngine/LumePlayer.swift
@@ -28,6 +28,10 @@ public final class LumePlayer {
public private(set) var isHardwareDecoding = false
/// Text of the currently visible subtitle cue(s), `nil` when none.
public private(set) var subtitleText: String?
+ /// Bitmaps of the currently visible subtitle cue(s) — DVB, PGS and DVD
+ /// subtitles, which have no text. `LumePlayerView` does not draw them:
+ /// overlay each image at its `rect` scaled to the video's display area.
+ public private(set) var subtitleImages: [SubtitleImage] = []
public var audioTracks: [TrackInfo] { mediaInfo?.audioTracks ?? [] }
public var subtitleTracks: [TrackInfo] { mediaInfo?.subtitleTracks ?? [] }
@@ -146,7 +150,10 @@ public final class LumePlayer {
public func selectSubtitleTrack(_ index: Int32?) {
let session = session
Task { await session?.selectSubtitleTrack(index) }
- if index == nil { subtitleText = nil }
+ if index == nil {
+ subtitleText = nil
+ subtitleImages = []
+ }
}
public func loadExternalSubtitles(url: String) async throws {
@@ -165,6 +172,8 @@ public final class LumePlayer {
position = 0
mediaInfo = nil
duration = nil
+ subtitleText = nil
+ subtitleImages = []
}
// MARK: Internals
@@ -210,10 +219,13 @@ public final class LumePlayer {
let now = session.renderer.currentTime
if MediaTime.isValid(now) {
let cues = session.subtitles.activeCues(at: now)
- let text = cues.map(\.text).joined(separator: "\n")
+ let text = cues.map(\.text).filter { !$0.isEmpty }.joined(separator: "\n")
subtitleText = text.isEmpty ? nil : text
+ let images = cues.flatMap(\.images)
+ if images != subtitleImages { subtitleImages = images }
} else {
subtitleText = nil
+ if !subtitleImages.isEmpty { subtitleImages = [] }
}
}
diff --git a/Sources/LumeEngineCore/Subtitle/SubtitleBitmap.swift b/Sources/LumeEngineCore/Subtitle/SubtitleBitmap.swift
new file mode 100644
index 0000000..9998156
--- /dev/null
+++ b/Sources/LumeEngineCore/Subtitle/SubtitleBitmap.swift
@@ -0,0 +1,163 @@
+internal import CFFmpeg
+import CoreGraphics
+import Foundation
+
+/// One bitmap of a DVB, PGS or DVD subtitle, ready to draw over the video.
+///
+/// `rect` is normalized to the picture the subtitle was authored for: (0, 0)
+/// is the top-left corner of the video, (1, 1) the bottom-right, so a host
+/// draws the image at `rect` scaled by the size of the video's display area
+/// (after aspect fitting). The authoring canvas — a DVB display definition
+/// (720×576 without one), a PGS presentation's video size, a DVD's `size:` —
+/// need not match the stream's pixel size; normalizing makes that moot.
+public struct SubtitleImage: Sendable, Equatable {
+ /// Premultiplied RGBA in sRGB.
+ public let image: CGImage
+ /// Position and size as fractions of the video picture.
+ public let rect: CGRect
+
+ public init(image: CGImage, rect: CGRect) {
+ self.image = image
+ self.rect = rect
+ }
+
+ /// Memory the bitmap holds, for the store's budget.
+ var byteCount: Int { image.bytesPerRow * image.height }
+}
+
+/// Palette-indexed `SUBTITLE_BITMAP` rects to `SubtitleImage`s.
+enum SubtitleBitmap {
+ /// The canvas when a decoder reports none — DVB's own default (ETSI EN
+ /// 300 743), and the PAL DVD frame; NTSC DVDs carry their size.
+ static let fallbackCanvas = (width: 720, height: 576)
+
+ /// The images of one decoded subtitle. `canvasWidth`/`canvasHeight` are
+ /// the codec context's dimensions after decoding, which the bitmap
+ /// decoders set to their authoring canvas.
+ static func images(of subtitle: AVSubtitle, canvasWidth: Int32, canvasHeight: Int32) -> [SubtitleImage] {
+ var rects: [UnsafeMutablePointer] = []
+ for index in 0.. SubtitleImage? {
+ let width = Int(rect.w)
+ let height = Int(rect.h)
+ let stride = Int(rect.linesize.0)
+ guard width > 0, height > 0, stride >= width,
+ let indices = rect.data.0, let paletteData = rect.data.1,
+ rect.nb_colors > 0
+ else { return nil }
+
+ let palette = UnsafeRawPointer(paletteData).withMemoryRebound(
+ to: UInt32.self, capacity: Int(rect.nb_colors)
+ ) { Array(UnsafeBufferPointer(start: $0, count: Int(rect.nb_colors))) }
+ let pixels = premultipliedPixels(
+ indices: UnsafeBufferPointer(start: indices, count: stride * height),
+ width: width, height: height, stride: stride, palette: palette
+ )
+ guard pixels.contains(where: { $0 >> 24 != 0 }),
+ let image = makeImage(pixels: pixels, width: width, height: height)
+ else { return nil } // fully transparent: nothing to draw
+
+ return SubtitleImage(
+ image: image,
+ rect: normalizedRect(
+ x: Int(rect.x), y: Int(rect.y), width: width, height: height,
+ canvasWidth: canvasWidth, canvasHeight: canvasHeight
+ )
+ )
+ }
+
+ /// Palette-indexed pixels to premultiplied ARGB words. FFmpeg's palettes
+ /// are straight-alpha `0xAARRGGBB`; Core Animation and Core Graphics
+ /// composite premultiplied, and straight alpha drawn as premultiplied
+ /// shows a bright fringe around every glyph's anti-aliased edge. An index
+ /// beyond the palette is transparent.
+ static func premultipliedPixels(
+ indices: UnsafeBufferPointer,
+ width: Int,
+ height: Int,
+ stride: Int,
+ palette: [UInt32]
+ ) -> [UInt32] {
+ let premultiplied = palette.map(premultiply)
+ var pixels = [UInt32](repeating: 0, count: width * height)
+ for row in 0.. UInt32 {
+ let alpha = argb >> 24
+ func channel(_ shift: UInt32) -> UInt32 {
+ ((argb >> shift & 0xFF) * alpha + 127) / 255
+ }
+ return alpha << 24 | channel(16) << 16 | channel(8) << 8 | channel(0)
+ }
+
+ /// ARGB words in native (little-endian) order are BGRA bytes in memory:
+ /// `byteOrder32Little` + `premultipliedFirst`, Core Graphics' native layout.
+ static func makeImage(pixels: [UInt32], width: Int, height: Int) -> CGImage? {
+ guard pixels.count == width * height,
+ let space = CGColorSpace(name: CGColorSpace.sRGB)
+ else { return nil }
+ let data = pixels.withUnsafeBytes { Data($0) }
+ guard let provider = CGDataProvider(data: data as CFData) else { return nil }
+ return CGImage(
+ width: width,
+ height: height,
+ bitsPerComponent: 8,
+ bitsPerPixel: 32,
+ bytesPerRow: width * 4,
+ space: space,
+ bitmapInfo: CGBitmapInfo(rawValue: CGBitmapInfo.byteOrder32Little.rawValue
+ | CGImageAlphaInfo.premultipliedFirst.rawValue),
+ provider: provider,
+ decode: nil,
+ shouldInterpolate: true,
+ intent: .defaultIntent
+ )
+ }
+
+ static func normalizedRect(
+ x: Int, y: Int, width: Int, height: Int,
+ canvasWidth: Int, canvasHeight: Int
+ ) -> CGRect {
+ guard canvasWidth > 0, canvasHeight > 0 else { return .zero }
+ let canvasW = Double(canvasWidth)
+ let canvasH = Double(canvasHeight)
+ return CGRect(
+ x: Double(x) / canvasW,
+ y: Double(y) / canvasH,
+ width: Double(width) / canvasW,
+ height: Double(height) / canvasH
+ )
+ }
+}
diff --git a/Sources/LumeEngineCore/Subtitle/SubtitleCue.swift b/Sources/LumeEngineCore/Subtitle/SubtitleCue.swift
index 9e4f274..3e53b56 100644
--- a/Sources/LumeEngineCore/Subtitle/SubtitleCue.swift
+++ b/Sources/LumeEngineCore/Subtitle/SubtitleCue.swift
@@ -1,8 +1,14 @@
import Foundation
import Synchronization
-/// One subtitle cue on the engine timeline. Text-only for now; bitmap cues
-/// (PGS/DVB) and full libass styling land with the subtitle pack (PLAN.md P5+).
+/// One subtitle cue on the engine timeline: text, bitmaps, or (rarely) both.
+/// Full libass styling lands with the subtitle pack (PLAN.md P5+).
+///
+/// Bitmap formats (DVB, PGS, DVD) arrive as `images` with empty `text`, so a
+/// consumer that only reads `text` shows nothing for them rather than
+/// something wrong. Drawing them is the host's job: `LumePlayerView` is bare
+/// video. Draw each image at its `rect` scaled to the video's display area,
+/// above the video.
public struct SubtitleCue: Sendable, Equatable, Identifiable {
public let id: UInt64
/// Engine µs.
@@ -10,12 +16,19 @@ public struct SubtitleCue: Sendable, Equatable, Identifiable {
public let end: Int64
/// Plain display text (override tags stripped, line breaks preserved).
public let text: String
+ /// Bitmaps to draw over the video, positioned relative to it.
+ public let images: [SubtitleImage]
- public init(id: UInt64, start: Int64, end: Int64, text: String) {
+ public init(id: UInt64, start: Int64, end: Int64, text: String, images: [SubtitleImage] = []) {
self.id = id
self.start = start
self.end = end
self.text = text
+ self.images = images
+ }
+
+ func ending(at time: Int64) -> SubtitleCue {
+ SubtitleCue(id: id, start: start, end: time, text: text, images: images)
}
}
diff --git a/Sources/LumeEngineCore/Subtitle/SubtitleDecoder.swift b/Sources/LumeEngineCore/Subtitle/SubtitleDecoder.swift
index 4878e26..cda9a33 100644
--- a/Sources/LumeEngineCore/Subtitle/SubtitleDecoder.swift
+++ b/Sources/LumeEngineCore/Subtitle/SubtitleDecoder.swift
@@ -7,7 +7,8 @@ import Foundation
/// so does the closed-caption lane, whose packets come from the video decoder
/// (`ClosedCaptionTap`) rather than the demuxer. Each decoded update goes to
/// the store as a `SubtitleEvent`; how long it shows is the store's policy.
-/// Bitmap codecs (PGS/DVB/VobSub) are recognized but skipped for now.
+/// Bitmap codecs (DVB, PGS, VobSub) produce cues with `images` instead of
+/// text (`SubtitleBitmap`).
public final class SubtitleDecoder: @unchecked Sendable {
public let store: SubtitleStore
@@ -204,9 +205,15 @@ public final class SubtitleDecoder: @unchecked Sendable {
lines.append(String(cString: text))
}
default:
- break // bitmap subtitle — deferred (libass/PGS pack)
+ break // bitmaps below
}
}
+ // Read after decoding: the bitmap decoders set the context's size to
+ // their authoring canvas as they parse it (DVB display definition,
+ // PGS presentation, DVD `size:`).
+ let images = SubtitleBitmap.images(
+ of: subtitle, canvasWidth: context.pointee.width, canvasHeight: context.pointee.height
+ )
var text = lines.joined(separator: "\n")
if isClosedCaptions {
// Caption rows are indented by column position, which a text
@@ -227,7 +234,8 @@ public final class SubtitleDecoder: @unchecked Sendable {
start: span.start,
end: span.end,
text: text.trimmingCharacters(in: .whitespacesAndNewlines),
- replacesPrevious: replacesPrevious
+ replacesPrevious: replacesPrevious,
+ images: images
))
}
@@ -246,6 +254,14 @@ public final class SubtitleDecoder: @unchecked Sendable {
avcodec_free_context(&pointer)
return false
}
+ if context.pointee.codec_id == AV_CODEC_ID_DVB_SUBTITLE {
+ // The DVB decoder takes its canvas from the first display
+ // definition only while the context has no size; the stream probe
+ // may have left one (often the 720×576 default), which would pin
+ // an HD page's bitmaps to an SD canvas.
+ context.pointee.width = 0
+ context.pointee.height = 0
+ }
context.pointee.pkt_timebase = lume_av_time_base_q()
var options: OpaquePointer?
for (key, value) in codecOptions {
diff --git a/Sources/LumeEngineCore/Subtitle/SubtitleTimeline.swift b/Sources/LumeEngineCore/Subtitle/SubtitleTimeline.swift
index d3f8381..cbcee35 100644
--- a/Sources/LumeEngineCore/Subtitle/SubtitleTimeline.swift
+++ b/Sources/LumeEngineCore/Subtitle/SubtitleTimeline.swift
@@ -8,14 +8,19 @@ struct SubtitleEvent: Sendable, Equatable {
/// exactly that — "until the next update" — and a decoder that turned it
/// into a number would leave the text up for 49 days (`UINT32_MAX` ms).
var end: Int64?
- /// Display text, trimmed. Empty on a replacing track means "clear the
- /// screen", which is how those formats end what is showing.
+ /// Display text, trimmed. Empty — with no images either — on a replacing
+ /// track means "clear the screen", which is how those formats end what is
+ /// showing.
var text: String
/// The track describes the whole screen with every update (closed
/// captions, DVB, PGS, DVD): an update ends everything shown before it.
/// Text formats (SRT, ASS, WebVTT, mov_text) overlap freely instead, each
/// cue carrying its own end.
var replacesPrevious: Bool
+ /// Bitmaps (DVB, PGS, DVD).
+ var images: [SubtitleImage] = []
+
+ var isEmpty: Bool { text.isEmpty && images.isEmpty }
/// Where an `AVSubtitle` sits on the engine timeline. `base` is the
/// subtitle's PTS in engine µs, display times are milliseconds after it.
@@ -66,8 +71,8 @@ struct SubtitleEvent: Sendable, Equatable {
/// seek, backfill seek) does not double its lines. On a replacing track a
/// different update at the same start replaces the stored one: the screen
/// at one instant has one state, and the latest decode of it wins.
-/// - At most `capacity` cues are kept; the ones farthest from where cues are
-/// being added go first.
+/// - At most `capacity` cues, holding at most `imageByteBudget` of bitmaps,
+/// are kept; the ones farthest from where cues are being added go first.
struct SubtitleTimeline: Sendable {
/// How long an open cue stays without a successor. Caption decoders on
/// televisions erase a caption left standing by about this long without
@@ -80,6 +85,12 @@ struct SubtitleTimeline: Sendable {
/// for a live track it is half an hour or more of roll-up captions.
static let capacity = 8_192
+ /// Bitmap memory kept. A full-HD PGS line is the better part of a
+ /// megabyte decoded, so a cue count alone would let a film's bitmap track
+ /// grow to gigabytes; this keeps the last minute or more of a live DVB
+ /// page, and a backward seek re-decodes whatever was dropped.
+ static let imageByteBudget = 64 << 20
+
/// How far back from an insertion point active cues are looked for.
/// Overlap is rare and short-lived; a replacing track has at most one.
private static let scanLimit = 64
@@ -89,15 +100,19 @@ struct SubtitleTimeline: Sendable {
/// The end is the limit, not the source's: the next update or a seek
/// may still bound it.
var provisional: Bool
+
+ var imageBytes: Int { cue.images.reduce(0) { $0 + $1.byteCount } }
}
private var entries: [Entry] = []
private var nextID: UInt64 = 0
+ /// Sum of `imageBytes` over `entries`.
+ private(set) var imageBytes = 0
var count: Int { entries.count }
mutating func apply(_ event: SubtitleEvent) {
- if event.text.isEmpty {
+ if event.isEmpty {
// A clear. Text tracks have no such thing; an empty text cue is
// just nothing to show.
guard event.replacesPrevious else { return }
@@ -119,13 +134,12 @@ struct SubtitleTimeline: Sendable {
}
guard end > event.start else { return }
- entries.insert(
- Entry(
- cue: SubtitleCue(id: nextID, start: event.start, end: end, text: event.text),
- provisional: event.end == nil
- ),
- at: index
+ let entry = Entry(
+ cue: SubtitleCue(id: nextID, start: event.start, end: end, text: event.text, images: event.images),
+ provisional: event.end == nil
)
+ entries.insert(entry, at: index)
+ imageBytes += entry.imageBytes
nextID += 1
trim(keeping: index)
}
@@ -140,6 +154,7 @@ struct SubtitleTimeline: Sendable {
mutating func removeAll() {
entries.removeAll()
+ imageBytes = 0
}
/// All cues covering `time` (engine µs), ordered by start.
@@ -175,7 +190,11 @@ struct SubtitleTimeline: Sendable {
var index = insertionIndex(for: event.start) - 1
while index >= 0 && entries[index].cue.start == event.start {
let entry = entries[index]
- if entry.cue.text == event.text {
+ // Bitmaps decoded twice are different objects; where and how big
+ // they are identifies them.
+ if entry.cue.text == event.text,
+ entry.cue.images.map(\.rect) == event.images.map(\.rect),
+ entry.cue.images.map(\.image.width) == event.images.map(\.image.width) {
// A stored end may have been bounded by a successor since, so
// it only has to match where both ends came from the source.
if event.replacesPrevious || event.end == nil || entry.provisional { return true }
@@ -189,7 +208,7 @@ struct SubtitleTimeline: Sendable {
private mutating func removeEntries(startingAt start: Int64) {
var index = insertionIndex(for: start) - 1
while index >= 0 && entries[index].cue.start == start {
- entries.remove(at: index)
+ imageBytes -= entries.remove(at: index).imageBytes
index -= 1
}
}
@@ -201,26 +220,40 @@ struct SubtitleTimeline: Sendable {
while index >= 0 && scanned < Self.scanLimit {
let entry = entries[index]
if entry.cue.start < time, entry.cue.end > time, !onlyProvisional || entry.provisional {
- entries[index].cue = SubtitleCue(
- id: entry.cue.id, start: entry.cue.start, end: time, text: entry.cue.text
- )
+ entries[index].cue = entry.cue.ending(at: time)
}
index -= 1
scanned += 1
}
}
- /// Drops cues beyond `capacity` from the end farther from `index`, where
- /// cues are being added — the front on a live timeline, the back after a
- /// seek far into the past of a long file. Drops an eighth more than
- /// needed so the shift is not paid on every insert.
+ /// Drops cues beyond `capacity` or `imageByteBudget` from the end farther
+ /// from `index`, where cues are being added — the front on a live
+ /// timeline, the back after a seek far into the past of a long file. Over
+ /// capacity it drops an eighth more than needed, so the shift is not paid
+ /// on every insert. The cue just added always stays.
private mutating func trim(keeping index: Int) {
- guard entries.count > Self.capacity else { return }
- let excess = entries.count - Self.capacity + Self.capacity / 8
- if index < entries.count / 2 {
- entries.removeLast(excess)
+ var kept = index
+ if entries.count > Self.capacity {
+ drop(entries.count - Self.capacity + Self.capacity / 8, keeping: &kept)
+ }
+ while imageBytes > Self.imageByteBudget, entries.count > 1 {
+ drop(1, keeping: &kept)
+ }
+ }
+
+ private mutating func drop(_ count: Int, keeping kept: inout Int) {
+ let count = min(count, entries.count - 1)
+ guard count > 0 else { return }
+ let removed: ArraySlice
+ if kept < entries.count / 2 {
+ removed = entries[(entries.count - count)...]
+ entries.removeLast(count)
} else {
- entries.removeFirst(excess)
+ removed = entries[.. (r: Int, g: Int, b: Int, a: Int)? {
+ guard let data = image.dataProvider?.data as Data? else { return nil }
+ let offset = y * image.bytesPerRow + x * 4
+ guard offset + 4 <= data.count else { return nil }
+ // byteOrder32Little + premultipliedFirst: B, G, R, A in memory.
+ return (Int(data[offset + 2]), Int(data[offset + 1]), Int(data[offset]), Int(data[offset + 3]))
+}
+
+@Suite("Bitmap subtitle conversion")
+struct BitmapSubtitleConversionTests {
+ @Test("palette colours are premultiplied")
+ func premultiply() {
+ #expect(SubtitleBitmap.premultiply(0xFF_FF_FF_FF) == 0xFF_FF_FF_FF)
+ #expect(SubtitleBitmap.premultiply(0x00_FF_FF_FF) == 0)
+ // Half-transparent red: colour scaled by alpha, alpha kept.
+ #expect(SubtitleBitmap.premultiply(0x80_FF_00_00) == 0x80_80_00_00)
+ #expect(SubtitleBitmap.premultiply(0x40_80_40_20) == 0x40_20_10_08)
+ }
+
+ @Test("indexed rows honour the stride; indices past the palette are transparent")
+ func indexedPixels() {
+ // Two rows of three pixels, stride four (one byte of padding each).
+ let indices: [UInt8] = [0, 1, 2, 9, 2, 1, 7, 9]
+ let palette: [UInt32] = [0x00_00_00_00, 0xFF_FF_FF_FF, 0x80_FF_00_00]
+ let pixels = indices.withUnsafeBufferPointer {
+ SubtitleBitmap.premultipliedPixels(indices: $0, width: 3, height: 2, stride: 4, palette: palette)
+ }
+ #expect(pixels == [0, 0xFF_FF_FF_FF, 0x80_80_00_00, 0x80_80_00_00, 0xFF_FF_FF_FF, 0])
+ }
+
+ @Test("an image round-trips as premultiplied BGRA")
+ func imageLayout() throws {
+ let image = try #require(SubtitleBitmap.makeImage(pixels: [0xFF_FF_FF_FF, 0x80_80_00_00], width: 2, height: 1))
+ #expect(image.width == 2 && image.height == 1)
+ #expect(image.alphaInfo == .premultipliedFirst)
+ let white = try #require(pixel(of: image, x: 0, y: 0))
+ #expect(white == (255, 255, 255, 255))
+ let red = try #require(pixel(of: image, x: 1, y: 0))
+ #expect(red == (128, 0, 0, 128))
+ }
+
+ @Test("rects are normalized to the authoring canvas")
+ func normalizedRect() {
+ let rect = SubtitleBitmap.normalizedRect(x: 480, y: 900, width: 960, height: 108, canvasWidth: 1920, canvasHeight: 1080)
+ #expect(rect == CGRect(x: 0.25, y: 0.8333333333333334, width: 0.5, height: 0.1))
+ #expect(SubtitleBitmap.normalizedRect(x: 0, y: 0, width: 1, height: 1, canvasWidth: 0, canvasHeight: 0) == .zero)
+ }
+
+ private func image(width: Int, height: Int) throws -> SubtitleImage {
+ let image = try #require(SubtitleBitmap.makeImage(
+ pixels: [UInt32](repeating: 0xFF_FF_FF_FF, count: width * height), width: width, height: height
+ ))
+ return SubtitleImage(image: image, rect: CGRect(x: 0.4, y: 0.9, width: 0.1, height: 0.05))
+ }
+
+ @Test("a bitmap update is a cue, not a clear, and a re-read does not double it")
+ func bitmapCuesInTimeline() throws {
+ var timeline = SubtitleTimeline()
+ let bitmap = try image(width: 8, height: 4)
+ timeline.apply(SubtitleEvent(start: 1_000_000, end: nil, text: "", replacesPrevious: true, images: [bitmap]))
+ timeline.apply(SubtitleEvent(start: 3_000_000, end: nil, text: "", replacesPrevious: true))
+ let shown = timeline.activeCues(at: 2_000_000)
+ #expect(shown.count == 1)
+ #expect(shown.first?.images.map(\.rect) == [bitmap.rect])
+ #expect(shown.first?.text == "")
+ #expect(timeline.activeCues(at: 3_000_000).isEmpty)
+
+ // The same update decoded again makes a new CGImage of the same size.
+ let again = try image(width: 8, height: 4)
+ timeline.apply(SubtitleEvent(start: 1_000_000, end: nil, text: "", replacesPrevious: true, images: [again]))
+ #expect(timeline.count == 1)
+ }
+
+ @Test("bitmap memory is bounded, dropping the oldest cues")
+ func imageBudget() throws {
+ var timeline = SubtitleTimeline()
+ // 1 MB per cue: the budget holds 64 of them.
+ let bitmap = try image(width: 512, height: 512)
+ for index in 0..<100 {
+ timeline.apply(SubtitleEvent(
+ start: Int64(index) * 1_000_000, end: nil, text: "", replacesPrevious: true, images: [bitmap]
+ ))
+ }
+ #expect(timeline.imageBytes <= SubtitleTimeline.imageByteBudget)
+ #expect(timeline.count == SubtitleTimeline.imageByteBudget / bitmap.byteCount)
+ #expect(timeline.activeCues(at: 99_500_000).count == 1)
+ #expect(timeline.activeCues(at: 500_000).isEmpty)
+ timeline.removeAll()
+ #expect(timeline.imageBytes == 0)
+ }
+}
+
+@Suite("Bitmap subtitles in playback", .serialized)
+struct BitmapSubtitlePlaybackTests {
+ private func eventually(
+ timeout: TimeInterval = 10,
+ _ condition: () async -> Bool
+ ) async -> Bool {
+ let deadline = Date(timeIntervalSinceNow: timeout)
+ while Date() < deadline {
+ if await condition() { return true }
+ try? await Task.sleep(for: .milliseconds(100))
+ }
+ return false
+ }
+
+ /// The fixtures carry the same two bitmaps (see `generate-fixtures.sh`):
+ /// 64x16 at (928, 1000) for 1–3 s, 48x8 at (936, 960) for 4–6 s, on a
+ /// 1920x1080 canvas over a 320x180 video — so a canvas taken from the
+ /// video instead of the subtitle would put them far off screen.
+ @Test(
+ "bitmap subtitles become positioned image cues",
+ .timeLimit(.minutes(1)),
+ arguments: ["pgs.mkv", "dvbsub.ts", "dvdsub.mkv"]
+ )
+ func bitmapCues(fixture: String) async throws {
+ var configuration = PlayerConfiguration()
+ configuration.muted = true
+ configuration.bufferTarget = 0.3
+ let session = PlayerSession(configuration: configuration)
+ let info = try await session.open(url: try Fixtures.path(fixture))
+ let track = try #require(info.subtitleTracks.first)
+
+ await session.selectSubtitleTrack(track.index)
+ let start = info.startTime
+ let decoded = await eventually {
+ session.subtitles.activeCues(at: start + 5_000_000).first?.images.isEmpty == false
+ }
+ #expect(decoded, "\(fixture): cues should decode, store has \(session.subtitles.count)")
+
+ let first = try #require(session.subtitles.activeCues(at: start + 2_000_000).first, "\(fixture): first cue")
+ #expect(first.text.isEmpty, "bitmap cues carry no text, so text consumers show nothing")
+ let image = try #require(first.images.first)
+ #expect(first.images.count == 1)
+ #expect(abs(first.start - (start + 1_000_000)) <= 40_000, "\(fixture): starts at \(first.start - start)")
+ #expect(abs(first.end - (start + 3_000_000)) <= 40_000, "\(fixture): ends at \(first.end - start)")
+
+ // Decoders may crop fully transparent borders (FFmpeg's DVD decoder
+ // drops the 8 transparent columns); the visible part must still land
+ // within a few canvas pixels of the original.
+ let tolerance = 4.0 / 1080
+ #expect(abs(image.rect.maxX - 992.0 / 1920) <= tolerance, "\(fixture): \(image.rect)")
+ #expect(abs(image.rect.maxY - 1016.0 / 1080) <= tolerance, "\(fixture): \(image.rect)")
+ #expect(image.rect.minX >= 928.0 / 1920 - tolerance, "\(fixture): \(image.rect)")
+ #expect(image.rect.minX <= 936.0 / 1920 + tolerance, "\(fixture): \(image.rect)")
+ #expect(abs(image.rect.minY - 1000.0 / 1080) <= tolerance, "\(fixture): \(image.rect)")
+ #expect(image.image.width == Int((image.rect.width * 1920).rounded()), "\(fixture): one image pixel per canvas pixel")
+
+ // Column 2 is transparent, column 16 opaque white, column 48
+ // half-transparent red (premultiplied). The DVB and DVD encoders
+ // requantize the palette, hence the tolerances.
+ let offsetX = Int(((image.rect.minX - 928.0 / 1920) * 1920).rounded())
+ let offsetY = Int(((image.rect.minY - 1000.0 / 1080) * 1080).rounded())
+ let row = 8 - offsetY
+ if offsetX <= 2 {
+ let clear = try #require(pixel(of: image.image, x: 2 - offsetX, y: row))
+ #expect(clear.a <= 16, "\(fixture): \(clear)")
+ }
+ let white = try #require(pixel(of: image.image, x: 16 - offsetX, y: row))
+ #expect(white.a >= 240 && white.r >= 220 && white.g >= 220 && white.b >= 220, "\(fixture): \(white)")
+ let red = try #require(pixel(of: image.image, x: 48 - offsetX, y: row))
+ #expect((100...160).contains(red.a) && red.r >= red.a - 30 && red.g <= 30 && red.b <= 30, "\(fixture): \(red)")
+ #expect(red.r <= red.a, "\(fixture): premultiplied colour never exceeds alpha")
+
+ #expect(session.subtitles.activeCues(at: start + 3_500_000).isEmpty, "\(fixture): cleared at 3 s")
+ let second = try #require(session.subtitles.activeCues(at: start + 5_000_000).first?.images.first)
+ #expect(abs(second.rect.width - 48.0 / 1920) <= tolerance, "\(fixture): \(second.rect)")
+ #expect(session.subtitles.activeCues(at: start + 6_500_000).isEmpty, "\(fixture): cleared at 6 s")
+
+ await session.shutdown()
+ }
+}
diff --git a/Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh b/Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh
index 9edace4..30a27fe 100644
--- a/Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh
+++ b/Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh
@@ -20,6 +20,8 @@
# hdr10.mp4 — HEVC Main 10, BT.2020 PQ, mastering display + content light level SEI, HDR10+ per frame
# surround51.ts — 20 s H.264 + AC-3 5.1 MPEG-TS, tone in the centre channel only
# captions.ts — H.264 TS with CEA-608 captions in A/53 SEI (1–3 s, 4–6 s; needs python3)
+# pgs.mkv, dvbsub.ts, dvdsub.mkv — the same two bitmap subtitles (1–3 s, 4–6 s)
+# as PGS, DVB and DVD subtitles on a 1920x1080 canvas (needs python3)
set -euo pipefail
FFMPEG="${FFMPEG:-ffmpeg}"
@@ -428,4 +430,77 @@ PY
fi
fi
+# Bitmap subtitles in the three formats broadcast and disc rips carry: PGS
+# (Blu-ray), DVB (European broadcast) and DVD. ffmpeg cannot render text into
+# subtitle bitmaps, but it converts between bitmap formats, so a short script
+# writes the PGS original by hand: a 64x16 bitmap (8 transparent, 24 opaque
+# white and 32 half-transparent red pixels per row) at (928, 1000) of a
+# 1920x1080 canvas shown 1–3 s, then a 48x8 white one at (936, 960) shown
+# 4–6 s, each ended by an empty presentation. A leading empty presentation at
+# 0 keeps ffmpeg from shifting the subtitles to start at 0, and
+# -fix_sub_duration turns PGS's "until the next one" into the durations the
+# DVB and DVD encoders need. The DVB conversion keeps the canvas as a display
+# definition segment.
+if [ ! -f "$OUT/pgs.mkv" ] || [ ! -f "$OUT/dvbsub.ts" ] || [ ! -f "$OUT/dvdsub.mkv" ]; then
+ if command -v python3 >/dev/null 2>&1; then
+ python3 -I - "$OUT/bitmap-subs.sup" <<'PY'
+import struct
+import sys
+
+def segment(kind, pts, payload):
+ return b'PG' + struct.pack('>IIBH', pts, 0, kind, len(payload)) + payload
+
+def display_set(pts, composition, position=None, palette=None, bitmap=None):
+ objects = 1 if bitmap else 0
+ pcs = struct.pack('>HHBHBBBB', 1920, 1080, 0x10, composition, 0x80 if bitmap else 0x00, 0, 0, objects)
+ if bitmap:
+ x, y = position
+ width, height, rle = bitmap
+ pcs += struct.pack('>HBBHH', 0, 0, 0, x, y)
+ out = segment(0x16, pts, pcs)
+ out += segment(0x17, pts, struct.pack('>BBHHHH', 1, 0, x, y, width, height))
+ pds = struct.pack('>BB', 0, 0) + b''.join(struct.pack('>BBBBB', *entry) for entry in palette)
+ out += segment(0x14, pts, pds)
+ ods = struct.pack('>HBB', 0, 0, 0xC0) + struct.pack('>I', len(rle) + 4)[1:]
+ out += segment(0x15, pts, ods + struct.pack('>HH', width, height) + rle)
+ else:
+ out = segment(0x16, pts, pcs)
+ out += segment(0x17, pts, struct.pack('>BBHHHH', 1, 0, 0, 0, 1, 1))
+ return out + segment(0x80, pts, b'')
+
+# (index, Y, Cr, Cb, alpha): transparent, opaque white, half-transparent red.
+palette = [(0, 16, 128, 128, 0), (1, 235, 128, 128, 255), (2, 63, 240, 102, 128)]
+# PGS run-length lines: 00 0n = n of colour 0, 00 8n c = n of colour c, 00 00 = end.
+first = (64, 16, bytes([0, 0x08, 0, 0x80 | 24, 1, 0, 0x80 | 32, 2, 0, 0]) * 16)
+second = (48, 8, bytes([0, 0x80 | 48, 1, 0, 0]) * 8)
+
+stream = display_set(0, 0)
+stream += display_set(90_000 * 1, 1, (928, 1000), palette, first)
+stream += display_set(90_000 * 3, 2)
+stream += display_set(90_000 * 4, 3, (936, 960), palette, second)
+stream += display_set(90_000 * 6, 4)
+open(sys.argv[1], 'wb').write(stream)
+PY
+ "$FFMPEG" -hide_banner -loglevel error -y \
+ -f lavfi -i "testsrc2=duration=8:size=320x180:rate=25" \
+ -i "$OUT/bitmap-subs.sup" -map 0:v -map 1:s \
+ -c:v libx264 -preset ultrafast -pix_fmt yuv420p -c:s copy \
+ "$OUT/pgs.mkv"
+ "$FFMPEG" -hide_banner -loglevel error -y \
+ -f lavfi -i "testsrc2=duration=8:size=320x180:rate=25" \
+ -fix_sub_duration -i "$OUT/bitmap-subs.sup" -map 0:v -map 1:s \
+ -c:v libx264 -preset ultrafast -pix_fmt yuv420p -c:s dvbsub \
+ -muxdelay 0 -f mpegts "$OUT/dvbsub.ts"
+ "$FFMPEG" -hide_banner -loglevel error -y \
+ -f lavfi -i "testsrc2=duration=8:size=320x180:rate=25" \
+ -fix_sub_duration -i "$OUT/bitmap-subs.sup" -map 0:v -map 1:s \
+ -c:v libx264 -preset ultrafast -pix_fmt yuv420p -c:s dvdsub \
+ "$OUT/dvdsub.mkv"
+ rm -f "$OUT/bitmap-subs.sup"
+ echo "generated: $OUT/pgs.mkv $OUT/dvbsub.ts $OUT/dvdsub.mkv"
+ else
+ echo "skipped: bitmap subtitle fixtures (needs python3)" >&2
+ fi
+fi
+
echo "fixtures ready in $OUT"
From d0b1464fb457ff9dca96a44bee796bd3b8ccfe11 Mon Sep 17 00:00:00 2001
From: Philipp Bischoff
Date: Mon, 12 Oct 2026 22:43:41 +0200
Subject: [PATCH 4/4] feat: lumectl, a command-line tool that reproduces stream
problems headlessly
`lumectl probe ` opens a source and prints what it carries; `lumectl
play ` drives a muted PlayerSession against it and prints every
PlayerEvent with a timestamp, the Diagnostics heartbeat with the delivery
rate, the StartupTimeline once the first frame shows, lastStallReason on
every stall and the typed EngineError (with its HTTP class) on failure.
Text or JSON Lines, optionally recorded to a file. Exit status separates
ok, open failure, playback failure and stall; Ctrl-C shuts the session
down (bounded) and prints the summary.
URLs are printed as scheme and host only unless --show-urls, and header
values never, because IPTV URLs and headers carry the account.
The logic lives in LumeCtlCore and is tested from LumeEngineCoreTests,
which already owns the fixtures and the torture server: argument parsing,
URL redaction, JSON output, exit codes, and end-to-end runs (probe, play,
refused open, silent source, live source that ends, interrupt, SIGINT on
the built binary).
---
AGENTS.md | 4 +
PLAN.md | 3 +-
Package.swift | 18 +-
README.md | 17 +
Sources/LumeCtlCore/Arguments.swift | 349 ++++++++++++++++
Sources/LumeCtlCore/Describe.swift | 267 ++++++++++++
Sources/LumeCtlCore/ExitCode.swift | 41 ++
Sources/LumeCtlCore/JSON.swift | 87 ++++
Sources/LumeCtlCore/LumeCtl.swift | 411 +++++++++++++++++++
Sources/LumeCtlCore/Reporter.swift | 66 +++
Sources/LumeCtlCore/RunControl.swift | 123 ++++++
Sources/LumeCtlCore/URLScrubber.swift | 77 ++++
Sources/lumectl/LumeCtlMain.swift | 79 ++++
Tests/LumeEngineCoreTests/LumeCtlTests.swift | 383 +++++++++++++++++
14 files changed, 1923 insertions(+), 2 deletions(-)
create mode 100644 Sources/LumeCtlCore/Arguments.swift
create mode 100644 Sources/LumeCtlCore/Describe.swift
create mode 100644 Sources/LumeCtlCore/ExitCode.swift
create mode 100644 Sources/LumeCtlCore/JSON.swift
create mode 100644 Sources/LumeCtlCore/LumeCtl.swift
create mode 100644 Sources/LumeCtlCore/Reporter.swift
create mode 100644 Sources/LumeCtlCore/RunControl.swift
create mode 100644 Sources/LumeCtlCore/URLScrubber.swift
create mode 100644 Sources/lumectl/LumeCtlMain.swift
create mode 100644 Tests/LumeEngineCoreTests/LumeCtlTests.swift
diff --git a/AGENTS.md b/AGENTS.md
index e986b78..44a7b68 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -41,8 +41,12 @@ swift test # full suite
swift test --filter TimestampUnwrapperTests # one suite
swift test --filter DecoderTests/testName # one test
swift run LumeEngineDemo # macOS demo app with diagnostics HUD
+swift run lumectl probe # tracks, codecs, live/seekable, HDR tags
+swift run lumectl play --duration 60 # headless run: events, heartbeat, startup, exit status
```
+`lumectl` reproduces a provider issue without the app: it drives a `PlayerSession` headlessly (muted) and prints what the engine reports — typed events, the `Diagnostics` heartbeat with the delivery rate, the `StartupTimeline`, `lastStallReason` on every stall, the `EngineError` (with its HTTP class) on failure. Exit status: 0 ok, 3 open failed, 4 playback failed, 5 stalled. URLs are redacted to scheme and host unless `--show-urls`, since IPTV URLs carry credentials — keep it that way. Its logic lives in `Sources/LumeCtlCore` (tested from `LumeCtlTests.swift`); `lumectl --help` lists the options. When a Lume report arrives, a `lumectl play` run against the same URL (with `--json` or `--record-events run.jsonl`) is usually the quickest way to tell an engine problem from a provider one.
+
Prerequisites:
- `BinaryDependencies/FFmpeg.xcframework` must exist before anything compiles. Build it once (or after bumping `build/versions.json`):
```bash
diff --git a/PLAN.md b/PLAN.md
index 1acf292..31a4e99 100644
--- a/PLAN.md
+++ b/PLAN.md
@@ -116,7 +116,8 @@ LumeEngine/
│ ├── LumeEngineRender/ # SystemRenderer, MetalRenderer, shaders
│ ├── LumeEngineSubtitles/ # libass bridge, parsers, cue model, styling
│ ├── LumeEngineMedia/ # Now Playing, PiP, audio session, route handling
-│ └── LumeEngine/ # public facade: LumePlayer, LumePlayerView, config, events
+│ ├── LumeEngine/ # public facade: LumePlayer, LumePlayerView, config, events
+│ └── LumeCtlCore/, lumectl/ # macOS triage CLI: headless session against a URL, events + diagnostics
├── BinaryDependencies/ # generated Package targets for ffmpeg/libass/dav1d xcframeworks
├── build/ # FFmpeg build pipeline (scripts, patches, configs)
├── DemoApp/ # multiplatform SwiftUI demo (streams menu, diagnostics HUD)
diff --git a/Package.swift b/Package.swift
index a2a7966..392cfee 100644
--- a/Package.swift
+++ b/Package.swift
@@ -85,6 +85,9 @@ let package = Package(
// namespace), so apps can also link engines embedding other FFmpeg builds
// without symbol collisions.
.library(name: "LumeEngine", type: .dynamic, targets: ["LumeEngine"]),
+ // macOS triage tool: `swift run lumectl --help`. Drives a session
+ // headlessly against a URL; nothing a consumer app links.
+ .executable(name: "lumectl", targets: ["lumectl"]),
],
targets: [
// FFmpeg 9.0.x static libraries: released artifact, or a local build when
@@ -116,11 +119,24 @@ let package = Package(
path: "DemoApp/Sources"
),
+ // lumectl's logic (argument parsing, URL redaction, the headless run,
+ // output), kept out of the executable so the tests can reach it.
+ .target(
+ name: "LumeCtlCore",
+ dependencies: ["LumeEngineCore"]
+ ),
+ .executableTarget(
+ name: "lumectl",
+ dependencies: ["LumeCtlCore"]
+ ),
+
.testTarget(
name: "LumeEngineCoreTests",
// CFFmpeg directly: some tests build FFmpeg structs (e.g. Dolby
// Vision metadata) to exercise the engine's readers of them.
- dependencies: ["LumeEngineCore", "LumeEngine", "CFFmpeg"],
+ // LumeCtlCore and lumectl: the tool's tests reuse the fixtures and
+ // the torture server, and the Ctrl-C test runs the built binary.
+ dependencies: ["LumeEngineCore", "LumeEngine", "CFFmpeg", "LumeCtlCore", "lumectl"],
exclude: ["Fixtures/generate-fixtures.sh"]
),
],
diff --git a/README.md b/README.md
index 849cb14..f536497 100644
--- a/README.md
+++ b/README.md
@@ -101,10 +101,27 @@ swift run LumeEngineDemo # macOS demo with diagnostics H
Testing needs a host `ffmpeg` CLI (`brew install ffmpeg`) to generate fixtures on first run — a build-time tool, unrelated to the FFmpeg the engine links. The full 10-slice matrix (`build/scripts/platforms.sh`) is only needed for release binaries, which CI builds on tags.
+## Debugging streams with lumectl
+
+`lumectl` (macOS) plays a URL headlessly through the engine and prints what it reports, so a stream problem — a slow start, stalls, a reconnect that replays media, timestamp seams, a hardware decoder falling back, an HTTP error — can be reproduced from a terminal without the app:
+
+```bash
+swift run lumectl probe 'http://provider.example:8080/live/user/pass/1234.ts'
+swift run lumectl play 'http://provider.example:8080/live/user/pass/1234.ts' --duration 60
+swift run lumectl play stream.m3u8 --json --record-events run.jsonl --interval 1
+```
+
+`probe` prints the container, duration, start time, live/seekable flags and every track (codec, language, resolution, frame rate, bit depth, HDR transfer and colour tags). `play` runs a muted session and prints every `PlayerEvent` with a timestamp, a diagnostics heartbeat (queue depths, renderer health, delivery rate, timeline seams, replayed packets), the startup timeline once the first frame shows, the stalled stage on every stall, and the typed error — with its HTTP status class — on failure. `--json` switches stdout to JSON Lines; `--record-events FILE` keeps a JSON Lines copy next to the readable output.
+
+The exit status tells the outcome apart for scripted repro loops: `0` ok, `3` the source never opened, `4` playback failed, `5` stalled (without failing), `2` bad arguments. Ctrl-C ends a run cleanly with a summary.
+
+URLs are printed as scheme and host only (`http://provider.example:8080/[redacted].ts`), because IPTV URLs carry the account; header values are never printed. Pass `--show-urls` to see them in full. `lumectl --help` lists every option: start position, mid-stream seek, software decoding, preferred audio languages, a single lane, custom headers, user agent, referer, reconnect and timeouts.
+
## Layout
- `Sources/LumeEngineCore` — demux, decode, render, session, subtitles, media integration (no UI)
- `Sources/LumeEngine` — public facade: `LumePlayer`, `LumePlayerView`
+- `Sources/lumectl`, `Sources/LumeCtlCore` — the `lumectl` command-line tool
- `build/` — FFmpeg build pipeline (versions manifest, scripts, patches)
- `DemoApp/` — macOS demo with diagnostics HUD
- `Tests/` — including a fault-injecting HTTP torture server
diff --git a/Sources/LumeCtlCore/Arguments.swift b/Sources/LumeCtlCore/Arguments.swift
new file mode 100644
index 0000000..44c99b4
--- /dev/null
+++ b/Sources/LumeCtlCore/Arguments.swift
@@ -0,0 +1,349 @@
+import Foundation
+import LumeEngineCore
+
+/// What the command line asked for. Parsing is pure (`Arguments.parse`),
+/// so every flag is unit-tested without opening a stream.
+public enum Command: Equatable, Sendable {
+ case help
+ case version
+ case probe(ProbeOptions)
+ case play(PlayOptions)
+}
+
+/// How to reach the source: everything `DemuxerOptions` exposes that a
+/// provider issue tends to hinge on.
+public struct SourceOptions: Equatable, Sendable {
+ public var url: String
+ /// In command-line order; a repeated name keeps the last value.
+ public var headers: [Header] = []
+ public var userAgent: String?
+ public var referer: String?
+ public var reconnect = true
+ /// Seconds; nil keeps the engine default.
+ public var ioTimeout: Double?
+ /// Seconds; nil keeps the engine default.
+ public var openTimeout: Double?
+
+ public struct Header: Equatable, Sendable {
+ public var name: String
+ public var value: String
+ }
+
+ public init(url: String) {
+ self.url = url
+ }
+
+ func apply(to options: inout DemuxerOptions) {
+ for header in headers {
+ options.httpHeaders[header.name] = header.value
+ }
+ if let userAgent { options.userAgent = userAgent }
+ if let referer { options.referer = referer }
+ options.enableReconnect = reconnect
+ if let ioTimeout { options.ioTimeout = Int64((ioTimeout * 1_000_000).rounded()) }
+ if let openTimeout { options.openTimeout = openTimeout }
+ }
+}
+
+public struct OutputOptions: Equatable, Sendable {
+ /// JSON Lines on stdout instead of text.
+ public var json = false
+ /// Print URLs as given instead of redacted.
+ public var showURLs = false
+ /// Also write every record as JSON Lines to this file.
+ public var recordPath: String?
+
+ public init() {}
+}
+
+public struct ProbeOptions: Equatable, Sendable {
+ public var source: SourceOptions
+ public var output = OutputOptions()
+
+ public init(source: SourceOptions) {
+ self.source = source
+ }
+
+ public var configuration: PlayerConfiguration {
+ var configuration = PlayerConfiguration()
+ configuration.muted = true
+ source.apply(to: &configuration.demuxer)
+ return configuration
+ }
+}
+
+public struct PlayOptions: Equatable, Sendable {
+ public var source: SourceOptions
+ public var output = OutputOptions()
+ /// Wall-clock seconds to play from `play()`; nil plays to the end (or
+ /// until Ctrl-C, for a live source).
+ public var duration: Double?
+ /// Seek here (seconds) once the first frame shows — a mid-stream seek,
+ /// as opposed to `start`.
+ public var seek: Double?
+ /// `PlayerConfiguration.startPosition`: the resume path, which seeks
+ /// before the first read.
+ public var start: Double?
+ public var software = false
+ public var audioLanguages: [String] = []
+ public var noAudio = false
+ public var noVideo = false
+ /// Seconds between diagnostics heartbeats; 0 turns them off.
+ public var interval: Double = 2
+ /// Seconds; nil keeps the engine default.
+ public var stallThreshold: Double?
+
+ public init(source: SourceOptions) {
+ self.source = source
+ }
+
+ public var configuration: PlayerConfiguration {
+ var configuration = PlayerConfiguration()
+ // Nothing should come out of the speakers while triaging; the audio
+ // lane still decodes and renders, so its timing is the real one.
+ configuration.muted = true
+ source.apply(to: &configuration.demuxer)
+ if let start { configuration.startPosition = start }
+ if software { configuration.hardwareDecode = .software }
+ configuration.preferredAudioLanguages = audioLanguages
+ configuration.enableAudio = !noAudio
+ configuration.enableVideo = !noVideo
+ if let stallThreshold { configuration.stallThreshold = stallThreshold }
+ return configuration
+ }
+}
+
+public struct UsageError: Error, Equatable, CustomStringConvertible {
+ public let message: String
+ public var description: String { message }
+
+ init(_ message: String) {
+ self.message = message
+ }
+}
+
+public enum Arguments {
+ /// Parses the arguments after the executable name.
+ public static func parse(_ arguments: [String]) throws(UsageError) -> Command {
+ guard let first = arguments.first else { return .help }
+ // `lumectl play --help` asks for help, not for a URL.
+ let options = arguments.prefix(while: { $0 != "--" })
+ if options.contains("-h") || options.contains("--help") { return .help }
+ switch first {
+ case "help":
+ return .help
+ case "--version", "version":
+ return .version
+ case "probe":
+ var reader = Reader(Array(arguments.dropFirst()))
+ return .probe(try parseProbe(&reader))
+ case "play":
+ var reader = Reader(Array(arguments.dropFirst()))
+ return .play(try parsePlay(&reader))
+ default:
+ throw UsageError("unknown command '\(first)' (expected probe or play)")
+ }
+ }
+
+ private static func parseProbe(_ reader: inout Reader) throws(UsageError) -> ProbeOptions {
+ var source = SourceOptions(url: "")
+ var output = OutputOptions()
+ var url: String?
+ while let argument = try reader.next() {
+ switch argument {
+ case .positional(let value):
+ try setURL(&url, value)
+ case .option(let name, let inline):
+ if try parseSourceOption(name, inline, &source, &reader) { continue }
+ if try parseOutputOption(name, inline, &output, &reader) { continue }
+ throw UsageError("unknown option '--\(name)' for probe")
+ }
+ }
+ guard let url else { throw UsageError("probe needs a URL or file path") }
+ source.url = url
+ var options = ProbeOptions(source: source)
+ options.output = output
+ return options
+ }
+
+ private static func parsePlay(_ reader: inout Reader) throws(UsageError) -> PlayOptions {
+ var options = PlayOptions(source: SourceOptions(url: ""))
+ var url: String?
+ while let argument = try reader.next() {
+ switch argument {
+ case .positional(let value):
+ try setURL(&url, value)
+ case .option(let name, let inline):
+ if try parseSourceOption(name, inline, &options.source, &reader) { continue }
+ if try parseOutputOption(name, inline, &options.output, &reader) { continue }
+ switch name {
+ case "duration":
+ options.duration = try seconds(name, reader.value(for: name, inline), allowZero: false)
+ case "seek":
+ options.seek = try seconds(name, reader.value(for: name, inline))
+ case "start":
+ options.start = try seconds(name, reader.value(for: name, inline))
+ case "interval":
+ options.interval = try seconds(name, reader.value(for: name, inline))
+ case "stall-threshold":
+ options.stallThreshold = try seconds(name, reader.value(for: name, inline), allowZero: false)
+ case "audio-lang":
+ options.audioLanguages = try languages(reader.value(for: name, inline))
+ case "software":
+ try noValue(name, inline)
+ options.software = true
+ case "no-audio":
+ try noValue(name, inline)
+ options.noAudio = true
+ case "no-video":
+ try noValue(name, inline)
+ options.noVideo = true
+ default:
+ throw UsageError("unknown option '--\(name)' for play")
+ }
+ }
+ }
+ guard let url else { throw UsageError("play needs a URL or file path") }
+ guard !(options.noAudio && options.noVideo) else {
+ throw UsageError("--no-audio and --no-video together leave nothing to play")
+ }
+ options.source.url = url
+ return options
+ }
+
+ /// Options both commands share. Returns false when `name` is not one.
+ private static func parseSourceOption(
+ _ name: String, _ inline: String?, _ source: inout SourceOptions, _ reader: inout Reader
+ ) throws(UsageError) -> Bool {
+ switch name {
+ case "header":
+ source.headers.append(try header(reader.value(for: name, inline)))
+ case "user-agent":
+ source.userAgent = try nonEmpty(name, reader.value(for: name, inline))
+ case "referer":
+ source.referer = try nonEmpty(name, reader.value(for: name, inline))
+ case "no-reconnect":
+ try noValue(name, inline)
+ source.reconnect = false
+ case "io-timeout":
+ source.ioTimeout = try seconds(name, reader.value(for: name, inline), allowZero: false)
+ case "open-timeout":
+ source.openTimeout = try seconds(name, reader.value(for: name, inline), allowZero: false)
+ default:
+ return false
+ }
+ return true
+ }
+
+ private static func parseOutputOption(
+ _ name: String, _ inline: String?, _ output: inout OutputOptions, _ reader: inout Reader
+ ) throws(UsageError) -> Bool {
+ switch name {
+ case "json":
+ try noValue(name, inline)
+ output.json = true
+ case "show-urls":
+ try noValue(name, inline)
+ output.showURLs = true
+ case "record-events":
+ output.recordPath = try nonEmpty(name, reader.value(for: name, inline))
+ default:
+ return false
+ }
+ return true
+ }
+
+ private static func setURL(_ url: inout String?, _ value: String) throws(UsageError) {
+ guard url == nil else { throw UsageError("unexpected extra argument '\(value)'") }
+ guard !value.isEmpty else { throw UsageError("the URL is empty") }
+ url = value
+ }
+
+ private static func seconds(_ name: String, _ text: String, allowZero: Bool = true) throws(UsageError) -> Double {
+ guard let value = Double(text), value.isFinite, value >= 0, allowZero || value > 0 else {
+ let expected = allowZero ? "a number of seconds ≥ 0" : "a number of seconds > 0"
+ throw UsageError("--\(name) expects \(expected), got '\(text)'")
+ }
+ return value
+ }
+
+ private static func header(_ text: String) throws(UsageError) -> SourceOptions.Header {
+ guard let colon = text.firstIndex(of: ":") else {
+ throw UsageError("--header expects 'Name: value', got '\(text)'")
+ }
+ let name = text[.. [String] {
+ let tags = text.split(separator: ",", omittingEmptySubsequences: false).map { $0.trimmingCharacters(in: .whitespaces).lowercased() }
+ guard !tags.isEmpty, tags.allSatisfy({ !$0.isEmpty }) else {
+ throw UsageError("--audio-lang expects a comma-separated list such as 'de,en', got '\(text)'")
+ }
+ return tags
+ }
+
+ private static func nonEmpty(_ name: String, _ text: String) throws(UsageError) -> String {
+ guard !text.isEmpty else { throw UsageError("--\(name) expects a value") }
+ return text
+ }
+
+ private static func noValue(_ name: String, _ inline: String?) throws(UsageError) {
+ if inline != nil { throw UsageError("--\(name) takes no value") }
+ }
+}
+
+/// Splits arguments into positionals and `--name` / `--name=value` options;
+/// everything after `--` is positional (a URL that starts with a dash).
+private struct Reader {
+ enum Argument {
+ case positional(String)
+ case option(name: String, inline: String?)
+ }
+
+ private var arguments: [String]
+ private var index = 0
+ private var positionalOnly = false
+
+ init(_ arguments: [String]) {
+ self.arguments = arguments
+ }
+
+ mutating func next() throws(UsageError) -> Argument? {
+ guard index < arguments.count else { return nil }
+ let argument = arguments[index]
+ index += 1
+ if positionalOnly { return .positional(argument) }
+ if argument == "--" {
+ positionalOnly = true
+ return try next()
+ }
+ guard argument.hasPrefix("--"), argument.count > 2 else {
+ if argument.hasPrefix("-"), argument.count > 1 {
+ throw UsageError("unknown option '\(argument)' (options are spelled --name)")
+ }
+ return .positional(argument)
+ }
+ let body = argument.dropFirst(2)
+ if let equals = body.firstIndex(of: "=") {
+ return .option(name: String(body[.. String {
+ if let inline { return inline }
+ guard index < arguments.count else { throw UsageError("--\(name) expects a value") }
+ let value = arguments[index]
+ index += 1
+ return value
+ }
+}
diff --git a/Sources/LumeCtlCore/Describe.swift b/Sources/LumeCtlCore/Describe.swift
new file mode 100644
index 0000000..207c7b4
--- /dev/null
+++ b/Sources/LumeCtlCore/Describe.swift
@@ -0,0 +1,267 @@
+import Foundation
+import LumeEngineCore
+
+/// Text and JSON renderings of the engine's public types. Anything that can
+/// carry a URL goes through `redact` first.
+struct Describe {
+ /// Redacts URLs inside free text unless the user asked to see them.
+ let showURLs: Bool
+
+ func url(_ url: String) -> String {
+ showURLs ? url : URLScrubber.redact(url)
+ }
+
+ func text(_ text: String) -> String {
+ showURLs ? text : URLScrubber.redactText(text)
+ }
+
+ // MARK: Media
+
+ func media(_ info: MediaInfo) -> (text: String, json: JSON) {
+ var lines: [String] = []
+ let duration = info.duration.map { String(format: "%.3f s", seconds($0)) } ?? "unknown (live)"
+ lines.append("format \(info.formatName)")
+ lines.append("duration \(duration)")
+ lines.append(String(format: "start %.3f s", seconds(info.startTime)))
+ lines.append("live \(info.isLive ? "yes" : "no") seekable \(info.isSeekable ? "yes" : "no")")
+ if info.bitrate >= 1_000_000 {
+ lines.append(String(format: "bitrate %.2f Mbit/s", Double(info.bitrate) / 1_000_000))
+ } else if info.bitrate >= 1000 {
+ lines.append(String(format: "bitrate %.0f kbit/s", Double(info.bitrate) / 1000))
+ } else if info.bitrate > 0 {
+ lines.append("bitrate \(info.bitrate) bit/s")
+ }
+ lines.append("tracks")
+ for track in info.tracks {
+ lines.append(" " + trackLine(track))
+ }
+ if !info.chapters.isEmpty {
+ lines.append("chapters")
+ for chapter in info.chapters {
+ lines.append(String(
+ format: " %.3f–%.3f s %@",
+ seconds(chapter.start), seconds(chapter.end), text(chapter.title ?? "")
+ ))
+ }
+ }
+ if !info.metadata.isEmpty {
+ lines.append("metadata")
+ for key in info.metadata.keys.sorted() {
+ lines.append(" \(key): \(text(info.metadata[key] ?? ""))")
+ }
+ }
+
+ let json = JSON.object([
+ ("format", .string(info.formatName)),
+ ("duration", .optional(info.duration.map(seconds))),
+ ("startTime", .double(seconds(info.startTime))),
+ ("live", .bool(info.isLive)),
+ ("seekable", .bool(info.isSeekable)),
+ ("bitrate", .int(info.bitrate)),
+ ("tracks", .array(info.tracks.map(trackJSON))),
+ ("chapters", .array(info.chapters.map { chapter in
+ .object([
+ ("title", .optional(chapter.title.map(text))),
+ ("start", .double(seconds(chapter.start))),
+ ("end", .double(seconds(chapter.end))),
+ ])
+ })),
+ ("metadata", .object(info.metadata.keys.sorted().map { ($0, .string(text(info.metadata[$0] ?? ""))) })),
+ ])
+ return (lines.joined(separator: "\n"), json)
+ }
+
+ private func trackLine(_ track: TrackInfo) -> String {
+ var parts = ["#\(track.index)", track.kind.rawValue, track.codecName]
+ if let language = track.language { parts.append(language) }
+ if let video = track.video {
+ parts.append("\(video.width)x\(video.height)")
+ if video.fps > 0 { parts.append(String(format: "%.6g fps", video.fps)) }
+ if video.bitDepth > 0 { parts.append("\(video.bitDepth)-bit") }
+ if let pixelFormat = video.pixelFormatName { parts.append(pixelFormat) }
+ parts.append(colour(video))
+ if video.rotation != 0 { parts.append("rotated \(video.rotation)°") }
+ }
+ if let audio = track.audio {
+ parts.append("\(audio.channels) ch")
+ parts.append("\(audio.sampleRate) Hz")
+ }
+ if track.bitrate > 0 {
+ parts.append(String(format: "%.0f kbit/s", Double(track.bitrate) / 1000))
+ }
+ if let title = track.title, !title.isEmpty { parts.append("\"\(text(title))\"") }
+ if track.isDefault { parts.append("[default]") }
+ if track.isForced { parts.append("[forced]") }
+ return parts.joined(separator: " ")
+ }
+
+ /// `HDR PQ bt2020/bt2020nc full` — enough to tell HDR10 from HLG from
+ /// SDR, and to spot an IPT base layer (Dolby Vision profile 5) that the
+ /// matrix tag gives away.
+ private func colour(_ video: TrackInfo.Video) -> String {
+ let range = video.colorRangeFull ? "full" : "limited"
+ let unspecified: UInt32 = 2
+ if video.colorPrimaries == unspecified, video.colorTransfer == unspecified, video.colorSpace == unspecified {
+ return "\(video.isHDR ? "HDR" : "SDR") untagged \(range)"
+ }
+ let dynamicRange: String
+ switch ColourNames.transfer(video.colorTransfer) {
+ case "pq": dynamicRange = "HDR PQ"
+ case "hlg": dynamicRange = "HDR HLG"
+ default: dynamicRange = video.isHDR ? "HDR" : "SDR"
+ }
+ return "\(dynamicRange) \(ColourNames.primaries(video.colorPrimaries))/\(ColourNames.matrix(video.colorSpace))"
+ + "/\(ColourNames.transfer(video.colorTransfer)) \(range)"
+ }
+
+ private func trackJSON(_ track: TrackInfo) -> JSON {
+ var members: [(String, JSON)] = [
+ ("index", .int(Int64(track.index))),
+ ("kind", .string(track.kind.rawValue)),
+ ("codec", .string(track.codecName)),
+ ("language", .optional(track.language)),
+ ("title", .optional(track.title.map(text))),
+ ("default", .bool(track.isDefault)),
+ ("forced", .bool(track.isForced)),
+ ("bitrate", .int(track.bitrate)),
+ ]
+ if let video = track.video {
+ members.append(("video", .object([
+ ("width", .int(Int64(video.width))),
+ ("height", .int(Int64(video.height))),
+ ("fps", .double(video.fps)),
+ ("bitDepth", .int(Int64(video.bitDepth))),
+ ("pixelFormat", .optional(video.pixelFormatName)),
+ ("rotation", .int(Int64(video.rotation))),
+ ("hdr", .bool(video.isHDR)),
+ ("primaries", .string(ColourNames.primaries(video.colorPrimaries))),
+ ("transfer", .string(ColourNames.transfer(video.colorTransfer))),
+ ("matrix", .string(ColourNames.matrix(video.colorSpace))),
+ ("fullRange", .bool(video.colorRangeFull)),
+ ])))
+ }
+ if let audio = track.audio {
+ members.append(("audio", .object([
+ ("channels", .int(Int64(audio.channels))),
+ ("sampleRate", .int(Int64(audio.sampleRate))),
+ ])))
+ }
+ return .object(members)
+ }
+
+ // MARK: Errors
+
+ func error(_ error: EngineError) -> (text: String, json: JSON) {
+ var detail: [String] = []
+ if let ffmpegCode = error.ffmpegCode { detail.append("ffmpeg \(ffmpegCode)") }
+ if let http = error.httpFailure { detail.append("http \(http.rawValue)") }
+ let suffix = detail.isEmpty ? "" : " (\(detail.joined(separator: ", ")))"
+ let message = text(error.message)
+ let json = JSON.object([
+ ("code", .string(error.code.rawValue)),
+ ("ffmpegCode", error.ffmpegCode.map { .int(Int64($0)) } ?? .null),
+ ("http", .optional(error.httpFailure?.rawValue)),
+ ("message", .string(message)),
+ ])
+ return ("\(error.code.rawValue)\(suffix): \(message)", json)
+ }
+
+ // MARK: Session state
+
+ func startup(_ startup: StartupTimeline) -> JSON {
+ .object([
+ ("opened", .optional(startup.opened)),
+ ("firstPacket", .optional(startup.firstPacket)),
+ ("firstVideoFrame", .optional(startup.firstVideoFrame)),
+ ("firstAudioFrame", .optional(startup.firstAudioFrame)),
+ ("firstFrameShown", .optional(startup.firstFrameShown)),
+ ])
+ }
+
+ func diagnostics(_ diagnostics: PlayerSession.Diagnostics, inputMbps: Double?) -> (text: String, json: JSON) {
+ func lane(_ queue: (count: Int, seconds: Double)?) -> JSON {
+ guard let queue else { return .null }
+ return .object([("count", .int(Int64(queue.count))), ("seconds", .double(queue.seconds))])
+ }
+ let health = diagnostics.rendererHealth
+ var line = text(diagnostics.description)
+ if let inputMbps { line += String(format: " in=%.2fMbit/s", inputMbps) }
+ let json = JSON.object([
+ ("state", .string(diagnostics.state)),
+ ("playhead", .double(diagnostics.playheadSeconds)),
+ ("videoFrames", lane(diagnostics.videoQueue)),
+ ("videoPackets", lane(diagnostics.videoPacketQueue)),
+ ("videoLead", .double(diagnostics.videoLeadSeconds)),
+ ("audioFrames", lane(diagnostics.audioQueue)),
+ ("audioPackets", lane(diagnostics.audioPacketQueue)),
+ ("audioLead", .double(diagnostics.audioLeadSeconds)),
+ ("renderers", .object([
+ ("video", .int(Int64(health.videoStatus))),
+ ("audio", .int(Int64(health.audioStatus))),
+ ("videoError", .optional(health.videoError.map(text))),
+ ("audioError", .optional(health.audioError.map(text))),
+ ])),
+ ("demuxEOF", .bool(diagnostics.demuxAtEOF)),
+ ("sourceLost", .bool(diagnostics.sourceLost)),
+ ("deliveredBytes", .int(diagnostics.deliveredBytes)),
+ ("inputMbps", .optional(inputMbps)),
+ ("timelineSeams", .int(Int64(diagnostics.timelineSeams))),
+ ("replayedPackets", .int(Int64(diagnostics.replayedPackets))),
+ ("lastStallReason", .optional(diagnostics.lastStallReason?.rawValue)),
+ ("startup", startup(diagnostics.startup)),
+ ])
+ return (line, json)
+ }
+
+ private func seconds(_ microseconds: Int64) -> Double {
+ Double(microseconds) / 1_000_000
+ }
+}
+
+/// Names for the ITU-T H.273 code points `TrackInfo.Video` carries raw.
+enum ColourNames {
+ static func primaries(_ value: UInt32) -> String {
+ switch value {
+ case 1: "bt709"
+ case 2: "unspecified"
+ case 5: "bt470bg"
+ case 6: "smpte170m"
+ case 9: "bt2020"
+ case 11: "dci-p3"
+ case 12: "display-p3"
+ default: "pri\(value)"
+ }
+ }
+
+ static func transfer(_ value: UInt32) -> String {
+ switch value {
+ case 1: "bt709"
+ case 2: "unspecified"
+ case 6: "smpte170m"
+ case 8: "linear"
+ case 13: "srgb"
+ case 14: "bt2020-10"
+ case 15: "bt2020-12"
+ case 16: "pq"
+ case 18: "hlg"
+ default: "trc\(value)"
+ }
+ }
+
+ static func matrix(_ value: UInt32) -> String {
+ switch value {
+ case 0: "rgb"
+ case 1: "bt709"
+ case 2: "unspecified"
+ case 5: "bt470bg"
+ case 6: "smpte170m"
+ case 9: "bt2020nc"
+ case 10: "bt2020c"
+ case 14: "ictcp"
+ // Dolby Vision's IPT base layer (profiles 5, 10.0, 20) when the
+ // stream tags it; many such encodes leave the matrix unspecified.
+ case 15: "ipt-c2"
+ default: "spc\(value)"
+ }
+ }
+}
diff --git a/Sources/LumeCtlCore/ExitCode.swift b/Sources/LumeCtlCore/ExitCode.swift
new file mode 100644
index 0000000..d87da64
--- /dev/null
+++ b/Sources/LumeCtlCore/ExitCode.swift
@@ -0,0 +1,41 @@
+/// lumectl's exit status: scripts reproducing a provider issue in a loop
+/// branch on these, so each failure class gets its own code.
+public enum ExitCode: Int32, Sendable {
+ /// Played for the requested time, to the end, or until Ctrl-C, with no
+ /// stall and no failure. Probe: the source opened.
+ case success = 0
+ /// lumectl itself could not run (e.g. the record file).
+ case toolFailure = 1
+ /// Bad command line.
+ case usage = 2
+ /// The source never opened (DNS, connect, HTTP status, probe, timeout).
+ case openFailed = 3
+ /// The session opened and then failed.
+ case playbackFailed = 4
+ /// The session stalled at least once and did not fail.
+ case stalled = 5
+ /// Ctrl-C before the source opened: no verdict to report.
+ case interrupted = 130
+
+ /// The verdict of a playback that got past `open`: a failure outranks a
+ /// stall (the stall usually preceded it), and Ctrl-C is how a live run
+ /// ends, so it is not a verdict of its own.
+ static func playback(failed: Bool, stalls: Int) -> ExitCode {
+ if failed { return .playbackFailed }
+ if stalls > 0 { return .stalled }
+ return .success
+ }
+
+ /// Short name used in the summary record.
+ var outcome: String {
+ switch self {
+ case .success: "ok"
+ case .toolFailure: "tool-failure"
+ case .usage: "usage"
+ case .openFailed: "open-failed"
+ case .playbackFailed: "playback-failed"
+ case .stalled: "stalled"
+ case .interrupted: "interrupted"
+ }
+ }
+}
diff --git a/Sources/LumeCtlCore/JSON.swift b/Sources/LumeCtlCore/JSON.swift
new file mode 100644
index 0000000..b1f6682
--- /dev/null
+++ b/Sources/LumeCtlCore/JSON.swift
@@ -0,0 +1,87 @@
+import Foundation
+
+/// A JSON value with ordered object keys.
+///
+/// lumectl's JSON Lines are read by people as often as by `jq`, and a record
+/// that starts with `t` and `type` scans far better than one sorted
+/// alphabetically — which is all `JSONSerialization` offers.
+public enum JSON: Equatable, Sendable {
+ case null
+ case bool(Bool)
+ case int(Int64)
+ case double(Double)
+ case string(String)
+ case array([JSON])
+ case object([(String, JSON)])
+
+ public static func == (lhs: JSON, rhs: JSON) -> Bool {
+ lhs.encoded == rhs.encoded
+ }
+
+ /// Compact single-line encoding (one record per line).
+ public var encoded: String {
+ var output = ""
+ write(to: &output)
+ return output
+ }
+
+ private func write(to output: inout String) {
+ switch self {
+ case .null:
+ output += "null"
+ case .bool(let value):
+ output += value ? "true" : "false"
+ case .int(let value):
+ output += String(value)
+ case .double(let value):
+ // JSON has no NaN or infinity; a missing measurement is null.
+ if value.isFinite {
+ output += String(format: "%.3f", value)
+ } else {
+ output += "null"
+ }
+ case .string(let value):
+ Self.writeString(value, to: &output)
+ case .array(let values):
+ output += "["
+ for (offset, value) in values.enumerated() {
+ if offset > 0 { output += "," }
+ value.write(to: &output)
+ }
+ output += "]"
+ case .object(let members):
+ output += "{"
+ for (offset, member) in members.enumerated() {
+ if offset > 0 { output += "," }
+ Self.writeString(member.0, to: &output)
+ output += ":"
+ member.1.write(to: &output)
+ }
+ output += "}"
+ }
+ }
+
+ private static func writeString(_ value: String, to output: inout String) {
+ output += "\""
+ for scalar in value.unicodeScalars {
+ switch scalar {
+ case "\"": output += "\\\""
+ case "\\": output += "\\\\"
+ case "\n": output += "\\n"
+ case "\r": output += "\\r"
+ case "\t": output += "\\t"
+ case _ where scalar.value < 0x20:
+ output += String(format: "\\u%04x", scalar.value)
+ default:
+ output.unicodeScalars.append(scalar)
+ }
+ }
+ output += "\""
+ }
+}
+
+extension JSON {
+ /// `nil` becomes `null`, so optional fields keep their key.
+ static func optional(_ value: Double?) -> JSON { value.map(JSON.double) ?? .null }
+ static func optional(_ value: String?) -> JSON { value.map(JSON.string) ?? .null }
+}
diff --git a/Sources/LumeCtlCore/LumeCtl.swift b/Sources/LumeCtlCore/LumeCtl.swift
new file mode 100644
index 0000000..4db49df
--- /dev/null
+++ b/Sources/LumeCtlCore/LumeCtl.swift
@@ -0,0 +1,411 @@
+import Foundation
+import LumeEngineCore
+import Synchronization
+
+/// lumectl: drives a `PlayerSession` headlessly against one URL, so a
+/// provider issue (startup stall, reconnect, replay, seam, decoder downgrade,
+/// HTTP error) can be reproduced from a terminal instead of the app.
+///
+/// It only observes. Like the app, it uses the session's public surface —
+/// events, `diagnostics`, `startup`, `lastStallReason` — and never retries:
+/// what the engine reports is what gets printed.
+public enum LumeCtl {
+ public static let usage = """
+ usage: lumectl probe [source options] [output options]
+ lumectl play [play options] [source options] [output options]
+
+ probe open the source, print what it carries, close it
+ play play it headlessly (muted) and print every event, a diagnostics
+ heartbeat, the startup timeline and the outcome
+
+ play options:
+ --duration S stop after S seconds of playback (default: to the
+ end, or Ctrl-C for a live source)
+ --start S open at S seconds (the resume path: seeks before
+ the first read)
+ --seek S seek to S seconds once the first frame shows
+ --interval S heartbeat every S seconds, 0 for none (default 2)
+ --software decode video in software (no VideoToolbox)
+ --audio-lang de,en preferred audio languages, in order
+ --no-audio video lane only
+ --no-video audio lane only
+ --stall-threshold S seconds without progress before a stall
+
+ source options:
+ --header 'Name: value' extra HTTP header (repeatable)
+ --user-agent UA
+ --referer URL
+ --no-reconnect disable FFmpeg's HTTP auto-reconnect
+ --io-timeout S blocking I/O timeout (default 15)
+ --open-timeout S deadline for open + probe (default 30)
+
+ output options:
+ --json JSON Lines on stdout instead of text
+ --record-events FILE also write every record to FILE as JSON Lines
+ --show-urls print URLs as given (default: scheme and host
+ only, since IPTV URLs carry credentials)
+
+ exit status:
+ 0 ok 1 lumectl failed 2 usage 3 open failed
+ 4 playback failed 5 stalled (and did not fail) 130 Ctrl-C before open
+
+ Ctrl-C ends a run cleanly (the session is shut down and the summary
+ printed); a second Ctrl-C exits at once.
+ """
+
+ /// Runs a parsed command to completion.
+ ///
+ /// - Parameter interrupts: yields once per Ctrl-C. The first one ends the
+ /// run the way `--duration` would; handling a second is the caller's.
+ public static func run(
+ _ command: Command,
+ interrupts: AsyncStream,
+ stdout: @escaping Reporter.Sink,
+ stderr: @escaping Reporter.Sink
+ ) async -> ExitCode {
+ switch command {
+ case .help:
+ stdout(usage)
+ return .success
+ case .version:
+ stdout("lumectl (LumeEngine) — \(FFmpegRuntime.versions)")
+ return .success
+ case .probe(let options):
+ guard let reporter = makeReporter(options.output, stdout: stdout, stderr: stderr) else {
+ return .toolFailure
+ }
+ defer { reporter.close() }
+ return await Run(
+ source: options.source, output: options.output,
+ configuration: options.configuration, reporter: reporter
+ ).probe(interrupts: interrupts)
+ case .play(let options):
+ guard let reporter = makeReporter(options.output, stdout: stdout, stderr: stderr) else {
+ return .toolFailure
+ }
+ defer { reporter.close() }
+ return await Run(
+ source: options.source, output: options.output,
+ configuration: options.configuration, reporter: reporter
+ ).play(options, interrupts: interrupts)
+ }
+ }
+
+ private static func makeReporter(
+ _ output: OutputOptions, stdout: @escaping Reporter.Sink, stderr: Reporter.Sink
+ ) -> Reporter? {
+ do {
+ return try Reporter(output: output, stdout: stdout)
+ } catch {
+ stderr("lumectl: \(error)")
+ return nil
+ }
+ }
+}
+
+/// One probe or play run.
+private struct Run {
+ let source: SourceOptions
+ let output: OutputOptions
+ let configuration: PlayerConfiguration
+ let reporter: Reporter
+
+ private var describe: Describe { Describe(showURLs: output.showURLs) }
+
+ // MARK: probe
+
+ func probe(interrupts: AsyncStream) async -> ExitCode {
+ reportStart("probe", extra: [])
+ let session = PlayerSession(configuration: configuration)
+ let control = RunControl()
+ let forwarder = forward(interrupts, to: control)
+ defer { forwarder.cancel() }
+
+ let result = await open(session, control: control)
+ control.stop(.ended)
+ await shutdown(session)
+ switch result {
+ case .opened(let info):
+ let media = describe.media(info)
+ reporter.emit("media", text: media.text, fields: [("media", media.json)])
+ return summarize(.success, control: control, position: nil)
+ case .failed(let error):
+ // Probe does not follow the event stream; tally the failure here.
+ control.setSessionState(.failed)
+ _ = control.recordError(error)
+ let described = describe.error(error)
+ reporter.emit("error", text: "open failed: \(described.text)", fields: [
+ ("fatal", .bool(true)), ("error", described.json),
+ ])
+ return summarize(.openFailed, control: control, position: nil)
+ case .interrupted:
+ return summarize(.interrupted, control: control, position: nil)
+ }
+ }
+
+ // MARK: play
+
+ func play(_ options: PlayOptions, interrupts: AsyncStream) async -> ExitCode {
+ reportStart("play", extra: [
+ ("duration", .optional(options.duration)),
+ ("start", .optional(options.start)),
+ ("seek", .optional(options.seek)),
+ ("software", .bool(options.software)),
+ ("audioLanguages", .array(options.audioLanguages.map(JSON.string))),
+ ("audio", .bool(!options.noAudio)),
+ ("video", .bool(!options.noVideo)),
+ ("interval", .double(options.interval)),
+ ("stallThreshold", .double(configuration.stallThreshold)),
+ ])
+ let session = PlayerSession(configuration: configuration)
+ let control = RunControl()
+ let forwarder = forward(interrupts, to: control)
+ defer { forwarder.cancel() }
+
+ // Subscribed before `open`, so the opening state changes print too.
+ let events = Task { await consumeEvents(of: session, control: control) }
+
+ switch await open(session, control: control) {
+ case .opened:
+ break
+ case .failed:
+ // The session reported the failure on its event stream already.
+ control.stop(.failed)
+ await shutdown(session, draining: events)
+ return summarize(.openFailed, control: control, position: nil)
+ case .interrupted:
+ await shutdown(session, draining: events)
+ return summarize(.interrupted, control: control, position: nil)
+ }
+
+ await session.play()
+ var helpers: [Task] = []
+ if let duration = options.duration {
+ helpers.append(Task {
+ try? await Task.sleep(for: .seconds(duration))
+ control.stop(.duration)
+ })
+ }
+ if options.interval > 0 {
+ helpers.append(Task { await heartbeat(session, every: options.interval) })
+ }
+ helpers.append(Task { await watchStartup(session, thenSeekTo: options.seek) })
+
+ let reason = await control.waitForStop()
+ for helper in helpers { helper.cancel() }
+ if reason == .interrupted {
+ reporter.emit("note", text: "interrupted, shutting down", fields: [("note", .string("interrupted"))])
+ }
+ let diagnostics = await session.diagnostics
+ let position = await session.position
+ let line = describe.diagnostics(diagnostics, inputMbps: nil)
+ reporter.emit("diagnostics", text: "final \(line.text)", fields: [("diagnostics", line.json)])
+ await shutdown(session, draining: events)
+
+ let tally = control.snapshot
+ return summarize(
+ .playback(failed: tally.failure != nil, stalls: tally.stalls.count),
+ control: control, position: position
+ )
+ }
+
+ // MARK: Steps
+
+ private enum OpenResult {
+ case opened(MediaInfo)
+ case failed(EngineError)
+ case interrupted
+ }
+
+ /// `open`, abandoned on Ctrl-C: cancelling the opening task aborts the
+ /// FFmpeg I/O, which can otherwise sit out the whole open timeout.
+ private func open(_ session: PlayerSession, control: RunControl) async -> OpenResult {
+ let url = source.url
+ let opening = Task { try await session.open(url: url) }
+ // Waits for the run's stop, which every run reaches; by then the
+ // open has long finished unless it was the Ctrl-C that stopped it.
+ Task {
+ _ = await control.waitForStop()
+ opening.cancel()
+ }
+ do {
+ return .opened(try await opening.value)
+ } catch {
+ if control.stopReason == .interrupted { return .interrupted }
+ let engineError = (error as? EngineError) ?? EngineError(code: .openFailed, message: "\(error)")
+ return .failed(engineError)
+ }
+ }
+
+ /// Prints events as they come and turns the terminal ones into a stop.
+ private func consumeEvents(of session: PlayerSession, control: RunControl) async {
+ for await event in session.events {
+ switch event {
+ case .stateChanged(let state):
+ control.setSessionState(state)
+ reporter.emit("state", text: "state \(state)", fields: [("state", .string("\(state)"))])
+ if state == .ended { control.stop(.ended) }
+
+ case .opened(let info):
+ // Printed from here rather than where `open` returns, so the
+ // records keep the order the session reported them in.
+ let media = describe.media(info)
+ reporter.emit("opened", text: "opened\n\(media.text)", fields: [("media", media.json)])
+ await reportSelection(session)
+
+ case .decoderDowngraded(let error):
+ control.countDowngrade()
+ let described = describe.error(error)
+ reporter.emit(
+ "decoderDowngraded", text: "decoder downgraded to software: \(described.text)",
+ fields: [("error", described.json)]
+ )
+
+ case .didSeek(let position):
+ reporter.emit(
+ "didSeek", text: String(format: "seeked to %.3f s", position),
+ fields: [("position", .double(position))]
+ )
+
+ case .stalled(let position):
+ let reason = await session.lastStallReason
+ control.countStall(reason)
+ reporter.emit(
+ "stalled",
+ text: String(format: "STALLED at %.3f s, stage: %@", position, reason?.rawValue ?? "unknown"),
+ fields: [("position", .double(position)), ("reason", .optional(reason?.rawValue))]
+ )
+
+ case .error(let error):
+ // A session that fails switches to `.failed` before it
+ // reports why; any other error left the session running.
+ let fatal = control.recordError(error)
+ let described = describe.error(error)
+ reporter.emit(
+ "error", text: "\(fatal ? "FAILED" : "error") \(described.text)",
+ fields: [("fatal", .bool(fatal)), ("error", described.json)]
+ )
+ if fatal { control.stop(.failed) }
+ }
+ }
+ }
+
+ private func heartbeat(_ session: PlayerSession, every interval: Double) async {
+ // Seeded here, so even the first heartbeat has a delivery rate.
+ var previous = (bytes: await session.diagnostics.deliveredBytes, at: reporter.elapsed)
+ while !Task.isCancelled {
+ try? await Task.sleep(for: .seconds(interval))
+ if Task.isCancelled { return }
+ let diagnostics = await session.diagnostics
+ let now = reporter.elapsed
+ // What the source delivered since the last beat: the number to
+ // hold against a provider's per-connection cap.
+ let mbps = now > previous.at
+ ? Double(diagnostics.deliveredBytes - previous.bytes) * 8 / 1_000_000 / (now - previous.at)
+ : nil
+ previous = (diagnostics.deliveredBytes, now)
+ let line = describe.diagnostics(diagnostics, inputMbps: mbps)
+ reporter.emit("heartbeat", text: "heartbeat \(line.text)", fields: [("diagnostics", line.json)])
+ }
+ }
+
+ /// Prints the startup timeline once the first frame shows, then makes
+ /// the `--seek` (a seek that lands mid-stream, which is the case that
+ /// differs from `--start`).
+ private func watchStartup(_ session: PlayerSession, thenSeekTo seek: Double?) async {
+ while !Task.isCancelled {
+ let startup = await session.startup
+ if startup.firstFrameShown != nil {
+ reporter.emit("startup", text: "startup \(startup)", fields: [("startup", describe.startup(startup))])
+ if let seek {
+ reporter.emit(
+ "seek", text: String(format: "seeking to %.3f s", seek),
+ fields: [("position", .double(seek))]
+ )
+ await session.seek(to: seek)
+ }
+ return
+ }
+ try? await Task.sleep(for: .milliseconds(50))
+ }
+ }
+
+ private func reportSelection(_ session: PlayerSession) async {
+ let video = await session.selectedVideoTrackIndex
+ let audio = await session.selectedAudioTrackIndex
+ let subtitle = await session.selectedSubtitleTrackIndex
+ func name(_ index: Int32?) -> String { index.map { "#\($0)" } ?? "none" }
+ func json(_ index: Int32?) -> JSON { index.map { .int(Int64($0)) } ?? .null }
+ reporter.emit(
+ "selected",
+ text: "selected video \(name(video)), audio \(name(audio)), subtitles \(name(subtitle))",
+ fields: [("video", json(video)), ("audio", json(audio)), ("subtitle", json(subtitle))]
+ )
+ }
+
+ /// Shuts the session down, bounded: a wedged teardown must not keep the
+ /// terminal hostage. The event stream ends with the session, so the
+ /// events still queued are printed before the summary.
+ private func shutdown(_ session: PlayerSession, draining events: Task? = nil) async {
+ let finished = await finishes(within: 5) { await session.shutdown() }
+ if !finished {
+ reporter.emit(
+ "note", text: "session shutdown did not finish within 5 s",
+ fields: [("note", .string("shutdown-timeout"))]
+ )
+ }
+ if let events {
+ _ = await finishes(within: 2) { await events.value }
+ events.cancel()
+ }
+ }
+
+ private func reportStart(_ command: String, extra: [(String, JSON)]) {
+ let headerNames = source.headers.map(\.name)
+ var text = "lumectl \(command) \(describe.url(source.url))"
+ if !headerNames.isEmpty {
+ text += "\n headers: \(headerNames.joined(separator: ", ")) (values hidden)"
+ }
+ reporter.emit("start", text: text, fields: [
+ ("command", .string(command)),
+ ("url", .string(describe.url(source.url))),
+ // Header values are where tokens and cookies travel.
+ ("headers", .array(headerNames.map(JSON.string))),
+ ("userAgent", .optional(source.userAgent)),
+ ("referer", .optional(source.referer.map(describe.url))),
+ ("reconnect", .bool(source.reconnect)),
+ ("ioTimeout", .optional(source.ioTimeout)),
+ ("openTimeout", .optional(source.openTimeout)),
+ ("ffmpeg", .string("\(FFmpegRuntime.versions)")),
+ ] + extra)
+ }
+
+ private func summarize(_ code: ExitCode, control: RunControl, position: Double?) -> ExitCode {
+ let tally = control.snapshot
+ let stallReasons = tally.stalls.map { $0?.rawValue ?? "unknown" }
+ var text = "summary \(code.outcome) (exit \(code.rawValue))"
+ if let position { text += String(format: ", position %.3f s", position) }
+ text += ", stalls \(tally.stalls.count)"
+ if !stallReasons.isEmpty { text += " [\(stallReasons.joined(separator: ", "))]" }
+ text += ", errors \(tally.errors), downgrades \(tally.downgrades)"
+ if let failure = tally.failure { text += "\n failure: \(describe.error(failure).text)" }
+ reporter.emit("summary", text: text, fields: [
+ ("outcome", .string(code.outcome)),
+ ("exitCode", .int(Int64(code.rawValue))),
+ ("position", .optional(position)),
+ ("stalls", .array(stallReasons.map(JSON.string))),
+ ("errors", .int(Int64(tally.errors))),
+ ("downgrades", .int(Int64(tally.downgrades))),
+ ("failure", tally.failure.map { describe.error($0).json } ?? .null),
+ ])
+ return code
+ }
+
+ private func forward(_ interrupts: AsyncStream, to control: RunControl) -> Task {
+ Task {
+ for await _ in interrupts {
+ control.stop(.interrupted)
+ }
+ }
+ }
+}
diff --git a/Sources/LumeCtlCore/Reporter.swift b/Sources/LumeCtlCore/Reporter.swift
new file mode 100644
index 0000000..3084c32
--- /dev/null
+++ b/Sources/LumeCtlCore/Reporter.swift
@@ -0,0 +1,66 @@
+import Foundation
+import Synchronization
+
+/// Where lumectl's records go: text or JSON Lines on stdout, and optionally
+/// JSON Lines into a file (`--record-events`), so a run can be watched live
+/// and attached to an issue afterwards.
+///
+/// Every record carries `t`, seconds since the run started — the one clock
+/// that lines up events, heartbeats and the startup timeline.
+public final class Reporter: Sendable {
+ /// Receives one finished line (no trailing newline).
+ public typealias Sink = @Sendable (String) -> Void
+
+ private let json: Bool
+ private let started: TimeInterval
+ private let stdout: Sink
+ private let recorder: Mutex
+
+ /// - Throws: when the record file cannot be created.
+ public init(output: OutputOptions, stdout: @escaping Sink) throws {
+ json = output.json
+ started = ProcessInfo.processInfo.systemUptime
+ self.stdout = stdout
+ var file: FileHandle?
+ if let path = output.recordPath {
+ let url = URL(fileURLWithPath: path)
+ guard FileManager.default.createFile(atPath: url.path, contents: nil) else {
+ throw UsageError("cannot create --record-events file '\(path)'")
+ }
+ file = try FileHandle(forWritingTo: url)
+ }
+ recorder = Mutex(file)
+ }
+
+ /// Seconds since the run started.
+ public var elapsed: Double {
+ ProcessInfo.processInfo.systemUptime - started
+ }
+
+ /// One record: `text` for the terminal, `fields` for JSON (after `t`
+ /// and `type`). Multi-line text keeps the timestamp on its first line.
+ func emit(_ type: String, text: String, fields: [(String, JSON)] = []) {
+ let t = elapsed
+ let record = JSON.object([("t", .double(t)), ("type", .string(type))] + fields).encoded
+ if json {
+ stdout(record)
+ } else {
+ let stamp = String(format: "[%8.3f] ", t)
+ let indent = String(repeating: " ", count: stamp.count)
+ let lines = text.split(separator: "\n", omittingEmptySubsequences: false)
+ stdout(lines.enumerated().map { ($0.offset == 0 ? stamp : indent) + $0.element }.joined(separator: "\n"))
+ }
+ recorder.withLock { file in
+ file?.write(Data((record + "\n").utf8))
+ }
+ }
+
+ /// Flushes and closes the record file.
+ func close() {
+ recorder.withLock { file in
+ try? file?.synchronize()
+ try? file?.close()
+ file = nil
+ }
+ }
+}
diff --git a/Sources/LumeCtlCore/RunControl.swift b/Sources/LumeCtlCore/RunControl.swift
new file mode 100644
index 0000000..b7a78c2
--- /dev/null
+++ b/Sources/LumeCtlCore/RunControl.swift
@@ -0,0 +1,123 @@
+import LumeEngineCore
+import Synchronization
+
+/// What a run has seen so far, and the one-shot signal that ends it.
+///
+/// Several tasks race to end a run — the duration timer, Ctrl-C, the event
+/// stream reporting `.ended` or a failure — and the first one wins; the
+/// others find the run already stopping.
+final class RunControl: Sendable {
+ enum Stop: Sendable, Equatable {
+ case duration
+ case ended
+ case failed
+ case interrupted
+ }
+
+ struct Tally: Sendable {
+ /// One entry per `.stalled`, with the stage `lastStallReason` named.
+ var stalls: [StallReason?] = []
+ var errors = 0
+ var downgrades = 0
+ /// The error the session failed with.
+ var failure: EngineError?
+ }
+
+ private struct State {
+ var stop: Stop?
+ var waiters: [CheckedContinuation] = []
+ var sessionState: PlayerSession.State = .idle
+ var tally = Tally()
+ }
+
+ private let state = Mutex(State())
+
+ /// Ends the run; only the first reason counts.
+ func stop(_ reason: Stop) {
+ let waiters = state.withLock { state -> [CheckedContinuation] in
+ guard state.stop == nil else { return [] }
+ state.stop = reason
+ let waiters = state.waiters
+ state.waiters.removeAll()
+ return waiters
+ }
+ for waiter in waiters {
+ waiter.resume(returning: reason)
+ }
+ }
+
+ var stopReason: Stop? {
+ state.withLock { $0.stop }
+ }
+
+ func waitForStop() async -> Stop {
+ await withCheckedContinuation { continuation in
+ let stopped = state.withLock { state -> Stop? in
+ if let stop = state.stop { return stop }
+ state.waiters.append(continuation)
+ return nil
+ }
+ if let stopped { continuation.resume(returning: stopped) }
+ }
+ }
+
+ var snapshot: Tally {
+ state.withLock { $0.tally }
+ }
+
+ func setSessionState(_ sessionState: PlayerSession.State) {
+ state.withLock { $0.sessionState = sessionState }
+ }
+
+ /// Counts an error; true when it is the one the session failed with.
+ func recordError(_ error: EngineError) -> Bool {
+ state.withLock { state in
+ state.tally.errors += 1
+ guard state.sessionState == .failed, state.tally.failure == nil else { return false }
+ state.tally.failure = error
+ return true
+ }
+ }
+
+ func countStall(_ reason: StallReason?) {
+ state.withLock { $0.tally.stalls.append(reason) }
+ }
+
+ func countDowngrade() {
+ state.withLock { $0.tally.downgrades += 1 }
+ }
+}
+
+/// Runs `operation` and waits at most `seconds` for it. True when it
+/// finished in time; when it did not, it keeps running unobserved.
+func finishes(within seconds: Double, _ operation: @escaping @Sendable () async -> Void) async -> Bool {
+ let outcome = Once()
+ return await withCheckedContinuation { continuation in
+ outcome.arm(continuation)
+ Task {
+ await operation()
+ outcome.resolve(true)
+ }
+ Task {
+ try? await Task.sleep(for: .seconds(seconds))
+ outcome.resolve(false)
+ }
+ }
+}
+
+/// A continuation resumed by whichever side gets there first.
+private final class Once: Sendable {
+ private let continuation = Mutex?>(nil)
+
+ func arm(_ continuation: CheckedContinuation) {
+ self.continuation.withLock { $0 = continuation }
+ }
+
+ func resolve(_ value: Bool) {
+ let continuation = self.continuation.withLock { stored in
+ defer { stored = nil }
+ return stored
+ }
+ continuation?.resume(returning: value)
+ }
+}
diff --git a/Sources/LumeCtlCore/URLScrubber.swift b/Sources/LumeCtlCore/URLScrubber.swift
new file mode 100644
index 0000000..ac9c1af
--- /dev/null
+++ b/Sources/LumeCtlCore/URLScrubber.swift
@@ -0,0 +1,77 @@
+/// Keeps credentials out of what lumectl prints.
+///
+/// IPTV providers put the account in the URL itself — Xtream-style paths
+/// (`/live///.ts`), `get.php?username=…&password=…`,
+/// tokens in the query, userinfo before the host — and terminal output ends
+/// up pasted into issues and chats. So a URL is reduced to what triage needs:
+/// scheme, host and port (which provider, which edge), plus the file
+/// extension (HLS playlist vs raw TS), and everything else is withheld.
+///
+/// Deliberately small and private to the tool: it only has to be right for
+/// the strings this tool prints.
+public enum URLScrubber {
+ static let placeholder = "[redacted]"
+
+ /// The URL reduced to `scheme://host[:port]`, with `/[redacted]` (plus a
+ /// short file extension when the path has one) standing for anything
+ /// after it. Strings without a scheme — local paths — come back as given:
+ /// a path names a file on the developer's own machine, not an account.
+ public static func redact(_ url: String) -> String {
+ guard let separator = url.range(of: "://") else { return url }
+ let scheme = url[.. String {
+ guard text.contains("://") else { return text }
+ // A scheme followed by `://` and everything up to whitespace or a
+ // character that commonly closes a quoted URL in a message. Built per
+ // call: `Regex` is not Sendable, so it cannot be a shared constant.
+ let embeddedURL = #/[A-Za-z][A-Za-z0-9+.\-]*://[^\s'"<>()\[\]{}]+/#
+ return text.replacing(embeddedURL) { match in
+ redact(String(text[match.range]))
+ }
+ }
+
+ private static func isScheme(_ candidate: Substring) -> Bool {
+ guard let first = candidate.first, first.isASCII, first.isLetter else { return false }
+ return candidate.allSatisfy { $0.isASCII && ($0.isLetter || $0.isNumber || "+-.".contains($0)) }
+ }
+
+ /// `.m3u8`, `.ts`, `.mkv` … of the last path component, when it looks
+ /// like a real extension (1–5 alphanumerics): it tells an HLS playlist
+ /// from a raw stream without saying which stream.
+ private static func fileExtension(of rest: Substring) -> String {
+ let pathEnd = rest.firstIndex(where: { "?#".contains($0) }) ?? rest.endIndex
+ let path = rest[...makeStream()
+ let sigint = installInterruptHandler(continuation)
+ let code = await LumeCtl.run(
+ command,
+ interrupts: interrupts,
+ stdout: { writeLine($0, to: .standardOutput) },
+ stderr: { writeLine($0, to: .standardError) }
+ )
+ sigint.cancel()
+ exit(code.rawValue)
+ }
+
+ /// The first Ctrl-C ends the run cleanly (bounded session shutdown, then
+ /// the summary); a second one means the user is done waiting.
+ private static func installInterruptHandler(
+ _ continuation: AsyncStream.Continuation
+ ) -> DispatchSourceSignal {
+ // Dispatch delivers the signal only once the default action is off.
+ signal(SIGINT, SIG_IGN)
+ let source = DispatchSource.makeSignalSource(signal: SIGINT, queue: DispatchQueue(label: "lumectl.sigint"))
+ let count = InterruptCount()
+ source.setEventHandler {
+ if count.increment() == 1 {
+ continuation.yield()
+ } else {
+ writeLine("lumectl: interrupted again, exiting without cleanup", to: .standardError)
+ exit(ExitCode.interrupted.rawValue)
+ }
+ }
+ source.resume()
+ return source
+ }
+
+ /// Unbuffered, so piped output (`| tee`, `| jq`) shows each record as it
+ /// happens rather than when a buffer fills.
+ private static func writeLine(_ line: String, to handle: FileHandle) {
+ handle.write(Data((line + "\n").utf8))
+ }
+}
+/// Ctrl-C presses so far; the handler is a `@Sendable` closure.
+private final class InterruptCount: Sendable {
+ private let count = Mutex(0)
+
+ func increment() -> Int {
+ count.withLock { count in
+ count += 1
+ return count
+ }
+ }
+}
+#else
+@main
+enum LumeCtlMain {
+ static func main() {
+ print("lumectl is macOS-only")
+ }
+}
+#endif
diff --git a/Tests/LumeEngineCoreTests/LumeCtlTests.swift b/Tests/LumeEngineCoreTests/LumeCtlTests.swift
new file mode 100644
index 0000000..db858f4
--- /dev/null
+++ b/Tests/LumeEngineCoreTests/LumeCtlTests.swift
@@ -0,0 +1,383 @@
+import Foundation
+import Testing
+@testable import LumeCtlCore
+import LumeEngineCore
+
+// lumectl lives in this test target rather than one of its own: its runs
+// need the generated fixtures and the torture server, both of which live here.
+
+@Suite("lumectl arguments")
+struct LumeCtlArgumentTests {
+ private func play(_ arguments: String...) throws -> PlayOptions {
+ guard case .play(let options) = try Arguments.parse(["play"] + arguments) else {
+ throw UsageError("not a play command")
+ }
+ return options
+ }
+
+ private func usageError(_ arguments: [String]) -> String? {
+ do {
+ _ = try Arguments.parse(arguments)
+ return nil
+ } catch {
+ return error.message
+ }
+ }
+
+ @Test("no arguments, help and version")
+ func helpAndVersion() throws {
+ #expect(try Arguments.parse([]) == .help)
+ #expect(try Arguments.parse(["--help"]) == .help)
+ #expect(try Arguments.parse(["help"]) == .help)
+ #expect(try Arguments.parse(["play", "http://h/x.ts", "-h"]) == .help)
+ #expect(try Arguments.parse(["--version"]) == .version)
+ }
+
+ @Test("play with every option, both spellings")
+ func playEverything() throws {
+ let options = try play(
+ "http://host/live.ts",
+ "--duration", "12.5", "--seek=30", "--start", "60", "--interval", "0",
+ "--software", "--audio-lang", "DE, en", "--no-video", "--stall-threshold=4",
+ "--header", "Authorization: Bearer a:b", "--header=X-Device: tv ",
+ "--user-agent", "Lume/1", "--referer", "http://ref/", "--no-reconnect",
+ "--io-timeout", "2.5", "--open-timeout", "10",
+ "--json", "--show-urls", "--record-events", "/tmp/run.jsonl"
+ )
+ #expect(options.source.url == "http://host/live.ts")
+ #expect(options.duration == 12.5)
+ #expect(options.seek == 30)
+ #expect(options.start == 60)
+ #expect(options.interval == 0)
+ #expect(options.software)
+ #expect(options.audioLanguages == ["de", "en"])
+ #expect(options.noVideo && !options.noAudio)
+ #expect(options.stallThreshold == 4)
+ #expect(options.source.headers == [
+ .init(name: "Authorization", value: "Bearer a:b"),
+ .init(name: "X-Device", value: "tv"),
+ ])
+ #expect(options.source.userAgent == "Lume/1")
+ #expect(options.source.referer == "http://ref/")
+ #expect(!options.source.reconnect)
+ #expect(options.source.ioTimeout == 2.5)
+ #expect(options.source.openTimeout == 10)
+ #expect(options.output.json && options.output.showURLs)
+ #expect(options.output.recordPath == "/tmp/run.jsonl")
+ }
+
+ @Test("defaults: muted, engine defaults kept")
+ func playDefaults() throws {
+ let options = try play("/path/basic.mp4")
+ #expect(options.duration == nil && options.seek == nil && options.start == nil)
+ #expect(options.interval == 2)
+ let configuration = options.configuration
+ let engineDefaults = PlayerConfiguration()
+ #expect(configuration.muted)
+ #expect(configuration.hardwareDecode == .videoToolbox)
+ #expect(configuration.enableAudio && configuration.enableVideo)
+ #expect(configuration.startPosition == 0)
+ #expect(configuration.stallThreshold == engineDefaults.stallThreshold)
+ #expect(configuration.demuxer.ioTimeout == engineDefaults.demuxer.ioTimeout)
+ #expect(configuration.demuxer.openTimeout == engineDefaults.demuxer.openTimeout)
+ #expect(configuration.demuxer.enableReconnect)
+ #expect(configuration.demuxer.httpHeaders.isEmpty)
+ }
+
+ @Test("options map onto PlayerConfiguration")
+ func configurationMapping() throws {
+ let configuration = try play(
+ "u", "--io-timeout", "2.5", "--open-timeout", "7", "--no-reconnect", "--software",
+ "--start", "42", "--audio-lang", "de,en", "--no-audio", "--stall-threshold", "3",
+ "--header", "A: 1", "--header", "A: 2", "--user-agent", "UA", "--referer", "R"
+ ).configuration
+ #expect(configuration.demuxer.ioTimeout == 2_500_000, "seconds become microseconds")
+ #expect(configuration.demuxer.openTimeout == 7)
+ #expect(!configuration.demuxer.enableReconnect)
+ #expect(configuration.hardwareDecode == .software)
+ #expect(configuration.startPosition == 42)
+ #expect(configuration.preferredAudioLanguages == ["de", "en"])
+ #expect(!configuration.enableAudio)
+ #expect(configuration.stallThreshold == 3)
+ #expect(configuration.demuxer.httpHeaders == ["A": "2"], "a repeated header keeps the last value")
+ #expect(configuration.demuxer.userAgent == "UA")
+ #expect(configuration.demuxer.referer == "R")
+ }
+
+ @Test("probe takes source and output options only")
+ func probe() throws {
+ let command = try Arguments.parse(["probe", "--json", "http://h/a.m3u8", "--user-agent", "UA"])
+ guard case .probe(let options) = command else {
+ Issue.record("expected probe, got \(command)")
+ return
+ }
+ #expect(options.source.url == "http://h/a.m3u8")
+ #expect(options.output.json)
+ #expect(options.configuration.demuxer.userAgent == "UA")
+ #expect(usageError(["probe", "u", "--duration", "3"])?.contains("unknown option '--duration'") == true)
+ }
+
+ @Test("-- ends options, so a URL may start with a dash")
+ func doubleDash() throws {
+ #expect(try play("--json", "--", "-odd-name.ts").source.url == "-odd-name.ts")
+ }
+
+ @Test("bad command lines say what is wrong")
+ func usageErrors() {
+ let cases: [([String], String)] = [
+ (["stream", "u"], "unknown command"),
+ (["play"], "needs a URL"),
+ (["play", "a", "b"], "unexpected extra argument 'b'"),
+ (["play", "u", "--duration"], "--duration expects a value"),
+ (["play", "u", "--duration", "-1"], "> 0"),
+ (["play", "u", "--duration", "0"], "> 0"),
+ (["play", "u", "--seek", "abc"], "≥ 0"),
+ (["play", "u", "--interval", "nan"], "≥ 0"),
+ (["play", "u", "--io-timeout", "inf"], "> 0"),
+ (["play", "u", "--header", "no colon"], "'Name: value'"),
+ (["play", "u", "--header", ": empty name"], "'Name: value'"),
+ (["play", "u", "--header", "A: 1\r\nB: 2"], "'Name: value'"),
+ (["play", "u", "--audio-lang", "de,,en"], "comma-separated"),
+ (["play", "u", "--software=yes"], "takes no value"),
+ (["play", "u", "--frobnicate"], "unknown option '--frobnicate'"),
+ (["play", "u", "-d", "3"], "spelled --name"),
+ (["play", "u", "--no-audio", "--no-video"], "nothing to play"),
+ (["play", ""], "empty"),
+ ]
+ for (arguments, expected) in cases {
+ let message = usageError(arguments)
+ #expect(message?.contains(expected) == true, "\(arguments) → \(message ?? "no error")")
+ }
+ }
+}
+
+@Suite("lumectl URL redaction")
+struct URLScrubberTests {
+ @Test("credentials in path, query and userinfo are withheld")
+ func redactsCredentials() {
+ let cases: [(String, String)] = [
+ ("http://provider.tv:8080/live/alice/s3cret/1234.ts", "http://provider.tv:8080/[redacted].ts"),
+ ("http://provider.tv/get.php?username=alice&password=s3cret&type=m3u", "http://provider.tv/[redacted].php"),
+ ("https://alice:s3cret@cdn.example.com/hls/index.m3u8?token=abc#t=1", "https://cdn.example.com/[redacted].m3u8"),
+ ("http://a:p@ss@host/x", "http://host/[redacted]"),
+ ("http://host:80/movie/alice/s3cret/99", "http://host:80/[redacted]"),
+ ("http://host/stream?auth=xyz", "http://host/[redacted]"),
+ ("rtmp://ingest.example.com/app/streamkey", "rtmp://ingest.example.com/[redacted]"),
+ ("HTTP://Host/path.with.dots/file.TS", "HTTP://Host/[redacted].TS"),
+ ("http://host/clip.verylongext", "http://host/[redacted]"),
+ ("http://host/v1.2/stream", "http://host/[redacted]"),
+ ]
+ for (input, expected) in cases {
+ #expect(URLScrubber.redact(input) == expected, "\(input)")
+ }
+ }
+
+ @Test("bare hosts and local paths are left alone")
+ func keepsWhatIsNotSecret() {
+ #expect(URLScrubber.redact("http://host:8080") == "http://host:8080")
+ #expect(URLScrubber.redact("http://host/") == "http://host/")
+ #expect(URLScrubber.redact("/Users/dev/TestStreams/basic.mp4") == "/Users/dev/TestStreams/basic.mp4")
+ #expect(URLScrubber.redact("file:///Users/dev/a.ts") == "file:///Users/dev/a.ts")
+ #expect(URLScrubber.redact("not a url") == "not a url")
+ #expect(URLScrubber.redact("1http://x/y") == "1http://x/y", "not a scheme")
+ }
+
+ @Test("URLs inside messages are redacted where they stand")
+ func redactsText() {
+ let message = "Failed to open segment 'http://cdn.tv/live/alice/s3cret/seg7.ts?t=1' (via https://u:p@edge.tv/k) for reading"
+ #expect(URLScrubber.redactText(message)
+ == "Failed to open segment 'http://cdn.tv/[redacted].ts' (via https://edge.tv/[redacted]) for reading")
+ #expect(URLScrubber.redactText("Connection refused") == "Connection refused")
+ #expect(URLScrubber.redactText("") == "")
+ }
+}
+
+@Suite("lumectl output")
+struct LumeCtlOutputTests {
+ @Test("JSON keeps key order and escapes")
+ func jsonEncoding() {
+ let value = JSON.object([
+ ("t", .double(1.23456)),
+ ("type", .string("a\"b\\c\nd\u{01}é")),
+ ("n", .null), ("ok", .bool(true)), ("i", .int(-5)),
+ ("nan", .double(.nan)), ("list", .array([.int(1), .optional(nil as String?)])),
+ ])
+ #expect(value.encoded
+ == #"{"t":1.235,"type":"a\"b\\c\nd\u0001é","n":null,"ok":true,"i":-5,"nan":null,"list":[1,null]}"#)
+ }
+
+ @Test("exit status: failure outranks stall, Ctrl-C is no verdict")
+ func exitCodes() {
+ #expect(ExitCode.playback(failed: false, stalls: 0) == .success)
+ #expect(ExitCode.playback(failed: false, stalls: 2) == .stalled)
+ #expect(ExitCode.playback(failed: true, stalls: 2) == .playbackFailed)
+ #expect(ExitCode.playback(failed: true, stalls: 0) == .playbackFailed)
+ let codes = [ExitCode.success, .toolFailure, .usage, .openFailed, .playbackFailed, .stalled, .interrupted]
+ #expect(Set(codes.map(\.rawValue)).count == codes.count, "every outcome has its own code")
+ }
+
+ @Test("errors carry the HTTP class and no URL")
+ func errorRendering() {
+ let forbidden = EngineError(
+ code: .openFailed, ffmpegCode: -(0xF8 | 0x34 << 8 | 0x30 << 16 | 0x33 << 24),
+ message: "open http://h/live/u/p/1.ts: Forbidden"
+ )
+ let rendered = Describe(showURLs: false).error(forbidden)
+ #expect(rendered.text.contains("http forbidden"))
+ #expect(!rendered.text.contains("/u/p/"))
+ #expect(rendered.json.encoded.contains(#""http":"forbidden""#))
+ #expect(Describe(showURLs: true).error(forbidden).text.contains("/live/u/p/1.ts"))
+ }
+}
+
+/// Collects what a run prints.
+private final class Output: Sendable {
+ private let lines = LockedBox<[String]>([])
+
+ var sink: Reporter.Sink {
+ { [lines] line in lines.set(lines.get() + [line]) }
+ }
+
+ var all: [String] { lines.get() }
+
+ /// JSON records of a `--json` run, by type.
+ func records(_ type: String) -> [[String: Any]] {
+ all.compactMap { line in
+ guard let data = line.data(using: .utf8),
+ let object = try? JSONSerialization.jsonObject(with: data) as? [String: Any],
+ object["type"] as? String == type else { return nil }
+ return object
+ }
+ }
+}
+
+@Suite("lumectl runs", .serialized)
+struct LumeCtlRunTests {
+ private func run(_ arguments: [String], interrupts: AsyncStream = AsyncStream { _ in }) async throws
+ -> (ExitCode, Output)
+ {
+ let output = Output()
+ let code = await LumeCtl.run(
+ try Arguments.parse(arguments), interrupts: interrupts, stdout: output.sink, stderr: output.sink
+ )
+ return (code, output)
+ }
+
+ @Test("probe prints the tracks", .timeLimit(.minutes(1)))
+ func probe() async throws {
+ let (code, output) = try await run(["probe", try Fixtures.path("multitrack.mkv"), "--json"])
+ #expect(code == .success)
+ let media = output.records("media").first?["media"] as? [String: Any]
+ let tracks = media?["tracks"] as? [[String: Any]] ?? []
+ #expect(tracks.map { $0["kind"] as? String } == ["video", "audio", "audio", "subtitle"])
+ #expect(tracks.compactMap { $0["language"] as? String }.contains("ger"))
+ #expect((media?["chapters"] as? [Any])?.count == 2)
+ #expect(output.records("summary").first?["exitCode"] as? Int == 0)
+ }
+
+ @Test("play prints startup, heartbeats and a clean summary", .timeLimit(.minutes(1)))
+ func play() async throws {
+ let (code, output) = try await run([
+ "play", try Fixtures.path("basic.mp4"), "--duration", "2", "--interval", "0.5", "--json",
+ ])
+ #expect(code == .success, "\(output.all.joined(separator: "\n"))")
+ #expect(output.records("startup").first.flatMap { $0["startup"] as? [String: Any] }?["firstFrameShown"] is Double)
+ #expect(output.records("heartbeat").count >= 2)
+ let states = output.records("state").compactMap { $0["state"] as? String }
+ #expect(states.contains("playing"))
+ let summary = output.records("summary").first
+ #expect(summary?["outcome"] as? String == "ok")
+ #expect((summary?["position"] as? Double ?? 0) > 1, "two seconds of playback")
+ }
+
+ @Test("a refused open exits 3 and never prints the credentials", .timeLimit(.minutes(1)))
+ func openFailure() async throws {
+ let server = try TortureHTTPServer(mode: .forbidden)
+ defer { server.stop() }
+ let url = "http://alice:pw@127.0.0.1:\(server.port)/live/alice/s3cret/1.ts?token=t0k"
+ let (code, output) = try await run(["play", url, "--no-reconnect", "--io-timeout", "2"])
+ #expect(code == .openFailed)
+ let text = output.all.joined(separator: "\n")
+ #expect(text.contains("http forbidden"), "\(text)")
+ for secret in ["alice", "pw", "s3cret", "t0k"] {
+ #expect(!text.contains(secret), "leaked '\(secret)': \(text)")
+ }
+ }
+
+ @Test("a source that goes silent exits 5 with the stage", .timeLimit(.minutes(1)))
+ func stall() async throws {
+ // ~6 s of a 12 s TS, then nothing on an open connection. Less than
+ // FFmpeg's ~5 s MPEG-TS probe window would hold `open` until its
+ // timeout instead. The watchdog reports the source within the
+ // threshold of it going quiet, while the buffer still plays out.
+ let server = try TortureHTTPServer(mode: .stallAfter(prefix: 700_000, file: Fixtures.url("wrap.ts")))
+ defer { server.stop() }
+ let (code, output) = try await run([
+ "play", server.url, "--duration", "5", "--stall-threshold", "2",
+ "--io-timeout", "60", "--no-reconnect", "--interval", "0", "--json",
+ ])
+ #expect(code == .stalled, "\(output.all.joined(separator: "\n"))")
+ #expect(output.records("stalled").first?["reason"] as? String == "source")
+ }
+
+ @Test("a live channel that ends exits 4", .timeLimit(.minutes(1)))
+ func liveSourceEnded() async throws {
+ let directory = try Fixtures.url("hls-live/index.m3u8").deletingLastPathComponent()
+ let server = try TortureHTTPServer(mode: .hlsLiveThenEnded(directory: directory))
+ defer { server.stop() }
+ let (code, output) = try await run([
+ "play", server.hlsURL, "--duration", "30", "--stall-threshold", "30", "--interval", "0", "--json",
+ ])
+ #expect(code == .playbackFailed, "\(output.all.joined(separator: "\n"))")
+ let failure = output.records("summary").first?["failure"] as? [String: Any]
+ #expect(failure?["code"] as? String == "sourceEnded")
+ }
+
+ @Test("an interrupt ends the run cleanly", .timeLimit(.minutes(1)))
+ func interrupt() async throws {
+ let (interrupts, continuation) = AsyncStream.makeStream()
+ let started = Date()
+ Task {
+ try? await Task.sleep(for: .seconds(1.5))
+ continuation.yield()
+ }
+ let (code, output) = try await run(
+ ["play", try Fixtures.path("basic.mp4"), "--interval", "0", "--json"], interrupts: interrupts
+ )
+ #expect(code == .success)
+ #expect(Date().timeIntervalSince(started) < 8, "a 10 s file must not play to its end")
+ #expect(output.records("note").first?["note"] as? String == "interrupted")
+ #expect(output.records("summary").count == 1)
+ }
+
+ #if os(macOS)
+ @Test("Ctrl-C on the real binary shuts down and exits 0", .timeLimit(.minutes(1)))
+ func sigint() async throws {
+ // The test bundle sits next to the products it was built with.
+ let binary = Bundle(for: BundleMarker.self).bundleURL
+ .deletingLastPathComponent().appendingPathComponent("lumectl")
+ try #require(FileManager.default.isExecutableFile(atPath: binary.path), "no lumectl at \(binary.path)")
+
+ let process = Process()
+ process.executableURL = binary
+ process.arguments = ["play", try Fixtures.path("basic.mp4"), "--interval", "0"]
+ let stdout = Pipe()
+ process.standardOutput = stdout
+ process.standardError = FileHandle.nullDevice
+ try process.run()
+ try await Task.sleep(for: .seconds(2))
+ process.interrupt()
+ let deadline = Date(timeIntervalSinceNow: 10)
+ while process.isRunning, Date() < deadline {
+ try await Task.sleep(for: .milliseconds(50))
+ }
+ if process.isRunning { process.terminate() }
+ let text = String(decoding: stdout.fileHandleForReading.readDataToEndOfFile(), as: UTF8.self)
+ #expect(process.terminationStatus == 0, "\(text)")
+ #expect(text.contains("interrupted, shutting down"))
+ #expect(text.contains("summary ok (exit 0)"))
+ }
+ #endif
+}
+
+private final class BundleMarker {}