Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,9 @@ This split is the core invariant of the codebase:
- **HDR metadata is written on every delivered buffer, from the frame.** `Decode/HDRMetadata` turns the frame's mastering display (ST 2086) and content light level side data into Apple's big-endian SEI payloads on the pixel buffer in `VideoDecoder.deliver`, and HDR10+ (ST 2094-40) into a T.35 message the `SampleBufferBuilder` attaches per sample (`kCMSampleAttachmentKey_HDR10PlusPerFrameData`). No path keeps it by itself: VideoToolbox, as FFmpeg drives it, leaves its surfaces without either key, and the engine's own buffers carry only what it writes. Read the frame, not the codec parameters — FFmpeg already folds the container's boxes into every frame and lets the bitstream's SEI override them. Keys are written *or removed* on every buffer (pools recycle), and the format description is rebuilt whenever they change, because `CMVideoFormatDescriptionMatchesImageBuffer` compares them; don't replace that match with a cheaper key.
- **Multi-layer HEVC reaches the codec as its base layer only.** `Decode/HEVCBaseLayerFilter` drops every `nuh_layer_id > 0` NAL unit from packets and extradata before `avcodec_send_packet`/`avcodec_open2`. FFmpeg files an MV-HEVC stream's second-view SPS/PPS (DV profile 20, spatial video) alongside the base layer's, its VideoToolbox glue packs all of them into one `hvcC`, VideoToolbox rejects that session, and FFmpeg falls back to software *without an error* — 4K HEVC in software, which stutters on Apple TV. The engine presents only the base view, so stripping loses nothing. Relatedly, `VideoDecoder.isHardwareActive` is judged from the frames' pixel format, not from whether a VT device was attached, and such a silent fallback emits `.downgradedToSoftware` — keep it that way, or this class of bug goes invisible again.
- **Dolby Vision with an IPT base layer is converted, not tagged.** Profiles 5, 10.0 and 20 (compatibility ID 0) carry IPTPQc2, which shows pink/green as YCbCr. No colour attachment fixes that — the RPU reshaping changes per scene and Apple's decoders don't apply it to decoded frames — so `Decode/DolbyVisionConverter` turns each frame into HDR10 on the GPU in `VideoDecoder.deliver`, the one point every frame (hardware, software, deinterlaced) passes with its side data. Whether a frame is IPT is decided *per frame* by `Decode/DolbyVisionDetection`, never by the configuration record alone — remuxes mislabel HDR10 as P5 and broadcasts drop RPUs without changing it, and converting HDR10 as IPT gives wrong colours. Order: an `AVCOL_SPC_IPT_C2` frame is IPT; a frame with its *own* RPU is what the RPU's `signal_color_space` says (2 = IPT); a frame without one that states a matrix (BT.2020 PQ) is that matrix; only an untagged frame without its own RPU falls back to the record (genuine P5 often leaves the VUI matrix unspecified). "Own" matters because FFmpeg's HEVC decoder attaches the last RPU's `AV_FRAME_DATA_DOVI_METADATA` to every later frame; only `AV_FRAME_DATA_DOVI_RPU_BUFFER` marks the frame that carried one (AV1 attaches metadata only to its own frame). HDR10-compatible profiles (8.1 etc.) must stay on the zero-copy path. GPU refusals are per-frame (iOS refuses a backgrounded app); only a converter that can't exist gives up for the session.
- **Display criteria describe the output, not the container.** `PlayerSession.displayCriteria` (pure resolution in `Media/ContentDisplayCriteria`) is what an Apple TV with Match Content should switch the TV to: the refresh rate *after* deinterlacing (field rate turns 1080i50 into 50 Hz) and the dynamic range of what is presented (IPT Dolby Vision profiles are converted, so HDR10; every other Dolby Vision profile is presented as its base layer, so HDR10, HLG or SDR by its compatibility ID and the frames' transfer function). Never ask for Dolby Vision mode while the engine has no Dolby Vision output path — that belongs to the PLAN.md §7 compressed-`dvh1` spike. Before the first frame it is predicted from the stream; once the renderer has been handed frames, their colour tags and measured cadence (`Render/FrameCadence`) decide. Rates snap to the film/broadcast grid or ask for none — never an odd rate. Applying it is tvOS-only and opt-in (`Media/DisplayCriteriaMatcher`, or `LumePlayerView(…, matchesDisplayCriteria: true)`): one matcher per player screen, kept across sessions so zapping between same-mode channels never switches the TV, reset only when the screen goes away. The mode-switch blank is sat out with `PlayerSession.holdStart(for:)`, which the session bounds itself and the stall watchdog ignores — keep both, or a host that never releases wedges playback, or a held start reads as a stall.
- **Dolby Vision with an IPT base layer is converted, not tagged.** Profiles 5, 10.0 and 20 (compatibility ID 0) carry IPTPQc2, which shows pink/green as YCbCr. No colour attachment fixes that — the RPU reshaping changes per scene and Apple's decoders don't apply it to decoded frames — so `Decode/DolbyVisionConverter` turns each frame into HDR10 on the GPU in `VideoDecoder.deliver`, the one point every frame (hardware, software, deinterlaced) passes with its side data. Detection is the stream's DV configuration record *or* an `AVCOL_SPC_IPT_C2` frame; many P5 encodes leave the VUI matrix unspecified, so don't reduce it to the tag. HDR10-compatible profiles (8.1 etc.) must stay on the zero-copy path. GPU refusals are per-frame (iOS refuses a backgrounded app); only a converter that can't exist gives up for the session.
- **Audio is fitted to the negotiated route width, never the route's capability.** `AudioDecoder` reads `AudioOutputRoute.outputChannels` (the session's `outputNumberOfChannels`, followed through route-change notifications) per frame and remixes per `AudioLayoutPolicy` with an explicit `DownmixMatrix` — a fold-down mixes centre and surrounds in, it never drops channels. Don't switch it to `maximumOutputNumberOfChannels`: a renderer fed more channels than the session outputs failed outright on tvOS. The engine only observes the audio session; setting its category, activation or preferred channel count is the host's job (Lume does it). When the route narrows, `SystemRenderer` keeps audio decoded for the wider route out of the renderer.
- **Zero global mutable state:** all configuration is `PlayerConfiguration` passed at session creation. No `static var` knobs.
- **No force unwraps / `as!` / `fatalError` in engine paths**; failures return typed `EngineError`. No `assertionFailure` in production channel/data paths — misuse degrades, never crashes.
- **Never scrape FFmpeg log strings for control flow** (`av_log` is diagnostics only). Silence is never a failure mode: pipelines are supervised, stalls are detected from playhead ground truth, and failures surface as typed `PlayerEvent`s on the session's `AsyncStream`.
Expand Down
2 changes: 2 additions & 0 deletions PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -190,13 +190,15 @@ Licensing note: engine code is **MIT** (`LICENSE`); FFmpeg is LGPL 2.1+, and the
## 6. Feature implementation notes (selected)

- **Now Playing (explicit requirement):** `NowPlayingCenter` module in `LumeEngineMedia` — `MPNowPlayingInfoCenter` (title/artwork/elapsed/duration/rate, live-stream flag) + `MPRemoteCommandCenter` (play/pause/seek/skip/next-episode hook), audio-session lifecycle, route-change and interruption handling. Works on the lock screen/Control Center including while audio/video routes to an Apple TV. Note: full-screen *video* AirPlay of FFmpeg-decoded content is not possible for third parties (system limitation); Lume's existing switch-to-AVPlayer-on-AirPlay stays, and the integration phase adds a shared app-level Now Playing service so lock-screen metadata is present in **both** engines (closing today's gap).
- **Audio width follows the route:** decoded audio leaves the engine in the layout `AudioLayoutPolicy` picks for the *negotiated* route width (`AVAudioSession.outputNumberOfChannels`, followed through route changes by `AudioOutputRoute`; stereo on macOS): discrete multichannel when the route carries it (HDMI receiver), an explicit ITU-R BS.775 fold-down (`DownmixMatrix`: centre and surrounds at −3 dB, LFE omitted, not normalized) when it does not, mono on both speakers of a stereo route. A route change refits at the next decoded frame; audio decoded for a wider route never reaches a narrower one. The engine observes the audio session only — category, activation and the preferred channel count are the host's. `PlayerSession.audioHealth` reports audio that died while video plays on (decoder gave up, renderer failed, or nothing reaching the renderer at a running playhead).
- **PiP:** `AVPictureInPictureController(contentSource: .init(sampleBufferDisplayLayer:playbackDelegate:))` from the SystemRenderer; subtitle cues composited into the sample buffers during PiP so subs survive.
- **Trickplay:** a `ThumbnailSession` — separate lightweight demux+decode (keyframes only, HW decode, downscale to ~320 px) filling a time-indexed cache; API `thumbnails(at:interval:)`. Works for VOD/catch-up; live uses the rewind buffer.
- **Live rewind (time-shift DVR):** disk-backed packet spool behind a custom AVIO layer; seeking within spool = local demux, seeking to live edge = passthrough. Configurable size/duration caps.
- **Zero-delay switching:** `prepare(next:)` opens a second session through first-frame-decoded, then atomically swaps renderer attachment; old session torn down async. (Also powers Lume's next-episode auto-advance and channel zapping.)
- **Precache:** same AVIO disk-cache layer, `precache(url:leadingSeconds:)` background task with LRU budget.
- **Upscaling:** MetalFX spatial scaler stage (D5), quality/perf switch, auto-disable on thermal pressure (`ProcessInfo.thermalState`).
- **HDR:** HDR10/HLG passthrough via sample-buffer attachments. DV with an HDR10-compatible base layer (P8.1, P10.1, …) plays as HDR10. DV with an IPTPQc2 base layer (P5, P10.0, P20 — compatibility ID 0) is converted to HDR10 before enqueue by a Metal compute pass (`Decode/DolbyVisionConverter`) that applies each frame's RPU reshaping (polynomial + MMR) and the IPT→LMS→BT.2020 matrices FFmpeg exports as `AV_FRAME_DATA_DOVI_METADATA`; the renderer then tone-maps it like any HDR10 stream. Dynamic DV display management (L1/L2/L8 trims) is not carried over. HDR10 static metadata (mastering display, MaxCLL/MaxFALL — from the bitstream's SEI or the container, falling back to a DV RPU's level 6 for the light level) is attached to every delivered pixel buffer, since neither VideoToolbox nor the engine's own buffers carry it otherwise, and so reaches the format description; HDR10+ dynamic metadata travels per sample as `kCMSampleAttachmentKey_HDR10PlusPerFrameData` (`Decode/HDRMetadata`). Whether the renderer acts on HDR10+ for decoded (not compressed) samples is still to be confirmed on an HDR10+ display. EDR headroom handling on macOS/visionOS Metal path.
- **Match Content (tvOS):** a sample-buffer pipeline gets no automatic display mode switching, so the engine derives `ContentDisplayCriteria` from its own output — refresh rate after deinterlacing (measured from enqueued frame PTS once there are frames, snapped to 23.976/24/25/29.97/30/50/59.94/60 or no preference), dynamic range of what is presented (IPT Dolby Vision profiles → HDR10 after conversion; other Dolby Vision profiles → their base layer's range by compatibility ID, cross-checked with the frames' transfer function; otherwise the frames' or the stream's transfer function) — and `DisplayCriteriaMatcher` sets it as the window's `AVDisplayManager.preferredDisplayCriteria` via `AVDisplayCriteria(refreshRate:formatDescription:)` (tvOS 17+). Dolby Vision mode is never requested: the engine presents the base layer, and only a real DV output path (§7 spike) should ask for it. The start is held, bounded, until the TV finishes switching. Opt-in; mid-stream changes are not chased (each switch blanks the TV).
- **360°/VR:** MetalRenderer projection (equirect sphere / fisheye), CoreMotion + gesture camera; visionOS immersive presentation later.
- **Blu-ray ISO/DVD:** libbluray/libdvdnav behind the `AVIOProtocol` seam; menu-less title playback first (longest-title heuristic), chapters mapped to the chapter API.
- **AI subtitles:** whisper.cpp (Metal) tapping the decoded-audio channel; streaming word-level segments feed the subtitle model as a synthetic track; on-device translation via Translation.framework.
Expand Down
25 changes: 25 additions & 0 deletions Sources/LumeEngine/LumePlayer.swift
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,24 @@ public final class LumePlayer {
/// layer tree directly (UIKit/AppKit).
public var displayLayer: LumeDisplayLayer? { session?.renderer.displayLayer }

/// tvOS: switch the TV to each loaded stream's refresh rate and dynamic
/// range (Match Content) while it plays in a `LumePlayerView`, which
/// supplies the window; `stop()` and the view going away hand the display
/// back. Set before `load`. Off by default; ignored on other platforms.
public var matchesDisplayCriteria = false {
didSet {
#if os(tvOS)
if !matchesDisplayCriteria { displayMatcher.reset() }
#endif
}
}

#if os(tvOS)
/// One per player, so loading the next stream does not hand the display
/// back in between (see `DisplayCriteriaMatcher`).
let displayMatcher = DisplayCriteriaMatcher()
#endif

private var session: PlayerSession?
private var eventTask: Task<Void, Never>?
private var tickTask: Task<Void, Never>?
Expand Down Expand Up @@ -86,6 +104,10 @@ public final class LumePlayer {

do {
let info = try await session.open(url: url)
#if os(tvOS)
// Before the app's play(): the start is held while the TV switches.
if matchesDisplayCriteria { await displayMatcher.match(session) }
#endif
mediaInfo = info
duration = info.duration.map(MediaTime.seconds)
return info
Expand Down Expand Up @@ -135,6 +157,9 @@ public final class LumePlayer {
}

public func stop() async {
#if os(tvOS)
displayMatcher.reset()
#endif
await teardownSession()
state = .idle
position = 0
Expand Down
53 changes: 49 additions & 4 deletions Sources/LumeEngine/LumePlayerView.swift
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,29 @@ public struct LumePlayerView: View {
}

public var body: some View {
if let layer = player.displayLayer {
LayerHostView(layer: layer, gravity: gravity)
ZStack {
if let layer = player.displayLayer {
#if os(tvOS)
LayerHostView(
layer: layer,
gravity: gravity,
displayMatcher: player.matchesDisplayCriteria ? player.displayMatcher : nil
)
.accessibilityLabel("Video")
} else {
Color.black
#else
LayerHostView(layer: layer, gravity: gravity)
.accessibilityLabel("Video")
#endif
} else {
Color.black
}
}
#if os(tvOS)
// The whole view, not the layer host: that one is swapped out while
// the player moves to the next stream, which must not hand the
// display back in between.
.onDisappear { [player] in player.displayMatcher.reset() }
#endif
}
}

Expand All @@ -33,21 +50,49 @@ import UIKit
private struct LayerHostView: UIViewRepresentable {
let layer: LumeDisplayLayer
let gravity: AVLayerVideoGravity
#if os(tvOS)
/// Given this view's window, so it knows which display to match.
var displayMatcher: DisplayCriteriaMatcher?
#endif

func makeUIView(context: Context) -> LayerHostUIView {
let view = LayerHostUIView()
view.install(layer: layer, gravity: gravity)
#if os(tvOS)
view.displayMatcher = displayMatcher
#endif
return view
}

func updateUIView(_ view: LayerHostUIView, context: Context) {
view.install(layer: layer, gravity: gravity)
#if os(tvOS)
view.displayMatcher = displayMatcher
#endif
}
}

final class LayerHostUIView: UIView {
private weak var hosted: LumeDisplayLayer?

#if os(tvOS)
weak var displayMatcher: DisplayCriteriaMatcher? {
didSet { lendWindow() }
}

override func didMoveToWindow() {
super.didMoveToWindow()
lendWindow()
}

/// Only ever sets a window: leaving one is not the end of playback (the
/// host is swapped while the next stream loads); the player view's
/// disappearance is.
private func lendWindow() {
if let window { displayMatcher?.window = window }
}
#endif

func install(layer: LumeDisplayLayer, gravity: AVLayerVideoGravity) {
layer.videoGravity = gravity
guard hosted !== layer else { return }
Expand Down
Loading
Loading