Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ Prerequisites:
build/scripts/build-ffmpeg.sh macos-arm64 # one slice is enough for local dev
build/scripts/make-xcframework.sh
```
- Tests need a host `ffmpeg` CLI (Homebrew) — fixtures are generated lazily on first test run by `Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh` into `TestStreams/generated/` (gitignored, idempotent). `Fixtures.swift` finds Homebrew ffmpeg automatically; override with the `FFMPEG` env var.
- Tests need a host `ffmpeg` CLI (Homebrew) — fixtures are generated lazily on first test run by `Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh` into `TestStreams/generated/` (gitignored, idempotent). `Fixtures.swift` finds Homebrew ffmpeg automatically; override with the `FFMPEG` env var. The closed-caption fixture (`captions.ts`) also needs `python3` (the one Xcode's command-line tools provide is enough), because no ffmpeg CLI option writes captions into the video; without it that fixture is skipped and its tests fail as missing.

CI (`.github/workflows/ci.yml`) builds the macOS FFmpeg slice (cached on `build/versions.json` + scripts + patches hashes) and runs `swift test` with a strict one-thread cooperative pool (see above), checks the Markdown docs offline (`.github/scripts/check-docs.py`: relative links and in-repo anchors resolve, `CLAUDE.md` is still the `AGENTS.md` symlink — run it after editing docs), then cross-compiles the library for iOS and tvOS via `xcodebuild` (compile-only, one cached slice each — visionOS is left to local verification because that matrix entry was never confirmed on a runner — the xrOS SDK ships with Xcode, but a visionOS *destination* also needs Xcode's visionOS platform component); tagged releases build all 10 platform slices and attach the xcframework.

Expand Down Expand Up @@ -85,6 +85,11 @@ This split is the core invariant of the codebase:
- **HDR metadata is written on every delivered buffer, from the frame.** `Decode/HDRMetadata` turns the frame's mastering display (ST 2086) and content light level side data into Apple's big-endian SEI payloads on the pixel buffer in `VideoDecoder.deliver`, and HDR10+ (ST 2094-40) into a T.35 message the `SampleBufferBuilder` attaches per sample (`kCMSampleAttachmentKey_HDR10PlusPerFrameData`). No path keeps it by itself: VideoToolbox, as FFmpeg drives it, leaves its surfaces without either key, and the engine's own buffers carry only what it writes. Read the frame, not the codec parameters — FFmpeg already folds the container's boxes into every frame and lets the bitstream's SEI override them. Keys are written *or removed* on every buffer (pools recycle), and the format description is rebuilt whenever they change, because `CMVideoFormatDescriptionMatchesImageBuffer` compares them; don't replace that match with a cheaper key.
- **Multi-layer HEVC reaches the codec as its base layer only.** `Decode/HEVCBaseLayerFilter` drops every `nuh_layer_id > 0` NAL unit from packets and extradata before `avcodec_send_packet`/`avcodec_open2`. FFmpeg files an MV-HEVC stream's second-view SPS/PPS (DV profile 20, spatial video) alongside the base layer's, its VideoToolbox glue packs all of them into one `hvcC`, VideoToolbox rejects that session, and FFmpeg falls back to software *without an error* — 4K HEVC in software, which stutters on Apple TV. The engine presents only the base view, so stripping loses nothing. Relatedly, `VideoDecoder.isHardwareActive` is judged from the frames' pixel format, not from whether a VT device was attached, and such a silent fallback emits `.downgradedToSoftware` — keep it that way, or this class of bug goes invisible again.
- **Dolby Vision with an IPT base layer is converted, not tagged.** Profiles 5, 10.0 and 20 (compatibility ID 0) carry IPTPQc2, which shows pink/green as YCbCr. No colour attachment fixes that — the RPU reshaping changes per scene and Apple's decoders don't apply it to decoded frames — so `Decode/DolbyVisionConverter` turns each frame into HDR10 on the GPU in `VideoDecoder.deliver`, the one point every frame (hardware, software, deinterlaced) passes with its side data. Whether a frame is IPT is decided *per frame* by `Decode/DolbyVisionDetection`, never by the configuration record alone — remuxes mislabel HDR10 as P5 and broadcasts drop RPUs without changing it, and converting HDR10 as IPT gives wrong colours. Order: an `AVCOL_SPC_IPT_C2` frame is IPT; a frame with its *own* RPU is what the RPU's `signal_color_space` says (2 = IPT); a frame without one that states a matrix (BT.2020 PQ) is that matrix; only an untagged frame without its own RPU falls back to the record (genuine P5 often leaves the VUI matrix unspecified). "Own" matters because FFmpeg's HEVC decoder attaches the last RPU's `AV_FRAME_DATA_DOVI_METADATA` to every later frame; only `AV_FRAME_DATA_DOVI_RPU_BUFFER` marks the frame that carried one (AV1 attaches metadata only to its own frame). HDR10-compatible profiles (8.1 etc.) must stay on the zero-copy path. GPU refusals are per-frame (iOS refuses a backgrounded app); only a converter that can't exist gives up for the session.
- **Display criteria describe the output, not the container.** `PlayerSession.displayCriteria` (pure resolution in `Media/ContentDisplayCriteria`) is what an Apple TV with Match Content should switch the TV to: the refresh rate *after* deinterlacing (field rate turns 1080i50 into 50 Hz) and the dynamic range of what is presented (IPT Dolby Vision profiles are converted, so HDR10; every other Dolby Vision profile is presented as its base layer, so HDR10, HLG or SDR by its compatibility ID and the frames' transfer function). Never ask for Dolby Vision mode while the engine has no Dolby Vision output path — that belongs to the PLAN.md §7 compressed-`dvh1` spike. Before the first frame it is predicted from the stream; once the renderer has been handed frames, their colour tags and measured cadence (`Render/FrameCadence`) decide. Rates snap to the film/broadcast grid or ask for none — never an odd rate. Applying it is tvOS-only and opt-in (`Media/DisplayCriteriaMatcher`, or `LumePlayerView(…, matchesDisplayCriteria: true)`): one matcher per player screen, kept across sessions so zapping between same-mode channels never switches the TV, reset only when the screen goes away. The mode-switch blank is sat out with `PlayerSession.holdStart(for:)`, which the session bounds itself and the stall watchdog ignores — keep both, or a host that never releases wedges playback, or a held start reads as a stall.
- **Dolby Vision with an IPT base layer is converted, not tagged.** Profiles 5, 10.0 and 20 (compatibility ID 0) carry IPTPQc2, which shows pink/green as YCbCr. No colour attachment fixes that — the RPU reshaping changes per scene and Apple's decoders don't apply it to decoded frames — so `Decode/DolbyVisionConverter` turns each frame into HDR10 on the GPU in `VideoDecoder.deliver`, the one point every frame (hardware, software, deinterlaced) passes with its side data. Detection is the stream's DV configuration record *or* an `AVCOL_SPC_IPT_C2` frame; many P5 encodes leave the VUI matrix unspecified, so don't reduce it to the tag. HDR10-compatible profiles (8.1 etc.) must stay on the zero-copy path. GPU refusals are per-frame (iOS refuses a backgrounded app); only a converter that can't exist gives up for the session.
- **Audio is fitted to the negotiated route width, never the route's capability.** `AudioDecoder` reads `AudioOutputRoute.outputChannels` (the session's `outputNumberOfChannels`, followed through route-change notifications) per frame and remixes per `AudioLayoutPolicy` with an explicit `DownmixMatrix` — a fold-down mixes centre and surrounds in, it never drops channels. Don't switch it to `maximumOutputNumberOfChannels`: a renderer fed more channels than the session outputs failed outright on tvOS. The engine only observes the audio session; setting its category, activation or preferred channel count is the host's job (Lume does it). When the route narrows, `SystemRenderer` keeps audio decoded for the wider route out of the renderer.
- **How long a subtitle shows is the timeline's decision, not the decoder's.** Decoders hand `Subtitle/SubtitleStore` a `SubtitleEvent` (start, end *or none*, text, whether the track replaces the whole screen with each update); `Subtitle/SubtitleTimeline` decides the rest: an open cue ends at the next update of its track (capped at 16 s), an empty update on a closed-caption/DVB/PGS/DVD track is a clear that ends what is showing, a seek ends open cues, re-read ranges are deduplicated, and the store is bounded. Don't turn `end_display_time == UINT32_MAX` into a number in a decoder or drop empty updates there — both put stuck or stacked cues on screen.
- **Closed captions leave the video lane through a tap, never a wait.** CEA-608 in A/53 side data (`AV_FRAME_DATA_A53_CC`) is lifted off each decoded frame in `VideoDecoder.receiveFrames` (before deinterlacing) by `Subtitle/ClosedCaptionTap`, stamped with the frame's PTS, and `trySend`-ed to an ordinary `SubtitleDecoder` running `ccaption` in real-time mode — a slow caption lane drops captions, never frames. The track is synthetic (`TrackInfo.embeddedClosedCaptionsIndex`) and appears in `PlayerSession.mediaInfo` once caption content is seen; `.opened` keeps the open-time track list, because no new `PlayerEvent` case may be added for it. Only CEA-608 field 1 (CC1/CC2) is decoded: FFmpeg has no CEA-708 (DTVCC) decoder, and teletext needs libzvbi, which the build does not include.
- **Zero global mutable state:** all configuration is `PlayerConfiguration` passed at session creation. No `static var` knobs.
- **No force unwraps / `as!` / `fatalError` in engine paths**; failures return typed `EngineError`. No `assertionFailure` in production channel/data paths — misuse degrades, never crashes.
- **Never scrape FFmpeg log strings for control flow** (`av_log` is diagnostics only). Silence is never a failure mode: pipelines are supervised, stalls are detected from playhead ground truth, and failures surface as typed `PlayerEvent`s on the session's `AsyncStream`.
Expand Down
3 changes: 3 additions & 0 deletions PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,7 @@ Brightness/contrast/saturation and MetalFX spatial upscaling operate CVPixelBuff

**D6 — One subtitle engine, composited as overlay.**
libass renders ASS/SSA (full effects, embedded fonts via attachment streams, word-by-word \k tags); PGS/DVB/VobSub decoded by FFmpeg to bitmaps; SRT/VTT parsed to styled cues rendered through the same layout box. Output = time-tagged `SubtitleImage` cues composited on a CALayer above the video surface (and into the PiP content when active; into Metal for HDR subtitle mode). Dual subtitle slots (main + secondary). External files attach as additional demux sessions.
**Cue lifecycle and closed captions (as built).** Every subtitle source — demuxed stream, sidecar file, or captions lifted from the video — reaches one time-indexed store as display updates; the store's `SubtitleTimeline` alone decides how long each shows (open cues end at the next update or a clear, capped; seeks end open cues; bounded size). CEA-608 captions carried as A/53 side data are tapped off decoded video frames without ever blocking the video lane and surface as a synthetic "CC1" subtitle track once seen, additively through `PlayerSession.mediaInfo` (no new event).
**System caption respect:** map `MACaptionAppearance` (user's Style settings) onto text-cue styling by default.

**D7 — Buffering is PTS-math, not heuristics.**
Expand Down Expand Up @@ -190,13 +191,15 @@ Licensing note: engine code is **MIT** (`LICENSE`); FFmpeg is LGPL 2.1+, and the
## 6. Feature implementation notes (selected)

- **Now Playing (explicit requirement):** `NowPlayingCenter` module in `LumeEngineMedia` — `MPNowPlayingInfoCenter` (title/artwork/elapsed/duration/rate, live-stream flag) + `MPRemoteCommandCenter` (play/pause/seek/skip/next-episode hook), audio-session lifecycle, route-change and interruption handling. Works on the lock screen/Control Center including while audio/video routes to an Apple TV. Note: full-screen *video* AirPlay of FFmpeg-decoded content is not possible for third parties (system limitation); Lume's existing switch-to-AVPlayer-on-AirPlay stays, and the integration phase adds a shared app-level Now Playing service so lock-screen metadata is present in **both** engines (closing today's gap).
- **Audio width follows the route:** decoded audio leaves the engine in the layout `AudioLayoutPolicy` picks for the *negotiated* route width (`AVAudioSession.outputNumberOfChannels`, followed through route changes by `AudioOutputRoute`; stereo on macOS): discrete multichannel when the route carries it (HDMI receiver), an explicit ITU-R BS.775 fold-down (`DownmixMatrix`: centre and surrounds at −3 dB, LFE omitted, not normalized) when it does not, mono on both speakers of a stereo route. A route change refits at the next decoded frame; audio decoded for a wider route never reaches a narrower one. The engine observes the audio session only — category, activation and the preferred channel count are the host's. `PlayerSession.audioHealth` reports audio that died while video plays on (decoder gave up, renderer failed, or nothing reaching the renderer at a running playhead).
- **PiP:** `AVPictureInPictureController(contentSource: .init(sampleBufferDisplayLayer:playbackDelegate:))` from the SystemRenderer; subtitle cues composited into the sample buffers during PiP so subs survive.
- **Trickplay:** a `ThumbnailSession` — separate lightweight demux+decode (keyframes only, HW decode, downscale to ~320 px) filling a time-indexed cache; API `thumbnails(at:interval:)`. Works for VOD/catch-up; live uses the rewind buffer.
- **Live rewind (time-shift DVR):** disk-backed packet spool behind a custom AVIO layer; seeking within spool = local demux, seeking to live edge = passthrough. Configurable size/duration caps.
- **Zero-delay switching:** `prepare(next:)` opens a second session through first-frame-decoded, then atomically swaps renderer attachment; old session torn down async. (Also powers Lume's next-episode auto-advance and channel zapping.)
- **Precache:** same AVIO disk-cache layer, `precache(url:leadingSeconds:)` background task with LRU budget.
- **Upscaling:** MetalFX spatial scaler stage (D5), quality/perf switch, auto-disable on thermal pressure (`ProcessInfo.thermalState`).
- **HDR:** HDR10/HLG passthrough via sample-buffer attachments. DV with an HDR10-compatible base layer (P8.1, P10.1, …) plays as HDR10. DV with an IPTPQc2 base layer (P5, P10.0, P20 — compatibility ID 0) is converted to HDR10 before enqueue by a Metal compute pass (`Decode/DolbyVisionConverter`) that applies each frame's RPU reshaping (polynomial + MMR) and the IPT→LMS→BT.2020 matrices FFmpeg exports as `AV_FRAME_DATA_DOVI_METADATA`; the renderer then tone-maps it like any HDR10 stream. Dynamic DV display management (L1/L2/L8 trims) is not carried over. HDR10 static metadata (mastering display, MaxCLL/MaxFALL — from the bitstream's SEI or the container, falling back to a DV RPU's level 6 for the light level) is attached to every delivered pixel buffer, since neither VideoToolbox nor the engine's own buffers carry it otherwise, and so reaches the format description; HDR10+ dynamic metadata travels per sample as `kCMSampleAttachmentKey_HDR10PlusPerFrameData` (`Decode/HDRMetadata`). Whether the renderer acts on HDR10+ for decoded (not compressed) samples is still to be confirmed on an HDR10+ display. EDR headroom handling on macOS/visionOS Metal path.
- **Match Content (tvOS):** a sample-buffer pipeline gets no automatic display mode switching, so the engine derives `ContentDisplayCriteria` from its own output — refresh rate after deinterlacing (measured from enqueued frame PTS once there are frames, snapped to 23.976/24/25/29.97/30/50/59.94/60 or no preference), dynamic range of what is presented (IPT Dolby Vision profiles → HDR10 after conversion; other Dolby Vision profiles → their base layer's range by compatibility ID, cross-checked with the frames' transfer function; otherwise the frames' or the stream's transfer function) — and `DisplayCriteriaMatcher` sets it as the window's `AVDisplayManager.preferredDisplayCriteria` via `AVDisplayCriteria(refreshRate:formatDescription:)` (tvOS 17+). Dolby Vision mode is never requested: the engine presents the base layer, and only a real DV output path (§7 spike) should ask for it. The start is held, bounded, until the TV finishes switching. Opt-in; mid-stream changes are not chased (each switch blanks the TV).
- **360°/VR:** MetalRenderer projection (equirect sphere / fisheye), CoreMotion + gesture camera; visionOS immersive presentation later.
- **Blu-ray ISO/DVD:** libbluray/libdvdnav behind the `AVIOProtocol` seam; menu-less title playback first (longest-title heuristic), chapters mapped to the chapter API.
- **AI subtitles:** whisper.cpp (Metal) tapping the decoded-audio channel; streaming word-level segments feed the subtitle model as a synthetic track; on-device translation via Translation.framework.
Expand Down
30 changes: 30 additions & 0 deletions Sources/LumeEngine/LumePlayer.swift
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,24 @@ public final class LumePlayer {
/// layer tree directly (UIKit/AppKit).
public var displayLayer: LumeDisplayLayer? { session?.renderer.displayLayer }

/// tvOS: switch the TV to each loaded stream's refresh rate and dynamic
/// range (Match Content) while it plays in a `LumePlayerView`, which
/// supplies the window; `stop()` and the view going away hand the display
/// back. Set before `load`. Off by default; ignored on other platforms.
public var matchesDisplayCriteria = false {
didSet {
#if os(tvOS)
if !matchesDisplayCriteria { displayMatcher.reset() }
#endif
}
}

#if os(tvOS)
/// One per player, so loading the next stream does not hand the display
/// back in between (see `DisplayCriteriaMatcher`).
let displayMatcher = DisplayCriteriaMatcher()
#endif

private var session: PlayerSession?
private var eventTask: Task<Void, Never>?
private var tickTask: Task<Void, Never>?
Expand Down Expand Up @@ -86,6 +104,10 @@ public final class LumePlayer {

do {
let info = try await session.open(url: url)
#if os(tvOS)
// Before the app's play(): the start is held while the TV switches.
if matchesDisplayCriteria { await displayMatcher.match(session) }
#endif
mediaInfo = info
duration = info.duration.map(MediaTime.seconds)
return info
Expand Down Expand Up @@ -135,6 +157,9 @@ public final class LumePlayer {
}

public func stop() async {
#if os(tvOS)
displayMatcher.reset()
#endif
await teardownSession()
state = .idle
position = 0
Expand Down Expand Up @@ -176,6 +201,11 @@ public final class LumePlayer {
private func tick() async {
guard let session else { return }
position = await session.position
// Tracks can turn up after open (closed captions in the video).
if let mediaInfo, let current = await session.mediaInfo,
current.tracks.count != mediaInfo.tracks.count {
self.mediaInfo = current
}

let now = session.renderer.currentTime
if MediaTime.isValid(now) {
Expand Down
Loading
Loading