Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,8 +41,12 @@ swift test # full suite
swift test --filter TimestampUnwrapperTests # one suite
swift test --filter DecoderTests/testName # one test
swift run LumeEngineDemo # macOS demo app with diagnostics HUD
swift run lumectl probe <url> # tracks, codecs, live/seekable, HDR tags
swift run lumectl play <url> --duration 60 # headless run: events, heartbeat, startup, exit status
```

`lumectl` reproduces a provider issue without the app: it drives a `PlayerSession` headlessly (muted) and prints what the engine reports — typed events, the `Diagnostics` heartbeat with the delivery rate, the `StartupTimeline`, `lastStallReason` on every stall, the `EngineError` (with its HTTP class) on failure. Exit status: 0 ok, 3 open failed, 4 playback failed, 5 stalled. URLs are redacted to scheme and host unless `--show-urls`, since IPTV URLs carry credentials — keep it that way. Its logic lives in `Sources/LumeCtlCore` (tested from `LumeCtlTests.swift`); `lumectl --help` lists the options. When a Lume report arrives, a `lumectl play` run against the same URL (with `--json` or `--record-events run.jsonl`) is usually the quickest way to tell an engine problem from a provider one.

Prerequisites:
- `BinaryDependencies/FFmpeg.xcframework` must exist before anything compiles. Build it once (or after bumping `build/versions.json`):
```bash
Expand All @@ -51,7 +55,7 @@ Prerequisites:
build/scripts/build-ffmpeg.sh macos-arm64 # one slice is enough for local dev
build/scripts/make-xcframework.sh
```
- Tests need a host `ffmpeg` CLI (Homebrew) — fixtures are generated lazily on first test run by `Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh` into `TestStreams/generated/` (gitignored, idempotent). `Fixtures.swift` finds Homebrew ffmpeg automatically; override with the `FFMPEG` env var. The closed-caption fixture (`captions.ts`) also needs `python3` (the one Xcode's command-line tools provide is enough), because no ffmpeg CLI option writes captions into the video; without it that fixture is skipped and its tests fail as missing.
- Tests need a host `ffmpeg` CLI (Homebrew) — fixtures are generated lazily on first test run by `Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh` into `TestStreams/generated/` (gitignored, idempotent). `Fixtures.swift` finds Homebrew ffmpeg automatically; override with the `FFMPEG` env var. The closed-caption and bitmap-subtitle fixtures (`captions.ts`, `pgs.mkv`, `dvbsub.ts`, `dvdsub.mkv`) also need `python3` (the one Xcode's command-line tools provide is enough), because no ffmpeg CLI option writes captions into the video or renders subtitle bitmaps; without it those fixtures are skipped and their tests fail as missing.

CI (`.github/workflows/ci.yml`) builds the macOS FFmpeg slice (cached on `build/versions.json` + scripts + patches hashes) and runs `swift test` with a strict one-thread cooperative pool (see above), checks the Markdown docs offline (`.github/scripts/check-docs.py`: relative links and in-repo anchors resolve, `CLAUDE.md` is still the `AGENTS.md` symlink — run it after editing docs), then cross-compiles the library for iOS and tvOS via `xcodebuild` (compile-only, one cached slice each — visionOS is left to local verification because that matrix entry was never confirmed on a runner — the xrOS SDK ships with Xcode, but a visionOS *destination* also needs Xcode's visionOS platform component); tagged releases build all 10 platform slices and attach the xcframework.

Expand Down Expand Up @@ -89,6 +93,7 @@ This split is the core invariant of the codebase:
- **Dolby Vision with an IPT base layer is converted, not tagged.** Profiles 5, 10.0 and 20 (compatibility ID 0) carry IPTPQc2, which shows pink/green as YCbCr. No colour attachment fixes that — the RPU reshaping changes per scene and Apple's decoders don't apply it to decoded frames — so `Decode/DolbyVisionConverter` turns each frame into HDR10 on the GPU in `VideoDecoder.deliver`, the one point every frame (hardware, software, deinterlaced) passes with its side data. Detection is the stream's DV configuration record *or* an `AVCOL_SPC_IPT_C2` frame; many P5 encodes leave the VUI matrix unspecified, so don't reduce it to the tag. HDR10-compatible profiles (8.1 etc.) must stay on the zero-copy path. GPU refusals are per-frame (iOS refuses a backgrounded app); only a converter that can't exist gives up for the session.
- **Audio is fitted to the negotiated route width, never the route's capability.** `AudioDecoder` reads `AudioOutputRoute.outputChannels` (the session's `outputNumberOfChannels`, followed through route-change notifications) per frame and remixes per `AudioLayoutPolicy` with an explicit `DownmixMatrix` — a fold-down mixes centre and surrounds in, it never drops channels. Don't switch it to `maximumOutputNumberOfChannels`: a renderer fed more channels than the session outputs failed outright on tvOS. The engine only observes the audio session; setting its category, activation or preferred channel count is the host's job (Lume does it). When the route narrows, `SystemRenderer` keeps audio decoded for the wider route out of the renderer.
- **How long a subtitle shows is the timeline's decision, not the decoder's.** Decoders hand `Subtitle/SubtitleStore` a `SubtitleEvent` (start, end *or none*, text, whether the track replaces the whole screen with each update); `Subtitle/SubtitleTimeline` decides the rest: an open cue ends at the next update of its track (capped at 16 s), an empty update on a closed-caption/DVB/PGS/DVD track is a clear that ends what is showing, a seek ends open cues, re-read ranges are deduplicated, and the store is bounded. Don't turn `end_display_time == UINT32_MAX` into a number in a decoder or drop empty updates there — both put stuck or stacked cues on screen.
- **Bitmap subtitles (DVB, PGS, DVD) are images on the cue, positioned by their own canvas.** `Subtitle/SubtitleBitmap` turns `SUBTITLE_BITMAP` rects into premultiplied sRGB `CGImage`s with a rect normalized to the canvas the decoder reports *after* decoding (DVB display definition, PGS presentation size, DVD `size:`), never to the video's pixel size — an HD DVB page over SD video, or PGS authored at 1080p over a 720p encode, would land off screen. The DVB context's size is zeroed before opening so the stream's display definition sets it. Bitmap cues carry empty `text` (text consumers like Lume show nothing rather than garbage); drawing them is the host's job, since `LumePlayerView` is bare video. The timeline bounds bitmap memory separately (`imageByteBudget`).
- **Closed captions leave the video lane through a tap, never a wait.** CEA-608 in A/53 side data (`AV_FRAME_DATA_A53_CC`) is lifted off each decoded frame in `VideoDecoder.receiveFrames` (before deinterlacing) by `Subtitle/ClosedCaptionTap`, stamped with the frame's PTS, and `trySend`-ed to an ordinary `SubtitleDecoder` running `ccaption` in real-time mode — a slow caption lane drops captions, never frames. The track is synthetic (`TrackInfo.embeddedClosedCaptionsIndex`) and appears in `PlayerSession.mediaInfo` once caption content is seen; `.opened` keeps the open-time track list, because no new `PlayerEvent` case may be added for it. Only CEA-608 field 1 (CC1/CC2) is decoded: FFmpeg has no CEA-708 (DTVCC) decoder, and teletext needs libzvbi, which the build does not include.
- **Zero global mutable state:** all configuration is `PlayerConfiguration` passed at session creation. No `static var` knobs.
- **No force unwraps / `as!` / `fatalError` in engine paths**; failures return typed `EngineError`. No `assertionFailure` in production channel/data paths — misuse degrades, never crashes.
Expand Down
5 changes: 3 additions & 2 deletions PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,8 @@ LumeEngine/
│ ├── LumeEngineRender/ # SystemRenderer, MetalRenderer, shaders
│ ├── LumeEngineSubtitles/ # libass bridge, parsers, cue model, styling
│ ├── LumeEngineMedia/ # Now Playing, PiP, audio session, route handling
│ └── LumeEngine/ # public facade: LumePlayer, LumePlayerView, config, events
│ ├── LumeEngine/ # public facade: LumePlayer, LumePlayerView, config, events
│ └── LumeCtlCore/, lumectl/ # macOS triage CLI: headless session against a URL, events + diagnostics
├── BinaryDependencies/ # generated Package targets for ffmpeg/libass/dav1d xcframeworks
├── build/ # FFmpeg build pipeline (scripts, patches, configs)
├── DemoApp/ # multiplatform SwiftUI demo (streams menu, diagnostics HUD)
Expand All @@ -142,7 +143,7 @@ Brightness/contrast/saturation and MetalFX spatial upscaling operate CVPixelBuff

**D6 — One subtitle engine, composited as overlay.**
libass renders ASS/SSA (full effects, embedded fonts via attachment streams, word-by-word \k tags); PGS/DVB/VobSub decoded by FFmpeg to bitmaps; SRT/VTT parsed to styled cues rendered through the same layout box. Output = time-tagged `SubtitleImage` cues composited on a CALayer above the video surface (and into the PiP content when active; into Metal for HDR subtitle mode). Dual subtitle slots (main + secondary). External files attach as additional demux sessions.
**Cue lifecycle and closed captions (as built).** Every subtitle source — demuxed stream, sidecar file, or captions lifted from the video — reaches one time-indexed store as display updates; the store's `SubtitleTimeline` alone decides how long each shows (open cues end at the next update or a clear, capped; seeks end open cues; bounded size). CEA-608 captions carried as A/53 side data are tapped off decoded video frames without ever blocking the video lane and surface as a synthetic "CC1" subtitle track once seen, additively through `PlayerSession.mediaInfo` (no new event).
**Cue lifecycle and closed captions (as built).** Every subtitle source — demuxed stream, sidecar file, or captions lifted from the video — reaches one time-indexed store as display updates; the store's `SubtitleTimeline` alone decides how long each shows (open cues end at the next update or a clear, capped; seeks end open cues; bounded size). CEA-608 captions carried as A/53 side data are tapped off decoded video frames without ever blocking the video lane and surface as a synthetic "CC1" subtitle track once seen, additively through `PlayerSession.mediaInfo` (no new event). PGS/DVB/DVD bitmaps reach the same store as `SubtitleImage`s (premultiplied `CGImage` + rect normalized to the subtitle's authoring canvas) on cues with empty text; compositing them is still the host's job until the overlay layer described above exists.
**System caption respect:** map `MACaptionAppearance` (user's Style settings) onto text-cue styling by default.

**D7 — Buffering is PTS-math, not heuristics.**
Expand Down
18 changes: 17 additions & 1 deletion Package.swift
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,9 @@ let package = Package(
// namespace), so apps can also link engines embedding other FFmpeg builds
// without symbol collisions.
.library(name: "LumeEngine", type: .dynamic, targets: ["LumeEngine"]),
// macOS triage tool: `swift run lumectl --help`. Drives a session
// headlessly against a URL; nothing a consumer app links.
.executable(name: "lumectl", targets: ["lumectl"]),
],
targets: [
// FFmpeg 9.0.x static libraries: released artifact, or a local build when
Expand Down Expand Up @@ -116,11 +119,24 @@ let package = Package(
path: "DemoApp/Sources"
),

// lumectl's logic (argument parsing, URL redaction, the headless run,
// output), kept out of the executable so the tests can reach it.
.target(
name: "LumeCtlCore",
dependencies: ["LumeEngineCore"]
),
.executableTarget(
name: "lumectl",
dependencies: ["LumeCtlCore"]
),

.testTarget(
name: "LumeEngineCoreTests",
// CFFmpeg directly: some tests build FFmpeg structs (e.g. Dolby
// Vision metadata) to exercise the engine's readers of them.
dependencies: ["LumeEngineCore", "LumeEngine", "CFFmpeg"],
// LumeCtlCore and lumectl: the tool's tests reuse the fixtures and
// the torture server, and the Ctrl-C test runs the built binary.
dependencies: ["LumeEngineCore", "LumeEngine", "CFFmpeg", "LumeCtlCore", "lumectl"],
exclude: ["Fixtures/generate-fixtures.sh"]
),
],
Expand Down
17 changes: 17 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,10 +101,27 @@ swift run LumeEngineDemo # macOS demo with diagnostics H

Testing needs a host `ffmpeg` CLI (`brew install ffmpeg`) to generate fixtures on first run — a build-time tool, unrelated to the FFmpeg the engine links. The full 10-slice matrix (`build/scripts/platforms.sh`) is only needed for release binaries, which CI builds on tags.

## Debugging streams with lumectl

`lumectl` (macOS) plays a URL headlessly through the engine and prints what it reports, so a stream problem — a slow start, stalls, a reconnect that replays media, timestamp seams, a hardware decoder falling back, an HTTP error — can be reproduced from a terminal without the app:

```bash
swift run lumectl probe 'http://provider.example:8080/live/user/pass/1234.ts'
swift run lumectl play 'http://provider.example:8080/live/user/pass/1234.ts' --duration 60
swift run lumectl play stream.m3u8 --json --record-events run.jsonl --interval 1
```

`probe` prints the container, duration, start time, live/seekable flags and every track (codec, language, resolution, frame rate, bit depth, HDR transfer and colour tags). `play` runs a muted session and prints every `PlayerEvent` with a timestamp, a diagnostics heartbeat (queue depths, renderer health, delivery rate, timeline seams, replayed packets), the startup timeline once the first frame shows, the stalled stage on every stall, and the typed error — with its HTTP status class — on failure. `--json` switches stdout to JSON Lines; `--record-events FILE` keeps a JSON Lines copy next to the readable output.

The exit status tells the outcome apart for scripted repro loops: `0` ok, `3` the source never opened, `4` playback failed, `5` stalled (without failing), `2` bad arguments. Ctrl-C ends a run cleanly with a summary.

URLs are printed as scheme and host only (`http://provider.example:8080/[redacted].ts`), because IPTV URLs carry the account; header values are never printed. Pass `--show-urls` to see them in full. `lumectl --help` lists every option: start position, mid-stream seek, software decoding, preferred audio languages, a single lane, custom headers, user agent, referer, reconnect and timeouts.

## Layout

- `Sources/LumeEngineCore` — demux, decode, render, session, subtitles, media integration (no UI)
- `Sources/LumeEngine` — public facade: `LumePlayer`, `LumePlayerView`
- `Sources/lumectl`, `Sources/LumeCtlCore` — the `lumectl` command-line tool
- `build/` — FFmpeg build pipeline (versions manifest, scripts, patches)
- `DemoApp/` — macOS demo with diagnostics HUD
- `Tests/` — including a fault-injecting HTTP torture server
Expand Down
Loading
Loading