Skip to content

feat: subtitle cue lifecycle and CEA-608 closed captions from the video - #52

Merged
bilipp merged 5 commits into
mainfrom
feat/subtitle-cues-and-captions
Oct 12, 2026
Merged

bilipp merged 5 commits into
mainfrom
feat/subtitle-cues-and-captions

Conversation

@bilipp

@bilipp bilipp commented Oct 12, 2026

Copy link
Copy Markdown
Owner

Stacked on #46.

Summary

Cue lifecycle. How long a subtitle shows is now decided in one place, a pure SubtitleTimeline behind SubtitleStore. Decoders hand it display updates (SubtitleEvent: start, end or none, text, and whether the track replaces the whole screen with each update). The cases this fixes, all of which reached the screen before:

  • Cues with no end (end_display_time == UINT32_MAX: PGS always, closed captions in real-time mode, DVB "until the next page") were turned into start + 49 days, so they never went away and stacked with every later cue. Open cues now end at the next update of their track, capped at 16 s (about how long a TV caption decoder leaves a caption without caption data).
  • Clears were dropped. The empty update that DVB, PGS, DVD and closed captions send to clear the screen was thrown away as "no text", so the previous cue stayed up for its whole timeout. On those tracks an empty update now ends what is showing.
  • Overlap on replacing tracks. Each update on a caption or bitmap track now ends everything shown before it, including when a re-read range (after a seek) delivers updates out of order. Text tracks (SRT/ASS/WebVTT/mov_text) keep their overlaps. Only their open cues are ended by the next cue.
  • Seeks. A seek ends cues whose end the source never gave. Cues with a real end stay, because they describe the timeline. Subtitle decoders now flush their codec state when the seek serial changes, so caption rows and PGS/DVB object sets don't carry across a seek. Re-read ranges stay deduplicated, matched on start and text when an end was bounded later. On a replacing track the latest decode of an instant wins.
  • Unbounded growth. The store keeps at most 8192 cues and drops them from the end farther from where cues are being added. That is the front on a live timeline, and the back after a seek far back in a long file. A day of live roll-up captions used to keep every update it ever decoded.

SubtitleStore is now checked Sendable (a Mutex over the timeline) instead of @unchecked Sendable.

CEA-608 closed captions from the video. US/ATSC broadcast and much IPTV carry captions as A/53 user data inside the video, not as a subtitle stream. FFmpeg's video decoders expose it as AV_FRAME_DATA_A53_CC side data.

  • ClosedCaptionTap reads that side data from every decoded frame in VideoDecoder.receiveFrames, before deinterlacing and Dolby Vision conversion. Each payload becomes a packet stamped with the frame's PTS (engine µs, display order) and goes out with trySend to an ordinary SubtitleDecoder running FFmpeg's ccaption. The video thread never waits: a lane that falls behind drops captions, never frames. A frame without captions costs one side-data lookup.
  • ccaption runs in real-time mode with field 1 (CC1/CC2) pinned. Buffered mode only emits a caption once the next change has ended it, which is too late for playback that runs a fraction of a second behind decode. Real-time mode stamps an update with the previous change's time, so caption cues are anchored at the packet that caused them instead.
  • Track discovery: no open-time probe can tell whether a stream has captions (FFmpeg 9 no longer reports it). So the synthetic track (TrackInfo.embeddedClosedCaptionsIndex, title "CC1", codec eia_608) appears in PlayerSession.mediaInfo once caption content (not just null padding) has been seen. .opened keeps the open-time list: no new PlayerEvent case. selectSubtitleTrack(TrackInfo.embeddedClosedCaptionsIndex) works before the track is listed too, so a host can keep captions on across channels. LumePlayer refreshes its mediaInfo when the track count changes.
  • No backfill seek when captions are selected: the captions ride in frames decoded only just ahead of the playhead, and live sources don't survive a seek.

API

Additive only; Lume compiles unchanged:

  • New: TrackInfo.embeddedClosedCaptionsIndex, TrackInfo.isEmbeddedClosedCaptions.
  • MediaInfo.tracks is now public internal(set) var (was let). Reading it is unchanged.
  • PlayerSession.mediaInfo may now list one more subtitle track than .opened did, once captions are detected.
  • SubtitleStore conforms to Sendable (was @unchecked Sendable). SubtitleDecoder's public init is unchanged (now a convenience init).
  • No new PlayerEvent cases, no renamed or removed API, no configuration changes.

Lume follow-up (not in this PR): Lume builds its subtitle menu from the MediaInfo that open returns. To offer CC it has to re-read session.mediaInfo (for example in its 100 ms tick, when the subtitle track count changes) and republish its track options.

FFmpeg build

No build changes. The shipped xcframework (all slices checked with nm) contains ccaption, dvbsub, dvdsub and pgssub. It has no teletext decoder (libzvbi_teletext needs libzvbi, which the build does not enable), so DVB teletext subtitle tracks still decode to nothing. FFmpeg has no CEA-708 (DTVCC) decoder at all; ccaption reads the 608 data that 708 streams carry alongside.

Tests

  • SubtitleTimelineTests (16, pure): span computation for UINT32_MAX, 0 and explicit ends; open cue ended by the next update; clears; the cap; out-of-order bounding; re-read dedupe; same-instant replacement; text overlap kept; open text cue ended by the next cue; seek closing only open cues; bounded growth in both directions.
  • ClosedCaptionTests: caption detection (null padding, field 2, DTVCC and invalid triples don't count); in-process ccaption decode of a pop-on script, with cue start and end exact to the frame; the tap's detection, forwarding and PTS/serial stamping on a real AVFrame with A/53 side data.
  • ClosedCaptionPlaybackTests (end to end, VideoToolbox and software): the CC track appears in mediaInfo and not in .opened, the cues decode, and start and end are within one frame of 1 s/3 s. A caption-free source lists no CC track.
  • New fixture captions.ts: an H.264 TS with CEA-608 pop-on captions in A/53 SEI. No ffmpeg CLI option writes captions into the video, so generate-fixtures.sh puts the SEI in front of each picture with a short python3 -I script. That makes python3 a fixture prerequisite (documented in AGENTS.md; the Xcode CLT python is enough). FFmpeg's own movie=…[out+subcc] reads the same two captions back from it.

Verification

  • swift test: 162 tests in 23 suites pass. One early run under parallel load failed the end-to-end caption test because the test checked a cue count before the post-seek pass had decoded. The test now waits on the second caption's erase instead; it passed in 3 isolated runs and in the full suite.
  • xcodebuild build for generic/platform=tvOS and generic/platform=iOS: succeeded with no new warnings (the SystemRenderer main-actor warnings were there before).

Not verified

  • Real ATSC and IPTV broadcasts with captions, on device. A/53 on VideoToolbox frames was verified on macOS only, though FFmpeg attaches SEI side data before the hwaccel, so tvOS and iOS should behave the same.
  • MPEG-2 and HEVC caption carriage. It goes through the same side-data type, but no fixture covers it.
  • Roll-up and paint-on captions in practice. Only pop-on is covered by fixtures, and roll-up is emitted at most every 200 ms (ccaption's real-time latency).
  • Lume UI: nothing in Lume offers the CC track yet (see the follow-up above).

Base automatically changed from feat/live-playout-and-startup-events to main October 12, 2026 22:14
…utput on tvOS

PlayerSession.displayCriteria describes the display mode the selected video
wants, from what the engine outputs: the refresh rate after deinterlacing
(measured from enqueued frames once there are any, snapped to the film and
broadcast grid or no preference) and the dynamic range after Dolby Vision
conversion. DisplayCriteriaMatcher (tvOS, opt-in) applies it through the
window's AVDisplayManager and holds the session's start, bounded, while the
TV switches; PlayerSession.holdStart/releaseStartHold carry that hold.
LumePlayer.matchesDisplayCriteria wires it into the facade. TrackInfo.Video
gains isInterlaced and the Dolby Vision configuration record.
The engine has no Dolby Vision output path: it presents the base layer of
every profile it does not convert. Dolby Vision streams now ask for that
base layer's range (compatibility ID before the first frame, the frames'
transfer function after), and the synthesized dvh1 description is gone.
DolbyVisionDetection.Record now comes from the public
DolbyVisionConfiguration instead of parsing AV_PKT_DATA_DOVI_CONF a second
time, and DolbyVisionConfiguration.baseLayerIsIPT asks Record.claimsIPT, so
the profile 5/10.0/20 rule lives in one place. Its doc now says what it is
since conversion became a per-frame decision: the stream's claim.
The audio decoder read the negotiated output width once, at creation, so a
route change mid-playback (HDMI receiver to AirPlay, HomePods) kept feeding
the old width. AudioOutputRoute now follows the shared audio session's
negotiated width through route-change notifications (stereo on macOS); the
decoder refits per AudioLayoutPolicy at the next frame boundary, with an
explicit ITU-R BS.775 DownmixMatrix (centre and surrounds -3 dB, LFE
omitted) set into swresample, and mono goes out on both speakers. When the
route narrows, the renderer keeps audio decoded for the wider route away
from the renderer.

PlayerSession.audioHealth (also in Diagnostics) reports audio that died
while video plays on: decoder failed, renderer failed, or nothing reaching
the renderer at a running playhead (AudioLossDetector), the latter also as
.error with the new EngineError.Code.audioLost. Additive API only.
Cue lifecycle: decoders now hand the store display updates (start, end or
none, text, whether the track replaces the whole screen), and a pure
SubtitleTimeline decides how long each shows. Open cues (UINT32_MAX end:
PGS, DVB, captions) end at the next update of their track instead of
staying up for 49 days; the empty update replacing tracks send to clear
the screen now ends what is showing instead of being dropped; open cues
are capped at 16 s and ended by a seek; re-read ranges stay deduplicated;
the store is bounded at 8192 cues so a live session no longer grows
without limit. Subtitle decoders flush their state across seek serials.
SubtitleStore is now checked Sendable (Mutex) instead of @unchecked.

Closed captions: CEA-608 carried as A/53 side data on decoded video frames
is lifted off in VideoDecoder.receiveFrames by a ClosedCaptionTap, stamped
with the frame's PTS and trySend-ed to a SubtitleDecoder running ccaption
in real-time mode, anchored at the packet time. The video lane never
waits on it. A synthetic "CC1" subtitle track
(TrackInfo.embeddedClosedCaptionsIndex) appears in PlayerSession.mediaInfo
once caption content is seen, and selectSubtitleTrack accepts it.

Stacked on #46.
@bilipp
bilipp force-pushed the feat/subtitle-cues-and-captions branch from 987170c to 6375c53 Compare October 12, 2026 22:35
@bilipp
bilipp merged commit 369328c into main Oct 12, 2026
5 checks passed
@bilipp
bilipp deleted the feat/subtitle-cues-and-captions branch October 12, 2026 22:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant