Repository navigation
feat: subtitle cue lifecycle and CEA-608 closed captions from the video - #52
Merged
Merged
Conversation
…utput on tvOS PlayerSession.displayCriteria describes the display mode the selected video wants, from what the engine outputs: the refresh rate after deinterlacing (measured from enqueued frames once there are any, snapped to the film and broadcast grid or no preference) and the dynamic range after Dolby Vision conversion. DisplayCriteriaMatcher (tvOS, opt-in) applies it through the window's AVDisplayManager and holds the session's start, bounded, while the TV switches; PlayerSession.holdStart/releaseStartHold carry that hold. LumePlayer.matchesDisplayCriteria wires it into the facade. TrackInfo.Video gains isInterlaced and the Dolby Vision configuration record.
The engine has no Dolby Vision output path: it presents the base layer of every profile it does not convert. Dolby Vision streams now ask for that base layer's range (compatibility ID before the first frame, the frames' transfer function after), and the synthesized dvh1 description is gone.
DolbyVisionDetection.Record now comes from the public DolbyVisionConfiguration instead of parsing AV_PKT_DATA_DOVI_CONF a second time, and DolbyVisionConfiguration.baseLayerIsIPT asks Record.claimsIPT, so the profile 5/10.0/20 rule lives in one place. Its doc now says what it is since conversion became a per-frame decision: the stream's claim.
The audio decoder read the negotiated output width once, at creation, so a route change mid-playback (HDMI receiver to AirPlay, HomePods) kept feeding the old width. AudioOutputRoute now follows the shared audio session's negotiated width through route-change notifications (stereo on macOS); the decoder refits per AudioLayoutPolicy at the next frame boundary, with an explicit ITU-R BS.775 DownmixMatrix (centre and surrounds -3 dB, LFE omitted) set into swresample, and mono goes out on both speakers. When the route narrows, the renderer keeps audio decoded for the wider route away from the renderer. PlayerSession.audioHealth (also in Diagnostics) reports audio that died while video plays on: decoder failed, renderer failed, or nothing reaching the renderer at a running playhead (AudioLossDetector), the latter also as .error with the new EngineError.Code.audioLost. Additive API only.
Cue lifecycle: decoders now hand the store display updates (start, end or none, text, whether the track replaces the whole screen), and a pure SubtitleTimeline decides how long each shows. Open cues (UINT32_MAX end: PGS, DVB, captions) end at the next update of their track instead of staying up for 49 days; the empty update replacing tracks send to clear the screen now ends what is showing instead of being dropped; open cues are capped at 16 s and ended by a seek; re-read ranges stay deduplicated; the store is bounded at 8192 cues so a live session no longer grows without limit. Subtitle decoders flush their state across seek serials. SubtitleStore is now checked Sendable (Mutex) instead of @unchecked. Closed captions: CEA-608 carried as A/53 side data on decoded video frames is lifted off in VideoDecoder.receiveFrames by a ClosedCaptionTap, stamped with the frame's PTS and trySend-ed to a SubtitleDecoder running ccaption in real-time mode, anchored at the packet time. The video lane never waits on it. A synthetic "CC1" subtitle track (TrackInfo.embeddedClosedCaptionsIndex) appears in PlayerSession.mediaInfo once caption content is seen, and selectSubtitleTrack accepts it. Stacked on #46.
bilipp
force-pushed
the
feat/subtitle-cues-and-captions
branch
from
October 12, 2026 22:35
987170c to
6375c53
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #46.
Summary
Cue lifecycle. How long a subtitle shows is now decided in one place, a pure
SubtitleTimelinebehindSubtitleStore. Decoders hand it display updates (SubtitleEvent: start, end or none, text, and whether the track replaces the whole screen with each update). The cases this fixes, all of which reached the screen before:end_display_time == UINT32_MAX: PGS always, closed captions in real-time mode, DVB "until the next page") were turned intostart + 49 days, so they never went away and stacked with every later cue. Open cues now end at the next update of their track, capped at 16 s (about how long a TV caption decoder leaves a caption without caption data).SubtitleStoreis now checkedSendable(aMutexover the timeline) instead of@unchecked Sendable.CEA-608 closed captions from the video. US/ATSC broadcast and much IPTV carry captions as A/53 user data inside the video, not as a subtitle stream. FFmpeg's video decoders expose it as
AV_FRAME_DATA_A53_CCside data.ClosedCaptionTapreads that side data from every decoded frame inVideoDecoder.receiveFrames, before deinterlacing and Dolby Vision conversion. Each payload becomes a packet stamped with the frame's PTS (engine µs, display order) and goes out withtrySendto an ordinarySubtitleDecoderrunning FFmpeg'sccaption. The video thread never waits: a lane that falls behind drops captions, never frames. A frame without captions costs one side-data lookup.ccaptionruns in real-time mode with field 1 (CC1/CC2) pinned. Buffered mode only emits a caption once the next change has ended it, which is too late for playback that runs a fraction of a second behind decode. Real-time mode stamps an update with the previous change's time, so caption cues are anchored at the packet that caused them instead.TrackInfo.embeddedClosedCaptionsIndex, title "CC1", codeceia_608) appears inPlayerSession.mediaInfoonce caption content (not just null padding) has been seen..openedkeeps the open-time list: no newPlayerEventcase.selectSubtitleTrack(TrackInfo.embeddedClosedCaptionsIndex)works before the track is listed too, so a host can keep captions on across channels.LumePlayerrefreshes itsmediaInfowhen the track count changes.API
Additive only; Lume compiles unchanged:
TrackInfo.embeddedClosedCaptionsIndex,TrackInfo.isEmbeddedClosedCaptions.MediaInfo.tracksis nowpublic internal(set) var(waslet). Reading it is unchanged.PlayerSession.mediaInfomay now list one more subtitle track than.openeddid, once captions are detected.SubtitleStoreconforms toSendable(was@unchecked Sendable).SubtitleDecoder's public init is unchanged (now a convenience init).PlayerEventcases, no renamed or removed API, no configuration changes.Lume follow-up (not in this PR): Lume builds its subtitle menu from the
MediaInfothatopenreturns. To offer CC it has to re-readsession.mediaInfo(for example in its 100 ms tick, when the subtitle track count changes) and republish its track options.FFmpeg build
No build changes. The shipped xcframework (all slices checked with
nm) containsccaption,dvbsub,dvdsubandpgssub. It has no teletext decoder (libzvbi_teletextneeds libzvbi, which the build does not enable), so DVB teletext subtitle tracks still decode to nothing. FFmpeg has no CEA-708 (DTVCC) decoder at all;ccaptionreads the 608 data that 708 streams carry alongside.Tests
SubtitleTimelineTests(16, pure): span computation forUINT32_MAX, 0 and explicit ends; open cue ended by the next update; clears; the cap; out-of-order bounding; re-read dedupe; same-instant replacement; text overlap kept; open text cue ended by the next cue; seek closing only open cues; bounded growth in both directions.ClosedCaptionTests: caption detection (null padding, field 2, DTVCC and invalid triples don't count); in-processccaptiondecode of a pop-on script, with cue start and end exact to the frame; the tap's detection, forwarding and PTS/serial stamping on a realAVFramewith A/53 side data.ClosedCaptionPlaybackTests(end to end, VideoToolbox and software): the CC track appears inmediaInfoand not in.opened, the cues decode, and start and end are within one frame of 1 s/3 s. A caption-free source lists no CC track.captions.ts: an H.264 TS with CEA-608 pop-on captions in A/53 SEI. No ffmpeg CLI option writes captions into the video, sogenerate-fixtures.shputs the SEI in front of each picture with a shortpython3 -Iscript. That makes python3 a fixture prerequisite (documented in AGENTS.md; the Xcode CLT python is enough). FFmpeg's ownmovie=…[out+subcc]reads the same two captions back from it.Verification
swift test: 162 tests in 23 suites pass. One early run under parallel load failed the end-to-end caption test because the test checked a cue count before the post-seek pass had decoded. The test now waits on the second caption's erase instead; it passed in 3 isolated runs and in the full suite.xcodebuild buildforgeneric/platform=tvOSandgeneric/platform=iOS: succeeded with no new warnings (theSystemRenderermain-actor warnings were there before).Not verified
ccaption's real-time latency).