Repository navigation
feat: DVB, PGS and DVD bitmap subtitles as positioned image cues - #55
Merged
Merged
Conversation
The audio decoder read the negotiated output width once, at creation, so a route change mid-playback (HDMI receiver to AirPlay, HomePods) kept feeding the old width. AudioOutputRoute now follows the shared audio session's negotiated width through route-change notifications (stereo on macOS); the decoder refits per AudioLayoutPolicy at the next frame boundary, with an explicit ITU-R BS.775 DownmixMatrix (centre and surrounds -3 dB, LFE omitted) set into swresample, and mono goes out on both speakers. When the route narrows, the renderer keeps audio decoded for the wider route away from the renderer. PlayerSession.audioHealth (also in Diagnostics) reports audio that died while video plays on: decoder failed, renderer failed, or nothing reaching the renderer at a running playhead (AudioLossDetector), the latter also as .error with the new EngineError.Code.audioLost. Additive API only.
Cue lifecycle: decoders now hand the store display updates (start, end or none, text, whether the track replaces the whole screen), and a pure SubtitleTimeline decides how long each shows. Open cues (UINT32_MAX end: PGS, DVB, captions) end at the next update of their track instead of staying up for 49 days; the empty update replacing tracks send to clear the screen now ends what is showing instead of being dropped; open cues are capped at 16 s and ended by a seek; re-read ranges stay deduplicated; the store is bounded at 8192 cues so a live session no longer grows without limit. Subtitle decoders flush their state across seek serials. SubtitleStore is now checked Sendable (Mutex) instead of @unchecked. Closed captions: CEA-608 carried as A/53 side data on decoded video frames is lifted off in VideoDecoder.receiveFrames by a ClosedCaptionTap, stamped with the frame's PTS and trySend-ed to a SubtitleDecoder running ccaption in real-time mode, anchored at the packet time. The video lane never waits on it. A synthetic "CC1" subtitle track (TrackInfo.embeddedClosedCaptionsIndex) appears in PlayerSession.mediaInfo once caption content is seen, and selectSubtitleTrack accepts it. Stacked on #46.
bilipp
force-pushed
the
feat/subtitle-cues-and-captions
branch
from
October 12, 2026 22:35
987170c to
6375c53
Compare
SUBTITLE_BITMAP rects from avcodec_decode_subtitle2 were dropped. They now become SubtitleImages on the cue: a premultiplied sRGB CGImage (palette ARGB -> premultiplied BGRA) and a rect normalized to the subtitle's authoring canvas as the decoder reports it after decoding (DVB display definition, PGS presentation size, DVD size:), with a 720x576 fallback grown to fit the rects. The DVB context's size is zeroed before opening so the stream's display definition decides it. SubtitleCue gains images (defaulted, empty for text cues); bitmap cues carry empty text, so text-only consumers keep working and show nothing. The timeline treats an update with images as content rather than a clear, deduplicates re-decoded bitmaps by position and size, and bounds bitmap memory at 64 MB on top of the cue count. LumePlayer exposes the active cues' images as subtitleImages; LumePlayerView has no subtitle overlay, so drawing stays with the host. Stacked on #46.
bilipp
force-pushed
the
feat/bitmap-subtitles
branch
from
October 12, 2026 22:38
27414a0 to
be16ad1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #46. This PR is built on top of the subtitle cue-lifecycle and closed-captions PR (branch
feat/subtitle-cues-and-captions) and targets that branch, because bitmap formats depend on its timeline: PGS has no end time, and DVB/PGS clear the screen with empty updates. Merge that PR first; GitHub then retargets this one.Summary
DVB, PGS and DVD subtitles reached
avcodec_decode_subtitle2, and theirSUBTITLE_BITMAPrects were then dropped. They now become images on the cue:SubtitleBitmap). Palette-indexed pixels (data[0], honouringlinesize) are mapped through the straight-alpha ARGB palette (data[1]) to premultiplied BGRA (byteOrder32Little | premultipliedFirst, sRGB), and wrapped in aCGImage. Straight alpha drawn as premultiplied leaves a bright fringe around every glyph. An index past the palette is treated as transparent, and fully transparent rects are skipped.rectis normalized (0…1, top-left origin) to the subtitle's authoring canvas, never the video's pixel size. The canvas is the size the decoder reports after decoding: the DVB display definition segment (720×576 by default), the PGS presentation's video size, or the DVDsize:. The DVB context's size is zeroed before opening so the stream's display definition decides it; otherwise a probe-time 720×576 would pin an HD page to an SD canvas. If no canvas is reported, it falls back to 720×576, grown to contain the rects.SubtitleCue.images: [SubtitleImage](defaulted to[]). Bitmap cues carry emptytext, so text consumers (Lume today) keep working and show nothing for them.CGImages. Bitmap memory gets its own bound (imageByteBudget, 64 MB) on top of the cue count: a full-HD PGS line is close to a megabyte decoded, so a count alone could grow to gigabytes on a film.LumePlayerViewhas no subtitle overlay (it is bare video), so drawing stays with the host, as documented onSubtitleCue/SubtitleImage: draw each image at itsrectscaled to the video's display area.LumePlayerexposes the active cues' images assubtitleImagesfor hosts using the facade. It changes only when the cue changes, so observation doesn't churn.API
Additive only; Lume compiles unchanged:
SubtitleImage(image: CGImage,rect: CGRect),SubtitleCue.images,LumePlayer.subtitleImages.SubtitleCue.initgains a defaultedimages:parameter, so existing calls compile unchanged.LumePlayer.subtitleTextnow skips empty cue texts when joining, andstop()clearssubtitleText/subtitleImages.PlayerEventcases, no renamed or removed API.FFmpeg build
No build changes.
dvbsub,pgssubanddvdsubare in every slice of the shipped xcframework (checked withnm).Tests
BitmapSubtitleConversionTests: premultiplication; stride and out-of-palette indices;CGImagebyte layout read back; rect normalization; a bitmap update is a cue and not a clear; re-decode dedupe; the image byte budget.BitmapSubtitlePlaybackTests, end to end overpgs.mkv,dvbsub.tsanddvdsub.mkv. All three carry the same two bitmaps on a 1920×1080 canvas over a 320×180 video, so a canvas taken from the video would put them far off screen. The tests check cue timing to within 40 ms, position and size to within a few canvas pixels, one image pixel per canvas pixel, pixel colours (transparent, opaque white, half-transparent red premultiplied), and the clears at 3 s and 6 s. FFmpeg's DVD decoder crops fully transparent borders, and the test allows for that.generate-fixtures.shwrites the PGS original by hand with a shortpython3 -Iscript (the same prerequisite as the caption fixture). ffmpeg then copies it into MKV and converts it to DVB (with a display definition segment) and DVD, using-fix_sub_duration.Verification
swift test: 169 tests in 25 suites pass.xcodebuild buildforgeneric/platform=tvOSandgeneric/platform=iOS: succeeded with no new warnings (theSystemRenderermain-actor warnings were there before).Not verified
imagesyet. Lume shows nothing for bitmap tracks, as before, until it composites them.