Skip to content

feat: DVB, PGS and DVD bitmap subtitles as positioned image cues - #55

Merged
bilipp merged 3 commits into
mainfrom
feat/bitmap-subtitles
Oct 12, 2026
Merged

bilipp merged 3 commits into
mainfrom
feat/bitmap-subtitles

Conversation

@bilipp

@bilipp bilipp commented Oct 12, 2026

Copy link
Copy Markdown
Owner

Stacked on #46. This PR is built on top of the subtitle cue-lifecycle and closed-captions PR (branch feat/subtitle-cues-and-captions) and targets that branch, because bitmap formats depend on its timeline: PGS has no end time, and DVB/PGS clear the screen with empty updates. Merge that PR first; GitHub then retargets this one.

Summary

DVB, PGS and DVD subtitles reached avcodec_decode_subtitle2, and their SUBTITLE_BITMAP rects were then dropped. They now become images on the cue:

  • Conversion (SubtitleBitmap). Palette-indexed pixels (data[0], honouring linesize) are mapped through the straight-alpha ARGB palette (data[1]) to premultiplied BGRA (byteOrder32Little | premultipliedFirst, sRGB), and wrapped in a CGImage. Straight alpha drawn as premultiplied leaves a bright fringe around every glyph. An index past the palette is treated as transparent, and fully transparent rects are skipped.
  • Positioning. Each image's rect is normalized (0…1, top-left origin) to the subtitle's authoring canvas, never the video's pixel size. The canvas is the size the decoder reports after decoding: the DVB display definition segment (720×576 by default), the PGS presentation's video size, or the DVD size:. The DVB context's size is zeroed before opening so the stream's display definition decides it; otherwise a probe-time 720×576 would pin an HD page to an SD canvas. If no canvas is reported, it falls back to 720×576, grown to contain the rects.
  • Cue model. SubtitleCue.images: [SubtitleImage] (defaulted to []). Bitmap cues carry empty text, so text consumers (Lume today) keep working and show nothing for them.
  • Timeline. An update that has images is content, not a clear. Re-decoded bitmaps (backward seek) are deduplicated by position and size, since decoding again makes new CGImages. Bitmap memory gets its own bound (imageByteBudget, 64 MB) on top of the cue count: a full-HD PGS line is close to a megabyte decoded, so a count alone could grow to gigabytes on a film.
  • Rendering. LumePlayerView has no subtitle overlay (it is bare video), so drawing stays with the host, as documented on SubtitleCue/SubtitleImage: draw each image at its rect scaled to the video's display area. LumePlayer exposes the active cues' images as subtitleImages for hosts using the facade. It changes only when the cue changes, so observation doesn't churn.

API

Additive only; Lume compiles unchanged:

  • New: SubtitleImage (image: CGImage, rect: CGRect), SubtitleCue.images, LumePlayer.subtitleImages.
  • SubtitleCue.init gains a defaulted images: parameter, so existing calls compile unchanged.
  • LumePlayer.subtitleText now skips empty cue texts when joining, and stop() clears subtitleText/subtitleImages.
  • No new PlayerEvent cases, no renamed or removed API.

FFmpeg build

No build changes. dvbsub, pgssub and dvdsub are in every slice of the shipped xcframework (checked with nm).

Tests

  • BitmapSubtitleConversionTests: premultiplication; stride and out-of-palette indices; CGImage byte layout read back; rect normalization; a bitmap update is a cue and not a clear; re-decode dedupe; the image byte budget.
  • BitmapSubtitlePlaybackTests, end to end over pgs.mkv, dvbsub.ts and dvdsub.mkv. All three carry the same two bitmaps on a 1920×1080 canvas over a 320×180 video, so a canvas taken from the video would put them far off screen. The tests check cue timing to within 40 ms, position and size to within a few canvas pixels, one image pixel per canvas pixel, pixel colours (transparent, opaque white, half-transparent red premultiplied), and the clears at 3 s and 6 s. FFmpeg's DVD decoder crops fully transparent borders, and the test allows for that.
  • Fixtures: ffmpeg cannot render text into subtitle bitmaps, so generate-fixtures.sh writes the PGS original by hand with a short python3 -I script (the same prerequisite as the caption fixture). ffmpeg then copies it into MKV and converts it to DVB (with a display definition segment) and DVD, using -fix_sub_duration.

Verification

  • swift test: 169 tests in 25 suites pass.
  • xcodebuild build for generic/platform=tvOS and generic/platform=iOS: succeeded with no new warnings (the SystemRenderer main-actor warnings were there before).

Not verified

  • Real broadcast DVB subtitles (multi-region pages, 2- and 4-bit CLUTs, page timeouts) and real Blu-ray PGS (cropping, multiple objects, forced flags). Only the generated fixtures are covered.
  • Drawing: no host draws images yet. Lume shows nothing for bitmap tracks, as before, until it composites them.
  • Memory and performance of conversion for large PGS bitmaps on Apple TV. Conversion is a per-pixel CPU loop on the subtitle thread, at subtitle rate.
  • Teletext stays undecodable: the build has no libzvbi.

The audio decoder read the negotiated output width once, at creation, so a
route change mid-playback (HDMI receiver to AirPlay, HomePods) kept feeding
the old width. AudioOutputRoute now follows the shared audio session's
negotiated width through route-change notifications (stereo on macOS); the
decoder refits per AudioLayoutPolicy at the next frame boundary, with an
explicit ITU-R BS.775 DownmixMatrix (centre and surrounds -3 dB, LFE
omitted) set into swresample, and mono goes out on both speakers. When the
route narrows, the renderer keeps audio decoded for the wider route away
from the renderer.

PlayerSession.audioHealth (also in Diagnostics) reports audio that died
while video plays on: decoder failed, renderer failed, or nothing reaching
the renderer at a running playhead (AudioLossDetector), the latter also as
.error with the new EngineError.Code.audioLost. Additive API only.
Cue lifecycle: decoders now hand the store display updates (start, end or
none, text, whether the track replaces the whole screen), and a pure
SubtitleTimeline decides how long each shows. Open cues (UINT32_MAX end:
PGS, DVB, captions) end at the next update of their track instead of
staying up for 49 days; the empty update replacing tracks send to clear
the screen now ends what is showing instead of being dropped; open cues
are capped at 16 s and ended by a seek; re-read ranges stay deduplicated;
the store is bounded at 8192 cues so a live session no longer grows
without limit. Subtitle decoders flush their state across seek serials.
SubtitleStore is now checked Sendable (Mutex) instead of @unchecked.

Closed captions: CEA-608 carried as A/53 side data on decoded video frames
is lifted off in VideoDecoder.receiveFrames by a ClosedCaptionTap, stamped
with the frame's PTS and trySend-ed to a SubtitleDecoder running ccaption
in real-time mode, anchored at the packet time. The video lane never
waits on it. A synthetic "CC1" subtitle track
(TrackInfo.embeddedClosedCaptionsIndex) appears in PlayerSession.mediaInfo
once caption content is seen, and selectSubtitleTrack accepts it.

Stacked on #46.
@bilipp
bilipp force-pushed the feat/subtitle-cues-and-captions branch from 987170c to 6375c53 Compare October 12, 2026 22:35
SUBTITLE_BITMAP rects from avcodec_decode_subtitle2 were dropped. They
now become SubtitleImages on the cue: a premultiplied sRGB CGImage
(palette ARGB -> premultiplied BGRA) and a rect normalized to the
subtitle's authoring canvas as the decoder reports it after decoding
(DVB display definition, PGS presentation size, DVD size:), with a 720x576
fallback grown to fit the rects. The DVB context's size is zeroed before
opening so the stream's display definition decides it.

SubtitleCue gains images (defaulted, empty for text cues); bitmap cues
carry empty text, so text-only consumers keep working and show nothing.
The timeline treats an update with images as content rather than a clear,
deduplicates re-decoded bitmaps by position and size, and bounds bitmap
memory at 64 MB on top of the cue count. LumePlayer exposes the active
cues' images as subtitleImages; LumePlayerView has no subtitle overlay, so
drawing stays with the host.

Stacked on #46.
@bilipp
bilipp force-pushed the feat/bitmap-subtitles branch from 27414a0 to be16ad1 Compare October 12, 2026 22:38
@bilipp
bilipp changed the base branch from feat/subtitle-cues-and-captions to main October 12, 2026 22:38
@bilipp
bilipp merged commit 74ce4e0 into main Oct 12, 2026
9 of 10 checks passed
@bilipp
bilipp deleted the feat/bitmap-subtitles branch October 12, 2026 22:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant