From ffb3d6bcd9bd0a63d520854c74abe3b250a0305c Mon Sep 17 00:00:00 2001
From: Philipp Bischoff
Date: Sun, 25 Oct 2026 21:51:34 +0100
Subject: [PATCH] fix: start the clock once every lane has a frame, at the
latest lane's first
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
The start gate counted packets as runway, so it could open before the video
decoder had produced anything — a cold VideoToolbox session after app launch
takes that long. Audio then played over a frozen picture, and the late video
frames were shown as fast as they decoded until they caught up with the clock.
The gate now also waits for every active lane to have a frame at its
renderer, and anchors the clock at the latest lane's first frame, so a live
TS joined mid-GOP (video waiting for its first keyframe) starts where the
picture does instead of under audio alone. A lane that cannot buffer more,
or every lane at EOF, still lets the clock start; a failed decoder marks its
lane EOF and leaves the gate. Demux EOF ends the buffering, not the wait for
a first frame.
Co-Authored-By: Claude
---
AGENTS.md | 1 +
.../Session/PlayerSession.swift | 51 +++++++++++++++++--
.../Fixtures/generate-fixtures.sh | 10 ++++
.../PlayerSessionTests.swift | 27 ++++++++++
4 files changed, 85 insertions(+), 4 deletions(-)
diff --git a/AGENTS.md b/AGENTS.md
index 44a7b68..a719053 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -83,6 +83,7 @@ This split is the core invariant of the codebase:
- **Timestamp unwrapping at the demux boundary:** `Time/TimestampUnwrapper` normalizes all PTS/DTS to monotonic 64-bit engine time immediately after `av_read_frame` (handles MPEG-TS 33-bit wraparound), and `Time/TimelineFolder` folds the other discontinuities a live source produces (encoder restarts, catch-up chunks that restart PTS) out of the timeline. A seam is detected per continuous stream but corrected on all streams alike, so A/V sync across it stays what the source had; the folder resets on seek. Before folding, `Demux/ReplayFilter` drops what a reconnected source sends a second time (same stream, timestamp and size, arriving behind the newest timestamp) — folded first, a replay would play twice as a seam. Nothing downstream ever sees raw container time.
- **RAII FFI wrappers:** `FFI/Packet`, `FFI/Frame` etc. wrap `AVPacket*`/`AVFrame*` with deinit-based free. `@unchecked Sendable` is allowed **only** on these audited immutable-after-construction wrappers and on `Channel` — never on state-bearing types. No `Unmanaged` outside FFI.
- **A/V sync is Apple's job:** decoded frames are enqueued into `AVSampleBufferRenderSynchronizer` (`Render/SystemRenderer`). Do not add drop-frame heuristics or hand-rolled clocks.
+- **The clock starts only once every lane has a frame at its renderer, and at the latest lane's first frame.** The buffer gate counts packets, so on its own it opens before a cold decoder (first VideoToolbox session after launch) has produced anything: audio plays over a frozen picture, then the late frames are shown as fast as they decode. A lane that joins later (live TS mid-GOP waiting for a keyframe) moves the start forward rather than leaving the picture empty. Only a lane that cannot buffer more or every lane at EOF lets the clock start without that.
- **Deinterlacing never costs the hardware decoder.** `Decode/VideoFilterGraph` (bwdif, falling back to yadif) sits between `avcodec_receive_frame` and delivery; VideoToolbox frames are downloaded with `av_hwframe_transfer_data` and filtered on the CPU. Do *not* "simplify" this into switching the codec to software when a stream is interlaced: that costs a mid-stream codec rebuild and keyframe resync at every interlaced/progressive splice, which live broadcasts (ad breaks in a 1080i match) are full of. Engagement is decided per frame from `AV_FRAME_FLAG_INTERLACED` (PLAN.md §3, failure 6) and, once engaged, everything stays on one path — `deint=interlaced` passes progressive frames through. Filter output carries the sink's time base, which a field-doubling filter *halves*; PTS and durations are rescaled back to engine µs in `VideoDecoder`.
- **Video decoders are chosen per path, never with `avcodec_find_decoder`.** `VideoDecoder.decoder(for:hardware:)` picks the decoder with a VideoToolbox config for the hardware path and a decoder that can run without hardware for the software path. For AV1 these are different decoders: FFmpeg's native `av1` is the only one with a VT hwaccel but returns `ENOSYS` without one, while `libdav1d` (registered first, so what `avcodec_find_decoder` returns) has no hardware path. The AV1 hardware path is only tried when `VTIsHardwareDecodeSupported` says the chip has it, since the native decoder has nothing to fall back to.
- **Tracks without a decoder are listed, never chosen.** `TrackInfo.isDecodable` asks the same lookups the lanes open with (`VideoDecoder.decoder(for:hardware:)` on both paths, `AudioDecoder`/`SubtitleDecoder.decoder(for:)`), and automatic selection passes over undecodable tracks (AC-4, MPEG-H in DVB/ATSC muxes) before language and default flag; only when nothing is decodable does it fall back to the old choice. Keeping such streams out of `avformat_find_stream_info` is not possible cleanly: its completeness check ignores `discard` (an AC-4 stream raises a TS probe from 5 s to 7 s of media, and pre-probe discard measured worse), so don't hack codecpar to get there.
diff --git a/Sources/LumeEngineCore/Session/PlayerSession.swift b/Sources/LumeEngineCore/Session/PlayerSession.swift
index e9a39a0..e25323e 100644
--- a/Sources/LumeEngineCore/Session/PlayerSession.swift
+++ b/Sources/LumeEngineCore/Session/PlayerSession.swift
@@ -1132,10 +1132,29 @@ public actor PlayerSession {
if state != .buffering { state = .buffering }
return
}
- // Start when every lane is as buffered as it can usefully get,
- // or when EOF means no more is coming.
- if allLanesReady || allEOF || demuxAtEOF {
- renderer.setRate(targetRate)
+ // Start when every lane is as buffered as it can usefully get and
+ // has something at its renderer, or when EOF means no more is
+ // coming. The runway counts packets, so a lane can be "ready"
+ // before its decoder has produced a frame — a cold VideoToolbox
+ // session takes that long on first launch. Started then, audio
+ // plays over a frozen picture and the late video frames are shown
+ // as fast as they decode until they catch up with the clock.
+ // Demux EOF ends the buffering, not the wait for a first frame:
+ // the decoders still hold what was read, and a lane that never
+ // produces a frame drains to EOF and leaves the gate.
+ let buffered = allLanesReady || demuxAtEOF
+ if (buffered && (presentableStart != nil || anyLaneFull)) || allEOF {
+ // A lane whose media starts later (live TS joined mid-GOP:
+ // video waits for its first keyframe) would show nothing
+ // until the clock got there; start where every lane has
+ // something to present instead. The audio before it is
+ // dropped by the renderer as late.
+ let now = renderer.currentTime
+ if let start = presentableStart, MediaTime.isValid(now), start > now {
+ renderer.setRate(targetRate, anchoredAt: start)
+ } else {
+ renderer.setRate(targetRate)
+ }
state = .playing
} else if state != .buffering {
state = .buffering
@@ -1234,4 +1253,28 @@ public actor PlayerSession {
$0.pipeline >= configuration.bufferTarget || $0.queueFull
}
}
+
+ /// Some lane cannot buffer more: waiting for another lane's first frame
+ /// would only park the demuxer behind it (a video lane that never
+ /// decodes is the watchdog's to report, not the start gate's to hide).
+ private var anyLaneFull: Bool {
+ laneDepths.contains(where: \.queueFull)
+ }
+
+ /// The earliest time at which every active lane has media at its
+ /// renderer — the latest of their first enqueued PTS since the last
+ /// flush. nil while some lane has handed its renderer nothing yet.
+ private var presentableStart: Int64? {
+ let lowWater = renderer.enqueuedLowWaterMark
+ var marks: [Int64] = []
+ if videoDecoder != nil, !videoAtEOF {
+ guard MediaTime.isValid(lowWater.video) else { return nil }
+ marks.append(lowWater.video)
+ }
+ if audioDecoder != nil, !audioAtEOF {
+ guard MediaTime.isValid(lowWater.audio) else { return nil }
+ marks.append(lowWater.audio)
+ }
+ return marks.max()
+ }
}
diff --git a/Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh b/Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh
index 30a27fe..eda7a57 100644
--- a/Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh
+++ b/Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh
@@ -94,6 +94,16 @@ gen av1.mkv \
-c:v libsvtav1 -preset 12 -pix_fmt yuv420p10le -g 24 \
-c:a aac -shortest
+# Video starting 2 s after audio, as a live TS joined mid-GOP delivers it
+# (the video decoder has nothing to show before the first keyframe): the
+# clock must start where the picture does, not under audio alone.
+gen videolate.ts \
+ -f lavfi -i "sine=frequency=440:duration=8" \
+ -itsoffset 2 -f lavfi -i "testsrc2=duration=6:size=320x180:rate=25" \
+ -map 1:v -map 0:a \
+ -c:v libx264 -preset ultrafast -pix_fmt yuv420p -g 25 \
+ -c:a aac -muxdelay 0 -f mpegts
+
gen interlaced.ts \
-f lavfi -i "testsrc2=duration=4:size=640x360:rate=50" \
-vf "interlace=scan=tff" \
diff --git a/Tests/LumeEngineCoreTests/PlayerSessionTests.swift b/Tests/LumeEngineCoreTests/PlayerSessionTests.swift
index f79dc52..a132e6d 100644
--- a/Tests/LumeEngineCoreTests/PlayerSessionTests.swift
+++ b/Tests/LumeEngineCoreTests/PlayerSessionTests.swift
@@ -168,6 +168,33 @@ struct PlayerSessionTests {
await session.shutdown()
}
+ @Test("the clock starts where every lane has a frame to present", .timeLimit(.minutes(1)))
+ func startsAtFirstPresentableFrame() async throws {
+ // No buffer target: the runway gate opens at once, so only waiting
+ // for the lanes' first frames keeps audio from starting under a
+ // picture that has nothing to show yet.
+ var configuration = PlayerConfiguration()
+ configuration.muted = true
+ configuration.bufferTarget = 0
+ let session = PlayerSession(configuration: configuration)
+ _ = try await session.open(url: try Fixtures.path("videolate.ts"))
+ await session.play()
+ let playing = await eventually(interval: .milliseconds(10)) { await session.state == .playing }
+ #expect(playing, "session must reach .playing")
+
+ let diagnostics = await session.diagnostics
+ #expect(diagnostics.startup.firstVideoFrame != nil, "the clock must not run before video has a frame")
+ // Video starts 2 s in; started at the audio's start instead, the
+ // picture would sit empty for two seconds of sound.
+ #expect(diagnostics.videoFirstSeconds >= 0)
+ #expect(
+ diagnostics.playheadSeconds >= diagnostics.videoFirstSeconds - 0.05,
+ "clock must start at the first video frame: playhead \(diagnostics.playheadSeconds), video from \(diagnostics.videoFirstSeconds)"
+ )
+ #expect(await session.position > 1.8, "playback must start where the picture does")
+ await session.shutdown()
+ }
+
@Test("audio the renderer drops on its own is enqueued again", .timeLimit(.minutes(1)))
func automaticAudioFlush() async throws {
let session = makeSession()