diff --git a/AGENTS.md b/AGENTS.md index 44a7b68..a719053 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -83,6 +83,7 @@ This split is the core invariant of the codebase: - **Timestamp unwrapping at the demux boundary:** `Time/TimestampUnwrapper` normalizes all PTS/DTS to monotonic 64-bit engine time immediately after `av_read_frame` (handles MPEG-TS 33-bit wraparound), and `Time/TimelineFolder` folds the other discontinuities a live source produces (encoder restarts, catch-up chunks that restart PTS) out of the timeline. A seam is detected per continuous stream but corrected on all streams alike, so A/V sync across it stays what the source had; the folder resets on seek. Before folding, `Demux/ReplayFilter` drops what a reconnected source sends a second time (same stream, timestamp and size, arriving behind the newest timestamp) — folded first, a replay would play twice as a seam. Nothing downstream ever sees raw container time. - **RAII FFI wrappers:** `FFI/Packet`, `FFI/Frame` etc. wrap `AVPacket*`/`AVFrame*` with deinit-based free. `@unchecked Sendable` is allowed **only** on these audited immutable-after-construction wrappers and on `Channel` — never on state-bearing types. No `Unmanaged` outside FFI. - **A/V sync is Apple's job:** decoded frames are enqueued into `AVSampleBufferRenderSynchronizer` (`Render/SystemRenderer`). Do not add drop-frame heuristics or hand-rolled clocks. +- **The clock starts only once every lane has a frame at its renderer, and at the latest lane's first frame.** The buffer gate counts packets, so on its own it opens before a cold decoder (first VideoToolbox session after launch) has produced anything: audio plays over a frozen picture, then the late frames are shown as fast as they decode. A lane that joins later (live TS mid-GOP waiting for a keyframe) moves the start forward rather than leaving the picture empty. Only a lane that cannot buffer more or every lane at EOF lets the clock start without that. - **Deinterlacing never costs the hardware decoder.** `Decode/VideoFilterGraph` (bwdif, falling back to yadif) sits between `avcodec_receive_frame` and delivery; VideoToolbox frames are downloaded with `av_hwframe_transfer_data` and filtered on the CPU. Do *not* "simplify" this into switching the codec to software when a stream is interlaced: that costs a mid-stream codec rebuild and keyframe resync at every interlaced/progressive splice, which live broadcasts (ad breaks in a 1080i match) are full of. Engagement is decided per frame from `AV_FRAME_FLAG_INTERLACED` (PLAN.md §3, failure 6) and, once engaged, everything stays on one path — `deint=interlaced` passes progressive frames through. Filter output carries the sink's time base, which a field-doubling filter *halves*; PTS and durations are rescaled back to engine µs in `VideoDecoder`. - **Video decoders are chosen per path, never with `avcodec_find_decoder`.** `VideoDecoder.decoder(for:hardware:)` picks the decoder with a VideoToolbox config for the hardware path and a decoder that can run without hardware for the software path. For AV1 these are different decoders: FFmpeg's native `av1` is the only one with a VT hwaccel but returns `ENOSYS` without one, while `libdav1d` (registered first, so what `avcodec_find_decoder` returns) has no hardware path. The AV1 hardware path is only tried when `VTIsHardwareDecodeSupported` says the chip has it, since the native decoder has nothing to fall back to. - **Tracks without a decoder are listed, never chosen.** `TrackInfo.isDecodable` asks the same lookups the lanes open with (`VideoDecoder.decoder(for:hardware:)` on both paths, `AudioDecoder`/`SubtitleDecoder.decoder(for:)`), and automatic selection passes over undecodable tracks (AC-4, MPEG-H in DVB/ATSC muxes) before language and default flag; only when nothing is decodable does it fall back to the old choice. Keeping such streams out of `avformat_find_stream_info` is not possible cleanly: its completeness check ignores `discard` (an AC-4 stream raises a TS probe from 5 s to 7 s of media, and pre-probe discard measured worse), so don't hack codecpar to get there. diff --git a/Sources/LumeEngineCore/Session/PlayerSession.swift b/Sources/LumeEngineCore/Session/PlayerSession.swift index e9a39a0..e25323e 100644 --- a/Sources/LumeEngineCore/Session/PlayerSession.swift +++ b/Sources/LumeEngineCore/Session/PlayerSession.swift @@ -1132,10 +1132,29 @@ public actor PlayerSession { if state != .buffering { state = .buffering } return } - // Start when every lane is as buffered as it can usefully get, - // or when EOF means no more is coming. - if allLanesReady || allEOF || demuxAtEOF { - renderer.setRate(targetRate) + // Start when every lane is as buffered as it can usefully get and + // has something at its renderer, or when EOF means no more is + // coming. The runway counts packets, so a lane can be "ready" + // before its decoder has produced a frame — a cold VideoToolbox + // session takes that long on first launch. Started then, audio + // plays over a frozen picture and the late video frames are shown + // as fast as they decode until they catch up with the clock. + // Demux EOF ends the buffering, not the wait for a first frame: + // the decoders still hold what was read, and a lane that never + // produces a frame drains to EOF and leaves the gate. + let buffered = allLanesReady || demuxAtEOF + if (buffered && (presentableStart != nil || anyLaneFull)) || allEOF { + // A lane whose media starts later (live TS joined mid-GOP: + // video waits for its first keyframe) would show nothing + // until the clock got there; start where every lane has + // something to present instead. The audio before it is + // dropped by the renderer as late. + let now = renderer.currentTime + if let start = presentableStart, MediaTime.isValid(now), start > now { + renderer.setRate(targetRate, anchoredAt: start) + } else { + renderer.setRate(targetRate) + } state = .playing } else if state != .buffering { state = .buffering @@ -1234,4 +1253,28 @@ public actor PlayerSession { $0.pipeline >= configuration.bufferTarget || $0.queueFull } } + + /// Some lane cannot buffer more: waiting for another lane's first frame + /// would only park the demuxer behind it (a video lane that never + /// decodes is the watchdog's to report, not the start gate's to hide). + private var anyLaneFull: Bool { + laneDepths.contains(where: \.queueFull) + } + + /// The earliest time at which every active lane has media at its + /// renderer — the latest of their first enqueued PTS since the last + /// flush. nil while some lane has handed its renderer nothing yet. + private var presentableStart: Int64? { + let lowWater = renderer.enqueuedLowWaterMark + var marks: [Int64] = [] + if videoDecoder != nil, !videoAtEOF { + guard MediaTime.isValid(lowWater.video) else { return nil } + marks.append(lowWater.video) + } + if audioDecoder != nil, !audioAtEOF { + guard MediaTime.isValid(lowWater.audio) else { return nil } + marks.append(lowWater.audio) + } + return marks.max() + } } diff --git a/Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh b/Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh index 30a27fe..eda7a57 100644 --- a/Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh +++ b/Tests/LumeEngineCoreTests/Fixtures/generate-fixtures.sh @@ -94,6 +94,16 @@ gen av1.mkv \ -c:v libsvtav1 -preset 12 -pix_fmt yuv420p10le -g 24 \ -c:a aac -shortest +# Video starting 2 s after audio, as a live TS joined mid-GOP delivers it +# (the video decoder has nothing to show before the first keyframe): the +# clock must start where the picture does, not under audio alone. +gen videolate.ts \ + -f lavfi -i "sine=frequency=440:duration=8" \ + -itsoffset 2 -f lavfi -i "testsrc2=duration=6:size=320x180:rate=25" \ + -map 1:v -map 0:a \ + -c:v libx264 -preset ultrafast -pix_fmt yuv420p -g 25 \ + -c:a aac -muxdelay 0 -f mpegts + gen interlaced.ts \ -f lavfi -i "testsrc2=duration=4:size=640x360:rate=50" \ -vf "interlace=scan=tff" \ diff --git a/Tests/LumeEngineCoreTests/PlayerSessionTests.swift b/Tests/LumeEngineCoreTests/PlayerSessionTests.swift index f79dc52..a132e6d 100644 --- a/Tests/LumeEngineCoreTests/PlayerSessionTests.swift +++ b/Tests/LumeEngineCoreTests/PlayerSessionTests.swift @@ -168,6 +168,33 @@ struct PlayerSessionTests { await session.shutdown() } + @Test("the clock starts where every lane has a frame to present", .timeLimit(.minutes(1))) + func startsAtFirstPresentableFrame() async throws { + // No buffer target: the runway gate opens at once, so only waiting + // for the lanes' first frames keeps audio from starting under a + // picture that has nothing to show yet. + var configuration = PlayerConfiguration() + configuration.muted = true + configuration.bufferTarget = 0 + let session = PlayerSession(configuration: configuration) + _ = try await session.open(url: try Fixtures.path("videolate.ts")) + await session.play() + let playing = await eventually(interval: .milliseconds(10)) { await session.state == .playing } + #expect(playing, "session must reach .playing") + + let diagnostics = await session.diagnostics + #expect(diagnostics.startup.firstVideoFrame != nil, "the clock must not run before video has a frame") + // Video starts 2 s in; started at the audio's start instead, the + // picture would sit empty for two seconds of sound. + #expect(diagnostics.videoFirstSeconds >= 0) + #expect( + diagnostics.playheadSeconds >= diagnostics.videoFirstSeconds - 0.05, + "clock must start at the first video frame: playhead \(diagnostics.playheadSeconds), video from \(diagnostics.videoFirstSeconds)" + ) + #expect(await session.position > 1.8, "playback must start where the picture does") + await session.shutdown() + } + @Test("audio the renderer drops on its own is enqueued again", .timeLimit(.minutes(1))) func automaticAudioFlush() async throws { let session = makeSession()