Skip to content

fix: start the clock once every lane has a frame, at the latest lane's first - #60

Open
bilipp wants to merge 1 commit into
mainfrom
fix/start-clock-at-first-presentable-frame
Open

bilipp wants to merge 1 commit into
mainfrom
fix/start-clock-at-first-presentable-frame

Conversation

@bilipp

@bilipp bilipp commented Oct 25, 2026

Copy link
Copy Markdown
Owner

Problem

Reported from Lume: sometimes the audio plays immediately while the video is frozen, then the video "rushes" (fast-forwards) for a few seconds until it matches the sound. It happens most on the first channel after launching the app.

PlayerSession's start gate (allLanesReady) counts undecoded packets as runway. So the video lane could count as ready before its decoder had produced a single frame. On the first start after launch, VideoToolbox is still initialising, which takes a while. The clock started anyway:

  1. Audio, which is cheap to decode, played right away.
  2. The video renderer had nothing to show, so the picture stayed frozen.
  3. When frames arrived, their PTS was already behind the clock. The display layer shows late frames as soon as they are enqueued, so they went by as fast as they decoded: the fast-forward.

A related case: a live TS joined mid-GOP delivers its first video keyframe a second or two after the first audio. The clock was anchored at the audio's start, so the picture stayed empty for that stretch.

Fix

In evaluatePlayback, the start condition is now: the buffer is ready and every active lane has a frame at its renderer (presentableStart). The clock is anchored at the latest lane's first enqueued PTS when that is ahead of the clock. The audio before it is dropped by the renderer as late.

Escape hatches, so the gate can't wedge:

  • A full lane (anyLaneFull): if one lane can't buffer more, waiting for another lane's first frame would only park the demuxer, so the clock starts. A video lane that never decodes is the stall watchdog's to report.
  • A failed decoder marks its lane EOF, so the lane drops out of the gate.
  • Every lane at EOF still starts the clock.

Demux EOF now ends only the buffering, not the wait for a first frame. The decoders still hold what was read, and they drain to EOF on demux EOF.

A rebuffer mid-play and a keyframe-backward seek landing are unaffected: in both, the first PTS is behind the clock, so the clock is not re-anchored.

The rule is recorded among the key invariants in AGENTS.md.

Tests

  • New fixture videolate.ts: a TS whose video starts 2 s after its audio. New test startsAtFirstPresentableFrame uses it with bufferTarget = 0.
    • On main it fails three ways: the clock runs before any video frame, from the audio's start, at position ≈ 0.
    • With the fix it passes: the clock starts at the first video frame, about 2.02 s in.
  • LIBDISPATCH_COOPERATIVE_POOL_STRICT=1 swift test: 284 of 285 pass. The one failure is "every decode path delivers the stream's HDR10 and HDR10+ metadata" (hardware path), which fails the same way on unmodified main on this machine.
  • Not yet checked on a device. On the Apple TV, the heartbeat's startup=[…] from a cold launch should show the first frame shown no earlier than video's first frame.

🤖 Generated with Claude Code

…s first

The start gate counted packets as runway, so it could open before the video
decoder had produced anything — a cold VideoToolbox session after app launch
takes that long. Audio then played over a frozen picture, and the late video
frames were shown as fast as they decoded until they caught up with the clock.

The gate now also waits for every active lane to have a frame at its
renderer, and anchors the clock at the latest lane's first frame, so a live
TS joined mid-GOP (video waiting for its first keyframe) starts where the
picture does instead of under audio alone. A lane that cannot buffer more,
or every lane at EOF, still lets the clock start; a failed decoder marks its
lane EOF and leaves the gate. Demux EOF ends the buffering, not the wait for
a first frame.

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant