Skip to content

Fix video republishing issues - #1008

Open
pblazej wants to merge 7 commits into
mainfrom
blaze/screenshare-reconnect
Open

Fix video republishing issues#1008
pblazej wants to merge 7 commits into
mainfrom
blaze/screenshare-reconnect

Conversation

@pblazej

@pblazej pblazej commented May 14, 2026

Copy link
Copy Markdown
Contributor

Summary

republishAllTracks() routes every local track through unpublish(), which calls track.stop() and tears down the capturer. For iOS broadcast extension and ReplayKit screen capture this closes the IPC and the extension self-terminates — the track gets re-added to the new publisher but no frames flow.

Fixed by giving unpublish() an internal stopTrack: variant and taking that path for screen shares, so the capturer keeps running while the track is properly detached from the old publisher and reattached to the new one.

  • unpublish(publication:notify:) stays as-is (public + ObjC selector unchanged) and forwards to a new internal _unpublish(publication:notify:stopTrack:). _unpublish is the previous body with the stop gated on the flag, so anything passing stopTrack: true behaves exactly as before.
  • Screen shares are unpublished with stopTrack: false. Everything else unpublish() does still happens: sender + simulcast-sender removal from the publisher, publisherShouldNegotiate(), onUnpublish(), didUnpublishTrack.
  • The detached publication then drops its track reference so it stops observing the capturer that is deliberately left running.
  • rtpSenderForCodec.removeAll() in _unpublish, once the simulcast senders have been removed from the publisher. _publish re-attaches the main rtpSender but never the per-codec ones, so a stale entry made _set(subscribedCodec:) report a match against a sender of the destroyed peer connection and publish(additionalVideoCodec:) never re-added the backup codec.
  • VideoCapturer.stopCapture() now nils the cached dimensions. It reset dimensionsCompleter but left _state.dimensions at its last value, and set(dimensions:) only resumes the completer when the value changes — so a source resuming at the same resolution never resolved dimensions again and _publish timed out in "Waiting for dimensions to resolve". CameraCapturer had its own set(dimensions: nil) on stop; BufferCapturer and BroadcastScreenCapturer did not, so no buffer-backed video track could be re-published after a full reconnect. This is the second half of the screen share failure.
  • Muted screen shares take the normal path instead of being detached and then skipped.
  • Per-track republish failures are logged individually with their source, and the first error is rethrown so both call sites' error handling stays reachable.
  • Camera/mic republish is untouched: same publication list, same stopTrack: true teardown, same per-track error log as the previous unpublishAll().

Resolves #1004

Why keeping the capturer alive works

Track.start() short-circuits on trackState == .started, so skipping stop() leaves the capturer's startStopCounter at 1 and _publish's track.start() is a no-op. Nothing else in _publish touches the capture source.

On the WebRTC side, Transport.remove(track:) is _pc.removeTrack(sender) followed by stopInternal() on the video transceiver. So the old sender's track is nulled (sink detached, encoder stops) and the transceiver is stopped, making its m-section recyclable in unified plan; the debounced negotiation lets the removal and the following addTransceiver coalesce into a single offer. The LKRTCVideoSource is owned by the VideoCapturer and the LKRTCVideoTrack by LocalVideoTrack — neither is owned by the peer connection — so the same media track is handed to the new PC with capture uninterrupted.

The two republish paths differ in whether transports are live, and both are now correct:

transport behavior
Full reconnect (Room+EngineDelegate) already nil (cleanUpRTC()) sender removal skipped, matching the "never removeTrack before close()" constraint in Transport.cleanUp
Room move (_republishLocalTracks) live sender removed and transceiver stopped, so _publish no longer adds a second transceiver for the same media track

E2EE also round-trips again: didUnpublishTrack disables and drops the old frame cryptor, didPublishTrack builds a fresh one bound to the new sender.

Test plan

  • iOS: start a screen share via broadcast extension, force a network outage long enough to trigger a full reconnect, confirm frames resume without re-prompting the broadcast picker
  • iOS: same scenario with in-app screen share (no extension)
  • macOS: start screen share, trigger full reconnect, confirm frames resume
  • Room move with an active screen share: confirm a single video m-line in the new offer and no duplicate upload
  • Camera track with preferredBackupCodec survives a full reconnect with the backup codec re-published
  • E2EE enabled: screen share stays decryptable for remote participants after a full reconnect
  • Regression: camera + mic still survive a full reconnect
  • Regression: stopping the broadcast from Control Center after a reconnect unpublishes exactly once

Tests

Tests/LiveKitCoreTests/Room/RepublishTracksTests.swift, parameterized over microphone / camera / screen share. Each case publishes the track, runs debug_simulate(scenario: .fullReconnect), and waits on a RoomDelegate-backed AsyncCompleter for the replacement publication. Screen share must return with its capturer untouched (startStopCounter still 1, no .stopped transition); every other source is expected to be torn down and started again. Frames come from a synthetic CVPixelBuffer feed, so the test needs no network and runs in ~3.5s.

Verified it fails without the fix: with keepCapturing forced to false, the screen share case fails on stopSpy.stops == 0 while the other two still pass.

Known gaps

  • If re-publishing the screen share itself fails, _publish's catch still calls track.stop() and kills the capture source. The error is at least surfaced now.

🤖 Generated with Claude Code

@pblazej

pblazej commented May 14, 2026

Copy link
Copy Markdown
Contributor Author

WebRTC semantics check

Verified against the libwebrtc tree that reusing the same LKRTCMediaStreamTrack across the old (closed) and new publisher peer connection is sound:

  • PeerConnection::Close() does not end attached tracks. pc/peer_connection.cc:1906-1912 only iterates transceivers and calls StopInternal(), which routes through RtpSenderBase::Stop() (pc/rtp_sender.cc:562-580). Stop() detaches the track and unregisters the sender as an observer, but never mutates track state.
  • Track state is driven by the source, not by the PC. pc/video_track.cc:147-152 is the only set_state(kEnded) call site for video, and it fires only when the underlying MediaSourceInterface::SourceState becomes kEnded. Closing the PC doesn't end the source, so the track stays kLive.
  • The video source is independent of any PC. pc/video_track.cc:60-67 forwards sink registration to video_source_->internal()->AddOrUpdateSink(...). The source outlives the PC; frames keep flowing.
  • Reattaching is supported. AddTransceiver(track, ...) on the new PC creates a fresh RtpSenderBase that registers as a new observer on the still-live track and adds its sink to the same source. No API or invariant prohibits reusing a track across PCs — only per-sender state is reset.
  • Dropping old RtpSender refs is safe. After Stop() the sender is inert (stopped_ = true, media_channel_ = nullptr); destruction via the Swift refcount drop has nothing further to clean up.

Ordering caveat that this fix relies on: cleanUpRTC() awaits _state.transport?.close() before republishAllTracks() runs, so the old PC is fully closed (all old senders Stop()'d, detached from the track) before AddTransceiver registers a new sender on the new PC. No double-sink, no race.

Net effect for screen share: LKRTCVideoSource keeps producing frames into the same LKRTCVideoTrack; closing the old PC detached the old sender but left the track kLive; re-adding the track to the new PC registers a new sender that picks up the same source — frames resume.

@pblazej
pblazej marked this pull request as ready for review May 14, 2026 09:58
@indenpendman

Copy link
Copy Markdown

This problem still exists

@pblazej
pblazej marked this pull request as draft June 9, 2026 12:03
pblazej and others added 2 commits June 23, 2026 09:44
`republishAllTracks()` routes every local track through `unpublish()`,
which calls `track.stop()` and tears down the capturer. For iOS broadcast
extension and ReplayKit screen capture this kills the IPC and the
extension self-terminates — the track gets re-added to the new publisher
but no frames flow.

Detach screen share publications from the old publisher peer connection
without going through `unpublish()` so the capturer keeps running, then
let `_publish` reattach it to the new publisher.

Resolves #1004

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
republishAllTracks() aborted the whole loop on the first _publish failure
and only logged a generic error at the call site, hiding which track failed.
Catch per track, log the source, and keep re-publishing the rest so one
track's failure no longer blocks the others.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@pblazej
pblazej marked this pull request as ready for review August 21, 2026 08:55

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 1 additional finding.

Open in Devin Review

@pblazej
pblazej force-pushed the blaze/screenshare-reconnect branch from 817bf1a to 2b0a4ff Compare August 21, 2026 09:02
The detach block added in 723e321 hand-rolled a partial unpublish, which
skipped four steps unpublish() performs: sender removal from the publisher,
didUnpublishTrack, publication.set(track: nil) and onUnpublish(). On the room
move path (_republishLocalTracks) transports are never torn down, so leaving
the sender in place kept the old sendonly transceiver attached to the running
capturer and _publish added a second one for the same media track.

Add an internal _unpublish(publication:notify:stopTrack:) and keep the public
unpublish() as a forwarder so the ObjC selector is unchanged. _unpublish is
the old body with the stop gated on the flag, so tracks that pass
stopTrack: true behave exactly as before. Screen shares take the same path
with stopTrack: false, which stops the transceiver and frees the m-line while
Track.start()'s already-started guard keeps the capturer counter at 1.

The detached publication also drops its track reference, so it stops
observing the capturer that is deliberately left running, and its per-codec
senders are cleared because _publish re-attaches the main sender but never
the rtpSenderForCodec entries.

Also restore the republish contract weakened in 7075330: per-track failures
are still logged individually, but the first error is rethrown so both call
sites' error handling is reachable again. Muted screen shares are no longer
detached-then-skipped, which silently dropped their publication with no
delegate event.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pblazej
pblazej force-pushed the blaze/screenshare-reconnect branch from 2b0a4ff to e586b70 Compare August 21, 2026 09:04
devin-ai-integration[bot]

This comment was marked as resolved.

pblazej and others added 3 commits August 21, 2026 11:28
republishAllTracks() cleared rtpSenderForCodec only for the screen share it
detaches. _publish re-attaches the main rtpSender but never the per-codec
entries, so after a full reconnect any other video track published with a
backup codec still pointed at senders of the destroyed peer connection:
VideoTrack._set(subscribedCodec:) matched one, reported the codec as still
sent, and publish(additionalVideoCodec:) never re-added it.

Clear the dict in _unpublish once the simulcast senders have been read and
removed, so it applies to every unpublished track.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
VideoCapturer.stopCapture() reset dimensionsCompleter but left _state.dimensions
at its last value, and set(dimensions:) only resumes the completer when the value
changes. A source that resumes at the same resolution therefore never resolved
dimensions again, so _publish's "Waiting for dimensions to resolve" timed out
after 10s and the re-publish failed.

CameraCapturer already worked around this with its own set(dimensions: nil) on
stop; BufferCapturer and BroadcastScreenCapturer did not, so no buffer-backed
video track could be re-published after a full reconnect — the second half of
the screen share failure. Nil the dimensions in the base class instead, which
resets the completer as a side effect.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Parameterized over microphone, camera and screen share: each track is published,
the room is put through debug_simulate(scenario: .fullReconnect), and the test
waits on a RoomDelegate-backed AsyncCompleter for the replacement publication
rather than polling. Screen share must come back with its capturer untouched
(startStopCounter still 1, no .stopped transition); every other source is
expected to be torn down and started again.

Writing it surfaced that the screen share predicate applied the name fallback to
any source, while Participant.getTrackPublication(source:) only applies it to
.unknown — createBufferTrack keeps the screen share name by default, so a
`source: .camera` track was matched as a screen share. Narrow the predicate to
match.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pblazej

pblazej commented Aug 21, 2026

Copy link
Copy Markdown
Contributor Author

Looked at whether the keep-alive should extend to camera tracks: the mechanism generalizes for free — widening the predicate to any unmuted video source is one line, and the parameterized test passes with the camera case flipped to uninterrupted — and the WebRTC side is cheap, since with no sender there are no sinks and AdaptedVideoTrackSource::AdaptFrame bails on !broadcaster_.frame_wanted() before any buffer wrapping, crop/scale or encode. For camera it would only be an optimization rather than a correctness fix, though: AVCaptureSession restarts fine, so the win is the ~few hundred ms of device enumeration + format matching + dimensionsCompleter.wait() on every republish, plus avoiding reacquisition failures like a device that went away mid-reconnect. The cost is that cleanUpParticipants(isFullReconnect: true) skips the local participant, so a reconnect that keeps failing leaves the camera live and its indicator lit for the whole retry cycle (10 attempts easing 0.3s → 7s), and nothing in the SDK observes AVCaptureSession interruption notifications today — a stop/start currently repairs an interrupted session by accident, and keep-alive would remove that, giving you a live sender with no frames. So this PR stays screen-share-only; if we want the camera latency win, the order should be interruption observation first, then an opt-in RoomOptions flag on top.

@pblazej pblazej changed the title Keep screen share alive across full reconnect Fix video republishing issues Aug 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Screen sharing does not recover after reconnecting from a network outage during a meeting

2 participants