realtime: orient capture dims to device orientation (iOS counterpart to decart-android#16) - #32
Open
nagar-decart wants to merge 1 commit into
Open
realtime: orient capture dims to device orientation (iOS counterpart to decart-android#16)#32nagar-decart wants to merge 1 commit into
nagar-decart wants to merge 1 commit into
Conversation
nagar-decart
force-pushed
the
fix/bug-b-orientation-aware-capture
branch
7 times, most recently
from
June 8, 2026 11:15
d979ee2 to
5353cc4
Compare
iOS counterpart to the Android fix in decart-android#16. Adds a `CameraCaptureOptions.orientedForDevice(...)` static factory in the SDK that takes the model's natural landscape (W,H) and transposes to portrait when the current interface orientation is portrait. The LiveKit camera capture session is then configured for the device's actual orientation; the published track's metadata carries the displayed dims directly (LiveKit transports track dims via the AddTrackRequest signaling protocol, not via SDP), instead of relying on the WebRTC CVO rotation extension to communicate orientation to the receiver. Why this matters: the realtime inference server's bridge eagerly publishes its output video track at the dims the input track advertised in its publication metadata (so the output SDP renegotiation overlaps the per-session time-to-first-output-frame on the inference side, saving ~500 ms TTFF). If those signaled dims don't match the orientation of the actual decoded input frames, the bridge republishes the output track at the correct orientation when the first model frame arrives. That unpublish-old + publish-new on the same participant triggers an intermittent transceiver-reuse race on the subscriber side: the stopped RtpReceiver from the eager subscription stays in the PeerConnection, libwebrtc reuses the transceiver slot for the new track, onAddTrack never fires for it, addSubscribedMediaTrack never runs, didSubscribeTrack never reaches the RoomDelegate, the renderer is never attached, and the app shows a black screen until session timeout. Empirically ~37 % rate on portrait-locked Lucy 2.1 sessions in production sampling. The same root cause was fixed in Android via decart-android#16; this is the iOS equivalent. The Example app's `startCapture()` now calls the new helper instead of constructing `CameraCaptureOptions` directly. Customer apps using the SDK should switch to the same helper to get the fix. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
nagar-decart
force-pushed
the
fix/bug-b-orientation-aware-capture
branch
from
June 8, 2026 15:58
5353cc4 to
b254bff
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CameraCaptureOptions.orientedForDevice(...)static factory in the SDK that produces capture dims matching the device's current interface orientation.startCapture()to use it.What was breaking
Existing pattern in
Example/Example/DecartSDK/RealtimeManager.swift:ModelDefinitionspecifies the model's natural landscape(W, H), e.g.(1088, 624). A portrait-locked app passes those dims unchanged → the LiveKitAVCaptureSessionis configured for landscape → frames are delivered post-rotation via WebRTC's CVO RTP extension → the publication metadata carries the unrotated landscape dims.Server-side, the realtime inference bridge eagerly publishes its output track at the dims the input track advertised in its publication metadata so the output SDP renegotiation overlaps the per-session time-to-first-output-frame on the inference side (~500 ms TTFF win). With landscape SDP + portrait actual frames, the bridge has to unpublish that track and republish at portrait when the first model frame arrives.
That unpublish-old + publish-new sequence on the same participant triggers an intermittent transceiver-reuse race in the subscriber
PeerConnection: the existing stoppedRtpReceiverstays in the PC, libwebrtc effectively reuses the transceiver slot for the new track,onAddTracknever fires again,addSubscribedMediaTracknever runs,didSubscribeTrack(_:)never reachesRoomDelegate, the renderer is never attached, and the app shows a black screen until session timeout.Empirically ~37 % of portrait Lucy 2.1 sessions in production. Detailed walk-through and the production trace evidence are in decart-android#16.
What this PR does
Adds an SDK-side helper that consumers (and the Example) can call instead of building
CameraCaptureOptionsdirectly:The helper reads the active
UIWindowScene.interfaceOrientation(falling back toUIScreen.main.bounds) and transposes the model's landscape dims to portrait when the device is portrait. The capture session is then configured for the actually-displayed orientation, the published track's metadata carries the displayed dims directly, the server's eager-publish prediction lands correctly on the first try, and no republish happens.Coverage
is_portrait=True✓is_portrait=False✓is_portrait=False✓is_portrait=True✓Test plan
lucy-2.1, run 10 sessions, confirmOUTPUT (REMOTE)renders every time. Pre-PR repro had a ~37 % black-screen rate.Republished LiveKit video trackline appears on the server (Datadog) trace for any of the test sessions.LocalVideoTrackcreation gets the fix when they switch toCameraCaptureOptions.orientedForDevice(...).Notes for SDK consumers
The fix is opt-in: customer apps that construct
CameraCaptureOptions(position:dimensions:fps:)directly will still hit the bug. We should update integration docs / release notes to point atorientedForDeviceonce this lands.🤖 Generated with Claude Code
Note
Medium Risk
Changes realtime video publish dimensions and orientation detection at session start; wrong orientation logic could still mis-advertise dims, but scope is limited to capture setup in the Example and the new public helper.
Overview
Adds
DecartCameraCapture.orientedDimensionsso LiveKit camera capture uses width/height that match the device’s interface orientation (transpose model landscape dims on portrait) instead of always using the model’s natural(W, H).The Example
RealtimeManager.startCapturenow passes those oriented dims intoCameraCaptureOptions, so published track metadata matches what the user sees and the server’s eager output publish no longer triggers a republish path linked to intermittent missing remote video (black screen).Reviewed by Cursor Bugbot for commit b254bff. Bugbot is set up for automated code reviews on this repo. Configure here.