Skip to content

realtime: orient capture dims to device orientation (iOS counterpart to decart-android#16) - #32

Open
nagar-decart wants to merge 1 commit into
mainfrom
fix/bug-b-orientation-aware-capture
Open

realtime: orient capture dims to device orientation (iOS counterpart to decart-android#16)#32
nagar-decart wants to merge 1 commit into
mainfrom
fix/bug-b-orientation-aware-capture

Conversation

@nagar-decart

@nagar-decart nagar-decart commented Jun 8, 2026

Copy link
Copy Markdown

Summary

  • Adds a CameraCaptureOptions.orientedForDevice(...) static factory in the SDK that produces capture dims matching the device's current interface orientation.
  • Switches the Example app's startCapture() to use it.
  • Closes the same Bug B class that decart-android#16 closed for Android: the server's eager-publish → republish dance no longer fires for portrait clients, so the LiveKit subscriber transceiver-reuse race that causes intermittent black-screen sessions never gets triggered.

What was breaking

Existing pattern in Example/Example/DecartSDK/RealtimeManager.swift:

let dimensions = Dimensions(width: Int32(model.width), height: Int32(model.height))
let captureOptions = CameraCaptureOptions(position: .front, dimensions: dimensions, fps: model.fps)

ModelDefinition specifies the model's natural landscape (W, H), e.g. (1088, 624). A portrait-locked app passes those dims unchanged → the LiveKit AVCaptureSession is configured for landscape → frames are delivered post-rotation via WebRTC's CVO RTP extension → the publication metadata carries the unrotated landscape dims.

Server-side, the realtime inference bridge eagerly publishes its output track at the dims the input track advertised in its publication metadata so the output SDP renegotiation overlaps the per-session time-to-first-output-frame on the inference side (~500 ms TTFF win). With landscape SDP + portrait actual frames, the bridge has to unpublish that track and republish at portrait when the first model frame arrives.

That unpublish-old + publish-new sequence on the same participant triggers an intermittent transceiver-reuse race in the subscriber PeerConnection: the existing stopped RtpReceiver stays in the PC, libwebrtc effectively reuses the transceiver slot for the new track, onAddTrack never fires again, addSubscribedMediaTrack never runs, didSubscribeTrack(_:) never reaches RoomDelegate, the renderer is never attached, and the app shows a black screen until session timeout.

Empirically ~37 % of portrait Lucy 2.1 sessions in production. Detailed walk-through and the production trace evidence are in decart-android#16.

What this PR does

Adds an SDK-side helper that consumers (and the Example) can call instead of building CameraCaptureOptions directly:

let captureOptions = CameraCaptureOptions.orientedForDevice(
    position: .front,
    modelWidth: model.width,
    modelHeight: model.height,
    fps: model.fps
)

The helper reads the active UIWindowScene.interfaceOrientation (falling back to UIScreen.main.bounds) and transposes the model's landscape dims to portrait when the device is portrait. The capture session is then configured for the actually-displayed orientation, the published track's metadata carries the displayed dims directly, the server's eager-publish prediction lands correctly on the first try, and no republish happens.

Coverage

Device Held Capture dims Publication metadata reports Server prediction Republish
Phone Portrait 624×1088 624×1088 is_portrait=True None
Phone Landscape 1088×624 1088×624 is_portrait=False None
iPad Landscape 1088×624 1088×624 is_portrait=False None
iPad Portrait 624×1088 624×1088 is_portrait=True None

Test plan

  • Build the Example on a portrait-locked iPhone with lucy-2.1, run 10 sessions, confirm OUTPUT (REMOTE) renders every time. Pre-PR repro had a ~37 % black-screen rate.
  • Run the Example on an iPad locked to landscape: confirm capture dims log landscape (no transpose) and the session works.
  • Confirm no Republished LiveKit video track line appears on the server (Datadog) trace for any of the test sessions.
  • Confirm a customer using the SDK with their own LocalVideoTrack creation gets the fix when they switch to CameraCaptureOptions.orientedForDevice(...).

Notes for SDK consumers

The fix is opt-in: customer apps that construct CameraCaptureOptions(position:dimensions:fps:) directly will still hit the bug. We should update integration docs / release notes to point at orientedForDevice once this lands.

🤖 Generated with Claude Code


Note

Medium Risk
Changes realtime video publish dimensions and orientation detection at session start; wrong orientation logic could still mis-advertise dims, but scope is limited to capture setup in the Example and the new public helper.

Overview
Adds DecartCameraCapture.orientedDimensions so LiveKit camera capture uses width/height that match the device’s interface orientation (transpose model landscape dims on portrait) instead of always using the model’s natural (W, H).

The Example RealtimeManager.startCapture now passes those oriented dims into CameraCaptureOptions, so published track metadata matches what the user sees and the server’s eager output publish no longer triggers a republish path linked to intermittent missing remote video (black screen).

Reviewed by Cursor Bugbot for commit b254bff. Bugbot is set up for automated code reviews on this repo. Configure here.

@nagar-decart
nagar-decart force-pushed the fix/bug-b-orientation-aware-capture branch 7 times, most recently from d979ee2 to 5353cc4 Compare June 8, 2026 11:15
iOS counterpart to the Android fix in decart-android#16.

Adds a `CameraCaptureOptions.orientedForDevice(...)` static factory in
the SDK that takes the model's natural landscape (W,H) and transposes
to portrait when the current interface orientation is portrait. The
LiveKit camera capture session is then configured for the device's
actual orientation; the published track's metadata carries the displayed dims directly (LiveKit transports track dims via the AddTrackRequest signaling protocol, not via SDP), instead of relying on the WebRTC CVO rotation extension to communicate orientation to the receiver.

Why this matters: the realtime inference server's bridge eagerly
publishes its output video track at the dims the input track advertised in its publication metadata (so the
output SDP renegotiation overlaps the per-session time-to-first-output-frame
on the inference side, saving ~500 ms TTFF).
If those signaled dims don't match the orientation of the actual
decoded input frames, the bridge republishes the output track at the
correct orientation when the first model frame arrives.

That unpublish-old + publish-new on the same participant triggers an
intermittent transceiver-reuse race on the subscriber side: the
stopped RtpReceiver from the eager subscription stays in the
PeerConnection, libwebrtc reuses the transceiver slot for the new
track, onAddTrack never fires for it, addSubscribedMediaTrack never
runs, didSubscribeTrack never reaches the RoomDelegate, the renderer
is never attached, and the app shows a black screen until session
timeout. Empirically ~37 % rate on portrait-locked Lucy 2.1 sessions
in production sampling. The same root cause was fixed in Android via
decart-android#16; this is the iOS equivalent.

The Example app's `startCapture()` now calls the new helper instead
of constructing `CameraCaptureOptions` directly. Customer apps using
the SDK should switch to the same helper to get the fix.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@nagar-decart
nagar-decart force-pushed the fix/bug-b-orientation-aware-capture branch from 5353cc4 to b254bff Compare June 8, 2026 15:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant