fix(iris): orient injected frames like a real sensor - #20
Conversation
A camera is bolted into the device, so its buffers always arrive in the sensor's own orientation and the scene inside them turns as the device does. Consumers depend on that: they read the interface orientation and rotate by the matching quarter turns to get an upright picture. The host webcam is bolted to nothing, so its frames arrive already upright and the consumer's correction becomes the error — 90 degrees out in portrait, the other way upside down, 180 out in the opposite landscape. Only one orientation ever looked right, and which one depended on the sensor the consumer assumed. Counter-rotate by the correction the consumer is about to apply so the two cancel, leaving the injected device behaving like hardware from the outside. This belongs here rather than in any individual app: a device published by an injector is still an AVCaptureDevice, and an app that special-cased where its pixels came from would be working around a bug rather than consuming a camera. The buffer keeps the sensor's fixed dimensions. The fake device advertises one format, and a frame whose width and height swapped underneath it would contradict its own activeFormat — so a rotated frame is scaled to cover the sensor rectangle and centre-cropped, which is what a real sensor shows in portrait anyway: same optics, narrower slice of the world. Landscape-right needs no turn, and that path is untouched: the IOSurface still reaches the app zero-copy. Rotation goes through a recycled buffer pool rather than allocating per frame, and any failure falls through to the unrotated frame — a picture the wrong way up beats no picture. UIKit is read only on the main thread, with the orientation published to the delivery queue as an atomic; an orientation UIKit doesn't resolve (face up, unknown) leaves the last good value alone instead of snapping to a default. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Cool! @meatpaste, a screen recording would be appreciated 😄 Time to use your legendary Claude screen recording skill |
iris-frame-orientation.mp4 |
UIKit's landscape constants are the mirror of the device orientations that produce them, so pairing them up by name gets the two landscape cases backwards. The sensor's native landscape — the case needing no turn — is UIInterfaceOrientationLandscapeLeft, not LandscapeRight. The effect was that landscapeLeft kept its 180-degree error and landscapeRight, correct before this branch, regressed to 180 out: it was computing 2 turns rather than taking the zero-turn path at all. Verified on an iPad (A16) against the multicamera plugin, filming all four orientations under this build and under 961abaf, with a test card as the frame source. All four are now upright; portrait and portraitUpsideDown are unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Follow-up in 2ff4f31: the two landscape cases were mapped backwards, and the video above is filmed against that fix rather than the original commit.
The symptom was asymmetric, which is what made it worth chasing:
Portrait and Corrected, this is the table — the two landscape rows swap against what the description above says:
So the "Zero-copy in landscape-right" property in the description is really zero-copy in Found it by filming all four orientations twice — once under this branch, once under 961abaf — with the two builds side by side and only the dylib swapped, using a test card as the frame source so any quarter turn is unambiguous. The frame source is mirror-invariant by design, since the app I tested with previews the front camera and iOS mirrors that. |
2 participants
The gap
A hardware camera is bolted into the device. Its buffers always arrive in the sensor's own orientation — landscape-right for a back-facing sensor — and the scene inside them turns as the device turns. Apps depend on that: they read the interface orientation and rotate by the matching quarter turns to get an upright picture.
The host webcam is bolted to nothing, so its frames arrive already upright and the app's correction becomes the error. Rotate the simulator with
sim camrunning and the picture is only ever right in one orientation:landscapeRightportraitportraitUpsideDownlandscapeLeftThis counter-rotates by whatever the app is about to apply, so the two cancel and the injected device behaves like hardware from the outside.
Why here and not in the app
A device published by an injector is still an
AVCaptureDevice. An app that special-cased where its pixels came from would be working around a bug rather than consuming a camera — and every app would have to do it separately. The injector is also the only layer that can: it sees both the host frame and the simulated device's orientation.I tried it the other way first, in the app's camera plugin, and its maintainer pushed back for exactly this reason. He was right.
Two properties preserved
Fixed dimensions. The synthesised device advertises one
activeFormat, and a frame whose width and height swapped underneath it would contradict that format. A rotated frame is therefore scaled to cover the sensor rectangle and centre-cropped — which is what a real sensor shows in portrait anyway: the same optics over a narrower slice of the world.Zero-copy in landscape-right. No turn is needed there, so that path is untouched and the
IOSurfacestill reaches the app unmodified. Only the rotated orientations pay for a render, and those recycle aCVPixelBufferPoolrather than allocating per frame. Any failure falls through to the unrotated frame — a picture the wrong way up beats no picture.UIKitis read only on the main thread, with the quarter-turn count published to the delivery queue as an atomic. An orientation UIKit doesn't resolve (face up, unknown) leaves the last good value in place rather than snapping to a default.One build change:
-framework UIKitadded to theIrisInjectlink step, which is new for the dylib.Type of change
How this has been tested
go build ./...,go vet ./...,go test ./...all cleanEnd to end against a real Flutter app — the multicamera plugin driving an iPad (A16) simulator, with
sim cam start --cameraon the built-in webcam:Test configuration
go test ./...green)Iris/Scripts/build.shNo automated test accompanies this: the behaviour needs a live
UIWindowSceneand a running capture session inside the Simulator, which the Swift test targets here don't stand up. Happy to add one if you have a pattern in mind.Docs updated: a new Frame orientation section in
docs/SIM_CAM_ARCHITECTURE.md, alongside the frame-delivery flow it modifies.