feat(sim): P1 private-API framebuffer streaming + HID injection, crash-isolated (CODE-396)#255
feat(sim): P1 private-API framebuffer streaming + HID injection, crash-isolated (CODE-396)#255AprilNEA wants to merge 6 commits into
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 00815b349c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| pub const RESULT: u8 = 0x81; | ||
| pub const SCREENSHOT: u8 = 0x82; | ||
| /// A streamed framebuffer frame: `[u16 LE udid_len][udid][jpeg]` (unsolicited, while a stream runs). | ||
| pub const STREAM_FRAME: u8 = 0x83; |
There was a problem hiding this comment.
The new frame type and P1 operations have no peer in the shipped daemon: in the inspected packages/host/sim, client.ts exposes only the P0 methods through screenshot (lines 69-105), codec.ts defines only RESULT and SCREENSHOT (lines 12-13), and handleFrame silently discards 0x83 via its default branch (lines 186-187). Because call is private, no product path can send tap or streamStart, and even an externally initiated stream would have every frame dropped; extend the TypeScript codec/client/backend and frame routing with this protocol change.
AGENTS.md reference: AGENTS.md:L27-L27
Useful? React with 👍 / 👎.
| ptr::null(), | ||
| ptr::null(), | ||
| ); | ||
| CFRelease(number); |
There was a problem hiding this comment.
CFDictionaryCreate receives null key/value callbacks, so the dictionary does not retain number; releasing it here leaves options containing a dangling CFNumber before encode_cgimage_jpeg passes the dictionary to CGImageDestinationCreateWithData. Every private JPEG capture therefore risks a use-after-free, which can crash-loop the worker or lose the requested quality; use Core Foundation retaining callbacks or keep the number alive until the dictionary consumer is finished.
Useful? React with 👍 / 👎.
| self.stopped.store(true, Ordering::Relaxed); | ||
| if let Some(manager) = self.manager.take() { | ||
| let _ = manager.join(); |
There was a problem hiding this comment.
If the worker opens a Screen but no framebuffer callback arrives, such as during a boot or shutdown transition, it remains alive without writing stdout and pump_worker blocks in read_exact. Setting this atomic does not wake that read or terminate the child, so dropping the stream from streamStop can block forever in manager.join() and never send its RESULT; close or kill the worker before joining, or make the read interruptible.
Useful? React with 👍 / 👎.
| if let Some(frame) = stream.latest() { | ||
| let ptr = Arc::as_ptr(&frame); | ||
| if last != Some(ptr) { | ||
| last = Some(ptr); |
There was a problem hiding this comment.
When the pusher misses an intermediate worker frame, the prior Arc can be freed and its allocation address reused by a later frame, so persisting only this raw pointer creates an ABA comparison that treats a new JPEG as already sent. With producer and poller rates aligned, allocator address alternation can repeatedly suppress visible updates; retain the previous Arc and compare it with Arc::ptr_eq, or deduplicate with a generation counter.
Useful? React with 👍 / 👎.
| if reg.streams.contains_key(udid) { | ||
| return Ok(json!({ "alreadyStreaming": true })); |
There was a problem hiding this comment.
When streamStart is called for an already-running UDID, this branch returns only { alreadyStreaming: true }, whereas both the first-call implementation and PROTOCOL.md promise { streaming, fps }. A caller validating the documented result will fail specifically on an idempotent retry; return the normal fields as well, with alreadyStreaming only as an optional addition if needed.
Useful? React with 👍 / 👎.
Greptile SummaryThis PR adds interactive iOS Simulator control through private macOS APIs. The main changes are:
Confidence Score: 5/5No additional blocking issue qualifies for this follow-up review. The remaining stream-delivery and lifecycle failures are already covered by existing review findings. No separate production failure requiring an independent fix was identified.
 T-Rex LogsWhat T-Rex did
Important Files Changed
Reviews (2): Last reviewed commit: "feat(sim): configurable capture scale (d..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| let fallback_interval = Duration::from_millis(500); | ||
| let mut last: Option<*const Vec<u8>> = None; |
There was a problem hiding this comment.
Fallback starts only after is_dead() becomes true. If the worker stays alive but framebuffer registration yields no callbacks, it emits no JPEGs and is never marked dead, so this loop sends no frames indefinitely instead of switching to simctl.
Artifacts
Repro: runnable extraction, compilation, and controlled stream harness
- Contains supporting evidence from the run (text/x-python; charset=utf-8).
- Contains supporting evidence from the run (text/x-rust; charset=utf-8).
Repro: successful runtime trace contrasting the silent-alive stream with the dead-stream control
- Keeps the command output available without making the summary code-heavy.
Reach the frame sink through a process global instead of a block-captured pointer (ROCK's XPC delivery zeroes the capture), consume the screen callbacks' delivered surface instead of polling framebufferSurface on the XPC proxy every frame, and build the ImageIO options dict with the standard CFType callbacks. Sustained 15fps headless with no crashes.
thread::sleep coalesces timers on macOS and overshoots by several ms, so a plain deadline loop drifts below target (60fps landed ~52-56). Pace on the mach absolute clock instead: locks lower rates exactly (30->29.5) and drives 60fps to the full-res pipeline ceiling (~58.5), with no busy-spin.
Downscale each frame before JPEG encode via a high-quality CoreGraphics resample. Native-resolution encode caps the stream near 55fps; scale below 1.0 lifts it to the display's 60Hz and cuts bandwidth (0.5 -> ~1/3 the bytes, 603x1311 locks 60fps). Threaded from streamStart through StreamParams to the worker; tap/swipe stay normalized so no coordinate change is needed.
1 participant
Summary
Stacked on #254. The P1 private-API layer for
crates/linkcode-sim: live framebuffer streaming and touch/button injection on iOS 26, ported from baguette (Apache-2.0;NOTICEadded). Rust FFI viaobjc2(dynamic ObjC messaging) +libloading/dlsym(private C symbols) — no Swift.What works, verified on real hardware (iPhone 17 Pro / iOS 26.5):
tap/swipe/button, stable (10 taps + 5 swipes + 3 buttons, no crashes). Uses the iOS 26IOHIDEventdigitizer recipe (the 9-arg mouse fn is misread as a Home gesture on 26). Runs in the sidecar main process; wired into the RPC (probenow reportsinteractive: true).IOSurface→ JPEG. Delivered live 1206×2622 frames (clock advancing across captures). Key recipe details cracked: XPC-proxy objects need rawobjc_msgSend(notobjc2's verified macro); the callback block needsBLOCK_HAS_SIGNATURE; surfaces are recycled buffers so the callback only memcpies pixels (the reader thread encodes); a registration-race gate stops callbacks touching a half-registered descriptor.The hard part — crash isolation (engineering best-practice):
The private framebuffer path intermittently aborts hard (
SIGABRT"unknown class" inside CoreSimulator's XPC machinery) — uncatchable in-process, and worse on the Xcode 26SimStreamProcessorera. So capture runs in a disposable worker subprocess: the sidecar spawnscapture-worker, reads length-prefixed JPEG frames, and respawns it with backoff on crashes; a crash-loop gives up gracefully. Input and the simctl lifecycle RPCs stay in the parent and are never taken down by a capture crash. When the private worker gives up, the stream degrades tosimctl io screenshot— slower, but frames never stop.New sidecar frames/ops (
PROTOCOL.md):STREAM_FRAME+tap/swipe/button/streamStart/streamStop.Verification
cargo fmt/clippy/testclean.streamStartdelivers frames (private fast-path when the sim state cooperates — 41 frames/5s observed earlier; simctl fallback when the worker gives up — verified frames keep flowing through a crash-loop while the sidecar stays up).Follow-up
The private framebuffer path is fragile on the current Xcode 26 / iOS 26
SimStreamProcessorsimulators (timing-sensitive hard crashes). It is fully contained by the isolation + graceful-degradation architecture, so it never threatens the sidecar; hardening the private path to reliably hit ≥10fps headless on this Xcode is a contained follow-up (the worker + supervisor are the stable foundation, and baguette upstream is the signal source when Xcode shifts these APIs).