feat(sim): P1 private-API framebuffer streaming + HID injection, crash-isolated (CODE-396)

crates/linkcode-sim/Cargo.toml

@@ -14,5 +14,10 @@ name = "linkcode-sim"
 path = "src/main.rs"
 
 [dependencies]
+block2 = "0.6.2"
+libc = "0.2.189"
+objc2 = { version = "0.6.4", features = ["exception"] }
+objc2-core-foundation = "0.3.2"
+objc2-foundation = "0.3.2"
 serde = { version = "1.0.228", features = ["derive"] }
 serde_json = "1.0.150"

crates/linkcode-sim/NOTICE

@@ -0,0 +1,15 @@
+linkcode-sim
+
+The private-framework interactive layer (framebuffer capture and HID injection under
+`src/private/`) ports recipes from baguette, an iOS Simulator control tool:
+
+    baguette — https://github.com/tddworks/baguette
+    Copyright (c) tddworks
+    Licensed under the Apache License, Version 2.0
+
+The undocumented CoreSimulator / SimulatorKit call sequences reproduced here — the IOHIDEvent
+digitizer tap recipe for iOS 26, the framebuffer descriptor enumeration and screen-callback
+registration, and the SimulatorKit framework path resolution — were derived from baguette's
+implementation. They are reimplemented in Rust; no baguette source is included verbatim.
+
+A copy of the Apache License 2.0 is available at https://www.apache.org/licenses/LICENSE-2.0.

crates/linkcode-sim/PROTOCOL.md

@@ -28,6 +28,7 @@ Sidecar to daemon:
 | ---: | --- | --- |
 | `0x81` | `RESULT` | JSON [`Result`](#result) |
 | `0x82` | `SCREENSHOT` | binary [`Screenshot body`](#screenshot-body) |
+| `0x83` | `STREAM_FRAME` | binary [`Stream frame body`](#stream-frame-body) (unsolicited, while a stream runs) |
 
 Unknown frame types are ignored. A malformed `REQUEST` fails only that request — the sidecar replies with an `invalidRequest` `RESULT` for its `requestId` and keeps serving. If the `requestId` itself cannot be recovered, the sidecar logs to stderr and the daemon's pending request is reclaimed by its own timeout.
 
@@ -58,6 +59,24 @@ Unknown frame types are ignored. A malformed `REQUEST` fails only that request 
 
 Per-op deadlines are enforced sidecar-side (`boot` 180s, `install` 120s, `screenshot` 30s, others 60s); a child that outlives its deadline is killed and reported as `timeout`.
 
+`probe` additionally reports `interactive: bool` — whether this host can drive simulators through the private-API path below (macOS with SimulatorKit). The daemon gates the interactive ops on it.
+
+### Ops (P1 — private API: HID injection + framebuffer streaming, macOS only)
+
+Off macOS, or when SimulatorKit is unavailable, every P1 op fails with `xcodeMissing`.
+
+| `type` | Params | Result |
+| --- | --- | --- |
+| `tap` | `udid`, `x`, `y` (normalised 0..1) | `{}` |
+| `swipe` | `udid`, `x0`, `y0`, `x1`, `y1`, `durationMs?` | `{}` |
+| `button` | `udid`, `button` (`home`/`lock`) | `{}` |
+| `streamStart` | `udid`, `fps?` (60), `quality?` (0.6), `scale?` (1.0) | `{ streaming, fps, scale }` — JPEG frames then arrive on `STREAM_FRAME`s |
+| `streamStop` | `udid` | `{}` |
+
+`scale` (0.1..1.0) downscales each frame before JPEG encode: at native resolution the encode bounds the frame rate near ~55 fps, so `scale` below 1.0 both lifts the achievable rate toward the display's 60 Hz and cuts bandwidth (e.g. `0.5` ≈ one third the bytes). `tap`/`swipe` stay in normalized 0..1 coordinates, so a downscaled stream needs no coordinate adjustment.
+
+Input (`tap`/`swipe`/`button`) runs in the sidecar's main process via a per-udid warmed HID client and is stable. Framebuffer streaming runs in a **crash-isolated worker subprocess**: the sidecar spawns it, reads its frames, and respawns it on the intermittent hard crashes of the private framebuffer path. If the worker crash-loops and gives up, the stream degrades to `simctl io screenshot` frames — slower, but frames never stop and the sidecar never crashes.
+
 ## Result
 
 Success:
@@ -92,6 +111,16 @@ A successful `screenshot` responds with raw image bytes instead of JSON, so capt
 
 A failed `screenshot` responds with a normal `RESULT` error. Exactly one of the two arrives per request.
 
+## Stream frame body
+
+While a `streamStart` stream runs, the sidecar pushes unsolicited `STREAM_FRAME`s (raw JPEG bytes, no base64):
+
+```text
+[u16 little-endian udid_length][udid UTF-8 bytes][image bytes]
+```
+
+Frames stop after `streamStop` (or when the daemon closes the pipe). The daemon routes them by `udid`.
+
 ## Lifecycle expectations
 
 - One sidecar process serves many concurrent requests; the daemon lazily starts it on first use.

crates/linkcode-sim/README.md

@@ -1,6 +1,6 @@
 # linkcode-sim
 
-The Link Code iOS Simulator host: one long-lived process the daemon spawns to drive Apple's iOS Simulator. P0 wraps the public `xcrun simctl` CLI only (lifecycle, install/launch, screenshot); the streaming/HID phase is CODE-396. The stdio wire contract is [`PROTOCOL.md`](./PROTOCOL.md); the TypeScript client is `@linkcode/sim` (`packages/host/sim`).
+The Link Code iOS Simulator host: one long-lived process the daemon spawns to drive Apple's iOS Simulator. P0 wraps the public `xcrun simctl` CLI (lifecycle, install/launch, screenshot); P1 adds live framebuffer streaming (JPEG `STREAM_FRAME`s) and touch/button injection through CoreSimulator/SimulatorKit private frameworks, in a crash-isolated capture worker. The stdio wire contract is [`PROTOCOL.md`](./PROTOCOL.md); the TypeScript client is `@linkcode/sim` (`packages/host/sim`).
 
 macOS-only at runtime (it needs Xcode's simulator tooling); it compiles and answers `probe` with a structured `xcodeMissing` error everywhere else, so workspace-wide `cargo test` stays green on any platform.
 
@@ -12,3 +12,24 @@ cargo test -p linkcode-sim --test device_loop -- --ignored  # full boot→instal
 ```
 
 Release binaries are staged by `apps/desktop/scripts/stage-sidecar.mts` into `apps/desktop/sidecar/${arch}` (macOS only) and shipped via electron-builder `extraResources`.
+
+## Benchmark
+
+`bench-encode` times the JPEG encode hot path — the per-frame CoreGraphics/ImageIO cost that bounds the framebuffer stream, since a single reader thread does the encode. `1000 / avg_ms` is the sustainable frame-rate ceiling. It needs no simulator: it encodes a synthetic BGRA frame across a resolution/quality sweep.
+
+```sh
+cargo build -p linkcode-sim --release
+./target/release/linkcode-sim bench-encode [iters]   # default 120 iters/config
+```
+
+On 2026-07-22 (macOS arm64), 150 iters/config:
+
+| Resolution | Quality | avg ms | p95 ms | Size | fps (avg) | fps (peak) |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: |
+| 1206×2622 (native) | 0.60 | 6.49 | 8.13 | 705K | 154 | 182 |
+| 904×1966 | 0.60 | 3.45 | 3.98 | 396K | 290 | 332 |
+| 603×1311 | 0.60 | 1.50 | 1.73 | 177K | 665 | 730 |
+| 402×874 | 0.60 | 0.77 | 0.95 | 79K | 1294 | 1441 |
+| 1206×2622 | 0.30 | 6.39 | 7.74 | 705K | 156 | 181 |
+
+Takeaways: the native-resolution encode ceiling (~154 fps) leaves ~2.5× headroom over 60 fps; cost is set by pixel count (the DCT), not JPEG quality (0.3 ≈ 0.6); downscaling is the real lever (half resolution ≈ 4× cheaper). The synthetic frame is high-entropy, so the numbers are conservative — a mostly-flat real screen encodes a touch faster. Use it to catch encode regressions and size the fps/resolution budget, not as an absolute.