feat(desktop,workbench,ui): iOS Simulator panel — on-demand section with live co-driving stream (CODE-397)#259
feat(desktop,workbench,ui): iOS Simulator panel — on-demand section with live co-driving stream (CODE-397)#259AprilNEA wants to merge 25 commits into
Conversation
Add STREAM_FRAME decode, tap/swipe/button/streamStart/streamStop, and a per-udid onFrame subscription so the engine can consume the sidecar's live framebuffer. Foundation for the CODE-397 simulator panel.
…backend port and service Extend SimulatorBackend and SimulatorService with claim-gated interactive ops and streamStart/streamStop plus a per-udid onFrame fan-out; dropping a device stops its stream idempotently. SimSidecarClient satisfies the widened port structurally. Transport wiring (wire variants, Hub targeting, client channel) lands next.
…ream wire (wire 45) Add tap/swipe/button/stream.start/stream.stop commands, a stream.started reply, and a session-scoped simulator.stream.frame push (base64 JPEG) — bump wire 44->45 (Invariant 1). The engine routes the commands and fans a device's frames out via the transport; the Hub scopes frames to attached sessions like agent.event (never a global broadcast); client-core gains the methods and a per-udid subscribeSimulatorFrames. Frames ride base64 in JSON like screenshots; a binary side-channel stays a remote/high-fps concern.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 289a0b2a52
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| /** Stop a device's framebuffer stream. Requires the session to hold the device. */ | ||
| async streamStop(sessionId: SessionId, udid: string): Promise<void> { | ||
| this.claim(sessionId, udid); |
There was a problem hiding this comment.
When releaseSession() has already run and the panel registry's deferred simulator.stream.stop arrives afterward, this call either recreates a claim for a user-booted device or clears the idle-reclaim timer for a service-booted device. Because the stopped session receives no second release callback, the device remains pinned to that dead session and subsequent sessions get conflict until the daemon restarts. streamStop should verify an existing owner without creating or refreshing a claim.
Useful? React with 👍 / 👎.
| /** Fan the device's frames out to the transport as session-scoped `simulator.stream.frame`s. | ||
| * Idempotent: a second `streamStart` for a device already fanning out keeps the one subscription. */ | ||
| private subscribeFrames(simulators: SimulatorService, sessionId: SessionId, udid: string): void { | ||
| if (this.frameSubs.has(udid)) return; |
There was a problem hiding this comment.
When a stream ends outside the simulator.stream.stop wire path—such as through session release or simulator shutdown—the backend stops, but this frameSubs entry is never removed. If another session later starts the same UDID, this early return retains the callback that captured the old sessionId, so every new frame is routed by the Hub to the old session and the current panel receives none. Remove these subscriptions during service lifecycle cleanup or replace them when the owning session changes.
Useful? React with 👍 / 👎.
| const device = pickDevice(devices, selectedUdid); | ||
| const udid = device?.udid ?? null; | ||
| const booted = device?.state === 'Booted'; | ||
| const canStream = sessionId !== null && udid !== null && booted; |
There was a problem hiding this comment.
On macOS hosts where simctl is available but SimulatorKit/private framebuffer access is unavailable, simulatorStatus().available remains true and this condition enables the live panel. The sidecar probe already returns interactive: false, but SimProbeSchema discards that field and the status contract never propagates it, so every stream start fails and Retry can never succeed. Surface and check the interactive capability before mounting the stream UI.
Useful? React with 👍 / 👎.
| (signal) => { | ||
| if (maskUrl == null) return; | ||
| const state = paintRef.current; | ||
| void fetch(maskUrl, { signal }) |
There was a problem hiding this comment.
In the desktop renderer, maskUrl is constructed as a data:image/png;base64,… URL, but apps/desktop/src/renderer/index.html does not allow data: in connect-src, which governs fetch(). The request is therefore rejected and swallowed by .catch(noop), leaving state.mask null and making the advertised device-specific outline silently fall back to generic rounding. Decode the base64 into a Blob directly or load it through a CSP-permitted image path.
Useful? React with 👍 / 👎.
Greptile SummaryThis PR adds an on-demand iOS Simulator panel with live co-driving. The main changes are:
Confidence Score: 5/5This looks safe to merge. No additional blocking issue was found in the updated stream lifecycle paths.
 T-Rex LogsWhat T-Rex did
Important Files Changed
Reviews (8): Last reviewed commit: "docs(sim): record the native-scroll-inje..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| if (entry) { | ||
| if (entry.closeTimer !== null) { | ||
| clearTimeout(entry.closeTimer); | ||
| entry.closeTimer = null; | ||
| } | ||
| entry.refCount += 1; | ||
| } else { | ||
| const created: RegistryEntry = { | ||
| refCount: 1, | ||
| closeTimer: null, | ||
| snapshot: { phase: 'starting', sessionId }, | ||
| listeners: new Set(), | ||
| }; | ||
| registry.set(udid, created); | ||
| entry = created; | ||
| startStream(client, registry, udid, created); |
There was a problem hiding this comment.
When the active thread changes while the same UDID remains acquired, this branch increments the refcount but keeps the first lease's sessionId. Later start and deferred-stop calls use that old claim, which can produce a claim conflict, stop the wrong session's stream, or leave the current panel without frames.
…keyboard (wire 48)
…r-move round-trip stutter
…modules Decouples the God component into framework-agnostic, unit-testable pure logic: device-geometry (coordinate mapping), device-compositor (canvas assembly), frame-decoder (H.264/JPEG decode). The component drops to event wiring + JSX.
…rolls don't stall
1 participant
Stacked on #255 (CODE-396). Closes CODE-397.
What
The desktop Simulator panel, end to end: an on-demand right-panel section summoned from a new "+" menu at the end of the section strip (mirroring the bottom panel's add-window flow), showing a live, touchable device screen.
simulator.tap/swipe/button/stream.start/stream.stopcommands,simulator.stream.startedreply, and session-scopedsimulator.stream.framepush — the Hub fans frames out likeagent.event, never as a global broadcast.@linkcode/simgains the STREAM_FRAME decoder and per-udidonFrame; client-core gainssubscribeSimulatorFrames.simulatorAddedmembership, active-section fallback).screenMasksidecar op rasterizes the devicetype bundle'sframebufferMaskPDF at runtime from the local Xcode install (publicsimctl list devicetypes→bundlePath; CoreGraphics render; nothing Apple-owned ships with Link Code). The renderer composites the whole device in native pixel space — mask-clipped screen, thin display band, titanium rim, side-button bumps — all concentric with the measured screen corners and scaling with the panel.Verification
pnpm -F @linkcode/desktop e2e:simulator(new): isolated daemon (profile-isolated appData) + built desktop → summon the section from "+" → real device list → wire-seeded pi session → reload restores the persisted section (v3) → live frames paint the canvas end-to-end →simulator.open-urldrives Safari to a real page → close restores the "+" menu. Full suite: 1798 tests, repo-wide typecheck/lint/format clean; sidecarcargo fmt/clippy/testclean.