Skip to content

ui: Add a Desktop surface for the sandbox's Computer Use desktop (VNC) - #48

Merged
MrgSub merged 2 commits into
mainfrom
devin/1789001051-desktop-surface
Sep 10, 2026
Merged

MrgSub merged 2 commits into
mainfrom
devin/1789001051-desktop-surface

Conversation

@devin-ai-integration

Copy link
Copy Markdown

Summary

A new right-pane surface, Desktop, that shows and drives the Keiki conversation sandbox's graphical desktop (Daytona Computer Use). Pairs with Mail-0/sms-kit#978, which mints the noVNC preview link.

Data path — no frames go through the platform:

Comet ── POST /conversations/:phone/desktop ──▶ sms-kit ──▶ Daytona getPreviewLink(6080)
Comet ── wss://<preview>/websockify  (x-daytona-preview-token) ──▶ noVNC/websockify ──▶ x11vnc

crates/ui/src/desktop.rs (new, self-contained):

  • connect() upgrades the preview URL to wss, sets the token header, and bridges the tungstenite socket to a tokio::io::duplex so vnc-rs's VncConnector sees a plain async stream.
  • run_session() on the Tokio runtime: negotiates Zrle/CopyRect/Raw/DesktopSize with PixelFormat::bgra(), polls VncEvents into a shared Framebuffer (BGRA, opaque), sends X11Event::Refresh on a ~60 Hz cadence, and drains an input channel so pointer/key events stay ordered.
  • Desktop entity: one Signal::Frame per burst → new RenderImage, previous one evicted via window.drop_image (otherwise the atlas leaks). Pointer positions map through the ObjectFit::Contain letterbox; wheel ticks are RFB buttons 4–7; keys go out as held modifiers ↓, key ↓↑, modifiers ↑ using the typed char's keysym (Latin‑1 direct, otherwise 0x01000000 + codepoint) or a named-key table.
  • Emits DesktopEvent::Gone when the open returns 404.

keiki-api: ConversationAction::Desktop + Client::open_desktop(...) -> Result<Option<DesktopViewer>, _> — 404 is Ok(None), the provider's word that the sandbox is gone.

shell.rs: RightSurface::Desktop(u64) wired into the picker, + menu, tab strip (icons::MONITOR), close, and render. Gated by desktop_offered() = Keiki chat and cached terminal availability (same sandbox record as the Terminal; no new availability endpoint). On Gone the shell drops the tab and flips the cached availability to false, so the picker stops offering it.

Deps: vnc-rs 0.5, image 0.25 (same major as gpui's, so its Frame is the one RenderImage::new takes), tokio-tungstenite (already in workspace).

Not touched: terminal transport, browser-view code, local-chat surfaces.

Verified: cargo fmt --check, clippy --workspace --all-targets (no new warnings), cargo test --workspace green. Not yet exercised against a live sandbox — the noVNC path (/websockify) and no-auth VNC assumption follow Daytona's launch.sh --vnc localhost:5901 --listen 6080.

Release Notes:

  • Added a Desktop surface that shows and controls a Keiki conversation's sandbox desktop over VNC.

Link to Devin session: https://app.devin.ai/sessions/4bd4a708feed454e9aa48d0b31cc6b4a
Open in Devin Desktop: https://app.devin.ai/desktop/session/4bd4a708feed454e9aa48d0b31cc6b4a?variant=devin
Requested by: @MrgSub

@devin-ai-integration

Copy link
Copy Markdown
Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR that start with 'DevinAI' or '@devin'.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@devin-ai-integration

Copy link
Copy Markdown
Author

Headed GPUI test against the paired local sms-kit branch (Mail-0/sms-kit#978) and a real Daytona XFCE sandbox.

  • Passed: gating (local chat / no-sandbox Keiki / sandbox Keiki) in picker and + menu; monitor tab → “Opening the desktop…” → live XFCE framebuffer.
  • Passed: remote typing (echo ok), window drag, wheel scrollback, Applications-menu click; close/reopen makes a fresh VNC connection; existing Terminal keeps working alongside. No panics or “desktop session ended” warnings.
  • Caveat: the very first click into a freshly opened Desktop (and once, into the existing Terminal) was swallowed until the Comet window had been clicked once — looks like X11 click-to-focus on the test box rather than the surface, but it is not pinned down.
  • Not exercised: production platform (no login on the box), and the cached-availability → 404 tab-invalidation path.
Remote typing Comet-driven menu click
Remote echo ok XFCE menu opened through Comet

Env notes: Vulkan/llvmpipe painted stale pixels; VK_ICD_FILENAMES=/dev/null (GL) fixed it. Sandbox ran x11vnc -display :0 -forever -shared -rfbport 5901 + websockify --web /usr/share/novnc 6080 localhost:5901, so the /websockify path and no-auth assumption held.

Desktop and Browser surfaces both landed additive changes in the same
spots (ConversationAction, RightSurface, picker/plus rows, icon map).
Desktop now uses the laptop icon since Browser took the monitor.
@MrgSub
MrgSub merged commit fa108d1 into main Sep 10, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant