Skip to content

feat(schema,engine,desktop): browser broker — wire contract, webview executor, code-mode execute tool for claude/pi (CODE-267)#188

Open
AprilNEA wants to merge 9 commits into
xuan/code-266from
xuan/code-267
Open

feat(schema,engine,desktop): browser broker — wire contract, webview executor, code-mode execute tool for claude/pi (CODE-267)#188
AprilNEA wants to merge 9 commits into
xuan/code-266from
xuan/code-267

Conversation

@AprilNEA

Copy link
Copy Markdown
Member

Summary

Phase B-1 of the in-app browser roadmap (CODE-267), stacked on #181 (CODE-266 multi-tab panel — its tab store is this feature's execution surface). Agents get a single code-mode execute tool driving the SAME browser tabs the user sees, per CODE-33's locked code-mode decision.

  • Wire 37→38, new browser domain (packages/schema/src/wire/browser.ts + src/browser.ts): browser.host.register (client-minted capability, mirrors terminal attachments), Hub-synthesized browser.host.detached, browser.host.changed broadcast, daemon→host browser.command/browser.command.result (closed op enum + closed error codes with retryable), and the client-side browser.execute/browser.executed pair (used by the E2E now, by the B-2 stdio bridge later).
  • Hub targeted routing (first daemon→specific-client delivery): the connection whose browser.host.register succeeds becomes THE single active host (last registration wins — commands have side effects and are never fanned out); browser.command frames are delivered only there; disconnect synthesizes browser.host.detached (guarded by hostId so a stale drop can't clear a newer host).
  • BrowserBrokerService (engine): pending-map correlation by daemon-minted commandId, 15 s timeout, fail-all on host detach/supersede/shutdown, availability broadcasts. dispatch never rejects — every failure is a closed-code result.
  • Desktop executor: client-core BrowserHostChannel registers via the workbench hook useBrowserHostRegistration (re-registers per connection generation); the renderer executor drives the CODE-266 tab store (open/select/close/navigate) and live webviews (executeJavaScript snapshot with @eN marker-attribute refs capped at 200, click/type with stale-ref discipline, capturePage screenshots, evaluate). Screenshot results are attached as real image content blocks.
  • Code-mode REPL (packages/engine/src/browser/repl-host.ts, reviving the sky-mcp skeleton): persistent node:vm context exposing a browser namespace + captured console; behavior discipline is embedded in the tool description. claude-code hosts it as an in-process MCP server (createSdkMcpServer, approvals ride the existing canUseTool flow); pi hosts it via customTools (TypeBox).
  • Default OFF: EngineDeps.browserToolsEnabled, daemon env gate LINKCODE_BROWSER_TOOLS=1. The Settings toggle is a follow-up; host registration itself is ungated (harmless).

Verification

  • Unit: broker (7 — timeout/detach/supersede/correlation/availability), Hub routing (5 — targeting/no-broadcast/last-wins/detach synthesis/stale-drop guard), wire schema (5). Full suite 1301 passed.
  • E2E pnpm -F @linkcode/desktop e2e:browser-broker (PASS): isolated daemon (wire 38) + real desktop; a raw wire client executes tabs.open → the desktop executor opens a visible tab and returns the tab list; tab.click without a snapshot → stale-ref; desktop quit → host-unavailable instead of a hang.
  • Remaining (manual): a real claude-code session with LINKCODE_BROWSER_TOOLS=1 driving browser.* end-to-end (needs a signed-in claude), and a pi smoke. pi has no per-call approval seam — known limitation, gate-guarded.

Linear: CODE-267

@linear-code

linear-code Bot commented Jul 17, 2026

Copy link
Copy Markdown
CODE-267 feat(schema,engine,desktop): browser broker — wire contract, desktop webview executor, code-mode execute tool for claude/pi

目标

Agent 自动化 Phase B-1:paseo 式 daemon broker——agent 经 code-mode execute 工具在持久 REPL 里用注入的 browser namespace 控制桌面端 Browser 面板的 webview,与用户共享同一浏览面。模型面形态守 CODE-33 的锁定决策:code-mode REPL(非离散工具);传输拓扑为 daemon broker + 桌面 webview 执行(2026-07-17 拍板,否决 Codex 式多后端抽象)。

前提事实(2026-07-17 实勘)

  • daemon→定向客户端 RPC 不存在;Hub 仅 replyTo/terminal-attachment/广播三种路由,无 client 身份。蓝本:terminal attachment 的 capability 定向(packages/transport/src/hub.ts)+ permission ask 的 requestId 关联(engine.ts session.asks)。
  • claude-code SDK 支持进程内 MCP(createSdkMcpServer,审批过 canUseTool);pi 支持 customTools(TypeBox,无审批缝)。StartOptions.mcpServers 是两端全空的死字段,本 feature 不走它。
  • sky-mcp REPL host 骨架(持久 node:vm + console 捕获)在 feat/sky-scaffold 分支(wire v9 时代),需复活到当前 master。

Tasks

Wire(WIRE_PROTOCOL_VERSION 37→38,一次 bump 含 B-2 的 browser.execute 预留)

  • 新建 packages/schema/src/wire/browser.ts + packages/schema/src/browser.ts
    • browser.host.register { clientReqId, attachmentId, attachmentSecret }(仅 desktop 注册;Hub 记录连接为 browser host,断开清除;单活宿主,后注册者胜——命令有副作用不能双发)。
    • browser.command { commandId, op, args }(daemon→host 定向)+ browser.command.result { commandId, ok, data? | error: { code, message, retryable } }(封闭错误码)。
    • browser.execute { clientReqId, op, args }(client→daemon,B-2 用)。
    • browser.host.changed { available } 广播。

Engine

  • packages/engine/src/browser/broker.tsBrowserBrokerService,对齐 terminal-service.ts 模式):pending map + 15s 超时,宿主断开全 reject;engine.ts handle() 加两个 case 委托。
  • browser namespace 单写一份(op 映射 broker.dispatch):tabs()/open(url)/tab.navigate|back|reload|close/tab.snapshot()@eN ref)/tab.click(ref)/tab.type(ref, text)/tab.screenshot()/tab.evaluate(js) + browser.documentation()(行为纪律全文:先 snapshot 才能 click/type,ref stale 即重拍)。
  • REPL host:复活 sky-mcp 骨架,claude-code 经 createSdkMcpServer 注入单一 execute 工具(同进程直接函数调用),pi 经 customTools

桌面执行端

  • client-core 新增 browser-host-channel.ts(capability 铸造同 terminal-channel.ts);workbench 运行时在连接就绪后注册。
  • 命令分发:tab 管理直接操作 CODE-266 的 browser tab store;页面命令三档执行(executeJavaScript snapshot/合成事件;CDP webContents.debugger 截图 + trusted input,经 system-plane IPC 由 main 按封闭命令表白名单执行;Electron 原生导航)。
  • snapshot:注入脚本收集可见可交互元素(上限 200)→ @eN ref → selector 映射存渲染器内存,URL 变化即 stale。

安全与默认值

  • 默认关闭(Settings Agents 组开关,关闭时不注册 execute 工具);仅 http(s)(复用 isAllowedBrowserUrl)。
  • 审批粒度注意:code-mode 下审批面收敛为「execute JS」单工具,依赖既有 approval-policy + 默认关闭兜底;per-op 细粒度审批归 CODE-33 三层审批体系。pi 无审批缝记 known limitation。

验收

  • broker 单测(超时/宿主断开/后注册者胜/result 关联);wire lockstep 全量重启验证。
  • 真机:claude-code 会话「打开 localhost:5173 截图并点某按钮」→ agent 写 browser.* JS,右面板可见 agent 开的 tab 与操作;桌面客户端退出 → 返回 unavailable 封闭错误而非挂死;开关关闭 → execute 工具不注册。pi 同链冒烟。

风险

  • CDP debugger attach 与 devtools 互斥(同一 debugger 通道),v1 文档化不解决;HQ tunnel 场景 broker 理论可用但首版不主动验证。

Review in Linear

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 38e3923d2f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +68 to +69
const script = new Script(`(async () => {\n${code}\n})()`);
const value = (await script.runInContext(this.vmContext)) as unknown;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve REPL bindings across execute calls

When an agent follows the documented persistent-REPL behavior and declares a normal binding such as const tabId = ... in one execute call, this per-call async IIFE makes that binding local to the wrapper and discards it as soon as the call returns; the next execute sees ReferenceError instead of the promised persistent variable. Only accidental globals persist, so the browser tool won't behave like the documented REPL for normal JavaScript.

Useful? React with 👍 / 👎.

Comment on lines +68 to +69
const script = new Script(`(async () => {\n${code}\n})()`);
const value = (await script.runInContext(this.vmContext)) as unknown;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Put a timeout around browser-tool code

With LINKCODE_BROWSER_TOOLS=1, model-provided code runs in the daemon process here with no vm timeout or other guard, so a non-yielding snippet such as while (true) {} blocks the daemon event loop. In that case the broker's 15s command timeout and socket message handling cannot fire, leaving every connected client hung until the daemon is killed.

Useful? React with 👍 / 👎.

@greptile-apps

greptile-apps Bot commented Jul 17, 2026

Copy link
Copy Markdown

Greptile Summary

This PR adds a browser broker so code-mode agents can drive the same desktop browser tabs the user sees. The main changes are:

  • New browser wire/schema contracts for host registration, targeted commands, command results, and execute replies.
  • A daemon broker that routes browser operations to one active desktop host with timeout and detach handling.
  • A desktop webview executor for tab actions, snapshots, clicks, typing, screenshots, and page evaluation.
  • Browser execute tool integrations for Claude Code and Pi behind the LINKCODE_BROWSER_TOOLS gate.
  • Unit and E2E coverage for broker routing, schema validation, and desktop round-trips.

Confidence Score: 4/5

Mostly safe, with one contained reliability issue to fix before merging.

Broker routing, schema, host channel, and desktop executor changes are scoped and covered by tests. The REPL execution path lacks bounds, so one bad browser execute call can hang the affected live session or daemon path.

packages/engine/src/browser/repl-host.ts

T-Rex T-Rex Logs

What T-Rex did

  • Verified bound REPL execution by running a narrow Node/tsx harness against the real BrowserReplHost, observing a never-settling promise after 353ms and a forced SIGKILL for a synchronous infinite loop after 503ms.
  • Inspected browser broker E2E setup and documented blockers: xvfb-run failed due to missing xauth, manual Xvfb fallback launched but ended with a generic Electron launch error, and a subsequent debug run surfaced missing libgtk-3.so.0.

View all artifacts

T-Rex Ran code and verified through T-Rex

Important Files Changed

Filename Overview
packages/engine/src/browser/repl-host.ts Adds the persistent browser REPL, but execution lacks a timeout and can wedge a session or daemon on non-terminating code.
packages/engine/src/browser/broker.ts Adds the daemon-side browser broker with single-host dispatch, pending correlation, timeout, and availability broadcasts.
packages/transport/src/hub.ts Adds single active browser host routing and disconnect detach synthesis in the hub.
apps/desktop/src/renderer/src/shell/browser/browser-command-executor.ts Implements desktop webview-backed browser operations with arg validation, ref snapshots, screenshots, and URL allow-listing.
packages/client-core/src/client/browser-host-channel.ts Adds client-side browser host registration and command-result settlement handling.
packages/agent-adapter/src/native/claude-code.ts Hosts the browser execute tool through an in-process Claude MCP server when enabled.
packages/agent-adapter/src/native/pi.ts Hosts the browser execute tool through Pi customTools when enabled.
packages/schema/src/wire/browser.ts Adds browser host registration, command, settlement, and execute wire variants.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
participant Agent as Agent code-mode execute tool
participant Engine as Engine BrowserReplHost/Broker
participant Hub as Transport Hub
participant Desktop as Desktop BrowserHostChannel
participant Webview as Visible Electron webview

Agent->>Engine: "browser.* JS call"
Engine->>Engine: dispatch(op,args), mint commandId
Engine->>Hub: browser.command(commandId, op, args)
Hub->>Desktop: targeted delivery to active browser host
Desktop->>Webview: execute store/webview operation
Webview-->>Desktop: operation result / screenshot / error
Desktop->>Hub: browser.command.result(commandId, result)
Hub->>Engine: forward settlement
Engine-->>Agent: resolve execute result

alt host disconnects or times out
    Hub->>Engine: browser.host.detached(hostId)
    Engine-->>Agent: closed-code error result
end
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
participant Agent as Agent code-mode execute tool
participant Engine as Engine BrowserReplHost/Broker
participant Hub as Transport Hub
participant Desktop as Desktop BrowserHostChannel
participant Webview as Visible Electron webview

Agent->>Engine: "browser.* JS call"
Engine->>Engine: dispatch(op,args), mint commandId
Engine->>Hub: browser.command(commandId, op, args)
Hub->>Desktop: targeted delivery to active browser host
Desktop->>Webview: execute store/webview operation
Webview-->>Desktop: operation result / screenshot / error
Desktop->>Hub: browser.command.result(commandId, result)
Hub->>Engine: forward settlement
Engine-->>Agent: resolve execute result

alt host disconnects or times out
    Hub->>Engine: browser.host.detached(hostId)
    Engine-->>Agent: closed-code error result
end
Loading

Reviews (1): Last reviewed commit: "test(desktop): browser broker end-to-end..." | Re-trigger Greptile

Comment on lines +68 to +69
const script = new Script(`(async () => {\n${code}\n})()`);
const value = (await script.runInContext(this.vmContext)) as unknown;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Bound REPL execution
execute awaits arbitrary model-supplied code without any timeout, so while (true) {} blocks the daemon event loop before the returned promise is created and await new Promise(() => {}) leaves the adapter tool call pending forever. Broker commands time out, but this REPL layer does not, so one bad browser.execute invocation can wedge the live session until the process is restarted.

Artifacts

Repro: BrowserReplHost no-timeout runtime harness

  • Contains supporting evidence from the run (text/javascript; charset=utf-8).

Repro: harness output showing pending promise watchdog and sync-loop SIGKILL

  • Keeps the command output available without making the summary code-heavy.

View artifacts

T-Rex Ran code and verified through T-Rex

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant