gpu-browser (1/2): the server for rendering on your GPU through an agent-socket link

.github/workflows/ci.yml

@@ -60,8 +60,19 @@ jobs:
       - run: npm ci
       - run: npm run ext:test:unit
 
+  gpu-browser-unit:
+    name: GPU browser — unit
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: '22'
+          cache: 'npm'
+      - run: npm run gpu:test:unit
+
   vendor-freshness:
-    name: chrome-extension/lib/sdk vendor-freshness
+    name: vendored SDK freshness (chrome-extension, gpu-browser)
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
@@ -75,11 +86,12 @@ jobs:
       # check below fails the build and the PR is blocked.
       - run: npm run build -w sdk
       - run: bash chrome-extension/scripts/vendor-sdk.sh
+      - run: bash gpu-browser/scripts/vendor-sdk.sh
       - name: Assert no drift in vendored SDK
         run: |
-          git diff --exit-code -- chrome-extension/lib/sdk \
-            || (echo "::error::Vendored SDK in chrome-extension/lib/sdk/ is stale."; \
-                echo "Run: npm run build -w sdk && bash chrome-extension/scripts/vendor-sdk.sh"; \
+          git diff --exit-code -- chrome-extension/lib/sdk gpu-browser/app/lib/sdk \
+            || (echo "::error::A vendored SDK copy (chrome-extension/lib/sdk/ or gpu-browser/app/lib/sdk/) is stale."; \
+                echo "Run: npm run build"; \
                 exit 1)
 
   chrome-ext-e2e:

CHANGELOG.md

@@ -1,5 +1,9 @@
 # Changelog
 
+## Unreleased
+
+- **GPU browser** (`gpu-browser/`, new): the server for letting an agent anywhere render on your machine's GPU through an agent-socket link: uploads served at `https://uploads.gpu-browser.internal/`, pages, JavaScript, screenshots (in parts when bigger than one reply), agent-defined tools for the session, and `/request` to ask the owner for more hosts. Every Chromium request goes through a per-run, password-protected filtering proxy (allowlisted public hosts only; loopback, private, link-local and IPv4-embedding IPv6 addresses refused on the address it connects to), with Chromium's sandbox on. Harness scenario 63 and unit tests. The installer and CLI that run it under its own OS account come next; until then it runs only as a development run.
+
 ## 0.3.0
 
 Chrome extension 0.5.0.

CONTRIBUTING.md

@@ -13,7 +13,7 @@ npm install
 npm test
 ```
 
-`npm test` type-checks every package and runs the SDK tests, the registry tests, the integration harness and the extension unit tests. The harness builds the SDK and starts its own `wrangler dev`, so nothing needs to be running first. To run part of it:
+`npm test` type-checks every package and runs the SDK tests, the registry tests, the integration harness, the extension unit tests and the GPU browser unit tests (`npm run gpu:test:unit`). The harness builds the SDK and starts its own `wrangler dev`, so nothing needs to be running first. To run part of it:
 
 ```bash
 npm run build -w sdk
@@ -32,7 +32,7 @@ To run the relay on its own: `npm run dev` (port 8787). Layout and deploy notes
 
 ## Changing the SDK
 
-The extension ships a copy of the compiled SDK in `chrome-extension/lib/sdk/`. After changing `sdk/src`, run `npm run build` and commit the updated copy; CI fails if it is stale.
+The extension and the GPU browser ship a copy of the compiled SDK, in `chrome-extension/lib/sdk/` and `gpu-browser/app/lib/sdk/`. After changing `sdk/src`, run `npm run build` and commit the updated copies; CI fails if either is stale.
 
 ## Pull requests
 

README.md

@@ -80,6 +80,7 @@ Details and how to report a vulnerability: [SECURITY.md](SECURITY.md).
 - [docs/protocol.md](docs/protocol.md): URLs, frames, limits, threat model
 - [docs/self-hosting.md](docs/self-hosting.md)
 - [CONTRIBUTING.md](CONTRIBUTING.md): `npm install`, `npm test`
+- [gpu-browser/](gpu-browser/README.md): let an agent anywhere render on your machine's GPU (server; the installer is in progress)
 - [agent-socket-channel](https://github.com/blitzdotdev/agent-socket-channel): a chat room where several AIs talk through agent-socket
 
 ## License

SECURITY.md

@@ -12,6 +12,7 @@ Include the affected component (relay, SDK, Chrome extension, registry), steps o
 - `@agent-socket/sdk` (`sdk/`).
 - The Chrome extension (`chrome-extension/`).
 - The registry at registry.agentsocket.dev and any deployment of `registry/`.
+- The GPU browser (`gpu-browser/`).
 
 ## How it is meant to work
 
@@ -26,6 +27,8 @@ The protocol and threat model are in [docs/protocol.md](docs/protocol.md). In sh
 - **Tool output is untrusted input for the AI**, and agent input is untrusted input for the app.
 - **The extension gives the link holder the tab.** `/eval` runs any JavaScript in the bound tab, with the user's logged-in session on that site. The `/navigate` check for local and private-network hosts is a guardrail, not a boundary, because `/eval` can navigate too. The site lock is the boundary for where the tab goes: every tool call checks the tab's origin against the sites the user allowed (at first the one at Connect) and runs page code only in the document it checked, so a navigation by `/eval`, a link or a redirect pauses the tools instead of extending them. Within an allowed site, `/eval` can do whatever that site's pages can (including the site's own cross-origin requests). Only the bound tab is reachable, and Stop, disconnecting, closing the tab or the session timer (60 minutes by default) ends access.
 
+- **The GPU browser gives the link holder a browser, not your machine.** The agent's JavaScript runs in pages of a separate headless Chromium (its sandbox on), not in your browser. Every request goes through the server's filtering proxy: hosts the owner allowed, ports 80/443, names only, and never an address that is loopback, private, link-local, CGNAT, multicast or an IPv6 form that embeds IPv4 (checked on the address it connects to). The proxy listens on loopback with a password that changes every run. Uploads are served from memory at an `.internal` origin. Widening the allowlist needs the owner's answer. The agent can use the machine's GPU, CPU and memory until the session ends.
+
 Reports that only restate these properties are not treated as vulnerabilities.
 
 ## Limits
@@ -68,3 +71,4 @@ There is no global cap on sessions beyond the per-IP rate limit.
 - Memory exhaustion or session disruption beyond the limits above.
 - The extension acting on a tab other than the bound one, or after the session ended.
 - Bypassing registry review or admin authentication.
+- A GPU browser page reaching a host that isn't allowed, a loopback or private address, or a local file, or another program using the GPU browser's proxy; an agent widening the allowlist without the owner's answer.

gpu-browser/README.md

@@ -0,0 +1,56 @@
+# GPU browser
+
+A Chromium on your own machine, with your machine's GPU, that an agent anywhere (a cloud Claude session, any chat that can make HTTPS calls) uses through an agent-socket link: it uploads a page and its assets, opens it, runs JavaScript in it and gets screenshots back. Made for WebGL/WebGPU work (three.js and the like), which cloud sandboxes render in software or not at all.
+
+It is a separate app from the [Chrome extension](../chrome-extension/README.md): the extension lets an agent use a tab of *your* browser; the GPU browser is a different, headless Chromium that never sees your tabs, logins or files, and keeps rendering while you use your computer.
+
+## Status
+
+| Part | Where | State |
+|---|---|---|
+| Server: the browser, its network boundary, the agent's tools, the link | [`app/`](app/README.md) | this folder |
+| Installer + CLI: `npx @agent-socket/gpu-browser`, creates the separate OS account it runs as, approvals in your terminal | `install/`, `bin/` | next PR |
+
+Until the CLI lands, the server only runs as a development run (below).
+
+## What the agent can do
+
+The link's `agents.md` explains it to the agent; in short:
+
+| Tool | |
+|---|---|
+| `GET /info` | GPU (WebGL renderer, WebGPU), versions, allowed hosts, open pages, limits, when the session ends |
+| `POST /files`, `GET /files`, `POST /files/delete` | Upload files; pages load them from `https://uploads.gpu-browser.internal/<path>` (served from memory, never the network). Big files go in base64 pieces (`append`). |
+| `POST /page` | Open an uploaded page (or an https page on an allowed host) in a fresh context, any viewport and scale |
+| `POST /eval` | Run JavaScript in a page; returns its value and the console output |
+| `POST /screenshot`, `POST /image` | PNG/JPEG, viewport, element, clip or full page; images too big for one reply (the relay's 4 MiB frame) come in parts |
+| `POST /close` | Close pages (idle ones close after 15 minutes) |
+| `POST /request` | Ask the owner to allow more hosts (a CDN, an API); returns a task the agent polls until the owner answers |
+| `POST /define`, `POST /undefine` | Turn a page script into a tool on the same link, for this session |
+
+## The boundary
+
+Fixed in code ([`app/policy.mjs`](app/policy.mjs), applied by [`app/server.mjs`](app/server.mjs)); everything inside it needs no approval:
+
+- **Network.** Every request Chromium makes goes through a proxy inside the server, protected by a password that changes every run (so no other program on the machine can use it). It allows only hosts on the allowlist (empty by default), ports 80 and 443, names (no IP addresses), and only when every address the name resolves to is public. Loopback, private ranges, link-local (cloud metadata), CGNAT, multicast and the IPv6 forms that embed an IPv4 address are refused, checked on the address it then connects to, so DNS rebinding doesn't get around it. Chromium resolves nothing itself, and QUIC and WebRTC UDP are off. Blocked requests are reported to the agent so it can upload the file instead or ask.
+- **Code.** The agent's code runs only inside pages, with Chromium's sandbox on. Pages open only http(s) URLs.
+- **Widening needs the owner.** `/request` sends an `ask` to whoever runs the server; the answer comes back on its stdin. Owner-supplied Chromium flags are limited to GPU settings; anything else (`--no-sandbox`, proxy or debugging flags) is dropped.
+- **The link** is a bearer credential, as everywhere in agent-socket, and dies when the server stops or its timer runs out.
+
+In an install (next PR) the server also runs as its own OS account, `agent-socket-gpu-browser`, which can't read your files.
+
+## Development run
+
+```bash
+npm install && npm run build            # from the repo root
+# the config line, then your terminal as the rest of stdin: type `answer <id> allow`, or Ctrl-D to stop
+( echo '{"relay":"https://agentsocket.dev","key":"'"$(openssl rand -hex 16)"'","minutes":60}'; cat ) \
+  | node gpu-browser/app/server.mjs     # the link arrives as a `link` event
+```
+
+The first line on stdin is the config; see the header of [`app/server.mjs`](app/server.mjs) for the whole control protocol. A development run may set `"chromium": {"executablePath": "...", "sandbox": false}` (CI, root); an install can't.
+
+## Tests
+
+- `npm run gpu:test:unit`: the boundary checks ([`test/`](test/README.md)).
+- `node harness/run.mjs 63` with `CHROMIUM_PATH` set: the server end to end against a local relay (uploads, pages, multi-part screenshots, the private-network refusal with a local server that must get 0 hits, the proxy's password, an approval round trip, forged events, `stop`).

gpu-browser/app/README.md

@@ -0,0 +1,12 @@
+# gpu-browser/app
+
+What runs as the `agent-socket-gpu-browser` account in an install (and as you in a development run). Copied as a whole into the install; nothing here is published on its own.
+
+| File | |
+|---|---|
+| `server.mjs` | The server: the proxy, Chromium (via `playwright-core`), the agent's tools, the agent-socket session. Its header describes the stdin/stdout protocol with the CLI. `--self-test` launches Chromium, draws a WebGL canvas and prints the GPU it got (for the installer). |
+| `policy.mjs` | The boundary checks (hosts, addresses, upload paths, tool paths, Chromium flags, text shown to the owner). No I/O; unit tested in `../test/`. |
+| `lib/sdk/` | This repo's `@agent-socket/sdk`, vendored by `../scripts/vendor-sdk.sh` (CI checks it is fresh). |
+| `package.json` | Its one dependency, `playwright-core`, pinned exactly. |
+
+See [../README.md](../README.md) for what it does and its security boundary.

gpu-browser/app/lib/README.md

@@ -0,0 +1,3 @@
+# gpu-browser/app/lib
+
+Code the server uses that doesn't live in this folder's sources: `sdk/` is this repo's `@agent-socket/sdk`, copied in by `../../scripts/vendor-sdk.sh` (see `sdk/VENDORED.md`). Don't edit it here.

gpu-browser/app/lib/sdk/VENDORED.md

@@ -0,0 +1,16 @@
+# Vendored SDK
+
+Files in this directory are mechanically copied from `../../../sdk/dist/`
+(the `@agent-socket/sdk` in this repo), so the GPU browser server runs the
+SDK it was tested with.
+
+**Do not edit them directly** — changes will be overwritten next time
+`gpu-browser/scripts/vendor-sdk.sh` runs.
+
+To refresh after changing the SDK source:
+
+```
+npm run build      # builds the SDK and vendors it here and into the extension
+```
+
+CI fails if the vendored files drift from `sdk/dist/`.

gpu-browser/app/lib/sdk/agents-md.js

@@ -0,0 +1,39 @@
+// defaultAgentsMd helper — emits a standard agents.md template that
+// app authors can use to brief any AI joining their app. App-specific
+// app name / description / capabilities / conventions are injected.
+//
+// NOTE: you don't strictly need this — the relay prepends a canonical "how to call
+// tools" contract (do-not-recite, $BASE, GET $BASE/tools.json, POST $BASE/<path>) to
+// any served agents.md that doesn't already include it. So a custom agentsMd can be
+// app-content-only and the agent will still drive the app. This template embeds
+// the FRAMEWORK_PREAMBLE (with its marker) at the top so the relay won't double
+// it up — single source of truth for the contract lives in ./preamble.ts.
+import { FRAMEWORK_PREAMBLE } from "./preamble.js";
+export function defaultAgentsMd(opts) {
+    const caps = (opts.capabilities ?? ["Call any tool listed in tools.json"])
+        .map((c) => `- ${c}`)
+        .join("\n");
+    const lims = (opts.limitations ?? [])
+        .map((c) => `- ${c}`)
+        .join("\n") || "_none specified_";
+    const conventions = opts.conventions ?? "Read tools.json before any write.";
+    // FRAMEWORK_PREAMBLE owns the calling contract + marker; what we emit
+    // here is app-content-only. The relay's dedup detects the embedded
+    // marker and skips its own prepend, so the served doc has the contract
+    // exactly once.
+    return `${FRAMEWORK_PREAMBLE}# ${opts.appName}
+
+${opts.appDescription}
+
+**Save this URL** in case your context is compacted: \`${opts.agentsMdUrl}\`
+
+## What you can do
+${caps}
+
+## What you cannot do
+${lims}
+
+## Conventions
+${conventions}
+`;
+}

gpu-browser/app/lib/sdk/backoff.js

@@ -0,0 +1,35 @@
+// Default backoff helpers for ConnectOptions.onDisconnect.
+/**
+ * Exponential backoff with ±25% jitter.
+ * Default: 1s → 2s → 4s → 8s → ... capped at maxMs (default 30s).
+ *
+ * Returns a DisconnectHandler that schedules the next reconnect via setTimeout,
+ * then calls reconnect(). Apps that want bespoke timing can supply their own.
+ */
+export function exponentialBackoff(opts = {}) {
+    const baseMs = opts.baseMs ?? 1000;
+    const maxMs = opts.maxMs ?? 30000;
+    const jitter = opts.jitter ?? 0.25;
+    return ({ attempt, reconnect }) => {
+        const exp = Math.min(maxMs, baseMs * Math.pow(2, Math.max(0, attempt - 1)));
+        const j = exp * jitter;
+        const delay = Math.max(0, exp + (Math.random() * 2 - 1) * j);
+        setTimeout(reconnect, delay);
+    };
+}
+/**
+ * Linear backoff: a fixed delay between every attempt.
+ */
+export function linearBackoff(opts = {}) {
+    const delay = opts.delayMs ?? 5000;
+    return ({ reconnect }) => {
+        setTimeout(reconnect, delay);
+    };
+}
+/**
+ * Reconnect immediately. Useful for tests or apps that want their own
+ * outer backoff. NOT recommended in production.
+ */
+export function noBackoff() {
+    return ({ reconnect }) => reconnect();
+}

gpu-browser/app/lib/sdk/heartbeat.js

@@ -0,0 +1,9 @@
+// The heartbeat frames, byte for byte. The relay registers this pair with
+// the Durable Object runtime (setWebSocketAutoResponse), which answers the
+// ping itself without waking a hibernated session; that only works for an
+// exact string match, so the SDK sends exactly HEARTBEAT_PING. Relays that
+// predate auto-response answer it like any ping ({type:"pong", id}), which is
+// the same string.
+export const HEARTBEAT_ID = "as_hb";
+export const HEARTBEAT_PING = `{"type":"ping","id":"${HEARTBEAT_ID}"}`;
+export const HEARTBEAT_PONG = `{"type":"pong","id":"${HEARTBEAT_ID}"}`;

gpu-browser/app/lib/sdk/index.js

@@ -0,0 +1,4 @@
+// Public entry point for @agent-socket/sdk.
+export { connect, endSession } from "./session.js";
+export { exponentialBackoff, linearBackoff, noBackoff } from "./backoff.js";
+export { defaultAgentsMd } from "./agents-md.js";