Skip to content

gpu-browser (1/2): the server for rendering on your GPU through an agent-socket link - #1

Open
repalash wants to merge 19 commits into
masterfrom
claude/gpu-browser
Open

repalash wants to merge 19 commits into
masterfrom
claude/gpu-browser

Conversation

@repalash

Copy link
Copy Markdown
Collaborator

What this PR does

Adds gpu-browser/: a server that runs a headless Chromium on the owner's machine. An agent anywhere uses it through an agent-socket link to upload a page, run it on the owner's GPU and get screenshots back.

Why

Cloud agents can't render WebGL/WebGPU properly: their sandboxes have no GPU. The Chrome extension can drive a real tab, but rendering there only works while that tab is in front and visible. When another window covers it, macOS reports the tab as hidden and Chrome throttles it. A separate headless Chromium avoids both problems.

Part 1 of 2. This PR is the server and its tests. Part 2 adds npx @agent-socket/gpu-browser: the installer that creates a dedicated OS account (agent-socket-gpu-browser) to run it, and the CLI that handles approvals in the terminal. Until part 2 lands, the server only runs as a development run (README).

What the agent gets (explained to it in the link's agents.md):

  • Uploads: /files puts files in memory, served to pages at https://uploads.gpu-browser.internal/.
  • Pages: /page opens one at any viewport size, and /eval runs JavaScript in it.
  • Screenshots: /screenshot captures one. An image too big for a single relay frame comes back in parts, fetched with /image, so there's no 2.8 MB ceiling.
  • Own tools: /define turns a page script into a tool on the same link, for this session only.
  • More hosts: /request asks the owner to allow hosts beyond the default. It returns a task that the agent polls until the owner answers.

The boundary (app/policy.mjs, applied in app/server.mjs):

  • Network. Every Chromium request goes through a filtering proxy inside the server, with a password that changes every run, so no other local program can use it (that includes aibox containers, which can reach the host's loopback). It allows:

    • only hosts on the allowlist, which is empty by default;
    • only ports 80 and 443;
    • only hostnames, never IP addresses;
    • nothing that resolves to a loopback, private, link-local, CGNAT or multicast address, or to an IPv6 form that embeds an IPv4 address. This is checked on the address it actually connects to, so DNS rebinding doesn't get around it.

    Chromium resolves nothing itself, and QUIC and WebRTC UDP are off.

  • Chromium's sandbox is on. Playwright turns it off by default. Only a development run may turn it off, for root or CI.

  • Owner-supplied Chromium flags are limited to GPU settings. Anything else, such as --no-sandbox or proxy and debugging flags, is dropped.

  • Events the server sends to the CLI carry a per-run key, so agent text that ends up in a log line can't pose as an event.

Large replies: this follows options 3 and 4 of issues/open/relay-oversized-reply-ends-session.md for this app. Every reply is measured against the relay's frame budget before it's sent, and over-size images are split. The SDK-level guard from that issue is still separate work.

Checklist

  • Affects: new gpu-browser/, plus harness, CI and docs. The SDK is vendored into gpu-browser/app/lib/sdk like the extension's copy, and covered by the same freshness check.
  • Tests cover the new behavior:
    • gpu-browser/test/policy.unit.mjs, 8 tests, in a new CI job;
    • harness scenario 63: uploads, pages, a 1200×1200 noise PNG returned in parts and reassembled, a local server that must get 0 hits, the blocked hosts reported to the agent, the proxy refusing a client without its password (407), an approval round trip over stdin, forged event lines, /define, and stop ending the session.
  • npm test passes locally: type check, SDK 35, registry 74, extension unit, GPU browser unit 8, and all 55 harness scenarios, run with --continue.
  • Needs a relay redeploy: no, relay code is unchanged.
  • No secrets or API tokens in the diff.
  • READMEs, docs and CHANGELOG updated:
    • a README in every new folder;
    • CHANGELOG Unreleased;
    • SECURITY.md scope, how it's meant to work, and what counts as a vulnerability;
    • CONTRIBUTING;
    • the harness README.
    • PRIVACY.md is the extension's store policy, so it waits for part 2, when the GPU browser becomes installable.

The design was audited before implementation: an adversarial review plus a fact check of the macOS behavior part 2 relies on. Part 2 carries the installer findings.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CpTxf6FhMdiFuXFsbUam53


Generated by Claude Code

…cket link

A Chromium on the owner's machine that an agent anywhere uses through a
link: uploads (served from memory at https://uploads.gpu-browser.internal),
pages, JavaScript, screenshots (in parts when one reply can't hold them),
agent-defined tools for the session, and /request to ask the owner for
more hosts. Cloud sandboxes render WebGL in software or not at all; the
Chrome extension needs the tab in front.

The boundary is fixed in app/policy.mjs and app/server.mjs: every Chromium
request goes through a per-run, password-protected filtering proxy
(allowlisted public hosts only, ports 80/443, names only; loopback,
private, link-local, CGNAT and IPv4-embedding IPv6 addresses refused on
the address it connects to), Chromium's sandbox is on, owner-supplied
Chromium flags are limited to GPU settings, and widening needs the
owner's answer on stdin. Events to the CLI carry a per-run key.

Tests: unit tests for the checks (npm run gpu:test:unit, new CI job) and
harness scenario 63, which runs the server against the local relay. The
SDK is vendored into app/lib/sdk like the extension's, with the same
freshness check. The installer and CLI that run it under its own OS
account are the next PR.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CpTxf6FhMdiFuXFsbUam53
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-11T19:44:05.361098Z 829e8d6 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 24130fa3c0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread gpu-browser/app/policy.mjs Resolved
Comment thread gpu-browser/app/server.mjs Resolved
Comment thread gpu-browser/app/server.mjs Resolved
…eserve page slots

- fec0::/10 (deprecated site-local, still routable inside networks) is
  refused like the other local ranges.
- An owner's answer given while the relay link is down was lost:
  completeTask throws then, and the request was already gone, so the
  agent's poll waited forever. Unsent answers now go out on reconnect.
- Concurrent /page (or defined-tool) calls all passed the 8-page check
  before any page existed. A slot is now taken before the first await;
  harness 63 opens 10 at once and expects exactly 8 (it opened 10 before).
- The server refuses a session length that isn't a number of minutes,
  instead of running without a timer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CpTxf6FhMdiFuXFsbUam53

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 55e553d901

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread gpu-browser/app/server.mjs Resolved
Comment thread gpu-browser/app/server.mjs Resolved
Comment thread gpu-browser/app/server.mjs Resolved
… relaunched browser

On macOS without a display session, closing a context could take Chromium
down, and the next /page failed with "browser has been closed". The GPU
probe now opens a second page in the keeper's context and closes only that
page; the keeper page is required (no longer best effort). A new context
on a browser that has just exited waits for the relaunch and retries once.
The self-test now opens and closes a context and checks the browser is
still there afterwards, so an install catches this.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CpTxf6FhMdiFuXFsbUam53

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 432eab01e1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread gpu-browser/app/policy.mjs Resolved
Comment thread gpu-browser/app/server.mjs Resolved
…ers; long timers

- connectAny drops its fallback once a socket connects, so a reset on a
  live tunnel can't open the next address and write a second 200; an
  upstream that connects after the client left is closed.
- task_complete is fire-and-forget, so the last 50 approval answers are
  sent again after every reconnect (completing a task twice stores the
  same reply).
- The session timer re-arms in steps of at most 2^31-1 ms, so sessions
  longer than ~24.8 days don't end at once.
- A self-test whose Chromium dies at launch launches it once more with
  Chromium's log on stderr and reports the end of it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CpTxf6FhMdiFuXFsbUam53
…ff, then end

- ::ffff:0:0:0/96 (IPv4-translated, SIIT) is refused like ::ffff:0:0/96.
- A failed relaunch is retried after 1, 2, 4, 8 and 16 s; if Chromium
  still doesn't start, the session ends instead of answering
  "unavailable" for good. A launch whose keeper page fails closes its
  browser, so a retry never leaves one behind.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CpTxf6FhMdiFuXFsbUam53
The self-test runs with Playwright's DEBUG=pw:browser and Chromium's
--enable-logging, keeps the last lines instead of printing them, and adds
them to the failure report (Chromium on CI runners died at launch or
after a context closed with nothing else to go on).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CpTxf6FhMdiFuXFsbUam53
…art from its stack

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CpTxf6FhMdiFuXFsbUam53
…ME sudo passed

macOS's sudoers keeps the caller's HOME; the server now takes the
account's home from the user database, so Chromium's profile and temp
files never point at the caller's home.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CpTxf6FhMdiFuXFsbUam53
Some sudoers keep XDG_CONFIG_HOME (GitHub's Ubuntu runners do). Chromium
then tries to create its crash reports under the caller's home, can't,
and dies at launch (strace on CI: mkdir /home/runner/.config EACCES,
then SIGTRAP). The installed server clears XDG_* and points the config
and cache dirs at its own account's home.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CpTxf6FhMdiFuXFsbUam53

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f8faa7fe58

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread gpu-browser/app/policy.mjs Resolved
Comment thread gpu-browser/app/server.mjs Resolved
Comment thread gpu-browser/app/server.mjs Resolved
Comment thread gpu-browser/app/server.mjs Resolved
…pups; more reserved IPv4

- stdin has one line/close listener from the start, so a "stop" (or the
  close) arriving in the same chunk as the config line is acted on; it
  was dropped and the link stayed up.
- Pages opened with window.open() are closed (and noted in the page's
  console report), so the page limit can't be bypassed.
- 192.0.2.0/24, 198.51.100.0/24, 203.0.113.0/24 (documentation) and
  192.88.99.0/24 (6to4 relay) are refused like the other non-global ranges.
- Up to 100 approval answers are kept for resending, as many tasks as
  the relay keeps for a session.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CpTxf6FhMdiFuXFsbUam53

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: dd32d085f7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread gpu-browser/app/server.mjs Resolved
Comment thread gpu-browser/app/policy.mjs Resolved
Comment thread gpu-browser/app/server.mjs Resolved
…al unicast only

- Concurrent /info calls share one in-flight GPU probe instead of each
  opening a page in the keeper's context.
- onSessionChanged gets tokensRemapped as a Map; it was read with
  Object.values (always empty), so a new link after an unresumable
  reconnect was never shown to the owner.
- IPv6 destinations must be global unicast (2000::/3) and outside the
  documentation, benchmarking, ORCHID and SRv6 SID ranges.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CpTxf6FhMdiFuXFsbUam53

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bb8fd83828

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread gpu-browser/app/server.mjs Resolved
Comment thread gpu-browser/app/server.mjs Resolved
…nding approval requests

- connectAny gives each vetted address at most 5 s before trying the
  next, so one that drops packets can't use up the page's navigation
  time, and a CONNECT whose client goes away cancels its pending attempt.
- At most 10 /request approvals wait at a time; more get 429
  too_many_requests before the owner is prompted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CpTxf6FhMdiFuXFsbUam53

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f28a614927

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread gpu-browser/app/policy.mjs Resolved
Comment thread gpu-browser/app/server.mjs Resolved
… 2001::/23

- Proxied http:// requests go through connectAny too: each vetted
  address gets 5 s, the next is tried on failure, and a pending attempt
  is cancelled when the browser side closes.
- 2001::/23 (IETF protocol assignments: Teredo, benchmarking, ORCHID,
  AMT, AS112 and anycast services, none of them websites) is refused as
  a whole instead of a few of its prefixes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CpTxf6FhMdiFuXFsbUam53
…wn temp dir

Two starts racing past the CLI's pgrep check could both run, the second
wiping the first's temp dir. The installed server now takes a pid file
in its home atomically (O_EXCL); a second one exits 3 "already running",
a stale one (dead pid) is taken over, and it's removed on exit. The
self-test, which an update runs while a server may be up, takes no lock
and uses tmp-self-test, so it no longer wipes a running server's files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CpTxf6FhMdiFuXFsbUam53

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5e70d8d552

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread gpu-browser/app/server.mjs Resolved
…at most 4 addresses

Both proxy paths check whether the client is still there after the DNS
lookup before starting connectAny, and at most the first 4 vetted
addresses are tried (5 s each), so cancelled requests to a name with many
blackholed addresses can't pile up sockets and timers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CpTxf6FhMdiFuXFsbUam53
…the install lock

After taking its pid file the server checks the installer's lock (a
symlink to the installer's pid); while that pid is alive it exits 3
"being updated". The installer checks for a running server's pid file
while holding the lock, so a start racing an update can't end up with its
files swapped underneath it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CpTxf6FhMdiFuXFsbUam53
repalash pushed a commit that referenced this pull request Oct 11, 2026
…nning server

- The install directory and its marker come first, and the marker
  journals the group, account and AppArmor profile before each is made,
  so a first install killed hard (SIGKILL, power loss) is finished or
  undone by the next run, including a group left between groupadd and
  useradd (tested: reused, then removed when that run failed). The
  journal is emptied once an install finishes.
- The swap is refused while a server runs (its pid file), checked while
  holding the install lock; the server refuses to start while the lock is
  held (PR #1). Tested both ways on an install.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CpTxf6FhMdiFuXFsbUam53

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: adf72ead63

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread gpu-browser/app/server.mjs Resolved
Two starts that both found the same stale pid file could each remove it,
the later removal deleting the earlier one's fresh file, and both run.
After a takeover the server waits 300 ms and re-reads the file before
touching its temp dir; if it isn't its own, it exits 3. Tested: two
starts racing over a stale file, three rounds, exactly one server each.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CpTxf6FhMdiFuXFsbUam53

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 829e8d65cf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread gpu-browser/app/server.mjs Resolved
…m runs

The server's checks of its own pid file and of the installer's lock
verify with ps that the pid is still server.mjs / install.sh, so a pid
reused by something else after a crash doesn't block starts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CpTxf6FhMdiFuXFsbUam53
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

… registered

A page counted in both while its navigation ran, so slow navigations
could hit too_many_pages with half the slots free. The reservation is
released as the page is added to the map (harness 63's 10-concurrent ->
exactly 8 still holds).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CpTxf6FhMdiFuXFsbUam53
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants