feat(ai-computer-use, ai-computer-playwright, ai-computer-sandbox, ai-computer-local): computer use on OpenAI and Anthropic, with Playwright, sandbox, and local drivers

.agent/domain-map.json

@@ -0,0 +1,161 @@
+{
+  "version": 1,
+  "domains": [
+    {
+      "id": "core",
+      "owns": ["packages/ai", "packages/ai-utils", "packages/ai-event-client"],
+      "talksTo": []
+    },
+    {
+      "id": "provider-adapters",
+      "owns": [
+        "packages/openai-base",
+        "packages/ai-openai",
+        "packages/ai-anthropic",
+        "packages/ai-gemini",
+        "packages/ai-ollama",
+        "packages/ai-bedrock",
+        "packages/ai-byteplus",
+        "packages/ai-cloudflare",
+        "packages/ai-cohere",
+        "packages/ai-elevenlabs",
+        "packages/ai-fal",
+        "packages/ai-grok",
+        "packages/ai-groq",
+        "packages/ai-llmgateway",
+        "packages/ai-lovable",
+        "packages/ai-mistral",
+        "packages/ai-ollaya",
+        "packages/ai-openrouter",
+        "packages/ai-perplexity",
+        "packages/ai-reactor",
+        "packages/ai-typesafe",
+        "packages/ai-vercel-gateway",
+        "packages/ai-vertex",
+        "packages/ai-worldlabs"
+      ],
+      "talksTo": ["core"]
+    },
+    {
+      "id": "harness",
+      "owns": [
+        "packages/ai-acp",
+        "packages/ai-claude-code",
+        "packages/ai-codex",
+        "packages/ai-opencode",
+        "packages/ai-grok-build"
+      ],
+      "talksTo": ["core", "sandbox"]
+    },
+    {
+      "id": "client",
+      "owns": [
+        "packages/ai-client",
+        "packages/ai-react",
+        "packages/ai-solid",
+        "packages/ai-vue",
+        "packages/ai-svelte",
+        "packages/ai-angular",
+        "packages/ai-preact",
+        "packages/ai-remix",
+        "packages/ai-octane",
+        "packages/ai-react-ui",
+        "packages/ai-solid-ui",
+        "packages/ai-vue-ui"
+      ],
+      "talksTo": ["core"]
+    },
+    {
+      "id": "code-mode",
+      "owns": [
+        "packages/ai-code-mode",
+        "packages/ai-code-mode-snippets",
+        "packages/ai-isolate-cloudflare",
+        "packages/ai-isolate-daytona",
+        "packages/ai-isolate-e2b",
+        "packages/ai-isolate-node",
+        "packages/ai-isolate-quickjs",
+        "packages/ai-isolate-quickjs-bun"
+      ],
+      "talksTo": ["core"]
+    },
+    {
+      "id": "sandbox",
+      "owns": [
+        "packages/ai-sandbox",
+        "packages/ai-sandbox-blaxel",
+        "packages/ai-sandbox-boxd",
+        "packages/ai-sandbox-cloudflare",
+        "packages/ai-sandbox-daytona",
+        "packages/ai-sandbox-docker",
+        "packages/ai-sandbox-e2b",
+        "packages/ai-sandbox-local-process",
+        "packages/ai-sandbox-railway",
+        "packages/ai-sandbox-sprites",
+        "packages/ai-sandbox-upstash-box",
+        "packages/ai-sandbox-vercel"
+      ],
+      "talksTo": ["core", "middleware", "persistence"]
+    },
+    {
+      "id": "middleware",
+      "owns": [
+        "packages/ai-compaction",
+        "packages/ai-memory",
+        "packages/ai-skills",
+        "packages/ai-mcp"
+      ],
+      "talksTo": ["core"]
+    },
+    {
+      "id": "persistence",
+      "owns": ["packages/ai-persistence", "packages/ai-durable-stream"],
+      "talksTo": ["core"]
+    },
+    {
+      "id": "devtools",
+      "owns": [
+        "packages/ai-devtools",
+        "packages/react-ai-devtools",
+        "packages/solid-ai-devtools",
+        "packages/svelte-ai-devtools",
+        "packages/preact-ai-devtools"
+      ],
+      "talksTo": ["core"]
+    },
+    {
+      "id": "computer-use",
+      "owns": [
+        "packages/ai-computer-use",
+        "packages/ai-computer-playwright",
+        "packages/ai-computer-sandbox"
+      ],
+      "talksTo": ["core"]
+    },
+    {
+      "id": "e2e",
+      "owns": ["testing/e2e"],
+      "talksTo": [
+        "client",
+        "code-mode",
+        "computer-use",
+        "core",
+        "harness",
+        "middleware",
+        "persistence",
+        "provider-adapters",
+        "sandbox"
+      ]
+    },
+    {
+      "id": "docs",
+      "owns": ["docs"],
+      "talksTo": []
+    },
+    {
+      "id": "model-sync",
+      "owns": ["scripts/model-sync"],
+      "talksTo": ["provider-adapters"]
+    }
+  ]
+}

.changeset/ai-computer-use.md

@@ -0,0 +1,45 @@
+---
+'@tanstack/ai-computer-use': minor
+'@tanstack/ai-computer-playwright': minor
+'@tanstack/ai-computer-sandbox': minor
+'@tanstack/ai-computer-local': minor
+'@tanstack/ai': minor
+'@tanstack/openai-base': minor
+'@tanstack/ai-openai': minor
+'@tanstack/ai-anthropic': minor
+---
+
+Add computer use that runs the same code on OpenAI and on Anthropic. A driver controls the machine. The Playwright and sandbox drivers give each thread its own machine.
+
+New packages:
+
+- `@tanstack/ai-computer-use` adds `createComputerUse()`. It returns `tools`, a middleware, a safety-check interrupt, and `close`. Each `threadId` gets its own machine, action queue, and approvals. A thread that stays idle for `idleTimeout` (10 minutes by default) closes.
+- `@tanstack/ai-computer-playwright` adds `playwrightComputer()`, which opens one browser page for each thread. `createPlaywrightComputerDriver()` controls one page that you give it. Both add a `navigate` tool that opens `http:` and `https:` URLs.
+- `@tanstack/ai-computer-sandbox` adds `sandboxComputer()`, which creates one Linux desktop sandbox for each thread from any `@tanstack/ai-sandbox` provider, like `e2bSandbox({ template: 'desktop' })`. `createSandboxComputerDriver()` controls one sandbox that you give it.
+- `@tanstack/ai-computer-local` adds `localComputer()`, which lets the model use the mouse, keyboard, and main screen of the machine that runs `chat()`. It works on Windows, macOS, and Linux with `@jitsi/robotjs`.
+
+Changes in existing packages:
+
+- `@tanstack/ai` adds `threadId` to `ToolExecutionContext`. It also adds the computer tool metadata reader in `@tanstack/ai/adapter-internals`.
+- `@tanstack/ai-anthropic` and `@tanstack/openai-base` send the generic computer tool to the model as the native computer tool of the provider.
+- `@tanstack/ai-openai` marks the models that support the `computer` tool.
+
+```ts
+const computerUse = createComputerUse({
+  driver: playwrightComputer({ startUrl: 'https://tanstack.com' }),
+})
+
+chat({
+  adapter,
+  messages,
+  threadId,
+  tools: computerUse.tools,
+  middleware: [computerUse.middleware],
+  interrupts: [computerUse.interrupt],
+})
+
+await computerUse.close(threadId) // one thread
+await computerUse.close() // every thread, and the browser
+```
+
+The `ai-core` agent skill routes computer use to `@tanstack/ai-computer-use` and its three drivers.

.changeset/anthropic-computer-toolset.md

@@ -0,0 +1,10 @@
+---
+'@tanstack/ai-anthropic': minor
+---
+
+Run computer use on every Claude model that supports it.
+
+- `computerUseTool()` takes `computer_toolset_20260801`, the only computer use version of Claude Opus 5.5, Sonnet 5.5, and Haiku 5.5. The model calls each action as its own tool. The adapter runs each call as one `computer` call with the action in `input.action`, and sends it back to Claude in the form that Claude sent. So one `execute` works for every version.
+- `computerUseTool()` takes `computer_20251124`, with its `computer-use-2025-11-24` beta header.
+- Each model now takes only the computer use versions that it supports. For example, `computer_20250124` on `claude-opus-4-6` is a type error, because Anthropic rejects that pair.
+- `@anthropic-ai/sdk` is now `^0.135.0`.

.changeset/anthropic-computer-use-beta.md

@@ -0,0 +1,5 @@
+---
+'@tanstack/ai-anthropic': patch
+---
+
+Send the `anthropic-beta` header that `computerUseTool()` needs: `computer-use-2025-01-24` for `computer_20250124`, and `computer-use-2024-10-22` for `computer_20241022`. Before, a request with the tool had no computer use beta, so Anthropic rejected it.

.changeset/daytona-spawn-unhandled-rejection.md

@@ -0,0 +1,5 @@
+---
+'@tanstack/ai-sandbox-daytona': patch
+---
+
+A background process from `spawn()` no longer crashes Node with an unhandled rejection when its status poll fails, for example after `destroy()`. `wait()` still rejects with the error.

.changeset/openai-computer-call-loop.md

@@ -0,0 +1,9 @@
+---
+'@tanstack/openai-base': patch
+---
+
+The computer use loop of `computerUseTool()` runs. A `computer_call` from the model is now a tool call to `computer_use_preview`, with the `action` and the `pending_safety_checks` as its input. The tool result goes back as `computer_call_output`, with the first image of the result as the screenshot. When the call does not fail, the output acknowledges the safety checks of the call.
+
+A failed computer call sends the newest screenshot of the conversation, because the failed call has no screenshot of its own. A computer call output has no text, so the reason for the failure goes to the model in a user message after the call outputs.
+
+A computer tool result with no image fails the run with an error. Before, the adapter sent an empty screenshot.

.changeset/openai-computer-tool.md

@@ -0,0 +1,6 @@
+---
+'@tanstack/openai-base': minor
+'@tanstack/ai-openai': minor
+---
+
+`computerUseTool({ type: 'computer' })` adds the computer tool of the newer OpenAI models, for example `gpt-6.1-sol`. The tool name is `computer`. Each call has the batched `actions` of the model and the `pending_safety_checks`, and the replay sends the actions back as they came. `computerUseTool({ type: 'computer_use_preview', ... })` for the `computer-use-preview` model works as before. The Responses tool choice type also accepts `computer`. Computer screenshots now go out with `detail: 'original'`, as in OpenAI's computer use guide.

.changeset/sandbox-skill-computer-use.md

@@ -0,0 +1,5 @@
+---
+'@tanstack/ai-sandbox': patch
+---
+
+The `ai-sandbox` skill now says how to use a sandbox provider for computer use, with `sandboxComputer()` from `@tanstack/ai-computer-sandbox`.

docs/adapters/anthropic.md

@@ -618,30 +618,75 @@ equivalent — see [Provider Skills](../tools/provider-skills.md).
 
 ### `computerUseTool`
 
-Allows Claude to observe a virtual desktop (screenshots) and interact with it
-via keyboard and mouse events. Provide the screen resolution so Claude can
-calculate accurate coordinates.
+Claude can look at a screen and use the mouse and the keyboard. Your app runs each action and sends back a screenshot. `createComputerUse` from `@tanstack/ai-computer-use` does this on a Playwright browser or an E2B desktop. Each thread gets its own machine:
 
 ```typescript
 import { chat } from "@tanstack/ai";
 import { anthropicText } from "@tanstack/ai-anthropic";
-import { computerUseTool } from "@tanstack/ai-anthropic/tools";
+import { createComputerUse } from "@tanstack/ai-computer-use";
+import { playwrightComputer } from "@tanstack/ai-computer-playwright";
+
+declare const threadId: string;
+
+const computerUse = createComputerUse({
+  driver: playwrightComputer({ startUrl: "https://tanstack.com" }),
+});
 
 const stream = chat({
-  adapter: anthropicText("claude-sonnet-4-6"),
-  messages: [{ role: "user", content: "Open the browser and go to example.com" }],
-  tools: [
-    computerUseTool({
-      type: "computer_20250124",
-      name: "computer",
-      display_width_px: 1024,
-      display_height_px: 768,
-    }),
-  ],
+  adapter: anthropicText("claude-opus-5-5"),
+  messages: [{ role: "user", content: "Find the TanStack AI docs" }],
+  threadId,
+  tools: computerUse.tools,
+  middleware: [computerUse.middleware],
+  interrupts: [computerUse.interrupt],
 });
 ```
 
-**Supported models:** Claude Sonnet 3.5 and above, with two exceptions. `claude-opus-5-fast` takes no provider tools. Claude Opus 5.5, Claude Sonnet 5.5, and Claude Haiku 5.5 accept only the `computer_toolset_20260801` toolset, which the adapter does not offer yet. See [Provider Tools](../tools/provider-tools.md#which-models-support-which-tools).
+The adapter sends the `computer` tool to Claude as the newest native computer tool that your model takes.
+
+- For the setup, the threads, and how to close the machines, see [Computer use](../computer-use/computer-use.md).
+- For your own machine, see [Custom driver](../computer-use/custom-driver.md).
+
+#### The raw tool
+
+`computerUseTool` from `@tanstack/ai-anthropic/tools` is the native tool with no `execute`. Use it when your app runs the actions itself. Pick the version that your model takes:
+
+| Version | Models |
+| --- | --- |
+| `computer_toolset_20260801` | Claude Opus 5.5, Sonnet 5.5, Haiku 5.5, Fable 5.1, Fable 5, Opus 5, Sonnet 5, Opus 4.8 |
+| `computer_20251124` | Claude Fable 5.1, Fable 5, Opus 5, Sonnet 5, Opus 4.8, Opus 4.7, Opus 4.6, Sonnet 4.6, Opus 4.5 |
+| `computer_20250124` | Claude Sonnet 4.5, Haiku 4.5, Opus 4.1 |
+
+```typescript
+import { computerUseTool } from "@tanstack/ai-anthropic/tools";
+
+const computer = computerUseTool({ type: "computer_toolset_20260801" });
+```
+
+The earlier versions also take a name and the screen size:
+
+```typescript
+import { computerUseTool } from "@tanstack/ai-anthropic/tools";
+
+const computer = computerUseTool({
+  type: "computer_20251124",
+  name: "computer",
+  display_width_px: 1280,
+  display_height_px: 800,
+});
+```
+
+Add an `execute` that runs the action and returns a result:
+
+- Every version calls `execute` with the action in `input.action`. With the toolset, Claude calls each action as its own tool. The adapter gives you each call as one `computer` call.
+- Return a screenshot as an image part for the `screenshot` action, and `"OK"` for the other actions.
+- Send screenshots that fit the image limits of the model. The toolset refuses a large screenshot. It does not make it smaller.
+- The toolset turns on `zoom` by default. Return an image for the `zoom` action, or turn it off with `configs: { zoom: { enabled: false } }`.
+- The adapter sends the beta header that an earlier version needs. The toolset needs no beta header.
+
+Claude gives coordinates in the pixels of your screenshots.
+
+A version that your model does not take is a type error. `claude-opus-5-fast` takes no provider tools. See [Provider Tools](../tools/provider-tools.md#which-models-support-which-tools).
 
 ### `bashTool`