feat(openai-base, ai-openai): add the computer tool by AlemTuzlak · Pull Request #1833 · TanStack/ai

AlemTuzlak · 2026-10-19T17:08:25Z

Apps can now give the newer OpenAI models, for example gpt-6.1-sol, the computer tool with computerUseTool({ type: 'computer' }). The model sends a batch of actions in each call, the app runs them, and the adapter sends the screenshot back. This PR builds on #1831 (fix/openai-computer-call-loop), which makes the computer use loop run.

🎯 Changes

The tool. computerUseTool() takes { type: 'computer' } next to { type: 'computer_use_preview', display_width, display_height, environment }. The tool name is the tool type, so the preview tool keeps its name.
Batched actions. A computer_call with actions becomes a tool call with { actions, pending_safety_checks }. The replay sends the actions back as they came. A call with one action (the preview model) works as before.
The call name. A computer_call is a call to the computer tool of the request: computer when the request has it, else computer_use_preview.
Tool choice. The Responses tool choice type accepts computer, as the OpenAI SDK does.
Screenshot detail. Each computer screenshot goes out with detail: 'original', as in OpenAI's computer use guide, so the model sees the screen at full size. The OpenAI SDK (also the newest, 7.39.0) and the API spec have no detail field yet. The adapter adds it to its own screenshot type.
Docs. The computerUseTool section of the OpenAI adapter page uses { type: 'computer' } with gpt-6.1-sol. It shows the preview tool for computer-use-preview, and its "Supported models" line names both.

✅ Checklist

I have followed the steps in the Contributing guide.
I have tested code changes locally with pnpm run test:pr, or these tests do not apply to this pull request.
I fully understand the code in this pull request, including any code generated with AI assistance.
Docs: I updated docs/ for this change, or this change is not user-facing.
Changeset: I added a changeset (pnpm changeset), or this PR does not change a published package.

🚀 Release Impact

This change affects published code, and I have generated a changeset.
This change is docs/CI/dev-only (no release).

Testing

Commands run:

pnpm --dir packages/openai-base run test:types, a build of @tanstack/openai-base, and pnpm --dir packages/ai-openai run test:types. Both pass.
vitest run --maxWorkers=2 in packages/openai-base on computer-use-loop, provider-tool-dispatch, responses-text, replay-parity, tool-choice-wire, and responses-empty-completed-output: 187 tests pass. In packages/ai-openai, tools-per-model-type-safety and openai-adapter: all pass.
test:oxlint of both packages (no errors), oxfmt --check on the changed files, kiira check on docs/adapters/openai.md (45 snippets pass), pnpm test:docs (no broken links), and pnpm test:knip (exit 0).
After the screenshot detail commit: computer-use-loop (3 tests pass), test:types of openai-base, oxlint on the changed files (no errors), and oxfmt --check on the changed files.
After the merge of fix(openai-base): run the computer use loop #1831 (a computer result is now the only result that reads the screenshot): computer-use-loop (4 tests), provider-tool-dispatch, and replay-parity pass (72 tests), and test:types of openai-base passes.
After the merge of the E2E commit of fix(openai-base): run the computer use loop #1831: the spec openai-computer-use-wire now also checks ?tool=computer (the batched actions on gpt-6.1-sol) and detail: 'original' on each screenshot. I type-checked the route, and ran oxfmt --check and oxlint on the changed files.

Not run locally: pnpm test:pr, E2E, and coverage. CI runs them. I type-checked the new E2E route. The spec runs in CI.

Manual test:

Run pnpm --dir packages/openai-base exec vitest run tests/computer-use-loop.test.ts.
Read the test "runs the batched actions of the computer tool". The first request sends { type: 'computer' }. The tool gets the two actions. The second request sends the computer_call with the same actions and the computer_call_output with the screenshot.

How this PR makes testing easy: tests/computer-use-loop.test.ts runs both tool types through chat() with a mock client. tests/provider-tool-dispatch.test.ts checks the wire shape of { type: 'computer' }, and tools-per-model-type-safety.test.ts checks that gpt-6.1-sol accepts it. testing/e2e/tests/openai-computer-use-wire.spec.ts runs both tool types through the real OpenAI SDK, against a custom fetch.

Risk / rollback

Low. The preview tool keeps its name, its wire shape, and its single action. The API spec does not list detail on computer_screenshot, but OpenAI's own samples send it. If the API rejects it, remove the detail line in screenshotOf. To undo all of it, revert this PR.

Public API change

Before

computerUseTool({
  type: 'computer_use_preview',
  display_width: 1024,
  display_height: 768,
  environment: 'browser',
})

After

computerUseTool({ type: 'computer' })

computerUseTool({ type: 'computer' }) sends the computer tool of the newer OpenAI models. Its tool name is computer. A computer_call is a call to the computer tool of the request, with the batched actions and the pending safety checks as input, and the replay sends the actions back as they came. The computer_use_preview tool works as before.

OpenAI's computer use guide sends each screenshot with detail: 'original', so the model sees the screen at full size. The SDK type has no detail field yet, so the adapter adds it to its own screenshot type.

changeset-bot · 2026-10-19T17:08:30Z

🦋 Changeset detected

Latest commit: f6ce006

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 10 packages

Name	Type
@tanstack/openai-base	Minor
@tanstack/ai-openai	Minor
@tanstack/ai-bedrock	Patch
@tanstack/ai-byteplus	Patch
@tanstack/ai-cloudflare	Patch
@tanstack/ai-grok	Patch
@tanstack/ai-groq	Patch
@tanstack/ai-llmgateway	Patch
@tanstack/ai-lovable	Patch
@tanstack/ai-vercel-gateway	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

nx-cloud · 2026-10-19T17:10:56Z

View your CI Pipeline Execution ↗ for commit ad8d630

Command	Status	Duration	Result
`nx run-many --targets=test:coverage --projects=...`	✅ Succeeded	1s	View ↗
`nx affected --targets=test:sherif,test:knip,tes...`	✅ Succeeded	59s	View ↗
`nx run-many --targets=build --exclude=examples/...`	✅ Succeeded	3s	View ↗

☁️ Nx Cloud last updated this comment at 2026-10-20 08:27:44 UTC

github-actions · 2026-10-19T17:13:43Z

Coverage

✅ Coverage held across 11 compared package(s).

Each package is measured twice in this job — on this PR and on its merge-base with main — and compared. A metric fails only when it is at or above 60% on the merge-base and under 60% on this PR. Other drops are listed but do not fail. Packages your PR doesn't affect are not measured and not listed.

package	statements	branches	functions	lines
ai-bedrock	90.11% (0.00)	82.82% (0.00)	94.20% (0.00)	91.33% (0.00)	ok
ai-byteplus	97.06% (0.00)	91.66% (0.00)	98.27% (0.00)	98.49% (0.00)	ok
ai-cloudflare	85.55% (0.00)	76.44% (0.00)	85.48% (0.00)	86.79% (0.00)	ok
ai-code-mode-snippets	87.13% (0.00)	75.67% (0.00)	91.11% (0.00)	88.39% (0.00)	ok
ai-grok	48.85% (0.00)	53.49% (0.00)	56.46% (0.00)	49.67% (0.00)	ok
ai-groq	52.54% (0.00)	47.73% (0.00)	67.39% (0.00)	54.66% (0.00)	ok
ai-llmgateway	97.67% (0.00)	100.00% (0.00)	88.88% (0.00)	97.67% (0.00)	ok
ai-lovable	73.21% (0.00)	55.63% (0.00)	85.07% (0.00)	73.96% (0.00)	ok
ai-openai	65.34% (0.00)	65.72% (0.00)	66.66% (0.00)	65.63% (0.00)	ok
ai-vercel-gateway	75.97% (0.00)	55.63% (0.00)	85.41% (0.00)	79.38% (0.00)	ok
openai-base	90.49% (+0.06)	78.78% (+0.23)	97.75% (+0.03)	92.61% (+0.04)	ok

pkg-pr-new · 2026-10-19T17:14:55Z

Open in StackBlitz

@tanstack/ai

npm i https://pkg.pr.new/@tanstack/ai@1833

@tanstack/ai-acp

npm i https://pkg.pr.new/@tanstack/ai-acp@1833

@tanstack/ai-angular

npm i https://pkg.pr.new/@tanstack/ai-angular@1833

@tanstack/ai-anthropic

npm i https://pkg.pr.new/@tanstack/ai-anthropic@1833

@tanstack/ai-bedrock

npm i https://pkg.pr.new/@tanstack/ai-bedrock@1833

@tanstack/ai-byteplus

npm i https://pkg.pr.new/@tanstack/ai-byteplus@1833

@tanstack/ai-claude-code

npm i https://pkg.pr.new/@tanstack/ai-claude-code@1833

@tanstack/ai-client

npm i https://pkg.pr.new/@tanstack/ai-client@1833

@tanstack/ai-cloudflare

npm i https://pkg.pr.new/@tanstack/ai-cloudflare@1833

@tanstack/ai-code-mode

npm i https://pkg.pr.new/@tanstack/ai-code-mode@1833

@tanstack/ai-code-mode-snippets

npm i https://pkg.pr.new/@tanstack/ai-code-mode-snippets@1833

@tanstack/ai-codex

npm i https://pkg.pr.new/@tanstack/ai-codex@1833

@tanstack/ai-cohere

npm i https://pkg.pr.new/@tanstack/ai-cohere@1833

@tanstack/ai-compaction

npm i https://pkg.pr.new/@tanstack/ai-compaction@1833

@tanstack/ai-devtools-core

npm i https://pkg.pr.new/@tanstack/ai-devtools-core@1833

@tanstack/ai-durable-stream

npm i https://pkg.pr.new/@tanstack/ai-durable-stream@1833

@tanstack/ai-elevenlabs

npm i https://pkg.pr.new/@tanstack/ai-elevenlabs@1833

@tanstack/ai-event-client

npm i https://pkg.pr.new/@tanstack/ai-event-client@1833

@tanstack/ai-fal

npm i https://pkg.pr.new/@tanstack/ai-fal@1833

@tanstack/ai-gemini

npm i https://pkg.pr.new/@tanstack/ai-gemini@1833

@tanstack/ai-grok

npm i https://pkg.pr.new/@tanstack/ai-grok@1833

@tanstack/ai-grok-build

npm i https://pkg.pr.new/@tanstack/ai-grok-build@1833

@tanstack/ai-groq

npm i https://pkg.pr.new/@tanstack/ai-groq@1833

@tanstack/ai-isolate-cloudflare

npm i https://pkg.pr.new/@tanstack/ai-isolate-cloudflare@1833

@tanstack/ai-isolate-daytona

npm i https://pkg.pr.new/@tanstack/ai-isolate-daytona@1833

@tanstack/ai-isolate-e2b

npm i https://pkg.pr.new/@tanstack/ai-isolate-e2b@1833

@tanstack/ai-isolate-node

npm i https://pkg.pr.new/@tanstack/ai-isolate-node@1833

@tanstack/ai-isolate-quickjs

npm i https://pkg.pr.new/@tanstack/ai-isolate-quickjs@1833

@tanstack/ai-isolate-quickjs-bun

npm i https://pkg.pr.new/@tanstack/ai-isolate-quickjs-bun@1833

@tanstack/ai-llmgateway

npm i https://pkg.pr.new/@tanstack/ai-llmgateway@1833

@tanstack/ai-lovable

npm i https://pkg.pr.new/@tanstack/ai-lovable@1833

@tanstack/ai-mcp

npm i https://pkg.pr.new/@tanstack/ai-mcp@1833

@tanstack/ai-memory

npm i https://pkg.pr.new/@tanstack/ai-memory@1833

@tanstack/ai-mistral

npm i https://pkg.pr.new/@tanstack/ai-mistral@1833

@tanstack/ai-octane

npm i https://pkg.pr.new/@tanstack/ai-octane@1833

@tanstack/ai-ollama

npm i https://pkg.pr.new/@tanstack/ai-ollama@1833

@tanstack/ai-ollaya

npm i https://pkg.pr.new/@tanstack/ai-ollaya@1833

@tanstack/ai-openai

npm i https://pkg.pr.new/@tanstack/ai-openai@1833

@tanstack/ai-opencode

npm i https://pkg.pr.new/@tanstack/ai-opencode@1833

@tanstack/ai-openrouter

npm i https://pkg.pr.new/@tanstack/ai-openrouter@1833

@tanstack/ai-perplexity

npm i https://pkg.pr.new/@tanstack/ai-perplexity@1833

@tanstack/ai-persistence

npm i https://pkg.pr.new/@tanstack/ai-persistence@1833

@tanstack/ai-preact

npm i https://pkg.pr.new/@tanstack/ai-preact@1833

@tanstack/ai-react

npm i https://pkg.pr.new/@tanstack/ai-react@1833

@tanstack/ai-react-ui

npm i https://pkg.pr.new/@tanstack/ai-react-ui@1833

@tanstack/ai-reactor

npm i https://pkg.pr.new/@tanstack/ai-reactor@1833

@tanstack/ai-remix

npm i https://pkg.pr.new/@tanstack/ai-remix@1833

@tanstack/ai-sandbox

npm i https://pkg.pr.new/@tanstack/ai-sandbox@1833

@tanstack/ai-sandbox-blaxel

npm i https://pkg.pr.new/@tanstack/ai-sandbox-blaxel@1833

@tanstack/ai-sandbox-boxd

npm i https://pkg.pr.new/@tanstack/ai-sandbox-boxd@1833

@tanstack/ai-sandbox-cloudflare

npm i https://pkg.pr.new/@tanstack/ai-sandbox-cloudflare@1833

@tanstack/ai-sandbox-daytona

npm i https://pkg.pr.new/@tanstack/ai-sandbox-daytona@1833

@tanstack/ai-sandbox-docker

npm i https://pkg.pr.new/@tanstack/ai-sandbox-docker@1833

@tanstack/ai-sandbox-e2b

npm i https://pkg.pr.new/@tanstack/ai-sandbox-e2b@1833

@tanstack/ai-sandbox-local-process

npm i https://pkg.pr.new/@tanstack/ai-sandbox-local-process@1833

@tanstack/ai-sandbox-railway

npm i https://pkg.pr.new/@tanstack/ai-sandbox-railway@1833

@tanstack/ai-sandbox-sprites

npm i https://pkg.pr.new/@tanstack/ai-sandbox-sprites@1833

@tanstack/ai-sandbox-upstash-box

npm i https://pkg.pr.new/@tanstack/ai-sandbox-upstash-box@1833

@tanstack/ai-sandbox-vercel

npm i https://pkg.pr.new/@tanstack/ai-sandbox-vercel@1833

@tanstack/ai-skills

npm i https://pkg.pr.new/@tanstack/ai-skills@1833

@tanstack/ai-solid

npm i https://pkg.pr.new/@tanstack/ai-solid@1833

@tanstack/ai-solid-ui

npm i https://pkg.pr.new/@tanstack/ai-solid-ui@1833

@tanstack/ai-svelte

npm i https://pkg.pr.new/@tanstack/ai-svelte@1833

@tanstack/ai-typesafe

npm i https://pkg.pr.new/@tanstack/ai-typesafe@1833

@tanstack/ai-utils

npm i https://pkg.pr.new/@tanstack/ai-utils@1833

@tanstack/ai-vercel-gateway

npm i https://pkg.pr.new/@tanstack/ai-vercel-gateway@1833

@tanstack/ai-vertex

npm i https://pkg.pr.new/@tanstack/ai-vertex@1833

@tanstack/ai-vue

npm i https://pkg.pr.new/@tanstack/ai-vue@1833

@tanstack/ai-vue-ui

npm i https://pkg.pr.new/@tanstack/ai-vue-ui@1833

@tanstack/ai-worldlabs

npm i https://pkg.pr.new/@tanstack/ai-worldlabs@1833

@tanstack/openai-base

npm i https://pkg.pr.new/@tanstack/openai-base@1833

@tanstack/preact-ai-devtools

npm i https://pkg.pr.new/@tanstack/preact-ai-devtools@1833

@tanstack/react-ai-devtools

npm i https://pkg.pr.new/@tanstack/react-ai-devtools@1833

@tanstack/solid-ai-devtools

npm i https://pkg.pr.new/@tanstack/solid-ai-devtools@1833

@tanstack/svelte-ai-devtools

npm i https://pkg.pr.new/@tanstack/svelte-ai-devtools@1833

commit: ad8d630

coderabbitai · 2026-10-19T17:23:41Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: TanStack/ai/.coderabbit.yaml
Review profile: CHILL
Plan: Advanced
Run ID: 0f5c11d9-d5c4-456c-b659-a5e75cb5674e

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

…r-tool # Conflicts: # packages/openai-base/src/adapters/responses-user-tools.ts # packages/openai-base/tests/computer-use-loop.test.ts

…r-tool

… the wire The route takes ?tool=computer for the computer tool of gpt-6.1-sol. The spec checks the batched actions, the computer tool entry, and detail: 'original' on each screenshot.

…r-tool

AlemTuzlak added 2 commits October 19, 2026 18:41

feat(openai-base): send computer screenshots with the original detail

becc71f

OpenAI's computer use guide sends each screenshot with detail: 'original', so the model sees the screen at full size. The SDK type has no detail field yet, so the adapter adds it to its own screenshot type.

Merge branch 'fix/openai-computer-call-loop' into feat/openai-compute…

f6ce006

…r-tool # Conflicts: # packages/openai-base/src/adapters/responses-user-tools.ts # packages/openai-base/tests/computer-use-loop.test.ts

github-actions Bot assigned jherr Oct 20, 2026

github-actions Bot requested a review from jherr October 20, 2026 00:23

github-actions Bot added the waiting-on: maintainer The ball is in the maintainers’ court label Oct 20, 2026

AlemTuzlak added 3 commits October 20, 2026 08:22

Merge branch 'fix/openai-computer-call-loop' into feat/openai-compute…

0bc3bf2

…r-tool

test(e2e): check the computer tool batch and the screenshot detail on…

7d482c8

… the wire The route takes ?tool=computer for the computer tool of gpt-6.1-sol. The spec checks the batched actions, the computer tool entry, and detail: 'original' on each screenshot.

Merge branch 'fix/openai-computer-call-loop' into feat/openai-compute…

ad8d630

…r-tool

Navigation Menu

Repository navigation

feat(openai-base, ai-openai): add the computer tool - #1833

feat(openai-base, ai-openai): add the computer tool#1833
AlemTuzlak wants to merge 6 commits into
fix/openai-computer-call-loopfrom
feat/openai-computer-tool

AlemTuzlak commented Oct 19, 2026 •

edited

Loading

Uh oh!

changeset-bot Bot commented Oct 19, 2026 •

edited

Loading

Uh oh!

nx-cloud Bot commented Oct 19, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 19, 2026 •

edited

Loading

Uh oh!

pkg-pr-new Bot commented Oct 19, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Oct 19, 2026 •

edited

Loading

Review skipped

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

AlemTuzlak commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🎯 Changes

✅ Checklist

🚀 Release Impact

Testing

Risk / rollback

Public API change

Uh oh!

changeset-bot Bot commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

nx-cloud Bot commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage

Uh oh!

pkg-pr-new Bot commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

AlemTuzlak commented Oct 19, 2026 •

edited

Loading

changeset-bot Bot commented Oct 19, 2026 •

edited

Loading

nx-cloud Bot commented Oct 19, 2026 •

edited

Loading

github-actions Bot commented Oct 19, 2026 •

edited

Loading

pkg-pr-new Bot commented Oct 19, 2026 •

edited

Loading

coderabbitai Bot commented Oct 19, 2026 •

edited

Loading