What happens to an agent's tool choice as you hand it more tools.
Seven frontier models, 665 trials, ten synthetic MCP servers built with deliberate semantic overlap — a notion_search that a model could plausibly confuse with drive_search, a linear_create_issue next to a github_create_issue. The question is not whether a model can call a tool. It is whether it still picks the right one when the menu gets long.
Single-shot, tool_choice: "auto", no retries, no per-model prompt engineering. Same system prompt and the same tool schemas across all three providers. Total cost $5.07. Zero API errors.
GPT-5 Mini beats GPT-5 at every tier. At the large tier it is 40/44 against 28/44 — thirteen items Mini got and the flagship didn't, against one the other way. Exact McNemar puts that at p = 0.0018, and it survives Holm correction across all 21 pairwise comparisons.
GPT-5 is also the only model in the set that gets worse as the tool surface grows: 76.5% at three servers, 63.6% at ten. The mechanism is visible in the raw results — it returns no tool call at all on create and update prompts rather than commit to one. It is not picking the wrong tool. It is declining to pick.
That is the result. It took me a while to see it, because I had written a leaderboard on top of it.
Sensible first-tool accuracy at the large tier (10 servers, 34 tools):
| Model | Score | Rate | Wilson 95% |
|---|---|---|---|
| Opus 4.7 | 43/44 | 97.7% | [88.2%, 99.6%] |
| Sonnet 4.6 | 43/44 | 97.7% | [88.2%, 99.6%] |
| Haiku 4.5 | 41/44 | 93.2% | [81.8%, 97.7%] |
| GPT-5 Mini | 40/44 | 90.9% | [78.8%, 96.4%] |
| Gemini 2.5 Pro | 33/44 | 75.0% | [60.6%, 85.4%] |
| GPT-5 | 28/44 | 63.6% | [48.9%, 76.2%] |
| Gemini 2.5 Flash | 26/44 | 59.1% | [44.4%, 72.3%] |
Those intervals are 11 to 28 percentage points wide. At n=44 the smallest gap this benchmark can detect at 80% power is 19 percentage points. Everything from Opus down to Gemini Pro sits inside the noise.
Worse, when I checked which items were doing the work:
- 14 of the 44 items carry no ranking information at all — 13 that every model passes, 1 that every model fails.
- Four items separate the top four models.
- Zero items separate first place from second.
- Opus and Sonnet have zero discordant pairs. They did not tie on the score. They returned the same answer on all 44 items.
I printed Opus above Sonnet. There is no observation in this benchmark that justifies that.
Full working in STATISTICS.md, regenerated from the committed data by error-bars:
python examples/from_mcp_tool_selection_eval.py \
results/cross-provider.json --tier large --metric sensible > trials.json
python -m errorbars trials.json -o STATISTICS.mdServers and tiers. Ten synthetic MCP servers — Notion, Gmail, Calendar, Drive, Slack, Linear, GitHub, Stripe, Figma, Todoist. Three tiers by server count: small (3 servers, 12 tools), medium (7 servers, 25 tools), large (10 servers, 34 tools).
Trial count. A case only runs at a tier where its target server is present, so the per-model case counts are 17 / 34 / 44 across the tiers — 95 per model, 665 across seven models. Not every case runs at every tier, which is why the arithmetic is 95×7 and not 44×3×7.
Categories. 24 ambiguous, 22 create, 18 search, 13 fetch, 11 list, 7 update. The ambiguous cases are the point: queries with more than one defensible answer, used to measure whether a model disambiguates or guesses.
Scoring. Strict means the first tool_use is in the accepted set. Sensible also credits a same-server search/list that precedes the correct create/update/fetch — reaching the right neighbourhood one call early. Overall: 454/665 strict (68.3%), 544/665 sensible (81.8%).
The sensible rule is reimplemented independently in the error-bars adapter and reproduces the harness's per-model counts exactly (43/43/41/40/33/28/26), so the statistics above are computed on the published headline metric rather than a proxy for it.
The fix for a benchmark this underpowered is not more items. It is harder items. Thirty-two percent of this one is dead weight and four items are carrying the entire top of the table. Adding another forty cases drawn from the same distribution would buy roughly nothing.
Separating 97.7% from 95.7% at 80% power needs about 1,200 items per model. Nobody hand-writes that. The honest move for the frontier tier is to stop ranking it and report the tie.
pnpm install
cp .env.example .env # ANTHROPIC_API_KEY, OPENAI_API_KEY, GEMINI_API_KEY
pnpm evalCheaper variants: pnpm eval -- --providers anthropic, pnpm eval -- --models gpt-5-mini, or pnpm eval:smoke (5 cases on Haiku, about a tenth of a cent).
results/cross-provider.json is the full run from April 2026, committed. A benchmark whose data is not in the repository is not a benchmark.
MIT.