Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 58 additions & 1 deletion evals/mcp-tool-use/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ Cases are JSON lines in `cases/`, one file per area. The case schema in `run.ts`
- `experience`: the real mistake, in a sentence. `source`: where it happened.
- `setup`: the fake thread, its `world` (repo, branch, files, PRs, pages), `priorTurns`, and `mocks` for tool results. With `world.backgroundTasks`, `t3_background_wait` answers from T3's own handler and ThreadManagementService, so its result and errors are the checkout's; a prior wait with `"result": "default"` gets that result too.
- `setup.priorTurns[].toolCalls[].resultFile`: load a repository-relative file as the tool result instead of copying its contents into the case. An empty `prompt` resumes a top-level thread after its last prior tool result without adding a user message. Side-chat and subagent setups require a nonempty prompt.
- `expect`: the checks, such as `calls`, `callsMustNot`, `maxCalls`, `order`, `shellMust`, `textMust`, `finalTextMust` (the last text, once the turn ended), and `linked`. A call's `args` match partially: `~text` is a case-insensitive substring, `"<=N"` / `">=N"` bound a number, `null` matches a missing key, and keys may be dotted paths. For `calls` and `callsMustNot`, `argsAnyOf` accepts any one of its partial argument objects, in addition to `args`. Only `calls` supports `success: true`, which requires a recorded successful result. `distinctCalls` supports only `tool` and `args`. Device cases also use `device` to check every child brief, forbid raw UI commands, bound failed device actions per user turn, and require artifact paths in completed final answers. Secrets cases use `secret` (see [Secrets cases](#secrets-cases)).
- `expect`: the checks, such as `calls`, `callsMustNot`, `maxCalls`, `order`, `shellMust`, `textMust`, `finalTextMust` (the last text, once the turn ended), and `linked`. A call's `args` match partially: `~text` is a case-insensitive substring, `re:` a case-sensitive regex, `"<=N"` / `">=N"` bound a number, `null` matches a missing key, and keys may be dotted paths. For `calls` and `callsMustNot`, `argsAnyOf` accepts any one of its partial argument objects, in addition to `args`. Only `calls` supports `success: true`, which requires a recorded successful result. `distinctCalls` supports only `tool` and `args`. Device cases also use `device` to check every child brief, forbid raw UI commands, bound failed device actions per user turn, and require artifact paths in completed final answers. Secrets cases use `secret` (see [Secrets cases](#secrets-cases)).

Check that the case builds with `--dry-run --case <id>`. Then run it on the text before your fix, where it must fail, and on the text after, where it must pass.

Expand Down Expand Up @@ -290,6 +290,63 @@ Passes per case, main's runtime text (3 samples) → this branch's "Link every o

After the change, the first reference to a file is an absolute link. The remaining failures come from two other kinds of span. Some repeat the file later as relative inline code (Sonnet's other-checkout failures). Others quote values rather than cite files, such as `main` set to `src/index.ts` or the import `./ws`. Chat still turns those spans into pills.

## PR tools vs raw CLIs

`cases/pr-tools-vs-cli.ndjson` asks whether T3's PR, CI and issue tools pay for the context they add. Each case is a forge task from a real failure in 2026-09/11 Claude and Codex sessions. `--arm raw` removes those tools and the `<pull_request_linking>` instructions, so the agent has only `gh`, `cafe` and the shell. `--arm tools` (the default) is T3 as shipped. Run both arms on the same model and compare:

```sh
pnpm mcp-tool-use --arm tools --cases mcp-tool-use/cases/pr-tools-vs-cli.ndjson --samples 3 --out runs/tools
pnpm mcp-tool-use --arm raw --cases mcp-tool-use/cases/pr-tools-vs-cli.ndjson --samples 3 --out runs/raw
pnpm mcp-tool-use:compare runs/tools/results.jsonl runs/raw/results.jsonl
```

`compare` regrades every row with this checkout's cases and prints, per case and arm: pass rate, model calls, forge calls (T3 forge tools plus `gh`/`cafe`/`glab`/`src` commands), failed forge calls, forge output in KB, requested sleep, `--help` calls, tokens and cost. Cost charges cache writes as cache reads, so the order the runs went in doesn't decide which arm looks cheaper.

Both arms see the same world. Shell mocks apply to either arm's commands, so an agent in the tools arm that reaches for `gh` hits the same rate limit. Where the real tools refuse, the tools arm is refused too: `world.unpushed` makes `pull_request_create` fail until a `git push`, and `pull_request_merge` compares `expectedHeadSha` with the full head, as `PullRequestService` does. A `{"$error": "..."}` mock fails any T3 tool with that text. `expectRaw` grades the raw arm when its right outcome differs: only T3 can watch a PR, so the raw arm passes `cli-babysit-green` by waiting on the checks.

Some cases are capability cases rather than mistake cases:

- `cli-rate-limited-read` and `cli-auth-missing` assume T3 is signed in through the T3 Code for GitHub app, whose quota and token are separate from `gh`'s.
- The git.cafe cases come in two forms. The plain ones say nothing about the forge. The `-instructed` ones put "use cafe" in the project instructions, as a maintained AGENTS.md would.

### Results

3 samples per case and arm, 2026-11-20, on T3 as of `6d6ca60` ("tools"). Codex models ran through CLIProxyAPI with `gpt-6.1-terra` as the shell. "Both can finish" leaves out the capability cases: rate limit, gh signed out, waiting on checks, and the uninstructed git.cafe cases. Means per sample; Sonnet cost is at warm-cache prices.

| Model | Arm | Pass | Model calls | Forge calls | Failed forge calls | Ktokens | Both can finish: pass, ktokens |
| ----------- | ----- | ----- | ----------- | ----------- | ------------------ | ------- | ------------------------------ |
| Sonnet 5.5 | raw | 34/42 | 3.5 | 2.0 | 0.4 | 81 | 27/27, 79 |
| Sonnet 5.5 | tools | 42/42 | 3.8 | 1.6 | 0.0 | 90 | 27/27, 89 |
| GPT-6 Luna | raw | 31/40 | 7.3 | 4.1 | 1.0 | 147 | 24/26, 109 |
| GPT-6 Luna | tools | 41/42 | 4.6 | 1.7 | 0.0 | 91 | 26/27, 97 |
| GPT-6.1 Sol | raw | 29/41 | 6.2 | 4.4 | 1.0 | 122 | 26/27, 92 |
| GPT-6.1 Sol | tools | 42/42 | 4.5 | 2.4 | 0.0 | 89 | 27/27, 99 |

Errored samples are left out.

**Where the tools win:**

- **They never failed a forge call.** The raw arm failed one call per sample on Codex and 0.4 on Sonnet.
- **Capability cases.**
- The raw arm cannot read checks while `gh` is rate-limited: 0–1 of 3 passed per model.
- It never found `cafe` on a git.cafe repo whose instructions don't name it: 0/3 on review comments for all three models.
- GPT-6.1 Sol's raw arm said "I'll check every minute" with nothing running, so it never reported green checks: 0/2.
- GPT-6 Luna's raw arm polled for 18 model calls on average.
- **On Codex, total tokens.** The tools cut them by 38% on Luna and 27% on Sol.

**Where it's even, or raw wins:**

- **Tasks both arms can finish.** Sonnet and Sol pass nearly every sample either way, at similar tokens; raw is about 10% lower.
- **One-line questions and plain merges.** The tools cost one extra model call to load the deferred schema.
- **Fixed overhead.** On Claude it is about 530 tokens per request: the linking block plus the deferred tool names.
- **git.cafe with "use cafe" in the project instructions.** Both arms pass at similar cost.

`cli-merge-at-head-moved`, a control where the head moved past the reviewed commit, was added after these runs. On GPT-6.1 Sol both arms passed it 3/3: neither merged.

**With the fixes.** "tools+fixes" (Sonnet, earlier git.cafe world) applied [#39899](https://github.com/pingdotgg/t3code/pull/39899), [#39901](https://github.com/pingdotgg/t3code/pull/39901), [#39903](https://github.com/pingdotgg/t3code/pull/39903) and [#39906](https://github.com/pingdotgg/t3code/pull/39906). It passed 42/42 with 2.5 model calls and 83 ktokens per sample, against 2.8 and 90 for "tools" on the same cases.

Limits: the shell is simulated, so a CLI's output is the simulator's reading of the world and of the error texts the cases copy from real sessions. Polling never really waits, so these runs measure what polling decides, not the wall-clock time it takes. The simulator also sees the whole world, and early runs leaked it into invented files, such as an AGENTS.md naming `cafe` or a Forgejo API on git.cafe. The git.cafe worlds now rule both out. Read failing and passing transcripts before you trust a new case.

## Where T3's text lives

- `packages/provider-core/src/server/runtimeInstructions.ts`
Expand Down
Loading
Loading