Multi-agent teams for real work: measured, not assumed.
Run a team of coding agents on a task in your repository: a single agent, a coder ⇄ reviewer
loop, or several agents with a judge. Workers are the coding CLIs you already use (Claude Code,
Codex) or a built-in agent that can use any model through the OpenAI Agents SDK. Each worker
gets its own git worktree. You get one patch back, plus an itemized record of every agent call:
tokens, cost and seconds. Your working tree is untouched unless you pass --apply.
The benchmark then answers the question most agent orchestrators skip: did the team beat a single agent on the same task, and what did the extra agents cost?
The evidence on teams of coding agents points both ways:
- A 260-configuration study found multi-agent setups 2–15% worse than single agents on SWE-bench Verified (Towards a Science of Scaling Agent Systems).
- Teams with a central coordinator and a reviewer reported 4–8 points better (icat-agent, Agyn).
- The best-known coding benchmark is contaminated: OpenAI stopped reporting SWE-bench Verified in February 2026.
So the answer depends on the topology, the models and the task, and published numbers may be contaminated. This project measures it on tasks created after the models' training cutoffs, with paired comparisons and cost.
Requirements: Python 3.10+, uv, git, and at least one logged-in
coding CLI (claude or codex). CLI harnesses use your existing login; no API keys needed.
uvx --from git+https://github.com/BichengWang/multi-agents multi-agents run \
--repo ~/code/myproject \
--team 'best:workers=claude@haiku+codex@gpt-6-luna' \
"Make the tests in test_slug.py pass without editing the tests."Real output from that command on a small repository:
Team: best:workers=claude@haiku+codex@gpt-6-luna,judge=claude@haiku
role ┃ harness@model ┃ tokens in/cached/out ┃ cost ┃ secs
━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━
worker │ claude@claude-haiku-5-5 │ 236k/201k/1.5k │ $0.0096 │ 34.8
judge │ claude@claude-haiku-5-5 │ 194k/157k/2.6k │ $0.0102 │ 42.8
worker │ codex@gpt-6-luna │ 107k/94k/667 │ $0.0025 │ 31.6
judge │ claude@claude-haiku-5-5 │ 192k/149k/3.8k │ $0.0119 │ 52.3
Total: 729k/604k/8.5k tokens, cost $0.0342, 85.0s
scores: [8.0, 8.0]
chosen: 1
Files changed: 1
slug.py +6 -3
The run prints the exact git apply command for the final patch and saves a JSON record
outside your repository (~/.cache/multi-agents/runs/).
| Spec | What happens |
|---|---|
solo:worker=H |
One agent works on the task. This is the baseline. |
review:coder=H,reviewer=H[,rounds=N] |
The coder works; the reviewer scores the patch in a separate copy and sends feedback; the coder resumes its session and revises. Returns the best-scoring patch, not the last one. |
best:workers=H+H[+H...][,judge=H] |
Workers run in parallel, each in its own worktree, possibly different CLIs and models. The judge scores each patch and the best one wins. |
plan:planner=H,workers=H[+H...][,integrator=H][,max_tasks=N] |
A planner splits the task into independent subtasks, workers solve them in parallel worktrees, and the patches are merged; an integrator resolves conflicts and gets one fix pass if the check fails. |
H is claude, codex or sdk, optionally with @model, for example claude@haiku,
codex@gpt-6-luna or sdk@litellm/openrouter/<model>. check (the --check command) can be the
reviewer or the judge: a test-driven loop with no LLM reviewer. Details:
team runner guide.
flowchart LR
T["Task"] --> Team["Team: solo / review / best / plan"]
Team --> H1["Claude Code"]
Team --> H2["Codex"]
Team --> H3["SDK agent (any model)"]
H1 --> W1["Worktree 1"]
H2 --> W2["Worktree 2"]
H3 --> W3["Worktree 3"]
Team --> R["Patch + record: runs, tokens, cost, time"]
--check runs your tests (or build, or lint) against the final patch in a separate worktree and
records pass/fail; with an LLM judge, candidates that fail it rank below those that pass. --issue
takes the task from a GitHub issue, and --pr commits the result to a new branch and opens a draft
pull request with the run summary, without touching your checkout.
multi-agents run --repo . --issue 42 --pr \
--team 'plan:planner=claude@haiku,workers=claude@haiku+codex@gpt-6-luna' \
--check 'python3 -m unittest -q'A real plan run on a repository with two unrelated bugs (without --issue/--pr):
role ┃ harness@model ┃ tokens in/cached/out ┃ cost ┃ secs
planner │ claude@claude-haiku-5-5 │ 187k/150k/2.0k │ $0.0101 │ 16.8
worker │ claude@claude-haiku-5-5 │ 141k/104k/510 │ $0.0088 │ 8.7
worker │ codex@gpt-6-luna │ 55k/37k/349 │ $0.0024 │ 9.7
Total: 385k/291k/2.9k tokens, cost $0.0212, 26.8s
subtask 1 [applied]: Fix word_count in textstats.py (claude@haiku)
subtask 2 [applied]: Fix format_cents in money.py (codex@gpt-6-luna)
check: passed (exit 0, 0.0s) python3 -m unittest -q
Issue text is untrusted input to the agents: don't combine --issue on public repositories with
--permissions bypass or the sdk harness.
This repository is also a plugin marketplace for both coding CLIs, so a running agent can hand a task to a team and then show you the patch. Install it once:
# Claude Code
claude plugin marketplace add BichengWang/multi-agents
claude plugin install multi-agents@multi-agents
# Codex
codex plugin marketplace add BichengWang/multi-agents
codex plugin add multi-agents@multi-agentsThen, inside Claude Code (the optional spec comes first; the default is a cheap coder ⇄ reviewer pair that pairs the other CLI with the one you are in):
/multi-agents:team make the tests pass
/multi-agents:team best:workers=claude@haiku+codex@gpt-6-luna fix the flaky retry in client.py
In Codex, ask for the skill by name, for example Use $multi-agents:multi-agents to make the tests pass. Either way the agent runs the published CLI, shows you the summary and the patch, and applies
it only after you say so. The skill is in plugins/multi-agents.
- Permissions. By default (
--permissions safe) each CLI runs inside its own sandbox: edits and test commands are confined to the worktree.--permissions bypassturns those checks off and is meant for disposable machines and containers only. Thesdkharness has no permission system. - Cost. Costs come from the CLI's own report or from list prices with a cited source. A run that cannot be priced shows as unpriced and makes the total unpriced; nothing is guessed.
The benchmark runs team specs against solo agents on GitHub issues opened after the models' training cutoffs (SWE-bench-Live MultiLang: Rust, Java, Go, C/C++, C#, JS/TS), each in a fresh Modal sandbox with git history stripped. Patches are graded with the dataset's official hidden tests. The report gives resolve rate with 95% confidence intervals, paired wins and losses against the solo baseline on the same tasks (exact McNemar test), cost per task and per resolved task, and wall time. Timeouts are graded rather than dropped, so slow teams cannot shrink the comparison to the easy tasks.
uv sync --locked --extra bench
python -m agent.bench tasks --since 2026-07-01 --limit 5 --seed 7 # no credentials neededSetup, methodology, cost estimates and limitations: benchmark guide. First results will be published here.
| Area | State |
|---|---|
Team runner (solo, review, best, plan), Claude Code and Codex harnesses |
Working; smoke-tested with real CLIs |
--check, --issue, --pr, plugins for both CLIs |
Working; smoke-tested end to end |
| SDK harness (any model) | Implemented and tested offline; not yet run against a live model |
| Benchmark | Built and tested offline; the Modal path has not run live yet |
| Results | Pending the first pilot run |
Plans are in docs/ROADMAP.md.
uv sync --locked --extra dev
make test # ruff + offline tests, no API keys needed
make check # also the GitHub workflow and attribution tests (Node.js 22+)Reusable orchestration patterns (RefineLoop, BestOfN, SequentialWorkflow, Router) are in
agent/workflows. Pre-pivot experiments are in this repository's history
at the pre-pivot tag.