Run Reason with Harbor on the three benchmarks of the Artificial Analysis Coding Agent Index (v1.5).
| Benchmark | Harbor dataset | Tasks |
|---|---|---|
| DeepSWE v1.1 | datacurve/deep-swe-1-1 |
113 |
| SWE-Atlas QnA | scale-ai/swe-atlas-qna |
124 |
| Terminal-Bench 4.0 | terminal-bench/terminal-bench@4 |
66 |
Harbor starts each task's container and grades it with the task's own verifier.
Reason works on that container as a normal Session, in a new, empty workspace
for every attempt. We recommend running a pinned harness release, which never
changes, so every rerun runs the same harness. The adapter (0.3.43 or later)
requests runtime/reason-agent-1.3 by default, so the commands below run it without naming it;
to rerun an earlier release, name it, for example --ak runtime_profile=runtime/reason-agent-1.2.
Results here are on the Artificial Analysis Coding Agent Index v1.5.
Each benchmark scores pass@1, averaged over three attempts per task, and the
Index is the equal-weight mean of the three. Timeouts score 0, provider refusals are redrawn
up to 10 times, and a Terminal-Bench attempt the reward-hacking judge fails
scores 0. See the methodology for the full rules and
their version history.
You need uv, Docker
and a Reason API key with run,
sessions:read, deployment:read, devices:read, devices:write,
devices:use and org:write, from an owner or admin of a workspace with paid
model access. Enable the model under
Models.
uv tool install "harbor[e2b,modal]==0.22.0" --with "reason-machines-harbor>=0.3.44"
export REASON_API_KEY=<YOUR-REASON-KEY>
# One DeepSWE task on local Docker, to check your setup.
# For your own provider key, connect it under Models and use --model byok:openai:gpt-6-luna
harbor run --dataset datacurve/deep-swe-1-1 --n-tasks 1 \
--agent reasonmachines_harbor:ReasonAgent \
--model openai/gpt-6-luna \
--ak reasoning_effort=xhigh \
--allow-agent-host api.reasonmachines.com \
--allow-agent-host relay.reasonmachines.comThe scripts here pin Harbor 0.22.0, which our published results ran on. For
your own runs, the adapter (0.3.44 or later) also works with Harbor up to
0.25. Drop --n-tasks 1 to run every task once. The worker connects out to the
Reason API and relay, so both --allow-agent-host flags are needed even on
offline tasks. Each trial's folder under jobs/ holds the Session's
trajectory (agent/trajectory.json), its link and cost (agent/session.json)
and the verifier's result.
New here? The guide walks through a run step by step, what each file does, the decisions behind them, and how we checked the token and cost numbers.
run.sh runs one benchmark or all three with the pinned task sets,
three attempts per task, then screens and scores the run.
run-pass.sh runs a whole publishable pass and its
preflight (see the methodology).
./run.sh deepswe # e2b, with the tasks e2b can't host on Modal
HARBOR_ENV=docker ./run.sh deepswe # everything on local Docker, no cloud accounts
./run-pass.sh openai/gpt-6-luna xhigh # all three benchmarks, e2b + Modal| Variable | Needed for | |
|---|---|---|
REASON_API_KEY |
every run | See quick start |
ANTHROPIC_API_KEY |
SWE-Atlas QnA grading; SCREEN_JUDGE=1 |
The grader is Claude Opus 4.5; the agent never sees this key |
E2B_API_KEY |
HARBOR_ENV=e2b (the default) and run-pass.sh |
From e2b |
| Modal login | the Modal half and run-pass.sh |
uvx modal token new, or MODAL_TOKEN_ID and MODAL_TOKEN_SECRET; MODAL_PROFILE picks the workspace |
run.sh settings, all optional:
| Variable | Default | Meaning |
|---|---|---|
MODEL |
openai/gpt-6-luna |
Model id, or byok:<provider>:<model> |
EFFORT |
xhigh |
minimal, low, medium, high, xhigh or max |
HARBOR_ENV |
e2b |
e2b (with Modal for the tasks in <benchmark>/modal-tasks.txt), modal or docker |
PART |
both |
With e2b: e2b skips the Modal tasks, modal runs only them |
ATTEMPTS |
3 |
Attempts per task (run-pass.sh reads it too) |
HARNESS |
the adapter's default release | A runtime profile, for example runtime/reason-agent-1.2 (run-pass.sh reads it too) |
RUN |
<model without provider>-<effort>, for example gpt-6-luna-xhigh |
Folder under jobs/ |
E2B_CONCURRENCY, MODAL_CONCURRENCY, DOCKER_CONCURRENCY |
10, 8, 2 |
Trials at a time |
SCREEN_JUDGE |
0 |
1 also runs the paid reward-hacking judge |
Extra arguments after the benchmark go to harbor run, for example
./run.sh deepswe -i datacurve/igel-persist-feature-schema or -l 5.
connect-nodus.sh connects Nodus
inference to the workspace of REASON_API_KEY as its Custom provider (an
OpenAI-compatible endpoint). Nodus serves only the model calls; the sandboxes
stay on e2b, Modal or Docker. It needs a NODUS_API_KEY with the
inference:invoke scope and a REASON_API_KEY with org:write, and it
replaces the workspace's existing Custom provider.
./connect-nodus.sh # NODUS_MODELS picks the models, default GLM-5.3, Kimi K3, DeepSeek V4 Pro
MODEL=byok:custom-endpoint:zai-org/glm-5.3 EFFORT=high ./run.sh deepsweThe Custom provider offers low, medium and high effort. score.py has
no Nodus prices, so it reports no cost for these runs.
Harbor's orchestrator holds every trial, so a laptop that sleeps or a cloud
session that parks kills the whole run.
e2b-orchestrator.py runs run.sh inside its own e2b
sandbox (4 CPUs, 8 GB) instead. A loop in that sandbox pushes its timeout an
hour ahead every five minutes while run.sh lives, up to --max-hours, then
leaves two hours to fetch the results:
uv run e2b-orchestrator.py start --max-hours 12 -- deepswe # with run.sh's variables set
uv run e2b-orchestrator.py status <sandbox-id>
uv run e2b-orchestrator.py fetch <sandbox-id> --kill # jobs/ and logs/ into this checkoutOnly run.sh's variables (the table above) are passed in, to a mode-600 file
in the sandbox. The orchestrator sandbox counts against e2b's concurrent
sandbox limit along with the trials.
Results land in jobs/<run>/<benchmark>/<part>/<job>/. Both scripts score
the run when they finish; to rescore, or to score a run you completed with
reruns:
uv run screening/screen.py jobs/<run> # reward-hacking scan
uv run score.py jobs/<run> # pass@1, intervals, builds and cost
uv run score.py --strict jobs/<run> # also check the run is complete on one configurationscore.py prices each model response from its PRICES table. A model not in
the table falls back to the cost Reason reported; add its list prices there to
price it from the trajectory.
missing_scope,forbidden: the key lacks a scope listed above, or isn't an owner or admin key.ephemeral_workspace_parent_payment_required: the workspace has no paid model access. Add billing, or bring your own provider key withbyok:.Reason Session ended: status=errorwith no code: adapter 0.3.45 and later name the cause (for exampleprovider_auth_failed, a connected provider key that was refused); the trial'sagent/session.jsonhas it too.execution_ineligibleat worker startup: upgrade the adapter to 0.3.43 or later.- "ANTHROPIC_API_KEY can't use the SWE-Atlas grader": Anthropic refused the key, or it has no access to Claude Opus 4.5. Without this check, every SWE-Atlas attempt would time out in grading.
- "the worker needs Reason egress on this offline task": add both
--allow-agent-hostflags. - Problems with the adapter or these scripts: open an issue.
@misc{reasonmachines2026benchmark,
title = {Reason on the Coding Agent Index},
author = {{Reason Machines}},
year = {2026},
howpublished = {\url{https://github.com/reason-machines/benchmark}}
}Apache-2.0. The benchmarks themselves are their authors'; see each benchmark's page for its terms.