PaperclipBench

How fast can an LLM agent beat Universal Paperclips?

What it measures

A milestone-ladder composite score (0–1000 base points) plus a clip bonus:

Milestone	Points
first_clip	5
auto_1	15
projects_open	30
stage2_enter	100
stage2_complete	150
probe_launched	250
space_25	100
space_50	75
space_75	75
endgame	200

• Clip bonus: min(200, floor(40 · log10(1 + totalClips))).
• Tiebreakers: equal primary scores rank by fewer tokens used, then fewer game ticks.
• Status outcomes: completed_endgame, completed_max, stuck, budget_exhausted, crashed.

Benchmark conditions

The fairness contract every run is held to:

• Vanilla game mirror, self-hosted. A stock copy of Universal Paperclips is fetched once via pnpm setup:game from an immutable upstream commit and served locally by the harness; the installed revision is recorded in packages/env/game/REVISION.
• Tick-rate patch honoring tickFactor. The game loop is patched so simulated ticks advance at a controlled rate: multi-fire timers get compensation and single-fire paths use one setTimeout, preserving the configured timer acceleration. Game timers pause before model observations and remain paused through model calls, retries, and action execution; provider latency contributes no simulated time. Host scheduling can still affect timer cadence, so this is not a claim of bit-for-bit deterministic replay.
• Identical action vocabulary & observation harnesses for all models. Every model sees the same snapshots (json or narrated) and can take only the same whitelisted actions.
• Decision cadence + wakeups. Decisions happen on a fixed tick cadence (default every 300 ticks), plus additive wakeup triggers (new_project_visible, stage_change, wire_exhausted).
• Per-run caps. Default limits: 400 decisions, 400 LLM calls, 2M total tokens, $10 estimated spend.
• Scripted-bot gate as environment proof. The scripted integration gate must reach stage 2 before model runs are trusted. It fails if the game mirror is missing. This verifies the stage-1 path, not full-game completion; later stages need separate validation before making full-game claims.

Setup

Requires Node >= 22. pnpm is enabled through Node's bundled corepack (no global install needed):

corepack enable pnpm        # or: npm i -g pnpm, or use bun as the package manager
pnpm i
pnpm setup:game
pnpm -C packages/env exec playwright install chromium
test -e .env || cp .env.example .env  # preserve existing keys; edit privately

Usage

Free pipeline check (no API key needed):

pnpm bench --agent=scripted --model=scripted --harness=narrated --max=150

Stage-2 integration gate (historically ~9 min; test timeout 2 hours; requires the fetched game):

pnpm vitest run packages/runner/tests/scripted-bot.test.ts

Paid smoke test (one short real-model run):

pnpm smoke openai/gpt-4o-mini narrated
# For other models, use an exact slug from packages/agents/src/registry.ts

Sweeps:

pnpm bench --model=<slug> --harness=json --seeds=1,2,3
pnpm bench --model=opencode/kimi-k2.6 --provider=opencode --harness=json

Providers

Two OpenAI-compatible gateways are supported; pick with --provider:

Provider	Key env var	Key source	Notes
openrouter	OPENROUTER_API_KEY	openrouter.ai/settings/keys	400+ models; free :free slugs rate-limited
opencode	OPENCODE_API_KEY	opencode.ai/auth	Zen curated gateway; per-request credits; some models free

Registry entries carry their provider (provider: "openrouter" | "opencode"); an explicit --provider overrides. Zen cost figures read $0 because Zen bills per request via credits.

Choosing a harness

--harness selects how the same game state is shown to the model — it is a first-class sweep dimension, so every {model × harness × seed} combination is its own tracked run:

• narrated — prose rendering resembling the game UI + JSON reply protocol (default; friendliest to weaker models)
• json — compact raw JSON observation, strict single-object reply
• tools — native OpenAI-style tool calling

Comparing a model across harnesses measures interface sensitivity; comparing models within one harness is the headline ranking.

Adding models

Edit MODEL_REGISTRY in packages/agents/src/registry.ts — add slug, provider, context window, and per-million-token prices. Prices are estimates used only for budget caps; re-verify them at openrouter.ai/models (Zen models bill per request via credits).

Data flow

Each run writes an event log (<run_id>.jsonl), periodic checkpoint.json, and a final summary.json under results/runs/<run_id>/. Completed runs are cached. Interrupted, crashed, or invalid run directories are never automatically resumed: checkpoints omit full random, timer, and conversation state. Preserve the artifacts and review prior spending before an explicitly planned retry in a fresh results root. Run IDs include the new timing policy, so old results are not reused.

Summaries record the effective environment, caps, decision budget, game revision, harness, agent type, and timing policy. Aggregation accepts one matching condition cohort and refuses to mix incompatible runs; legacy summaries without provenance are excluded and counted.

For a dedicated results root containing one cohort:

pnpm --filter @pb/score aggregate --results=/absolute/path/to/results --configVersion=v1

Use an absolute results path because the package command runs from packages/score. If several cohorts exist, add --conditionId=<sha256> to select one. Inspect results.json for excluded counts and the expected run count before using the ranking. The smoke script also aggregates its selected cohort.

Architecture map

Package	Role
packages/env	Playwright-driven self-hosted game mirror, action vocabulary, snapshot extraction, tick patch
packages/harnesses	Observation/decision formats: json, narrated, tools
packages/agents	OpenRouter client, model registry, token/USD caps
packages/runner	Run orchestration, event logs, diagnostic checkpoints, CLI, scripted bot
packages/score	Milestone ladder, primary score, summarizer, results aggregation

Results viewer

pnpm web        # serves apps/web + results/ at http://127.0.0.1:4173

results/results.json is the single data artifact (leaderboard rows sorted by score, then tokens, then ticks). The page is dependency-free static HTML - deployable to any static host alongside a committed results.json.

Cheap / free models

Free (:free) OpenRouter slugs are registered with $0 pricing (openrouter/free, z-ai/glm-5.2:free, nvidia/nemotron-3-super-120b-a12b:free). They are rate-limited (20 req/min; 50 req/day, or 1,000/day after a one-time $10 credit purchase) - use small budgets:

pnpm bench --model=openrouter/free --harness=narrated --max=40

Any other OpenAI-compatible gateway works via the client's baseUrl option.

Attribution & license

Universal Paperclips © Frank Lantz / Everybody House Games — http://www.decisionproblem.com/paperclips/. This repo automates a privately-fetched copy of the game for research purposes only; game code/assets are never committed or redistributed. All code in this repository is MIT licensed.

Benchmark execution handoff

See the execution plan for setup, gates, isolated canaries, approved run matrices, stop conditions, and result checks. The plan is instructions for a later operator; writing it does not execute any benchmark.

Known limitations

• Text-only observation harnesses in v1; no vision.
• With an honest-payout economy, full-game completion is rare without strong play — most runs will land mid-ladder under the default caps.
• Registry prices are estimates; treat USD figures as approximate.