Every number here compares a System 1 decision model against a chat model in the same agent, on the same tools and the same seeds. The model is the only difference between the two columns.
One episode per agent on one seed, both sides on the wall clock. The replays are under assets/demos/ and in the
README.
| scenario | Jev | chat model | speedup | Jev cost | chat model cost | more expensive | score, Jev / chat model |
|---|---|---|---|---|---|---|---|
| Ticket router, 30 tickets | 12.7 s | 65.3 s | 5.14× | $0.000764 | $0.014800 | 19.4× | 30 / 30 |
| Desktop, Windows Calculator | 17.2 s | 30.1 s | 1.75× | $0.000447 | $0.003002 | 6.7× | 1 / 1 |
| ALFWorld | 7.4 s | 25.7 s | 3.47× | $0.000221 | $0.002800* | 12.7× | 1 / 1 |
| 2048, 20 moves | 27.1 s | 68.5 s | 2.53× | $0.000447 | $0.006139 | 13.7× | 64 / 80 |
| Millionaire | 15.3 s | 21.6 s | 1.41× | $0.000168 | $0.000892 | 5.3× | $1,000 / $0 |
| Blackjack | 2.3 s | 14.7 s | 6.39× | $0.000021 | $0.000531 | 25.3× | 1 / 1 |
* Estimated; the chat-model run recorded no cost. A score of 1 is a won hand, a solved household task or a window that shows the expected result. The browser row is in the next section.
The allrecipes agent runs the first Allrecipes task of the WebVoyager
task set verbatim (Allrecipes--0 in data/WebVoyager_data.jsonl; He et al., 2024;
Apache License 2.0, attribution in NOTICE): a vegetarian lasagna with more than 100 reviews, a rating of at least
4.5 stars, suitable for 6 people, answered in text. Both models ran headed on
2026-09-22 (the site answers a headless Chromium with a bot wall) on the Playwright MCP without its settle sleeps,
with a frame after every browser call from the run's own MCP session: scripts/browser_showcase.sh allrecipes 2.
The chat model on both sides is Claude Fable 5.1 through OpenRouter, at $10 per million input tokens, $50 per
million output tokens and $0.25 per million cached input tokens; Jev's decisions go to TypeSafe at $0.042 per
million input tokens. The Jev run pays the chat model for one typed value, the search text, and for the answer.
| run | model | wall s | decisions | chat calls | chat tokens in / cached / out | Jev tokens | cost | answer |
|---|---|---|---|---|---|---|---|---|
| 0 | jev | 35.7 | 5 | 4 | 7,946 / 1,210 / 454 | 23,654 | $0.0914 | Easy Vegetarian Spinach Lasagna: 4.6 stars, 117 ratings, serves 6 |
| 1 | jev | 40.4 | 4 | 4 | 7,335 / 0 / 440 | 18,785 | $0.0961 | the same recipe |
| 2 | jev | 25.4 | 4 | 4 | 7,335 / 7,295 / 365 | 18,785 | $0.0213 | the same recipe |
| 0 | llm | 116.9 | 11 | 12 | 238,211 / 123,989 / 4,811 | 0 | $1.4138 | the same recipe; the harness's own verdict on the run was blocked |
| 1 | llm | 143.8 | 9 | 10 | 200,547 / 91,155 / 5,355 | 0 | $1.3845 | the same recipe |
| 2 | llm | 138.7 | 10 | 12 | 230,599 / 115,042 / 6,200 | 0 | $1.4943 | the same recipe |
Every answer meets the task's three conditions, checked by hand on the recipe page: 4.6 stars, 117 ratings (85 of
them written reviews; the WebVoyager judge of the September 19 batch accepted the ratings count as reviews),
6 servings. The README row and the replay are Jev's run 0 against the chat model's run 2, the median run of each
model by wall clock; the chat model's cost is 16.4 times Jev's. The records of those two runs are under results/allrecipes/.
Two more pairs ran the same afternoon without records in that folder. The first, before the probe dropped escaped markup from control labels: Jev answered in 31.9 s of process time for $0.102 with the same recipe, after two clicks on a card whose label was an image tag; the chat model missed the serving count: 120.8 s and $1.212 for Vegetarian Four Cheese Lasagna, 4.6 stars, 243 ratings, 8 servings. The second, whose records a later run of the script overwrote: Jev 40.3 s, 5 decisions and $0.096; the chat model 104.9 s, 8 decisions and $1.096; both the spinach lasagna.
wall s is the task's own clock, written to answer.json: the browser's start, the navigation, every decision and
the final answer. Of the Jev run's cost, the decisions themselves are a tenth of a cent; the rest is the chat
model's typed value and answer. Jev's run 2 cost a fifth of run 1 because it read 7,295 of its 7,335 prompt tokens
from the cache. The frames add one screenshot per browser call on both sides. Each run begins with a
chat-model call by the harness that probes the model's image support. OpenRouter refuses that call for this model and
the harness continues without it. Both sides pay it.
Over 150 moves of 2048 the cost gap widens with the transcript. Seed 0: Jev scored 1,104 in 244 s for $0.003; the chat model scored 1,188 in 363 s for $0.31.
The same showcase on ALFWorld, Blackjack and Millionaire is under results/<eval>/showcase/replay.gif, written by
scripts/showcase.sh.
The rail scores 20 of 20 on its labelled set at a median of 464 ms (s1a run injection_guard).
The protocol in evals/README.md quotes nothing under ten episodes and asks for 500 Blackjack hands. The series of
2026-09-19, from job folders on the author's machine; the chat model of the llm rows is not recorded in them:
| eval | model | N | score, 95 % CI | s per episode | steps | $ per episode |
|---|---|---|---|---|---|---|
| Blackjack | jev | 100 | -0.06 [-0.25, 0.13] | 0.6 | 1.5 | 0.0000 |
| Blackjack | llm | 100 | -0.06 [-0.25, 0.13] | 3.0 | 1.5 | 0.0006 |
| Blackjack | basic strategy | 100 | -0.06 [-0.25, 0.13] | 0.0 | 1.5 | 0 |
| ALFWorld text | jev | 12 | 0.75 [0.50, 1.00] | 3.4 | 20.3 | 0.0015 |
| ALFWorld text | llm | 12 | 0.917 [0.75, 1.00] | 9.2 | 12.6 | 0.0247 |
| ALFWorld text | oracle plan | 12 | 0.917 [0.75, 1.00] | 0.4 | 19.3 | 0 |
| 2048 | jev | 5 | 1115 [924, 1238] | 168.9 | 146.6 | 0.0032 |
On Blackjack the three models play the identical basic strategy over 100 hands. Jev takes a fifth of the chat
model's time per hand. On ALFWorld the chat model wins two more games of twelve. It spends 16 times the dollars.
uv run python -m evals.table evals/results prints this table for any run. laya and cua have no
numbers yet.
Goal for every arm: open Google Flights, find one-way flights from Zurich to London on September 20, 2026, one adult, economy, stop when matching flight options are visible. The clock runs from the first decision request to the final DONE. Every run uses the same Chrome (--remote-debugging-port=9222, scratch profile), the same decisions backend (typesafe/jev-1.13 through OpenRouter's /api/alpha/decisions), and the same chat model for typed values (google/gemini-2.5-flash). A run counts as verified when the results page shows flight options for that route and date. Measured 2026-09-19 on one MacBook Air, the twelve runs back to back.
- A, jev-ultrafast: browser-use's own agent and loop, Chrome through browser-harness. Reference.
- B, S1A on the browser-use driver: this repo's policy inside openJiuwen's browser subagent, on the agtai fork at tag
jj-bu-baseline(agtai/agent-core PR #6), where the driver is a browser-use sidecar over CDP. - C, S1A on the stock Playwright runtime: the same policy through the decision-policy slot (ThinkFlowLab/agent-core#1), openJiuwen's Playwright runtime, official
@playwright/mcp@0.0.78. - C', S1A on Playwright without the settle sleeps: arm C with one change in the MCP server:
waitForCompletionno longer sleeps 500 ms before and after waiting for in-flight requests.scripts/pw_mcp_nosettle.shprepares that copy; it is a patched npm package. - C'', S1A on Playwright, no settle sleeps, batched actions: arm C' with
--batch on: each step is onebrowser_run_code_unsafecall that performs the action and returns the next probe, one transport round trip per step where C' uses two. The runtime's target validation is skipped on that path; the batched code re-stamps and re-finds the target by role and label when a re-render dropped the stamp.
Per run, S1A arms record the profiler JSON (s1a run flights --model jev --batch on --profile-out run.json); arm A records jev-ultrafast's state.json. scripts/summarize_runs.py prints the table from those files. The S1A records live under docs/results/flights/. The arm A records stay out of the tree because each holds a page screenshot. The shipped flights agent computes its date as the first Sunday at least 28 days after the run day; the runs below used September 20, 2026. The records' visible_flights and screenshots show prices in yen because the runs were made from Japan; Google picks the currency from the run's location.
| run | window s | decisions | median ms | decision s | probe s | tool s | value wait s | verified |
|---|---|---|---|---|---|---|---|---|
| A-1 | 13.8 | 21 (11 not executed) | 462 | 10.4 | n/a | n/a | n/a | yes |
| A-2 | 13.1 | 19 (9 not executed) | 496 | 10.1 | n/a | n/a | n/a | yes |
| A-3 | 12.6 | 18 (8 not executed) | 470 | 9.3 | n/a | n/a | n/a | yes |
| B-1 | 10.1 | 12 | 455 | 5.9 | 3.7 | 0.5 | 0.0 | yes |
| B-2 | 11.2 | 12 | 510 | 6.3 | 4.3 | 0.5 | 0.0 | yes |
| B-3 | 10.1 | 12 | 438 | 5.4 | 4.1 | 0.5 | 0.0 | yes |
| C-1 | 30.0 | 11 | 460 | 5.5 | 13.5 | 9.8 | 0.0 | yes |
| C-2 | 50.9 | 4 | 730 | 3.0 | 30.0 | 15.1 | 0.0 | no |
| C-3 | 42.7 | 13 | 493 | 6.7 | 19.3 | 14.8 | 0.0 | yes |
| C'-1 | 10.6 | 12 | 461 | 5.4 | 3.1 | 1.9 | 0.0 | yes |
| C'-2 | 11.1 | 12 | 471 | 6.1 | 3.0 | 1.8 | 0.0 | yes |
| C'-3 | 11.5 | 13 | 469 | 6.1 | 3.2 | 2.0 | 0.0 | yes |
Medians of the window: A 13.1 s, B 10.1 s, C 36.4 s over the two verified runs, C' 11.1 s.
decisions counts requests to the decisions endpoint. For arm A, "not executed" are decisions jev-ultrafast discarded because the page had changed before the answer arrived. jev, probe and tool are critical-path seconds spent waiting on the decisions endpoint, on the page probe, and on the browser action; value wait is time blocked on the chat model for a typed value (zero with prefetch on).
- Arms B and C' run the same policy and the same 12 decisions; the difference between them is the driver. C' is 1.0 s behind B on the median, and its per-step cost is probe 0.25 s plus action 0.15 s against B's probe 0.33 s plus action 0.04 s.
- Arm C is the same code as C' on the stock server. Its two verified runs took 30.0 and 42.7 s and one run blocked. Probes and clicks make up the difference: 13.5 to 19.3 s of probes and 9.8 to 14.8 s of clicks against 3.1 s and 1.9 s in C'.
- Jev itself costs the same everywhere: 11 to 13 decisions at 440 to 510 ms through OpenRouter, 5.4 to 6.7 s of each window. Direct TypeSafe access measured 363 to 399 ms per decision on the agtai/agent-core fork (its F_04 note); the same arithmetic puts B and C' near 9 s there.
- Arm A makes 18 to 21 decisions per run and discards 8 to 11 of them; S1A makes 12 and discards none: it probes after the page settles and asks once per step.
Same task and Chrome, later the same day, decisions from api.typesafe.ai (jev-latest) instead of OpenRouter. Three runs per arm, back to back.
| run | window s | decisions | median ms | decision s | probe s | tool s | value wait s | verified |
|---|---|---|---|---|---|---|---|---|
| A-direct-1 | 14.1 | 21 (11 not executed) | 402 | 9.1 | n/a | n/a | n/a | yes |
| A-direct-2 | 12.1 | 18 (7 not executed) | 393 | 7.7 | n/a | n/a | n/a | yes |
| A-direct-3 | 14.2 | 18 (7 not executed) | 406 | 8.7 | n/a | n/a | n/a | yes |
| B-direct-1 | 11.9 | 12 | 405 | 5.6 | 5.5 | 0.6 | 0.0 | yes |
| B-direct-2 | 9.9 | 12 | 415 | 5.6 | 3.8 | 0.4 | 0.0 | yes |
| B-direct-3 | 11.8 | 12 | 418 | 5.2 | 5.7 | 0.8 | 0.0 | yes |
| C'-direct-1 | 34.3 | 11 | 414 | 7.0 | 15.1 | 11.8 | 0.0 | no |
| C'-direct-2 | 13.6 | 12 | 374 | 5.4 | 5.8 | 2.1 | 0.0 | yes |
| C'-direct-3 | 14.9 | 11 | 393 | 4.7 | 6.9 | 3.0 | 0.0 | yes |
| C''-direct-1 | 9.5 | 12 | 360 | 4.5 | 0.6 | 4.3 | 0.0 | yes |
| C''-direct-2 | 10.4 | 13 | 386 | 5.3 | 0.6 | 4.4 | 0.0 | yes |
| C''-direct-3 | 10.5 | 12 | 392 | 4.8 | 0.6 | 5.0 | 0.0 | yes |
Medians of the window: A 14.1 s, B 11.8 s, C' 14.2 s over its two verified runs, C'' 10.4 s.
For C'' the tool column holds the batched calls, each of which contains the action, the policy's in-page settle and the probe; probe is only the first probe of the run.
- The direct endpoint answers in 360 to 418 ms against 440 to 510 ms through OpenRouter, worth about 1 s per run. Every arm is slower on this afternoon's runs than on the morning's; the same code at tag
jj-bu-baselinemeasured 7.3 s (the fork's F_04 note) on the direct backend the day before and 11.8 s today, with 3.8 to 5.7 s of probe time per run. The cause is outside the agents: the page and the network of the day. Inference, not measured. - C'' is 1.4 s faster than B on the same clock and 3.7 s faster than jev-ultrafast. Its steps cost one round trip each: 12 batched calls in 4.3 to 5.0 s, about 0.36 s per step including the in-page settle, plus 12 decisions in 4.5 to 5.3 s.
- The first batched attempt, without stamp recovery, blocked two runs of three: a re-render between the probe and the click dropped the stamped selector, and the 5 s click timeout followed by the WAIT budget ended the run. With the re-stamp, the by-label recovery and a 2 s action timeout, three of three passed and no batched call reported an error.
- C' with the direct backend had one unverified run whose tab was reported hidden five times; the Chrome window was occluded during that run. C'' runs bring Chrome to the front before each run.
What is left in a C'' window is Jev (about 4.8 s for 12 decisions) and the policy's own settle windows inside each batched call (DOM-quiet up to 500 ms, autocomplete up to 900 ms). The remaining levers are the settle windows and the step count. Transport is down to one round trip per step, about 0.36 s including the in-page settle.
Arm B, tag jj-bu-baseline, direct backend, prefetch on: 7.2 / 8.5 / 7.3 s on 2026-09-18 (the fork's F_04 numbers, records B-direct-daybefore-*.json) and 11.9 / 9.9 / 11.8 s on 2026-09-19 afternoon. Same 12 decisions, same 12 steps, same Chrome setup. The profiler splits each run into what the policy waited for.
| run | window s | jev median ms | jev s | probe s | probe median ms | probe max ms | tool s |
|---|---|---|---|---|---|---|---|
| day before, direct, run 1 | 7.2 | 363 | 4.5 | 2.3 | 171 | 498 | 0.3 |
| day before, direct, run 2 | 8.5 | 384 | 5.8 | 2.3 | 170 | 488 | 0.4 |
| day before, direct, run 3 | 7.3 | 363 | 4.5 | 2.6 | 202 | 478 | 0.2 |
| morning, OpenRouter, run 1 | 10.1 | 455 | 5.9 | 3.7 | 211 | 725 | 0.5 |
| morning, OpenRouter, run 2 | 11.2 | 510 | 6.3 | 4.3 | 229 | 1028 | 0.5 |
| morning, OpenRouter, run 3 | 10.1 | 438 | 5.4 | 4.1 | 231 | 1055 | 0.5 |
| afternoon, direct, run 1 | 11.9 | 405 | 5.6 | 5.5 | 356 | 1025 | 0.6 |
| afternoon, direct, run 2 | 9.9 | 415 | 5.6 | 3.8 | 219 | 1134 | 0.4 |
| afternoon, direct, run 3 | 11.8 | 418 | 5.2 | 5.7 | 357 | 1231 | 0.8 |
jev is time waiting on the decisions endpoint. probe is time the policy spent waiting for the page after each action: the probe script waits for readyState, a 60 ms DOM-quiet window (cap 500 ms) and rendered autocomplete options (cap 900 ms), and the policy re-probes up to 1,000 ms more when an action's effect has not shown. tool is the driver executing the click or fill.
Per step, probe wait in ms (day before / morning / afternoon, one run each):
| step | action | day before | morning | afternoon |
|---|---|---|---|---|
| 1 | Change ticket type | 115 | 109 | 181 |
| 2 | One way | 101 | 116 | 551 |
| 3 | Where from? | 157 | 189 | 262 |
| 4 | Zürich, Switzerland | 243 | 238 | 367 |
| 5 | Where to? | 150 | 158 | 281 |
| 6 | London, United Kingdom | 261 | 233 | 296 |
| 7 | Departure | 148 | 173 | 345 |
| 8 | Sunday, September 20 | 187 | 725 | 531 |
| 9 | Done (date picker) | 185 | 714 | 694 |
| 10 | Search | 498 | 612 | 1025 |
| 11 | WAIT | 242 | 188 | 321 |
| 12 | DONE | 157 | 328 | 831 |
The three runs of each day agree with the one shown (scripts/summarize_runs.py --steps prints all of them).
Components of the 4 s gap between the day before and the afternoon, median runs:
- Decisions endpoint, about +0.7 s. 363 ms per decision the day before, 405 to 418 ms the afternoon (and 438 to 510 ms through OpenRouter in the morning). Over twelve decisions that is 0.5 to 1.0 s. Outside the agent.
- Google's price fetches, about +1.5 to 2.5 s. Steps 8, 9, 10 and 12 are the steps after which Google loads prices: the calendar grid after a date is picked, the results after Search. Their probe waits went from 185 / 185 / 498 / 157 ms to 700 / 700 / 1,000 / 800 ms. The morning runs, made before any other load on the machine, already showed those waits. Outside the agent; the policy's settle caps bound how long it waits for them.
- Load on the machine, about +0.5 to 1.0 s. Steps 1 to 7, which fetch nothing, took 100 to 260 ms in the morning and 100 to 550 ms in the afternoon with run-to-run scatter. A game (Slay the Spire 2, started 12:50) and Chrome at 130 % CPU were running during every afternoon run and none of the morning ones. Controllable by not running other things.
- Driver, about +0.3 s. Clicks and fills went from 0.2 to 0.4 s per run to 0.4 to 0.8 s, in step with the machine load.
The agent's own levers are the settle caps (500 ms DOM quiet, 900 ms autocomplete, 1,000 ms post-action) and the step count; lowering the caps trades reliability on the price-loading steps. The endpoint's latency and Google's fetches, about 3 s of the 4 s gap, are outside it.
Playwright MCP wraps browser_click, browser_evaluate and browser_run_code in waitForCompletion: run the action, sleep 500 ms, wait for every request started in that window (up to 5 s), then sleep 500 ms again if any request was seen. browser_press_key, browser_type and browser_snapshot skip it. Each probe and each click pays the full 1.0 s: Google Flights always has a request in flight. Measured on this page through the runtime: run_code and browser_evaluate 1,000 ms flat, press_key and snapshot about 100 ms. No CLI option changes it.
The stock run also blocked once (C-2): three clicks on the ticket-type control timed out after 5 s each waiting for the stamped element, with probes of 5 to 13 s in between. The likely cause, inferred from the timings and not measured: the extra second between probe and click gives Google time to re-render the control and drop the stamp. C' and B, with a shorter probe-to-click gap, did not hit it in three runs each.
npx -y @playwright/mcp@0.0.78 --version # populates the npx cache
CACHE="$(dirname "$(find ~/.npm/_npx -path '*/node_modules/@playwright/mcp/cli.js' | head -1)")/../../.."
cp -R "$CACHE" /tmp/pw-mcp-nosettle
# In /tmp/pw-mcp-nosettle/node_modules/playwright-core/lib/coreBundle.js, delete the two
# `await tab2.waitForTimeout(500);` lines inside `async function waitForCompletion`.
PLAYWRIGHT_MCP_COMMAND=node PLAYWRIGHT_MCP_ARGS=/tmp/pw-mcp-nosettle/node_modules/@playwright/mcp/cli.js \
s1a run flights --model jev --batch off --profile-out run.json # TYPESAFE_API_KEY unset: the OpenRouter proxyEvery run writes a record: the profiler JSON for S1A arms (--profile-out), state.json for jev-ultrafast.
scripts/compare_runs.py takes any set of them as label=glob, represents each arm by its median run, and
prints one table with each component next to its delta against the first arm; --svg also writes the chart
(a -dark sibling for dark surfaces). The S1A records are in docs/results/flights/. The chart below also took the
arm A records as a sixth label=glob; "day before" is 2026-09-18,
"today" is 2026-09-19. These runs predate the whole-page answer step (commit ec947e6). That step adds one probe and
one chat call to every DONE, about 0.5 to 1 s per run.
uv run python scripts/compare_runs.py --svg docs/results/flights/compare.svg \
"S1A browser-use driver, day before=docs/results/flights/B-direct-daybefore-*.json" \
"S1A browser-use driver, today=docs/results/flights/B-direct-[123].json" \
"S1A Playwright stock MCP, today (OpenRouter)=docs/results/flights/C-openrouter-*.json" \
"S1A Playwright no settle, today=docs/results/flights/Cprime-direct-*.json" \
"S1A Playwright no settle, batched, today=docs/results/flights/Cbatch-direct-*.json"
| arm | runs | total s | waiting on the model s | waiting on the page s | browser actions s | other s | decisions |
|---|---|---|---|---|---|---|---|
| S1A browser-use driver, day before | 3 of 3 verified, 7.2 to 8.5 s | 7.3 | 4.5 | 2.6 | 0.2 | 0.0 | 12 × 363 ms |
| S1A browser-use driver, today | 3 of 3 verified, 9.9 to 11.9 s | 11.8 (+4.5) | 5.2 (+0.7) | 5.7 (+3.0) | 0.8 (+0.7) | 0.2 (+0.1) | 12 × 418 ms |
| S1A Playwright stock MCP, today (OpenRouter) | 2 of 3 verified, 30.0 to 50.9 s | 42.7 (+35.3) | 6.7 (+2.2) | 21.0 (+18.4) | 14.8 (+14.7) | 0.1 (+0.0) | 13 × 493 ms |
| S1A Playwright no settle, today | 2 of 3 verified, 13.6 to 34.3 s | 14.9 (+7.6) | 4.7 (+0.2) | 7.2 (+4.5) | 3.0 (+2.8) | 0.1 (+0.0) | 11 × 393 ms |
| S1A Playwright no settle, batched, today | 3 of 3 verified, 9.5 to 10.5 s | 10.4 (+3.1) | 5.3 (+0.8) | 0.6 (-2.0) | 4.4 (+4.2) | 0.2 (+0.1) | 13 × 386 ms |
| jev-ultrafast, today | 3 of 3 verified, 12.1 to 14.2 s | 14.1 (+6.8) | 9.1 (+4.7) | n/a | n/a | 5.0 (+5.0) | 21 × 402 ms |
Each arm is its median run; its components add up to its total. Deltas in parentheses are against the first arm. total: seconds from the first decision request to the final answer. waiting on the model: time blocked on the decision request. waiting on the page: time the policy waited for the page after an action (DOM quiet, autocomplete, tab activation, typed values). browser actions: clicks and fills executing in the browser. other: everything else. decisions: requests to the endpoint in the median run × median latency. In the batched arm, page waits sit in browser actions because each action call also runs the next probe. jev-ultrafast's page and browser time sits in other because it records only its decisions.