Skip to content

[Docs] Add the 2026-09-23 rerun of the Google Flights benchmark - #8

Merged
Yunaik merged 1 commit into
ThinkFlowLab:mainfrom
opPO333:feature/update-benchmarks-rerun
Sep 25, 2026
Merged

Yunaik merged 1 commit into
ThinkFlowLab:mainfrom
opPO333:feature/update-benchmarks-rerun

Conversation

@opPO333

@opPO333 opPO333 commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

Why

There was a concern that network latency to the TypeSafe and OpenRouter servers might delay the model's
responses. The first numbers were recorded in China, so we gathered a new set of data from a different
location to verify this.

How

We ran every arm again on 2026-09-23 from a Windows 11 PC. Google places this PC in Poland. Each arm ran
three times on the direct TypeSafe backend and three times through OpenRouter. The baseline is the runs
of 2026-09-18 and 2026-09-19 already in the doc. The arms were jev-ultrafast (A), the S1A policy on the
browser-use driver at tag jj-bu-baseline (B), and this repo on the stock Playwright MCP (C), the
no-settle copy (C') and the no-settle copy with batched actions (C''). All runs used the same Chrome
window, remote debugging on port 9222, and one scratch profile. Every arm searched for the same date,
October 25, 2026.

Before we edited the docs, a script checked all 27 result files. It checked the end status, the verified
flag, the final URL with the date in it, the number of decisions, and the median latency. We then ran
scripts/summarize_runs.py and scripts/compare_runs.py on the committed records to make the tables in
the doc.

What we found:

  • The route to the model servers did not slow the model down. Each decision took 354 to 426 ms direct
    and 391 to 466 ms through OpenRouter. In the baseline it took 360 to 418 ms and 438 to 510 ms. The
    "waiting on the model" column is the same in both sets of runs. OpenRouter adds about 1 s per run over
    twelve decisions, as before. This PC is in Poland, so the test covers the route from Poland only.
  • Page loading got much faster for the stock Playwright MCP arm. In the baseline the median run took
    42.7 s and only two of three runs passed. Now it takes 26.2 s direct and 28.9 s through OpenRouter. All
    three runs pass. Page waits dropped from 21.0 s to 11.3 s. Clicks dropped from 14.8 s to 9.1 s. The
    MCP server code is the same. Google's pages answered faster from here.
  • Arm B took 8.0 s direct and 8.5 s through OpenRouter. In the baseline it took 11.8 s on 2026-09-19 and
    7.3 s on 2026-09-18.
  • jev-ultrafast took 10.3 s, down from 14.1 s in the baseline. It threw away 5 to 8 decisions per run
    instead of 7 to 11.
  • C'' is 1.3 s slower than its baseline. 0.7 s of that comes from the whole-page answer step (commit
    ec947e6). That step was added after the baseline runs.

Things a reviewer should know. The doc states all of them.

  • Google showed a cookie consent page to this PC, as it does to any new browser in the EU. We clicked
    "Reject all" once before the runs. The timed window starts after navigation, so this does not touch
    any number.
  • Google's menus do not open in a background tab on this Chrome build. jev-ultrafast opens its tab in the
    background, so its first three runs got stuck after one click. We changed one line so the tab opens in
    the foreground. Its decision loop, clock and model calls are unchanged. Every S1A arm already runs in
    the visible tab.
  • Arm B's sidecar processes stay alive after a run ends. Two OpenRouter runs timed out until we killed
    the old processes. Those runs do not count. The doc lists every run we dropped.

Read docs/benchmarks.md first. The new parts are the sections "Rerun of 2026-09-23" and "Every arm in
one table".

What

  • docs/benchmarks.md: a new section with the 27-row run table, the per-step waits of the median direct
    runs, and a short list of findings. The closing table now holds the baseline rows, the old
    jev-ultrafast row, and the nine rerun rows. It also has the command that rebuilds the table from the
    tree and the new chart. The heading "Six arms, one table" is now "Every arm in one table".
  • docs/results/flights/rerun-2026-09-23/: the 24 S1A records plus compare.svg and compare-dark.svg.
    They sit in a subfolder so the old glob patterns still match only the baseline records. The
    jev-ultrafast records stay out of the tree, as before.
  • README.md: the two links to the Flights comparison now mention the rerun. The README table compares
    agents against the chat model. The rerun does not change those numbers.
  • CHANGELOG.md: one entry under Unreleased.

No code changed.

Verification

  • uv run ruff format --check . && uv run ruff check . && uv run ty check pass.
  • uv run pytest -q: 479 passed, 41 skipped, 3 failed. The three failures only happen on Windows and
    happen on main too. They are the exit code test for an unusable S1A_HOME and two socket
    timeouts in the BrowserHands tests. scripts/smoke.sh fails its first check on Windows because
    s1a list prints CRLF line endings. This branch changes only docs and data files.
  • CHANGELOG.md and the docs say what the code does now.
  • A script checked all 27 records before the docs were edited. The tables were rebuilt from the
    committed records with the repo's own scripts.

@Yunaik

Yunaik commented Sep 24, 2026

Copy link
Copy Markdown
Collaborator

please don't push the results with 38k lines of code in here, but a compressed version or the code to reproduce the code.

We ran every arm of the Google Flights comparison again on 2026-09-23.
The machine was a Windows 11 PC in Poland. Each arm ran three times on
the direct TypeSafe backend and three times through OpenRouter. All 27
runs passed their checks. The baseline is the runs of 2026-09-18 and
2026-09-19 already in the doc.

docs/benchmarks.md gets a new section for the rerun. It has a table
with every run, the per-step waits of the median runs, and a short list
of what we found. The closing table now shows the baseline rows and the
rerun rows together. The 24 S1A records and the chart are under
docs/results/flights/rerun-2026-09-23/. The old glob patterns still
match only the baseline records. README.md and CHANGELOG.md point at
the rerun.

The model answered as fast as in the baseline. Each decision took 354
to 426 ms direct and 391 to 466 ms through OpenRouter. The stock
Playwright MCP arm went from a baseline median of 42.7 s to 26.2 s. All
three of its runs passed. The extra time in the baseline was page waits
and clicks.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@opPO333
opPO333 force-pushed the feature/update-benchmarks-rerun branch from 057e9d5 to c21bf68 Compare September 24, 2026 16:46
@opPO333

opPO333 commented Sep 24, 2026

Copy link
Copy Markdown
Contributor Author

Done. I removed the raw JSON files and replaced them with a single compressed archive .tar.gz. The commit has been amended.

@Yunaik Yunaik left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks

@Yunaik
Yunaik merged commit a3367a2 into ThinkFlowLab:main Sep 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants