[Docs] Add the 2026-09-23 rerun of the Google Flights benchmark - #8
Merged
Yunaik merged 1 commit intoSep 25, 2026
Merged
Conversation
Collaborator
|
please don't push the results with 38k lines of code in here, but a compressed version or the code to reproduce the code. |
We ran every arm of the Google Flights comparison again on 2026-09-23. The machine was a Windows 11 PC in Poland. Each arm ran three times on the direct TypeSafe backend and three times through OpenRouter. All 27 runs passed their checks. The baseline is the runs of 2026-09-18 and 2026-09-19 already in the doc. docs/benchmarks.md gets a new section for the rerun. It has a table with every run, the per-step waits of the median runs, and a short list of what we found. The closing table now shows the baseline rows and the rerun rows together. The 24 S1A records and the chart are under docs/results/flights/rerun-2026-09-23/. The old glob patterns still match only the baseline records. README.md and CHANGELOG.md point at the rerun. The model answered as fast as in the baseline. Each decision took 354 to 426 ms direct and 391 to 466 ms through OpenRouter. The stock Playwright MCP arm went from a baseline median of 42.7 s to 26.2 s. All three of its runs passed. The extra time in the baseline was page waits and clicks. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
opPO333
force-pushed
the
feature/update-benchmarks-rerun
branch
from
September 24, 2026 16:46
057e9d5 to
c21bf68
Compare
Contributor
Author
|
Done. I removed the raw JSON files and replaced them with a single compressed archive .tar.gz. The commit has been amended. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
There was a concern that network latency to the TypeSafe and OpenRouter servers might delay the model's
responses. The first numbers were recorded in China, so we gathered a new set of data from a different
location to verify this.
How
We ran every arm again on 2026-09-23 from a Windows 11 PC. Google places this PC in Poland. Each arm ran
three times on the direct TypeSafe backend and three times through OpenRouter. The baseline is the runs
of 2026-09-18 and 2026-09-19 already in the doc. The arms were jev-ultrafast (A), the S1A policy on the
browser-use driver at tag
jj-bu-baseline(B), and this repo on the stock Playwright MCP (C), theno-settle copy (C') and the no-settle copy with batched actions (C''). All runs used the same Chrome
window, remote debugging on port 9222, and one scratch profile. Every arm searched for the same date,
October 25, 2026.
Before we edited the docs, a script checked all 27 result files. It checked the end status, the verified
flag, the final URL with the date in it, the number of decisions, and the median latency. We then ran
scripts/summarize_runs.pyandscripts/compare_runs.pyon the committed records to make the tables inthe doc.
What we found:
and 391 to 466 ms through OpenRouter. In the baseline it took 360 to 418 ms and 438 to 510 ms. The
"waiting on the model" column is the same in both sets of runs. OpenRouter adds about 1 s per run over
twelve decisions, as before. This PC is in Poland, so the test covers the route from Poland only.
42.7 s and only two of three runs passed. Now it takes 26.2 s direct and 28.9 s through OpenRouter. All
three runs pass. Page waits dropped from 21.0 s to 11.3 s. Clicks dropped from 14.8 s to 9.1 s. The
MCP server code is the same. Google's pages answered faster from here.
7.3 s on 2026-09-18.
instead of 7 to 11.
ec947e6). That step was added after the baseline runs.
Things a reviewer should know. The doc states all of them.
"Reject all" once before the runs. The timed window starts after navigation, so this does not touch
any number.
background, so its first three runs got stuck after one click. We changed one line so the tab opens in
the foreground. Its decision loop, clock and model calls are unchanged. Every S1A arm already runs in
the visible tab.
the old processes. Those runs do not count. The doc lists every run we dropped.
Read
docs/benchmarks.mdfirst. The new parts are the sections "Rerun of 2026-09-23" and "Every arm inone table".
What
docs/benchmarks.md: a new section with the 27-row run table, the per-step waits of the median directruns, and a short list of findings. The closing table now holds the baseline rows, the old
jev-ultrafast row, and the nine rerun rows. It also has the command that rebuilds the table from the
tree and the new chart. The heading "Six arms, one table" is now "Every arm in one table".
docs/results/flights/rerun-2026-09-23/: the 24 S1A records pluscompare.svgandcompare-dark.svg.They sit in a subfolder so the old glob patterns still match only the baseline records. The
jev-ultrafast records stay out of the tree, as before.
README.md: the two links to the Flights comparison now mention the rerun. The README table comparesagents against the chat model. The rerun does not change those numbers.
CHANGELOG.md: one entry under Unreleased.No code changed.
Verification
uv run ruff format --check . && uv run ruff check . && uv run ty checkpass.uv run pytest -q: 479 passed, 41 skipped, 3 failed. The three failures only happen on Windows andhappen on
maintoo. They are the exit code test for an unusableS1A_HOMEand two sockettimeouts in the
BrowserHandstests.scripts/smoke.shfails its first check on Windows becauses1a listprints CRLF line endings. This branch changes only docs and data files.CHANGELOG.mdand the docs say what the code does now.committed records with the repo's own scripts.