feat(eval): /eco token-compression benchmark — measured, README showcase#712
Merged
Conversation
Replicate the experiment behind RTK's README savings table (https://github.com/rtk-ai/rtk estimates -80% over a modeled 30-minute session) with real measurements, and showcase the results in the READMEs. eval/eco/ benchmark: - capture_corpus.py (stdlib-only) captures 27 real operations: failing + green pytest/go/jest runs from sample projects with genuine bugs (RTK's "never synthetic" fixture rule), npm/pip installs, git clone/status/ commit/push --progress, repo-scale log/diff/ls/find/cat/grep, a ~35k-line macOS system log, and small must-pass-through rows. Marker-file guard refuses to rmtree a --workdir it didn't create; missing toolchains skip; partial failure exits non-zero. - measure.py replays each capture through the production path (baseline = _assemble_bash_body(truncate_output(...)); compress_bash_output; the mapper's returnCodeInterpretation suffix on both sides) and counts tiktoken cl100k_base tokens of the exact wire content. Tee-filename entropy (time_ns/pid/counter) is pinned and the tee dir fixed, so reruns on an unchanged corpus are byte-identical. Never-worse asserted corpus-wide. Percentages floor so display never overstates savings. - Committed results: corpus total 92,989 -> 17,767 tokens (-80%), filter hits -88%, and an honestly conservative recompute of RTK's own session model at -19% (eco compresses results only; small outputs pass through). READMEs: new /eco section (before/after pytest example, measured table, RTK-model comparison in <details>, safety guarantees, reproduce commands) in README.md and README_ZH.md, compact translated sections in AR/FR/HI/PT/ RU, /eco rows in the REPL command tables, eval/README.md pointer. src/eco: npm >=9 lowercased its log prefixes — noise pattern now matches "npm warn" alongside WARN/notice (npm error still always survives); found via the real npm-install capture. New regression test pins both casings. Critic-reviewed: two REVISE rounds (determinism + rmtree guard majors, wire-exactness and display-floor nits) then APPROVE with independent re-execution of the capture and byte-identical measure verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ericleepi314
force-pushed
the
worktree-RTK-token-compression
branch
from
1926da8 to
8f0dba1
Compare
ericleepi314
added a commit
that referenced
this pull request
Jul 14, 2026
Replace the DeepSeek Prefix Cache banner with the measured /eco story (same session, 80% fewer Bash-output tokens; 92,989 -> 17,767 across 27 real operations), with an explicit #eco-benchmark anchor down to the full benchmark section. The DeepSeek economics remain in the 2026-06-18 news item, which the banner and the /eco section now point at. Add a 2026-07-13 news item (#708 + #712) to README.md, docs/NEWS.md, and README_ZH.md; both README news lists stay at the 10 most recent (the GLM-5.2 item rolls off — already archived in docs/NEWS.md). Wording is engine-exact per critic review: failure summaries keep the error lines that matter (kept lines never rewritten), every LOSSY compression tees the full output to disk (safe-loss ceremony strips do not claim recoverability), and the RTK-model recompute names its -19%. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ericleepi314
added a commit
that referenced
this pull request
Jul 14, 2026
Replace the DeepSeek Prefix Cache banner with the measured /eco story (same session, 80% fewer Bash-output tokens; 92,989 -> 17,767 across 27 real operations), with an explicit #eco-benchmark anchor down to the full benchmark section. The DeepSeek economics remain in the 2026-06-18 news item, which the banner and the /eco section now point at. Add a 2026-07-13 news item (#708 + #712) to README.md, docs/NEWS.md, and README_ZH.md; both README news lists stay at the 10 most recent (the GLM-5.2 item rolls off — already archived in docs/NEWS.md). Wording is engine-exact per critic review: failure summaries keep the error lines that matter (kept lines never rewritten), every LOSSY compression tees the full output to disk (safe-loss ceremony strips do not claim recoverability), and the RTK-model recompute names its -19%. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ericleepi314
added a commit
that referenced
this pull request
Jul 14, 2026
…713) Replace the DeepSeek Prefix Cache banner with the measured /eco story (same session, 80% fewer Bash-output tokens; 92,989 -> 17,767 across 27 real operations), with an explicit #eco-benchmark anchor down to the full benchmark section. The DeepSeek economics remain in the 2026-06-18 news item, which the banner and the /eco section now point at. Add a 2026-07-13 news item (#708 + #712) to README.md, docs/NEWS.md, and README_ZH.md; both README news lists stay at the 10 most recent (the GLM-5.2 item rolls off — already archived in docs/NEWS.md). Wording is engine-exact per critic review: failure summaries keep the error lines that matter (kept lines never rewritten), every LOSSY compression tees the full output to disk (safe-loss ceremony strips do not claim recoverability), and the RTK-model recompute names its -19%. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Replicates the experiment behind RTK's README savings table — RTK estimates -80% over a modeled 30-minute session — with real measurements of our
/ecocompression (#708), and showcases the results in the READMEs.Benchmark (
eval/eco/)capture_corpus.py(stdlib-only): captures 27 real operations — failing/greenpytest/go test/jestruns from sample projects with genuine bugs (RTK's "never synthetic" fixture rule),npm/pipinstalls, git clone/status/commit/push--progress, repo-scalelog/diff/ls/find/cat/grep, a ~35k-line macOS system log, and small must-pass-through rows. Marker-file guard refuses tormtreea--workdirit didn't create.measure.py: replays each capture through the production path (_assemble_bash_body(truncate_output(...))baseline →compress_bash_output→returnCodeInterpretationsuffix on both sides) and counts tiktokencl100k_basetokens of the exact wire content. Tee-filename entropy is pinned → reruns on an unchanged corpus are byte-identical. Never-worse asserted corpus-wide; percentage display floors (never overstates).Results (committed, reproducible)
/ecoThe -19% row is the honest recompute of RTK's averaged session model:
/ecocompresses results only (never rewrites commands) and passes small/mid outputs through untouched — real sessions are fat-tailed, which is what the corpus measures. Both numbers are published side by side.READMEs
Full
/ecosection (before/after pytest example, measured table, RTK comparison in<details>, safety guarantees, reproduce commands) inREADME.md+README_ZH.md; compact translated sections in AR/FR/HI/PT/RU;/ecorows in the REPL command tables.Production fix found by the benchmark
npm ≥9 lowercased its log prefixes: the noise pattern now matches
npm warnalongsideWARN/notice(npm erroralways survives). Regression test pins both casings. 54/54 eco tests pass.Review
Critic loop: two REVISE rounds (determinism + rmtree-guard majors, wire-exactness nits), then APPROVE with independent re-execution of the capture and byte-identical measure verification.
🤖 Generated with Claude Code