Skip to content

docs(benchmark): add rtk (Rust Token Killer) as a 4th SWE-bench arm#20

Merged
OsherElhadad merged 1 commit into
mainfrom
docs/benchmark-rtk-arm
Jul 27, 2026
Merged

docs(benchmark): add rtk (Rust Token Killer) as a 4th SWE-bench arm#20
OsherElhadad merged 1 commit into
mainfrom
docs/benchmark-rtk-arm

Conversation

@OsherElhadad

Copy link
Copy Markdown
Collaborator

Summary

Adds rtk (rtk-ai/rtk, "Rust Token Killer") as a 4th arm to the live SWE-bench Verified study, alongside baseline, context-guru, and headroom.

rtk is architecturally distinct from the two request-stream proxies: it is a Claude Code PreToolUse hook that rewrites Bash commands (catrtk read, pytestrtk pytest, greprtk grep, …) and compresses their output at the shell, before it enters the transcript. It is not on the network path — so it adds 0 ms request latency, runs no model ($0 tool cost), and is cache-safe by construction (the compressed form is what gets cached; there is no cached prefix to invalidate).

Result (matched 50 tasks, all four arms, 0 exceptions)

dimension baseline context-guru headroom rtk
reward (solved / 50) 43 44 40 43
billed cost $31.98 $27.77 (−13.2%) $30.30 (−5.3%) $29.09 (−9.0%)
cache-read 102.8M 84.5M 96.4M 91.7M
mean steps 36.1 31.1 35.1 33.2
added latency / req 117 ms 63 ms 0 ms
tool LLM cost $0.31 $0 $0

rtk is the efficiency surprise — a simple deterministic shell filter is the 2nd-cheapest arm (−9.0%), reward-neutral vs baseline (43 = 43), at zero request-path latency and $0 tool cost, beating the headroom proxy on both cost and reward. Its 65.8% bash-output reduction (dominated by catread and grep) nets to −9% on the bill because bash output is a small slice of a ~98%-cached agent's context; its ceiling is that only Bash-tool output passes the hook (Claude Code's built-in Read/Grep/Glob bypass it), which is why context-guru's whole-request compaction goes deeper.

Changes

  • New docs/results/rtk.md — per-task table, rtk's own savings ledger, per-command breakdown, and real before→after examples.
  • Four-way updates to comparison.md, components.md, RESULTS.md, README.md, the home page, charts.js, and the mkdocs nav.
  • REPRODUCE.md documents the rtk arm end-to-end.
  • Harness: deploy/harbor/swebench_rtk.py + a committed reference copy of the custom claude-code-rtk Harbor agent; deep_analysis.py / deep_plots.py extended to a 4th arm (figures regenerated).
  • mkdocs build --clean is warning-free.

Notes

  • Model routing for the rtk arm is identical to the baseline (same off passthrough proxy) — the only variable is the in-container bash compression.
  • The content-removed metric is not comparable across arm kinds: the proxies measure against the whole request; rtk's 65.8% is of bash output only.

Assisted-By: Claude (Opus 4.8)

Adds rtk to the live SWE-bench Verified study alongside baseline,
context-guru, and headroom. rtk is architecturally distinct from the two
request-stream proxies: it is a Claude Code PreToolUse hook that rewrites
Bash commands (cat->rtk read, pytest->rtk pytest, grep->rtk grep, ...) and
compresses their output at the shell, before it enters the transcript — so
it adds 0 ms request latency, runs no model ($0 tool cost), and is
cache-safe by construction.

Harness (deploy/harbor/): swebench_rtk.py routes model traffic through the
same `off` passthrough proxy as the baseline (only variable = in-container
bash compression) and runs a custom claude-code-rtk Harbor agent
(committed reference copy: claude_code_rtk_agent.py) that uploads the
static-musl rtk binary and installs the hook per trial. deep_analysis.py /
deep_plots.py extended to a 4th arm.

Result (matched 50, all four arms 0 exceptions): rtk 43/50 solved,
$29.09 billed (-9.0% vs baseline), 91.7M cache-read (-10.8%), 33.2 steps,
0 ms/req, $0 tool cost. It is the 2nd-cheapest arm and reward-neutral vs
baseline, beating the headroom proxy on both cost and reward. Its 65.8%
bash-output reduction nets to -9% on the bill because bash output is a
small slice of a ~98%-cached agent; its ceiling is that only Bash-tool
output passes the hook (built-in Read/Grep/Glob bypass it).

Docs: new results/rtk.md; comparison.md, components.md, RESULTS.md,
README.md, home page, charts.js, and mkdocs nav updated to four-way;
REPRODUCE.md documents the rtk arm. mkdocs build clean.

Assisted-By: Claude (Opus 4.8)
Signed-off-by: Osher Elhadad <osher.elhadad@live.biu.ac.il>
@OsherElhadad
OsherElhadad merged commit e7c5990 into main Jul 27, 2026
6 checks passed
@github-project-automation github-project-automation Bot moved this from New/ToDo to Done in Rossoctl Issue Prioritization Jul 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants