Skip to content

Make e5 runnable: readiness gate, cost estimate, root-safe tests - #173

Merged
sykosyber merged 1 commit into
mainfrom
claude/e5-runnable
Nov 4, 2026
Merged

sykosyber merged 1 commit into
mainfrom
claude/e5-runnable

Conversation

@sykosyber

Copy link
Copy Markdown
Member

Admission

Class: R
Agent-platform: claude
Reviewed-by: none recorded
Admitted-by:
Evidence: tested

Objective

Factory plan, priority 3 (measured quality): make e5 (single vs repair) ready to run for real, and say what it will cost. This PR runs nothing paid. Starting the run spends money, which makes it Class X, so a named human starts it.

Before: e5 had never recorded a preflight. On a container running as root, preflight failed. Nothing checked that the adapter could actually reach a model. The adapter needs the anthropic package, which this repository does not install. Without it, every run records a provider error at $0, and because a run that left a bundle is never run again, an e5 started that way would lock in 48 zeros.

After: before the first run, the runner checks each arm and stops with adapter_unready if any arm can't run. In that case it writes no bundle and no spend line. --dry prints the cost estimate and each arm's readiness. The tests that failed only under root now check permission bits.

What changed

  • syberlabs/eval/arms.py: readiness(arm), which checks that:

    • the interpreter exists;
    • the adapter path is absolute and exists;
    • the required package imports in a child process.

    A missing ANTHROPIC_API_KEY or ANTHROPIC_AUTH_TOKEN is only a warning, because the adapter also accepts an ant auth login profile.

  • syberlabs/eval/runner.py: estimate() gives two figures:

    • the worst case, bounded by the per-run cap and what is left of total_dollars;
    • a nominal expected spend.

    It also adds the readiness gate before the first run. Both appear in the run result.

  • syberlabs/eval/cli.py: the dry run prints the estimate and readiness.

  • tests/test_eval_core.py: 5 new tests (CostAndReadiness) and estimate assertions on the e5 dry run.

  • tests/test_eval_runner.py: checks read-only staging by permission bits instead of os.access, which root bypasses.

  • docs/EVAL.md: what --dry prints, and a new "Running e5" section with the steps, variables, commands and cost.

What did not change

  • evals/e5-single-vs-repair/spec.json and registration.json, the task fixtures, and the adapter.
  • Nothing the GitHits pilot depends on: githits-pilot/, syberlabs/githits*.py, claude_code.py and query_guard.py are untouched. The pilot runs on its pinned commit b298cb6 in any case.
  • e5's harness was not deleted. Delete code no deployed app reaches: harnesses, examples, jev_api, unused symbols #166 removed benchmarks/model_vs_patch.py, the older E3 harness, which e5 never uses. Nothing was restored.

Evidence

PYTHONPATH=. python3 -m unittest discover -s tests -q
  1242 tests, 2 failures (Playwright browser tests: the node package is installed but its browsers are not, and the download is blocked here), 25 skipped, 5 expected failures
  (before: 1237 tests, 4 failures)
PYTHONPATH=. python3 -m conformance.run
  23 golden traces matched
PYTHONPATH=. python3 -m unittest tests.test_eval_core tests.test_eval_runner -q   (after rebasing on main with #154)
  Ran 42 tests, OK

Preflight passed on this commit, with node removed from PATH so the browser tests skipped as designed:

  • tests 0 (226 s)
  • conformance 0
  • scenarios 0
  • eval_tests 0

It was written into a temporary copy of evals/, not this repository.

Two simulated runs used the unmodified spec, the real adapter and anthropic 2.4.0. They ran against a stand-in Messages server on 127.0.0.1 with a fake key, so no money was spent:

  • v3-journal_chain seed 1:
    • repair was accepted in 2 calls, 3/3 hidden checks, $0.0076;
    • single was not acceptable, $0.0027.
  • v3-gaps_renumber seed 2, through the full CLI and the preflight gate:
    • repair was accepted and certified, 4/4 hidden checks, $0.0106.

A non-dry run with an interpreter lacking anthropic stopped adapter_unready and left the spend ledger unchanged.

The dry-run estimate for e5:

estimate: worst case $10.0000 for 48 pending run(s): 48 x $1.0000 per-run cap = $48.0000, bounded by the $10.0000 left of total_dollars
estimate: expected $0.4320 to $1.5120 at 2000 tokens in and 500 out per call at the spec's prices; a prior, not a measurement

For comparison, E3 measured Sonnet 5.5 on the v3 fixtures at about $0.0099 a call. That suggests about $0.35 for e5's v3 half. No v4 dollar figures are recorded in this repository.

Known limitations

  • No real API call has been made.

  • No preflight has run with the browser tests actually running.

  • The Windows behaviour of the permission-bits check is reasoned, not run.

  • To run e5, the owner needs to:

    • install anthropic;
    • set ANTHROPIC_API_KEY;
    • leave ANTHROPIC_BASE_URL unset;
    • optionally set SYBERLABS_CERTIFICATE_SEED;
    • then follow "Running e5" in docs/EVAL.md.

    The cap is $10.00, plus at most one call's cost.


Generated by Claude Code

… fix root-only test failures

e5's harness was not deleted: the eval runs through syberlabs/eval, the
adapter and benchmarks/fixtures.py, all present. What stood between it and
a real run:

- Preflight had never been recorded, and it failed in a container running
  as root: two eval tests asserted read-only staging with os.access, which
  root bypasses. They now check the mode bits, which hold on POSIX and
  Windows alike.
- Nothing checked that the adapter could run. adapters/anthropic_adapter.py
  needs the anthropic package, which this repository does not install; a
  run without it records a provider error at $0, and a run that left a
  bundle is never run again, so an eval started that way would lock in 48
  zeros. The runner now asks arms.readiness before its first run (the
  interpreter and adapter file exist, the path is absolute, the package
  imports in a child process) and stops adapter_unready instead; an unset
  ANTHROPIC_API_KEY/ANTHROPIC_AUTH_TOKEN is a warning. No model is called.
- `eval run <id> --dry` now prints the worst case (runs x per-run dollars,
  bounded by total_dollars left) and a nominal expected range from
  arms.NOMINAL_CALL at the spec's prices: e5 is $10.00 worst case, $0.43 to
  $1.51 nominal.

docs/EVAL.md gains "Running e5": the commands, the variables by name, and
the cap. spec.json, registration.json and evals/tasks are unchanged, so the
registration digests hold.

Class: R
Claim ceiling: tested (unit tests; two runs against a local stand-in for
the Messages API through the real adapter and SDK). Not measured.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sykosyber
sykosyber marked this pull request as ready for review November 4, 2026 05:22
@sykosyber
sykosyber merged commit c721d08 into main Nov 4, 2026
25 checks passed
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-04T05:23:41.919391Z b572239 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants