Repository navigation
Make e5 runnable: readiness gate, cost estimate, root-safe tests - #173
Merged
Merged
Conversation
… fix root-only test failures e5's harness was not deleted: the eval runs through syberlabs/eval, the adapter and benchmarks/fixtures.py, all present. What stood between it and a real run: - Preflight had never been recorded, and it failed in a container running as root: two eval tests asserted read-only staging with os.access, which root bypasses. They now check the mode bits, which hold on POSIX and Windows alike. - Nothing checked that the adapter could run. adapters/anthropic_adapter.py needs the anthropic package, which this repository does not install; a run without it records a provider error at $0, and a run that left a bundle is never run again, so an eval started that way would lock in 48 zeros. The runner now asks arms.readiness before its first run (the interpreter and adapter file exist, the path is absolute, the package imports in a child process) and stops adapter_unready instead; an unset ANTHROPIC_API_KEY/ANTHROPIC_AUTH_TOKEN is a warning. No model is called. - `eval run <id> --dry` now prints the worst case (runs x per-run dollars, bounded by total_dollars left) and a nominal expected range from arms.NOMINAL_CALL at the spec's prices: e5 is $10.00 worst case, $0.43 to $1.51 nominal. docs/EVAL.md gains "Running e5": the commands, the variables by name, and the cap. spec.json, registration.json and evals/tasks are unchanged, so the registration digests hold. Class: R Claim ceiling: tested (unit tests; two runs against a local stand-in for the Messages API through the real adapter and SDK). Not measured. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sykosyber
marked this pull request as ready for review
November 4, 2026 05:22
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Admission
Class: R
Agent-platform: claude
Reviewed-by: none recorded
Admitted-by:
Evidence: tested
Objective
Factory plan, priority 3 (measured quality): make e5 (single vs repair) ready to run for real, and say what it will cost. This PR runs nothing paid. Starting the run spends money, which makes it Class X, so a named human starts it.
Before: e5 had never recorded a preflight. On a container running as root, preflight failed. Nothing checked that the adapter could actually reach a model. The adapter needs the
anthropicpackage, which this repository does not install. Without it, every run records a provider error at $0, and because a run that left a bundle is never run again, an e5 started that way would lock in 48 zeros.After: before the first run, the runner checks each arm and stops with
adapter_unreadyif any arm can't run. In that case it writes no bundle and no spend line.--dryprints the cost estimate and each arm's readiness. The tests that failed only under root now check permission bits.What changed
syberlabs/eval/arms.py:readiness(arm), which checks that:A missing
ANTHROPIC_API_KEYorANTHROPIC_AUTH_TOKENis only a warning, because the adapter also accepts anant auth loginprofile.syberlabs/eval/runner.py:estimate()gives two figures:total_dollars;It also adds the readiness gate before the first run. Both appear in the run result.
syberlabs/eval/cli.py: the dry run prints the estimate and readiness.tests/test_eval_core.py: 5 new tests (CostAndReadiness) and estimate assertions on the e5 dry run.tests/test_eval_runner.py: checks read-only staging by permission bits instead ofos.access, which root bypasses.docs/EVAL.md: what--dryprints, and a new "Running e5" section with the steps, variables, commands and cost.What did not change
evals/e5-single-vs-repair/spec.jsonandregistration.json, the task fixtures, and the adapter.githits-pilot/,syberlabs/githits*.py,claude_code.pyandquery_guard.pyare untouched. The pilot runs on its pinned commit b298cb6 in any case.benchmarks/model_vs_patch.py, the older E3 harness, which e5 never uses. Nothing was restored.Evidence
Preflight passed on this commit, with node removed from PATH so the browser tests skipped as designed:
It was written into a temporary copy of
evals/, not this repository.Two simulated runs used the unmodified spec, the real adapter and
anthropic2.4.0. They ran against a stand-in Messages server on 127.0.0.1 with a fake key, so no money was spent:v3-journal_chainseed 1:v3-gaps_renumberseed 2, through the full CLI and the preflight gate:A non-dry run with an interpreter lacking
anthropicstoppedadapter_unreadyand left the spend ledger unchanged.The dry-run estimate for e5:
For comparison, E3 measured Sonnet 5.5 on the v3 fixtures at about $0.0099 a call. That suggests about $0.35 for e5's v3 half. No v4 dollar figures are recorded in this repository.
Known limitations
No real API call has been made.
No preflight has run with the browser tests actually running.
The Windows behaviour of the permission-bits check is reasoned, not run.
To run e5, the owner needs to:
anthropic;ANTHROPIC_API_KEY;ANTHROPIC_BASE_URLunset;SYBERLABS_CERTIFICATE_SEED;The cap is $10.00, plus at most one call's cost.
Generated by Claude Code