From b5722398e6f1b349c2e874cba534d5936c350575 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 4 Nov 2026 01:19:43 +0000 Subject: [PATCH] Make e5 runnable: stop runs that cannot reach a model, estimate cost, fix root-only test failures e5's harness was not deleted: the eval runs through syberlabs/eval, the adapter and benchmarks/fixtures.py, all present. What stood between it and a real run: - Preflight had never been recorded, and it failed in a container running as root: two eval tests asserted read-only staging with os.access, which root bypasses. They now check the mode bits, which hold on POSIX and Windows alike. - Nothing checked that the adapter could run. adapters/anthropic_adapter.py needs the anthropic package, which this repository does not install; a run without it records a provider error at $0, and a run that left a bundle is never run again, so an eval started that way would lock in 48 zeros. The runner now asks arms.readiness before its first run (the interpreter and adapter file exist, the path is absolute, the package imports in a child process) and stops adapter_unready instead; an unset ANTHROPIC_API_KEY/ANTHROPIC_AUTH_TOKEN is a warning. No model is called. - `eval run --dry` now prints the worst case (runs x per-run dollars, bounded by total_dollars left) and a nominal expected range from arms.NOMINAL_CALL at the spec's prices: e5 is $10.00 worst case, $0.43 to $1.51 nominal. docs/EVAL.md gains "Running e5": the commands, the variables by name, and the cap. spec.json, registration.json and evals/tasks are unchanged, so the registration digests hold. Class: R Claim ceiling: tested (unit tests; two runs against a local stand-in for the Messages API through the real adapter and SDK). Not measured. Co-Authored-By: Claude Opus 5.5 --- docs/EVAL.md | 30 +++++++++++++- syberlabs/eval/arms.py | 51 ++++++++++++++++++++++++ syberlabs/eval/cli.py | 22 ++++++++++ syberlabs/eval/runner.py | 68 ++++++++++++++++++++++++++++++- tests/test_eval_core.py | 84 ++++++++++++++++++++++++++++++++++++++- tests/test_eval_runner.py | 3 +- 6 files changed, 254 insertions(+), 4 deletions(-) diff --git a/docs/EVAL.md b/docs/EVAL.md index a75be679..658e78d4 100644 --- a/docs/EVAL.md +++ b/docs/EVAL.md @@ -43,7 +43,7 @@ syberlabs eval score [--json] # the gate, quality and efficie syberlabs eval report [--partial] [--json] # the paired analysis and the decision; report.md and report.json ``` -`--evals DIR` (before the subcommand) names another evals directory. In `--only`, an empty part or `*` matches anything. `--dry` prints the registered order with each run's state, `done` or `pending`, and runs nothing. It works before a preflight. `run` exits 1 when the eval stopped on its cap. `--hidden-root` is the host root under which each run's kit home and hidden copy get random names (default `/syberlabs-eval-host`). +`--evals DIR` (before the subcommand) names another evals directory. In `--only`, an empty part or `*` matches anything. `--dry` prints the registered order with each run's state, `done` or `pending`, and runs nothing. It works before a preflight. It also prints the pending runs' worst-case dollars (runs times the per-run `dollars`, bounded by what is left of `total_dollars`) and a nominal expected range (2000 tokens in and 500 out per call, `arms.NOMINAL_CALL`, at the spec's prices, from one call per run to each arm's K; a prior, not a measurement), and what would make every call of an arm fail before it reaches a model: an interpreter or adapter file that is not there, a relative adapter path, or a package the adapter needs that the interpreter cannot import (`anthropic` for `adapters/anthropic_adapter.py`). A run that is not dry stops `adapter_unready` on any of these before its first run, because each such run would leave a bundle scored 0 that is never run again; an unset `ANTHROPIC_API_KEY` and `ANTHROPIC_AUTH_TOKEN` is a warning only. `run` exits 1 when the eval stopped on its cap. `--hidden-root` is the host root under which each run's kit home and hidden copy get random names (default `/syberlabs-eval-host`). ## The spec @@ -222,3 +222,31 @@ syberlabs eval report e5-single-vs-repair ## Not here yet The agent-facing safety tasks (`evals/tasks/safety-*`, from the deterministic scenarios) and the validation of the gate on them, the gate rules that are `not_computable` for some arms, and what else is open are in `spec/GAPS.md`, 60 and 62. + +## Running e5 + +Running E5 spends money and is Class X: a named human runs it. Everything up to the spend has been exercised (2026-11-04, Linux, Python 3.13): the preflight (passed with node absent, so the browser tests skipped), and two runs through the unmodified spec, the real adapter and the `anthropic` SDK (2.4.0) against a local stand-in server, which spent nothing. Only the paid calls have not run. + +What it needs: + +- A clean checkout of the commit you will run (nothing changed outside `evals/`), from which the preflight has passed. The browser tests need Playwright's browsers when the `playwright` node package is installed (`npx playwright install chromium`), and skip when node or that package is absent. +- The `anthropic` package importable by the interpreter that runs `syberlabs`, because the spec's `{python}` is that interpreter: `python -m pip install anthropic`. Without it, the dry run says `not ready` and the run stops `adapter_unready` before its first run. +- `ANTHROPIC_API_KEY`, the adapter's credential (the SDK also reads `ANTHROPIC_AUTH_TOKEN`). SyberEval never writes it; the bundle's credential scan records only variable names. +- Optionally `SYBERLABS_CERTIFICATE_SEED`, the factory key, to certify accepted runs; unset, they are recorded `uncertified`. +- `ANTHROPIC_BASE_URL` unset, unless it names the real API: the SDK sends every call wherever it points. + +The commands, from the checkout (`python` is the interpreter with `anthropic` installed): + +```sh +export PYTHONPATH=. +python -m syberlabs eval preflight e5-single-vs-repair # about four minutes; writes evals/e5-single-vs-repair/preflight.json +python -m syberlabs eval run e5-single-vs-repair --dry # the order, the cost estimate, and readiness: expect no "not ready" line +python -m syberlabs eval run e5-single-vs-repair --only 'v3-journal_chain,repair,1' # optional: one paid run first +python -m syberlabs eval run e5-single-vs-repair # resumable by run id; stops at the cap +python -m syberlabs eval score e5-single-vs-repair +python -m syberlabs eval report e5-single-vs-repair +``` + +The cost: 48 runs, at most $1.00 each, and **$10.00 for the eval**, which the runner enforces from its spend ledger. The governor lets the call that crosses a ceiling complete, so recorded spend can pass the cap by one call's cost. The dry run prints the nominal expected spend as **$0.43 to $1.51** (one call per run up to six per `repair` run, at $0.009 a call). For comparison, E3 measured Claude Sonnet 5.5 at medium effort, K = 6, on the four v3 fixtures (three repeats, 2026-10-22 and 2026-10-28, [MODEL_COMPARISON.md](MODEL_COMPARISON.md)) at $0.119 for 12 `single` runs ($0.0099 a call) and $0.23 for 12 `repair` runs. That suggests about $0.35 for E5's v3 half. No v4 dollars are recorded in this repository. Dollars are the harness's pricing of the recorded tokens, not an invoice. + +The runs write `runs/`, `spend.jsonl`, `index.json`, `preflight.json` and the report under `evals/e5-single-vs-repair/`. Publishing a number from them is Class P. diff --git a/syberlabs/eval/arms.py b/syberlabs/eval/arms.py index b8f9070f..916d4226 100644 --- a/syberlabs/eval/arms.py +++ b/syberlabs/eval/arms.py @@ -42,7 +42,9 @@ import json import os import re +import shutil import statistics +import subprocess import sys from contextlib import contextmanager from pathlib import Path @@ -59,6 +61,12 @@ WITHHELD_ENV = ("GITHUB_TOKEN", "GH_TOKEN") NOMINAL_CALL = (2000, 500) # tokens in, out: Jev's prior price of one mutation HIDDEN_PREFIX = "hidden." # tasks.HIDDEN_PREFIX: the contract names every hidden check hidden. +# Adapters this checkout ships whose every call fails before it reaches a model unless the adapter's interpreter can +# import a package; the repository has no runtime dependencies, so nothing installs it (``readiness``). +ADAPTER_REQUIRES = {"anthropic_adapter.py": "anthropic"} +# The variables an adapter's SDK reads its credential from. None set is a warning, not a refusal: the adapter also +# names an ``ant auth login`` profile, which this check cannot see. +ADAPTER_CREDENTIALS = {"anthropic_adapter.py": ("ANTHROPIC_API_KEY", "ANTHROPIC_AUTH_TOKEN")} def check(arm: dict, budget: dict) -> list[str]: @@ -116,6 +124,49 @@ def resolve(argv: list[str], harness: Path = HARNESS) -> list[str]: return [sys.executable if arg == "{python}" else arg.replace("{harness}", str(harness)) for arg in argv] +def readiness(arm: dict, harness: Path = HARNESS, _imports: dict | None = None) -> dict: + """Why every call of a JSON-adapter ``arm`` would fail before it reaches a model (``blocking``), and what may + make it fail (``warnings``), without running the adapter or calling a model: the interpreter is found; each + ``.py`` file the argv names is an absolute path that exists (the adapter runs with the run's repository as its + working directory, where a relative path does not exist: E3, 2026-10-21); the interpreter can import the package + an adapter of ``ADAPTER_REQUIRES`` needs (a child process that only imports it); one of the adapter's + ``ADAPTER_CREDENTIALS`` is set. ``_imports`` caches the import probe by (interpreter, module).""" + blocking, warnings = [], [] + if arm["kind"] not in JSON_ADAPTER_KINDS: + return {"blocking": blocking, "warnings": warnings} + argv, label = resolve(arm["adapter"], harness), arm["name"] + if not argv or not (Path(argv[0]).is_file() or shutil.which(argv[0])): + return {"blocking": [f"{label}: interpreter {argv[0] if argv else None!r} not found"], "warnings": warnings} + imports = {} if _imports is None else _imports + for arg in argv[1:]: + path = Path(arg) + if path.suffix != ".py": + continue + if not path.is_absolute(): + blocking.append(f"{label}: adapter {arg} is a relative path; the adapter runs in the run's repository") + continue + if not path.is_file(): + blocking.append(f"{label}: adapter {arg} not found") + continue + module = ADAPTER_REQUIRES.get(path.name) + if module: + key = (argv[0], module) + if key not in imports: + try: + imports[key] = subprocess.run([argv[0], "-c", f"import {module}"], capture_output=True, + timeout=120).returncode == 0 + except (OSError, subprocess.TimeoutExpired): + imports[key] = False + if not imports[key]: + blocking.append(f"{label}: {path.name} needs the {module} package, which {argv[0]} cannot import " + f"(install it: {argv[0]} -m pip install {module})") + names = ADAPTER_CREDENTIALS.get(path.name) + if names and not any(os.environ.get(name) for name in names): + warnings.append(f"{label}: none of {', '.join(names)} is set; every call fails unless another credential " + f"source the SDK reads serves") + return {"blocking": blocking, "warnings": warnings} + + def jev_prices(prices: dict | None) -> dict | None: """Jev's price prior per action kind from a contract price table: one nominal call at the mean of the rates.""" if not prices: diff --git a/syberlabs/eval/cli.py b/syberlabs/eval/cli.py index be26cf17..66531033 100644 --- a/syberlabs/eval/cli.py +++ b/syberlabs/eval/cli.py @@ -119,4 +119,26 @@ def run(args) -> int: print(f"{found['eval']}: {len(found['runs'])} of {found['planned']} listed, spent " f"${found['spent_dollars']:.4f} of ${found['total_dollars']:.4f}; preflight {found['preflight']}" + (f"; stopped: {found['stopped']}" if found["stopped"] else "")) + if args.dry: + for line in describe_estimate(found["estimate"]): + print(line) + for line in found["readiness"]["blocking"]: + print(f"not ready: {line} (a run would stop adapter_unready)") + for line in found["readiness"]["warnings"]: + print(f"warning: {line}") return 1 if found["stopped"] else 0 + + +def describe_estimate(cost: dict) -> list[str]: + """``runner.estimate`` as the lines a dry run prints.""" + lines = [f"estimate: worst case ${cost['worst_case_dollars']:.4f} for {cost['runs']} pending run(s): " + f"{cost['runs']} x ${cost['per_run_dollars']:.4f} per-run cap = ${cost['uncapped_dollars']:.4f}, " + f"bounded by the ${cost['remaining_dollars']:.4f} left of total_dollars (the call that crosses a " + f"ceiling completes, so recorded spend can pass it by one call's cost)"] + expected = cost["expected_dollars"] + if expected: + lines.append(f"estimate: expected ${expected['low']:.4f} to ${expected['high']:.4f} at {expected['basis']}") + elif cost["runs"]: + lines.append("estimate: no expected figure (no nominal basis for arm(s) " + + ", ".join(cost["no_nominal_basis"]) + "); only the cap is known") + return lines diff --git a/syberlabs/eval/runner.py b/syberlabs/eval/runner.py index 2bcac7d5..d60165ca 100644 --- a/syberlabs/eval/runner.py +++ b/syberlabs/eval/runner.py @@ -52,6 +52,12 @@ runner holds an eval at a time (``runs/.lock``; a second is refused ``eval_running``), and the preflight gate is asked again before every run, so a checkout that moves or changes mid-eval stops it. +Before its first run, the runner asks ``arms.readiness`` of every arm a pending run uses (``readiness``): an +interpreter or adapter file that is not there, a relative adapter path, or a package the adapter needs that the +interpreter cannot import would make every call fail before a model and leave 0s that are never run again, so a run +that is not dry stops ``adapter_unready`` instead; an unset credential variable is logged as a warning. ``estimate`` +gives the pending runs' worst-case dollars (the cap) and a nominal expected range; a dry run prints both. + ``preflight`` runs the deterministic suites (``scripts/run_shards.py``, ``conformance.run``, ``benchmarks/run_scenarios.py``, and this package's tests) from the checkout and records the result for the commit in ``evals//preflight.json``. @@ -481,6 +487,12 @@ def _run(self, only: str | None, dry: bool, require_preflight: bool, gate: dict budget = self.doc["budget"] spent, unknown = self.spent(done) rows, stopped = [], None + pending = [item for item in planned if item.run_id not in done] + cost = estimate(self.doc, pending, spent) + ready = readiness(self.doc, pending, self.harness) + if not dry: + for line in ready["warnings"]: + self.log("warning: " + line) seed = None if not dry and os.environ.get(self.seed_env): from syberlabs.certificate import load_seed @@ -493,6 +505,10 @@ def _run(self, only: str | None, dry: bool, require_preflight: bool, gate: dict if dry: rows.append({"run_id": item.run_id, "order_index": item.order_index, "state": "pending"}) continue + if ready["blocking"]: # every call would fail before a model: a run would leave a bundle that is a 0 + stopped = "adapter_unready" + self.log("stop: adapter_unready (" + "; ".join(ready["blocking"]) + ")") + break if unknown: stopped = "eval_spend_unknown" self.log("stop: eval_spend_unknown (" + "; ".join(unknown[:3]) + ")") @@ -525,7 +541,7 @@ def _run(self, only: str | None, dry: bool, require_preflight: bool, gate: dict return {"eval": self.eval_id, "spec_digest": self.registration["spec_digest"], "dry": dry, "preflight": ("not required" if gate is None else gate.get("refused", "passed")), "planned": len(planned), "runs": rows, "stopped": stopped, "spent_dollars": round(spent, 6), - "total_dollars": budget["total_dollars"]} + "total_dollars": budget["total_dollars"], "estimate": cost, "readiness": ready} # One run @@ -695,6 +711,56 @@ def _certify(self, kit, thread, chosen, receipt, seed, partial: Path, repo_dir: return {"state": "certified:" + public.hex()[:10], "path": "certificate.json"} +def estimate(doc: dict, pending: list[Planned], spent: float = 0.0) -> dict: + """What the ``pending`` runs may spend, before any of them runs; never a bill and never a measurement. + + ``worst_case_dollars`` is the cap the runner enforces: the runs times the spec's per-run ``dollars``, bounded by + what is left of ``total_dollars`` after ``spent``; the governor lets the call that crosses a ceiling complete, so + recorded spend can pass it by one call's cost. ``expected_dollars`` is a nominal range, priced by the spec's table + at ``arms.NOMINAL_CALL`` tokens per call (Jev's prior price of one call), from one call per run to each arm's K + calls (a run's ceiling bounds each run's share); it is None when a pending run's arm is not a JSON-adapter arm + (a Claude Code call has no nominal size here) or its model is not in the table, and then only the cap is known.""" + budget = doc["budget"] + by_name = {arm["name"]: arm for arm in doc["arms"]} + remaining = max(0.0, budget["total_dollars"] - spent) + uncapped = len(pending) * budget["dollars"] + low = high = 0.0 + missing = [] + for item in pending: + arm = by_name[item.arm] + rate = (budget.get("prices") or {}).get(arm["model"]) + if arm["kind"] not in arms.JSON_ADAPTER_KINDS or not rate: + missing.append(item.arm) + continue + tokens_in, tokens_out = arms.NOMINAL_CALL + call = (tokens_in * rate["input_per_million"] + tokens_out * rate["output_per_million"]) / 1e6 + low += min(call, budget["dollars"]) + high += min(call * arms.calls(arm, budget), budget["dollars"]) + expected = None + if pending and not missing: + expected = {"low": round(min(low, remaining), 6), "high": round(min(high, remaining), 6), + "basis": f"{arms.NOMINAL_CALL[0]} tokens in and {arms.NOMINAL_CALL[1]} out per call " + f"(arms.NOMINAL_CALL) at the spec's prices, from 1 call per run to each arm's K; " + f"a prior, not a measurement"} + return {"runs": len(pending), "per_run_dollars": budget["dollars"], "uncapped_dollars": round(uncapped, 6), + "remaining_dollars": round(remaining, 6), "worst_case_dollars": round(min(uncapped, remaining), 6), + "expected_dollars": expected, "no_nominal_basis": sorted(set(missing))} + + +def readiness(doc: dict, pending: list[Planned], harness: Path = HARNESS) -> dict: + """``arms.readiness`` of every arm a pending run uses, in spec order: what would make every call fail before it + reaches a model (``blocking``; a run that is not dry stops ``adapter_unready`` before its first run) and what may + (``warnings``). It imports a package in a child process and calls no model.""" + used, imports = {item.arm for item in pending}, {} + found = {"blocking": [], "warnings": []} + for arm in doc["arms"]: + if arm["name"] in used: + result = arms.readiness(arm, harness, imports) + found["blocking"] += result["blocking"] + found["warnings"] += result["warnings"] + return found + + def run_eval(evals_root: str | Path, eval_id: str, *, only: str | None = None, dry: bool = False, require_preflight: bool = True, **options) -> dict: return EvalRun(evals_root, eval_id, **options).run(only=only, dry=dry, require_preflight=require_preflight) diff --git a/tests/test_eval_core.py b/tests/test_eval_core.py index 3b3af1de..de72832f 100644 --- a/tests/test_eval_core.py +++ b/tests/test_eval_core.py @@ -252,7 +252,8 @@ def test_hidden_checks_are_staged_outside_and_named_by_absolute_path(self): "the copy re-hashes to the task") for path in staged.rglob("*"): if path.is_file(): - self.assertFalse(os.access(path, os.W_OK), f"{path.name} is read-only") + # The mode bits, not os.access: root may write a read-only file, and a container runs as root. + self.assertEqual(path.stat().st_mode & 0o222, 0, f"{path.name} is read-only") checks, required = tasks.contract_checks(task, staged) self.assertEqual(sorted(checks), ["hidden.discount", "hidden.shipping", "hidden.tax"]) self.assertEqual(required, sorted(checks)) @@ -516,6 +517,78 @@ def test_the_default_preflight_names_the_four_suites(self): self.assertIn("test_eval_*.py", argv["eval_tests"]) +class CostAndReadiness(Workspace): + """A dry run states what the pending runs may spend; a run whose adapter cannot reach a model never starts.""" + + def register(self, **changes): + doc = tiny_spec(self.ids, self.digests, **changes) + spec.register(self.write_spec(doc)) + return doc + + def test_a_dry_run_estimates_the_cap_and_a_nominal_range(self): + self.register() + cost = runner.run_eval(self.root, "tiny", require_preflight=False, dry=True)["estimate"] + # 8 runs at a $0.50 per-run cap under a $5.00 total; one nominal call is 2000 in and 500 out at $2 and $10 + # per million, $0.009: 8 runs at one call each, up to single 4 x 1 call and repair 4 x 3 calls. + self.assertEqual((cost["runs"], cost["uncapped_dollars"], cost["remaining_dollars"], + cost["worst_case_dollars"]), (8, 4.0, 5.0, 4.0)) + self.assertEqual((cost["expected_dollars"]["low"], cost["expected_dollars"]["high"]), (0.072, 0.144)) + self.assertIn("not a measurement", cost["expected_dollars"]["basis"]) + one = runner.run_eval(self.root, "tiny", require_preflight=False, dry=True, only="v1-pricing,repair,1") + self.assertEqual((one["estimate"]["worst_case_dollars"], one["estimate"]["expected_dollars"]["high"]), + (0.5, 0.027)) + + def test_the_total_cap_bounds_the_worst_case_and_an_unpriced_arm_has_no_expected_figure(self): + doc = tiny_spec(self.ids, self.digests) + doc["budget"] = dict(doc["budget"], total_dollars=1.0) + cost = runner.estimate(doc, runner.plan(doc), spent=0.25) + self.assertEqual((cost["uncapped_dollars"], cost["remaining_dollars"], cost["worst_case_dollars"]), + (4.0, 0.75, 0.75)) + doc["arms"] = [dict(doc["arms"][0], model="unpriced"), doc["arms"][1]] + cost = runner.estimate(doc, runner.plan(doc)) + self.assertIsNone(cost["expected_dollars"]) + self.assertEqual(cost["no_nominal_basis"], ["single"]) + self.assertEqual(runner.estimate(doc, [])["worst_case_dollars"], 0.0) + from syberlabs.eval.cli import describe_estimate + self.assertIn("only the cap is known", describe_estimate(cost)[-1]) + + def assert_stops_before_any_run(self, blocking: str): + dry = runner.run_eval(self.root, "tiny", require_preflight=False, dry=True) + self.assertTrue(any(blocking in line for line in dry["readiness"]["blocking"]), dry["readiness"]) + self.assertEqual({row["state"] for row in dry["runs"]}, {"pending"}) + found = runner.run_eval(self.root, "tiny", require_preflight=False, seed_env="SYBERLABS_EVAL_NO_SEED") + self.assertEqual((found["stopped"], found["runs"]), ("adapter_unready", [])) + runs = self.root / "tiny" / "runs" + self.assertEqual(sorted(p.name for p in runs.iterdir()) if runs.is_dir() else [], [runner.LOCK], + "the runner's lock and no run, no bundle") + self.assertFalse((self.root / "tiny" / runner.SPEND_FILE).exists(), "nothing charged") + + def test_a_missing_adapter_file_stops_the_eval_before_its_first_run(self): + missing = ["{python}", "{harness}/benchmarks/no_such_adapter.py"] + doc = tiny_spec(self.ids, self.digests) + self.register(arms=[dict(arm, adapter=missing) for arm in doc["arms"]]) + self.assert_stops_before_any_run("no_such_adapter.py not found") + + def test_a_relative_adapter_path_stops_the_eval_before_its_first_run(self): + relative = ["{python}", "benchmarks/simulated_model.py"] # E3 2026-10-21: the adapter runs in the run's repo + doc = tiny_spec(self.ids, self.digests) + self.register(arms=[dict(arm, adapter=relative) for arm in doc["arms"]]) + self.assert_stops_before_any_run("relative path") + + def test_a_package_the_adapter_needs_and_cannot_import_stops_the_eval(self): + from unittest import mock + self.register() + with mock.patch.dict(arms.ADAPTER_REQUIRES, {"simulated_model.py": "syberlabs_eval_no_such_module"}): + self.assert_stops_before_any_run("cannot import") + with mock.patch.dict(arms.ADAPTER_CREDENTIALS, {"simulated_model.py": ("SYBERLABS_EVAL_NO_SUCH_KEY",)}): + os.environ.pop("SYBERLABS_EVAL_NO_SUCH_KEY", None) + found = runner.run_eval(self.root, "tiny", require_preflight=False, dry=True) + self.assertEqual(found["readiness"]["blocking"], []) + self.assertEqual(len(found["readiness"]["warnings"]), 2, "an unset credential warns once per arm") + self.assertEqual(arms.readiness({"name": "cc", "kind": "claude-code"}), + {"blocking": [], "warnings": []}, "only JSON-adapter arms are probed") + + class Terminal(unittest.TestCase): def cli(self, *args): from syberlabs.cli import main @@ -575,6 +648,15 @@ def test_tasks_list_and_a_dry_run_of_e5(self): self.assertEqual(found["preflight"], "preflight_missing") self.assertEqual([row["order_index"] for row in found["runs"]], list(range(48))) self.assertEqual([row["run_id"] for row in found["runs"]], [p.run_id for p in runner.plan(doc)]) + # the cap: 48 runs at $1.00 bounded by the $10.00 total; nominal: 2000 in and 500 out at $2 and $10 per + # million is $0.009 a call, 48 runs at one call each up to single 24 x 1 and repair 24 x 6 calls + cost = found["estimate"] + self.assertEqual((cost["uncapped_dollars"], cost["worst_case_dollars"]), (48.0, 10.0)) + self.assertEqual((cost["expected_dollars"]["low"], cost["expected_dollars"]["high"]), (0.432, 1.512)) + code, out = self.cli("eval", "--evals", str(evals), "run", "e5-single-vs-repair", "--dry") + self.assertEqual(code, 0) + self.assertIn("estimate: worst case $10.0000 for 48 pending run(s)", out) + self.assertIn("estimate: expected $0.4320 to $1.5120", out) code, _ = self.cli("eval", "--evals", str(evals), "run", "e5-single-vs-repair", "--dry", "--only", "v3-journal_chain,repair,") self.assertEqual(code, 0) diff --git a/tests/test_eval_runner.py b/tests/test_eval_runner.py index 31f88352..bd208778 100644 --- a/tests/test_eval_runner.py +++ b/tests/test_eval_runner.py @@ -252,7 +252,8 @@ def test_the_agent_never_holds_a_hidden_check_and_cannot_shadow_one(self): self.assertEqual(path.parent.resolve(), self.host.resolve()) self.assertRegex(path.name, r"^[0-9a-f]{32}$") self.assertEqual(sorted(p.name for p in work.iterdir()), ["repo"]) - self.assertTrue(all(not os.access(p, os.W_OK) for p in staged.rglob("*") if p.is_file()), "read-only") + # The mode bits, not os.access: root may write a read-only file, and a container runs as root. + self.assertTrue(all(p.stat().st_mode & 0o222 == 0 for p in staged.rglob("*") if p.is_file()), "read-only") integrity = bundle["hidden_integrity"] self.assertEqual([(h["point"], h["match"]) for h in integrity["hashes"]], [(point, True) for point in ("staged", "after_propose", "after_check", "before_accept")])