Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 29 additions & 1 deletion docs/EVAL.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ syberlabs eval score <id> [--json] # the gate, quality and efficie
syberlabs eval report <id> [--partial] [--json] # the paired analysis and the decision; report.md and report.json
```

`--evals DIR` (before the subcommand) names another evals directory. In `--only`, an empty part or `*` matches anything. `--dry` prints the registered order with each run's state, `done` or `pending`, and runs nothing. It works before a preflight. `run` exits 1 when the eval stopped on its cap. `--hidden-root` is the host root under which each run's kit home and hidden copy get random names (default `<system temp>/syberlabs-eval-host`).
`--evals DIR` (before the subcommand) names another evals directory. In `--only`, an empty part or `*` matches anything. `--dry` prints the registered order with each run's state, `done` or `pending`, and runs nothing. It works before a preflight. It also prints the pending runs' worst-case dollars (runs times the per-run `dollars`, bounded by what is left of `total_dollars`) and a nominal expected range (2000 tokens in and 500 out per call, `arms.NOMINAL_CALL`, at the spec's prices, from one call per run to each arm's K; a prior, not a measurement), and what would make every call of an arm fail before it reaches a model: an interpreter or adapter file that is not there, a relative adapter path, or a package the adapter needs that the interpreter cannot import (`anthropic` for `adapters/anthropic_adapter.py`). A run that is not dry stops `adapter_unready` on any of these before its first run, because each such run would leave a bundle scored 0 that is never run again; an unset `ANTHROPIC_API_KEY` and `ANTHROPIC_AUTH_TOKEN` is a warning only. `run` exits 1 when the eval stopped on its cap. `--hidden-root` is the host root under which each run's kit home and hidden copy get random names (default `<system temp>/syberlabs-eval-host`).

## The spec

Expand Down Expand Up @@ -222,3 +222,31 @@ syberlabs eval report e5-single-vs-repair
## Not here yet

The agent-facing safety tasks (`evals/tasks/safety-*`, from the deterministic scenarios) and the validation of the gate on them, the gate rules that are `not_computable` for some arms, and what else is open are in `spec/GAPS.md`, 60 and 62.

## Running e5

Running E5 spends money and is Class X: a named human runs it. Everything up to the spend has been exercised (2026-11-04, Linux, Python 3.13): the preflight (passed with node absent, so the browser tests skipped), and two runs through the unmodified spec, the real adapter and the `anthropic` SDK (2.4.0) against a local stand-in server, which spent nothing. Only the paid calls have not run.

What it needs:

- A clean checkout of the commit you will run (nothing changed outside `evals/`), from which the preflight has passed. The browser tests need Playwright's browsers when the `playwright` node package is installed (`npx playwright install chromium`), and skip when node or that package is absent.
- The `anthropic` package importable by the interpreter that runs `syberlabs`, because the spec's `{python}` is that interpreter: `python -m pip install anthropic`. Without it, the dry run says `not ready` and the run stops `adapter_unready` before its first run.
- `ANTHROPIC_API_KEY`, the adapter's credential (the SDK also reads `ANTHROPIC_AUTH_TOKEN`). SyberEval never writes it; the bundle's credential scan records only variable names.
- Optionally `SYBERLABS_CERTIFICATE_SEED`, the factory key, to certify accepted runs; unset, they are recorded `uncertified`.
- `ANTHROPIC_BASE_URL` unset, unless it names the real API: the SDK sends every call wherever it points.

The commands, from the checkout (`python` is the interpreter with `anthropic` installed):

```sh
export PYTHONPATH=.
python -m syberlabs eval preflight e5-single-vs-repair # about four minutes; writes evals/e5-single-vs-repair/preflight.json
python -m syberlabs eval run e5-single-vs-repair --dry # the order, the cost estimate, and readiness: expect no "not ready" line
python -m syberlabs eval run e5-single-vs-repair --only 'v3-journal_chain,repair,1' # optional: one paid run first
python -m syberlabs eval run e5-single-vs-repair # resumable by run id; stops at the cap
python -m syberlabs eval score e5-single-vs-repair
python -m syberlabs eval report e5-single-vs-repair
```

The cost: 48 runs, at most $1.00 each, and **$10.00 for the eval**, which the runner enforces from its spend ledger. The governor lets the call that crosses a ceiling complete, so recorded spend can pass the cap by one call's cost. The dry run prints the nominal expected spend as **$0.43 to $1.51** (one call per run up to six per `repair` run, at $0.009 a call). For comparison, E3 measured Claude Sonnet 5.5 at medium effort, K = 6, on the four v3 fixtures (three repeats, 2026-10-22 and 2026-10-28, [MODEL_COMPARISON.md](MODEL_COMPARISON.md)) at $0.119 for 12 `single` runs ($0.0099 a call) and $0.23 for 12 `repair` runs. That suggests about $0.35 for E5's v3 half. No v4 dollars are recorded in this repository. Dollars are the harness's pricing of the recorded tokens, not an invoice.

The runs write `runs/`, `spend.jsonl`, `index.json`, `preflight.json` and the report under `evals/e5-single-vs-repair/`. Publishing a number from them is Class P.
51 changes: 51 additions & 0 deletions syberlabs/eval/arms.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,9 @@
import json
import os
import re
import shutil
import statistics
import subprocess
import sys
from contextlib import contextmanager
from pathlib import Path
Expand All @@ -59,6 +61,12 @@
WITHHELD_ENV = ("GITHUB_TOKEN", "GH_TOKEN")
NOMINAL_CALL = (2000, 500) # tokens in, out: Jev's prior price of one mutation
HIDDEN_PREFIX = "hidden." # tasks.HIDDEN_PREFIX: the contract names every hidden check hidden.<name>
# Adapters this checkout ships whose every call fails before it reaches a model unless the adapter's interpreter can
# import a package; the repository has no runtime dependencies, so nothing installs it (``readiness``).
ADAPTER_REQUIRES = {"anthropic_adapter.py": "anthropic"}
# The variables an adapter's SDK reads its credential from. None set is a warning, not a refusal: the adapter also
# names an ``ant auth login`` profile, which this check cannot see.
ADAPTER_CREDENTIALS = {"anthropic_adapter.py": ("ANTHROPIC_API_KEY", "ANTHROPIC_AUTH_TOKEN")}


def check(arm: dict, budget: dict) -> list[str]:
Expand Down Expand Up @@ -116,6 +124,49 @@ def resolve(argv: list[str], harness: Path = HARNESS) -> list[str]:
return [sys.executable if arg == "{python}" else arg.replace("{harness}", str(harness)) for arg in argv]


def readiness(arm: dict, harness: Path = HARNESS, _imports: dict | None = None) -> dict:
"""Why every call of a JSON-adapter ``arm`` would fail before it reaches a model (``blocking``), and what may
make it fail (``warnings``), without running the adapter or calling a model: the interpreter is found; each
``.py`` file the argv names is an absolute path that exists (the adapter runs with the run's repository as its
working directory, where a relative path does not exist: E3, 2026-10-21); the interpreter can import the package
an adapter of ``ADAPTER_REQUIRES`` needs (a child process that only imports it); one of the adapter's
``ADAPTER_CREDENTIALS`` is set. ``_imports`` caches the import probe by (interpreter, module)."""
blocking, warnings = [], []
if arm["kind"] not in JSON_ADAPTER_KINDS:
return {"blocking": blocking, "warnings": warnings}
argv, label = resolve(arm["adapter"], harness), arm["name"]
if not argv or not (Path(argv[0]).is_file() or shutil.which(argv[0])):
return {"blocking": [f"{label}: interpreter {argv[0] if argv else None!r} not found"], "warnings": warnings}
imports = {} if _imports is None else _imports
for arg in argv[1:]:
path = Path(arg)
if path.suffix != ".py":
continue
if not path.is_absolute():
blocking.append(f"{label}: adapter {arg} is a relative path; the adapter runs in the run's repository")
continue
if not path.is_file():
blocking.append(f"{label}: adapter {arg} not found")
continue
module = ADAPTER_REQUIRES.get(path.name)
if module:
key = (argv[0], module)
if key not in imports:
try:
imports[key] = subprocess.run([argv[0], "-c", f"import {module}"], capture_output=True,
timeout=120).returncode == 0
except (OSError, subprocess.TimeoutExpired):
imports[key] = False
if not imports[key]:
blocking.append(f"{label}: {path.name} needs the {module} package, which {argv[0]} cannot import "
f"(install it: {argv[0]} -m pip install {module})")
names = ADAPTER_CREDENTIALS.get(path.name)
if names and not any(os.environ.get(name) for name in names):
warnings.append(f"{label}: none of {', '.join(names)} is set; every call fails unless another credential "
f"source the SDK reads serves")
return {"blocking": blocking, "warnings": warnings}


def jev_prices(prices: dict | None) -> dict | None:
"""Jev's price prior per action kind from a contract price table: one nominal call at the mean of the rates."""
if not prices:
Expand Down
22 changes: 22 additions & 0 deletions syberlabs/eval/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -119,4 +119,26 @@ def run(args) -> int:
print(f"{found['eval']}: {len(found['runs'])} of {found['planned']} listed, spent "
f"${found['spent_dollars']:.4f} of ${found['total_dollars']:.4f}; preflight {found['preflight']}"
+ (f"; stopped: {found['stopped']}" if found["stopped"] else ""))
if args.dry:
for line in describe_estimate(found["estimate"]):
print(line)
for line in found["readiness"]["blocking"]:
print(f"not ready: {line} (a run would stop adapter_unready)")
for line in found["readiness"]["warnings"]:
print(f"warning: {line}")
return 1 if found["stopped"] else 0


def describe_estimate(cost: dict) -> list[str]:
"""``runner.estimate`` as the lines a dry run prints."""
lines = [f"estimate: worst case ${cost['worst_case_dollars']:.4f} for {cost['runs']} pending run(s): "
f"{cost['runs']} x ${cost['per_run_dollars']:.4f} per-run cap = ${cost['uncapped_dollars']:.4f}, "
f"bounded by the ${cost['remaining_dollars']:.4f} left of total_dollars (the call that crosses a "
f"ceiling completes, so recorded spend can pass it by one call's cost)"]
expected = cost["expected_dollars"]
if expected:
lines.append(f"estimate: expected ${expected['low']:.4f} to ${expected['high']:.4f} at {expected['basis']}")
elif cost["runs"]:
lines.append("estimate: no expected figure (no nominal basis for arm(s) "
+ ", ".join(cost["no_nominal_basis"]) + "); only the cap is known")
return lines
68 changes: 67 additions & 1 deletion syberlabs/eval/runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,12 @@
runner holds an eval at a time (``runs/.lock``; a second is refused ``eval_running``), and the preflight gate is
asked again before every run, so a checkout that moves or changes mid-eval stops it.

Before its first run, the runner asks ``arms.readiness`` of every arm a pending run uses (``readiness``): an
interpreter or adapter file that is not there, a relative adapter path, or a package the adapter needs that the
interpreter cannot import would make every call fail before a model and leave 0s that are never run again, so a run
that is not dry stops ``adapter_unready`` instead; an unset credential variable is logged as a warning. ``estimate``
gives the pending runs' worst-case dollars (the cap) and a nominal expected range; a dry run prints both.

``preflight`` runs the deterministic suites (``scripts/run_shards.py``, ``conformance.run``,
``benchmarks/run_scenarios.py``, and this package's tests) from the checkout and records the result for the
commit in ``evals/<id>/preflight.json``.
Expand Down Expand Up @@ -481,6 +487,12 @@ def _run(self, only: str | None, dry: bool, require_preflight: bool, gate: dict
budget = self.doc["budget"]
spent, unknown = self.spent(done)
rows, stopped = [], None
pending = [item for item in planned if item.run_id not in done]
cost = estimate(self.doc, pending, spent)
ready = readiness(self.doc, pending, self.harness)
if not dry:
for line in ready["warnings"]:
self.log("warning: " + line)
seed = None
if not dry and os.environ.get(self.seed_env):
from syberlabs.certificate import load_seed
Expand All @@ -493,6 +505,10 @@ def _run(self, only: str | None, dry: bool, require_preflight: bool, gate: dict
if dry:
rows.append({"run_id": item.run_id, "order_index": item.order_index, "state": "pending"})
continue
if ready["blocking"]: # every call would fail before a model: a run would leave a bundle that is a 0
stopped = "adapter_unready"
self.log("stop: adapter_unready (" + "; ".join(ready["blocking"]) + ")")
break
if unknown:
stopped = "eval_spend_unknown"
self.log("stop: eval_spend_unknown (" + "; ".join(unknown[:3]) + ")")
Expand Down Expand Up @@ -525,7 +541,7 @@ def _run(self, only: str | None, dry: bool, require_preflight: bool, gate: dict
return {"eval": self.eval_id, "spec_digest": self.registration["spec_digest"], "dry": dry,
"preflight": ("not required" if gate is None else gate.get("refused", "passed")),
"planned": len(planned), "runs": rows, "stopped": stopped, "spent_dollars": round(spent, 6),
"total_dollars": budget["total_dollars"]}
"total_dollars": budget["total_dollars"], "estimate": cost, "readiness": ready}

# One run

Expand Down Expand Up @@ -695,6 +711,56 @@ def _certify(self, kit, thread, chosen, receipt, seed, partial: Path, repo_dir:
return {"state": "certified:" + public.hex()[:10], "path": "certificate.json"}


def estimate(doc: dict, pending: list[Planned], spent: float = 0.0) -> dict:
"""What the ``pending`` runs may spend, before any of them runs; never a bill and never a measurement.

``worst_case_dollars`` is the cap the runner enforces: the runs times the spec's per-run ``dollars``, bounded by
what is left of ``total_dollars`` after ``spent``; the governor lets the call that crosses a ceiling complete, so
recorded spend can pass it by one call's cost. ``expected_dollars`` is a nominal range, priced by the spec's table
at ``arms.NOMINAL_CALL`` tokens per call (Jev's prior price of one call), from one call per run to each arm's K
calls (a run's ceiling bounds each run's share); it is None when a pending run's arm is not a JSON-adapter arm
(a Claude Code call has no nominal size here) or its model is not in the table, and then only the cap is known."""
budget = doc["budget"]
by_name = {arm["name"]: arm for arm in doc["arms"]}
remaining = max(0.0, budget["total_dollars"] - spent)
uncapped = len(pending) * budget["dollars"]
low = high = 0.0
missing = []
for item in pending:
arm = by_name[item.arm]
rate = (budget.get("prices") or {}).get(arm["model"])
if arm["kind"] not in arms.JSON_ADAPTER_KINDS or not rate:
missing.append(item.arm)
continue
tokens_in, tokens_out = arms.NOMINAL_CALL
call = (tokens_in * rate["input_per_million"] + tokens_out * rate["output_per_million"]) / 1e6
low += min(call, budget["dollars"])
high += min(call * arms.calls(arm, budget), budget["dollars"])
expected = None
if pending and not missing:
expected = {"low": round(min(low, remaining), 6), "high": round(min(high, remaining), 6),
"basis": f"{arms.NOMINAL_CALL[0]} tokens in and {arms.NOMINAL_CALL[1]} out per call "
f"(arms.NOMINAL_CALL) at the spec's prices, from 1 call per run to each arm's K; "
f"a prior, not a measurement"}
return {"runs": len(pending), "per_run_dollars": budget["dollars"], "uncapped_dollars": round(uncapped, 6),
"remaining_dollars": round(remaining, 6), "worst_case_dollars": round(min(uncapped, remaining), 6),
"expected_dollars": expected, "no_nominal_basis": sorted(set(missing))}


def readiness(doc: dict, pending: list[Planned], harness: Path = HARNESS) -> dict:
"""``arms.readiness`` of every arm a pending run uses, in spec order: what would make every call fail before it
reaches a model (``blocking``; a run that is not dry stops ``adapter_unready`` before its first run) and what may
(``warnings``). It imports a package in a child process and calls no model."""
used, imports = {item.arm for item in pending}, {}
found = {"blocking": [], "warnings": []}
for arm in doc["arms"]:
if arm["name"] in used:
result = arms.readiness(arm, harness, imports)
found["blocking"] += result["blocking"]
found["warnings"] += result["warnings"]
return found


def run_eval(evals_root: str | Path, eval_id: str, *, only: str | None = None, dry: bool = False,
require_preflight: bool = True, **options) -> dict:
return EvalRun(evals_root, eval_id, **options).run(only=only, dry=dry, require_preflight=require_preflight)
Expand Down
Loading
Loading