diff --git a/docs/EVAL.md b/docs/EVAL.md index a75be67..658e78d 100644 --- a/docs/EVAL.md +++ b/docs/EVAL.md @@ -43,7 +43,7 @@ syberlabs eval score [--json] # the gate, quality and efficie syberlabs eval report [--partial] [--json] # the paired analysis and the decision; report.md and report.json ``` -`--evals DIR` (before the subcommand) names another evals directory. In `--only`, an empty part or `*` matches anything. `--dry` prints the registered order with each run's state, `done` or `pending`, and runs nothing. It works before a preflight. `run` exits 1 when the eval stopped on its cap. `--hidden-root` is the host root under which each run's kit home and hidden copy get random names (default `/syberlabs-eval-host`). +`--evals DIR` (before the subcommand) names another evals directory. In `--only`, an empty part or `*` matches anything. `--dry` prints the registered order with each run's state, `done` or `pending`, and runs nothing. It works before a preflight. It also prints the pending runs' worst-case dollars (runs times the per-run `dollars`, bounded by what is left of `total_dollars`) and a nominal expected range (2000 tokens in and 500 out per call, `arms.NOMINAL_CALL`, at the spec's prices, from one call per run to each arm's K; a prior, not a measurement), and what would make every call of an arm fail before it reaches a model: an interpreter or adapter file that is not there, a relative adapter path, or a package the adapter needs that the interpreter cannot import (`anthropic` for `adapters/anthropic_adapter.py`). A run that is not dry stops `adapter_unready` on any of these before its first run, because each such run would leave a bundle scored 0 that is never run again; an unset `ANTHROPIC_API_KEY` and `ANTHROPIC_AUTH_TOKEN` is a warning only. `run` exits 1 when the eval stopped on its cap. `--hidden-root` is the host root under which each run's kit home and hidden copy get random names (default `/syberlabs-eval-host`). ## The spec @@ -222,3 +222,31 @@ syberlabs eval report e5-single-vs-repair ## Not here yet The agent-facing safety tasks (`evals/tasks/safety-*`, from the deterministic scenarios) and the validation of the gate on them, the gate rules that are `not_computable` for some arms, and what else is open are in `spec/GAPS.md`, 60 and 62. + +## Running e5 + +Running E5 spends money and is Class X: a named human runs it. Everything up to the spend has been exercised (2026-11-04, Linux, Python 3.13): the preflight (passed with node absent, so the browser tests skipped), and two runs through the unmodified spec, the real adapter and the `anthropic` SDK (2.4.0) against a local stand-in server, which spent nothing. Only the paid calls have not run. + +What it needs: + +- A clean checkout of the commit you will run (nothing changed outside `evals/`), from which the preflight has passed. The browser tests need Playwright's browsers when the `playwright` node package is installed (`npx playwright install chromium`), and skip when node or that package is absent. +- The `anthropic` package importable by the interpreter that runs `syberlabs`, because the spec's `{python}` is that interpreter: `python -m pip install anthropic`. Without it, the dry run says `not ready` and the run stops `adapter_unready` before its first run. +- `ANTHROPIC_API_KEY`, the adapter's credential (the SDK also reads `ANTHROPIC_AUTH_TOKEN`). SyberEval never writes it; the bundle's credential scan records only variable names. +- Optionally `SYBERLABS_CERTIFICATE_SEED`, the factory key, to certify accepted runs; unset, they are recorded `uncertified`. +- `ANTHROPIC_BASE_URL` unset, unless it names the real API: the SDK sends every call wherever it points. + +The commands, from the checkout (`python` is the interpreter with `anthropic` installed): + +```sh +export PYTHONPATH=. +python -m syberlabs eval preflight e5-single-vs-repair # about four minutes; writes evals/e5-single-vs-repair/preflight.json +python -m syberlabs eval run e5-single-vs-repair --dry # the order, the cost estimate, and readiness: expect no "not ready" line +python -m syberlabs eval run e5-single-vs-repair --only 'v3-journal_chain,repair,1' # optional: one paid run first +python -m syberlabs eval run e5-single-vs-repair # resumable by run id; stops at the cap +python -m syberlabs eval score e5-single-vs-repair +python -m syberlabs eval report e5-single-vs-repair +``` + +The cost: 48 runs, at most $1.00 each, and **$10.00 for the eval**, which the runner enforces from its spend ledger. The governor lets the call that crosses a ceiling complete, so recorded spend can pass the cap by one call's cost. The dry run prints the nominal expected spend as **$0.43 to $1.51** (one call per run up to six per `repair` run, at $0.009 a call). For comparison, E3 measured Claude Sonnet 5.5 at medium effort, K = 6, on the four v3 fixtures (three repeats, 2026-10-22 and 2026-10-28, [MODEL_COMPARISON.md](MODEL_COMPARISON.md)) at $0.119 for 12 `single` runs ($0.0099 a call) and $0.23 for 12 `repair` runs. That suggests about $0.35 for E5's v3 half. No v4 dollars are recorded in this repository. Dollars are the harness's pricing of the recorded tokens, not an invoice. + +The runs write `runs/`, `spend.jsonl`, `index.json`, `preflight.json` and the report under `evals/e5-single-vs-repair/`. Publishing a number from them is Class P. diff --git a/syberlabs/eval/arms.py b/syberlabs/eval/arms.py index b8f9070..916d422 100644 --- a/syberlabs/eval/arms.py +++ b/syberlabs/eval/arms.py @@ -42,7 +42,9 @@ import json import os import re +import shutil import statistics +import subprocess import sys from contextlib import contextmanager from pathlib import Path @@ -59,6 +61,12 @@ WITHHELD_ENV = ("GITHUB_TOKEN", "GH_TOKEN") NOMINAL_CALL = (2000, 500) # tokens in, out: Jev's prior price of one mutation HIDDEN_PREFIX = "hidden." # tasks.HIDDEN_PREFIX: the contract names every hidden check hidden. +# Adapters this checkout ships whose every call fails before it reaches a model unless the adapter's interpreter can +# import a package; the repository has no runtime dependencies, so nothing installs it (``readiness``). +ADAPTER_REQUIRES = {"anthropic_adapter.py": "anthropic"} +# The variables an adapter's SDK reads its credential from. None set is a warning, not a refusal: the adapter also +# names an ``ant auth login`` profile, which this check cannot see. +ADAPTER_CREDENTIALS = {"anthropic_adapter.py": ("ANTHROPIC_API_KEY", "ANTHROPIC_AUTH_TOKEN")} def check(arm: dict, budget: dict) -> list[str]: @@ -116,6 +124,49 @@ def resolve(argv: list[str], harness: Path = HARNESS) -> list[str]: return [sys.executable if arg == "{python}" else arg.replace("{harness}", str(harness)) for arg in argv] +def readiness(arm: dict, harness: Path = HARNESS, _imports: dict | None = None) -> dict: + """Why every call of a JSON-adapter ``arm`` would fail before it reaches a model (``blocking``), and what may + make it fail (``warnings``), without running the adapter or calling a model: the interpreter is found; each + ``.py`` file the argv names is an absolute path that exists (the adapter runs with the run's repository as its + working directory, where a relative path does not exist: E3, 2026-10-21); the interpreter can import the package + an adapter of ``ADAPTER_REQUIRES`` needs (a child process that only imports it); one of the adapter's + ``ADAPTER_CREDENTIALS`` is set. ``_imports`` caches the import probe by (interpreter, module).""" + blocking, warnings = [], [] + if arm["kind"] not in JSON_ADAPTER_KINDS: + return {"blocking": blocking, "warnings": warnings} + argv, label = resolve(arm["adapter"], harness), arm["name"] + if not argv or not (Path(argv[0]).is_file() or shutil.which(argv[0])): + return {"blocking": [f"{label}: interpreter {argv[0] if argv else None!r} not found"], "warnings": warnings} + imports = {} if _imports is None else _imports + for arg in argv[1:]: + path = Path(arg) + if path.suffix != ".py": + continue + if not path.is_absolute(): + blocking.append(f"{label}: adapter {arg} is a relative path; the adapter runs in the run's repository") + continue + if not path.is_file(): + blocking.append(f"{label}: adapter {arg} not found") + continue + module = ADAPTER_REQUIRES.get(path.name) + if module: + key = (argv[0], module) + if key not in imports: + try: + imports[key] = subprocess.run([argv[0], "-c", f"import {module}"], capture_output=True, + timeout=120).returncode == 0 + except (OSError, subprocess.TimeoutExpired): + imports[key] = False + if not imports[key]: + blocking.append(f"{label}: {path.name} needs the {module} package, which {argv[0]} cannot import " + f"(install it: {argv[0]} -m pip install {module})") + names = ADAPTER_CREDENTIALS.get(path.name) + if names and not any(os.environ.get(name) for name in names): + warnings.append(f"{label}: none of {', '.join(names)} is set; every call fails unless another credential " + f"source the SDK reads serves") + return {"blocking": blocking, "warnings": warnings} + + def jev_prices(prices: dict | None) -> dict | None: """Jev's price prior per action kind from a contract price table: one nominal call at the mean of the rates.""" if not prices: diff --git a/syberlabs/eval/cli.py b/syberlabs/eval/cli.py index be26cf1..6653103 100644 --- a/syberlabs/eval/cli.py +++ b/syberlabs/eval/cli.py @@ -119,4 +119,26 @@ def run(args) -> int: print(f"{found['eval']}: {len(found['runs'])} of {found['planned']} listed, spent " f"${found['spent_dollars']:.4f} of ${found['total_dollars']:.4f}; preflight {found['preflight']}" + (f"; stopped: {found['stopped']}" if found["stopped"] else "")) + if args.dry: + for line in describe_estimate(found["estimate"]): + print(line) + for line in found["readiness"]["blocking"]: + print(f"not ready: {line} (a run would stop adapter_unready)") + for line in found["readiness"]["warnings"]: + print(f"warning: {line}") return 1 if found["stopped"] else 0 + + +def describe_estimate(cost: dict) -> list[str]: + """``runner.estimate`` as the lines a dry run prints.""" + lines = [f"estimate: worst case ${cost['worst_case_dollars']:.4f} for {cost['runs']} pending run(s): " + f"{cost['runs']} x ${cost['per_run_dollars']:.4f} per-run cap = ${cost['uncapped_dollars']:.4f}, " + f"bounded by the ${cost['remaining_dollars']:.4f} left of total_dollars (the call that crosses a " + f"ceiling completes, so recorded spend can pass it by one call's cost)"] + expected = cost["expected_dollars"] + if expected: + lines.append(f"estimate: expected ${expected['low']:.4f} to ${expected['high']:.4f} at {expected['basis']}") + elif cost["runs"]: + lines.append("estimate: no expected figure (no nominal basis for arm(s) " + + ", ".join(cost["no_nominal_basis"]) + "); only the cap is known") + return lines diff --git a/syberlabs/eval/runner.py b/syberlabs/eval/runner.py index 2bcac7d..d60165c 100644 --- a/syberlabs/eval/runner.py +++ b/syberlabs/eval/runner.py @@ -52,6 +52,12 @@ runner holds an eval at a time (``runs/.lock``; a second is refused ``eval_running``), and the preflight gate is asked again before every run, so a checkout that moves or changes mid-eval stops it. +Before its first run, the runner asks ``arms.readiness`` of every arm a pending run uses (``readiness``): an +interpreter or adapter file that is not there, a relative adapter path, or a package the adapter needs that the +interpreter cannot import would make every call fail before a model and leave 0s that are never run again, so a run +that is not dry stops ``adapter_unready`` instead; an unset credential variable is logged as a warning. ``estimate`` +gives the pending runs' worst-case dollars (the cap) and a nominal expected range; a dry run prints both. + ``preflight`` runs the deterministic suites (``scripts/run_shards.py``, ``conformance.run``, ``benchmarks/run_scenarios.py``, and this package's tests) from the checkout and records the result for the commit in ``evals//preflight.json``. @@ -481,6 +487,12 @@ def _run(self, only: str | None, dry: bool, require_preflight: bool, gate: dict budget = self.doc["budget"] spent, unknown = self.spent(done) rows, stopped = [], None + pending = [item for item in planned if item.run_id not in done] + cost = estimate(self.doc, pending, spent) + ready = readiness(self.doc, pending, self.harness) + if not dry: + for line in ready["warnings"]: + self.log("warning: " + line) seed = None if not dry and os.environ.get(self.seed_env): from syberlabs.certificate import load_seed @@ -493,6 +505,10 @@ def _run(self, only: str | None, dry: bool, require_preflight: bool, gate: dict if dry: rows.append({"run_id": item.run_id, "order_index": item.order_index, "state": "pending"}) continue + if ready["blocking"]: # every call would fail before a model: a run would leave a bundle that is a 0 + stopped = "adapter_unready" + self.log("stop: adapter_unready (" + "; ".join(ready["blocking"]) + ")") + break if unknown: stopped = "eval_spend_unknown" self.log("stop: eval_spend_unknown (" + "; ".join(unknown[:3]) + ")") @@ -525,7 +541,7 @@ def _run(self, only: str | None, dry: bool, require_preflight: bool, gate: dict return {"eval": self.eval_id, "spec_digest": self.registration["spec_digest"], "dry": dry, "preflight": ("not required" if gate is None else gate.get("refused", "passed")), "planned": len(planned), "runs": rows, "stopped": stopped, "spent_dollars": round(spent, 6), - "total_dollars": budget["total_dollars"]} + "total_dollars": budget["total_dollars"], "estimate": cost, "readiness": ready} # One run @@ -695,6 +711,56 @@ def _certify(self, kit, thread, chosen, receipt, seed, partial: Path, repo_dir: return {"state": "certified:" + public.hex()[:10], "path": "certificate.json"} +def estimate(doc: dict, pending: list[Planned], spent: float = 0.0) -> dict: + """What the ``pending`` runs may spend, before any of them runs; never a bill and never a measurement. + + ``worst_case_dollars`` is the cap the runner enforces: the runs times the spec's per-run ``dollars``, bounded by + what is left of ``total_dollars`` after ``spent``; the governor lets the call that crosses a ceiling complete, so + recorded spend can pass it by one call's cost. ``expected_dollars`` is a nominal range, priced by the spec's table + at ``arms.NOMINAL_CALL`` tokens per call (Jev's prior price of one call), from one call per run to each arm's K + calls (a run's ceiling bounds each run's share); it is None when a pending run's arm is not a JSON-adapter arm + (a Claude Code call has no nominal size here) or its model is not in the table, and then only the cap is known.""" + budget = doc["budget"] + by_name = {arm["name"]: arm for arm in doc["arms"]} + remaining = max(0.0, budget["total_dollars"] - spent) + uncapped = len(pending) * budget["dollars"] + low = high = 0.0 + missing = [] + for item in pending: + arm = by_name[item.arm] + rate = (budget.get("prices") or {}).get(arm["model"]) + if arm["kind"] not in arms.JSON_ADAPTER_KINDS or not rate: + missing.append(item.arm) + continue + tokens_in, tokens_out = arms.NOMINAL_CALL + call = (tokens_in * rate["input_per_million"] + tokens_out * rate["output_per_million"]) / 1e6 + low += min(call, budget["dollars"]) + high += min(call * arms.calls(arm, budget), budget["dollars"]) + expected = None + if pending and not missing: + expected = {"low": round(min(low, remaining), 6), "high": round(min(high, remaining), 6), + "basis": f"{arms.NOMINAL_CALL[0]} tokens in and {arms.NOMINAL_CALL[1]} out per call " + f"(arms.NOMINAL_CALL) at the spec's prices, from 1 call per run to each arm's K; " + f"a prior, not a measurement"} + return {"runs": len(pending), "per_run_dollars": budget["dollars"], "uncapped_dollars": round(uncapped, 6), + "remaining_dollars": round(remaining, 6), "worst_case_dollars": round(min(uncapped, remaining), 6), + "expected_dollars": expected, "no_nominal_basis": sorted(set(missing))} + + +def readiness(doc: dict, pending: list[Planned], harness: Path = HARNESS) -> dict: + """``arms.readiness`` of every arm a pending run uses, in spec order: what would make every call fail before it + reaches a model (``blocking``; a run that is not dry stops ``adapter_unready`` before its first run) and what may + (``warnings``). It imports a package in a child process and calls no model.""" + used, imports = {item.arm for item in pending}, {} + found = {"blocking": [], "warnings": []} + for arm in doc["arms"]: + if arm["name"] in used: + result = arms.readiness(arm, harness, imports) + found["blocking"] += result["blocking"] + found["warnings"] += result["warnings"] + return found + + def run_eval(evals_root: str | Path, eval_id: str, *, only: str | None = None, dry: bool = False, require_preflight: bool = True, **options) -> dict: return EvalRun(evals_root, eval_id, **options).run(only=only, dry=dry, require_preflight=require_preflight) diff --git a/tests/test_eval_core.py b/tests/test_eval_core.py index 3b3af1d..de72832 100644 --- a/tests/test_eval_core.py +++ b/tests/test_eval_core.py @@ -252,7 +252,8 @@ def test_hidden_checks_are_staged_outside_and_named_by_absolute_path(self): "the copy re-hashes to the task") for path in staged.rglob("*"): if path.is_file(): - self.assertFalse(os.access(path, os.W_OK), f"{path.name} is read-only") + # The mode bits, not os.access: root may write a read-only file, and a container runs as root. + self.assertEqual(path.stat().st_mode & 0o222, 0, f"{path.name} is read-only") checks, required = tasks.contract_checks(task, staged) self.assertEqual(sorted(checks), ["hidden.discount", "hidden.shipping", "hidden.tax"]) self.assertEqual(required, sorted(checks)) @@ -516,6 +517,78 @@ def test_the_default_preflight_names_the_four_suites(self): self.assertIn("test_eval_*.py", argv["eval_tests"]) +class CostAndReadiness(Workspace): + """A dry run states what the pending runs may spend; a run whose adapter cannot reach a model never starts.""" + + def register(self, **changes): + doc = tiny_spec(self.ids, self.digests, **changes) + spec.register(self.write_spec(doc)) + return doc + + def test_a_dry_run_estimates_the_cap_and_a_nominal_range(self): + self.register() + cost = runner.run_eval(self.root, "tiny", require_preflight=False, dry=True)["estimate"] + # 8 runs at a $0.50 per-run cap under a $5.00 total; one nominal call is 2000 in and 500 out at $2 and $10 + # per million, $0.009: 8 runs at one call each, up to single 4 x 1 call and repair 4 x 3 calls. + self.assertEqual((cost["runs"], cost["uncapped_dollars"], cost["remaining_dollars"], + cost["worst_case_dollars"]), (8, 4.0, 5.0, 4.0)) + self.assertEqual((cost["expected_dollars"]["low"], cost["expected_dollars"]["high"]), (0.072, 0.144)) + self.assertIn("not a measurement", cost["expected_dollars"]["basis"]) + one = runner.run_eval(self.root, "tiny", require_preflight=False, dry=True, only="v1-pricing,repair,1") + self.assertEqual((one["estimate"]["worst_case_dollars"], one["estimate"]["expected_dollars"]["high"]), + (0.5, 0.027)) + + def test_the_total_cap_bounds_the_worst_case_and_an_unpriced_arm_has_no_expected_figure(self): + doc = tiny_spec(self.ids, self.digests) + doc["budget"] = dict(doc["budget"], total_dollars=1.0) + cost = runner.estimate(doc, runner.plan(doc), spent=0.25) + self.assertEqual((cost["uncapped_dollars"], cost["remaining_dollars"], cost["worst_case_dollars"]), + (4.0, 0.75, 0.75)) + doc["arms"] = [dict(doc["arms"][0], model="unpriced"), doc["arms"][1]] + cost = runner.estimate(doc, runner.plan(doc)) + self.assertIsNone(cost["expected_dollars"]) + self.assertEqual(cost["no_nominal_basis"], ["single"]) + self.assertEqual(runner.estimate(doc, [])["worst_case_dollars"], 0.0) + from syberlabs.eval.cli import describe_estimate + self.assertIn("only the cap is known", describe_estimate(cost)[-1]) + + def assert_stops_before_any_run(self, blocking: str): + dry = runner.run_eval(self.root, "tiny", require_preflight=False, dry=True) + self.assertTrue(any(blocking in line for line in dry["readiness"]["blocking"]), dry["readiness"]) + self.assertEqual({row["state"] for row in dry["runs"]}, {"pending"}) + found = runner.run_eval(self.root, "tiny", require_preflight=False, seed_env="SYBERLABS_EVAL_NO_SEED") + self.assertEqual((found["stopped"], found["runs"]), ("adapter_unready", [])) + runs = self.root / "tiny" / "runs" + self.assertEqual(sorted(p.name for p in runs.iterdir()) if runs.is_dir() else [], [runner.LOCK], + "the runner's lock and no run, no bundle") + self.assertFalse((self.root / "tiny" / runner.SPEND_FILE).exists(), "nothing charged") + + def test_a_missing_adapter_file_stops_the_eval_before_its_first_run(self): + missing = ["{python}", "{harness}/benchmarks/no_such_adapter.py"] + doc = tiny_spec(self.ids, self.digests) + self.register(arms=[dict(arm, adapter=missing) for arm in doc["arms"]]) + self.assert_stops_before_any_run("no_such_adapter.py not found") + + def test_a_relative_adapter_path_stops_the_eval_before_its_first_run(self): + relative = ["{python}", "benchmarks/simulated_model.py"] # E3 2026-10-21: the adapter runs in the run's repo + doc = tiny_spec(self.ids, self.digests) + self.register(arms=[dict(arm, adapter=relative) for arm in doc["arms"]]) + self.assert_stops_before_any_run("relative path") + + def test_a_package_the_adapter_needs_and_cannot_import_stops_the_eval(self): + from unittest import mock + self.register() + with mock.patch.dict(arms.ADAPTER_REQUIRES, {"simulated_model.py": "syberlabs_eval_no_such_module"}): + self.assert_stops_before_any_run("cannot import") + with mock.patch.dict(arms.ADAPTER_CREDENTIALS, {"simulated_model.py": ("SYBERLABS_EVAL_NO_SUCH_KEY",)}): + os.environ.pop("SYBERLABS_EVAL_NO_SUCH_KEY", None) + found = runner.run_eval(self.root, "tiny", require_preflight=False, dry=True) + self.assertEqual(found["readiness"]["blocking"], []) + self.assertEqual(len(found["readiness"]["warnings"]), 2, "an unset credential warns once per arm") + self.assertEqual(arms.readiness({"name": "cc", "kind": "claude-code"}), + {"blocking": [], "warnings": []}, "only JSON-adapter arms are probed") + + class Terminal(unittest.TestCase): def cli(self, *args): from syberlabs.cli import main @@ -575,6 +648,15 @@ def test_tasks_list_and_a_dry_run_of_e5(self): self.assertEqual(found["preflight"], "preflight_missing") self.assertEqual([row["order_index"] for row in found["runs"]], list(range(48))) self.assertEqual([row["run_id"] for row in found["runs"]], [p.run_id for p in runner.plan(doc)]) + # the cap: 48 runs at $1.00 bounded by the $10.00 total; nominal: 2000 in and 500 out at $2 and $10 per + # million is $0.009 a call, 48 runs at one call each up to single 24 x 1 and repair 24 x 6 calls + cost = found["estimate"] + self.assertEqual((cost["uncapped_dollars"], cost["worst_case_dollars"]), (48.0, 10.0)) + self.assertEqual((cost["expected_dollars"]["low"], cost["expected_dollars"]["high"]), (0.432, 1.512)) + code, out = self.cli("eval", "--evals", str(evals), "run", "e5-single-vs-repair", "--dry") + self.assertEqual(code, 0) + self.assertIn("estimate: worst case $10.0000 for 48 pending run(s)", out) + self.assertIn("estimate: expected $0.4320 to $1.5120", out) code, _ = self.cli("eval", "--evals", str(evals), "run", "e5-single-vs-repair", "--dry", "--only", "v3-journal_chain,repair,") self.assertEqual(code, 0) diff --git a/tests/test_eval_runner.py b/tests/test_eval_runner.py index 31f8835..bd20877 100644 --- a/tests/test_eval_runner.py +++ b/tests/test_eval_runner.py @@ -252,7 +252,8 @@ def test_the_agent_never_holds_a_hidden_check_and_cannot_shadow_one(self): self.assertEqual(path.parent.resolve(), self.host.resolve()) self.assertRegex(path.name, r"^[0-9a-f]{32}$") self.assertEqual(sorted(p.name for p in work.iterdir()), ["repo"]) - self.assertTrue(all(not os.access(p, os.W_OK) for p in staged.rglob("*") if p.is_file()), "read-only") + # The mode bits, not os.access: root may write a read-only file, and a container runs as root. + self.assertTrue(all(p.stat().st_mode & 0o222 == 0 for p in staged.rglob("*") if p.is_file()), "read-only") integrity = bundle["hidden_integrity"] self.assertEqual([(h["point"], h["match"]) for h in integrity["hashes"]], [(point, True) for point in ("staged", "after_propose", "after_check", "before_accept")])