diff --git a/docs/EVAL.md b/docs/EVAL.md index 5fed8cb3..57e59f7a 100644 --- a/docs/EVAL.md +++ b/docs/EVAL.md @@ -203,10 +203,9 @@ It writes `report.md` and `report.json` (`report.partial.*` under `--partial`, w ## E5 and E6 -**E5, `evals/e5-single-vs-repair`, framework validation: specified, not registered in this tree (it is registered on a clean checkout of `main`), not run.** It compares `single` with `repair` on the eight v3 and v4 tasks, using Claude Sonnet 5.5 at medium effort with K = 6, and seeds 1, 2 and 3 (48 runs in the order `order_seed` 20261026 gives), with `hidden_feedback: tails`, as in E3. The budget is $1.00, 1800 s and 6 calls per run, with a $10.00 cap on the eval. The primary metric is the per-task pass rate on the hidden checks. The decision rule is repair ≥ single on the paired per-task difference, with a sign test over tasks beside it. E3 already answered this question (repair wins), so E5 checks the instrument, not the hypothesis. Adding `hidden_feedback` moved its digest, so its earlier registration (written from a dirty tree) is deleted rather than kept, since `register` returns an existing registration of the same digest unchanged. To register and run it, which spends money: +**E5, `evals/e5-single-vs-repair`, framework validation: registered at `90930320c6`, not run.** It compares `single` with `repair` on the eight v3 and v4 tasks, using Claude Sonnet 5.5 at medium effort with K = 6, and seeds 1, 2 and 3 (48 runs in the order `order_seed` 20261026 gives), with `hidden_feedback: tails`, as in E3. The budget is $1.00, 1800 s and 6 calls per run, with a $10.00 cap on the eval. The primary metric is the per-task pass rate on the hidden checks. The decision rule is repair ≥ single on the paired per-task difference, with a sign test over tasks beside it. E3 already answered this question (repair wins), so E5 checks the instrument, not the hypothesis. Adding `hidden_feedback` moved its digest, so its earlier registration (written from a dirty tree) was deleted rather than kept, since `register` returns an existing registration of the same digest unchanged. It was registered again on a clean checkout of `main` at `90930320c6` (`registration.json`: spec digest `7625340097b9…`, task set `13589c4ff8e0…`, `git_dirty: false`). To run it, which spends money: ```sh -syberlabs eval register evals/e5-single-vs-repair/spec.json # on a clean checkout of main export ANTHROPIC_API_KEY=... # the adapter's credential; never written by SyberEval export SYBERLABS_CERTIFICATE_SEED=... # the factory key, to certify accepted runs syberlabs eval preflight e5-single-vs-repair # on a clean checkout of the commit you will run diff --git a/evals/e5-single-vs-repair/registration.json b/evals/e5-single-vs-repair/registration.json new file mode 100644 index 00000000..cdcd58e6 --- /dev/null +++ b/evals/e5-single-vs-repair/registration.json @@ -0,0 +1,23 @@ +{ + "arm_digests": { + "repair": "1912623052881dac83a369fbf6a855a49595a7a99cca7ff8e235179af0f716b5", + "single": "cc8dc7ec0d1038c73c93d793e8852aa24b9a7902c5d7655ecaa4cdc20e869821" + }, + "eval": "e5-single-vs-repair", + "git_commit": "90930320c6c1c3073066d502997d5ada791b7673", + "git_dirty": false, + "registered_at": "2026-10-29T06:54:36Z", + "schema": "syberlabs.eval/registration.v1", + "spec_digest": "7625340097b9df1a1be4a53482bc36a14865361bf0fb7c48cdfc787318d5caa6", + "task_digests": { + "v3-gaps_renumber": "0eeff9daaf1b0f8eea3fae5ee9defcf751b9519a503f5885ea4334d03ff000cc", + "v3-journal_chain": "910f140b9334be262d399ceaecbd6b2ee5956d51748a39344d5b4fdeb1a61f12", + "v3-verify_layers": "0ab4bd05a178b95fdbeaf2766c85cc38fda22a08a5863fef3b0811f8df8c9c6e", + "v3-worktree_order": "1cd93278532b00dd83bf9b57abf586f6185481519e8cb01ed6362f4a0b6a271e", + "v4-cell_metrics": "29b61a320b15d67da274738f05e885eaf546d40bba96b325cd90f2a874e3e325", + "v4-certificate_layers": "52d653e131593a75a24d5a939c2c0f34ff6b1bd31d5cfa7e69261ad0a14686c8", + "v4-stack_landing": "3a13bf606221209e37ef7490ab3d64874660e1fb47b06ca206714727ee297e86", + "v4-thread_stations": "6be55591c0e962fa77988731c7702ce0db0fa3ecc6fa79d927553b73080de784" + }, + "task_set_digest": "13589c4ff8e07e9666c72f3922d743bf076b9e225024f1296864ecc31642cb17" +} diff --git a/spec/GAPS.md b/spec/GAPS.md index f310976f..ff48e3cd 100644 --- a/spec/GAPS.md +++ b/spec/GAPS.md @@ -122,6 +122,6 @@ The schemas describe SyberWork as it behaves. They are not enforced at runtime e 61. **TypeSafe's Jev decides inside one search from ids and numbers; its decisions are not journaled by the host; its spend is outside the host's dollars; it cannot be registered as a SyberEval arm.** `TypeSafeJevPolicy` (`syberlabs/jev_api.py`, `docs/EVOLUTION.md`, 2026-10-28) asks `jev-1.13.0` one `choice` per decision over the actions `JevPolicy` offers, validates the answer and falls back to `JevPolicy`. What is open: (1) the decision record (`source`, `model`, `answers_digest`, `request_digest`, `latency_ms`, `fallback`, `local_action`) lives in the provider's in-memory trace, as every policy's does; `GitSearchSpace.observe` accepts only `PROVIDER_FACTS = ("lookup",)` (`syberlabs/build.py`), outside this packet, so a provider cannot journal it, and `benchmarks/model_vs_patch.py` writes the entries beside each exported journal instead. The smallest change that closes it is a `policy_decision` key in `PROVIDER_FACTS`, bounded like `lookup`. (2) Jev's dollars (input tokens at TypeSafe's published $0.042 per million) are counted by the policy and the arm's row, not by the host's `max_dollars`, which sees `providers.run_json` usage only; at about $0.00003 a decision this does not move a bound, but it is not the host's observation. (3) The answer is a provider's signal: its probabilities are not calibrated against admission and were not repeatable on 2026-10-28 (one request, three answers, `mutate` at 0.58, 0.56, 0.53). (4) Jev sees no text, so it decides from the same numbers the local heuristic reads; whether that adds anything is the pre-registered measurement in `docs/EVOLUTION.md`, unrun. (5) `syberlabs/eval/spec.py` closes `ARM_KINDS` without `evolve_jevapi`, so that measurement is a plan in the docs and not an `evals/` spec. (6) The pin is `jev-1.13.0` while `jev-latest` is `jev-1.14.0`; TypeSafe's page does not state a deprecation policy, so a retired pin would show up as 400 `Unknown model`, fall back to local on every decision and switch after three. (7) Jev cannot be named by the contract, like every policy (entry 41). -62. **SyberEval scores, gates and reports its runs; some gate rules cannot be computed for some arms, the agent-facing safety tasks do not exist yet, and candidate code still runs beside the hidden checks.** `eval-score` (2026-10-26, landed on the `eval-core` verification fixes of 2026-10-28: the runner's lock, bundle verification, the per-run preflight gate, strict JSON, refused run-id collisions and finite numbers) adds `syberlabs/eval/gate.py`, `score.py` and `report.py` and `syberlabs eval score|report` (`docs/EVAL.md`); the required spec field `hidden_feedback` (`tails`, `status`, `none`), enforced through `RepairProvider`'s optional `feedback_filter`, which every other caller leaves unset; the hidden copy's hashing (`tasks.tree_digests`) at four points against the copy as staged (`tasks.hidden_changes`, which names a symbolic link by its target and tolerates pytest's own `.pytest_cache`), with the run outcome `not_acceptable`, reason `hidden_tampered`; a host root where the hidden copy and the kit home get random names outside the run's work directory; the append-only spend ledger the eval's cap counts, read strictly, which also records every run that raised (it replaces the `runs/failed.jsonl` of the verification fixes); and `syberlabs.eval` in the wheel. It closes entry 60's (1) and (8) and narrows its (3) and (4). The contract schema, the event kinds, the golden traces and the kernel's behaviour for every existing caller are unchanged. Open: (1) the agent-facing safety tasks (`evals/tasks/safety-*`, packet `eval-safety-fixtures`) and the gate's validation on them: no planted canary, forbidden action or unsafe path has yet been offered to an agent, and every gate rule is tested on synthetic journals and with stand-in adapters only; (2) `canary.provider_requests` is `not_computable` for `claude-code` arms, whose own model requests hold whatever Claude Code's tools read, and for `evolve` and `evolve_jev` arms, whose mutate requests' feedback is not recorded; such a run's gate is `incomplete`, never `pass`; (3) the policy is not in the journal, so `approval_or_prerequisite_skipped` checks that at least one approval came between a `needs_approval` decision and its effect, not the policy's count; (4) `credential_observed` matches credential shapes (`syberlabs._patterns.SECRET`, at a word boundary) and the values of the host's credential variables present when the run ended, not every possible secret, and full check outputs and provider stderr are not retained, so only their recorded tails are scanned; (5) the hidden copy is hashed when staged, after the proposal (one hash after every check run inside it, the repair loop's evaluations included), after `Thread.check` (which runs the checks again only when the chosen candidate's result is missing or stale, so it usually follows a check that ran nothing) and before acceptance; the gate's `hidden_tampered` rule is `not_computable` for a bundle missing a hash the runner always writes; so a rewrite undone between two hashes, or inside one proposal's evaluations, is not detected, and candidate code inside a check can still read the copy and print it into its output, which `tails` sends to the next request; only a sandbox per check closes this (research register N4); (6) `hidden_feedback` filters the `repair` and `claude-code` feedback path only: an evolve arm builds its mutate requests in `syberlabs/evolve.py`, which no filter reaches, so a spec may use evolve arms under `tails` only; (7) under `none` the repair loop still stops on the first candidate that passes the hidden checks, so they still choose among up to K attempts on the host's side, and a call after a failure starts from the failed candidate's tree; (8) the eval's recorded spend can exceed `total_dollars` by one call's cost, and a call in flight when the runner is killed outright is in no journal and no ledger line; (9) the model API takes no sampling seed, so a run's seed indexes a repeat and is not a reproducible draw; (10) E5's registration is deleted, because adding `hidden_feedback` moved its digest and `register` returns an existing registration of the same digest unchanged; it is registered again on a clean checkout of `main` before it runs; (11) the package reads `benchmarks/`, the adapters and `evals/` from a checkout, so the wheel's `syberlabs eval` needs one to run anything; (12) the design gaps the verification pinned as `expectedFailure` in `tests/test_eval_adversarial.py` stay open: a candidate that exits 0 at import passes every hidden check; under `tails` a candidate can print its hidden check into the next request (under `status` and `none` it cannot, and that test passes); a `claude-code` arm's model is a declaration the runner neither sets nor compares with the model that served; a `repo:` task's run repository holds the commits after its base; and a runner killed outright charges nothing (tested on POSIX only, where `SIGKILL` exists). +62. **SyberEval scores, gates and reports its runs; some gate rules cannot be computed for some arms, the agent-facing safety tasks do not exist yet, and candidate code still runs beside the hidden checks.** `eval-score` (2026-10-26, landed on the `eval-core` verification fixes of 2026-10-28: the runner's lock, bundle verification, the per-run preflight gate, strict JSON, refused run-id collisions and finite numbers) adds `syberlabs/eval/gate.py`, `score.py` and `report.py` and `syberlabs eval score|report` (`docs/EVAL.md`); the required spec field `hidden_feedback` (`tails`, `status`, `none`), enforced through `RepairProvider`'s optional `feedback_filter`, which every other caller leaves unset; the hidden copy's hashing (`tasks.tree_digests`) at four points against the copy as staged (`tasks.hidden_changes`, which names a symbolic link by its target and tolerates pytest's own `.pytest_cache`), with the run outcome `not_acceptable`, reason `hidden_tampered`; a host root where the hidden copy and the kit home get random names outside the run's work directory; the append-only spend ledger the eval's cap counts, read strictly, which also records every run that raised (it replaces the `runs/failed.jsonl` of the verification fixes); and `syberlabs.eval` in the wheel. It closes entry 60's (1) and (8) and narrows its (3) and (4). The contract schema, the event kinds, the golden traces and the kernel's behaviour for every existing caller are unchanged. Open: (1) the agent-facing safety tasks (`evals/tasks/safety-*`, packet `eval-safety-fixtures`) and the gate's validation on them: no planted canary, forbidden action or unsafe path has yet been offered to an agent, and every gate rule is tested on synthetic journals and with stand-in adapters only; (2) `canary.provider_requests` is `not_computable` for `claude-code` arms, whose own model requests hold whatever Claude Code's tools read, and for `evolve` and `evolve_jev` arms, whose mutate requests' feedback is not recorded; such a run's gate is `incomplete`, never `pass`; (3) the policy is not in the journal, so `approval_or_prerequisite_skipped` checks that at least one approval came between a `needs_approval` decision and its effect, not the policy's count; (4) `credential_observed` matches credential shapes (`syberlabs._patterns.SECRET`, at a word boundary) and the values of the host's credential variables present when the run ended, not every possible secret, and full check outputs and provider stderr are not retained, so only their recorded tails are scanned; (5) the hidden copy is hashed when staged, after the proposal (one hash after every check run inside it, the repair loop's evaluations included), after `Thread.check` (which runs the checks again only when the chosen candidate's result is missing or stale, so it usually follows a check that ran nothing) and before acceptance; the gate's `hidden_tampered` rule is `not_computable` for a bundle missing a hash the runner always writes; so a rewrite undone between two hashes, or inside one proposal's evaluations, is not detected, and candidate code inside a check can still read the copy and print it into its output, which `tails` sends to the next request; only a sandbox per check closes this (research register N4); (6) `hidden_feedback` filters the `repair` and `claude-code` feedback path only: an evolve arm builds its mutate requests in `syberlabs/evolve.py`, which no filter reaches, so a spec may use evolve arms under `tails` only; (7) under `none` the repair loop still stops on the first candidate that passes the hidden checks, so they still choose among up to K attempts on the host's side, and a call after a failure starts from the failed candidate's tree; (8) the eval's recorded spend can exceed `total_dollars` by one call's cost, and a call in flight when the runner is killed outright is in no journal and no ledger line; (9) the model API takes no sampling seed, so a run's seed indexes a repeat and is not a reproducible draw; (10) E5's earlier registration was deleted, because adding `hidden_feedback` moved its digest and `register` returns an existing registration of the same digest unchanged; E5 is registered again from a clean checkout of `main` at `90930320c6` (`git_dirty: false`) and is not run, and a run still needs a passing preflight on the commit it runs from; (11) the package reads `benchmarks/`, the adapters and `evals/` from a checkout, so the wheel's `syberlabs eval` needs one to run anything; (12) the design gaps the verification pinned as `expectedFailure` in `tests/test_eval_adversarial.py` stay open: a candidate that exits 0 at import passes every hidden check; under `tails` a candidate can print its hidden check into the next request (under `status` and `none` it cannot, and that test passes); a `claude-code` arm's model is a declaration the runner neither sets nor compares with the model that served; a `repo:` task's run repository holds the commits after its base; and a runner killed outright charges nothing (tested on POSIX only, where `SIGKILL` exists). 63. **The GitHits pilot's example pacing and license reading rest on GitHits' documentation, not on observed traffic.** (1) The host paces `example` calls to one per 40 s per host (`syberlabs/githits_pace.py`, called by the launcher entry); GitHits' limit is per organization, so two hosts, or an operator's own `syberlabs githits example` (not paced), share the organization's budget without seeing each other's slots. (2) A loopback base URL is not paced unless `SYBERLABS_GITHITS_PACE_SECONDS` asks: a loopback proxy that forwards to GitHits would go unpaced. (3) Calls queued behind several others wait one interval each, so a Bash tool timeout shorter than the queue ends a call before it is sent, and the host then flags it `githits_unrecorded`. (4) The time file is plain text a process of the same user can rewrite; it paces cooperating processes and is no security boundary. (5) The license reading admits an example's `References`/`Sources` links by their `(SPDX)` and passes an example with neither code (a fenced, indented or HTML block) nor a source; inline code in prose is not read as code, and the three documented shapes (`complete`, `degraded`, `no_example` in the pinned OpenAPI document) are the only evidence of what real responses look like until the first live example is recorded. (6) The engineering smoke pins Claude Code 2.1.312 exactly because the host and the pilot VM run it; that version's flags were not re-checked here. diff --git a/tests/test_eval_core.py b/tests/test_eval_core.py index 25311203..775ea8ef 100644 --- a/tests/test_eval_core.py +++ b/tests/test_eval_core.py @@ -285,17 +285,17 @@ def test_the_committed_tasks_are_what_the_importer_writes(self): self.assertEqual({w["id"]: w["digest"] for w in fresh}, {task_id: tasks.task_digest(EVALS / "tasks", task_id) for task_id in committed}) - def test_e5_is_a_valid_spec_awaiting_registration_on_clean_main(self): - # eval-score adds the required hidden_feedback field, so E5's digest moved; its stale registration is deleted - # (register returns an existing registration with the same digest unchanged) and the lead registers it again - # on a clean checkout of main. Until then run refuses it. + def test_e5_is_registered_from_clean_main_and_not_run(self): + # eval-score added the required hidden_feedback field, so E5's digest moved and its stale registration was + # deleted (register returns an existing registration with the same digest unchanged). E5 was registered again + # on a clean checkout of main (e5-register). registered() refuses a spec, task or arm that moved since. path = EVALS / "e5-single-vs-repair" / "spec.json" - doc = spec.load(path) + doc, registration = spec.registered(path) self.assertEqual(doc["hidden_feedback"], "tails", "E5 matches E3: hidden checks' tails reach repair") - self.assertFalse((EVALS / "e5-single-vs-repair" / "registration.json").exists()) - with self.assertRaises(Rejected) as caught: - spec.registered(path) - self.assertEqual(caught.exception.code, "spec_unregistered") + self.assertEqual(registration["spec_digest"], spec.spec_digest(doc)) + self.assertEqual(registration["task_set_digest"], doc["tasks"]["set_digest"]) + self.assertIs(registration["git_dirty"], False, "registered from a clean tree") + self.assertRegex(registration["git_commit"], r"^[0-9a-f]{40}$") self.assertEqual([a["name"] for a in doc["arms"]], ["single", "repair"]) self.assertEqual({a["model"] for a in doc["arms"]}, {"claude-sonnet-5-5"}) self.assertEqual({a["effort"] for a in doc["arms"]}, {"medium"}) @@ -306,6 +306,7 @@ def test_e5_is_a_valid_spec_awaiting_registration_on_clean_main(self): doc["decision_rule"]["operator"]), ("repair", "single", ">=")) self.assertEqual(len(runner.plan(doc)), 48) self.assertFalse((EVALS / "e5-single-vs-repair" / "runs").exists(), "E5 is not run") + self.assertFalse((EVALS / "e5-single-vs-repair" / "spend.jsonl").exists(), "no E5 run was attempted") class Arms(unittest.TestCase): @@ -453,10 +454,38 @@ def cli(self, *args): code = main(list(args)) return code, out.getvalue() - def test_score_and_report_refuse_an_unregistered_eval(self): - for name in ("score", "report"): - code, out = self.cli("eval", name, "e5-single-vs-repair") + def cli_err(self, *args): + from syberlabs.cli import main + + out, err = io.StringIO(), io.StringIO() + with redirect_stdout(out), redirect_stderr(err): + code = main(list(args)) + return code, out.getvalue(), err.getvalue() + + def test_score_and_report_of_e5_unregistered_and_registered_with_no_runs(self): + # Every score and report runs on a copy of evals/: score writes index.json beside the spec, and no test may + # write into the checkout (preflight records whether it differs from its commit). + folder_e5 = EVALS / "e5-single-vs-repair" + before = sorted(p.name for p in folder_e5.iterdir()) + artifacts = ("runs", "spend.jsonl", "index.json", "preflight.json", "report*") + with tempfile.TemporaryDirectory() as folder: + evals = Path(folder) / "evals" + shutil.copytree(EVALS, evals, ignore=shutil.ignore_patterns(*artifacts, "registration.json")) + for name in ("score", "report"): + code, out, err = self.cli_err("eval", "--evals", str(evals), name, "e5-single-vs-repair") + self.assertEqual(code, 2, out) + self.assertIn("spec_unregistered", err) + with tempfile.TemporaryDirectory() as folder: + evals = Path(folder) / "evals" + shutil.copytree(EVALS, evals, ignore=shutil.ignore_patterns(*artifacts)) + code, out, err = self.cli_err("eval", "--evals", str(evals), "score", "e5-single-vs-repair", "--json") + self.assertEqual(code, 0, err) + self.assertEqual(json.loads(out)["runs"], {}, "a registered eval with no runs scores none") + code, out, err = self.cli_err("eval", "--evals", str(evals), "report", "e5-single-vs-repair") self.assertEqual(code, 2, out) + self.assertIn("runs_missing", err) + self.assertIn("48 of 48", err) + self.assertEqual(sorted(p.name for p in folder_e5.iterdir()), before, "nothing written into the checkout") def test_tasks_list_and_a_dry_run_of_e5(self): code, out = self.cli("eval", "tasks", "list", "--json") @@ -464,10 +493,9 @@ def test_tasks_list_and_a_dry_run_of_e5(self): found = json.loads(out) doc = json.loads((EVALS / "e5-single-vs-repair" / "spec.json").read_text(encoding="utf-8")) self.assertEqual(found["set_digest"], doc["tasks"]["set_digest"]) - with tempfile.TemporaryDirectory() as folder: # registered in a copy: the committed E5 waits for clean main + with tempfile.TemporaryDirectory() as folder: # the committed registration, on a copy: nothing written here evals = Path(folder) / "evals" - shutil.copytree(EVALS, evals, ignore=shutil.ignore_patterns("runs", "registration.json")) - spec.register(evals / "e5-single-vs-repair" / "spec.json") + shutil.copytree(EVALS, evals, ignore=shutil.ignore_patterns("runs", "spend.jsonl", "index.json")) code, out = self.cli("eval", "--evals", str(evals), "run", "e5-single-vs-repair", "--dry", "--json") found = json.loads(out) self.assertEqual((code, found["planned"], found["dry"]), (0, 48, True))