diff --git a/docs/EVAL.md b/docs/EVAL.md index 5af01bd1..62ac6217 100644 --- a/docs/EVAL.md +++ b/docs/EVAL.md @@ -1,10 +1,10 @@ # SyberEval: pre-registered evaluations through the real Build Thread -SyberEval measures SyberWork's agents as governed workflows. It asks whether a provider configuration, run through the real Build Thread, produces admitted changes that are correct, safe and cheap. This page covers the core: specs, tasks, arms, the runner, its evidence bundles, and the preflight gate (package `syberlabs/eval/`, packet `eval-core`). Scoring, the journal safety gate and the report come in the next packet, `eval-score`. The agent-facing safety tasks come in `eval-safety-fixtures` (`spec/GAPS.md`, 60). The E3 harness (`benchmarks/model_vs_patch.py`, [MODEL_COMPARISON.md](MODEL_COMPARISON.md)) is still the tool for quick research. SyberEval is the instrument whose numbers are published. +SyberEval measures SyberWork's agents as governed workflows. It asks whether a provider configuration, run through the real Build Thread, produces admitted changes that are correct, safe and cheap. This page covers specs, tasks, arms, the runner and its evidence bundles, the preflight gate, how much of a hidden check reaches the proposer (`hidden_feedback`), the spend ledger, the safety gate, the scorer and the report (package `syberlabs/eval/`). The agent-facing safety tasks are not here yet (`spec/GAPS.md`, 62). The E3 harness (`benchmarks/model_vs_patch.py`, [MODEL_COMPARISON.md](MODEL_COMPARISON.md)) is still the tool for quick research. SyberEval is the instrument whose numbers are published. ## Principles -1. **Three outcomes, never blended.** Safety is a hard gate. Task quality is host-verified acceptance on hidden checks, with partial results labelled partial. Efficiency is the nine metrics of [METRICS.md](METRICS.md), and unknown stays unknown. `eval-core` records all three. `eval-score` computes them. +1. **Three outcomes, never blended.** Safety is a hard gate. Task quality is host-verified acceptance on hidden checks, with partial results labelled partial. Efficiency is the nine metrics of [METRICS.md](METRICS.md), and unknown stays unknown. The runner records all three; `eval score` computes them and `eval report` reports them side by side. 2. **Pre-registered.** The question, arms, task set (by digest), metrics, budget, seeds, run order, exclusions, decision rule and reporting commitments are frozen in a spec whose digest every bundle cites. `run` refuses an unregistered or modified spec, and a task that changed since registration. 3. **Real workflow.** Every run is a Build Thread: a contract, host checks on the exact candidate tree, admission, a journal and a certificate. The admission decision is part of the result, and so are refused and blocked proposals. 4. **Hidden acceptance.** A task's acceptance checks are held by the host outside the run's repository. They are not in the agent's worktree, its requests or its candidates (see Hidden checks for what that does and does not cover). @@ -20,10 +20,14 @@ evals/ /spec.json the spec (schema syberlabs.eval/spec.v1) /registration.json written once by register /preflight.json the last preflight, for one commit + /spend.jsonl the spend ledger: one line per run attempt, appended, never rewritten /runs//bundle.json one per (task, arm, seed), with the files beside it + /runs//score.json written by score, beside the bundle and never part of it + /index.json written by score: every run's bundle and score digests + /report.md, report.json written by report (report.partial.* under --partial) ``` -A run id is `..s`. The package runs from a SyberWork checkout, not from the wheel: it reads `benchmarks/fixtures.py` and the adapters, and `pyproject.toml` packages `syberlabs` without its subpackages. The `syberlabs` command in a wheel install has no `eval` subcommand. +A run id is `..s`. The wheel carries `syberlabs.eval`, so an installed `syberlabs` has the `eval` subcommand, but an eval runs from a SyberWork checkout: it reads `benchmarks/fixtures.py`, the adapters and `evals/` from there. ## Commands @@ -32,11 +36,12 @@ syberlabs eval tasks import-fixtures [--fixtures NAME ...] [--force] # default: syberlabs eval tasks list [ID ...] [--json] # each task's digest and the set digest of those listed syberlabs eval register evals//spec.json # validate, check the set digest, write registration.json syberlabs eval preflight # the deterministic suites, recorded for this commit -syberlabs eval run [--only TASK,ARM,SEED] [--dry] [--work DIR] [--keep-work] [--seed-env NAME] [--json] -syberlabs eval score | report # exit 2: "not yet: eval-score packet" +syberlabs eval run [--only TASK,ARM,SEED] [--dry] [--work DIR] [--keep-work] [--hidden-root DIR] [--seed-env NAME] [--json] +syberlabs eval score [--json] # the gate, quality and efficiency of every run; index.json +syberlabs eval report [--partial] [--json] # the paired analysis and the decision; report.md and report.json ``` -`--evals DIR` (before the subcommand) names another evals directory. In `--only`, an empty part or `*` matches anything. `--dry` prints the registered order with each run's state, `done` or `pending`, and runs nothing. It works before a preflight. `run` exits 1 when the eval stopped on its cap. +`--evals DIR` (before the subcommand) names another evals directory. In `--only`, an empty part or `*` matches anything. `--dry` prints the registered order with each run's state, `done` or `pending`, and runs nothing. It works before a preflight. `run` exits 1 when the eval stopped on its cap. `--hidden-root` is the host root under which each run's kit home and hidden copy get random names (default `/syberlabs-eval-host`). ## The spec @@ -54,6 +59,9 @@ syberlabs eval score | report # exit 2: "not yet: eval-score | `exclusions` | `[{id, rule}]` | | `decision_rule` | `{metric, comparison: paired_difference, treatment, baseline, operator, threshold, unit: task, aggregation: mean_over_seeds, test}`; the metric is the primary one, and treatment and baseline are two different arms of the spec | | `reporting` | what is reported whatever the outcome | +| `hidden_feedback` | `tails`, `status` or `none`: how much of a hidden check's result reaches the proposer (below). Required, and part of the digest | + +Every number must be finite (`NaN` passes every comparison, so a `NaN` cap would never stop). The cross-field rules (`spec.check`) also refuse two cells that share a run id (`..s`, where task ids and arm names may hold dots), two `githits` arms whose GitHits agent labels are the same (`arms.githits_arm` rewrites every character outside `[A-Za-z0-9_-]` and cuts the name to 16 characters), and an evolve arm under any `hidden_feedback` but `tails`. Spec, task and bundle files are read strictly (`spec.read_json`): a key repeated in one object, `NaN` and `Infinity` are refused, so what a reader sees in the file is what the digest freezes. `register` writes `registration.json`: `{spec_digest, task_digests, task_set_digest, arm_digests, registered_at, git_commit, git_dirty}`. The spec digest is `syberlabs.canonical.digest` of the parsed spec, so whitespace and line endings do not move it and any field does. Registering the same spec again returns the registration. Registering a changed spec is refused (`already_registered`), because a changed question is a new eval id. Before it runs anything, `run` refuses `spec_unregistered`, `spec_modified` (the spec, or an arm, no longer digests to its registration) and `task_modified`. @@ -65,7 +73,7 @@ syberlabs eval score | report # exit 2: "not yet: eval-score - `objective` and `scope` (`paths`, `exclude`, `max_files`, `max_diff_bytes`: the run contract's mutable scope and the paths a provider may read). - `checks`: visible checks, which run like any contract check. They may be empty, and they may not name `{hidden}`. - `hidden.checks`: the acceptance checks. In their argv, `{python}` is the host's interpreter and `{hidden}` is the absolute path of the host's copy of `hidden/`. -- `safety` (`expectations`, `canaries`: strings planted in protected data, which `eval-score` searches outbound records for), `provenance`, `contamination` (`status`: `unknown`, `private`, `public_repository` or `public_benchmark`, plus a `note`), and `labels`. +- `safety` (`expectations`; `canaries`: strings planted in protected data, which the safety gate searches outbound records for; optionally `forbidden_actions`: contract actions a planner task forbids, which the gate fails a run for admitting), `provenance`, `contamination` (`status`: `unknown`, `private`, `public_repository` or `public_benchmark`, plus a `note`), and `labels`. The task digest is `canonical.digest({"task": , "hidden": {: }})` over every file under `hidden/`. A hidden file that is UTF-8 without NUL bytes is text, and its CRLF becomes LF before it is digested and staged, so a Windows checkout and a POSIX one agree. Any other file is used byte for byte. @@ -73,21 +81,40 @@ The task digest is `canonical.digest({"task": , "hidden": {