diff --git a/docs/EVAL.md b/docs/EVAL.md index 287bdb4..dfc848e 100644 --- a/docs/EVAL.md +++ b/docs/EVAL.md @@ -199,7 +199,7 @@ It writes `report.md` and `report.json` (`report.partial.*` under `--partial`, w ## Preflight -`eval preflight ` runs `scripts/run_shards.py`, `python -m conformance.run`, `benchmarks/run_scenarios.py` and `python -m unittest discover -s tests -p "test_eval_*.py"` from the checkout, with `PYTHONPATH` set to it and with the certificate seed and GitHub tokens removed from the environment. It writes `preflight.json` with the commit, whether the checkout differs from it outside `evals/`, and each command's exit code, seconds, output digest and output tail. `run` refuses `preflight_missing`, `preflight_failed`, `preflight_stale` (the record names another commit) and `preflight_dirty` (the checkout differs from its commit outside `evals/`, now or when the preflight ran). Untracked files under `evals/` are not dirt, because that is where runs write. +`eval preflight ` runs `scripts/run_shards.py`, `python -m conformance.run`, `benchmarks/run_scenarios.py` and `python -m unittest discover -s tests -p "test_eval_*.py"` from the checkout, with `PYTHONPATH` set to it and with the certificate seed and GitHub tokens removed from the environment. It writes `preflight.json` with the commit, whether the checkout differs from it outside `evals/`, and each command's exit code, seconds, output digest and output tail. `run` refuses `registration_dirty` (the registration's `git_dirty` is not `false` or its `git_commit` is not a commit id, so its digests were computed by code no commit holds; delete `registration.json` before any run and register again from a clean checkout), `preflight_missing`, `preflight_failed`, `preflight_stale` (the record names another commit) and `preflight_dirty` (the checkout differs from its commit outside `evals/`, now or when the preflight ran). A dry run reports the first refusal in its `preflight` field; `score` and `report` do not ask about the registration's origin. Untracked files under `evals/` are not dirt, because that is where runs write. ## E5 and E6 diff --git a/docs/EVOLUTION.md b/docs/EVOLUTION.md index d0e0d9d..7d9212e 100644 --- a/docs/EVOLUTION.md +++ b/docs/EVOLUTION.md @@ -157,7 +157,7 @@ It takes the highest score (ties in the order evaluate, crossover, mutate, migra Simulated, and nothing about a model: the profiles were chosen so the policies can be told apart, the in-memory world is not the Build Thread, and the fixture has three independent rules that favour recombination by construction. On the real Build Thread the module checks, over two seeds with a priced contract and a command mutator that reports usage, that the host's attributions reach `OperatorStats.dollars` kind by kind, that every Jev decision entry carries the audit fields, that the signal stays within the bound, and that Jev spends fewer host-observed dollars than Empirical. The recorded-equivalence tests for `FixedPolicy` are unchanged and still pass. -**Decision rule.** From the packet's measurement section: "The decision rule for Jev: it is kept as the provider's default if, on the v2 fixtures, `evolve_jev` solves at least as many as `evolve` with at most half the dollars, over the three repeats; otherwise it stays available and not default." That is: Jev becomes the default only if it solves at least as many v2 fixtures as `fixed` with at most half the dollars over three repeats. The E3 v2 pilot of 2026-10-21 (Claude Sonnet 5.5, medium, K = 6, one repeat; `docs/MODEL_COMPARISON.md`) solved all four v2 fixtures single-shot, 16 of 16 runs in 28 calls for about $0.24, with `evolve` costing four to six times `single` on every fixture, so by the plan's own rule the v2 set is uninformative for that model and the paid Jev-against-fixed comparison will not run on v2 at Sonnet 5.5 as this packet's spec assumed. The simulated comparison above is the evidence in this packet; the paid comparison is deferred to a fixture set on which `single` fails (Haiku 5.5 on v2, or a v3 set), a choice the owner makes. Until that run is made and reported, `fixed` is the default and the table above is not evidence for the rule. +**Decision rule.** From the packet's measurement section: "The decision rule for Jev: it is kept as the provider's default if, on the v2 fixtures, `evolve_jev` solves at least as many as `evolve` with at most half the dollars, over the three repeats; otherwise it stays available and not default." That is: Jev becomes the default only if it solves at least as many v2 fixtures as `fixed` with at most half the dollars over three repeats. The E3 v2 pilot of 2026-10-21 (Claude Sonnet 5.5, medium, K = 6, one repeat; `docs/MODEL_COMPARISON.md`) solved all four v2 fixtures single-shot, 16 of 16 runs in 28 calls for about $0.24, with `evolve` costing four to six times `single` on every fixture, so by the plan's own rule the v2 set is uninformative for that model and the paid Jev-against-fixed comparison will not run on v2 at Sonnet 5.5 as this packet's spec assumed. The simulated comparison above is the evidence in this packet; the paid comparison is deferred to a fixture set on which `single` fails (Haiku 5.5 on v2, or a v3 set), a choice the owner makes. Until that run is made and reported, `fixed` is the default and the table above is not evidence for the rule. That run was made on the v3 fixtures on 2026-10-28 (results under "Measurement plan" below): the rule was not met, and `fixed` stays the default. ### TypeSafe's Jev API as a policy @@ -186,9 +186,9 @@ Simulated, and nothing about a model: the profiles were chosen so the policies c | Price | $0.042 per million input tokens, output free (docs.typesafe.ai/models, read 2026-10-28): about $0.00003 a decision; the 28 calls cost about $0.0006 | | Data terms | docs.typesafe.ai/legal points to a Data Processing Agreement and a Privacy Policy that commits not to train on user data; retention periods are not stated on that page; zero data retention is offered to enterprise customers | -What the calls say about the choice, and no more: on these synthetic states Jev chose `mutate` or `evaluate` every time; it gave `crossover` 0.02 to 0.15 on the 19 three- and four-option answers and 0.37 on the one two-option answer (`crossover` against `mutate`, `tests/fixtures/jev_api/choice_two_options.json`), where the local policy prices a crossover (no model call) far below a mutation. Whether that is better or worse on a real fixture is what the measurement below is for. +What the calls say about the choice, and no more: on these synthetic states Jev chose `mutate` or `evaluate` every time; it gave `crossover` 0.02 to 0.15 on the 19 three- and four-option answers and 0.37 on the one two-option answer (`crossover` against `mutate`, `tests/fixtures/jev_api/choice_two_options.json`), where the local policy prices a crossover (no model call) far below a mutation. Whether that is better or worse on a real fixture is what the measurement below was for; on the v3 fixtures Jev chose `mutate` on every one of the 21 decisions it was asked, and the local policy would have chosen `crossover` on each. -**Measurement plan (pre-registered 2026-10-28, re-targeted to v3 the same day; not run).** It continues the `jev-policy` measurement and needs the owner's go through the lead, because it spends on Anthropic's API. +**Measurement plan (pre-registered 2026-10-28, re-targeted to v3 the same day; run 2026-10-28 on `main` at `5ebbbce`, results after the plan).** It continues the `jev-policy` measurement and needs the owner's go through the lead, because it spends on Anthropic's API. - *Set*: the four **v3** fixtures (`gaps_renumber`, `journal_chain`, `worktree_order`, `verify_layers`). v2 is uninformative at this model and at Haiku 5.5, and the paid Jev-against-fixed comparison goes to v3 by the owner's decision (`docs/MODEL_COMPARISON.md`, "Where that run goes"); v4 is the fallback. - *Arms*: `single`, `evolve` (`FixedPolicy` revision 2), `evolve_jev` (local `JevPolicy`) and `evolve_jevapi` (`TypeSafeJevPolicy`, `jev-1.13.0`), through `benchmarks/model_vs_patch.py` and the same Build Thread; K = 6 model calls per run; Claude Sonnet 5.5 at medium effort (`SYBERLABS_MODEL=claude-sonnet-5-5`, `SYBERLABS_EFFORT=medium`), with `ANTHROPIC_API_KEY` and `TYPESAFE_API_KEY` in that shell only. @@ -205,6 +205,26 @@ What the calls say about the choice, and no more: on these synthetic states Jev - Budget line for the owner: **$5.40 Anthropic hard ceiling** (expected $1.90) and **$0.15 Jev**. - *Not expressible as a SyberEval spec yet*: `syberlabs/eval/spec.py` closes `ARM_KINDS` to `single`, `repair`, `evolve`, `evolve_jev` and `claude-code`, so neither `evolve_jevapi` nor a policy choice can be registered under `evals/`; this plan is the pre-registration (`spec/GAPS.md` 61). +**Results (run 2026-10-28 on `main` at `5ebbbce`; neither rule met; no default changes).** The owner approved the run on 2026-10-28 with the ceilings above. It ran from a fresh clone of `main` at `5ebbbce6e0` (#149), the commit that registered this plan, so the plan text it read is the registered text; the section's SHA-256 at that commit, from its first line to the next heading, is `5e8f98a4b6e4421b57467fadd8cfe85b91d134b3fd780920aed06a79cfb363c1`. Source: the officer workspace's `reports/2026-10-28-jev-v3-measurement.md`, with the harness output, per-run journals, Jev decision records and guard logs under `reports/receipts/2026-10-28-jev-v3-measurement/`. Claude Sonnet 5.5 at medium effort served all 64 rows; TypeSafe served `jev-1.13.0` on all 27 answered calls. One substitution in form: the adapter was named by its absolute path in the clone (`C:/Python312/python.exe /adapters/anthropic_adapter.py`), because the harness runs the adapter from the fixture worktree (`docs/MODEL_COMPARISON.md`); arms, fixtures, K, repeats and model are as registered. + +- *Pilot* (`--repeats 1`, seed 1): informative. `single` solved 1 of 4, six `evolve*` runs needed more than one candidate, and Jev was asked 6 decisions, each offering `{crossover, mutate}`. The measurement ran; the pilot is counted in the cost and not read by the rules. +- *Measurement* (`--repeats 3`, seeds 1 to 3), per arm over the four fixtures and three repeats: + +| Arm | Solved | Anthropic calls | Anthropic $ | Jev calls | Jev $ | Total $ | +| --- | --- | --- | --- | --- | --- | --- | +| `single` | 4/12 | 12 | 0.119276 | | | 0.119276 | +| `evolve` (`fixed/2`) | 11/12 | 40 | 0.441426 | | | 0.441426 | +| `evolve_jev` (`jev/1`) | 10/12 | 42 | 0.483434 | | | 0.483434 | +| `evolve_jevapi` (`jev_api/1`) | 9/12 | 46 | 0.525576 | 21 | 0.000804468 | 0.526380 | + +Per fixture, solved (`single` / `evolve` / `evolve_jev` / `evolve_jevapi`): `gaps_renumber` 1 / 3 / 2 / 2, `journal_chain` 3 / 3 / 3 / 3, `worktree_order` 0 / 2 / 2 / 1, `verify_layers` 0 / 3 / 3 / 3. + +- *Rule carried over* (`evolve_jev` against `evolve`): **not met**. 10 solves against 11, and $0.483434 against a bar of half of $0.441426 ($0.220713), a ratio of 1.095. `evolve_jev` stays available and not default; `fixed` stays the provider's default. +- *New rule* (`evolve_jevapi` against `evolve_jev`): **measured, and not met**. Jev was asked 21 decisions, 0 fell back and no run switched to local, so the comparison counts as measured; it agreed with `local_action` on 0 of 21, so the "no difference may be claimed" clause does not apply. 9 solves against 10, and $0.526380 (Anthropic plus Jev) against $0.483434. TypeSafe's Jev does not replace the local heuristic; `repair` stays the default proposer path. +- *Jev's choices*: every asked decision (21, all on `worktree_order` and `verify_layers`) offered exactly `{crossover, mutate}`. Jev chose `mutate` on all 21, and the local policy would have chosen `crossover` on all 21. Its probability for `crossover` had a median of 0.37 (0.24 to 0.49); median latency 188.8 ms; 19,154 input tokens. On `gaps_renumber` and `journal_chain` no decision offered two actions, so the two failed `gaps_renumber` runs (seed 1, in both Jev arms) did not depend on the chooser. +- *Spend*, pilot and measurement: **$1.987102** Anthropic (178 calls, the harness's pricing of the adapter's reported usage; `host_dollars` equals `dollars` on every row) and **$0.001034670** Jev (27 calls, input tokens at $0.042 per million), 36.8% and 0.69% of the ceilings. Neither is an invoice. +- *Claim ceiling*: one pre-registered run at small n (12 runs per arm per rule) of four arms on the four v3 fixtures, at Claude Sonnet 5.5 medium, K = 6 and `jev-1.13.0`. The deciding gaps are one solve and 9 to 10% in dollars, within run-to-run variation for a model that takes no seed. "Not met" says the pre-registered bars were not cleared on this sample, not that `evolve_jev` or Jev is worse; it says nothing about another fixture set, model, effort, K or Jev version. + ### Measured on a toy fixture `examples/evolve.py` is a pricing module with three independently checked rules and a stand-in mutator. `benchmarks/evolve_baseline.py` compares the provider with and without crossover on 30 seeds, with the same budget. The result is in `benchmarks/results/evolve-baseline.txt`: diff --git a/docs/MODEL_COMPARISON.md b/docs/MODEL_COMPARISON.md index 2f58faf..8fc15a8 100644 --- a/docs/MODEL_COMPARISON.md +++ b/docs/MODEL_COMPARISON.md @@ -169,7 +169,7 @@ What it shows about the **harness and the policies on these profiles**: on `muta **Measurement procedure and decision rule**, copied from the packet spec (`jev-policy`, "Measurement"): "Run the three `evolve*` arms plus `single` on the v2 fixtures (step 2) at K = 6, one repeat first, Sonnet 5.5 medium. The decision rule for Jev: it is kept as the provider's default if, on the v2 fixtures, `evolve_jev` solves at least as many as `evolve` with at most half the dollars, over the three repeats; otherwise it stays available and not default. Budget line: **$15** of the $50 cap (see the allocation in the review). The run is a named human's act with the key in that shell, or the orchestrator's under the owner's standing "ready for experimentation", stated in the results packet either way." -**Where that run goes.** The v2 set is uninformative at every tier measured on 2026-10-21: `single` solved all four v2 fixtures at Claude Sonnet 5.5 (16 of 16 runs, medium, K = 6, one repeat) and at Claude Haiku 5.5 (8 of 8, two repeats, about $0.003), so a comparison on v2 could only add single-shot failures on tasks every arm solves at its first candidate. The paid Jev-against-fixed comparison therefore goes to a set on which `single` fails, the v3 fixtures (a parallel packet), by the owner's decision; until it is run and reported, `fixed` stays the provider's default and the simulated table above is not evidence for the rule. +**Where that run goes.** The v2 set is uninformative at every tier measured on 2026-10-21: `single` solved all four v2 fixtures at Claude Sonnet 5.5 (16 of 16 runs, medium, K = 6, one repeat) and at Claude Haiku 5.5 (8 of 8, two repeats, about $0.003), so a comparison on v2 could only add single-shot failures on tasks every arm solves at its first candidate. The paid Jev-against-fixed comparison therefore goes to a set on which `single` fails, the v3 fixtures (a parallel packet), by the owner's decision; until it is run and reported, `fixed` stays the provider's default and the simulated table above is not evidence for the rule. It was run on 2026-10-28 (the note of that date below): the rule was not met, and `fixed` stays the default. ## Note of 2026-10-22: every `evolve*` row so far was taken with an empty scope in the mutate request @@ -199,6 +199,20 @@ The wrong readings the oracle holds: `stack_landing` writes `*2026-10-22, narrow **What is measured, and what is not.** With the simulated adapter on 2026-10-22 (K = 6, one repeat, arms `single`, `repair`, `evolve`), 0 of 12 runs solved: `single` 1 call and 1 evaluation on each fixture, `repair` 6 calls and 3 evaluations (median), `evolve` 6 calls and 2 evaluations (the stand-in knows only line swaps). `tests/test_model_comparison.py` (`FixturesV4`) re-derives the pins from the kit (`dsse.pae`, the certificate constants, `DEFAULT_TARGET_REF`, `certificate.summary`, `checks.SETUP_NAMES` and `EMPTY_DIGEST`, `PROMOTION_RANK`, `dashboard.TONE`, `inspector.INCOMPLETE_REMEDY`). Nothing here says what a model does. "Parts per dollar" in the prediction needs a per-part count the harness summary does not print (the per-part states are in each exported journal's `candidate_evaluated` events); that column belongs to the packet that owns `benchmarks/model_vs_patch.py` (`spec/GAPS.md`, 47). The `evolve_jev` arm is on #68's branch, so the v4 measurement runs on the stack, not on this branch alone. Decision of 2026-10-23 (`repair-proposer`): on the v3 and v4 sets (Claude Sonnet 5.5, K = 6 and K = 12; the v4 note and its rows are in the `e3-v4-multipart` packet, #75, a parallel stack of 2026-10-22, with the receipts in the officer workspace) `repair` beat every evolve arm on solves and on parts per dollar at every budget tested, and beat `single` on solves; the prediction that evolution overtakes repair at a larger budget was falsified at K = 12 (42 against 65 parts per dollar, repair ahead). The owner's decision: the default proposer path is repair (`syberlabs.providers.RepairProvider`, `syberlabs propose --repair`), and the evolve arms stay as research. No row in this file is rewritten by that decision. +## Note of 2026-10-28: the pre-registered Jev measurement on v3, neither rule met + +The measurement plan in `docs/EVOLUTION.md` (pre-registered 2026-10-28) ran the same day on `main` at `5ebbbce` (#149): arms `single`, `evolve` (`fixed/2`), `evolve_jev` (local `JevPolicy`) and `evolve_jevapi` (TypeSafe's Jev, `jev-1.13.0`), the four v3 fixtures, Claude Sonnet 5.5 at medium effort, K = 6, a one-repeat pilot (informative) and then three repeats, which the rules read. Source: the officer workspace's `reports/2026-10-28-jev-v3-measurement.md`, with the harness output, per-run journals, Jev decision records and guard logs under `reports/receipts/2026-10-28-jev-v3-measurement/`. The adapter was named by its absolute path in the clone, as the first finding above requires; nothing else differs from the plan's command. + +``` +arm policy solved calls anthropic$ jev calls jev$ total$ +single - 4/12 12 0.119276 - - 0.119276 +evolve fixed/2 11/12 40 0.441426 - - 0.441426 +evolve_jev jev/1 10/12 42 0.483434 - - 0.483434 +evolve_jevapi jev_api/1 9/12 46 0.525576 21 0.000804468 0.526380 +``` + +Three repeats, seeds 1 to 3, summed over `gaps_renumber`, `journal_chain`, `worktree_order` and `verify_layers`; no row has a `provider_error`. The rule carried over from `jev-policy` (`evolve_jev` solves at least as many as `evolve` with at most half the dollars) is **not met**: 10 against 11, and 109.5% of `evolve`'s dollars. The new rule (`evolve_jevapi` solves at least as many as `evolve_jev` with at most its dollars, Anthropic plus Jev) is **measured and not met**: 0 fallbacks on 21 asked decisions and no switch to local, agreement with `local_action` 0 of 21, 9 against 10 solves, $0.526380 against $0.483434. TypeSafe's Jev chose `mutate` on all 21 decisions it was asked, where the local policy would have chosen `crossover`. Defaults are unchanged: `fixed` stays the evolutionary provider's default, `evolve_jev` stays available and not default, the local heuristic stays the chooser behind it, and `repair` stays the default proposer path. Spend, pilot included: $1.987102 Anthropic and $0.001034670 Jev, the harness's pricing and TypeSafe's published input rate, not invoices. At 12 runs per arm the deciding gaps (one solve, 9 to 10% in dollars) are within run-to-run variation; the rules were applied as written, and "not met" says only that the pre-registered bars were not cleared on this sample. The full result, per fixture and per decision, is in `docs/EVOLUTION.md` under the plan. + ## What would count as evidence A real-model run on these fixtures would show only whether the arms differ on toy tasks. The roadmap's bar is a real repository with several productive alternatives, a baseline, a failure case, and a decision that changed for a measured reason. Until that run exists, the claim in `docs/ROADMAP_STATUS.md` stays at what the pilot supports: **pilot measured 2026-10-21; uninformative by rule; harder fixtures next**. diff --git a/spec/GAPS.md b/spec/GAPS.md index 28ce3d0..a17969a 100644 --- a/spec/GAPS.md +++ b/spec/GAPS.md @@ -118,7 +118,7 @@ The schemas describe SyberWork as it behaves. They are not enforced at runtime e 59. **The GitHits API client is wired into Claude Code as arm B's only lookup route; its contamination and license checks are filters on what returns, and its lookups are counted from the client's own records.** `githits-client` (2026-10-26) adds `syberlabs/githits.py` and `syberlabs githits search|example|read|list|grep` (`docs/GITHITS.md`): a standard-library client for five read-only GitHits REST operations, with a SyberLabs exclusion in three layers (a target refused before any request, a result whose identity is under an excluded owner dropped and counted, an example or read body that cites one dropped whole as `contamination_blocked`), a permissive-only license policy (`license_mode: strict` on examples; client-side, a result whose SPDX license is not on the allow list dropped and counted as `results_dropped_license`), and an HMAC-signed record per call (`query_guard.sign`, now shared). `githits-wiring` (2026-10-26) closes the first two deferred items: (1) `ClaudeCodeProvider(githits=True)` (`--githits`) writes a per-call, read-only launcher outside the worktree that runs the host checkout's client (`syberlabs/githits_cli_entry.py` under `python -I`), allows `Bash(:*)` and nothing else that runs code, matches `Bash` in the query guard (only the launcher with plain arguments passes, and its arguments are judged by the guard's rules), and journals one `lookup` per authentic client record (tool `githits.`), `signal.tool_usage.githits` being their number and `githits_lookups` reading those facts; (2) GitHits' shipped skills (`githits-com/githits-cli`, Apache-2.0) are adapted into `skills/githits/SKILL.md` with their license and notice, and appended to arm B's system prompt (`signal.harness.skill`). The same packet fixes #129's three low findings (nested SPDX expressions, the token scrubbed before truncation and out of `X-Request-ID`, `SYBERLABS_GITHITS_EXCLUDE` adding to `SyberLabs` instead of replacing it) and adds `syberlabs githits doctor` (one `GET /v1/languages`). `githits-pilot-hardening` (2026-10-26) narrows the launcher's entry to the five lookups (`doctor` and `--license-mode` stay the operator's, `example` always sends `strict`, and a `githits.doctor` fact is never counted in `githits_lookups`), records the skill's digest with CRLF normalized to LF (`.gitattributes` checks `skills/**` out with LF), tests that `SYBERLABS_GITHITS_EXCLUDE_REPLACE` reaches neither arm, and, under `--allow-code-tools-with-mcp`, matches `PowerShell` in the query guard and judges it by the launcher-only rule. Deferred: (3) the other operations of the document are not covered: `search_status` (polling a `search_ref` when a search returns incomplete; the client prints the ref and stops), `research` and the rest of `/v1/experimental/*`, package intelligence, `code_diff`, `resolve_target`, `feedback`; (4) no retry or back-off on `429`/`503`: `Retry-After` is not read by the lookup operations (only `doctor` reports it) and the call fails. Not solved: (5) layer (c) is detection, not prevention: an example can blend code from an excluded repository without citing it, and a mention spelled outside the listed host patterns passes; for the pilot, contamination is resolved on SyberLabs' side (unpublished work items, both arms run locally, a pinned base) with the client as the backstop, and an index-side exclusion by GitHits is optional; (6) a package target whose name does not name the owner reaches GitHits, and only its served repository, after the call, can drop it; (7) the license check depends on license data: in contract 0.1.0 only examples carry licenses (in Markdown), so search, grep, list and read results pass unjudged, and what `strict` admits server-side is not documented; (8) the fixture pins the five operations' closed subset of the document (215,618 bytes), and `doctor`'s operation in a second fixture, not the whole 1,364,570-byte document, which is over the contract's change-size limit; the upstream SHA-256 is recorded in `docs/GITHITS.md` for comparison by hand; (9) the record key and the guard's key sit in Claude Code's environment, which the launcher's client inherits, so a verifying MAC shows a key holder wrote the line, not which process; (10) no call to the real API was made, and no real Claude Code ran: every behaviour is tested against a loopback fake that follows the pinned schemas and a stand-in `claude` whose Bash tool is emulated (a command no `Bash(:*)` rule names is denied, the launcher runs as `sh` runs it); whether Claude Code 2.1.309 matches `Bash(:*)` against a Windows path written with forward slashes, and runs the launcher through Git Bash, is for the first live call (`syberlabs githits doctor` first, then one guarded lookup); (11) a record cannot be tied to the tool use that ran it (the client has no tool use id), so the host reconciles counts: authentic records against launcher calls the guard passed; fewer records is `signal.githits.mismatch: githits_unrecorded` (a client that failed before recording, a usage error, a denied permission), more is `githits_unexpected_records`, every record of the call is then `unguarded` and the search stops `provider_guard_inactive`; (12) closed by the two-arm accounting repair: failed calls journal unreconciled records as `unguarded` or `githits.unrecorded` when none exist, and a `no_change` refusal with missing records journals a synthetic gap; the metric remains unknown without a candidate signal; (13) `X-GitHits-Session-ID` is `-`, not `/`, because GitHits accepts only `[A-Za-z0-9_-]{1,64}`; (14) closed by the two-arm accounting repair: when client-record facts exist, `githits_lookups` also includes the host's MCP lookup attempts once; mixed routes still raise `two_routes`; (15) arm A (no `--githits` and no `--mcp-config`) receives no `GITHITS_*` or `SYBERLABS_GITHITS_*` variable since `githits-pilot-hardening`, the token included: least privilege was chosen over the byte-identical environment `githits-wiring` kept, so arm A's environment differs from before this route by exactly those variables (its argv does not); an `--mcp-config` run keeps them, because a GitHits MCP server may read its token from that environment; (16) the skill rides in `--append-system-prompt`, so it is part of every arm B request's system prompt, and arm A's harness note has no skill. -60. **SyberEval runs and records; it does not yet score, gate on safety or report, and its hidden checks are hidden from the agent's worktree, requests and candidates, not from a process that can read the host's disk.** `eval-core` (2026-10-26) adds `syberlabs/eval/` and `syberlabs eval register|tasks|run|preflight` (`docs/EVAL.md`): pre-registered specs with frozen digests, tasks whose acceptance checks the host stages outside the run's repository and appends to the run contract as required `hidden.*` checks by absolute path, arms built into the kernel's own providers, a runner that takes every task x arm x seed through an unattended Build Thread in the registered order and writes a digested bundle per run, an eval-wide dollar cap, and a preflight gate; the eight v3 and v4 fixtures are imported as tasks and E5 is registered, not run. The contract schema, the event kinds, the golden traces and the kernel are unchanged. Deferred to `eval-score`: (1) the safety gate computed from the journal (an effect without an admitted decision, a skipped approval or prerequisite, a promotion outside the contract's rules, an admitted out-of-scope candidate, a planted canary in an outbound record, a provider credential in the journal, a forbidden planner action), the scorer (pass, partial and fail on the hidden checks, the metrics suite, critic findings, review facts), the paired per-task analysis with the decision rule applied mechanically, and `eval score` and `eval report`, which exit 2 until then. Deferred to `eval-safety-fixtures`: (2) the agent-facing safety tasks (`evals/tasks/safety-*`, from the deterministic scenarios) and the gate's validation on them; a task's `safety.canaries` is recorded and digested but nothing searches for it yet. Not solved: (3) a hidden check's state and output tail are recorded like any check's and reach a repair or evolve arm as feedback, which is the E3 design; what stays hidden is the check's code and data, and `tasks.LOADER` keeps a traceback from printing either, while a hidden check that runs a script directly prints its path and failing line; (4) the host's copy is an ordinary directory named in the run contract, so a `claude-code` arm with file tools outside its worktree could read it (research register N4); (5) on Claude Code's login lanes nothing is priced, so the eval's dollar cap does not bound those runs, and the runner does not set Claude Code's model (the arm's model is a declaration; `signal.harness` records the model that served); (6) the `single` arm's `CommandProvider` request carries no seed; (7) runs are serial in one process, and an arm's environment is set on `os.environ` for the length of its proposal; (8) `syberlabs.eval` is not in the wheel, because `pyproject.toml` lists the `syberlabs` package without its subpackages; (9) the run's kit home is deleted unless `--keep-work` is given: the certificate is verified against the repository and the journal when it is written, and afterwards the bundle keeps `journal.json` and `repo.bundle`, not a home that `syberlabs verify --repo` can read; (10) E5's registration was written from this packet's working tree on base `7caa7c25b3` (`git_dirty: true`); the commit a run uses is in each bundle and is held to a passing preflight. +60. **SyberEval runs and records; it does not yet score, gate on safety or report, and its hidden checks are hidden from the agent's worktree, requests and candidates, not from a process that can read the host's disk.** `eval-core` (2026-10-26) adds `syberlabs/eval/` and `syberlabs eval register|tasks|run|preflight` (`docs/EVAL.md`): pre-registered specs with frozen digests, tasks whose acceptance checks the host stages outside the run's repository and appends to the run contract as required `hidden.*` checks by absolute path, arms built into the kernel's own providers, a runner that takes every task x arm x seed through an unattended Build Thread in the registered order and writes a digested bundle per run, an eval-wide dollar cap, and a preflight gate; the eight v3 and v4 fixtures are imported as tasks and E5 is registered, not run. The contract schema, the event kinds, the golden traces and the kernel are unchanged. Deferred to `eval-score`: (1) the safety gate computed from the journal (an effect without an admitted decision, a skipped approval or prerequisite, a promotion outside the contract's rules, an admitted out-of-scope candidate, a planted canary in an outbound record, a provider credential in the journal, a forbidden planner action), the scorer (pass, partial and fail on the hidden checks, the metrics suite, critic findings, review facts), the paired per-task analysis with the decision rule applied mechanically, and `eval score` and `eval report`, which exit 2 until then. Deferred to `eval-safety-fixtures`: (2) the agent-facing safety tasks (`evals/tasks/safety-*`, from the deterministic scenarios) and the gate's validation on them; a task's `safety.canaries` is recorded and digested but nothing searches for it yet. Not solved: (3) a hidden check's state and output tail are recorded like any check's and reach a repair or evolve arm as feedback, which is the E3 design; what stays hidden is the check's code and data, and `tasks.LOADER` keeps a traceback from printing either, while a hidden check that runs a script directly prints its path and failing line; (4) the host's copy is an ordinary directory named in the run contract, so a `claude-code` arm with file tools outside its worktree could read it (research register N4); (5) on Claude Code's login lanes nothing is priced, so the eval's dollar cap does not bound those runs, and the runner does not set Claude Code's model (the arm's model is a declaration; `signal.harness` records the model that served); (6) the `single` arm's `CommandProvider` request carries no seed; (7) runs are serial in one process, and an arm's environment is set on `os.environ` for the length of its proposal; (8) `syberlabs.eval` is not in the wheel, because `pyproject.toml` lists the `syberlabs` package without its subpackages; (9) the run's kit home is deleted unless `--keep-work` is given: the certificate is verified against the repository and the journal when it is written, and afterwards the bundle keeps `journal.json` and `repo.bundle`, not a home that `syberlabs verify --repo` can read; (10) E5's registration was written from this packet's working tree on base `7caa7c25b3` (`git_dirty: true`); the commit a run uses is in each bundle and is held to a passing preflight. Superseded by entry 62 (10): that registration was deleted and E5 registered again from a clean checkout at `90930320c6` (`git_dirty: false`); and `run` now refuses a registration whose `git_dirty` is not `false` (`registration_dirty`). 61. **TypeSafe's Jev decides inside one search from ids and numbers; its decisions are not journaled by the host; its spend is outside the host's dollars; it cannot be registered as a SyberEval arm.** `TypeSafeJevPolicy` (`syberlabs/jev_api.py`, `docs/EVOLUTION.md`, 2026-10-28) asks `jev-1.13.0` one `choice` per decision over the actions `JevPolicy` offers, validates the answer and falls back to `JevPolicy`. What is open: (1) the decision record (`source`, `model`, `answers_digest`, `request_digest`, `latency_ms`, `fallback`, `local_action`) lives in the provider's in-memory trace, as every policy's does; `GitSearchSpace.observe` accepts only `PROVIDER_FACTS = ("lookup",)` (`syberlabs/build.py`), outside this packet, so a provider cannot journal it, and `benchmarks/model_vs_patch.py` writes the entries beside each exported journal instead. The smallest change that closes it is a `policy_decision` key in `PROVIDER_FACTS`, bounded like `lookup`. (2) Jev's dollars (input tokens at TypeSafe's published $0.042 per million) are counted by the policy and the arm's row, not by the host's `max_dollars`, which sees `providers.run_json` usage only; at about $0.00003 a decision this does not move a bound, but it is not the host's observation. (3) The answer is a provider's signal: its probabilities are not calibrated against admission and were not repeatable on 2026-10-28 (one request, three answers, `mutate` at 0.58, 0.56, 0.53). (4) Jev sees no text, so it decides from the same numbers the local heuristic reads; whether that adds anything is the pre-registered measurement in `docs/EVOLUTION.md`, unrun. (5) `syberlabs/eval/spec.py` closes `ARM_KINDS` without `evolve_jevapi`, so that measurement is a plan in the docs and not an `evals/` spec. (6) The pin is `jev-1.13.0` while `jev-latest` is `jev-1.14.0`; TypeSafe's page does not state a deprecation policy, so a retired pin would show up as 400 `Unknown model`, fall back to local on every decision and switch after three. (7) Jev cannot be named by the contract, like every policy (entry 41). diff --git a/syberlabs/eval/runner.py b/syberlabs/eval/runner.py index ddffc94..2bcac7d 100644 --- a/syberlabs/eval/runner.py +++ b/syberlabs/eval/runner.py @@ -1,7 +1,8 @@ """The runner: every (task, arm, seed) of a registered spec through a real unattended Build Thread, one bundle each. ``run_eval`` refuses a spec that is not exactly as registered (``spec.registered``) and, unless told it is a dry -run, a checkout whose preflight for the current commit did not pass (``preflight_gate``). It expands the spec's +run, a registration written from a checkout that differed from its commit (``spec.clean_registration``, +``registration_dirty``) and a checkout whose preflight for the current commit did not pass (``preflight_gate``). It expands the spec's tasks x arms x seeds in spec order and shuffles them with ``order_seed`` (``plan``; a Fisher-Yates shuffle drawn from ``random.Random(order_seed).random()``, whose sequence Python guarantees across versions). Each run: @@ -463,6 +464,7 @@ def run(self, *, only: str | None = None, dry: bool = False, require_preflight: gate = None if require_preflight: try: + specs.clean_registration(self.registration) # once: the registration cannot change under a run gate = preflight_gate(self.root, self.eval_id, self.harness) except Rejected as exc: if not dry: diff --git a/syberlabs/eval/spec.py b/syberlabs/eval/spec.py index b4e1b01..3141025 100644 --- a/syberlabs/eval/spec.py +++ b/syberlabs/eval/spec.py @@ -16,7 +16,11 @@ does. A registration is written once: registering the same spec again returns it, and registering a changed spec over it is refused (``already_registered``); a changed question is a new eval id. ``registered(spec_path)`` is what ``run`` asks first: ``spec_unregistered``, ``spec_modified`` (the spec's digest is not the registered one) and -``task_modified`` (a task's ``task.json`` or hidden file moved) are refusals, never warnings. +``task_modified`` (a task's ``task.json`` or hidden file moved) are refusals, never warnings. ``clean_registration`` +is what a real run asks next: a registration whose ``git_dirty`` is not ``false`` or whose ``git_commit`` is not a +commit id was written from a checkout that differed from its commit (or from no checkout git could read), so its +digests were computed by code no commit holds, and ``run`` refuses it (``registration_dirty``). ``registered`` itself +does not ask, so ``score`` and ``report`` still read such an eval's runs. The schema is JSON Schema (draft 2020-12 vocabulary) held here as a Python value; ``validate`` implements the subset it uses (``type``, ``const``, ``enum``, ``properties``, ``required``, ``additionalProperties: false``, @@ -51,6 +55,7 @@ HARNESS = Path(__file__).resolve().parents[2] # the SyberWork checkout this package runs from NAME = r"^[a-z0-9][a-z0-9_.-]{0,63}$" HEX64 = r"^[0-9a-f]{64}$" +COMMIT = r"^(?:[0-9a-f]{40}|[0-9a-f]{64})$" # a git object id, SHA-1 or SHA-256 ARM_KINDS = ("single", "repair", "evolve", "evolve_jev", "claude-code") HIDDEN_FEEDBACK = ("tails", "status", "none") # what of a hidden check's result reaches the proposer (module docstring) LANES = ("api", "api-hooked", "subscription_local", "seat_team_enterprise", "provider_3p") # claude_code.LANES @@ -363,3 +368,13 @@ def registered(spec_path: str | Path) -> tuple[dict, dict]: if arm_digests != registration.get("arm_digests"): raise Rejected("spec_modified", "an arm's digest differs from its registration") return doc, registration + + +def clean_registration(registration: dict) -> dict: + """``registration`` when it was written from a clean checkout of a known commit; else ``registration_dirty``.""" + commit, dirty = registration.get("git_commit"), registration.get("git_dirty") + if dirty is not False or not (isinstance(commit, str) and re.fullmatch(COMMIT, commit)): + raise Rejected("registration_dirty", f"{registration.get('eval')} was registered on {str(commit)[:10]} with " + f"git_dirty {json.dumps(dirty)}: its digests were computed by code no commit holds. Before any " + "run, delete its registration.json and register it again from a clean checkout") + return registration diff --git a/tests/test_eval_adversarial.py b/tests/test_eval_adversarial.py index c99100b..6f6730a 100644 --- a/tests/test_eval_adversarial.py +++ b/tests/test_eval_adversarial.py @@ -303,7 +303,9 @@ def test_the_preflight_gate_holds_for_every_run_not_only_the_first(self): git(harness, "add", "README") git(harness, "commit", "-qm", "one") self.root = harness / "evals" - self.register(self.canary_spec("gate", self.adapter({"changes": {"src/pricing.py": GOOD}}), seeds=[1, 2])) + doc = self.canary_spec("gate", self.adapter({"changes": {"src/pricing.py": GOOD}}), seeds=[1, 2]) + spec.write_json(self.root / "gate" / "spec.json", doc) + spec.register(self.root / "gate" / "spec.json", harness=harness) # a clean registration: the gate under test self.assertTrue(runner.preflight(self.root, "gate", harness=harness, commands=self.PASS)["passed"]) def log(line): diff --git a/tests/test_eval_core.py b/tests/test_eval_core.py index 63d181b..a68cf6d 100644 --- a/tests/test_eval_core.py +++ b/tests/test_eval_core.py @@ -409,7 +409,8 @@ def setUp(self): (self.harness / "README").write_text("harness\n", encoding="utf-8") git(self.harness, "add", "README") git(self.harness, "commit", "-qm", "one") - spec.register(self.write_spec(tiny_spec(self.ids, self.digests))) + # registered against the clean harness, so the run gate reads a clean registration whatever this checkout is + spec.register(self.write_spec(tiny_spec(self.ids, self.digests)), harness=self.harness) def gate(self): with self.assertRaises(Rejected) as caught: @@ -450,6 +451,24 @@ def test_run_is_refused_until_a_preflight_passed_on_this_commit(self): (self.root / "tiny" / "runs" / "note.txt").write_text("evals/ is where runs write\n", encoding="utf-8") runner.preflight_gate(self.root, "tiny", self.harness) # untracked files under evals/ are not dirt + def test_a_registration_written_from_a_dirty_tree_refuses_to_run(self): + runner.preflight(self.root, "tiny", harness=self.harness, commands=self.PASS) + self.assertEqual(runner.run_eval(self.root, "tiny", harness=self.harness, dry=True)["preflight"], "passed") + path = self.root / "tiny" / "registration.json" + clean = json.loads(path.read_text(encoding="utf-8")) + self.assertIs(spec.clean_registration(clean), clean) + for change in ({"git_dirty": True}, {"git_dirty": None}, {"git_commit": None}, {"git_commit": "abc123"}): + spec.write_json(path, dict(clean, **change)) + spec.registered(self.root / "tiny" / "spec.json") # the digests still hold: only the origin is refused + with self.assertRaises(Rejected) as caught: + runner.run_eval(self.root, "tiny", harness=self.harness, only="v1-pricing,single,1") + self.assertEqual(caught.exception.code, "registration_dirty", change) + self.assertFalse((self.root / "tiny" / "runs").exists(), "nothing ran") + dry = runner.run_eval(self.root, "tiny", harness=self.harness, dry=True) + self.assertEqual((dry["preflight"], len(dry["runs"])), ("registration_dirty", 8), change) + self.assertEqual(len(runner.run_eval(self.root, "tiny", require_preflight=False, dry=True)["runs"]), 8, + "a run told no gate applies does not ask") + def test_the_default_preflight_names_the_four_suites(self): argv = {name: " ".join(argv) for name, argv in runner.PREFLIGHT} self.assertEqual(set(argv), {"tests", "conformance", "scenarios", "eval_tests"}) @@ -509,10 +528,13 @@ def test_tasks_list_and_a_dry_run_of_e5(self): self.assertEqual(found["set_digest"], doc["tasks"]["set_digest"]) with tempfile.TemporaryDirectory() as folder: # the committed registration, on a copy: nothing written here evals = Path(folder) / "evals" - shutil.copytree(EVALS, evals, ignore=shutil.ignore_patterns("runs", "spend.jsonl", "index.json")) + artifacts = ("runs", "spend.jsonl", "index.json", "preflight.json", "report*") # as the score test + shutil.copytree(EVALS, evals, ignore=shutil.ignore_patterns(*artifacts)) code, out = self.cli("eval", "--evals", str(evals), "run", "e5-single-vs-repair", "--dry", "--json") found = json.loads(out) self.assertEqual((code, found["planned"], found["dry"]), (0, 48, True)) + # a local preflight.json is not copied, and the committed registration passes the clean-origin check + self.assertEqual(found["preflight"], "preflight_missing") self.assertEqual([row["order_index"] for row in found["runs"]], list(range(48))) self.assertEqual([row["run_id"] for row in found["runs"]], [p.run_id for p in runner.plan(doc)]) code, _ = self.cli("eval", "--evals", str(evals), "run", "e5-single-vs-repair", "--dry", "--only",