Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/EVAL.md
Original file line number Diff line number Diff line change
Expand Up @@ -199,7 +199,7 @@ It writes `report.md` and `report.json` (`report.partial.*` under `--partial`, w

## Preflight

`eval preflight <id>` runs `scripts/run_shards.py`, `python -m conformance.run`, `benchmarks/run_scenarios.py` and `python -m unittest discover -s tests -p "test_eval_*.py"` from the checkout, with `PYTHONPATH` set to it and with the certificate seed and GitHub tokens removed from the environment. It writes `preflight.json` with the commit, whether the checkout differs from it outside `evals/`, and each command's exit code, seconds, output digest and output tail. `run` refuses `preflight_missing`, `preflight_failed`, `preflight_stale` (the record names another commit) and `preflight_dirty` (the checkout differs from its commit outside `evals/`, now or when the preflight ran). Untracked files under `evals/` are not dirt, because that is where runs write.
`eval preflight <id>` runs `scripts/run_shards.py`, `python -m conformance.run`, `benchmarks/run_scenarios.py` and `python -m unittest discover -s tests -p "test_eval_*.py"` from the checkout, with `PYTHONPATH` set to it and with the certificate seed and GitHub tokens removed from the environment. It writes `preflight.json` with the commit, whether the checkout differs from it outside `evals/`, and each command's exit code, seconds, output digest and output tail. `run` refuses `registration_dirty` (the registration's `git_dirty` is not `false` or its `git_commit` is not a commit id, so its digests were computed by code no commit holds; delete `registration.json` before any run and register again from a clean checkout), `preflight_missing`, `preflight_failed`, `preflight_stale` (the record names another commit) and `preflight_dirty` (the checkout differs from its commit outside `evals/`, now or when the preflight ran). A dry run reports the first refusal in its `preflight` field; `score` and `report` do not ask about the registration's origin. Untracked files under `evals/` are not dirt, because that is where runs write.

## E5 and E6

Expand Down
26 changes: 23 additions & 3 deletions docs/EVOLUTION.md
Original file line number Diff line number Diff line change
Expand Up @@ -157,7 +157,7 @@ It takes the highest score (ties in the order evaluate, crossover, mutate, migra

Simulated, and nothing about a model: the profiles were chosen so the policies can be told apart, the in-memory world is not the Build Thread, and the fixture has three independent rules that favour recombination by construction. On the real Build Thread the module checks, over two seeds with a priced contract and a command mutator that reports usage, that the host's attributions reach `OperatorStats.dollars` kind by kind, that every Jev decision entry carries the audit fields, that the signal stays within the bound, and that Jev spends fewer host-observed dollars than Empirical. The recorded-equivalence tests for `FixedPolicy` are unchanged and still pass.

**Decision rule.** From the packet's measurement section: "The decision rule for Jev: it is kept as the provider's default if, on the v2 fixtures, `evolve_jev` solves at least as many as `evolve` with at most half the dollars, over the three repeats; otherwise it stays available and not default." That is: Jev becomes the default only if it solves at least as many v2 fixtures as `fixed` with at most half the dollars over three repeats. The E3 v2 pilot of 2026-10-21 (Claude Sonnet 5.5, medium, K = 6, one repeat; `docs/MODEL_COMPARISON.md`) solved all four v2 fixtures single-shot, 16 of 16 runs in 28 calls for about $0.24, with `evolve` costing four to six times `single` on every fixture, so by the plan's own rule the v2 set is uninformative for that model and the paid Jev-against-fixed comparison will not run on v2 at Sonnet 5.5 as this packet's spec assumed. The simulated comparison above is the evidence in this packet; the paid comparison is deferred to a fixture set on which `single` fails (Haiku 5.5 on v2, or a v3 set), a choice the owner makes. Until that run is made and reported, `fixed` is the default and the table above is not evidence for the rule.
**Decision rule.** From the packet's measurement section: "The decision rule for Jev: it is kept as the provider's default if, on the v2 fixtures, `evolve_jev` solves at least as many as `evolve` with at most half the dollars, over the three repeats; otherwise it stays available and not default." That is: Jev becomes the default only if it solves at least as many v2 fixtures as `fixed` with at most half the dollars over three repeats. The E3 v2 pilot of 2026-10-21 (Claude Sonnet 5.5, medium, K = 6, one repeat; `docs/MODEL_COMPARISON.md`) solved all four v2 fixtures single-shot, 16 of 16 runs in 28 calls for about $0.24, with `evolve` costing four to six times `single` on every fixture, so by the plan's own rule the v2 set is uninformative for that model and the paid Jev-against-fixed comparison will not run on v2 at Sonnet 5.5 as this packet's spec assumed. The simulated comparison above is the evidence in this packet; the paid comparison is deferred to a fixture set on which `single` fails (Haiku 5.5 on v2, or a v3 set), a choice the owner makes. Until that run is made and reported, `fixed` is the default and the table above is not evidence for the rule. That run was made on the v3 fixtures on 2026-10-28 (results under "Measurement plan" below): the rule was not met, and `fixed` stays the default.

### TypeSafe's Jev API as a policy

Expand Down Expand Up @@ -186,9 +186,9 @@ Simulated, and nothing about a model: the profiles were chosen so the policies c
| Price | $0.042 per million input tokens, output free (docs.typesafe.ai/models, read 2026-10-28): about $0.00003 a decision; the 28 calls cost about $0.0006 |
| Data terms | docs.typesafe.ai/legal points to a Data Processing Agreement and a Privacy Policy that commits not to train on user data; retention periods are not stated on that page; zero data retention is offered to enterprise customers |

What the calls say about the choice, and no more: on these synthetic states Jev chose `mutate` or `evaluate` every time; it gave `crossover` 0.02 to 0.15 on the 19 three- and four-option answers and 0.37 on the one two-option answer (`crossover` against `mutate`, `tests/fixtures/jev_api/choice_two_options.json`), where the local policy prices a crossover (no model call) far below a mutation. Whether that is better or worse on a real fixture is what the measurement below is for.
What the calls say about the choice, and no more: on these synthetic states Jev chose `mutate` or `evaluate` every time; it gave `crossover` 0.02 to 0.15 on the 19 three- and four-option answers and 0.37 on the one two-option answer (`crossover` against `mutate`, `tests/fixtures/jev_api/choice_two_options.json`), where the local policy prices a crossover (no model call) far below a mutation. Whether that is better or worse on a real fixture is what the measurement below was for; on the v3 fixtures Jev chose `mutate` on every one of the 21 decisions it was asked, and the local policy would have chosen `crossover` on each.

**Measurement plan (pre-registered 2026-10-28, re-targeted to v3 the same day; not run).** It continues the `jev-policy` measurement and needs the owner's go through the lead, because it spends on Anthropic's API.
**Measurement plan (pre-registered 2026-10-28, re-targeted to v3 the same day; run 2026-10-28 on `main` at `5ebbbce`, results after the plan).** It continues the `jev-policy` measurement and needs the owner's go through the lead, because it spends on Anthropic's API.

- *Set*: the four **v3** fixtures (`gaps_renumber`, `journal_chain`, `worktree_order`, `verify_layers`). v2 is uninformative at this model and at Haiku 5.5, and the paid Jev-against-fixed comparison goes to v3 by the owner's decision (`docs/MODEL_COMPARISON.md`, "Where that run goes"); v4 is the fallback.
- *Arms*: `single`, `evolve` (`FixedPolicy` revision 2), `evolve_jev` (local `JevPolicy`) and `evolve_jevapi` (`TypeSafeJevPolicy`, `jev-1.13.0`), through `benchmarks/model_vs_patch.py` and the same Build Thread; K = 6 model calls per run; Claude Sonnet 5.5 at medium effort (`SYBERLABS_MODEL=claude-sonnet-5-5`, `SYBERLABS_EFFORT=medium`), with `ANTHROPIC_API_KEY` and `TYPESAFE_API_KEY` in that shell only.
Expand All @@ -205,6 +205,26 @@ What the calls say about the choice, and no more: on these synthetic states Jev
- Budget line for the owner: **$5.40 Anthropic hard ceiling** (expected $1.90) and **$0.15 Jev**.
- *Not expressible as a SyberEval spec yet*: `syberlabs/eval/spec.py` closes `ARM_KINDS` to `single`, `repair`, `evolve`, `evolve_jev` and `claude-code`, so neither `evolve_jevapi` nor a policy choice can be registered under `evals/`; this plan is the pre-registration (`spec/GAPS.md` 61).

**Results (run 2026-10-28 on `main` at `5ebbbce`; neither rule met; no default changes).** The owner approved the run on 2026-10-28 with the ceilings above. It ran from a fresh clone of `main` at `5ebbbce6e0` (#149), the commit that registered this plan, so the plan text it read is the registered text; the section's SHA-256 at that commit, from its first line to the next heading, is `5e8f98a4b6e4421b57467fadd8cfe85b91d134b3fd780920aed06a79cfb363c1`. Source: the officer workspace's `reports/2026-10-28-jev-v3-measurement.md`, with the harness output, per-run journals, Jev decision records and guard logs under `reports/receipts/2026-10-28-jev-v3-measurement/`. Claude Sonnet 5.5 at medium effort served all 64 rows; TypeSafe served `jev-1.13.0` on all 27 answered calls. One substitution in form: the adapter was named by its absolute path in the clone (`C:/Python312/python.exe <clone>/adapters/anthropic_adapter.py`), because the harness runs the adapter from the fixture worktree (`docs/MODEL_COMPARISON.md`); arms, fixtures, K, repeats and model are as registered.

- *Pilot* (`--repeats 1`, seed 1): informative. `single` solved 1 of 4, six `evolve*` runs needed more than one candidate, and Jev was asked 6 decisions, each offering `{crossover, mutate}`. The measurement ran; the pilot is counted in the cost and not read by the rules.
- *Measurement* (`--repeats 3`, seeds 1 to 3), per arm over the four fixtures and three repeats:

| Arm | Solved | Anthropic calls | Anthropic $ | Jev calls | Jev $ | Total $ |
| --- | --- | --- | --- | --- | --- | --- |
| `single` | 4/12 | 12 | 0.119276 | | | 0.119276 |
| `evolve` (`fixed/2`) | 11/12 | 40 | 0.441426 | | | 0.441426 |
| `evolve_jev` (`jev/1`) | 10/12 | 42 | 0.483434 | | | 0.483434 |
| `evolve_jevapi` (`jev_api/1`) | 9/12 | 46 | 0.525576 | 21 | 0.000804468 | 0.526380 |

Per fixture, solved (`single` / `evolve` / `evolve_jev` / `evolve_jevapi`): `gaps_renumber` 1 / 3 / 2 / 2, `journal_chain` 3 / 3 / 3 / 3, `worktree_order` 0 / 2 / 2 / 1, `verify_layers` 0 / 3 / 3 / 3.

- *Rule carried over* (`evolve_jev` against `evolve`): **not met**. 10 solves against 11, and $0.483434 against a bar of half of $0.441426 ($0.220713), a ratio of 1.095. `evolve_jev` stays available and not default; `fixed` stays the provider's default.
- *New rule* (`evolve_jevapi` against `evolve_jev`): **measured, and not met**. Jev was asked 21 decisions, 0 fell back and no run switched to local, so the comparison counts as measured; it agreed with `local_action` on 0 of 21, so the "no difference may be claimed" clause does not apply. 9 solves against 10, and $0.526380 (Anthropic plus Jev) against $0.483434. TypeSafe's Jev does not replace the local heuristic; `repair` stays the default proposer path.
- *Jev's choices*: every asked decision (21, all on `worktree_order` and `verify_layers`) offered exactly `{crossover, mutate}`. Jev chose `mutate` on all 21, and the local policy would have chosen `crossover` on all 21. Its probability for `crossover` had a median of 0.37 (0.24 to 0.49); median latency 188.8 ms; 19,154 input tokens. On `gaps_renumber` and `journal_chain` no decision offered two actions, so the two failed `gaps_renumber` runs (seed 1, in both Jev arms) did not depend on the chooser.
- *Spend*, pilot and measurement: **$1.987102** Anthropic (178 calls, the harness's pricing of the adapter's reported usage; `host_dollars` equals `dollars` on every row) and **$0.001034670** Jev (27 calls, input tokens at $0.042 per million), 36.8% and 0.69% of the ceilings. Neither is an invoice.
- *Claim ceiling*: one pre-registered run at small n (12 runs per arm per rule) of four arms on the four v3 fixtures, at Claude Sonnet 5.5 medium, K = 6 and `jev-1.13.0`. The deciding gaps are one solve and 9 to 10% in dollars, within run-to-run variation for a model that takes no seed. "Not met" says the pre-registered bars were not cleared on this sample, not that `evolve_jev` or Jev is worse; it says nothing about another fixture set, model, effort, K or Jev version.

### Measured on a toy fixture

`examples/evolve.py` is a pricing module with three independently checked rules and a stand-in mutator. `benchmarks/evolve_baseline.py` compares the provider with and without crossover on 30 seeds, with the same budget. The result is in `benchmarks/results/evolve-baseline.txt`:
Expand Down
16 changes: 15 additions & 1 deletion docs/MODEL_COMPARISON.md
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,7 @@ What it shows about the **harness and the policies on these profiles**: on `muta

**Measurement procedure and decision rule**, copied from the packet spec (`jev-policy`, "Measurement"): "Run the three `evolve*` arms plus `single` on the v2 fixtures (step 2) at K = 6, one repeat first, Sonnet 5.5 medium. The decision rule for Jev: it is kept as the provider's default if, on the v2 fixtures, `evolve_jev` solves at least as many as `evolve` with at most half the dollars, over the three repeats; otherwise it stays available and not default. Budget line: **$15** of the $50 cap (see the allocation in the review). The run is a named human's act with the key in that shell, or the orchestrator's under the owner's standing "ready for experimentation", stated in the results packet either way."

**Where that run goes.** The v2 set is uninformative at every tier measured on 2026-10-21: `single` solved all four v2 fixtures at Claude Sonnet 5.5 (16 of 16 runs, medium, K = 6, one repeat) and at Claude Haiku 5.5 (8 of 8, two repeats, about $0.003), so a comparison on v2 could only add single-shot failures on tasks every arm solves at its first candidate. The paid Jev-against-fixed comparison therefore goes to a set on which `single` fails, the v3 fixtures (a parallel packet), by the owner's decision; until it is run and reported, `fixed` stays the provider's default and the simulated table above is not evidence for the rule.
**Where that run goes.** The v2 set is uninformative at every tier measured on 2026-10-21: `single` solved all four v2 fixtures at Claude Sonnet 5.5 (16 of 16 runs, medium, K = 6, one repeat) and at Claude Haiku 5.5 (8 of 8, two repeats, about $0.003), so a comparison on v2 could only add single-shot failures on tasks every arm solves at its first candidate. The paid Jev-against-fixed comparison therefore goes to a set on which `single` fails, the v3 fixtures (a parallel packet), by the owner's decision; until it is run and reported, `fixed` stays the provider's default and the simulated table above is not evidence for the rule. It was run on 2026-10-28 (the note of that date below): the rule was not met, and `fixed` stays the default.

## Note of 2026-10-22: every `evolve*` row so far was taken with an empty scope in the mutate request

Expand Down Expand Up @@ -199,6 +199,20 @@ The wrong readings the oracle holds: `stack_landing` writes `*2026-10-22, narrow
**What is measured, and what is not.** With the simulated adapter on 2026-10-22 (K = 6, one repeat, arms `single`, `repair`, `evolve`), 0 of 12 runs solved: `single` 1 call and 1 evaluation on each fixture, `repair` 6 calls and 3 evaluations (median), `evolve` 6 calls and 2 evaluations (the stand-in knows only line swaps). `tests/test_model_comparison.py` (`FixturesV4`) re-derives the pins from the kit (`dsse.pae`, the certificate constants, `DEFAULT_TARGET_REF`, `certificate.summary`, `checks.SETUP_NAMES` and `EMPTY_DIGEST`, `PROMOTION_RANK`, `dashboard.TONE`, `inspector.INCOMPLETE_REMEDY`). Nothing here says what a model does. "Parts per dollar" in the prediction needs a per-part count the harness summary does not print (the per-part states are in each exported journal's `candidate_evaluated` events); that column belongs to the packet that owns `benchmarks/model_vs_patch.py` (`spec/GAPS.md`, 47). The `evolve_jev` arm is on #68's branch, so the v4 measurement runs on the stack, not on this branch alone.
Decision of 2026-10-23 (`repair-proposer`): on the v3 and v4 sets (Claude Sonnet 5.5, K = 6 and K = 12; the v4 note and its rows are in the `e3-v4-multipart` packet, #75, a parallel stack of 2026-10-22, with the receipts in the officer workspace) `repair` beat every evolve arm on solves and on parts per dollar at every budget tested, and beat `single` on solves; the prediction that evolution overtakes repair at a larger budget was falsified at K = 12 (42 against 65 parts per dollar, repair ahead). The owner's decision: the default proposer path is repair (`syberlabs.providers.RepairProvider`, `syberlabs propose --repair`), and the evolve arms stay as research. No row in this file is rewritten by that decision.

## Note of 2026-10-28: the pre-registered Jev measurement on v3, neither rule met

The measurement plan in `docs/EVOLUTION.md` (pre-registered 2026-10-28) ran the same day on `main` at `5ebbbce` (#149): arms `single`, `evolve` (`fixed/2`), `evolve_jev` (local `JevPolicy`) and `evolve_jevapi` (TypeSafe's Jev, `jev-1.13.0`), the four v3 fixtures, Claude Sonnet 5.5 at medium effort, K = 6, a one-repeat pilot (informative) and then three repeats, which the rules read. Source: the officer workspace's `reports/2026-10-28-jev-v3-measurement.md`, with the harness output, per-run journals, Jev decision records and guard logs under `reports/receipts/2026-10-28-jev-v3-measurement/`. The adapter was named by its absolute path in the clone, as the first finding above requires; nothing else differs from the plan's command.

```
arm policy solved calls anthropic$ jev calls jev$ total$
single - 4/12 12 0.119276 - - 0.119276
evolve fixed/2 11/12 40 0.441426 - - 0.441426
evolve_jev jev/1 10/12 42 0.483434 - - 0.483434
evolve_jevapi jev_api/1 9/12 46 0.525576 21 0.000804468 0.526380
```

Three repeats, seeds 1 to 3, summed over `gaps_renumber`, `journal_chain`, `worktree_order` and `verify_layers`; no row has a `provider_error`. The rule carried over from `jev-policy` (`evolve_jev` solves at least as many as `evolve` with at most half the dollars) is **not met**: 10 against 11, and 109.5% of `evolve`'s dollars. The new rule (`evolve_jevapi` solves at least as many as `evolve_jev` with at most its dollars, Anthropic plus Jev) is **measured and not met**: 0 fallbacks on 21 asked decisions and no switch to local, agreement with `local_action` 0 of 21, 9 against 10 solves, $0.526380 against $0.483434. TypeSafe's Jev chose `mutate` on all 21 decisions it was asked, where the local policy would have chosen `crossover`. Defaults are unchanged: `fixed` stays the evolutionary provider's default, `evolve_jev` stays available and not default, the local heuristic stays the chooser behind it, and `repair` stays the default proposer path. Spend, pilot included: $1.987102 Anthropic and $0.001034670 Jev, the harness's pricing and TypeSafe's published input rate, not invoices. At 12 runs per arm the deciding gaps (one solve, 9 to 10% in dollars) are within run-to-run variation; the rules were applied as written, and "not met" says only that the pre-registered bars were not cleared on this sample. The full result, per fixture and per decision, is in `docs/EVOLUTION.md` under the plan.

## What would count as evidence

A real-model run on these fixtures would show only whether the arms differ on toy tasks. The roadmap's bar is a real repository with several productive alternatives, a baseline, a failure case, and a decision that changed for a measured reason. Until that run exists, the claim in `docs/ROADMAP_STATUS.md` stays at what the pilot supports: **pilot measured 2026-10-21; uninformative by rule; harder fixtures next**.
Loading
Loading