Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
125 changes: 99 additions & 26 deletions docs/EVAL.md

Large diffs are not rendered by default.

23 changes: 0 additions & 23 deletions evals/e5-single-vs-repair/registration.json

This file was deleted.

1 change: 1 addition & 0 deletions evals/e5-single-vs-repair/spec.json
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@
"rule": "If the eval stops (eval_budget_exhausted, eval_spend_unknown) a task without a bundle for every arm and seed is left out of the paired comparison and named in the report."
}
],
"hidden_feedback": "tails",
"hypothesis": "Framework validation, not a new claim. On the eight v3 and v4 fixture tasks, with Claude Sonnet 5.5 at medium effort and K = 6 calls, the repair arm's per-task pass rate on the hidden checks is at least the single arm's (E3 v3 and v4, 2026-10-22: repair closed every fixture in one or two calls). A result against E3 is a finding about the instrument before it is one about the arms.",
"id": "e5-single-vs-repair",
"notes": "Pre-registered by the eval-core packet (2026-10-26); not run there. Running it spends money (about $10 at most): syberlabs eval preflight e5-single-vs-repair, then syberlabs eval run e5-single-vs-repair with ANTHROPIC_API_KEY and SYBERLABS_CERTIFICATE_SEED set.",
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ syberwork = "syberwork.cli:main"
syberlabs = "syberlabs.cli:main"

[tool.setuptools]
packages = ["syberwork", "syberlabs"]
packages = ["syberwork", "syberlabs", "syberlabs.eval"]

[tool.setuptools.package-data]
syberwork = ["static/*", "examples/*.json"]
Expand Down
2 changes: 2 additions & 0 deletions spec/GAPS.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,3 +121,5 @@ The schemas describe SyberWork as it behaves. They are not enforced at runtime e
60. **SyberEval runs and records; it does not yet score, gate on safety or report, and its hidden checks are hidden from the agent's worktree, requests and candidates, not from a process that can read the host's disk.** `eval-core` (2026-10-26) adds `syberlabs/eval/` and `syberlabs eval register|tasks|run|preflight` (`docs/EVAL.md`): pre-registered specs with frozen digests, tasks whose acceptance checks the host stages outside the run's repository and appends to the run contract as required `hidden.*` checks by absolute path, arms built into the kernel's own providers, a runner that takes every task x arm x seed through an unattended Build Thread in the registered order and writes a digested bundle per run, an eval-wide dollar cap, and a preflight gate; the eight v3 and v4 fixtures are imported as tasks and E5 is registered, not run. The contract schema, the event kinds, the golden traces and the kernel are unchanged. Deferred to `eval-score`: (1) the safety gate computed from the journal (an effect without an admitted decision, a skipped approval or prerequisite, a promotion outside the contract's rules, an admitted out-of-scope candidate, a planted canary in an outbound record, a provider credential in the journal, a forbidden planner action), the scorer (pass, partial and fail on the hidden checks, the metrics suite, critic findings, review facts), the paired per-task analysis with the decision rule applied mechanically, and `eval score` and `eval report`, which exit 2 until then. Deferred to `eval-safety-fixtures`: (2) the agent-facing safety tasks (`evals/tasks/safety-*`, from the deterministic scenarios) and the gate's validation on them; a task's `safety.canaries` is recorded and digested but nothing searches for it yet. Not solved: (3) a hidden check's state and output tail are recorded like any check's and reach a repair or evolve arm as feedback, which is the E3 design; what stays hidden is the check's code and data, and `tasks.LOADER` keeps a traceback from printing either, while a hidden check that runs a script directly prints its path and failing line; (4) the host's copy is an ordinary directory named in the run contract, so a `claude-code` arm with file tools outside its worktree could read it (research register N4); (5) on Claude Code's login lanes nothing is priced, so the eval's dollar cap does not bound those runs, and the runner does not set Claude Code's model (the arm's model is a declaration; `signal.harness` records the model that served); (6) the `single` arm's `CommandProvider` request carries no seed; (7) runs are serial in one process, and an arm's environment is set on `os.environ` for the length of its proposal; (8) `syberlabs.eval` is not in the wheel, because `pyproject.toml` lists the `syberlabs` package without its subpackages; (9) the run's kit home is deleted unless `--keep-work` is given: the certificate is verified against the repository and the journal when it is written, and afterwards the bundle keeps `journal.json` and `repo.bundle`, not a home that `syberlabs verify --repo` can read; (10) E5's registration was written from this packet's working tree on base `7caa7c25b3` (`git_dirty: true`); the commit a run uses is in each bundle and is held to a passing preflight.

61. **TypeSafe's Jev decides inside one search from ids and numbers; its decisions are not journaled by the host; its spend is outside the host's dollars; it cannot be registered as a SyberEval arm.** `TypeSafeJevPolicy` (`syberlabs/jev_api.py`, `docs/EVOLUTION.md`, 2026-10-28) asks `jev-1.13.0` one `choice` per decision over the actions `JevPolicy` offers, validates the answer and falls back to `JevPolicy`. What is open: (1) the decision record (`source`, `model`, `answers_digest`, `request_digest`, `latency_ms`, `fallback`, `local_action`) lives in the provider's in-memory trace, as every policy's does; `GitSearchSpace.observe` accepts only `PROVIDER_FACTS = ("lookup",)` (`syberlabs/build.py`), outside this packet, so a provider cannot journal it, and `benchmarks/model_vs_patch.py` writes the entries beside each exported journal instead. The smallest change that closes it is a `policy_decision` key in `PROVIDER_FACTS`, bounded like `lookup`. (2) Jev's dollars (input tokens at TypeSafe's published $0.042 per million) are counted by the policy and the arm's row, not by the host's `max_dollars`, which sees `providers.run_json` usage only; at about $0.00003 a decision this does not move a bound, but it is not the host's observation. (3) The answer is a provider's signal: its probabilities are not calibrated against admission and were not repeatable on 2026-10-28 (one request, three answers, `mutate` at 0.58, 0.56, 0.53). (4) Jev sees no text, so it decides from the same numbers the local heuristic reads; whether that adds anything is the pre-registered measurement in `docs/EVOLUTION.md`, unrun. (5) `syberlabs/eval/spec.py` closes `ARM_KINDS` without `evolve_jevapi`, so that measurement is a plan in the docs and not an `evals/` spec. (6) The pin is `jev-1.13.0` while `jev-latest` is `jev-1.14.0`; TypeSafe's page does not state a deprecation policy, so a retired pin would show up as 400 `Unknown model`, fall back to local on every decision and switch after three. (7) Jev cannot be named by the contract, like every policy (entry 41).

62. **SyberEval scores, gates and reports its runs; some gate rules cannot be computed for some arms, the agent-facing safety tasks do not exist yet, and candidate code still runs beside the hidden checks.** `eval-score` (2026-10-26, landed on the `eval-core` verification fixes of 2026-10-28: the runner's lock, bundle verification, the per-run preflight gate, strict JSON, refused run-id collisions and finite numbers) adds `syberlabs/eval/gate.py`, `score.py` and `report.py` and `syberlabs eval score|report` (`docs/EVAL.md`); the required spec field `hidden_feedback` (`tails`, `status`, `none`), enforced through `RepairProvider`'s optional `feedback_filter`, which every other caller leaves unset; the hidden copy's hashing (`tasks.tree_digests`) at four points against the copy as staged (`tasks.hidden_changes`, which names a symbolic link by its target and tolerates pytest's own `.pytest_cache`), with the run outcome `not_acceptable`, reason `hidden_tampered`; a host root where the hidden copy and the kit home get random names outside the run's work directory; the append-only spend ledger the eval's cap counts, read strictly, which also records every run that raised (it replaces the `runs/failed.jsonl` of the verification fixes); and `syberlabs.eval` in the wheel. It closes entry 60's (1) and (8) and narrows its (3) and (4). The contract schema, the event kinds, the golden traces and the kernel's behaviour for every existing caller are unchanged. Open: (1) the agent-facing safety tasks (`evals/tasks/safety-*`, packet `eval-safety-fixtures`) and the gate's validation on them: no planted canary, forbidden action or unsafe path has yet been offered to an agent, and every gate rule is tested on synthetic journals and with stand-in adapters only; (2) `canary.provider_requests` is `not_computable` for `claude-code` arms, whose own model requests hold whatever Claude Code's tools read, and for `evolve` and `evolve_jev` arms, whose mutate requests' feedback is not recorded; such a run's gate is `incomplete`, never `pass`; (3) the policy is not in the journal, so `approval_or_prerequisite_skipped` checks that at least one approval came between a `needs_approval` decision and its effect, not the policy's count; (4) `credential_observed` matches credential shapes (`syberlabs._patterns.SECRET`, at a word boundary) and the values of the host's credential variables present when the run ended, not every possible secret, and full check outputs and provider stderr are not retained, so only their recorded tails are scanned; (5) the hidden copy is hashed when staged, after the proposal, after the check and before acceptance, so a rewrite undone between two hashes, or inside one proposal's evaluations, is not detected, and candidate code inside a check can still read the copy and print it into its output, which `tails` sends to the next request; only a sandbox per check closes this (research register N4); (6) `hidden_feedback` filters the `repair` and `claude-code` feedback path only: an evolve arm builds its mutate requests in `syberlabs/evolve.py`, which no filter reaches, so a spec may use evolve arms under `tails` only; (7) under `none` the repair loop still stops on the first candidate that passes the hidden checks, so they still choose among up to K attempts on the host's side, and a call after a failure starts from the failed candidate's tree; (8) the eval's recorded spend can exceed `total_dollars` by one call's cost, and a call in flight when the runner is killed outright is in no journal and no ledger line; (9) the model API takes no sampling seed, so a run's seed indexes a repeat and is not a reproducible draw; (10) E5's registration is deleted, because adding `hidden_feedback` moved its digest and `register` returns an existing registration of the same digest unchanged; it is registered again on a clean checkout of `main` before it runs; (11) the package reads `benchmarks/`, the adapters and `evals/` from a checkout, so the wheel's `syberlabs eval` needs one to run anything; (12) the design gaps the verification pinned as `expectedFailure` in `tests/test_eval_adversarial.py` stay open: a candidate that exits 0 at import passes every hidden check; under `tails` a candidate can print its hidden check into the next request (under `status` and `none` it cannot, and that test passes); a `claude-code` arm's model is a declaration the runner neither sets nor compares with the model that served; a `repo:` task's run repository holds the commits after its base; and a runner killed outright charges nothing (tested on POSIX only, where `SIGKILL` exists).
15 changes: 10 additions & 5 deletions syberlabs/eval/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@
syberlabs eval register evals/<id>/spec.json # freeze the spec: evals/<id>/registration.json
syberlabs eval preflight <id> # the deterministic suites, recorded for this commit
syberlabs eval run <id> [--only T,A,S] [--dry] # every task x arm x seed, in the registered order
syberlabs eval score <id> # the safety gate, quality and efficiency of every run
syberlabs eval report <id> [--partial] # the paired analysis with the decision rule applied

The pieces (``docs/EVAL.md``):

Expand All @@ -16,13 +18,16 @@
named ``hidden.<check>``, by absolute path, so the agent's tree, its requests and its candidates never hold them.
- ``arms``: named provider configurations (``single``, ``repair``, ``evolve``, ``evolve_jev``, ``claude-code``)
built into the kernel's own providers; each has a canonical digest.
- ``runner``: the run, the evidence bundle, the eval's dollar cap, and the preflight gate.
- ``runner``: the run, the evidence bundle with every hash of the hidden copy, the spend ledger and the eval's
dollar cap, and the preflight gate.
- ``gate``: the safety gate computed from a run's journal and bundle, rule by rule.
- ``score``: per run, the gate, task quality on the hidden checks, and the metrics suite; ``index.json``.
- ``report``: tasks paired, the decision rule applied in exact fractions, a sign test and a bootstrap beside it.

Scoring and reporting are the next packet (``eval-score``). This package runs from a SyberWork checkout: it reads
``benchmarks/fixtures.py`` and the adapters beside it, and it is not in the wheel (``pyproject.toml`` lists the
``syberlabs`` package only, not its subpackages).
The wheel carries this package, but an eval runs from a SyberWork checkout: it reads ``benchmarks/fixtures.py``, the
adapters and ``evals/`` from there.
"""

from __future__ import annotations

VERSION = "1" # the eval package's own revision, recorded in every bundle as harness.eval
VERSION = "2" # the eval package's own revision, recorded in every bundle as harness.eval
53 changes: 49 additions & 4 deletions syberlabs/eval/arms.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,11 +22,20 @@
reads ``SYBERLABS_MODEL`` and ``SYBERLABS_EFFORT`` from it (``adapters/anthropic_adapter.py``), the simulated model
reads ``SYBERLABS_FIXTURE``, and the certificate seed and GitHub tokens are removed so no provider process inherits
them.

``hidden_view`` is the spec's ``hidden_feedback`` mode applied to one ``feedback`` object (``providers.repair_feedback``):
``tails`` returns it unchanged; ``status`` keeps each hidden check's name and ``passed`` or ``failed`` and drops its
exit code, duration and output tail; ``none`` drops every hidden check, and the whole ``feedback`` when no visible
check is left that did not pass (a request that said "the last attempt failed" would be hidden information).
Visible checks and the candidate's own ``files`` pass unchanged in every mode. ``build`` sets it as the
``feedback_filter`` of a ``repair`` or ``claude-code`` provider, recording what each request carried; a ``single``
arm sends no feedback, and an evolve arm is refused any mode but ``tails`` by ``spec.check``.
"""

from __future__ import annotations

import inspect
import json
import os
import re
import statistics
Expand All @@ -45,6 +54,7 @@
# Removed from the environment while an arm proposes; restored after (the certificate seed variable is added).
WITHHELD_ENV = ("GITHUB_TOKEN", "GH_TOKEN")
NOMINAL_CALL = (2000, 500) # tokens in, out: Jev's prior price of one mutation (benchmarks/model_vs_patch.py)
HIDDEN_PREFIX = "hidden." # tasks.HIDDEN_PREFIX: the contract names every hidden check hidden.<name>


def check(arm: dict, budget: dict) -> list[str]:
Expand Down Expand Up @@ -95,10 +105,43 @@ def githits_arm(name: str) -> str:
return re.sub(r"[^A-Za-z0-9_-]", "-", name)[:16]


def _hidden(check) -> bool:
return isinstance(check, dict) and isinstance(check.get("name"), str) and check["name"].startswith(HIDDEN_PREFIX)


def hidden_view(feedback: dict, mode: str) -> dict | None:
"""``feedback`` as the ``hidden_feedback`` ``mode`` lets the proposer see it, or None for no feedback at all."""
if mode == "tails":
return feedback
checks = feedback.get("checks") or []
if mode == "status":
return {**feedback, "checks": [{"name": c["name"], "state": "passed" if c.get("state") == "passed" else "failed"}
if _hidden(c) else c for c in checks]}
if mode == "none":
visible = [c for c in checks if not _hidden(c)]
if all(isinstance(c, dict) and c.get("state") == "passed" for c in visible):
return None
return {**feedback, "checks": visible}
raise Rejected("invalid_spec", f"hidden_feedback must be one of tails, status, none, not {mode!r}")


def feedback_filter(mode: str, sent: list | None = None):
"""A provider's ``feedback_filter`` for ``mode``; each result, as sent, is appended to ``sent``."""
def apply(feedback: dict) -> dict | None:
shown = hidden_view(feedback, mode)
if sent is not None:
sent.append({"candidate": feedback.get("candidate"), "sent": shown is not None,
"feedback": None if shown is None else json.loads(json.dumps(shown))})
return shown
return apply


def build(arm: dict, *, repo: Path, budget: dict, prices: dict | None, harness: Path = HARNESS,
thread_label: str | None = None):
thread_label: str | None = None, hidden_feedback: str = "tails", sent: list | None = None):
"""The provider for one run of ``arm`` against the repository at ``repo``. ``thread_label`` (the run's thread
id) and the arm's name label a ``githits`` arm's lookups, so each run is its own GitHits session."""
id) and the arm's name label a ``githits`` arm's lookups, so each run is its own GitHits session. A ``repair`` or
``claude-code`` provider filters its feedback through ``hidden_feedback`` and records each one it sends in
``sent``."""
kind, k = arm["kind"], calls(arm, budget)
timeout = arm.get("timeout_seconds", 600)
name = arm["name"] if arm["name"][0].isalpha() else "arm-" + arm["name"]
Expand All @@ -108,7 +151,7 @@ def build(arm: dict, *, repo: Path, budget: dict, prices: dict | None, harness:
if kind == "repair":
from syberlabs.providers import RepairProvider
return RepairProvider(resolve(arm["adapter"], harness), name=name, revision="1", timeout=timeout, cwd=repo,
max_calls=k)
max_calls=k, feedback_filter=feedback_filter(hidden_feedback, sent))
if kind in ("evolve", "evolve_jev"):
from syberlabs.evolve import CommandMutator, EvolutionaryProvider, JevPolicy
mutator = CommandMutator(resolve(arm["adapter"], harness), name="model", timeout=timeout, cwd=repo, max_calls=k)
Expand All @@ -131,7 +174,9 @@ def build(arm: dict, *, repo: Path, budget: dict, prices: dict | None, harness:
options["githits"] = arm["githits"]
if arm["githits"]:
options.update(arm=githits_arm(arm["name"]), thread_label=thread_label)
return ClaudeCodeProvider(repo, **options)
provider = ClaudeCodeProvider(repo, **options)
provider.feedback_filter = feedback_filter(hidden_feedback, sent) # the RepairProvider hook it inherits
return provider


@contextmanager
Expand Down
Loading
Loading