Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
119 changes: 95 additions & 24 deletions docs/EVAL.md

Large diffs are not rendered by default.

23 changes: 0 additions & 23 deletions evals/e5-single-vs-repair/registration.json

This file was deleted.

1 change: 1 addition & 0 deletions evals/e5-single-vs-repair/spec.json
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@
"rule": "If the eval stops (eval_budget_exhausted, eval_spend_unknown) a task without a bundle for every arm and seed is left out of the paired comparison and named in the report."
}
],
"hidden_feedback": "tails",
"hypothesis": "Framework validation, not a new claim. On the eight v3 and v4 fixture tasks, with Claude Sonnet 5.5 at medium effort and K = 6 calls, the repair arm's per-task pass rate on the hidden checks is at least the single arm's (E3 v3 and v4, 2026-10-22: repair closed every fixture in one or two calls). A result against E3 is a finding about the instrument before it is one about the arms.",
"id": "e5-single-vs-repair",
"notes": "Pre-registered by the eval-core packet (2026-10-26); not run there. Running it spends money (about $10 at most): syberlabs eval preflight e5-single-vs-repair, then syberlabs eval run e5-single-vs-repair with ANTHROPIC_API_KEY and SYBERLABS_CERTIFICATE_SEED set.",
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ syberwork = "syberwork.cli:main"
syberlabs = "syberlabs.cli:main"

[tool.setuptools]
packages = ["syberwork", "syberlabs"]
packages = ["syberwork", "syberlabs", "syberlabs.eval"]

[tool.setuptools.package-data]
syberwork = ["static/*", "examples/*.json"]
Expand Down
2 changes: 2 additions & 0 deletions spec/GAPS.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,3 +119,5 @@ The schemas describe SyberWork as it behaves. They are not enforced at runtime e
59. **The GitHits API client is wired into Claude Code as arm B's only lookup route; its contamination and license checks are filters on what returns, and its lookups are counted from the client's own records.** `githits-client` (2026-10-26) adds `syberlabs/githits.py` and `syberlabs githits search|example|read|list|grep` (`docs/GITHITS.md`): a standard-library client for five read-only GitHits REST operations, with a SyberLabs exclusion in three layers (a target refused before any request, a result whose identity is under an excluded owner dropped and counted, an example or read body that cites one dropped whole as `contamination_blocked`), a permissive-only license policy (`license_mode: strict` on examples; client-side, a result whose SPDX license is not on the allow list dropped and counted as `results_dropped_license`), and an HMAC-signed record per call (`query_guard.sign`, now shared). `githits-wiring` (2026-10-26) closes the first two deferred items: (1) `ClaudeCodeProvider(githits=True)` (`--githits`) writes a per-call, read-only launcher outside the worktree that runs the host checkout's client (`syberlabs/githits_cli_entry.py` under `python -I`), allows `Bash(<launcher>:*)` and nothing else that runs code, matches `Bash` in the query guard (only the launcher with plain arguments passes, and its arguments are judged by the guard's rules), and journals one `lookup` per authentic client record (tool `githits.<op>`), `signal.tool_usage.githits` being their number and `githits_lookups` reading those facts; (2) GitHits' shipped skills (`githits-com/githits-cli`, Apache-2.0) are adapted into `skills/githits/SKILL.md` with their license and notice, and appended to arm B's system prompt (`signal.harness.skill`). The same packet fixes #129's three low findings (nested SPDX expressions, the token scrubbed before truncation and out of `X-Request-ID`, `SYBERLABS_GITHITS_EXCLUDE` adding to `SyberLabs` instead of replacing it) and adds `syberlabs githits doctor` (one `GET /v1/languages`). `githits-pilot-hardening` (2026-10-26) narrows the launcher's entry to the five lookups (`doctor` and `--license-mode` stay the operator's, `example` always sends `strict`, and a `githits.doctor` fact is never counted in `githits_lookups`), records the skill's digest with CRLF normalized to LF (`.gitattributes` checks `skills/**` out with LF), tests that `SYBERLABS_GITHITS_EXCLUDE_REPLACE` reaches neither arm, and, under `--allow-code-tools-with-mcp`, matches `PowerShell` in the query guard and judges it by the launcher-only rule. Deferred: (3) the other operations of the document are not covered: `search_status` (polling a `search_ref` when a search returns incomplete; the client prints the ref and stops), `research` and the rest of `/v1/experimental/*`, package intelligence, `code_diff`, `resolve_target`, `feedback`; (4) no retry or back-off on `429`/`503`: `Retry-After` is not read by the lookup operations (only `doctor` reports it) and the call fails. Not solved: (5) layer (c) is detection, not prevention: an example can blend code from an excluded repository without citing it, and a mention spelled outside the listed host patterns passes; for the pilot, contamination is resolved on SyberLabs' side (unpublished work items, both arms run locally, a pinned base) with the client as the backstop, and an index-side exclusion by GitHits is optional; (6) a package target whose name does not name the owner reaches GitHits, and only its served repository, after the call, can drop it; (7) the license check depends on license data: in contract 0.1.0 only examples carry licenses (in Markdown), so search, grep, list and read results pass unjudged, and what `strict` admits server-side is not documented; (8) the fixture pins the five operations' closed subset of the document (215,618 bytes), and `doctor`'s operation in a second fixture, not the whole 1,364,570-byte document, which is over the contract's change-size limit; the upstream SHA-256 is recorded in `docs/GITHITS.md` for comparison by hand; (9) the record key and the guard's key sit in Claude Code's environment, which the launcher's client inherits, so a verifying MAC shows a key holder wrote the line, not which process; (10) no call to the real API was made, and no real Claude Code ran: every behaviour is tested against a loopback fake that follows the pinned schemas and a stand-in `claude` whose Bash tool is emulated (a command no `Bash(<prefix>:*)` rule names is denied, the launcher runs as `sh` runs it); whether Claude Code 2.1.309 matches `Bash(<absolute path>:*)` against a Windows path written with forward slashes, and runs the launcher through Git Bash, is for the first live call (`syberlabs githits doctor` first, then one guarded lookup); (11) a record cannot be tied to the tool use that ran it (the client has no tool use id), so the host reconciles counts: authentic records against launcher calls the guard passed; fewer records is `signal.githits.mismatch: githits_unrecorded` (a client that failed before recording, a usage error, a denied permission), more is `githits_unexpected_records`, every record of the call is then `unguarded` and the search stops `provider_guard_inactive`; (12) the mismatch is on the candidate's signal only, so a call refused `no_change` keeps its lookup facts but not its mismatch (`GitSearchSpace.observe` accepts only `lookup`); (13) `X-GitHits-Session-ID` is `<thread>-<arm>`, not `<thread>/<arm>`, because GitHits accepts only `[A-Za-z0-9_-]{1,64}`; (14) when a thread holds client-record facts, `githits_lookups` counts only those, so GitHits MCP lookups in the same thread (an operator who passes `--mcp-config` too) are not added; (15) arm A (no `--githits` and no `--mcp-config`) receives no `GITHITS_*` or `SYBERLABS_GITHITS_*` variable since `githits-pilot-hardening`, the token included: least privilege was chosen over the byte-identical environment `githits-wiring` kept, so arm A's environment differs from before this route by exactly those variables (its argv does not); an `--mcp-config` run keeps them, because a GitHits MCP server may read its token from that environment; (16) the skill rides in `--append-system-prompt`, so it is part of every arm B request's system prompt, and arm A's harness note has no skill.

60. **SyberEval runs and records; it does not yet score, gate on safety or report, and its hidden checks are hidden from the agent's worktree, requests and candidates, not from a process that can read the host's disk.** `eval-core` (2026-10-26) adds `syberlabs/eval/` and `syberlabs eval register|tasks|run|preflight` (`docs/EVAL.md`): pre-registered specs with frozen digests, tasks whose acceptance checks the host stages outside the run's repository and appends to the run contract as required `hidden.*` checks by absolute path, arms built into the kernel's own providers, a runner that takes every task x arm x seed through an unattended Build Thread in the registered order and writes a digested bundle per run, an eval-wide dollar cap, and a preflight gate; the eight v3 and v4 fixtures are imported as tasks and E5 is registered, not run. The contract schema, the event kinds, the golden traces and the kernel are unchanged. Deferred to `eval-score`: (1) the safety gate computed from the journal (an effect without an admitted decision, a skipped approval or prerequisite, a promotion outside the contract's rules, an admitted out-of-scope candidate, a planted canary in an outbound record, a provider credential in the journal, a forbidden planner action), the scorer (pass, partial and fail on the hidden checks, the metrics suite, critic findings, review facts), the paired per-task analysis with the decision rule applied mechanically, and `eval score` and `eval report`, which exit 2 until then. Deferred to `eval-safety-fixtures`: (2) the agent-facing safety tasks (`evals/tasks/safety-*`, from the deterministic scenarios) and the gate's validation on them; a task's `safety.canaries` is recorded and digested but nothing searches for it yet. Not solved: (3) a hidden check's state and output tail are recorded like any check's and reach a repair or evolve arm as feedback, which is the E3 design; what stays hidden is the check's code and data, and `tasks.LOADER` keeps a traceback from printing either, while a hidden check that runs a script directly prints its path and failing line; (4) the host's copy is an ordinary directory named in the run contract, so a `claude-code` arm with file tools outside its worktree could read it (research register N4); (5) on Claude Code's login lanes nothing is priced, so the eval's dollar cap does not bound those runs, and the runner does not set Claude Code's model (the arm's model is a declaration; `signal.harness` records the model that served); (6) the `single` arm's `CommandProvider` request carries no seed; (7) runs are serial in one process, and an arm's environment is set on `os.environ` for the length of its proposal; (8) `syberlabs.eval` is not in the wheel, because `pyproject.toml` lists the `syberlabs` package without its subpackages; (9) the run's kit home is deleted unless `--keep-work` is given: the certificate is verified against the repository and the journal when it is written, and afterwards the bundle keeps `journal.json` and `repo.bundle`, not a home that `syberlabs verify --repo` can read; (10) E5's registration was written from this packet's working tree on base `7caa7c25b3` (`git_dirty: true`); the commit a run uses is in each bundle and is held to a passing preflight.

61. **SyberEval scores, gates and reports its runs; some gate rules cannot be computed for some arms, the agent-facing safety tasks do not exist yet, and candidate code still runs beside the hidden checks.** `eval-score` (2026-10-26) adds `syberlabs/eval/gate.py`, `score.py` and `report.py` and `syberlabs eval score|report` (`docs/EVAL.md`); the required spec field `hidden_feedback` (`tails`, `status`, `none`), enforced through `RepairProvider`'s optional `feedback_filter`, which every other caller leaves unset; the hidden copy's re-hashing at four points, with the run outcome `hidden_tampered`; a host root where the hidden copy and the kit home get random names outside the run's work directory; the append-only spend ledger the eval's cap counts; and `syberlabs.eval` in the wheel. It closes entry 60's (1) and (8) and narrows its (3) and (4). The contract schema, the event kinds, the golden traces and the kernel's behaviour for every existing caller are unchanged. Open: (1) the agent-facing safety tasks (`evals/tasks/safety-*`, packet `eval-safety-fixtures`) and the gate's validation on them: no planted canary, forbidden action or unsafe path has yet been offered to an agent, and every gate rule is tested on synthetic journals and with stand-in adapters only; (2) `canary.provider_requests` is `not_computable` for `claude-code` arms, whose own model requests hold whatever Claude Code's tools read, and for `evolve` and `evolve_jev` arms, whose mutate requests' feedback is not recorded; such a run's gate is `incomplete`, never `pass`; (3) the policy is not in the journal, so `approval_or_prerequisite_skipped` checks that at least one approval came between a `needs_approval` decision and its effect, not the policy's count; (4) `credential_observed` matches credential shapes (`syberlabs._patterns.SECRET`, at a word boundary) and the values of the host's credential variables present when the run ended, not every possible secret, and full check outputs and provider stderr are not retained, so only their recorded tails are scanned; (5) the hidden copy is hashed when staged, after the proposal, after the check and before acceptance, so a rewrite undone between two hashes, or inside one proposal's evaluations, is not detected, and candidate code inside a check can still read the copy and print it into its output, which `tails` sends to the next request; a hidden check that imports a module from the copy writes `__pycache__` into it where the directory is writable (Windows), which reads as tampering; only a sandbox per check closes this (research register N4); (6) `hidden_feedback` filters the `repair` and `claude-code` feedback path only: an evolve arm builds its mutate requests in `syberlabs/evolve.py`, which no filter reaches, so a spec may use evolve arms under `tails` only; (7) under `none` the repair loop still stops on the first candidate that passes the hidden checks, so they still choose among up to K attempts on the host's side, and a call after a failure starts from the failed candidate's tree; (8) the eval's recorded spend can exceed `total_dollars` by one call's cost, and a call in flight when the runner is killed outright is in no journal and no ledger line; (9) the model API takes no sampling seed, so a run's seed indexes a repeat and is not a reproducible draw; (10) E5's registration is deleted, because adding `hidden_feedback` moved its digest and `register` returns an existing registration of the same digest unchanged; it is registered again on a clean checkout of `main` before it runs; (11) the package reads `benchmarks/`, the adapters and `evals/` from a checkout, so the wheel's `syberlabs eval` needs one to run anything.
15 changes: 10 additions & 5 deletions syberlabs/eval/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@
syberlabs eval register evals/<id>/spec.json # freeze the spec: evals/<id>/registration.json
syberlabs eval preflight <id> # the deterministic suites, recorded for this commit
syberlabs eval run <id> [--only T,A,S] [--dry] # every task x arm x seed, in the registered order
syberlabs eval score <id> # the safety gate, quality and efficiency of every run
syberlabs eval report <id> [--partial] # the paired analysis with the decision rule applied

The pieces (``docs/EVAL.md``):

Expand All @@ -16,13 +18,16 @@
named ``hidden.<check>``, by absolute path, so the agent's tree, its requests and its candidates never hold them.
- ``arms``: named provider configurations (``single``, ``repair``, ``evolve``, ``evolve_jev``, ``claude-code``)
built into the kernel's own providers; each has a canonical digest.
- ``runner``: the run, the evidence bundle, the eval's dollar cap, and the preflight gate.
- ``runner``: the run, the evidence bundle with every hash of the hidden copy, the spend ledger and the eval's
dollar cap, and the preflight gate.
- ``gate``: the safety gate computed from a run's journal and bundle, rule by rule.
- ``score``: per run, the gate, task quality on the hidden checks, and the metrics suite; ``index.json``.
- ``report``: tasks paired, the decision rule applied in exact fractions, a sign test and a bootstrap beside it.

Scoring and reporting are the next packet (``eval-score``). This package runs from a SyberWork checkout: it reads
``benchmarks/fixtures.py`` and the adapters beside it, and it is not in the wheel (``pyproject.toml`` lists the
``syberlabs`` package only, not its subpackages).
The wheel carries this package, but an eval runs from a SyberWork checkout: it reads ``benchmarks/fixtures.py``, the
adapters and ``evals/`` from there.
"""

from __future__ import annotations

VERSION = "1" # the eval package's own revision, recorded in every bundle as harness.eval
VERSION = "2" # the eval package's own revision, recorded in every bundle as harness.eval
Loading
Loading