Repository navigation
Add the SyberEval safety gate, scorer and paired report, with the hidden_feedback spec field, hidden-check integrity re-hashing and an append-only spend ledger. Each is computed from the journal and t - #146
Conversation
…n_feedback, hidden-copy re-hashing and the spend ledger SyberLabs-Thread: eff81f77-a4b1-4c44-906a-0d57bf9dbced SyberLabs-Candidate: c1 SyberLabs-Provider: claude@opus-5.5
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ae117c1f72
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| """Which host credential variables (by name) have a value in which evidence file of ``folder``; never a value.""" | ||
| names = sorted(k for k, v in os.environ.items() if CREDENTIAL_NAME.search(k) and _looks_secret(v)) | ||
| found = [] | ||
| for path in sorted(p for p in folder.iterdir() if p.is_file() and p.suffix == ".json"): |
There was a problem hiding this comment.
Scan bundled candidate trees for credential values
When a provider that inherits a host credential writes its value into a candidate file, this scan checks only .json evidence; the value is serialized in repo.bundle, which the gate also excludes, so credential_observed can pass while the run artifact contains the credential. Scan the bundled candidate contents too, or reject such candidates before scoring.
AGENTS.md reference: AGENTS.md:L210-L215
Useful? React with 👍 / 👎.
| if found.get("bundle_digest") != bundle_digest: | ||
| unscored.append(item.run_id) |
There was a problem hiding this comment.
Verify score hashes before using them
If a score.json is edited after eval score, this check accepts it whenever its embedded bundle_digest still matches; it never compares the score's current hash with the expected score hash recorded in index.json. Altered metrics or gate results can therefore drive the report's decision despite the indexed score digest disagreeing. Verify the score against the index before consuming it.
AGENTS.md reference: AGENTS.md:L198-L199
Useful? React with 👍 / 👎.
| write_json(partial / "bundle.json", bundle) | ||
| except BaseException: | ||
| self._charge(item, thread, "completed", None, budget["dollars"]) |
There was a problem hiding this comment.
Recover spend from partial bundles before charging
If the process is killed after writing bundle.json but before _charge completes, the run's spend remains in <run>.partial; resume ignores that directory when calculating spend and deletes it before charging, so the same run can be retried without counting the earlier dollars and exceed total_dollars. Recover the partial's recorded spend before deleting it, or make charging durable before writing the bundle.
Useful? React with 👍 / 👎.
|
Scope: owner direction in session 2026-10-26 (SyberFactory, packet Packet
What changed
Files (18):
Size: 236,930 diff bytes. There is no change to the contract schema, the event kinds, the golden traces, Evidence
Tamper detection (
Spend ledger (
Gate rules.
Every rule has a failing and a passing test on synthetic journals ( End-to-end report. A local probe run of the same tiny simulated eval (pricing and slugify, single vs repair, seed 1) printed:
The committed test asserts the same relations, not these numbers: the simulated model's draws are not pinned by the tests. Commands and results
Claim ceiling: implemented and tested, on CI (Linux 3.11–3.13, Windows) and locally, with the simulated adapter and stand-in adapters. Not measured: no eval with a model has been run, E5 included, and the gate has not been validated on agent-facing safety tasks. Known limitations
Overlap with #144 (read from
|
| File | Duplicates #144 | Would conflict |
|---|---|---|
syberlabs/eval/runner.py |
Tamper detection: #144 compares tasks.tree_digests of the copy before and after the check, and sets status: not_acceptable, reason: hidden_tampered plus hidden_intact / hidden_changed. This PR hashes at four points against the registered task digest, sets status: hidden_tampered, and records every hash in hidden_integrity. Spend for raised runs: #144 has runs/failed.jsonl beside bundle sums; this PR has evals/<id>/spend.jsonl for every attempt, and the cap counts the ledger. |
The module docstring; run(), which #144 splits into run/_run under its lock and re-checks preflight per run; the run_one body (staging, the arms.build call that both extend: thread_label there, hidden_feedback/sent here, and the accept condition); the except BaseException handler (_failed vs _charge); and the imports. #144's lock, verify/bundle_tampered, per-run preflight, githits session label and strict read_json have no counterpart here. This PR's ledger, ceiling, host root, feedback record and credential scan have none there. |
syberlabs/eval/tasks.py |
Both add a copy re-hash helper: #144 tree_digests and hidden_changes (symlinks named, .pytest_cache tolerated); this PR staged_files, copy_digest and release, plus read-only staging. |
The module docstring paragraph both rewrite. #144's materialize --force and hidden_path touch other functions. |
syberlabs/eval/spec.py |
None. | The module docstring's cross-field-rules sentence, which both rewrite; check(), where both append rules (#144: run-id collisions and finite numbers; this PR: evolve arms only under tails). The hidden_feedback schema field is unique to this PR. |
syberlabs/eval/arms.py |
None. | The build() signature, which both extend (thread_label vs hidden_feedback, sent); imports. |
- Close eleven SyberEval core holes found by adversarial verification and pin five design gaps for the GitHits pilot (stacked on #137) #144's tests.
tests/test_eval_adversarial.pywould needhidden_feedbackin its specs once this lands, because the field is required. - The two outcome schemas differ. Close eleven SyberEval core holes found by adversarial verification and pin five design gaps for the GitHits pilot (stacked on #137) #144 uses status
not_acceptablewith reasonhidden_tampered; this PR uses statushidden_tampered. Reconciliation should pick one. - Design points from Close eleven SyberEval core holes found by adversarial verification and pin five design gaps for the GitHits pilot (stacked on #137) #144 worth keeping in a reconciliation:
- its lock against two runners;
- bundle verification at load;
tree_digestsnaming symlinks:staged_fileshere follows them;- tolerating
.pytest_cache.
- Exact decision rule. This report already decides in
fractions.Fraction. TheStatisticstest is Close eleven SyberEval core holes found by adversarial verification and pin five design gaps for the GitHits pilot (stacked on #137) #144's tie example, and the float-per-task-means mutation is caught.
Unresolved questions
- Does the lead accept that evolve arms are refused under
status/none, or want a follow-up packet that allowsevolve.pyfor a hook? - Does the lead accept moving the kit home under the host root together with the hidden copy?
- Should
canary.recordstreat the journal's check tails as outbound (current behaviour), or count only lookup records, publish arguments and provider requests? - How should this PR and Close eleven SyberEval core holes found by adversarial verification and pin five design gaps for the GitHits pilot (stacked on #137) #144 be reconciled?
- The scratch branches
syberlabs/eff81f77-ciandsyberlabs/eff81f77-biteare left for the orchestrator to delete.
Next permissible packet: eval-safety-fixtures, the agent-facing safety tasks and the gate's validation on them (GAPS 61 (1)). After merge, the lead registers E5 on a clean checkout of main, runs eval preflight, and then runs it. That run is the paid act, about $10 at most.
…rt, hidden_feedback and spend ledger) on (#150) Objective: Land eval-score (#146: the SyberEval safety gate, scorer, paired report, hidden_feedback and spend ledger) on top of the eval-core fixes of #148, with one hidden-copy integrity mechanism and one spend ledger Class: R Agent-platform: claude@opus-5.5 Admitted-by: sykosyber Reviewed-by: rules@2 (critic check critic.rules: high 0, medium 0, low 0) Evidence: tested Accepted in SyberLabs thread `a248e101-021e-4a08-9b35-0fefe78d839a` as candidate `c1` (commit `098a4f323a3e942ef1660ea5e4089e9258591f31`). Host checks on that tree: conformance passed, critic.rules passed, scenarios passed, tests passed. Reviewed-by: claude (independent verifier session, SyberFactory workflow) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Objective: Add the SyberEval safety gate, scorer and paired report, with the hidden_feedback spec field, hidden-check integrity re-hashing and an append-only spend ledger. Each is computed from the journal and the run bundle, with every rule the journal cannot support reported not_computable.
Class: R
Agent-platform: claude@opus-5.5
Admitted-by: sykosyber
Reviewed-by: rules@2 (critic check critic.rules: high 0, medium 0, low 0)
Evidence: tested
Accepted in SyberLabs thread
eff81f77-a4b1-4c44-906a-0d57bf9dbcedas candidatec1(commitae117c1f7269f896834386d9960a246f2e8f4991).Host checks on that tree: conformance passed, critic.rules passed, scenarios passed, tests passed.