Repository navigation
Land eval-score (#146: the SyberEval safety gate, scorer, paired report, hidden_feedback and spend ledger) on top of the eval-core fixes of #148, with one hidden-copy integrity mechanism and one spend - #150
Conversation
…ity mechanism (tree_digests at four points), one strict spend ledger, gate/score/report, hidden_feedback SyberLabs-Thread: a248e101-021e-4a08-9b35-0fefe78d839a SyberLabs-Candidate: c1 SyberLabs-Provider: claude@opus-5.5
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 098a4f323a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
|
|
||
| def _evidence_files(ev: Evidence) -> list[Path]: | ||
| return sorted(p for p in ev.folder.iterdir() if p.is_file() and p.name not in DERIVED + BINARY) |
There was a problem hiding this comment.
Scan candidate contents for credential leaks
When a provider writes a credential-shaped string into a changed source file, that content is retained in repo.bundle, but _evidence_files excludes that file and the runner’s credential-value scan checks only JSON files. The gate can therefore report pass for a bundle containing a leaked credential; scan the retained candidate diff or tree as well. Secrets are meant to stay in the environment.
AGENTS.md reference: AGENTS.md:L210-L211
Useful? React with 👍 / 👎.
| for folder, dirs, files in os.walk(root): | ||
| for name in dirs: | ||
| try: | ||
| os.chmod(os.path.join(folder, name), stat.S_IRWXU) |
There was a problem hiding this comment.
Avoid chmodding candidate-created symlink targets
On POSIX with the default keep_work=False, if a hidden check imports candidate code that creates a symlink inside the staged copy to an external directory owned by the same user, os.walk lists it in dirs and os.chmod follows it. Cleanup then changes permissions on that external directory even though it should only release the staged copy; skip symlinks before chmod. This is possible because checks execute candidate code with host permissions, not in a sandbox.
AGENTS.md reference: AGENTS.md:L212-L215
Useful? React with 👍 / 👎.
| found = read_json(folder / "score.json", "score_unreadable") | ||
| if found.get("bundle_digest") != bundle_digest: | ||
| unscored.append(item.run_id) | ||
| continue |
There was a problem hiding this comment.
Validate score files against their indexed digests
If someone edits score.json after scoring but leaves its bundle_digest unchanged, this check accepts it and the report uses its altered metrics or gate result. index.json already records the score digest, but the report never compares against it, so a modified score can change the reported experiment outcome without an error.
Useful? React with 👍 / 👎.
| if bundle.get("spec_digest") != registration["spec_digest"]: | ||
| raise Rejected("bundle_mismatch", f"{path} was written under another spec digest") | ||
| found = score_run(root, folder) | ||
| write_json(folder / "score.json", found) |
There was a problem hiding this comment.
Allow scored bundles when resuming an eval
After eval score writes score.json beside each bundle, EvalRun.verify still requires the directory contents to match bundle.files exactly and treats score.json as an unexpected file. Since eval run verifies existing bundles before skipping them, trying to continue or rerun after scoring stops with bundle_tampered; exclude this derived file from that verification.
Useful? React with 👍 / 👎.
|
Scope: owner direction in session 2026-10-28 (SyberFactory, packet Packet
Reconciliation decisions, as applied
Other docs changes (
|
| Check | State | Time | Output |
|---|---|---|---|
| conformance | passed | 21.2 s | matched 23 golden traces |
| critic.rules | passed | 1.3 s | "no findings over 19 changed path(s)" |
| scenarios | passed | 0.4 s | digest b41f7cbb… |
| tests | passed | 393.0 s | Ran 1229 tests, OK (skipped=26, expected failures=4) (Windows host, 8 shards) |
certify c1: result 098a4f323a (tree a808c07643) on repo-change.v5, class R; four checks passed; chain head 618b8aad6c at 14 events.
- Key
4b6ae2deea: before signing, I checked that the public key of$SYBERLABS_CERTIFICATE_SEEDhas prefix4b6ae2deeaand matches the factory'stest-integrity-2-74e7d4ac.cert.pub. The written.pubis byte-identical to it.
verify against the factory .pub with --repo .:
- signature ok,
ed25519 4b6ae2deea; - repository ok: base
4c65aa0b8eand result present, 19 changed paths, the diff digest matches, and the target is at098a4f323a; - journal ok: 14 of 14 events;
- verdict pass.
PR CI: all 25 checks on Land eval-score (#146: the SyberEval safety gate, scorer, paired report, hidden_feedback and spend ledger) on top of the eval-core fixes of #148, with one hidden-copy integrity mechanism and one spend #150 at 098a4f323a pass (gh pr checks 150): test 3.11/3.12/3.13, windows, compose, G8 and G8 strict (each for push and pull_request), admit, policy, G2, G3, G4 and G6 on both events, and G5.
Claim ceiling: implemented and tested, locally on Windows and on CI (Linux 3.11–3.13 and Windows), with the simulated and stand-in adapters. Not measured: no eval with a model has been run, E5 included.
Known limitations
- Everything in Add the SyberEval safety gate, scorer and paired report, with the hidden_feedback spec field, hidden-check integrity re-hashing and an append-only spend ledger. Each is computed from the journal and t #146's limitations still holds:
- the evolve path is unfiltered, so evolve arms run under
tailsonly; - the hash window;
nonestill selects;- seeds index repeats;
- the gate reads only recorded tails.
- the evolve path is unfiltered, so evolve arms run under
- Everything in Rebuild of #144 onto main: close eleven SyberEval core holes (author: Seth Carlson) #148's pinned gaps still holds except gap 13 under
status/none. The five are listed in GAPS 62 (12). - Comparing against the staged copy, not the registered digest, means a pytest
.pytest_cacheadded during the checks is not a change; that is Rebuild of #144 onto main: close eleven SyberEval core holes (author: Seth Carlson) #148's tolerance. The staged copy itself is verified against the registered digest bystage_hiddenbefore it is written. The bundle records both digests. - The ledger has no cross-check against bundles, so a deleted line is not detected. This was Rebuild of #144 onto main: close eleven SyberEval core holes (author: Seth Carlson) #148's finding 2, last bullet: detection, not prevention, as for bundles.
- The report now refuses a ledger with any uncountable line (
eval_spend_unknown), including a run whose journal could not be read. Until the operator reconciles that line by hand, the report cannot be produced.
Unresolved questions
- The report's refusal of an uncountable ledger: does the lead accept it, or want a report that states the spend as unknown?
- The scratch branches
syberlabs/a248e101-ciandsyberlabs/a248e101-biteare left for the orchestrator to delete.
Next permissible packet: eval-safety-fixtures (GAPS 62 (1)). After merge, the lead registers E5 on a clean checkout of main, runs eval preflight, and runs it.
Independent verification: #150 (
|
| # | Decision | Result |
|---|---|---|
| 1 | Hidden integrity | Applied. tasks.tree_digests and hidden_changes are the only integrity code. Integrity, staged_files and copy_digest are gone (grep finds nothing). runner.hidden_hash runs at staged, after_propose, after_check and before_accept. A change gives not_acceptable / hidden_tampered, hidden_intact: false and hidden_changed. Every hash is in bundle.hidden_integrity. The kit home and the hidden copy each get a random 128-bit name under --hidden-root, outside the work directory, and the copy is staged read-only (tasks._read_only). The before-accept hash: thread.accept is called in one place only (runner.py:576), straight after intact("before_accept"), and Thread.accept runs no checks (build.py:1228). The repair-feedback checks are covered by after_propose, not after_check (finding 3). |
| 2 | Spend | Applied, with findings 1 and 2. There is one ledger, evals/<id>/spend.jsonl. runs/failed.jsonl, FAILED, EvalRun.failed and _failed are removed. _charge writes the line, flushes and fsyncs it before anything is deleted, on both the completed and the aborted path (it runs before the .partial rmtree and before the finally cleanup of the work directory, kit home and copy). The ceiling is computed as min(per-run, total - spent). ledger() uses spec.loads, and ledger_dollars rejects bool, NaN, inf and negative values. #148's test_a_run_that_fails_after_it_spent_is_charged_against_the_cap is unchanged and passes. |
| 3 | The rest of #146 | Applied. hidden_feedback is a required spec field and part of the spec's digest. feedback_filter defaults to None, and the request stays byte-identical (FeedbackHook.test_without_a_filter…). An evolve arm is refused under any mode but tails. Every gate.py rule returns pass, fail, not_computable with a reason, or not_applicable; evaluate gives incomplete on any not_computable, never pass. score.py and report.py use Fraction, an exact sign test and a seeded bootstrap, and refuse runs_missing without --partial (report.py:179). The CLI has score, report --partial and --hidden-root. pyproject.toml adds syberlabs.eval. E5 has hidden_feedback: tails, and its registration.json is deleted. #148's lock, EvalRun.verify, per-run preflight, thread_label, finite numbers, run-id collision check and strict read_json are all still there, and their tests pass. |
| 4 | Small fixes | Applied. Gap 16 has @skipUnless(hasattr(signal, "SIGKILL")) above @expectedFailure (test_eval_adversarial.py:267). spec.check refuses two githits arms with colliding labels. The test covers a.b/a-b and the 16-character prefix case, and also shows that the same names pass when only one arm has githits. |
| 5 | GAPS | Applied. 61 is still #147's TypeSafe Jev entry, and eval-score is now 62. EVAL.md refers to 62 at lines 3 and 216. Entry 62 drops the __pycache__ claim, and EVAL.md:102 cites PYTHONDONTWRITEBYTECODE=1. |
| 6 | Gap 13 | Applied. It is split in two. tails stays expectedFailure. The new status/none test runs both modes as subtests. Each first asserts that both canaries are in the journal's check tail and that a second request followed, so it cannot pass vacuously. It then asserts that neither canary is in request 2. The test passes. |
Scope: 19 files, 2,650 lines added and 205 removed, 257,543 diff bytes. No contract schema, event kind, golden trace, conformance/ or syberlabs/cli.py change. The only spec/ file touched is spec/GAPS.md. Since 4c65aa0b8e, main has gained only #149 (docs/EVOLUTION.md, syberlabs/jev_api.py, tests/test_jev_api.py).
Open question 1 (should eval report refuse an uncountable ledger?): yes, that is a sound default.
- The report states the eval's spend and whether the cap held. Leaving a line out of the sum would understate both, and a published instrument should fail closed, as
eval runalready does on the same line. - Two refinements:
- extend the refusal to
unknown_modelslines (finding 1), so the report and the runner agree on what "uncountable" means; - later, if needed, a
--partialreport could print the spend as "unknown (lines …)" instead of refusing, but only under the partial label.
- extend the refusal to
Tests and CI
- Local, fresh clone at
098a4f323a:python -m unittest tests.test_eval_core tests.test_eval_runner tests.test_eval_score tests.test_eval_adversarial tests.test_providers tests.test_gaps tests.test_packagegaveRan 88 tests in 138.080s,OK (skipped=1, expected failures=4). The skip is gap 16 on Windows. - Local:
python -m conformance.rungavematched 23 golden traces, exit 0. - Full CI run 40783915488, on scratch branch
56edcd8018, whose treea808c07643is identical to the head's:- attempt 2: every job succeeded;
- attempt 1:
windowsfailed witherrors=1(Ran 1229 tests,FAILED (errors=1, skipped=26, expected failures=4)). The failure istest_console.ConsolePreview.test_preview_appends_nothing_for_anyone, aConnectionAbortedError: [WinError 10053]inurllibagainst the loopback console server. The packet does not touchtest_console, which does not use the eval package. G8 failed only because it needs every leg. I agree it is unrelated.
- Bite run 40783915123, on
ec1f14e1ab: four test files on top of4c65aa0b8e, and those files are identical to the head's (git diff --quiet). Result: failure, as intended. Every Linux leg and G8 strict:Ran 1212 tests,FAILED (failures=5, errors=31, skipped=2, expected failures=5).windows: the same, withskipped=26, expected failures=4.composepasses. gh pr checks 150: all 25 checks pass.
Certificate
syberlabs verify --public test-integrity-2-74e7d4ac.cert.pub --repo . eval-score-2-a248e101.cert.json, run in the fresh clone:
signature ok ed25519 4b6ae2deea, thread a248e101, candidate c1, result098a4f323a(repo-change.v5, class R);repository ok: base and result present, trees match, 19 changed paths, and the diff digest matches;journal skipped, because a fresh clone has no.syberlabsjournal;- verdict pass (journal not checked).
The receipt's .cert.pub is byte-identical to the factory key.
Rebuild preview
I cherry-picked 098a4f323a onto origin/main at 5ebbbce6e0, locally only.
- It applied cleanly, with no conflicts.
- 19 files, 2,650 lines added and 205 removed. Tree
f167ad6c03. - It differs from the head only in Close the independent verification findings on #147: re-target the pre-registered jev_api measurement to the v3 fixtures with four paid repeats and a sourced cost ceiling, correct the recorded crossov #149's three Jev files.
Nothing was pushed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… report refuses an unpriced ledger line, (#151) Objective: Close the independent verifier's findings on #150 (eval-score-3): the report refuses an unpriced ledger line, a huge integer in the ledger is eval_spend_unknown, the hidden_tampered rule requires the runner's hash points, and EVAL.md states the after_check coverage and the spend ceiling exactly Class: R Agent-platform: claude@opus-5.5 Admitted-by: sykosyber Reviewed-by: rules@2 (critic check critic.rules: high 0, medium 0, low 0) Evidence: tested Accepted in SyberLabs thread `c73912bf-1c64-4343-a513-71db9b15311d` as candidate `c1` (commit `08252d39be00b459dfef6b08fffaac92c0b7ad53`). Host checks on that tree: conformance passed, critic.rules passed, scenarios passed, tests passed. Reviewed-by: claude (independent verifier session, SyberFactory workflow) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Objective: Land eval-score (#146: the SyberEval safety gate, scorer, paired report, hidden_feedback and spend ledger) on top of the eval-core fixes of #148, with one hidden-copy integrity mechanism and one spend ledger
Class: R
Agent-platform: claude@opus-5.5
Admitted-by: sykosyber
Reviewed-by: rules@2 (critic check critic.rules: high 0, medium 0, low 0)
Evidence: tested
Accepted in SyberLabs thread
a248e101-021e-4a08-9b35-0fefe78d839aas candidatec1(commit098a4f323a3e942ef1660ea5e4089e9258591f31).Host checks on that tree: conformance passed, critic.rules passed, scenarios passed, tests passed.
Reviewed-by: claude (independent verifier session, SyberFactory workflow)