Repository navigation
Rebuild of #144 onto main: close eleven SyberEval core holes (author: Seth Carlson) - #148
Conversation
…pin five design gaps Independent verification of the eval core (#137) before the GitHits A/B pilot. Every finding has a test in tests/test_eval_adversarial.py; each fixed one failed before its fix. Fixes are confined to syberlabs/eval/*. Fixed: - run ids could collide (task and arm names may hold dots): one cell's bundle overwrote another's; spec.check now refuses colliding run ids - a run that raised after spending was charged nothing, so every retry spent again under the cap; runs/failed.jsonl records its spend and the cap counts it - candidate code could rewrite the host's copy of the hidden checks and get a later candidate accepted; the runner refuses to accept on a changed copy (not_acceptable, hidden_tampered) - every run of a githits arm shared X-GitHits-Session-ID s1-b and agent syberfactory/b; the run's thread and the arm's name now label them - NaN passed every numeric bound of the spec schema (a NaN total_dollars disabled the cap); numbers must be finite and read_json is strict (no NaN, Infinity or repeated keys) - the preflight gate was checked once per invocation; it is checked before every run - bundles were never verified: edited evidence or dollars, a bundle in another run's directory and a .partial were read as done; EvalRun.verify checks them (bundle_tampered) - two runners could run one eval at once and pass the cap together; runs/.lock - a fixture's base honoured the host's global git ignore rules (an empty base on a host ignoring *.py); git add --force - write_task wrote hidden files named ../ or absolute outside the task Pinned as expectedFailure (design decisions): a candidate that exits 0 at import passes every hidden check; a candidate can print its hidden check into the next repair request; a claude-code arm's served model is never compared with its declared model; repo: task repositories hold commits after the base; a runner killed outright charges nothing to the cap. Rebuilt onto main from #144 by the SyberLabs lead's branch operator. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Independent verification of #148 (head
|
| # | Fix | Code | Test | Result |
|---|---|---|---|---|
| 1 | Run-id collisions refused | spec.py:218-224 (Counter over run_id) |
test_two_cells_that_would_share_a_run_id_are_refused |
Pass; matches the claim |
| 2 | Spend of a raised run charged | runner.py:446-448,461-477 (_failed), :318, :332 (eval_spend_unknown) |
test_a_run_that_fails_after_it_spent_is_charged_against_the_cap |
Pass. Narrower than it sounds: see finding 2 |
| 3 | Hidden-copy tamper detection | tasks.py:385-411, runner.py:388,406-413 |
test_a_candidate_that_rewrites_…, test_only_pytest_s_own_cache_… |
Pass. Detection only, as stated (finding 5) |
| 4 | A GitHits session per run | arms.py:93-95,133, runner.py:395-396 |
test_each_run_of_the_githits_arm_has_its_own_githits_session |
Pass. Its session half is already on main (finding 3) |
| 5 | Non-finite numbers refused | spec.py:162-163, :244-253 (parse_constant) |
test_a_number_that_is_not_finite_is_not_a_valid_spec |
Pass |
| 6 | Preflight gate before every run | runner.py:341-347 |
test_the_preflight_gate_holds_for_every_run_not_only_the_first |
Pass. Applies when require_preflight (the default) |
| 7 | EvalRun.verify, bundle_tampered, .partial skipped |
runner.py:224-275 |
Bundles.* (4 tests) |
Pass. Unsigned, as stated |
| 8 | runs/.lock, eval_running |
runner.py:64-75,182-204,310 |
test_a_second_runner_cannot_run_the_same_eval_at_once |
Pass on Windows (local and CI) and Linux (CI) |
| 9 | git add -A --force |
tasks.py:322 |
test_a_fixture_s_base_holds_every_fixture_file_… |
Pass |
| 10 | Hidden paths confined under hidden/ |
tasks.py:379-382,417 |
test_write_task_keeps_hidden_files_inside_the_task |
Pass |
| 11 | Strict read_json (duplicate keys) |
spec.py:238-253 |
test_a_spec_with_a_duplicate_key_is_refused |
Pass |
Pinned gaps (12–16), with markers cleared in a scratch run
| # | Test | Fails at | Matches the claim |
|---|---|---|---|
| 12 | …exits_at_import… |
line 425, assertNotEqual(status, "accepted") |
Yes |
| 13 | …carry_a_hidden_check… |
line 439, assertNotIn(CANARY_CODE, second) |
Yes |
| 14 | …served_by_another_model… |
line 501, assertFalse(… not in served) |
Yes |
| 15 | …repo_task_s_repository… |
line 527, assertNotEqual(returncode, 0) |
Yes |
| 16 | …killed_midway… |
Windows: line 275 AttributeError (no SIGKILL); POSIX: line 278 |
Windows: no (finding 4) |
Tests
- Local (fresh clone, Windows, Python 3.13):
tests.test_eval_adversarial tests.test_eval_core tests.test_eval_runner tests.test_gapsran 50 tests, OK (5 expected failures).python -m conformance.runmatched 23 golden traces, exit 0. - CI on
92ed764420: all 25 check runs pass, across the push and pull_request runs.- The full suite is 1200 tests on 3.11, 3.12 and 3.13, on strict 3.12 and on Windows. All are OK with 5 expected failures; skips are 2 on Linux and 25 on Windows. The base had 1180 tests, so the 20 new tests are present.
- Conformance matched 23 golden traces.
admitis acceptable (all_checks_passed, ruleadmission.passed). It moved a local ref only; nothing was pushed or merged.
Merge preview: #146 (ae117c1f72) onto this head
A local trial with no push, then aborted, conflicted in 5 files:
spec/GAPS.md: 1 hunk. Entry 61 appears twice: Add a TypeSafe-backed branch policy jev_api behind the BranchPolicy seam that asks TypeSafe Jev to choose among the local JevPolicy's offered actions from ids and numbers only, validates every answer, #147's TypeSafe Jev entry, now on main, and Add the SyberEval safety gate, scorer and paired report, with the hidden_feedback spec field, hidden-check integrity re-hashing and an append-only spend ledger. Each is computed from the journal and t #146's eval-score entry.syberlabs/eval/arms.py: 2 hunks.githits_armandbuild(…, thread_label=)against Add the SyberEval safety gate, scorer and paired report, with the hidden_feedback spec field, hidden-check integrity re-hashing and an append-only spend ledger. Each is computed from the journal and t #146'shidden_view/feedback_filterandbuild(…, hidden_feedback=, sent=).- The
claude-codereturn: the githits options here against Add the SyberEval safety gate, scorer and paired report, with the hidden_feedback spec field, hidden-check integrity re-hashing and an append-only spend ledger. Each is computed from the journal and t #146'sprovider.feedback_filter.
syberlabs/eval/runner.py: 10 hunks.- The module docstring (×2): the tamper and cap prose.
_exclusiveagainstIntegrity,recorded_spendandcredential_scan.- The
spentcomputation (bundles plusfailed.jsonl, againstself.spent(done)). - The
eval_spend_unknowntest. - The per-run preflight against the
ceilingcomputation. - Staging:
tree_digestsagainstIntegrity.take("staged"). - The
arms.buildcall (thread_labelagainsthidden_feedback/sent). - Accept and tamper:
hidden_changesagainstintegrity.take("after_check"/"before_accept"). - The
except BaseExceptionblock (_failedagainst_charge).
syberlabs/eval/spec.py: 1 hunk. The module docstring: finite numbers, run ids and strictread_jsonhere, against the evolve-only-under-tailsrule in Add the SyberEval safety gate, scorer and paired report, with the hidden_feedback spec field, hidden-check integrity re-hashing and an append-only spend ledger. Each is computed from the journal and t #146.syberlabs/eval/tasks.py: 1 hunk. The module docstring. Underneath it, bothcopy_digest/release(Add the SyberEval safety gate, scorer and paired report, with the hidden_feedback spec field, hidden-check integrity re-hashing and an append-only spend ledger. Each is computed from the journal and t #146) andtree_digests/hidden_changes/hidden_path(Rebuild of #144 onto main: close eleven SyberEval core holes (author: Seth Carlson) #148) auto-merge side by side. That is two integrity mechanisms to reduce to one.- Semantic overlap outside the conflict markers.
runs/failed.jsonlagainstevals/<id>/spend.jsonl, and the tamper outcome shapes (finding 1). Add the SyberEval safety gate, scorer and paired report, with the hidden_feedback spec field, hidden-check integrity re-hashing and an append-only spend ledger. Each is computed from the journal and t #146 also editstests/test_eval_runner.pyandtests/test_eval_core.py, which merge cleanly with this PR's new test file. The adversarial tests' assertions would need re-pointing to whichever shape survives.
Reviewed-by: claude (independent verifier session, SyberFactory workflow)
…rt, hidden_feedback and spend ledger) on (#150) Objective: Land eval-score (#146: the SyberEval safety gate, scorer, paired report, hidden_feedback and spend ledger) on top of the eval-core fixes of #148, with one hidden-copy integrity mechanism and one spend ledger Class: R Agent-platform: claude@opus-5.5 Admitted-by: sykosyber Reviewed-by: rules@2 (critic check critic.rules: high 0, medium 0, low 0) Evidence: tested Accepted in SyberLabs thread `a248e101-021e-4a08-9b35-0fefe78d839a` as candidate `c1` (commit `098a4f323a3e942ef1660ea5e4089e9258591f31`). Host checks on that tree: conformance passed, critic.rules passed, scenarios passed, tests passed. Reviewed-by: claude (independent verifier session, SyberFactory workflow) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rebuilt commit:
92ed764420ec6c14633fb23a9c78078df3c1a6f6(author Seth Carlson), cherry-picked from35b8ce311a60a1abbcadb60f2a80d5d1bc82d186onto maind85f3d2825484fa5ee246585a6f43e3ad4114723with no conflicts. Its diff against main is byte-identical to #144's own diff (same five files, same--full-indexpatch, same stable patch-ida98987dc8bf17cc21d1a7294733363f2ffaa78c7). The only change to the commit message is one added line: "Rebuilt onto main from #144 by the SyberLabs lead's branch operator."#144's body, quoted in full:
🤖 Generated with Claude Code
Reviewed-by: claude (independent verifier session, SyberFactory workflow)