Repository navigation
Close eleven SyberEval core holes found by adversarial verification and pin five design gaps for the GitHits pilot (stacked on #137) - #144
Closed
sdcarlson wants to merge 1 commit into
Conversation
…pin five design gaps Independent verification of the eval core (#137) before the GitHits A/B pilot. Every finding has a test in tests/test_eval_adversarial.py; each fixed one failed before its fix. Fixes are confined to syberlabs/eval/*. Fixed: - run ids could collide (task and arm names may hold dots): one cell's bundle overwrote another's; spec.check now refuses colliding run ids - a run that raised after spending was charged nothing, so every retry spent again under the cap; runs/failed.jsonl records its spend and the cap counts it - candidate code could rewrite the host's copy of the hidden checks and get a later candidate accepted; the runner refuses to accept on a changed copy (not_acceptable, hidden_tampered) - every run of a githits arm shared X-GitHits-Session-ID s1-b and agent syberfactory/b; the run's thread and the arm's name now label them - NaN passed every numeric bound of the spec schema (a NaN total_dollars disabled the cap); numbers must be finite and read_json is strict (no NaN, Infinity or repeated keys) - the preflight gate was checked once per invocation; it is checked before every run - bundles were never verified: edited evidence or dollars, a bundle in another run's directory and a .partial were read as done; EvalRun.verify checks them (bundle_tampered) - two runners could run one eval at once and pass the cap together; runs/.lock - a fixture's base honoured the host's global git ignore rules (an empty base on a host ignoring *.py); git add --force - write_task wrote hidden files named ../ or absolute outside the task Pinned as expectedFailure (design decisions): a candidate that exits 0 at import passes every hidden check; a candidate can print its hidden check into the next repair request; a claude-code arm's served model is never compared with its declared model; repo: task repositories hold commits after the base; a runner killed outright charges nothing to the cap. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
This was referenced Oct 27, 2026
Member
sykosyber
added a commit
that referenced
this pull request
Oct 28, 2026
…rebuild of #144) (#148) Independent verification of the eval core (#137) before the GitHits A/B pilot. Every finding has a test in tests/test_eval_adversarial.py; each fixed one failed before its fix. Fixes are confined to syberlabs/eval/*. Fixed: - run ids could collide (task and arm names may hold dots): one cell's bundle overwrote another's; spec.check now refuses colliding run ids - a run that raised after spending was charged nothing, so every retry spent again under the cap; runs/failed.jsonl records its spend and the cap counts it - candidate code could rewrite the host's copy of the hidden checks and get a later candidate accepted; the runner refuses to accept on a changed copy (not_acceptable, hidden_tampered) - every run of a githits arm shared X-GitHits-Session-ID s1-b and agent syberfactory/b; the run's thread and the arm's name now label them - NaN passed every numeric bound of the spec schema (a NaN total_dollars disabled the cap); numbers must be finite and read_json is strict (no NaN, Infinity or repeated keys) - the preflight gate was checked once per invocation; it is checked before every run - bundles were never verified: edited evidence or dollars, a bundle in another run's directory and a .partial were read as done; EvalRun.verify checks them (bundle_tampered) - two runners could run one eval at once and pass the cap together; runs/.lock - a fixture's base honoured the host's global git ignore rules (an empty base on a host ignoring *.py); git add --force - write_task wrote hidden files named ../ or absolute outside the task Pinned as expectedFailure (design decisions): a candidate that exits 0 at import passes every hidden check; a candidate can print its hidden check into the next repair request; a claude-code arm's served model is never compared with its declared model; repo: task repositories hold commits after the base; a runner killed outright charges nothing to the cap. Rebuilt onto main from #144 by the SyberLabs lead's branch operator. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz
Member
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Independent adversarial verification of #137 (the SyberEval core) before the GitHits A/B pilot runs on it. Every finding below has a test in the new
tests/test_eval_adversarial.py, run against real Build Threads with stand-in models (no model, no network). Each test for a fixed bug failed before its fix. A test markedexpectedFailurepins a gap that needs a design decision: I left that behaviour unchanged, and the test fails at exactly the assertion it names.Fixes are confined to
syberlabs/eval/*.docs/EVAL.mdandspec/GAPS.mdare not edited; the last section lists the sentences they need.Findings
<task>.<arm>.s<seed>, and both names may contain dots. Taskpwith armx.rand taskp.xwith armrboth givep.x.r.s1. The second run overwrites the first one's bundle, and on resume both cells read as done from one bundle: one run is missing and another is counted twice.test_two_cells_that_would_share_a_run_id_are_refusedspec.checkrefuses colliding run ids, soregisterandrunrefuse toototal_dollars. A Ctrl-C, a harness error or a certificate that fails to verify after the model calls leaves no bundle, so each retry spent again under a cap that recorded nothing.test_a_run_that_fails_after_it_spent_is_charged_against_the_capruns/failed.jsonlrecords the host-observed dollars of each run that raised, and the cap counts them. An unreadable spend stops the eval (eval_spend_unknown)__main__.path,/proc/self/cmdline). c1 rewrites every check topass, c2 changes nothing that matters, and c2 is accepted. EVAL.md's "Not editable" held only for the candidate's own tree.test_a_candidate_that_rewrites_the_host_copy_of_a_check_is_not_acceptednot_acceptablewith reasonhidden_tampered, plushidden_intactandhidden_changed. pytest's own.pytest_cacheis tolerated; a plantedconftest.pyor__pycache__is not. A rewrite that is undone before the checks end is not detected (see 12 and 13)githitsarm shares one GitHits session.arms.buildpassed the provider neitherthread_labelnorarm, so every run of every task and seed sentX-GitHits-Session-ID: s1-bandX-GitHits-Agent: syberfactory/b.test_each_run_of_the_githits_arm_has_its_own_githits_session<thread id>-<arm>and the agent issyberfactory/<arm name>NaNpasses the spec schema. It passesminimum,maximumandexclusiveMinimum, because every comparison with it is false. A spec withtotal_dollars: NaNregisters and its cap never stops, and aNaNorInfinitythreshold makes the decision vacuous.test_a_number_that_is_not_finite_is_not_a_valid_specread_jsonrefusesNaNandInfinityrunstarted. A checkout moved or edited mid-eval (git pull, or an edited adapter, which is re-read from disk on every call) ran every later run under a stale preflight.test_the_preflight_gate_holds_for_every_run_not_only_the_firstoutcome.dollars(which bought room under the cap), a bundle copied into another run's directory, and a<run>.partialthat already heldbundle.jsonwere all read as done.Bundles.*(4 tests)EvalRun.verifychecks identity and digests against the plan and registration, every file's SHA-256, and that dollars, unknown models and acceptance match the journal (bundle_tampered)..partialdirectories are skipped. Nothing signsbundle.json, so a rewrite that keeps every digest consistent still passes.partial.test_a_second_runner_cannot_run_the_same_eval_at_onceruns/.lock(flock, ormsvcrton Windows). A second runner is refused witheval_runninggit add -Aapplies~/.config/git/ignoreandcore.excludesFile. A host that ignores*.pycommitted an empty base for every arm.source_digestpins the files, not the commit made from them.test_a_fixture_s_base_holds_every_fixture_file_whatever_the_host_ignoresgit add -A --forcewrite_taskwrites hidden files outside the task. A hidden file named with../.., an absolute path or a backslash path is written outside the task directory. This is the import API the pilot's 21 items would use.test_write_task_keeps_hidden_files_inside_the_taskhidden/json.loadskeeps the last value, so a reader can seethreshold: 0.0while the digest freezes-1.0.test_a_spec_with_a_duplicate_key_is_refusedread_jsonis strictos._exit(0)orsys.exit(0)in the module the checks import is accepted.spec/GAPS.md58 answers this fortests, not for hidden checks.test_a_candidate_that_exits_at_import_does_not_pass_the_hidden_checks__main__.bodyand its data fromsys.argv, then prints them. The output tail is the repair arm's feedback, so request 2 holds both canaries. EVAL.md's "Not in a request" holds only for a candidate that does not look.test_a_candidate_cannot_carry_a_hidden_check_into_its_next_requestclaude-codearm's model is a declaration only. The runner does not set the model, and nothing compares the served model (signal.harness.model) with the arm's registeredmodel. A changed Claude Code default runs both pilot arms on another model, and the bundle still names the declared one.test_a_run_served_by_another_model_than_its_arm_declares_is_not_reported_as_that_armANTHROPIC_MODELor--model, or refuse a mismatch). Pinnedrepo:task's run repository holds the commits after its base.materializeclones every ref, which includes the upstream fix once it is merged. Claude Code's file tools under--restrictedcannot reach it. Candidate code in checks, an arm withBashandrepo.bundlecan.test_a_repo_task_s_repository_holds_nothing_after_its_basetest_a_run_killed_midway_is_charged_against_the_capDesign gaps reported without a test
The pilot's decision rule does not fit the schema. The rule is "B's MEDIAN paired cost per delivered item is lower AND B delivers at least as many items", and
decision_rulecannot express it:metricmust be the primary metric, so delivered items and cost per delivered item cannot both be decided on.aggregationcan only bemean_over_seeds.apiorprovider_3plane, or a rule onsignal.estimated_usd.For the
eval-scorepacket: compute the rule in exact fractions. Per-task seed means A (0, 0, 1) and B (0, 1/3, 2/3) are an exact tie, but in floats the paired mean is -1.85e-17, so E5's>= 0would read the tie as a loss.Nothing holds the harness constant across an eval's runs. That covers the harness commit, adapter contents, Claude Code's version and the defaults filled in by code. For example, a
claude-codearm's timeout defaults to 600 s here and 1800 s in the provider, and that default is not in the arm digest. Bundles record these, andrundoes not refuse a mix. Seeds, run order and arms are registered, but the code the arms run is not.--onlyruns cells out of the registered order. The bundle records the plannedorder_indexand leaves no trace of the selection or the executed sequence apart from timestamps.The registration is not anchored. Deleting
registration.jsonand registering again before the first run leaves no local trace. A dirty registration is accepted silently, and E5's is one (git_dirty: true). A completed bundle deleted by hand is re-run without trace, which allows re-rolling.Candidate code in checks can also forge the run's journal and read the runner's secrets. The journal is an unsigned hash chain at a path known relative to the hidden copy (
../../kit/journal), and the session accepts records appended by another process. The code can also read the runner's initial environment (/proc/<ppid>/environ: the certificate seed and API keys). Points 3, 12 and 13 share this root cause, and only a sandbox per check closes it (N4).Per-run limits can be exceeded. Per-run dollars can overshoot by one call, which for
claude-codeis a whole session. Per-run seconds can overshoot by one call's timeout, plus the final check.CRLF normalization alters hidden data. It changes hidden data whose CRLF is significant, such as HTTP or email fixtures.
-textin.gitattributesforhidden/would avoid it.Docs to update (not edited here)
docs/EVAL.md, Hidden checks:hidden_tampered.docs/EVAL.md, the dollar cap: failed runs (runs/failed.jsonl), the lock, the per-run preflight gate andbundle_tampered.spec/GAPS.md60: add rows 12–16 above and the decision-rule gap.Checks
PYTHONPATH=. python3 -m unittest tests.test_eval_core tests.test_eval_runner tests.test_eval_adversarial: 46 tests, OK (5 expected failures).env -u ANTHROPIC_BASE_URL -u CLAUDE_CODE_PROVIDER_MANAGED_BY_HOST PYTHONPATH=. python3 -m unittest discover -s tests): 1033 tests, OK (24 skipped, 5 expected failures).PYTHONPATH=. python3 -m conformance.run: matched 23 golden traces, exit 0.https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz
Generated by Claude Code