Skip to content

Rebuild of #144 onto main: close eleven SyberEval core holes (author: Seth Carlson) - #148

Merged
sykosyber merged 1 commit into
mainfrom
syberlabs/rebuild-144
Oct 28, 2026
Merged

sykosyber merged 1 commit into
mainfrom
syberlabs/rebuild-144

Conversation

@sykosyber

@sykosyber sykosyber commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

Note: Rebuild of #144 onto current main after #137 was squash-merged; Seth Carlson's commit unchanged. #144 stays open until this lands; the owner authorised the rebuild.

Rebuilt commit: 92ed764420ec6c14633fb23a9c78078df3c1a6f6 (author Seth Carlson), cherry-picked from 35b8ce311a60a1abbcadb60f2a80d5d1bc82d186 onto main d85f3d2825484fa5ee246585a6f43e3ad4114723 with no conflicts. Its diff against main is byte-identical to #144's own diff (same five files, same --full-index patch, same stable patch-id a98987dc8bf17cc21d1a7294733363f2ffaa78c7). The only change to the commit message is one added line: "Rebuilt onto main from #144 by the SyberLabs lead's branch operator."

#144's body, quoted in full:

Independent adversarial verification of #137 (the SyberEval core) before the GitHits A/B pilot runs on it. Every finding below has a test in the new tests/test_eval_adversarial.py, run against real Build Threads with stand-in models (no model, no network). Each test for a fixed bug failed before its fix. A test marked expectedFailure pins a gap that needs a design decision: I left that behaviour unchanged, and the test fails at exactly the assertion it names.

Fixes are confined to syberlabs/eval/*. docs/EVAL.md and spec/GAPS.md are not edited; the last section lists the sentences they need.

Findings

# Sev Finding Repro (test) Status
1 High Two cells can share a run id. A run id is <task>.<arm>.s<seed>, and both names may contain dots. Task p with arm x.r and task p.x with arm r both give p.x.r.s1. The second run overwrites the first one's bundle, and on resume both cells read as done from one bundle: one run is missing and another is counted twice. test_two_cells_that_would_share_a_run_id_are_refused Fixed: spec.check refuses colliding run ids, so register and run refuse too
2 High A run that raises after spending is charged nothing. Only bundles counted toward total_dollars. A Ctrl-C, a harness error or a certificate that fails to verify after the model calls leaves no bundle, so each retry spent again under a cap that recorded nothing. test_a_run_that_fails_after_it_spent_is_charged_against_the_cap Fixed: runs/failed.jsonl records the host-observed dollars of each run that raised, and the cap counts them. An unreadable spend stops the eval (eval_spend_unknown)
3 High Candidate code can rewrite the host's copy of the hidden checks. The copy is an ordinary directory, and its path is in the check's own process (the loader's __main__.path, /proc/self/cmdline). c1 rewrites every check to pass, c2 changes nothing that matters, and c2 is accepted. EVAL.md's "Not editable" held only for the candidate's own tree. test_a_candidate_that_rewrites_the_host_copy_of_a_check_is_not_accepted Fixed (detection): the runner digests the staged copy and refuses to accept if it changed. The outcome becomes not_acceptable with reason hidden_tampered, plus hidden_intact and hidden_changed. pytest's own .pytest_cache is tolerated; a planted conftest.py or __pycache__ is not. A rewrite that is undone before the checks end is not detected (see 12 and 13)
4 Medium Every run of a githits arm shares one GitHits session. arms.build passed the provider neither thread_label nor arm, so every run of every task and seed sent X-GitHits-Session-ID: s1-b and X-GitHits-Agent: syberfactory/b. test_each_run_of_the_githits_arm_has_its_own_githits_session Fixed: the session is <thread id>-<arm> and the agent is syberfactory/<arm name>
5 Medium NaN passes the spec schema. It passes minimum, maximum and exclusiveMinimum, because every comparison with it is false. A spec with total_dollars: NaN registers and its cap never stops, and a NaN or Infinity threshold makes the decision vacuous. test_a_number_that_is_not_finite_is_not_a_valid_spec Fixed: numbers must be finite, and read_json refuses NaN and Infinity
6 Medium The preflight gate was checked only when run started. A checkout moved or edited mid-eval (git pull, or an edited adapter, which is re-read from disk on every call) ran every later run under a stale preflight. test_the_preflight_gate_holds_for_every_run_not_only_the_first Fixed: the gate is checked again before every run, and the eval stops with the gate's code
7 Medium Bundles were never verified. An edited journal, trace or repository bundle, an edited outcome.dollars (which bought room under the cap), a bundle copied into another run's directory, and a <run>.partial that already held bundle.json were all read as done. Bundles.* (4 tests) Fixed: EvalRun.verify checks identity and digests against the plan and registration, every file's SHA-256, and that dollars, unknown models and acceptance match the journal (bundle_tampered). .partial directories are skipped. Nothing signs bundle.json, so a rewrite that keeps every digest consistent still passes
8 Medium Two runners can run one eval at once. Each counts only the bundles it found at start, so both pass the cap check. Both start the same run, and the second deletes the first one's .partial. test_a_second_runner_cannot_run_the_same_eval_at_once Fixed: runs/.lock (flock, or msvcrt on Windows). A second runner is refused with eval_running
9 Medium A fixture's base depends on the host's global git ignore rules. git add -A applies ~/.config/git/ignore and core.excludesFile. A host that ignores *.py committed an empty base for every arm. source_digest pins the files, not the commit made from them. test_a_fixture_s_base_holds_every_fixture_file_whatever_the_host_ignores Fixed: git add -A --force
10 Low write_task writes hidden files outside the task. A hidden file named with ../.., an absolute path or a backslash path is written outside the task directory. This is the import API the pilot's 21 items would use. test_write_task_keeps_hidden_files_inside_the_task Fixed: hidden paths must be relative and stay under hidden/
11 Low A duplicate key in a spec or task file. json.loads keeps the last value, so a reader can see threshold: 0.0 while the digest freezes -1.0. test_a_spec_with_a_duplicate_key_is_refused Fixed: read_json is strict
12 High A candidate that exits 0 at import passes every hidden check. Checks are judged by exit status alone, so os._exit(0) or sys.exit(0) in the module the checks import is accepted. spec/GAPS.md 58 answers this for tests, not for hidden checks. test_a_candidate_that_exits_at_import_does_not_pass_the_hidden_checks Design (sandbox, or a loader with a completion protocol, which changes task digests and E5's registration). Pinned
13 High A candidate can carry a hidden check into its next request. Code run by a hidden check reads the check's source from __main__.body and its data from sys.argv, then prints them. The output tail is the repair arm's feedback, so request 2 holds both canaries. EVAL.md's "Not in a request" holds only for a candidate that does not look. test_a_candidate_cannot_carry_a_hidden_check_into_its_next_request Design (sandbox, or feedback the candidate cannot write). Pinned
14 High A claude-code arm's model is a declaration only. The runner does not set the model, and nothing compares the served model (signal.harness.model) with the arm's registered model. A changed Claude Code default runs both pilot arms on another model, and the bundle still names the declared one. test_a_run_served_by_another_model_than_its_arm_declares_is_not_reported_as_that_arm Design (set it with ANTHROPIC_MODEL or --model, or refuse a mismatch). Pinned
15 Medium A repo: task's run repository holds the commits after its base. materialize clones every ref, which includes the upstream fix once it is merged. Claude Code's file tools under --restricted cannot reach it. Candidate code in checks, an arm with Bash and repo.bundle can. test_a_repo_task_s_repository_holds_nothing_after_its_base Design (keep ancestors only, at the cost of fetch or GC time). Pinned
16 Medium A runner killed outright charges nothing. SIGKILL, an OOM kill or a power cut runs no handler, and its spend is counted nowhere. test_a_run_killed_midway_is_charged_against_the_cap Design (reserve the per-run dollars at start, or stop until reconciled). Pinned

Design gaps reported without a test

  • The pilot's decision rule does not fit the schema. The rule is "B's MEDIAN paired cost per delivered item is lower AND B delivers at least as many items", and decision_rule cannot express it:

    1. There is one rule object, so two conditions cannot be joined with AND.
    2. metric must be the primary metric, so delivered items and cost per delivered item cannot both be decided on.
    3. No field names the statistic across tasks. A mean is implied (E5 says "mean over tasks" only in its reporting text), so a median cannot be stated.
    4. aggregation can only be mean_over_seeds.
    5. A ratio metric has no convention for an item that one arm did not deliver.
    6. On Claude Code's login lanes cost is unpriced, so the pilot needs the api or provider_3p lane, or a rule on signal.estimated_usd.

    For the eval-score packet: compute the rule in exact fractions. Per-task seed means A (0, 0, 1) and B (0, 1/3, 2/3) are an exact tie, but in floats the paired mean is -1.85e-17, so E5's >= 0 would read the tie as a loss.

  • Nothing holds the harness constant across an eval's runs. That covers the harness commit, adapter contents, Claude Code's version and the defaults filled in by code. For example, a claude-code arm's timeout defaults to 600 s here and 1800 s in the provider, and that default is not in the arm digest. Bundles record these, and run does not refuse a mix. Seeds, run order and arms are registered, but the code the arms run is not.

  • --only runs cells out of the registered order. The bundle records the planned order_index and leaves no trace of the selection or the executed sequence apart from timestamps.

  • The registration is not anchored. Deleting registration.json and registering again before the first run leaves no local trace. A dirty registration is accepted silently, and E5's is one (git_dirty: true). A completed bundle deleted by hand is re-run without trace, which allows re-rolling.

  • Candidate code in checks can also forge the run's journal and read the runner's secrets. The journal is an unsigned hash chain at a path known relative to the hidden copy (../../kit/journal), and the session accepts records appended by another process. The code can also read the runner's initial environment (/proc/<ppid>/environ: the certificate seed and API keys). Points 3, 12 and 13 share this root cause, and only a sandbox per check closes it (N4).

  • Per-run limits can be exceeded. Per-run dollars can overshoot by one call, which for claude-code is a whole session. Per-run seconds can overshoot by one call's timeout, plus the final check.

  • CRLF normalization alters hidden data. It changes hidden data whose CRLF is significant, such as HTTP or email fixtures. -text in .gitattributes for hidden/ would avoid it.

Docs to update (not edited here)

  • docs/EVAL.md, Hidden checks:
    • "Not editable" now holds only for a rewrite that stays. Name hidden_tampered.
    • "Not in a request" fails for a candidate that prints what its check holds.
    • The host path does not appear "only in the run contract". It is in every check process.
  • docs/EVAL.md, the dollar cap: failed runs (runs/failed.jsonl), the lock, the per-run preflight gate and bundle_tampered.
  • spec/GAPS.md 60: add rows 12–16 above and the decision-rule gap.

Checks

  • PYTHONPATH=. python3 -m unittest tests.test_eval_core tests.test_eval_runner tests.test_eval_adversarial: 46 tests, OK (5 expected failures).
  • Full suite (env -u ANTHROPIC_BASE_URL -u CLAUDE_CODE_PROVIDER_MANAGED_BY_HOST PYTHONPATH=. python3 -m unittest discover -s tests): 1033 tests, OK (24 skipped, 5 expected failures).
  • PYTHONPATH=. python3 -m conformance.run: matched 23 golden traces, exit 0.
  • E5's registration, the committed tasks' digests and the golden traces are unchanged.

https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz


Generated by Claude Code

🤖 Generated with Claude Code

Reviewed-by: claude (independent verifier session, SyberFactory workflow)

…pin five design gaps

Independent verification of the eval core (#137) before the GitHits A/B pilot.
Every finding has a test in tests/test_eval_adversarial.py; each fixed one
failed before its fix. Fixes are confined to syberlabs/eval/*.

Fixed:
- run ids could collide (task and arm names may hold dots): one cell's bundle
  overwrote another's; spec.check now refuses colliding run ids
- a run that raised after spending was charged nothing, so every retry spent
  again under the cap; runs/failed.jsonl records its spend and the cap counts it
- candidate code could rewrite the host's copy of the hidden checks and get a
  later candidate accepted; the runner refuses to accept on a changed copy
  (not_acceptable, hidden_tampered)
- every run of a githits arm shared X-GitHits-Session-ID s1-b and agent
  syberfactory/b; the run's thread and the arm's name now label them
- NaN passed every numeric bound of the spec schema (a NaN total_dollars
  disabled the cap); numbers must be finite and read_json is strict
  (no NaN, Infinity or repeated keys)
- the preflight gate was checked once per invocation; it is checked before
  every run
- bundles were never verified: edited evidence or dollars, a bundle in another
  run's directory and a .partial were read as done; EvalRun.verify checks them
  (bundle_tampered)
- two runners could run one eval at once and pass the cap together; runs/.lock
- a fixture's base honoured the host's global git ignore rules (an empty base
  on a host ignoring *.py); git add --force
- write_task wrote hidden files named ../ or absolute outside the task

Pinned as expectedFailure (design decisions): a candidate that exits 0 at
import passes every hidden check; a candidate can print its hidden check into
the next repair request; a claude-code arm's served model is never compared
with its declared model; repo: task repositories hold commits after the base;
a runner killed outright charges nothing to the cap.

Rebuilt onto main from #144 by the SyberLabs lead's branch operator.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review 🔄 Running since 2026-10-28T22:10:15.907182Z 92ed764 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@sykosyber

Copy link
Copy Markdown
Member Author

Independent verification of #148 (head 92ed764420)

Verdict: mergeable as is. I checked all 11 fixes against the code, and each one does what #144's table says. Each fix's named test exercises it and passes. The 5 pinned gaps are expectedFailure. Four of them fail at exactly the assertion named. The fifth (16) does too on POSIX, but on Windows it errors earlier (finding 4). No other module's behaviour changes, the golden traces match, and every CI job is green. Nothing below blocks the merge. Finding 1 is input for reconciling with #146, not a defect of this PR.

Method: I read the code and re-ran the tests in a fresh clone on Windows 11 with Python 3.13. I wrote no attack code and made no model, Jev or GitHits call. I made two scratch checks, neither of which touched the PR. First, I ran the five gap tests with their markers cleared, to see where each fails. Second, I ran fix 4's test against main's arms.py.

Identity

Findings, ranked

  1. Reconciliation with Add the SyberEval safety gate, scorer and paired report, with the hidden_feedback spec field, hidden-check integrity re-hashing and an append-only spend ledger. Each is computed from the journal and t #146 (no defect here). Add the SyberEval safety gate, scorer and paired report, with the hidden_feedback spec field, hidden-check integrity re-hashing and an append-only spend ledger. Each is computed from the journal and t #146 rebuilds the same two mechanisms differently:

    tests/test_eval_adversarial.py:395-398 pins this PR's shape. Whichever PR lands second has to choose one mechanism of each kind, not keep both. A trial merge of Add the SyberEval safety gate, scorer and paired report, with the hidden_feedback spec field, hidden-check integrity re-hashing and an append-only spend ledger. Each is computed from the journal and t #146 onto this head conflicts in 5 files; the list follows the findings.

  2. Low: runs/failed.jsonl is read without the strictness this PR adds elsewhere. EvalRun.failed (runner.py:284) uses plain json.loads, so NaN and Infinity are accepted. runner.py:318 then sums float(f.get("dollars") or 0.0) with no finite or sign check. The effects:

    • A NaN row makes spent NaN, and the cap comparison is then always false. That is the bug class fix 5 closes for the spec.
    • A negative row buys room under the cap.
    • A non-numeric value raises a bare ValueError instead of a coded refusal.
    • Unlike bundles (fix 7), a row is cross-checked against nothing, so a deleted line goes undetected.

    The file is host-written, so this only matters under the same tamper model as fix 7. Suggested follow-up: read the ledger with read_json-style strictness and require finite dollars ≥ 0.

  3. Low/info: on current main, fix 4's session half is already covered by Integrate honest two-arm GitHits accounting and dry-run exit status #145. ClaudeCodeProvider._session (syberlabs/claude_code.py:777-784) now falls back to the space's _thread. I ran the test with main's arms.py and without thread_label. The two-sessions assertions passed, and only the agent assertion failed (test_eval_adversarial.py:487: syberfactory/b against syberfactory/on). So the live part of fix 4 is the X-GitHits-Agent label. Passing thread_label is now belt-and-braces. A separate small point: githits_arm (arms.py:93-95) sanitises names and cuts them to 16 characters. Two distinct arm names can therefore share one agent label, for example a.b and a-b, or two names that differ only after 16 characters. The spec does not refuse that.

  4. Low (test portability): gap 16's test fails at the wrong line on Windows. test_a_run_killed_midway_is_charged_against_the_cap raises AttributeError: module 'signal' has no attribute 'SIGKILL' at test_eval_adversarial.py:275, not at the named assertion on line 278. The ADAPTER also uses signal.SIGKILL at line 55. On Windows the expectedFailure therefore passes for the wrong reason. On POSIX it reaches line 278. Suggested fix: skipUnless(hasattr(signal, "SIGKILL")).

  5. Info: the tamper check is narrower than "tamper-proof", as the PR states.

    • It runs once, after the final thread.check and immediately before thread.accept (runner.py:406-408). This is the only accept site in syberlabs/eval, so every accepting path is covered.
    • The baseline is taken before propose, so a rewrite made during the repair arm's feedback checks is also caught.
    • A rewrite that is undone before the end is not caught (the PR's own caveat, linked to 12 and 13).
    • .pytest_cache is tolerated at any depth and with any contents (tasks.py:410). That is wider than "pytest's rootdir", but harmless today because pytest loads no conftest.py from dot-directories.
    • The __pycache__ rule cannot misfire on legitimate checks, because syberlabs/checks.py:83 sets PYTHONDONTWRITEBYTECODE=1.
  6. Info.

    • spec.check imports runner.run_id (spec.py:218). That adds a spec→runner import cycle, resolved lazily.
    • The lock (runner.py:182-204,310) is not taken for --dry or register. Its behaviour matches the claim on both platforms: flock with LOCK_NB on POSIX, and msvcrt.locking(LK_NBLCK) on byte 0 on Windows. The in-process second-runner test passed locally on Windows and in CI on Linux and Windows.

Per-fix check

# Fix Code Test Result
1 Run-id collisions refused spec.py:218-224 (Counter over run_id) test_two_cells_that_would_share_a_run_id_are_refused Pass; matches the claim
2 Spend of a raised run charged runner.py:446-448,461-477 (_failed), :318, :332 (eval_spend_unknown) test_a_run_that_fails_after_it_spent_is_charged_against_the_cap Pass. Narrower than it sounds: see finding 2
3 Hidden-copy tamper detection tasks.py:385-411, runner.py:388,406-413 test_a_candidate_that_rewrites_…, test_only_pytest_s_own_cache_… Pass. Detection only, as stated (finding 5)
4 A GitHits session per run arms.py:93-95,133, runner.py:395-396 test_each_run_of_the_githits_arm_has_its_own_githits_session Pass. Its session half is already on main (finding 3)
5 Non-finite numbers refused spec.py:162-163, :244-253 (parse_constant) test_a_number_that_is_not_finite_is_not_a_valid_spec Pass
6 Preflight gate before every run runner.py:341-347 test_the_preflight_gate_holds_for_every_run_not_only_the_first Pass. Applies when require_preflight (the default)
7 EvalRun.verify, bundle_tampered, .partial skipped runner.py:224-275 Bundles.* (4 tests) Pass. Unsigned, as stated
8 runs/.lock, eval_running runner.py:64-75,182-204,310 test_a_second_runner_cannot_run_the_same_eval_at_once Pass on Windows (local and CI) and Linux (CI)
9 git add -A --force tasks.py:322 test_a_fixture_s_base_holds_every_fixture_file_… Pass
10 Hidden paths confined under hidden/ tasks.py:379-382,417 test_write_task_keeps_hidden_files_inside_the_task Pass
11 Strict read_json (duplicate keys) spec.py:238-253 test_a_spec_with_a_duplicate_key_is_refused Pass

Pinned gaps (12–16), with markers cleared in a scratch run

# Test Fails at Matches the claim
12 …exits_at_import… line 425, assertNotEqual(status, "accepted") Yes
13 …carry_a_hidden_check… line 439, assertNotIn(CANARY_CODE, second) Yes
14 …served_by_another_model… line 501, assertFalse(… not in served) Yes
15 …repo_task_s_repository… line 527, assertNotEqual(returncode, 0) Yes
16 …killed_midway… Windows: line 275 AttributeError (no SIGKILL); POSIX: line 278 Windows: no (finding 4)

Tests

  • Local (fresh clone, Windows, Python 3.13): tests.test_eval_adversarial tests.test_eval_core tests.test_eval_runner tests.test_gaps ran 50 tests, OK (5 expected failures). python -m conformance.run matched 23 golden traces, exit 0.
  • CI on 92ed764420: all 25 check runs pass, across the push and pull_request runs.
    • The full suite is 1200 tests on 3.11, 3.12 and 3.13, on strict 3.12 and on Windows. All are OK with 5 expected failures; skips are 2 on Linux and 25 on Windows. The base had 1180 tests, so the 20 new tests are present.
    • Conformance matched 23 golden traces.
    • admit is acceptable (all_checks_passed, rule admission.passed). It moved a local ref only; nothing was pushed or merged.

Merge preview: #146 (ae117c1f72) onto this head

A local trial with no push, then aborted, conflicted in 5 files:

Reviewed-by: claude (independent verifier session, SyberFactory workflow)

@sykosyber
sykosyber merged commit 4c65aa0 into main Oct 28, 2026
25 checks passed
sykosyber added a commit that referenced this pull request Oct 29, 2026
…rt, hidden_feedback and spend ledger) on (#150)

Objective: Land eval-score (#146: the SyberEval safety gate, scorer, paired report, hidden_feedback and spend ledger) on top of the eval-core fixes of #148, with one hidden-copy integrity mechanism and one spend ledger

Class: R
Agent-platform: claude@opus-5.5
Admitted-by: sykosyber
Reviewed-by: rules@2 (critic check critic.rules: high 0, medium 0, low 0)
Evidence: tested

Accepted in SyberLabs thread `a248e101-021e-4a08-9b35-0fefe78d839a` as candidate `c1` (commit `098a4f323a3e942ef1660ea5e4089e9258591f31`).
Host checks on that tree: conformance passed, critic.rules passed, scenarios passed, tests passed.

Reviewed-by: claude (independent verifier session, SyberFactory workflow)


Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants