Skip to content

Close eleven SyberEval core holes found by adversarial verification and pin five design gaps for the GitHits pilot (stacked on #137) - #144

Closed
sdcarlson wants to merge 1 commit into
syberlabs/3d8561e8from
fix/syber-eval-verification
Closed

sdcarlson wants to merge 1 commit into
syberlabs/3d8561e8from
fix/syber-eval-verification

Conversation

@sdcarlson

Copy link
Copy Markdown
Contributor

Independent adversarial verification of #137 (the SyberEval core) before the GitHits A/B pilot runs on it. Every finding below has a test in the new tests/test_eval_adversarial.py, run against real Build Threads with stand-in models (no model, no network). Each test for a fixed bug failed before its fix. A test marked expectedFailure pins a gap that needs a design decision: I left that behaviour unchanged, and the test fails at exactly the assertion it names.

Fixes are confined to syberlabs/eval/*. docs/EVAL.md and spec/GAPS.md are not edited; the last section lists the sentences they need.

Findings

# Sev Finding Repro (test) Status
1 High Two cells can share a run id. A run id is <task>.<arm>.s<seed>, and both names may contain dots. Task p with arm x.r and task p.x with arm r both give p.x.r.s1. The second run overwrites the first one's bundle, and on resume both cells read as done from one bundle: one run is missing and another is counted twice. test_two_cells_that_would_share_a_run_id_are_refused Fixed: spec.check refuses colliding run ids, so register and run refuse too
2 High A run that raises after spending is charged nothing. Only bundles counted toward total_dollars. A Ctrl-C, a harness error or a certificate that fails to verify after the model calls leaves no bundle, so each retry spent again under a cap that recorded nothing. test_a_run_that_fails_after_it_spent_is_charged_against_the_cap Fixed: runs/failed.jsonl records the host-observed dollars of each run that raised, and the cap counts them. An unreadable spend stops the eval (eval_spend_unknown)
3 High Candidate code can rewrite the host's copy of the hidden checks. The copy is an ordinary directory, and its path is in the check's own process (the loader's __main__.path, /proc/self/cmdline). c1 rewrites every check to pass, c2 changes nothing that matters, and c2 is accepted. EVAL.md's "Not editable" held only for the candidate's own tree. test_a_candidate_that_rewrites_the_host_copy_of_a_check_is_not_accepted Fixed (detection): the runner digests the staged copy and refuses to accept if it changed. The outcome becomes not_acceptable with reason hidden_tampered, plus hidden_intact and hidden_changed. pytest's own .pytest_cache is tolerated; a planted conftest.py or __pycache__ is not. A rewrite that is undone before the checks end is not detected (see 12 and 13)
4 Medium Every run of a githits arm shares one GitHits session. arms.build passed the provider neither thread_label nor arm, so every run of every task and seed sent X-GitHits-Session-ID: s1-b and X-GitHits-Agent: syberfactory/b. test_each_run_of_the_githits_arm_has_its_own_githits_session Fixed: the session is <thread id>-<arm> and the agent is syberfactory/<arm name>
5 Medium NaN passes the spec schema. It passes minimum, maximum and exclusiveMinimum, because every comparison with it is false. A spec with total_dollars: NaN registers and its cap never stops, and a NaN or Infinity threshold makes the decision vacuous. test_a_number_that_is_not_finite_is_not_a_valid_spec Fixed: numbers must be finite, and read_json refuses NaN and Infinity
6 Medium The preflight gate was checked only when run started. A checkout moved or edited mid-eval (git pull, or an edited adapter, which is re-read from disk on every call) ran every later run under a stale preflight. test_the_preflight_gate_holds_for_every_run_not_only_the_first Fixed: the gate is checked again before every run, and the eval stops with the gate's code
7 Medium Bundles were never verified. An edited journal, trace or repository bundle, an edited outcome.dollars (which bought room under the cap), a bundle copied into another run's directory, and a <run>.partial that already held bundle.json were all read as done. Bundles.* (4 tests) Fixed: EvalRun.verify checks identity and digests against the plan and registration, every file's SHA-256, and that dollars, unknown models and acceptance match the journal (bundle_tampered). .partial directories are skipped. Nothing signs bundle.json, so a rewrite that keeps every digest consistent still passes
8 Medium Two runners can run one eval at once. Each counts only the bundles it found at start, so both pass the cap check. Both start the same run, and the second deletes the first one's .partial. test_a_second_runner_cannot_run_the_same_eval_at_once Fixed: runs/.lock (flock, or msvcrt on Windows). A second runner is refused with eval_running
9 Medium A fixture's base depends on the host's global git ignore rules. git add -A applies ~/.config/git/ignore and core.excludesFile. A host that ignores *.py committed an empty base for every arm. source_digest pins the files, not the commit made from them. test_a_fixture_s_base_holds_every_fixture_file_whatever_the_host_ignores Fixed: git add -A --force
10 Low write_task writes hidden files outside the task. A hidden file named with ../.., an absolute path or a backslash path is written outside the task directory. This is the import API the pilot's 21 items would use. test_write_task_keeps_hidden_files_inside_the_task Fixed: hidden paths must be relative and stay under hidden/
11 Low A duplicate key in a spec or task file. json.loads keeps the last value, so a reader can see threshold: 0.0 while the digest freezes -1.0. test_a_spec_with_a_duplicate_key_is_refused Fixed: read_json is strict
12 High A candidate that exits 0 at import passes every hidden check. Checks are judged by exit status alone, so os._exit(0) or sys.exit(0) in the module the checks import is accepted. spec/GAPS.md 58 answers this for tests, not for hidden checks. test_a_candidate_that_exits_at_import_does_not_pass_the_hidden_checks Design (sandbox, or a loader with a completion protocol, which changes task digests and E5's registration). Pinned
13 High A candidate can carry a hidden check into its next request. Code run by a hidden check reads the check's source from __main__.body and its data from sys.argv, then prints them. The output tail is the repair arm's feedback, so request 2 holds both canaries. EVAL.md's "Not in a request" holds only for a candidate that does not look. test_a_candidate_cannot_carry_a_hidden_check_into_its_next_request Design (sandbox, or feedback the candidate cannot write). Pinned
14 High A claude-code arm's model is a declaration only. The runner does not set the model, and nothing compares the served model (signal.harness.model) with the arm's registered model. A changed Claude Code default runs both pilot arms on another model, and the bundle still names the declared one. test_a_run_served_by_another_model_than_its_arm_declares_is_not_reported_as_that_arm Design (set it with ANTHROPIC_MODEL or --model, or refuse a mismatch). Pinned
15 Medium A repo: task's run repository holds the commits after its base. materialize clones every ref, which includes the upstream fix once it is merged. Claude Code's file tools under --restricted cannot reach it. Candidate code in checks, an arm with Bash and repo.bundle can. test_a_repo_task_s_repository_holds_nothing_after_its_base Design (keep ancestors only, at the cost of fetch or GC time). Pinned
16 Medium A runner killed outright charges nothing. SIGKILL, an OOM kill or a power cut runs no handler, and its spend is counted nowhere. test_a_run_killed_midway_is_charged_against_the_cap Design (reserve the per-run dollars at start, or stop until reconciled). Pinned

Design gaps reported without a test

  • The pilot's decision rule does not fit the schema. The rule is "B's MEDIAN paired cost per delivered item is lower AND B delivers at least as many items", and decision_rule cannot express it:

    1. There is one rule object, so two conditions cannot be joined with AND.
    2. metric must be the primary metric, so delivered items and cost per delivered item cannot both be decided on.
    3. No field names the statistic across tasks. A mean is implied (E5 says "mean over tasks" only in its reporting text), so a median cannot be stated.
    4. aggregation can only be mean_over_seeds.
    5. A ratio metric has no convention for an item that one arm did not deliver.
    6. On Claude Code's login lanes cost is unpriced, so the pilot needs the api or provider_3p lane, or a rule on signal.estimated_usd.

    For the eval-score packet: compute the rule in exact fractions. Per-task seed means A (0, 0, 1) and B (0, 1/3, 2/3) are an exact tie, but in floats the paired mean is -1.85e-17, so E5's >= 0 would read the tie as a loss.

  • Nothing holds the harness constant across an eval's runs. That covers the harness commit, adapter contents, Claude Code's version and the defaults filled in by code. For example, a claude-code arm's timeout defaults to 600 s here and 1800 s in the provider, and that default is not in the arm digest. Bundles record these, and run does not refuse a mix. Seeds, run order and arms are registered, but the code the arms run is not.

  • --only runs cells out of the registered order. The bundle records the planned order_index and leaves no trace of the selection or the executed sequence apart from timestamps.

  • The registration is not anchored. Deleting registration.json and registering again before the first run leaves no local trace. A dirty registration is accepted silently, and E5's is one (git_dirty: true). A completed bundle deleted by hand is re-run without trace, which allows re-rolling.

  • Candidate code in checks can also forge the run's journal and read the runner's secrets. The journal is an unsigned hash chain at a path known relative to the hidden copy (../../kit/journal), and the session accepts records appended by another process. The code can also read the runner's initial environment (/proc/<ppid>/environ: the certificate seed and API keys). Points 3, 12 and 13 share this root cause, and only a sandbox per check closes it (N4).

  • Per-run limits can be exceeded. Per-run dollars can overshoot by one call, which for claude-code is a whole session. Per-run seconds can overshoot by one call's timeout, plus the final check.

  • CRLF normalization alters hidden data. It changes hidden data whose CRLF is significant, such as HTTP or email fixtures. -text in .gitattributes for hidden/ would avoid it.

Docs to update (not edited here)

  • docs/EVAL.md, Hidden checks:
    • "Not editable" now holds only for a rewrite that stays. Name hidden_tampered.
    • "Not in a request" fails for a candidate that prints what its check holds.
    • The host path does not appear "only in the run contract". It is in every check process.
  • docs/EVAL.md, the dollar cap: failed runs (runs/failed.jsonl), the lock, the per-run preflight gate and bundle_tampered.
  • spec/GAPS.md 60: add rows 12–16 above and the decision-rule gap.

Checks

  • PYTHONPATH=. python3 -m unittest tests.test_eval_core tests.test_eval_runner tests.test_eval_adversarial: 46 tests, OK (5 expected failures).
  • Full suite (env -u ANTHROPIC_BASE_URL -u CLAUDE_CODE_PROVIDER_MANAGED_BY_HOST PYTHONPATH=. python3 -m unittest discover -s tests): 1033 tests, OK (24 skipped, 5 expected failures).
  • PYTHONPATH=. python3 -m conformance.run: matched 23 golden traces, exit 0.
  • E5's registration, the committed tasks' digests and the golden traces are unchanged.

https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz


Generated by Claude Code

…pin five design gaps

Independent verification of the eval core (#137) before the GitHits A/B pilot.
Every finding has a test in tests/test_eval_adversarial.py; each fixed one
failed before its fix. Fixes are confined to syberlabs/eval/*.

Fixed:
- run ids could collide (task and arm names may hold dots): one cell's bundle
  overwrote another's; spec.check now refuses colliding run ids
- a run that raised after spending was charged nothing, so every retry spent
  again under the cap; runs/failed.jsonl records its spend and the cap counts it
- candidate code could rewrite the host's copy of the hidden checks and get a
  later candidate accepted; the runner refuses to accept on a changed copy
  (not_acceptable, hidden_tampered)
- every run of a githits arm shared X-GitHits-Session-ID s1-b and agent
  syberfactory/b; the run's thread and the arm's name now label them
- NaN passed every numeric bound of the spec schema (a NaN total_dollars
  disabled the cap); numbers must be finite and read_json is strict
  (no NaN, Infinity or repeated keys)
- the preflight gate was checked once per invocation; it is checked before
  every run
- bundles were never verified: edited evidence or dollars, a bundle in another
  run's directory and a .partial were read as done; EvalRun.verify checks them
  (bundle_tampered)
- two runners could run one eval at once and pass the cap together; runs/.lock
- a fixture's base honoured the host's global git ignore rules (an empty base
  on a host ignoring *.py); git add --force
- write_task wrote hidden files named ../ or absolute outside the task

Pinned as expectedFailure (design decisions): a candidate that exits 0 at
import passes every hidden check; a candidate can print its hidden check into
the next repair request; a claude-code arm's served model is never compared
with its declared model; repo: task repositories hold commits after the base;
a runner killed outright charges nothing to the cap.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-26T22:47:48.248251Z 35b8ce3 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@sykosyber

Copy link
Copy Markdown
Member

Heads-up: #137 was squash-merged, so this PR's base branch is stale. With Mateo's OK, we rebuilt your commit unchanged onto main as #148, keeping you as author, so it can merge ahead of #146. Your branch was not touched. If you'd rather land it from your branch instead, say so and we'll close #148.

sykosyber added a commit that referenced this pull request Oct 28, 2026
…rebuild of #144) (#148)

Independent verification of the eval core (#137) before the GitHits A/B pilot.
Every finding has a test in tests/test_eval_adversarial.py; each fixed one
failed before its fix. Fixes are confined to syberlabs/eval/*.

Fixed:
- run ids could collide (task and arm names may hold dots): one cell's bundle
  overwrote another's; spec.check now refuses colliding run ids
- a run that raised after spending was charged nothing, so every retry spent
  again under the cap; runs/failed.jsonl records its spend and the cap counts it
- candidate code could rewrite the host's copy of the hidden checks and get a
  later candidate accepted; the runner refuses to accept on a changed copy
  (not_acceptable, hidden_tampered)
- every run of a githits arm shared X-GitHits-Session-ID s1-b and agent
  syberfactory/b; the run's thread and the arm's name now label them
- NaN passed every numeric bound of the spec schema (a NaN total_dollars
  disabled the cap); numbers must be finite and read_json is strict
  (no NaN, Infinity or repeated keys)
- the preflight gate was checked once per invocation; it is checked before
  every run
- bundles were never verified: edited evidence or dollars, a bundle in another
  run's directory and a .partial were read as done; EvalRun.verify checks them
  (bundle_tampered)
- two runners could run one eval at once and pass the cap together; runs/.lock
- a fixture's base honoured the host's global git ignore rules (an empty base
  on a host ignoring *.py); git add --force
- write_task wrote hidden files named ../ or absolute outside the task

Pinned as expectedFailure (design decisions): a candidate that exits 0 at
import passes every hidden check; a candidate can print its hidden check into
the next repair request; a claude-code arm's served model is never compared
with its declared model; repo: task repositories hold commits after the base;
a runner killed outright charges nothing to the cap.

Rebuilt onto main from #144 by the SyberLabs lead's branch operator.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz
@sykosyber

Copy link
Copy Markdown
Member

Landed as #148 (merge commit 4c65aa0), your commit unchanged. Closing this one as superseded; thanks Seth.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants