Skip to content

Register the pre-registered eval E5 on clean main (90930320c6) and commit its registration, with the committed-E5 tests, docs/EVAL.md and GAPS 62 (10) updated to a registered, unrun E5 - #152

Merged
sykosyber merged 2 commits into
mainfrom
syberlabs/7c38fc5e
Oct 31, 2026

Conversation

@sykosyber

Copy link
Copy Markdown
Member

Objective: Register the pre-registered eval E5 on clean main (9093032) and commit its registration, with the committed-E5 tests, docs/EVAL.md and GAPS 62 (10) updated to a registered, unrun E5

Class: R
Agent-platform: claude@opus-5.5
Admitted-by: sykosyber
Reviewed-by: rules@2 (critic check critic.rules: high 0, medium 0, low 0)
Evidence: tested

Accepted in SyberLabs thread 7c38fc5e-4eb7-4814-bb99-e3b9a073dfd4 as candidate c1 (commit dfdfddfb7375c343599187dad14bd7fea30b4265).
Host checks on that tree: conformance passed, critic.rules passed, scenarios passed, tests passed.

… tests, EVAL.md and GAPS 62 (10) updated

SyberLabs-Thread: 7c38fc5e-4eb7-4814-bb99-e3b9a073dfd4
SyberLabs-Candidate: c1
SyberLabs-Provider: claude@opus-5.5
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-29T07:04:24.321563Z dfdfddf PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@sykosyber

Copy link
Copy Markdown
Member Author

Scope: owner direction in session 2026-10-28 (lead decision: widen packet e5-register to 4 files). This PR registers the pre-registered eval E5 on clean main and commits the registration. E5 is not run.

Registration (evals/e5-single-vs-repair/registration.json)

The registration was written on a fresh clone of main at 90930320c6 before anything else was touched. git status --porcelain was empty beforehand.

  • spec_digest: 7625340097b9df1a1be4a53482bc36a14865361bf0fb7c48cdfc787318d5caa6
  • task_set_digest: 13589c4ff8e07e9666c72f3922d743bf076b9e225024f1296864ecc31642cb17 (8 tasks)
  • arm_digests: single cc8dc7ec0d…, repair 1912623052…
  • git_commit: 90930320c6c1c3073066d502997d5ada791b7673
  • git_dirty: false
  • registered_at: 2026-10-29T06:54:36Z

E5, as the spec defines it:

  • 48 runs: 8 tasks (v3 and v4) x arms single and repair x seeds 1, 2 and 3, with order_seed 20261026 and hidden_feedback: tails.
  • Budget per run: $1.00, 1800 s and 6 calls. The eval is capped at total_dollars $10.00 (strict).
  • Sonnet 5.5 is priced at $2 per million input tokens and $10 per million output tokens.

What changed (4 files)

  1. evals/e5-single-vs-repair/registration.json: new, as above.
  2. tests/test_eval_core.py:
    • CommittedEvals.test_e5_is_registered_from_clean_main_and_not_run replaces test_e5_is_a_valid_spec_awaiting_registration_on_clean_main. It asserts:

      • spec.registered() accepts E5, which refuses a spec, task or arm that has moved;
      • the registration's spec_digest equals the spec's current digest, and task_set_digest equals the spec's set digest;
      • git_dirty is False and git_commit is a 40-hex SHA;
      • there is no runs/ and no spend.jsonl;
      • the spec facts that were checked before are unchanged.
    • Terminal.test_score_and_report_of_e5_unregistered_and_registered_with_no_runs replaces test_score_and_report_refuse_an_unregistered_eval. It works on temporary copies of evals/ and asserts:

      • with the registration left out of the copy, score and report exit 2 with spec_unregistered;
      • with it, score --json exits 0 with runs == {}, and report exits 2 with runs_missing, "48 of 48";
      • the checkout's E5 folder lists the same files before and after.

      The old test ran eval score on the checkout itself. With a registration present, that wrote index.json into the tree.

    • Terminal.test_tasks_list_and_a_dry_run_of_e5 now uses the committed registration on a copy, instead of registering inside the copy.

  3. docs/EVAL.md:
    • E5's status reads "registered at 90930320c6, not run", with the digests and git_dirty: false.
    • The register line is gone from the E5 command block.
  4. spec/GAPS.md, entry 62 (10): updated to match.

Commands and results

  • Targeted, local (Windows 11, CPython 3.12): tests.test_eval_core and tests.test_gaps gave Ran 26 tests, OK.
    • With the registration removed, the new tests fail: 1 failure and 2 errors in test_eval_core. So they depend on the registration.
    • No bite run on CI. This is a data and test-expectation change.
  • Full suite, GitHub Actions, scratch branch syberlabs/7c38fc5e-ci, commit f793cd0, tree 30dd76c602 (identical to the head's): https://github.com/SyberLabs/SyberWork/actions/runs/40832177779.
    • Still in progress when this comment was written.
    • The scratch branch's factory-gates, g2, g3 and g4 workflows all succeeded.
  • python -m conformance.run: matched 23 golden traces.
  • python benchmarks/run_scenarios.py: 11 replayed, exit 0, digest b41f7cbb… (unchanged).
  • Thread checks on c1 (4 files, 22,867 diff bytes): conformance, critic.rules, scenarios and tests all passed. findings c1: no findings.
  • certify c1: result dfdfddfb73 (tree 30dd76c602) on repo-change.v5, class R; four checks passed; chain head bd88941250 at 14 events.
    • Signed with key 4b6ae2deea. I checked the key against the factory's test-integrity-2-74e7d4ac.cert.pub before signing, and the written .pub is byte-identical to it.
  • verify against that .pub with --repo .: signature ok; repository ok (4 paths, diff digest match); journal ok (14 of 14 events). Verdict: pass.
  • eval preflight e5-single-vs-repair, run on a clean detached worktree at the head dfdfddfb73, with the seed and GitHub tokens unset: passed, dirty: false.
    • tests: exit 0, Ran 1232, OK (skipped=26, expected failures=4).
    • conformance: exit 0.
    • scenarios: exit 0.
    • eval_tests: exit 0, Ran 76.
    • Afterwards the only untracked file was preflight.json. No test wrote anything into the checkout.
  • This preflight is evidence only. preflight.json is not committed: EVAL.md calls it "the last preflight, for one commit", and run refuses preflight_stale. So E5's run needs a new preflight on post-merge main.
  • Publish: push_branch and open_pull_request both succeeded on the first attempt.

Claim ceiling

Implemented and tested. Nothing measured: E5 is registered and not run.

Known limitations

  • The registration records 90930320c6 as its commit. Merging this PR makes a new commit, so E5's run must preflight on that merge commit. The registration itself stays valid, because register and run check digests, not the commit.
  • The probe of 2026-10-29 confirmed that on main before this PR, register followed by preflight failed: the two old tests failed once registration.json existed.

Next permissible packet

The paid E5 run, after the owner's go: eval preflight, then eval run, score and report on post-merge main. The cap is $10.00, plus at most one call's cost.

Never merged, never force-pushed, nothing pushed to main, no setting changed, PR body untouched, no model, Jev or GitHits call made.

@sykosyber

Copy link
Copy Markdown
Member Author

Independent verification: PR #152 at dfdfddfb73 (base 90930320c6)

Verdict: pass with findings. All findings are informational. None blocks the merge.

Claims checked

  1. Diff scope: exactly 4 files: evals/e5-single-vs-repair/registration.json (new), tests/test_eval_core.py, docs/EVAL.md, and spec/GAPS.md. In GAPS only line 125 (entry 62) changes, and within it only item (10). No other entry is renumbered. The PR is mergeable and its base is current main.

  2. Registration:

    • spec.spec_digest(spec.load(...)) on the committed spec gives 7625340097b9df1a…, which matches.
    • The spec's declared set digest and the recomputed task digests give 13589c4ff8e0…, which matches.
    • spec.registered() accepts the registration.
    • git_commit is 90930320c6… and git_dirty is false.
    • spec.json and evals/tasks are byte-identical to base.
    • The spec gives 48 planned runs, dollars 1.0, total_dollars 10.0, strict true and hidden_feedback tails.
    • Running spec.register on a temp copy of evals/ with no registration reproduces the same spec_digest, task_set_digest, all 8 task_digests, both arm_digests, schema and eval. Only the commit and timestamp differ, as expected.
  3. Flipped tests:

    • The tests assert that E5 is registered and not run: no runs/ and no spend.jsonl.
    • Score and report run only on tempfile copies, and the test asserts the E5 folder listing is unchanged.
    • git status --porcelain was empty after every local run.
    • On a copy with registration.json deleted, 3 tests fail: 2 errors and 1 failure.
    • The dry-run test now uses the committed registration instead of registering in the copy. register is still covered by the SpecRegistration tests, so no coverage is lost.
  4. Preflight gate: run calls preflight_gate (syberlabs/eval/runner.py:463-475) and calls it again before every run (:504-510). The gate requires preflight.json (:750), with commit == HEAD (:758) and a clean tree outside evals/ (:761). preflight.json is not committed, so a real run needs a fresh preflight on the commit it runs from, which after merge is the merge commit.

  5. Docs: docs/EVAL.md:206 and spec/GAPS.md:125 (62(10)) describe the state accurately: registered at 90930320c6, git_dirty: false, not run, and a preflight is needed before a run.

  6. Local runs (Python 3.12, Windows):

    • tests.test_eval_core and tests.test_gaps: 26 OK.
    • test_eval_adversarial, test_eval_runner and test_eval_score: 54 OK (1 skipped, 4 expected failures).
    • conformance.run: 23 golden traces matched.
    • benchmarks/run_scenarios.py: 11 scenarios replayed, exit 0.

    CI:

    • Run 40832177779 on syberlabs/7c38fc5e-ci succeeded. Its head commit is f793cd083e, whose tree 30dd76c602 is identical to this PR head's tree.
    • All 25 checks on this PR pass, including both windows jobs. None failed or is pending.
  7. Certificate: syberlabs verify --public <cert.pub> --repo . gives:

    • signature ok: ed25519 4b6ae2deea, thread 7c38fc5e, candidate c1, result dfdfddfb73, class R.
    • repository ok: base and result present, trees match, 4 changed paths and diff digest match.
    • journal not checked, because no journal was available.
    • verdict pass.

Findings (all informational)

  1. Info: the clean-tree origin is held by the test, not the code. syberlabs/eval/spec.py:345 registered() checks only the spec, task and arm digests. Nothing at run time reads git_commit or git_dirty. The clean-tree claim rests on the assertion in tests/test_eval_core.py:288 and on review. This is acceptable, because the commit actually used is held to the preflight gate.
  2. Info: an older GAPS item keeps the historical wording. spec/GAPS.md:121 (60(10)) still says E5's registration was written from a dirty tree at 7caa7c25b3. That is accurate history for that entry and is superseded by 62(10), but a reader of 60 alone could be misled.
  3. Info: the dry-run test copies evals/ with fewer ignore patterns than the score test. tests/test_eval_core.py:498 copies evals/ without ignoring preflight.json or report*, unlike :467-480. A local preflight.json would be copied in. Under --dry a refused or present gate does not change the 48 planned runs, so this is harmless.

Not done: E5 was not run, no paid model, Jev or GitHits was called, and nothing was pushed or merged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant