Skip to content

Record the v3 Jev measurement's results in docs/EVOLUTION.md and docs/MODEL_COMPARISON.md, and close three findings from PR #152's verifier: refuse an eval registration made from a dirty tree, cross-r - #161

Merged
sykosyber merged 1 commit into
mainfrom
syberlabs/4eb3eadc
Nov 2, 2026

Conversation

@sykosyber

Copy link
Copy Markdown
Member

Objective: Record the v3 Jev measurement's results in docs/EVOLUTION.md and docs/MODEL_COMPARISON.md, and close three findings from PR #152's verifier: refuse an eval registration made from a dirty tree, cross-reference GAPS 60 (10) to 62 (10), and make the dry-run test ignore preflight and report files. No defaults change and no new GAPS entry is added.

Class: R
Agent-platform: claude@opus-5.5
Admitted-by: sykosyber
Reviewed-by: rules@2 (critic check critic.rules: high 0, medium 0, low 0)
Evidence: tested

Accepted in SyberLabs thread 4eb3eadc-37d6-4437-b198-b18027e7e449 as candidate c1 (commit 44cdb4e488f49de64f0f98f0b53a32a70110ffc1).
Host checks on that tree: conformance passed, critic.rules passed, scenarios passed, tests passed.

…egistration at run; GAPS 60 (10) cross-reference; dry-run test ignores preflight and reports

SyberLabs-Thread: 4eb3eadc-37d6-4437-b198-b18027e7e449
SyberLabs-Candidate: c1
SyberLabs-Provider: claude@opus-5.5
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-02T08:08:12.623725Z 44cdb4e PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@sykosyber

Copy link
Copy Markdown
Member Author

Scope: owner direction in the current session (SyberFactory consolidate, bug-proof and elevate workflow), packet eval-docs-followups, Class R. Produced by Build Thread 4eb3eadc on a fresh clone, contract repo-change v5, unattended runner, base main at dc69499491 (#156).

What changed

  1. The v3 Jev measurement's results are in the docs. docs/EVOLUTION.md has a new "Results" block under the measurement plan. The plan heading now says "run 2026-10-28 on main at 5ebbbce" where it said "not run". Two other sentences that said the run was still to come now point to the result. docs/MODEL_COMPARISON.md has a new "Note of 2026-10-28" and the "Where that run goes" paragraph points to it. Every number was copied from reports/2026-10-28-jev-v3-measurement.md in the officer workspace and checked against measure-r3.json in its receipts. I re-summed the 48 rows per arm and got the same results:

    • single 4/12, $0.119276
    • evolve 11/12, $0.441426
    • evolve_jev 10/12, $0.483434
    • evolve_jevapi 9/12, $0.525576 Anthropic plus $0.000804468 Jev, $0.526380 in total

    Neither pre-registered rule was met. TypeSafe Jev chose mutate on 21 of 21 asked decisions and agreed with local_action on 0 of 21. Defaults are unchanged. The docs also record the plan-section SHA-256 at 5ebbbce and the one stated substitution (the adapter was given by its absolute path).

  2. PR Register the pre-registered eval E5 on clean main (90930320c6) and commit its registration, with the committed-E5 tests, docs/EVAL.md and GAPS 62 (10) updated to a registered, unrun E5 #152 verifier, item (a). I added spec.clean_registration(registration). It refuses with registration_dirty when git_dirty is not false or git_commit is not a 40- or 64-hex commit id. EvalRun.run calls it once, before the preflight gate, whenever the gate is required. A dry run reports the refusal in its preflight field. I deliberately left registered() unchanged, so score and report still read such an eval. This is in the docstrings and in docs/EVAL.md.

  3. Item (b). spec/GAPS.md entry 60 item (10): only that line changed. It now cross-references entry 62 (10) and names the new refusal. No GAPS entry was added and no other entry changed.

  4. Item (c). The E5 dry-run test copies evals/ ignoring preflight.json and report*, as the score test does. It now also asserts preflight == "preflight_missing", which shows that the committed E5 registration passes the new clean-origin check.

Tests: there is one new test, Preflight.test_a_registration_written_from_a_dirty_tree_refuses_to_run. It covers git_dirty true and null, plus a missing and a malformed commit, through both a real run and a dry run, and confirms that nothing ran. Two existing tests that run the gate now register against their own clean temp harness instead of the developer's checkout, so they no longer depend on whether that checkout is dirty: Preflight.setUp in test_eval_core and the adversarial test_the_preflight_gate_holds_for_every_run_not_only_the_first.

Files (8, +86/-11)

  • docs/EVAL.md
  • docs/EVOLUTION.md
  • docs/MODEL_COMPARISON.md
  • spec/GAPS.md
  • syberlabs/eval/runner.py
  • syberlabs/eval/spec.py
  • tests/test_eval_adversarial.py
  • tests/test_eval_core.py

Commands run (Windows, Python 3.12.6, clone C:/Users/MATEO/AppData/Local/Temp/sfv/evf)

  • python -m unittest discover -s tests -p "test_eval_*.py": Ran 76 tests in 108.602s / OK (skipped=1, expected failures=4).
  • python -m unittest tests.test_gaps tests.test_model_comparison: Ran 41 tests / OK.
  • python -m conformance.run: matched 23 golden traces.
  • python benchmarks/run_scenarios.py: scenarios: 11 replayed, exit 0, digest b41f7cbb…1f44bb.
  • Thread checks on c1, run by the host on the candidate tree:
    • tests (sharded): Ran 1263 tests in 407.515s / OK (skipped=26, expected failures=4)
    • conformance: matched 23 golden traces
    • scenarios: pass
    • critic.rules: no findings over 8 changed path(s)
  • run --once: check c1 -> acceptable. A second run --once: accept c1 -> succeeded, target refs/heads/syberlabs/4eb3eadc -> 44cdb4e488, spend $0.0000 / $5.0000.
  • certify c1: ed25519 key 3312866a18 (the rotated factory key, loaded from the User scope and never printed), result 44cdb4e488, chain head 9deddceb31 at 13 events.
  • verify --public factory-key-current.pub --repo .: signature ok, repository ok, journal ok (13 of 13), verdict pass.
  • CI on scratch branches:
    • syberlabs/4eb3eadc-ci (full change): ci success, plus factory-gates and the g2, g3 and g4 checks.
    • syberlabs/4eb3eadc-bite (test changes only, on old code): ci failed on 3.11, 3.12, 3.13, Windows and G8, each with exactly one error: the new test, AttributeError: module 'syberlabs.eval.spec' has no attribute 'clean_registration'.
  • conformance.clean_install: not run. No build, cli, publish, gitspace or checks module was touched.

Claim ceiling. The code change is implemented and tested. The docs change transcribes a measurement made on 2026-10-28; I made no model call and no Jev call for it.

Known limitations

  • The bite fails because the new function is missing, not on a behaviour assertion. A stub clean_registration that does nothing would still fail the test on the run and dry-run assertions, but CI has not shown that.
  • spec/GAPS.md entry 61 (4) still says the Jev measurement is "unrun". The packet allowed editing only entry 60's line, so it needs a one-line follow-up after the parallel GAPS packet lands.
  • docs/ROADMAP_STATUS.md was not touched, because Add the factory: Space-Time Parsing of a repository on one ledger (v0 slice) #154 edits it.
  • The accepted commit message is generated by the product and has no Co-Authored-By trailer.

Unresolved questions. Should score and report also flag a dirty-origin registration, or is refusing at run time enough? Today they ignore it.

Freeze. Nothing on the pilot-freeze list (SW-01/03/04/05/07/08/10, FS-01/02) was touched.

Next permissible packet. After the parallel GAPS packet merges: amend GAPS 61 (4) to point at the result, and optionally record the measurement in docs/ROADMAP_STATUS.md once #154 lands.

🤖 Generated with Claude Code

@sykosyber

Copy link
Copy Markdown
Member Author

Independent verification of PR #161 (eval-docs-followups, head 44cdb4e488, base main at dc69499491): pass with findings (low or informational only).

I checked the claims read-only in a fresh clone at C:/Users/MATEO/AppData/Local/Temp/v-eval-docs-followups (Windows, Python 3.12) and deleted the clone afterwards. I made no model calls and no GitHits calls.

1. Brief items

  • Docs (item 1): correct. I checked every figure in docs/EVOLUTION.md and docs/MODEL_COMPARISON.md against reports/2026-10-28-jev-v3-measurement.md. That covers the per-arm table, the per-fixture solves, 1.095 and $0.220713, 21 decisions with 0 agreeing, the crossover median of 0.37 (range 0.24 to 0.49), 188.8 ms, 19,154 tokens, $1.987102 and $0.001034670, 36.8% and 0.69%, the plan SHA 5e8f98a4…, 64 rows and 27 calls, and the pilot's 1/4, six runs and 6 asked decisions. All match. Commit 5ebbbce is cited. The heading no longer says "not run", and the two sentences that said the run was pending now point to the result.
  • 2(a): done in code. spec.clean_registration is at syberlabs/eval/spec.py:373-380. EvalRun.run calls it at syberlabs/eval/runner.py:467 whenever preflight is required. The CLI run always requires preflight, and only tests pass require_preflight=False. The committed E5 registration (90930320c6, git_dirty: false) passes the check.
  • 2(b): done. Only line 121 of spec/GAPS.md (entry 60) changed. It now cross-references entry 62 (10). tests.test_gaps passes.
  • 2(c): done. tests/test_eval_core.py:531 ignores preflight.json and report* when copying, and the test now asserts preflight_missing.
  • No new GAPS entry was added.

2. Does the new test fail on base and pass on head? Yes. With the base syberlabs/ and the head tests, the new test errors with an AttributeError. I then disabled the assertIs(spec.clean_registration(...)) line, to rule out a failure that only shows the function is missing. On base code the test then fails on behaviour: AssertionError: Rejected not raised at tests/test_eval_core.py:463. On head it passes.

3. Freeze list: nothing touched. The diff covers 8 files: eval spec.py and runner.py, docs, GAPS 60 and the two eval test files. None of the SW-01/03/04/05/07/08/10 or FS-01/02 code was changed.

4. Local runs on head

  • tests.test_eval_core and tests.test_eval_adversarial: 44 tests, OK (1 skipped, 4 expected failures).
  • All test_eval_*: 76 tests, OK.
  • The same 76 with a deliberately dirtied tracked file: still OK, which confirms the change to the clean temp harness.
  • tests.test_model_comparison and tests.test_jev_api: 65 tests, OK.
  • python -m conformance.run: matched 23 golden traces.
  • benchmarks/run_scenarios.py: 11 replayed, exit 0, digest b41f7cbb…1f44bb.

5. CI: every check passed. That includes test 3.11, 3.12 and 3.13, windows, G2 to G8, admit, compose and policy.

6. Certificate: verified. I verified eval-docs-followups-4eb3eadc.cert.json against factory-key-current.pub with signature ed25519 3312866a18. The repository check matched base, result, trees, the 8 paths and the diff digest. In the executor's clone the journal check passed (13 of 23 events certified, head 9deddceb31), with verdict pass. The .pub written beside the cert differs from factory-key-current.pub only in its line ending (LF against CRLF).

7. Conflict with the parallel GAPS packet: none expected. I simulated an entry appended after entry 64 on base and merged this PR into it. Git auto-merged with no conflict, because the two hunks are separate.

Findings

  • Low: spec/GAPS.md:123 (entry 61 (4)) still calls the Jev measurement "unrun". This is out of scope by the brief and the executor disclosed it. It needs a one-line follow-up after the parallel GAPS packet lands.
  • Info: the bite in CI showed only the AttributeError. The behavioural failure on base is confirmed locally, as described in section 2.
  • Info: score and report do not flag a registration made from a dirty tree. This is a deliberate choice and is documented in docs/EVAL.md (Preflight) and syberlabs/eval/spec.py:19-23. The owner may want to decide whether to keep it.
  • Info: docs/ROADMAP_STATUS.md was not updated, deferred to Add the factory: Space-Time Parsing of a repository on one ledger (v0 slice) #154.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant