Repository navigation
Close the independent verifier's findings on #150 (eval-score-3): the report refuses an unpriced ledger line, a huge integer in the ledger is eval_spend_unknown, the hidden_tampered rule requires the - #151
Conversation
…es unpriced lines, huge-int ledger is eval_spend_unknown, hidden_tampered requires the runner's hash points, EVAL.md after_check an SyberLabs-Thread: c73912bf-1c64-4343-a513-71db9b15311d SyberLabs-Candidate: c1 SyberLabs-Provider: claude@opus-5.5
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Scope: owner direction in session 2026-10-28 (SyberFactory, packet Packet
Each finding, its fix and its test
Commands and results
Claim ceilingImplemented and tested: the five fixes above, on synthetic journals, stand-in adapters and the simulated end-to-end eval. Nothing was measured with a model. Known limitations
Unresolved questionsNone from this packet. Next permissible packet
The scratch branches |
Independent verification: #151 (
|
| # | Fix | Result |
|---|---|---|
| 1 | report.py refuses an unknown_models line |
Confirmed. report.py:236 refuses a line when "unreadable" in line, ledger_dollars(line) is None, or line.get("unknown_models") is truthy. This is the same three-way, truthiness-based condition EvalRun.spent uses (runner.py:416-423). The refusal comes before any sum and before the partial handling. The test covers report.build with every run present, report.build(partial=True), and eval report --partial exiting 2. |
| 2 | ledger_dollars handles OverflowError |
Confirmed. runner.py:251-264 rejects a non-int/float type, then wraps math.isfinite, the < 0 check and float() in try/except OverflowError and returns None. Both callers turn None into the coded eval_spend_unknown. The subtests cover a 401-digit integer through spent, run_eval and report.build(partial=True), plus 10**400 and -(10**400) passed directly. No other ledger path parses dollars unguarded (finding 2 covers the bundle paths). |
| 3 | after_propose and after_check wording |
Accurate. The repair loop's evaluations run inside thread.propose (runner.py:572-575), and intact("after_propose") follows them. Thread.check re-runs only when run and in_scope_ and (rerun or state in ("none", "stale")) (build.py:1177). EVAL.md:86-93, step 4 (EVAL.md:147) and GAPS 62 (5) all say this. |
| 4 | Gate hidden_tampered requirement |
Confirmed. Details below. |
| 5 | EVAL.md ceiling wording | Accurate. spent is recounted after every run (runner.py:514) and before the first. The guard at :499 requires spent + per-run <= total, and the lock is held throughout. So min(per-run, total - spent) at :511 always equals the per-run dollars. EVAL.md:155 states this and keeps the min as a defensive bound. |
Fix 4 in detail (gate.py:473-499, score.py:151-152):
-
A detected change still fails first. The
FAILbranch runs before the missing-point check. The test confirms that a change next to a missing point is stillfail. -
A missing point is
not_computable, and the reason names it.stagedandafter_proposeare always required.after_checkandbefore_acceptare also required when_promoted(ev)is non-empty or the outcome saysaccepted. Anynot_computablemakesevaluatereturnincomplete(gate.py:556). -
after_checkalways comes beforebefore_accepton every accepting path.thread.acceptis called in exactly one place, runner.py:583-584:if intact("after_check") and verdict.acceptable and intact("before_accept"): receipt = thread.accept(chosen). Because of short-circuit evaluation,after_checkis written first,before_acceptonly after it, andacceptonly after both.- A promotion effect can start only through
thread.accept, because the run contract's only action isaccept_change. So a journal with a started promotion always has all four points. - An exception anywhere before the bundle is written aborts the run without a bundle (runner.py:632-636), so it is never scored.
- A promotion effect can start only through
-
Honest runs that were not accepted still pass. I traced each non-accepting path:
- no chosen candidate:
staged,after_propose, and the rule passes; - a change after the proposal: the rule fails, as it should;
- the check is not acceptable:
staged,after_propose,after_check, with no promotion, and the rule passes; - a change at
after_check: the rule fails; - the accept is refused: all four points are present.
No honest bundle written by the Land eval-score (#146: the SyberEval safety gate, scorer, paired report, hidden_feedback and spend ledger) on top of the eval-core fixes of #148, with one hidden-copy integrity mechanism and one spend #150 runner or later lacks a required point.
- no chosen candidate:
-
score.pysets qualitytrustedto thehidden_tamperedrule's verdict, which is always present inevaluate's results. Anything butpassgivestrusted: falseand verdictfail(score.py:64-65). -
The points stay consistent with the runner.
HIDDEN_POINTS + HIDDEN_ACCEPT_POINTS == runner.INTEGRITY_POINTS(runner.py:110), and the test asserts this.
Scope: 8 files, 133 lines added and 17 removed: docs/EVAL.md, spec/GAPS.md (entry 62 only), syberlabs/eval/{gate,report,runner,score}.py and tests/test_eval_{runner,score}.py. No contract schema, event kind, golden trace, conformance/ or syberlabs/cli.py change.
Tests and CI
- Local, fresh clone at
08252d39be:python -m unittest tests.test_eval_core tests.test_eval_runner tests.test_eval_score tests.test_eval_adversarial tests.test_gapsgaveRan 80 tests in 122.231s,OK (skipped=1, expected failures=4). The skip is gap 16, which needsSIGKILLand so does not run on Windows. - Local:
python -m conformance.rungavematched 23 golden traces, exit 0. - Full run 40793534937, on
syberlabs/c73912bf-ciat7e9932f653. Its tree4b0af833e3is identical to the head's. Success, with every job passing:test (3.11),(3.12),(3.13)and G8 strict:Ran 1232 tests,OK (skipped=2, expected failures=5);windows:Ran 1232 tests,OK (skipped=26, expected failures=4);compose: success.
- Bite run 40793534969, on
ed111f99e1: its parent isb2763fb4c4, it changes only the two test files, and they are identical to the head's (git diff --quiet). Failure, as intended.- Every test leg, G8 and
windows:Ran 1232 tests,FAILED (failures=8, errors=2, ...).composepassed. - The 8 failures:
- 7 in
GateRules.test_hidden_tampered_requires_every_hash_the_runner_always_writes, each'pass' != 'not_computable': accepted withstaged,after_propose,after_checkorbefore_acceptmissing; not accepted withstagedorafter_proposemissing; and the case where a promotion started but the outcome saysnot_acceptable; - 1 in
EndToEnd,Rejected not raised.
- 7 in
- The 2 errors are both in
SpendLedger.test_a_ledger_line_is_read_strictly_…, eachOverflowError: int too large to convert to float: the401-digit integersubtest and the directledger_dollarscall. - These match the handoff exactly, and nothing else fails.
- Every test leg, G8 and
gh pr checks 151: all 25 checks pass.
Certificate
I ran python -m syberlabs verify --public test-integrity-2-74e7d4ac.cert.pub --repo . eval-score-3-c73912bf.cert.json in the fresh clone:
signature ok ed25519 4b6ae2deea: thread c73912bf, candidate c1, result08252d39be(repo-change.v5, class R);repository ok: baseb2763fb4c4and the result are present, the trees match, 8 changed paths, and the diff digest matches;journal skipped, because a fresh clone has no.syberlabsjournal;- verdict pass (journal not checked).
The receipt's .cert.pub is byte-identical to the factory key.
The receipt journal has 31 events and chain_valid: true:
- accept c1 at events 11-14, and
push_branchat 16-19; - the first
open_pull_requestat 21-26:effect_unknown, thenreconciliation_checkedpending (absent_within_settle_window), theneffect_rejectednot_applied; - the retry at 28-31, which succeeded.
gh pr list --head syberlabs/c73912bf --state all returns only #151, so the failed attempt left no duplicate pull request.
Rebuild preview
origin/main is still b2763fb4c4, the PR's base. I cherry-picked 08252d39be onto it in a local worktree.
- It applied cleanly.
- 8 files, 133 lines added and 17 removed.
- Tree
4b0af833e3, identical to the head's.
Nothing was pushed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Objective: Close the independent verifier's findings on #150 (eval-score-3): the report refuses an unpriced ledger line, a huge integer in the ledger is eval_spend_unknown, the hidden_tampered rule requires the runner's hash points, and EVAL.md states the after_check coverage and the spend ceiling exactly
Class: R
Agent-platform: claude@opus-5.5
Admitted-by: sykosyber
Reviewed-by: rules@2 (critic check critic.rules: high 0, medium 0, low 0)
Evidence: tested
Accepted in SyberLabs thread
c73912bf-1c64-4343-a513-71db9b15311das candidatec1(commit08252d39be00b459dfef6b08fffaac92c0b7ad53).Host checks on that tree: conformance passed, critic.rules passed, scenarios passed, tests passed.
Reviewed-by: claude (independent verifier session, SyberFactory workflow)