Skip to content

Close the independent verifier's findings on #150 (eval-score-3): the report refuses an unpriced ledger line, a huge integer in the ledger is eval_spend_unknown, the hidden_tampered rule requires the - #151

Merged
sykosyber merged 2 commits into
mainfrom
syberlabs/c73912bf
Oct 29, 2026

Conversation

@sykosyber

@sykosyber sykosyber commented Oct 29, 2026 •

Copy link
Copy Markdown
Member

Objective: Close the independent verifier's findings on #150 (eval-score-3): the report refuses an unpriced ledger line, a huge integer in the ledger is eval_spend_unknown, the hidden_tampered rule requires the runner's hash points, and EVAL.md states the after_check coverage and the spend ceiling exactly

Class: R
Agent-platform: claude@opus-5.5
Admitted-by: sykosyber
Reviewed-by: rules@2 (critic check critic.rules: high 0, medium 0, low 0)
Evidence: tested

Accepted in SyberLabs thread c73912bf-1c64-4343-a513-71db9b15311d as candidate c1 (commit 08252d39be00b459dfef6b08fffaac92c0b7ad53).
Host checks on that tree: conformance passed, critic.rules passed, scenarios passed, tests passed.

Reviewed-by: claude (independent verifier session, SyberFactory workflow)

…es unpriced lines, huge-int ledger is eval_spend_unknown, hidden_tampered requires the runner's hash points, EVAL.md after_check an

SyberLabs-Thread: c73912bf-1c64-4343-a513-71db9b15311d
SyberLabs-Candidate: c1
SyberLabs-Provider: claude@opus-5.5
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-29T00:35:48.484668Z 08252d3 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@sykosyber

Copy link
Copy Markdown
Member Author

Scope: owner direction in session 2026-10-28 (SyberFactory, packet eval-score-3). It closes the independent verifier's findings 1 to 5 on #150 (merged as b2763fb4c4): #150 (comment).

Packet eval-score-3, thread c73912bf, candidate c1 (head 08252d39be, tree 4b0af833e3).

Each finding, its fix and its test

Finding Fix Test
1 (medium). report.build added up a ledger line whose unknown_models is non-empty. syberlabs/eval/report.py: the refusal condition now also takes line.get("unknown_models"), which is what EvalRun.spent stops on. So the report refuses that line with eval_spend_unknown, as docs/EVAL.md already said. test_eval_score.EndToEnd: with every run present, an unknown_models: ["m"] line makes report.build raise eval_spend_unknown. With a run missing, report.build(partial=True) raises it too, eval report --partial exits 2, and EvalRun.spent returns the same line as unknown.
2 (low to medium). A 401-digit integer in dollars raised a bare OverflowError from math.isfinite. syberlabs/eval/runner.py ledger_dollars: the math.isfinite / float conversion catches OverflowError and returns None. The line is unknown, so EvalRun.spent and report.build give the coded eval_spend_unknown. test_eval_runner.SpendLedger.test_a_ledger_line_is_read_strictly_...: a new 401-digit integer subtest (spent reports it unknown, sums only the good line, and run_eval stops eval_spend_unknown). ledger_dollars also returns None on 10**400 and -(10**400). EndToEnd: the same line makes report.build(partial=True) raise eval_spend_unknown.
3 (low). EVAL.md:86 read as if candidate code ran just before after_check. Docs only. docs/EVAL.md lists the four points and what each covers. after_propose follows every check run inside propose, the repair loop's evaluations included, all together. after_check follows Thread.check, which runs the checks again only when the chosen candidate's result is missing or stale, so for a repair or single arm it usually follows a check that ran nothing. Step 4 of the run says the same. spec/GAPS.md 62 (5) says the same. No code change. None (docs).
4 (low). hidden_tampered passed on whatever hashes were present, and score.py:148 read fewer signals than the gate. syberlabs/eval/gate.py: the rule now requires staged and after_propose (gate.HIDDEN_POINTS) for every run. When the journal shows a started promotion, or the outcome says accepted, it also requires after_check and before_accept (gate.HIDDEN_ACCEPT_POINTS). The runner writes after_check before every before_accept, so it is required alongside it. A missing point gives not_computable with the points named, and the run is incomplete, never pass. A detected change still fails first. syberlabs/eval/score.py: quality is trusted exactly when the gate's hidden_tampered rule passed. A fail or a not_computable gives trusted: false and the verdict fail. test_eval_score.GateRules.test_hidden_tampered_requires_every_hash_the_runner_always_writes: with each of the four points dropped in turn, on an accepted journal and on one with no promotion. The test also covers a promotion started while the outcome says not_acceptable (the journal decides), a change beside a missing point (still fail), and that the gate's points equal runner.INTEGRITY_POINTS. EndToEnd: each real bundle is edited by hand to drop each required point, then scored with score.score_run. The result is hidden_tampered: not_computable, gate incomplete, and quality trusted: false, fail. With the bundle restored, quality is trusted again.
5 (info). EVAL.md:148 described a ceiling below per-run dollars that cannot happen. Docs only. Within one runner the min(per-run, remaining) ceiling always equals the per-run dollars. The spend is recounted just before the budget check, the check requires spend + per-run <= total, and the lock keeps other runners out. The min stays as a defensive bound. None (docs).

Commands and results

  • Targeted, local (Windows 11, CPython 3.12.6): python -m unittest tests.test_eval_core tests.test_eval_runner tests.test_eval_score tests.test_eval_adversarial tests.test_providers tests.test_gaps tests.test_package gave Ran 89 tests in 140.703s, OK (skipped=1, expected failures=4).
  • Conformance, local: python -m conformance.run gave matched 23 golden traces, exit 0.
  • Scenarios, local: python benchmarks/run_scenarios.py gave scenarios: 11 replayed, exit 0, digest b41f7cbb2e70447c4e57bf7c253b976b81cd7ab1f95daf7b9c459564db1f44bb.
  • Thread checks on c1 (first syberlabs run --once): conformance, critic.rules, scenarios and tests (sharded) all passed, so c1 was acceptable. The second run --once gave accept c1 -> succeeded.
  • Full suite, GitHub Actions, scratch branch syberlabs/c73912bf-ci (7e9932f653, tree 4b0af833e3, the same as c1's): https://github.com/SyberLabs/SyberWork/actions/runs/40793534937, success.
    • test (3.11), (3.12) and (3.13): Ran 1232 tests, OK (skipped=2, expected failures=5).
    • G8 strict 3.12: the same.
    • windows: Ran 1232 tests, OK (skipped=26, expected failures=4).
    • compose: success.
  • Bite, GitHub Actions, scratch branch syberlabs/c73912bf-bite (ed111f99e1: only the two test files, on b2763fb4c4): https://github.com/SyberLabs/SyberWork/actions/runs/40793534969, failure, as intended.
    • Every test leg, G8 strict and windows: Ran 1232 tests, FAILED (failures=8, errors=2, ...). compose passed.
    • The failures and errors:
      • GateRules.test_hidden_tampered_requires_every_hash_the_runner_always_writes, 7 failures, each 'pass' != 'not_computable': accepted with staged, after_propose, after_check or before_accept missing; no promotion with staged or after_propose missing; and the case where the journal shows a promotion while the outcome says not_acceptable.
      • EndToEnd.test_run_score_and_report_cite_every_digest_and_apply_the_rule, Rejected not raised on the unknown_models line (finding 1).
      • SpendLedger.test_a_ledger_line_is_read_strictly_..., 2 errors, OverflowError: int too large to convert to float: the 401-digit integer subtest and the direct ledger_dollars(10**400) (finding 2).
  • Certificate: syberlabs certify c1 --seed-env SYBERLABS_CERTIFICATE_SEED used the factory key (ed25519 4b6ae2deea, checked against test-integrity-2-74e7d4ac.cert.pub before signing). syberlabs verify c73912bf-c1.json --public test-integrity-2-74e7d4ac.cert.pub --repo . gave:
    • signature ok (thread c73912bf, candidate c1, result 08252d39be, repo-change.v5, class R);
    • repository ok (trees match, 8 changed paths, the diff digest matches);
    • journal ok (14 of 14 events certified, head 318fcb9f29);
    • verdict pass.
  • Publish: push_branch succeeded. The first open_pull_request was recorded effect_unknown (destination_error). syberlabs recover first gave pending (absent_within_settle_window), and after the 120-second window not_applied (destination_state_unchanged); no pull request existed. The second attempt opened this one. TLS verification was never disabled.
  • gh pr checks 151: all 25 checks pass.

Claim ceiling

Implemented and tested: the five fixes above, on synthetic journals, stand-in adapters and the simulated end-to-end eval. Nothing was measured with a model.

Known limitations

  • The hidden_tampered requirement depends on the bundle's hidden_integrity and journal, neither of which is signed. A consistent rewrite of a bundle that adds the missing hashes with matching digests is still not detected. That is the same limit as EvalRun.verify ("detects edits and does not prevent a consistent rewrite").
  • Finding 3 is documented, not changed: there is still no hash after each repair evaluation. feedback_filter remains the natural hook if the lead wants one.
  • A bundle written before eval-score, or edited to drop a hash, is now scored with untrusted quality (fail) where it used to be trusted. No committed eval holds such a bundle.

Unresolved questions

None from this packet.

Next permissible packet

eval-safety-fixtures (GAPS 62 (1)), or the E5 re-registration on a clean main, as the lead orders.

The scratch branches syberlabs/c73912bf-ci and syberlabs/c73912bf-bite are left for the orchestrator to delete.

@sykosyber

sykosyber commented Oct 29, 2026 •

Copy link
Copy Markdown
Member Author

Independent verification: #151 (eval-score-3) at 08252d39be

Verdict: pass, with two non-blocking notes. I checked all five fixes from the #150 verification against the code, and each is in place. The local tests, conformance, both CI runs, gh pr checks and the certificate all check out. A rebuild onto current main is clean. The new hidden_tampered requirement cannot make an honest run that was not accepted read as incomplete.

Reviewed: head 08252d39be00b459dfef6b08fffaac92c0b7ad53, tree 4b0af833e3, base b2763fb4c4, thread c73912bf. I worked in a fresh clone on Windows 11 with CPython 3.12.6. I wrote no new attack code, made no model, Jev or GitHits calls, and pushed nothing.

Findings, most important first

  1. Low (evidence, not code): no bite result on record shows that the score.py change is needed. (tests/test_eval_score.py:505-512 and 553-571, syberlabs/eval/score.py:151-152)
    • In bite run 40793534969, EndToEnd fails at its first new assertion: the unknown_models line at :505-512 gives Rejected not raised.
    • So the later assertions at :553-571 never ran against the old code. Those assertions edit a bundle by hand and expect quality trusted: false. GateRules tests the gate only.
    • On reading, the code is correct. score.py now takes trusted from the gate's hidden_tampered verdict, so a missing point makes quality untrusted, which the old signals did not do. The bite just does not demonstrate it on its own.
    • Not blocking. If the lead wants that evidence, a bite with only score.py reverted would show it.
  2. Info: two bundle-dollar readers still call bare float(). (syberlabs/eval/runner.py:384 EvalRun.dollars, syberlabs/eval/report.py:262 per_arm_bundle_dollars)
    • Both read a bundle's outcome.dollars, not a ledger line, so they are outside finding 2.
    • Every ledger path is now guarded: EvalRun.spent (:416) and report.build (:236) both go through ledger_dollars, and the sum at report.py:242 runs only after the refusal.
    • I did not trace whether a 401-digit integer could reach outcome.dollars in a bundle that EvalRun.verify accepts, since verify requires it to equal the journal's spend.

Each fix

# Fix Result
1 report.py refuses an unknown_models line Confirmed. report.py:236 refuses a line when "unreadable" in line, ledger_dollars(line) is None, or line.get("unknown_models") is truthy. This is the same three-way, truthiness-based condition EvalRun.spent uses (runner.py:416-423). The refusal comes before any sum and before the partial handling. The test covers report.build with every run present, report.build(partial=True), and eval report --partial exiting 2.
2 ledger_dollars handles OverflowError Confirmed. runner.py:251-264 rejects a non-int/float type, then wraps math.isfinite, the < 0 check and float() in try/except OverflowError and returns None. Both callers turn None into the coded eval_spend_unknown. The subtests cover a 401-digit integer through spent, run_eval and report.build(partial=True), plus 10**400 and -(10**400) passed directly. No other ledger path parses dollars unguarded (finding 2 covers the bundle paths).
3 after_propose and after_check wording Accurate. The repair loop's evaluations run inside thread.propose (runner.py:572-575), and intact("after_propose") follows them. Thread.check re-runs only when run and in_scope_ and (rerun or state in ("none", "stale")) (build.py:1177). EVAL.md:86-93, step 4 (EVAL.md:147) and GAPS 62 (5) all say this.
4 Gate hidden_tampered requirement Confirmed. Details below.
5 EVAL.md ceiling wording Accurate. spent is recounted after every run (runner.py:514) and before the first. The guard at :499 requires spent + per-run <= total, and the lock is held throughout. So min(per-run, total - spent) at :511 always equals the per-run dollars. EVAL.md:155 states this and keeps the min as a defensive bound.

Fix 4 in detail (gate.py:473-499, score.py:151-152):

  • A detected change still fails first. The FAIL branch runs before the missing-point check. The test confirms that a change next to a missing point is still fail.

  • A missing point is not_computable, and the reason names it. staged and after_propose are always required. after_check and before_accept are also required when _promoted(ev) is non-empty or the outcome says accepted. Any not_computable makes evaluate return incomplete (gate.py:556).

  • after_check always comes before before_accept on every accepting path. thread.accept is called in exactly one place, runner.py:583-584: if intact("after_check") and verdict.acceptable and intact("before_accept"): receipt = thread.accept(chosen). Because of short-circuit evaluation, after_check is written first, before_accept only after it, and accept only after both.

    • A promotion effect can start only through thread.accept, because the run contract's only action is accept_change. So a journal with a started promotion always has all four points.
    • An exception anywhere before the bundle is written aborts the run without a bundle (runner.py:632-636), so it is never scored.
  • Honest runs that were not accepted still pass. I traced each non-accepting path:

    • no chosen candidate: staged, after_propose, and the rule passes;
    • a change after the proposal: the rule fails, as it should;
    • the check is not acceptable: staged, after_propose, after_check, with no promotion, and the rule passes;
    • a change at after_check: the rule fails;
    • the accept is refused: all four points are present.

    No honest bundle written by the Land eval-score (#146: the SyberEval safety gate, scorer, paired report, hidden_feedback and spend ledger) on top of the eval-core fixes of #148, with one hidden-copy integrity mechanism and one spend #150 runner or later lacks a required point.

  • score.py sets quality trusted to the hidden_tampered rule's verdict, which is always present in evaluate's results. Anything but pass gives trusted: false and verdict fail (score.py:64-65).

  • The points stay consistent with the runner. HIDDEN_POINTS + HIDDEN_ACCEPT_POINTS == runner.INTEGRITY_POINTS (runner.py:110), and the test asserts this.

Scope: 8 files, 133 lines added and 17 removed: docs/EVAL.md, spec/GAPS.md (entry 62 only), syberlabs/eval/{gate,report,runner,score}.py and tests/test_eval_{runner,score}.py. No contract schema, event kind, golden trace, conformance/ or syberlabs/cli.py change.

Tests and CI

  • Local, fresh clone at 08252d39be: python -m unittest tests.test_eval_core tests.test_eval_runner tests.test_eval_score tests.test_eval_adversarial tests.test_gaps gave Ran 80 tests in 122.231s, OK (skipped=1, expected failures=4). The skip is gap 16, which needs SIGKILL and so does not run on Windows.
  • Local: python -m conformance.run gave matched 23 golden traces, exit 0.
  • Full run 40793534937, on syberlabs/c73912bf-ci at 7e9932f653. Its tree 4b0af833e3 is identical to the head's. Success, with every job passing:
    • test (3.11), (3.12), (3.13) and G8 strict: Ran 1232 tests, OK (skipped=2, expected failures=5);
    • windows: Ran 1232 tests, OK (skipped=26, expected failures=4);
    • compose: success.
  • Bite run 40793534969, on ed111f99e1: its parent is b2763fb4c4, it changes only the two test files, and they are identical to the head's (git diff --quiet). Failure, as intended.
    • Every test leg, G8 and windows: Ran 1232 tests, FAILED (failures=8, errors=2, ...). compose passed.
    • The 8 failures:
      • 7 in GateRules.test_hidden_tampered_requires_every_hash_the_runner_always_writes, each 'pass' != 'not_computable': accepted with staged, after_propose, after_check or before_accept missing; not accepted with staged or after_propose missing; and the case where a promotion started but the outcome says not_acceptable;
      • 1 in EndToEnd, Rejected not raised.
    • The 2 errors are both in SpendLedger.test_a_ledger_line_is_read_strictly_…, each OverflowError: int too large to convert to float: the 401-digit integer subtest and the direct ledger_dollars call.
    • These match the handoff exactly, and nothing else fails.
  • gh pr checks 151: all 25 checks pass.

Certificate

I ran python -m syberlabs verify --public test-integrity-2-74e7d4ac.cert.pub --repo . eval-score-3-c73912bf.cert.json in the fresh clone:

  • signature ok ed25519 4b6ae2deea: thread c73912bf, candidate c1, result 08252d39be (repo-change.v5, class R);
  • repository ok: base b2763fb4c4 and the result are present, the trees match, 8 changed paths, and the diff digest matches;
  • journal skipped, because a fresh clone has no .syberlabs journal;
  • verdict pass (journal not checked).

The receipt's .cert.pub is byte-identical to the factory key.

The receipt journal has 31 events and chain_valid: true:

  • accept c1 at events 11-14, and push_branch at 16-19;
  • the first open_pull_request at 21-26: effect_unknown, then reconciliation_checked pending (absent_within_settle_window), then effect_rejected not_applied;
  • the retry at 28-31, which succeeded.

gh pr list --head syberlabs/c73912bf --state all returns only #151, so the failed attempt left no duplicate pull request.

Rebuild preview

origin/main is still b2763fb4c4, the PR's base. I cherry-picked 08252d39be onto it in a local worktree.

  • It applied cleanly.
  • 8 files, 133 lines added and 17 removed.
  • Tree 4b0af833e3, identical to the head's.

Nothing was pushed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant