Skip to content

Close the independent verification findings on #147: re-target the pre-registered jev_api measurement to the v3 fixtures with four paid repeats and a sourced cost ceiling, correct the recorded crossov - #149

Merged
sykosyber merged 2 commits into
mainfrom
syberlabs/7ca8f3c4
Oct 28, 2026

Conversation

@sykosyber

@sykosyber sykosyber commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

Objective: Close the independent verification findings on #147: re-target the pre-registered jev_api measurement to the v3 fixtures with four paid repeats and a sourced cost ceiling, correct the recorded crossover range, make the Jev timeout a wall-clock deadline per decision, document fallback.response_digest, and replace the assert in JevFailure with a real check.

Class: R
Agent-platform: claude@opus-5.5
Admitted-by: sykosyber
Reviewed-by: rules@2 (critic check critic.rules: high 0, medium 0, low 0)
Evidence: tested

Accepted in SyberLabs thread 7ca8f3c4-534e-492e-bc25-2f5440e2146b as candidate c1 (commit 78c116c90fc9c0262e3c6906d96ddf2c176a201c).
Host checks on that tree: conformance passed, critic.rules passed, scenarios passed, tests passed.
Reviewed-by: claude (independent verifier session, SyberFactory workflow)

…sover range, wall-clock deadline per Jev decision, response_digest documented, closed-word check without assert

SyberLabs-Thread: 7ca8f3c4-534e-492e-bc25-2f5440e2146b
SyberLabs-Candidate: c1
SyberLabs-Provider: claude@opus-5.5
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-28T22:33:11.427184Z 78c116c PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 78c116c90f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread docs/EVOLUTION.md
Comment on lines +203 to +204
- Hard ceiling: every run spends its whole budget, 1 call for `single` and K = 6 for each of the three evolve arms, so 19 calls per fixture, 76 per repeat and **304 calls** for four repeats. Each call is priced at **$0.0176**, the highest dollars per call of any of the 84 Sonnet 5.5 v3 rows (`evolve` on `gaps_renumber`, $0.0353 for two calls). 304 x $0.0176 = **$5.36**, stated as $5.40. Nothing in the harness stops the total at that number: K bounds each run's calls, and the operator stops the run if the rows' dollars pass it.
- Jev: asked decisions are not bounded by K, because a Jev-chosen crossover uses no model call. The code bound per `evolve_jevapi` run is K mutate decisions plus one decision per generation, with the harness's 200 generations, so at most 206 calls. That makes at most 16 runs x 206 = **3,296 calls**, at no more than 1,000 input tokens each, and **$0.14** at $0.042 per million. Expected: about 10 asked decisions per run (the v3 `evolve_jev` runs took at most 5 model calls), 16 runs and about 750 tokens each, about $0.005.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Enforce the stated spend ceilings

If any Anthropic call exceeds the historical $0.0176 maximum or a Jev request exceeds the assumed 1,000 input tokens, the four-repeat batch can exceed these figures: the harness limits calls per run and sets a $50 per-run budget, but it has no aggregate spend cap or 1,000-token cap, and reports costs only after completing the batch. These are estimates, not hard ceilings; label them accordingly or enforce a stop before presenting them as the paid approval budget. The repository requires paid model runs to report their expected cost.

AGENTS.md reference: AGENTS.md:L175-L177

Useful? React with 👍 / 👎.

@sykosyber

Copy link
Copy Markdown
Member Author

Scope: owner direction in session 2026-10-28 (route (c) for #147). Packet jev-api-2 lands the fixes from the independent verification of #147 (#147 (comment)) as a follow-up to the merged #147.

What changed (3 files: docs/EVOLUTION.md, syberlabs/jev_api.py, tests/test_jev_api.py)

  1. The measurement plan now targets v3 (gaps_renumber, journal_chain, worktree_order, verify_layers). Verifier finding 2.
    • It cites the owner's decision in docs/MODEL_COMPARISON.md ("Where that run goes").
    • Arms, K = 6 and Sonnet 5.5 medium are unchanged.
    • Both rules are kept. Rule 1 is verbatim from the spec and applied on v3.
    • The uninformative clause is restated for v3: if single solves all four in the pilot, or Jev is never offered two actions, the measurement moves to v4 on the owner's go.
    • For reference only, not a prediction: on the v3 rows of 2026-10-22, evolve_jev spent $0.373 against evolve's $0.469.
  2. Repeats and cost. Finding 4.
    • A pilot runs --repeats 1, then the measurement runs --repeats 3 (seed 1 runs again, since the harness has no seed offset). So four repeats are paid and three decide.
    • Anthropic: expected $1.91, hard ceiling $5.40 (304 calls x $0.0176).
    • Source: the E3 v3 Sonnet 5.5 rows of 2026-10-22. That is 84 rows, with mutate feedback and the base evaluation. The ceiling's per-call price is the highest in those rows.
    • Jev asked decisions are not bounded by K. The code bound is K plus 200 generations, at most 3,296 calls, a ceiling of $0.14 (stated as $0.15); expected about $0.005.
  3. Crossover range corrected. Finding 3.
    • 0.02 to 0.15 on the 19 three- and four-option answers, and 0.37 on the one two-option answer (choice_two_options.json).
    • Added: confidence = (p_max − 1/k)/(1 − 1/k) on all 20 answers.
  4. The timeout is a wall-clock deadline per decision. Finding 5.
    • The transport runs on a daemon thread joined with timeout. Past it, the decision falls back with jev_timeout, detail deadline (a new closed word).
    • The thread is abandoned, not cancelled, and a late answer is discarded. The docs say so.
    • The same value is still urllib's per-socket-operation timeout (detail timeout). Stdlib only.
    • New test: a transport blocked for 10 s, with a timeout of 0.2 s, falls back in under 3 s and takes the local action.
  5. fallback.response_digest is documented in the module docstring and docs/EVOLUTION.md. Finding 6.
  6. JevFailure checks its closed words with a ValueError, not an assert, so the check survives python -O; a new test covers it. Finding 7.

GAPS entry 61 is unchanged. Nothing outside these 3 files differs from main.

Correction to #147's handoff: four open_pull_request attempts failed, and the fifth opened #147. The handoff said three. The thread journal shows effect_unknown at seq 26, 32, 41 and 56, and effect_succeeded at 69.

Commands and results

  • Targeted, local (Windows 11, CPython 3.12.6): test_jev_api, test_jev, test_gaps, test_package and test_evolve gave Ran 75 tests, OK.

  • Mutation check (on a copy of the identical tree): 17 of 18 caught. The two new mutations, "deadline not enforced" and "closed words not checked", are both caught. The one survivor, "non-finite accepted", is equivalent, as before.

  • Full suite, GitHub Actions, scratch syberlabs/7ca8f3c4-ci (a2d7247, tree 2736444a9a, identical to the head's): https://github.com/SyberLabs/SyberWork/actions/runs/40777504544, success.

    • test (3.11), test (3.12), test (3.13) and G8 strict: Ran 1182 tests, OK (skipped=2).
    • windows: OK (skipped=25).
    • compose and G8: success.
  • Bite, GitHub Actions, scratch syberlabs/7ca8f3c4-bite (f0d19ec: only tests/test_jev_api.py, on main): https://github.com/SyberLabs/SyberWork/actions/runs/40777504565, failure, as intended.

    • Every leg: Ran 1182 tests, FAILED (failures=1, errors=1). The two are exactly the two new tests:
      • test_a_slow_answer_falls_back_at_the_wall_clock_deadline errors;
      • test_a_failure_takes_only_closed_words_without_relying_on_assert fails.
    • There is no import error, and no other test fails. compose passes.
  • PR CI: all 25 checks on Close the independent verification findings on #147: re-target the pre-registered jev_api measurement to the v3 fixtures with four paid repeats and a sourced cost ceiling, correct the recorded crossov #149 at 78c116c90f pass (gh pr checks 149).

  • python -m conformance.run: matched 23 golden traces.

  • python benchmarks/run_scenarios.py: 11 replayed, exit 0, digest b41f7cbb… (unchanged).

  • Thread checks on c1 (3 files, 25,075 diff bytes):

    • conformance passed (matched 23 golden traces);
    • critic.rules passed ("no findings over 3 changed path(s)");
    • scenarios passed;
    • tests passed (Ran 1182 tests, OK (skipped=25)).

    findings c1: no findings.

  • certify c1: result 78c116c90f (tree 2736444a9a) on repo-change.v5, class R; four checks passed; chain head 25d04f2bc0 at 14 events.

    • Signed with key 4b6ae2deea. I checked it against the factory's test-integrity-2-74e7d4ac.cert.pub before signing, and the written .pub is byte-identical.
  • verify against that .pub with --repo .: signature ok, repository ok (3 paths, diff digest match), journal ok (14 of 14 events). Verdict pass.

  • Publish. push_branch succeeded.

Claim ceiling

Implemented and tested on CI and locally, against recorded responses and stand-in transports. Not measured. The plan is pre-registered and unrun, and it needs the owner's go.

Known limitations

Next permissible packet

The paid v3 measurement, after the owner's go: $5.40 Anthropic hard ceiling (expected $1.90) plus $0.15 Jev.

Never merged, never force-pushed, nothing pushed to main, no setting changed, no Anthropic or Jev call made.

@sykosyber

Copy link
Copy Markdown
Member Author

Independent verification: #149 (jev-api-2, thread 7ca8f3c4, candidate c1)

Verdict: PASS. Mergeable as is. Every claim checked holds, and #147 findings 2 to 7 are closed. The findings below are one low docs wording issue and informational notes. None of them blocks the merge.

  • Head 78c116c90f (tree 2736444a9a), base main d85f3d2825, 3 files.
  • Fresh clone. No Jev, model or GitHits call was made. Tests ran with TYPESAFE_API_KEY removed from the child environment, and the child confirmed it was absent.

Findings, ranked

  1. Low (docs): docs/EVOLUTION.md:203 calls the $5.40 a "hard ceiling", but it is an empirical planning ceiling.
    • $0.0176 is the highest per-run average dollars per call in the 84 rows (evolve on gaps_renumber, baseeval-r1, $0.03526 for 2 calls). It is not the most that one call can cost.
    • The adapter allows max_tokens=16000 (adapters/anthropic_adapter.py:164) at $10/M output (benchmarks/model_vs_patch.py:83). That is about $0.16 or more per call in the worst case.
    • The paragraph already says that nothing in the harness stops the total and that the operator stops the run, so this is disclosed. Calling it "planning ceiling" or "observed-rate ceiling" would make the doc exact.
    • The arithmetic itself is right (see check 3).
  2. Info (docs): the Jev $0.14 ceiling, docs/EVOLUTION.md:204, assumes at most 1,000 input tokens per request. That is not a code bound.
    • MAX_REQUEST is 65,536 bytes (syberlabs/jev_api.py:69). Observed requests were 705 to 760 tokens, and a population of at most 4 keeps them small.
    • The decision bound "K plus one per generation, at most 206 calls" is a sound upper bound:
      • model_vs_patch passes no seeds (benchmarks/model_vs_patch.py:226-229), so no Evaluate is offered.
      • Fills and slot mutations each use a K call.
      • Crossovers advance the generation (syberlabs/evolve.py:453,514).
  3. Info (docs): docs/EVOLUTION.md:202 gives evolve as $0.0391. The receipts give $0.0390495, which rounds to $0.0390. The total is unaffected: $1.9086 is still stated as $1.91.
  4. Info (code): for a silent server, the detail will almost always be deadline rather than timeout.
    • In syberlabs/jev_api.py:297-301, join(self.timeout) starts before urllib's socket timer inside the worker, so the deadline almost always expires first.
    • Both details have reason jev_timeout, so the plan's "not measured" clause, which counts any reason, is unaffected. The docs say both are possible, which is true.
  5. Info (code): abandoned workers can pile up within one search if timeouts alternate with successes.
    • max_failures counts only failures in a row, so it does not cap this.
    • Each worker ends at urllib's per-operation timeout. Until then its frame holds the Request, whose Authorization header has the key. It is never recorded or raised.
    • The workers are daemon threads, so they do not block interpreter exit (syberlabs/jev_api.py:297). The docstring discloses that a slow call is abandoned, not cancelled.
  6. Info: the confidence = (p_max − 1/k)/(1 − 1/k) claim (docs/EVOLUTION.md:182) checks out on all 9 answered fixtures in the repository. The other 11 of the 20 answers are in transcripts outside the repository, and I did not check them.

Per check

  1. Scope: OK. git diff --stat d85f3d2825 78c116c90f lists exactly docs/EVOLUTION.md, syberlabs/jev_api.py and tests/test_jev_api.py (+91/−15).
  2. Measurement plan: OK.
    • docs/EVOLUTION.md:193 targets v3 (gaps_renumber, journal_chain, worktree_order, verify_layers), with v4 as the fallback. The v4 fixture names exist under evals/tasks/.
    • It cites the owner's decision, docs/MODEL_COMPARISON.md:172 ("Where that run goes").
    • Rule 1 is verbatim. It matches the spec quote in docs/MODEL_COMPARISON.md:170 and docs/EVOLUTION.md:160, and is applied on v3.
    • Rule 2 says "on v3" (:198).
    • The uninformative clause is restated for v3 (:200).
    • The supporting figures match the receipts: single 3 of 12 on v3; evolve arms 12, 11 and 12 of 12; evolve_jev $0.373 against evolve $0.469 (80%); evolve_jev used at most 5 calls.
  3. Cost arithmetic: OK, recomputed from the receipts. The 84 rows are feedback-r3 (60), baseeval-r1 (20) and the Sonnet single probe (4), all claude-sonnet-5-5.
    • Mean dollars per run in the r3 rows: single 0.010103, evolve 0.039050, evolve_jev 0.031087.
    • Per fixture per repeat: single + evolve + evolve_jev + evolve = 0.11929. Per repeat: 0.4772. Four repeats: $1.9086.
    • Highest per call: 0.01763 (baseeval-r1, gaps_renumber, evolve, $0.03526 for 2 calls). 304 × 0.01763 = $5.3595.
    • Calls: 4 × (1 + 3 × 6) = 76 per repeat, so 304 for four repeats.
    • Jev: 16 runs × 206 = 3,296 calls. At 1,000 tokens each and $0.042/M that is $0.138, stated as $0.15.
    • Four repeats are paid and three decide. The harness numbers seeds range(1, repeats + 1) (benchmarks/model_vs_patch.py:337), so there is no seed offset, as the plan states.
  4. Crossover range: OK. tests/fixtures/jev_api/choice_two_options.json has crossover 0.37 against mutate 0.63. The three- and four-option fixtures range from 0.02 to 0.15.
  5. Wall-clock deadline: OK, no race found.
    • box is local to each call. After JevFailure("jev_timeout", "deadline") the deciding thread never reads it again, so a late answer cannot be used.
    • The worker writes only to box, and http_transport has no shared state. self.calls is incremented before the call starts, so no shared state changes after the fallback.
    • If the worker finishes exactly at the boundary, it is either read (thread dead) or discarded (alive), which is the conservative result either way.
    • The thread is a daemon, so it does not block exit.
    • The thread name is the constant jev-api-call. The worker catches everything, so threading.excepthook never prints, and the re-raised JevFailure is dropped by decide. There is no key in any thread record.
  6. JevFailure: OK. syberlabs/jev_api.py:95-96 is a real ValueError. Under python -O (sys.flags.optimize=1), tests.test_jev_api ran 28 tests, OK, and both new tests pass under -O.
  7. Docs: OK. response_digest is in the module docstring (syberlabs/jev_api.py:33-34) and in docs/EVOLUTION.md:172.
  8. Tests: OK.
    • Locally on Windows 11 with CPython 3.12.6: tests.test_jev_api tests.test_jev tests.test_evolve gave Ran 70 tests in 313.729s, OK. python -m conformance.run gave matched 23 golden traces.
    • CI full run 40777504544 was on a2d7247, tree 2736444a9a, the same as the head.
      • test (3.11/3.12/3.13) and G8 strict: Ran 1182 tests, OK (skipped=2).
      • windows: OK (skipped=25).
      • compose and G8: success.
    • CI bite run 40777504565 was on f0d19ec, which is main plus only tests/test_jev_api.py, byte-identical to the head's.
      • Every test leg: FAILED (failures=1, errors=1).
      • The two are exactly test_a_slow_answer_falls_back_at_the_wall_clock_deadline (ERROR, TypeError because on main the 10 s answer is used and there is no fallback) and test_a_failure_takes_only_closed_words_without_relying_on_assert (FAIL).
      • compose passed.
    • gh pr checks 149: 25 of 25 pass. Merge state CLEAN.
  9. Certificate: OK.
  10. Rebuild preview: clean. origin/main is still d85f3d2825, because Rebuild of #144 onto main: close eleven SyberEval core holes (author: Seth Carlson) #148 is open. Cherry-picking 78c116c90f onto it is clean, 3 files, with tree 2736444a9a, identical to the head. Rebuild of #144 onto main: close eleven SyberEval core holes (author: Seth Carlson) #148 touches only syberlabs/eval/* and tests/test_eval_adversarial.py, so neither PR overlaps the other. Nothing was pushed.

Reviewed-by: claude (independent verifier session, SyberFactory workflow)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant