E9: calibration on representative benign traffic, frozen before any score - #23
Merged
Merged
Conversation
…core This implements the owner's E9 decision of 2026-09-17. The guards, pins, budget and exemption are E8's, and one variable changes: the benign population the operating point is calibrated on. E8 calibrated on OR-Bench's seemingly-toxic benign prompts, and G2's threshold went to 0.999485. E9 calibrates on the first user turns of real English WildChat-1M conversations, from three pinned chronological shards. OR-Bench stays as a stress stratum, which is reported and decides nothing. contract.json holds E8's value in every field except id and status, and the tests assert it. OASST1 and OASST2 were excluded before any score. G1 trains on a filtered OASST1 through VMware/open-instruct, and OASST2 holds no English first turn outside OASST1's trees. The release's toxic flag is not a filter: dropping toxic traffic would lower the thresholds and push width toward admission. Every partition is drawn now: 800 calibration items, 400 stress items, and for each of E8's three candidates, 200 scouting and 1000 measurement items. They are pairwise disjoint and outside every prior freeze. itw-jailbreak has only 964 measurement items left, so it cannot be admitted; that was settled before any score. Admission requires a scouting width at or above the contract's 0.10 floor, a point estimate on 200 items, and that limitation is declared. If nothing clears, the failure is the result and the local-path stop is recorded. score.py refuses every E9 set until the frozen contract, freeze.json and the set's item list are committed, and freeze.json is on origin/main. No item has been scored.
distribute.py run rebinds E9's preregistration and freeze sources from pending_commit to 6ac65de. The ledger snapshot now counts E9's committed documents, and the conformance record re-hashes the snapshot it reads. No approval, dispatch or publication record changed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This implements the owner's E9 decision of 2026-09-17. No item has been scored. It freezes the design that the scoring will follow, so it records no result.
One variable changes. The benign population the operating point is calibrated on. E8 calibrated on OR-Bench's seemingly-toxic benign prompts, and G2's threshold went to 0.999485. E9 calibrates on representative benign traffic: the first user turns of real English WildChat-1M conversations (
7d6490e4, ODC-BY, ungated), shards 0, 7 and 13. OR-Bench stays as a stress stratum, which is reported and decides nothing.contract.jsonholds E8's value in every field exceptidandstatus, and the test suite asserts it.Excluded before any score:
VMware/open-instruct.The toxic flag is deliberately not a filter. Dropping toxic traffic would lower the thresholds and push width toward admission.
Partitions, all drawn now, pairwise disjoint and outside the 5,920 prior frozen hashes:
itw-jailbreakhas only 964 measurement items left and cannot be admitted, which was settled before any score.Admission. A pool is admitted if its scouting width at the calibration thresholds is at or above the contract's 0.10 floor. That width is a point estimate on 200 items, and the limitation is declared. If nothing clears, the failure is the result: the local-path stop is recorded, and E10 (a new guard) becomes the justified next step.
Order of operations.
score.pyrefuses every E9 set until the frozen contract,freeze.jsonand the set's item list are committed, andfreeze.jsonis onorigin/main. Merging this PR before any score gives the freeze a timestamp outside the repository.Gates: at
17fb5f2,verification_manifest.pyran 83 checks, exit 0 (.venv, Python 3.14).evidence_guard.py --base 9bb2affpasses. No trust-root or workflow file is touched, so E9's tests live in the existingtests/test_e8_runner.pyrather than a new manifest row. There are no co-author trailers.Merge as a merge commit.