Bundle 69: Curation Review Protocol V2 + Holdout-Safe Calibration

[nulleimy]

Implements the Skill Tester-approved Protocol V2.4 design from #159 on a governed feature branch.

Scope:

• separate Curation Review V2 from transition feasibility and human execution;
• distinct packet/review schema and contamination-safe calibration input;
• development vs holdout evidence roles;
• representative vs diagnostic selection scopes;
• holdout freeze/binding contract;
• machine system outcomes preserved in denominator;
• personal vs multi-DJ evaluation scope;
• A/B assignment balance diagnostics;
• anti-survivorship / anti-leakage tests;
• no optimizer/ranking mutation, audio reads, MIR execution, network/cloud upload, release/deploy/production/PDM authority.

Governing tester blocker: #159.

MERGE_AUTHORIZATION=NO
OPTIMIZER_RANKING_ACTIVATION=NO
RELEASE_AUTHORIZATION=NO
DEPLOY_AUTHORIZATION=NO
PRODUCTION_ACTIVATION=NO
PDM_TRAINING=NO

[nulleimy]

Skill Tester implementation audit — GREEN TESTS, NOT READY YET

Exact HEAD audited: 931c525d57a59edfcc14025bcd32faa5fc48e6aa
CI #929 is green on Python 3.11/3.12 with 506 passed on each version, but independent spec-coverage review found three missing fail-closed enforcement points from the approved Protocol V2.4 design.

GAP 1 — development cases can still be relabeled as holdout

The current HoldoutValidationManifest rejects label leakage and binds case/scenario identity, but it does not bind a registry of prior development case/scenario identities. A caller could construct a new holdout manifest over the already-used R2 development cases and technically receive holdout semantics.

Required fix: holdout validation must bind a non-empty prior-development exclusion registry and reject selected case IDs or scenario fingerprints present in that registry.

GAP 2 — representative selection independence is documentation-only

selection_scope=representative_holdout exists, but the contract does not constrain the actual selection inputs. The approved design requires representative selection to reject challenger score/preference/disagreement/failure-class dependent selection.

Required fix: add an auditable selection-basis contract with enumerated allowed/disallowed inputs; representative holdout constructor must reject model-outcome-dependent selection inputs. Diagnostic challenge sets may permit them but cannot claim representative performance.

GAP 3 — assignment seed commitment is not verifiable

CurationBlindAssignmentV2 carries an assignment fingerprint and balance is checked, but there is no seed-commitment field or deterministic builder/proof that reviewer-specific A/B mappings came from the frozen private assignment seed lineage.

Required fix: bind assignment seed commitment and add deterministic assignment-generation/verification semantics (or equivalent proof) while preserving algorithm identity hiding. Aggregate calibration must reject assignment commitment mismatch.

Tester status

CI_GREEN=YES
SPEC_COVERAGE_COMPLETE=NO
READY_FOR_REVIEW=NO

No test expectation will be weakened. These gaps are being implemented before Ready-for-Review.

[nulleimy]

Bundle 69 final verification receipt — SKILL TESTER PASS FOR REVIEW

Exact feature HEAD: aca2eacc54583e6aa9152e3756abc996c8288052
Canonical base: f3292d6ac3db81e9d741eedfd2467ecc90c933c0
PR: #161

Executed evidence:

• CI #939 SUCCESS
  • Python 3.11.16: compile PASS; critical Ruff PASS; 512 passed, 6 warnings; artifact ID 9489037041; ZIP SHA-256 5f4a45bd4c6a764ae72786a41c5638a403201bf7549a64505ef1dbc286c6d641
  • Python 3.12.14: compile PASS; critical Ruff PASS; 512 passed, 5 warnings; artifact ID 9489037043; ZIP SHA-256 da3d73e8cf3246028a50411da2c91e10727d16da5c835f8f5c2c41d6e0c7e5a2
• PR Guard #338 PASS
• post-Ready PR Guard #339 PASS
• post-Ready PR Guard #340 PASS
• exact compare: ahead 13 / behind 0; 8 files added; 0 pre-existing files modified
• review threads 0; review submissions 0

Independent authority verification:

• set_engine.py unchanged blob dd8f0cdb672202289530579d092cf0feb5cedf25
• set_path_optimizer.py unchanged blob 819f6399181f4b37bb26ed8ccefb6cc0f4a9607f
• legacy Bundle63 human_dj_review_execution.py unchanged blob 195a4f5f6ef62421be07413807072244f48c4b57
• Bundle67 competitive_curation.py unchanged blob faadcae932e5f7b907503cbf2cd5bfdaab36462e
• Bundle68 human_preference_calibration.py unchanged blob 61899fda033c5d77513a780602c227ccabe36b79

Skill Tester implementation findings resolved before Ready:

• transition-execution contamination separated from curation truth
• legacy schema cannot become V2 curation truth
• old R2 cases remain development-only
• representative vs diagnostic selection scope enforced
• representative selection rejects challenger-dependent inputs
• prior-development case/scenario overlap rejected from holdout
• deterministic counterbalanced reviewer assignment recomputed from private seed commitment
• non-reviewable system outcomes remain in denominator and cannot become human abstain
• holdout candidate/config/policy mismatch fails closed
• purity audit covers review, calibration, and holdout-guard services

Operational caveat retained: a real holdout must persist an immutable freeze receipt before the first human label. The pure protocol does not claim wall-clock proof by itself.

SKILL_TESTER_IMPLEMENTATION_VERDICT=PASS_FOR_REVIEW
READY_FOR_REVIEW=YES
MERGE=NO
OPTIMIZER_ACTIVATION=NO
RELEASE=NO
DEPLOY=NO
PRODUCTION=NO
PDM_TRAINING=NO

Merge still requires separate explicit MERGE GO.