Human Review Protocol R2 design: separate curation, transition feasibility, and execution

[nulleimy]

Status

DESIGN ONLY — no implementation authorization in this issue.

Source blocker: #159
Canonical checkpoint: feature/bundle-0-bootstrap @ f3292d6ac3db81e9d741eedfd2467ecc90c933c0

Goal

Replace the experimentally contaminated Human DJ Review R1 calibration path with a versioned R2 protocol that cleanly separates what APPLAYLIST chooses from how a human DJ happens to execute a mix.

Core invariant:

CURATION_REVIEW != TRANSITION_FEASIBILITY_REVIEW != HUMAN_EXECUTION_REVIEW

Historical Bundle 63 / R1 evidence remains immutable. R2 is a new protocol/schema, not an in-place reinterpretation of R1.

Proposed R2 evidence model

A. CurationReviewR2 — primary calibration truth for Competitive Curation

Evaluates the ordered set/segment only. No live transition execution is required.

Required dimensions, each 1..5:

• energy_flow
• dramaturgical_fit
• set_coherence
• alternative_usefulness

Required case-level fields:

• curation_preference = plan_a | plan_b | tie | abstain
• curation_confidence = 0..1
• optional reason codes / note
• observed timestamp
• algorithm identity hidden

The reviewer instruction must explicitly say: judge track choice, ordering, progression and usefulness; do not use how well you personally executed a mix as the basis for curation preference.

Bundle 67 / Competitive Curation calibration may consume only curation_preference, never an overall mixed preference.

B. TransitionFeasibilityReviewR2 — separate APPLAYLIST transition-quality evidence

Evaluates a proposed transition plan, not arbitrary live DJ execution.

A transition is assessable only when the review artifact binds a deterministic TransitionReviewSpec, including at minimum:

• outgoing track/segment identity
• incoming track/segment identity
• proposed outgoing mix window
• proposed incoming mix window
• transition duration or bars
• tempo handling / target BPM when relevant
• transition strategy
• explicit evidence refs

Optional technique fields such as EQ/FX/stems/loop policy may be absent only if the dimension being rated does not depend on them.

Candidate dimensions:

• phrase_window_quality
• energy_handoff
• spectral_compatibility
• tempo_key_feasibility
• transition_strategy_suitability
• vocal_collision_risk only when explicit vocal evidence exists

Each dimension uses:

• numeric rating when assessable, or
• explicit not_assessable with reason code.

No forced score is allowed for missing vocal/phrase/spectral evidence.

Transition preference, if collected, must be separate:

• transition_preference = plan_a | plan_b | tie | abstain | not_assessable

C. HumanExecutionReviewR2 — optional separate experiment

Measures the DJ's actual performance, not optimizer/curation truth.

Execution ratings are valid only if one of these is true:

1. the exact execution recipe is standardized and bound to the review, or
2. the research question explicitly studies free-form human execution.

Possible execution dimensions:

• execution_smoothness
• execution_phrase_accuracy
• execution_energy_control

These values MUST NOT calibrate Bundle 67 Competitive Curation.

Preference isolation rules

1. Remove the concept of a single mixed preference from R2.
2. curation_preference is the only human target for Bundle 67 curation calibration.
3. transition_preference is reserved for a future transition challenger/transition-planning model.
4. execution outcome cannot alter curation calibration labels.
5. ties and abstains remain first-class values.
6. not_assessable remains distinct from tie/abstain.

R1 compatibility / migration

• R1 rows and ledgers remain immutable historical evidence.
• No R1 transition_smoothness or phrase_alignment score may be reclassified as R2 transition feasibility without a bound TransitionReviewSpec.
• Existing qualitative track-selection notes may be retained as qualitative curation evidence only.
• Existing unfilled 12-case CSV must not be auto-filled or coerced into R2.
• New R2 reviewer artifacts receive new schema/protocol IDs and separate append-only ledgers/reports.

Calibration R3 design implications

Bundle 68 R2 calibration must be superseded for future real evidence by a curation-only calibration path that:

• accepts complete CurationReviewR2 evidence;
• does not require transition/execution dimensions;
• binds blind assignment -> source plan/path exactly;
• computes agreement, decisive agreement, tie behavior, abstain handling, confidence weighting, set-role coverage;
• cannot read/use transition or execution preference fields even if present elsewhere;
• cannot authorize optimizer activation.

Proposed tester acceptance properties BEFORE implementation

The Skill Tester must attempt to falsify this design against at least these attacks:

1. Same track order, excellent DJ execution vs poor DJ execution -> curation label MUST remain identical.
2. Missing transition spec -> transition feasibility MUST become not_assessable, never fabricated numeric evidence.
3. Missing explicit vocal evidence -> vocal collision MUST be not_assessable.
4. Reviewer prefers A only because they personally mixed A better -> that must not be representable as curation truth without an explicit curation judgment.
5. Transition A better, curation B better -> protocol must preserve both without collapsing into one winner.
6. All four curation dimensions complete but transition dimensions absent -> curation calibration must still be valid.
7. Complete transition evidence but curation abstain -> no curation accuracy contribution.
8. Legacy R1 submission -> cannot be silently accepted as R2.
9. Duplicate/replayed review identity -> fail closed.
10. Algorithm identity exposed -> blinded calibration invalid.
11. A/B slot inversion -> deterministic source identity resolution must remain correct.
12. Qualitative note only -> retained as note, never converted to score.
13. HumanExecutionReview changes -> must not change curation report hash/verdict.
14. Transition review changes -> must not change curation report hash/verdict.
15. Curation review changes -> should change curation calibration deterministically.

Design gate

No implementation branch should be opened until a separate Skill Tester adversarial review of this proposal returns no unresolved HIGH/CRITICAL experimental-validity findings.

Authority

IMPLEMENTATION_AUTHORIZATION=NO
MERGE_AUTHORIZATION=NO
RELEASE_AUTHORIZATION=NO
DEPLOY_AUTHORIZATION=NO
PRODUCTION_ACTIVATION=NO
OPTIMIZER_RANKING_ACTIVATION=NO
PDM_TRAINING=NO

[nulleimy]

Skill Tester adversarial review — PASS 1

Verdict: DESIGN FAIL — implementation must NOT start.

The separation of curation / transition feasibility / execution is directionally correct, but the proposal still has unresolved experimental-validity defects.

CRITICAL C1 — circular validation / holdout contamination

Bundle 67 Competitive Curation was explicitly designed and hardened using qualitative findings from the existing 12 R2 cases (UKG saturation, sudden rave-techno drift, under-energy peak behavior, near-equivalent alternatives, etc.). Those same 12 cases therefore cannot serve as an independent validation set for whether Bundle 67 agrees with human judgment.

Required correction:

• classify the existing 12 cases as development/regression evidence only;
• preserve them for deterministic regression testing and protocol rehearsal;
• create a fresh blinded holdout case set that was not used to design/tune Bundle 67 or calibration thresholds;
• freeze policy/thresholds before revealing holdout human labels;
• never report holdout performance from a dataset after using its labels to tune the challenger.

HIGH H1 — reviewer-memory leakage

The original reviewer has already seen/heard and qualitatively judged many of the 12 cases. Algorithm identity blinding does not remove prior case familiarity.

Required correction:

• fresh holdout cases for formal validation;
• new blind assignment seed/IDs;
• explicitly record whether reviewer previously encountered a case;
• previously exposed cases cannot count as blind holdout validation.

HIGH H2 — cognitive contamination between curation and transition review

A separate curation_preference field is necessary but not sufficient. If the reviewer first performs or hears transitions, execution quality can still influence the later curation judgment.

Required correction:

• curation review is a separate artifact/session and MUST be locked first;
• transition review is unlocked only after curation submission is immutable;
• no transition recipe, rendered transition, strategy explanation, source path identity, or execution result is visible during curation judgment;
• timestamps/session order are evidenced.

HIGH H3 — TransitionReviewSpec is underspecified

mix window is ambiguous without a coordinate system and analysis binding.

Required correction for each transition spec:

• exact analysis revision / evidence fingerprint for both tracks;
• window coordinates with explicit unit and reference (e.g. seconds from canonical media start, plus optional beat/bar coordinates bound to beat-grid revision);
• incoming/outgoing segment IDs;
• exact target BPM or explicit tempo_unchanged;
• duration in a canonical unit;
• strategy version;
• deterministic spec fingerprint.

HIGH H4 — plan metadata alone cannot support all proposed human auditory ratings

A deterministic plan can describe a transition, but a human cannot reliably judge spectral_compatibility, actual energy handoff, or execution smoothness from metadata alone.

Required correction:

• distinguish evidence-derived feasibility from human audition review;
• human auditory transition ratings require a deterministic local rendered preview OR a standardized execution recipe;
• absent that, auditory dimensions are not_assessable;
• no cloud/audio upload is implied.

HIGH H5 — tempo_key_feasibility combines two independent constructs

A good tempo match can hide a bad harmonic relation or vice versa.

Required correction: split into at least:

• tempo_feasibility
• harmonic_compatibility
  with independent missing-evidence semantics.

HIGH H6 — n=12 point-estimate agreement is too unstable for strong conclusions

Even after fixing leakage, raw agreement rates on a tiny dataset can swing materially with one case.

Required correction:

• predeclare minimum holdout sample size by intended claim;
• report uncertainty interval (e.g. Wilson interval) for agreement rates;
• no SUPPORTS_FURTHER_EVALUATION based only on a point estimate when the lower confidence bound is below the policy floor;
• thresholds are frozen before holdout labels are opened.

HIGH H7 — target population is ambiguous

A single reviewer can validate personalization to that DJ, but not market-wide DJ quality.

Required correction: split claim scopes:

• PERSONAL_DJ_CALIBRATION: one identified reviewer can be valid for that person's preference model;
• GENERAL_DJ_PRODUCT_VALIDATION: requires multiple independent DJs and inter-rater/disagreement reporting.

No result from one reviewer may be described as general DJ consensus.

MEDIUM M1 — ordering / carryover effects

Required correction:

• Plan A/B slot assignment randomized per case;
• playback/list presentation order counterbalanced where possible;
• preserve assignment fingerprint;
• report repeated-track overlap/familiarity across holdout cases.

MEDIUM M2 — near-equivalent alternatives

Near-equivalent A/B cases are valid evidence about weak alternative generation, but they are weak evidence for choosing one ranking policy over another.

Required correction:

• flag meaningful_difference_status before human labels are opened;
• keep ties as product evidence;
• do not silently treat near-equivalent cases as decisive calibration opportunities.

MEDIUM M3 — preference/dimension inconsistency

A reviewer may prefer A while dimension means favor B. Do not auto-correct either signal.

Required correction:

• preserve explicit preference as primary pairwise label;
• surface dimension/preference disagreement as diagnostic evidence;
• never rewrite human preference from dimension arithmetic.

Tester gate

PROTOCOL_R2_DESIGN_VALIDITY=FAIL

Implementation remains unauthorized until C1 + H1–H7 are addressed and a second independent tester pass finds no unresolved CRITICAL/HIGH defects.

[nulleimy]

Protocol R2.1 design revision — response to Skill Tester PASS 1

This revision supersedes the original proposal for implementation planning. No code is authorized yet.

1. Dataset classes are now explicit

DEVELOPMENT / REGRESSION SET

The existing 12 historical R2 cases are permanently classified as:

DATASET_ROLE=DEVELOPMENT_REGRESSION

They may be used for:

• regression tests;
• reproducing known UKG/rave/energy/near-equivalent failure modes;
• reviewer-UX rehearsal;
• protocol debugging.

They MUST NOT contribute to independent validation agreement metrics for Bundle 67 because their qualitative labels influenced its design.

FRESH PERSONAL HOLDOUT

For the first genuine one-DJ validation:

• minimum 24 fresh cases;
• minimum 4 fresh cases per set role (opening/build/mid_set/peak/reset/closing);
• no case previously reviewed qualitatively or numerically by that reviewer;
• fresh blind assignment IDs/seed;
• source optimizer paths and Bundle 67 shadow results frozen before human labels exist;
• policy + thresholds + case manifest hashed/frozen before label reveal.

This dataset may support only PERSONAL_DJ_CALIBRATION conclusions.

GENERAL PRODUCT HOLDOUT

Any claim about general DJ preference requires a separate dataset with at least:

• 60 fresh blinded cases total;
• all six set roles covered;
• at least 5 independent DJs;
• reviewer-level agreement/disagreement reported;
• no result described as DJ consensus when it is only one person's preference.

These are minimum protocol floors, not automatic activation thresholds.

2. Pre-registration / freeze order

For every formal holdout run, immutable order is:

1. freeze canonical code SHA;
2. freeze library/snapshot identity;
3. freeze fresh case manifest;
4. freeze blinded assignment manifest;
5. compute/freeze source optimizer path identities;
6. compute/freeze Bundle 67 shadow comparison outputs WITHOUT human labels;
7. freeze calibration policy/thresholds;
8. hash the entire pre-registration manifest;
9. only then open CurationReviewR2 to the reviewer.

Any tuning after step 9 invalidates that holdout for independent validation and reclassifies it as development evidence.

3. Curation session is physically isolated

CurationReviewR2 is a separate reviewer artifact/session.

During curation judgment it exposes only:

• blinded Plan A / Plan B track order/display identity required for musical judgment;
• set role;
• four curation dimensions;
• curation preference/confidence;
• neutral reviewer instructions.

It MUST NOT expose:

• transition recipes;
• rendered transitions;
• transition strategy names;
• optimizer/path/result identity;
• shadow scores;
• prior transition/execution ratings;
• algorithm identity.

The curation submission is append-only and locked before any transition artifact for the same case becomes available.

Required fields include:

• prior_case_exposure = yes | no | unsure
• curation_session_id
• observed_at

prior_case_exposure != no disqualifies the case from fresh blind-holdout metrics but preserves it as observational evidence.

4. Curation truth is isolated

Required curation dimensions remain:

• energy_flow
• dramaturgical_fit
• set_coherence
• alternative_usefulness

Pairwise truth:

• curation_preference = plan_a | plan_b | tie | abstain
• curation_confidence = 0..1

Human dimension arithmetic NEVER rewrites explicit preference. A disagreement between dimension means and explicit preference is reported as diagnostic evidence.

5. TransitionReviewSpecR2 is fully bound

Every assessable transition spec requires:

• spec_id + deterministic fingerprint;
• outgoing/incoming track IDs;
• outgoing/incoming segment IDs;
• outgoing/incoming analysis revisions;
• evidence fingerprints;
• canonical window coordinates in seconds from canonical media start;
• optional beat/bar coordinates only when bound to a beat-grid revision;
• duration in seconds and, when available, bars;
• exact target BPM OR explicit tempo_unchanged;
• strategy ID + strategy version;
• explicit technique policy fields when required by the rated dimension.

Changing any bound field creates a new spec identity.

6. Evidence-derived feasibility and human auditory review are separate

TransitionFeasibilityEvidenceR2

Machine/evidence assessment; may evaluate whatever source evidence actually supports.

Dimensions are separate:

• phrase_window_feasibility
• energy_handoff_feasibility
• spectral_compatibility_evidence
• tempo_feasibility
• harmonic_compatibility
• transition_strategy_suitability
• vocal_collision_risk only with explicit vocal evidence

Every dimension is either:

• measured/derived value with evidence refs, or
• not_assessable(reason_code).

HumanTransitionAuditionReviewR2

Human auditory judgment is valid only against:

• a deterministic local rendered preview with render fingerprint; OR
• a standardized execution recipe whose complete parameters are bound to the review.

No preview/standardized recipe => auditory transition dimension is not_assessable.

Local-only by default; no cloud audio upload is authorized.

7. HumanExecutionReviewR2 remains separate

Free-form DJ execution can be recorded as user-research evidence, but:

HumanExecutionReviewR2 -> NEVER -> CurationCalibrationR3

A standardized execution study may compare execution methods, but it is not curation truth.

8. Meaningful-difference status is frozen before labels

Each holdout case receives a pre-label status derived without human preference:

• meaningfully_distinct
• near_equivalent
• not_proven_missing_evidence

Near-equivalent cases remain valuable product evidence and ties remain valid, but they are separately reported rather than silently counted as strong decisive ranking tests.

9. Statistical reporting

For personal holdout:

• n >= 24 fresh eligible cases;
• all six roles covered;
• point estimates AND 95% Wilson intervals for exact/decisive agreement;
• no conclusion above SUPPORTS_FURTHER_EVALUATION from personal holdout alone;
• lower confidence bound, not point estimate alone, must satisfy any pre-registered agreement floor.

For general-product holdout:

• n >= 60 across >=5 DJs;
• aggregate + per-reviewer metrics;
• reviewer disagreement reported;
• confidence intervals reported;
• no automatic optimizer activation.

Exact agreement floors and false-winner limits are versioned policy values and MUST be frozen in the pre-registration manifest before labels are opened. Changing a floor after labels invalidates the holdout.

10. Calibration isolation invariants

CurationCalibrationR3 must satisfy all of these by construction/tests:

• accepts only CurationReviewR2 human targets;
• cannot import/read HumanExecutionReviewR2 fields;
• cannot import/read HumanTransitionAuditionReviewR2 preference fields;
• transition/execution artifact changes cannot change curation calibration report bytes/hash/verdict;
• legacy R1 review cannot satisfy R2 schema;
• A/B assignment inversion resolves source identity deterministically;
• duplicate review IDs fail closed;
• exposed algorithm identity invalidates blinded evidence;
• qualitative notes never become numeric scores;
• holdout case eligibility is checked against prior exposure and dataset role;
• activation authority remains false.

Revised design state

PROTOCOL_R2_1_IMPLEMENTATION_AUTHORIZATION=NO

Next gate: Skill Tester PASS 2 must attack this revised design. Any unresolved HIGH/CRITICAL finding keeps implementation blocked.

[nulleimy]

Skill Tester adversarial review — PASS 2 over R2.1

Verdict: DESIGN FAIL — improved, but 3 HIGH findings remain.

R2.1 resolves the original CRITICAL circular-validation defect and the initial transition/curation mixing defects. The following are still blocking formal implementation.

HIGH H8 — holdout case-selection bias is not fully frozen

R2.1 freezes the case manifest before Bundle 67 shadow outputs, which is necessary, but it does not yet freeze the case generation/selection algorithm and acceptance criteria.

A human/operator could unknowingly select cases that look interesting after inspecting source paths, style composition, or other results correlated with challenger difficulty.

Required correction:

• pre-register a versioned HoldoutCaseSamplingPolicy before generating cases;
• freeze random seed / deterministic sampling material;
• freeze eligibility and engineering-acceptance criteria;
• case inclusion/exclusion reasons must be generated without Bundle 67 shadow scores or human labels;
• rejected cases remain in a selection ledger with reason codes so cherry-picking is auditable;
• after any human label is opened, no replacement case may enter the same holdout except under a pre-registered deterministic replacement rule triggered by a technical invalidity unrelated to preference.

HIGH H9 — curation session can still be contaminated by the reviewer's own mixing

The artifact hides transition information, but a reviewer could still load Plan A/B into Traktor and manually mix while deciding curation preference. The resulting preference could again include execution quality.

Required correction:

• define explicit curation_judgment_mode = sequence_only for formal calibration;
• reviewer attests transition_execution_used=false and transition_preview_heard=false before submission;
• if either is true/unknown, preserve the review but mark it ineligible for clean curation holdout metrics;
• transition audition may begin only after immutable curation submission;
• reviewer instructions must explain that energy/dramaturgy/coherence are judged from selection/order/progression, not from how successfully the DJ blended tracks.

This is an evidence limitation, not a claim that cognition can be perfectly controlled; the protocol must at least make contamination explicit and filterable.

HIGH H10 — general-DJ statistics ignore clustered/repeated-review structure

For GENERAL_DJ_PRODUCT_VALIDATION, multiple DJs may review overlapping cases. Treating all review rows as independent and applying a simple pooled Wilson interval would overstate certainty.

Required correction:

• personal calibration may use case-level binomial/Wilson reporting for one reviewer;
• general validation must report per-reviewer metrics and a macro reviewer average;
• uncertainty must account for reviewer clustering / repeated cases (e.g. reviewer-cluster bootstrap or another pre-registered cluster-aware method);
• no pooled-row confidence claim that assumes all rows are independent;
• report reviewer disagreement / tie distributions explicitly.

MEDIUM M4 — algorithm-recognition suspicion should be recorded

Fresh cases reduce memory leakage but do not guarantee the reviewer cannot infer strategy from repeated behavioral patterns.

Recommended:

• after curation submission, optionally collect algorithm_identity_guess = a | b | unknown and confidence;
• never reveal correctness during the run;
• report guess accuracy as a blinding diagnostic, not as a reason to rewrite human labels.

PASS 2 summary

Resolved from PASS 1:

• C1 circular validation: RESOLVED by development-vs-holdout split
• H1 prior case exposure: RESOLVED
• H2 artifact/session isolation: PARTIAL -> H9 remains
• H3 TransitionReviewSpec coordinates/fingerprint: RESOLVED
• H4 metadata vs auditory review: RESOLVED
• H5 tempo/key conflation: RESOLVED
• H6 uncertainty requirement: RESOLVED for personal; H10 remains for general clustered data
• H7 personal vs general claims: RESOLVED

Current gate:

PROTOCOL_R2_1_DESIGN_VALIDITY=FAIL

Implementation remains blocked until H8-H10 are resolved.

[nulleimy]

Protocol R2.2 design revision — response to Skill Tester PASS 2

No implementation authorization yet.

A. HoldoutCaseSamplingPolicy is now a first-class frozen artifact

Every formal holdout run requires a versioned policy frozen before case generation.

Required fields:

• sampling_policy_id
• sampling_policy_version
• canonical code SHA
• snapshot/library fingerprint
• eligible track/candidate scope fingerprint
• required set-role quotas
• deterministic sampling seed/material
• source case-generation algorithm/version
• engineering-acceptance criteria
• technical-invalidity criteria
• deterministic replacement order/reservoir
• explicit forbidden selection inputs

Forbidden selection inputs include:

• Bundle 67 shadow score/preference
• human preference/ratings/notes
• prior holdout outcome
• any manual 'looks musically better/worse' selection after path generation

A HoldoutSelectionLedger records every candidate case considered with:

• candidate/case identity
• selected/rejected
• machine-checkable reason code
• sampling ordinal
• no human preference fields

No rejected candidate disappears from provenance.

Replacement rule

Before labels are opened, the policy freezes an ordered fallback reservoir.

After label opening, a case can be replaced only when:

• technical invalidity matches a pre-registered reason code; and
• replacement is the next eligible case in the already-frozen fallback order.

The invalid case and its reason remain in the ledger. Preference or challenger difficulty can never trigger replacement.

meaningful_difference_status is NOT an inclusion/exclusion criterion for the formal random holdout. It is computed after selection, before human labels, and used only for stratified reporting.

B. Formal curation judgment mode is sequence-only

Every formal CurationReviewR2 submission includes:

• curation_judgment_mode = sequence_only
• transition_execution_used = false
• transition_preview_heard = false
• prior_case_exposure = no

The reviewer UI requires explicit attestation immediately before immutable submission.

If any value is false/true/unknown in a way that violates clean sequence-only review:

• preserve the review as observational evidence;
• mark clean_holdout_eligible=false;
• exclude it from formal clean curation agreement denominators;
• do not delete/rewrite it.

Transition audition artifacts for a case are unavailable until the curation submission has been append-only committed.

Reviewer instruction is explicit:

Judge track choice, ordering, energy progression, dramaturgy, coherence and usefulness. Do not perform or audition transitions while deciding curation preference.

C. General-product analysis becomes cluster-aware

Personal DJ calibration

For one reviewer:

• case-level agreement metrics;
• point estimate + 95% Wilson interval;
• role-stratified reporting;
• no general-population claim.

General DJ product validation

For multiple reviewers/cases:

• per-reviewer exact/decisive/tie/abstain metrics;
• macro-average across reviewers;
• per-case disagreement distribution;
• cluster-aware uncertainty using a pre-registered reviewer-cluster bootstrap (deterministic bootstrap seed and replicate count frozen before labels) or another explicitly versioned cluster-aware method;
• simple pooled-row Wilson intervals are forbidden as the primary general-DJ uncertainty estimate when rows are clustered/repeated.

The analysis policy records its method/version and bootstrap seed before label opening.

D. Blinding diagnostics

After immutable curation judgment, and before revealing any algorithm identity, the reviewer may optionally record:

• algorithm_identity_guess_for_a = greedy | beam | unknown
• algorithm_identity_guess_confidence_a
• equivalent for B

Guess correctness is evaluated only after the review run is closed/unblinded for analysis.

Blinding diagnostics:

• never rewrite preference;
• never reveal correctness during review;
• are reported separately from curation agreement.

E. Updated formal holdout freeze sequence

1. canonical SHA frozen
2. snapshot/library fingerprint frozen
3. HoldoutCaseSamplingPolicy frozen
4. candidate reservoir + HoldoutSelectionLedger generated
5. selected case manifest + fallback order frozen
6. blind assignment manifest frozen
7. source optimizer path/result identities frozen
8. Bundle 67 shadow comparisons frozen
9. meaningful-difference stratification frozen (reporting only)
10. calibration policy + statistical-analysis policy frozen
11. complete pre-registration manifest hash written
12. CurationReviewR2 opened
13. curation submission append-only committed
14. only then transition/audition phase may open

Any tuning or case-selection change after step 12 reclassifies affected data out of independent clean-holdout evidence.

R2.2 invariants added for implementation tests

• changing shadow scores cannot change selected holdout cases;
• changing human labels cannot change holdout selection/replacement order;
• replacing an invalid case skips only according to frozen fallback order;
• transition preview availability before curation commit must fail closed;
• a curation review with transition_execution_used=true is preserved but excluded from clean metrics;
• transition/execution evidence mutation cannot change clean curation report hash;
• general-DJ primary uncertainty cannot use independent-row Wilson on clustered repeated-review data;
• blinding guess cannot influence preference or eligibility.

State

PROTOCOL_R2_2_IMPLEMENTATION_AUTHORIZATION=NO

Next gate: independent Skill Tester PASS 3. Implementation opens only if no unresolved HIGH/CRITICAL experimental-validity findings remain.

[nulleimy]

Skill Tester adversarial review — PASS 3 over R2.2

Verdict: DESIGN PASS FOR IMPLEMENTATION — no unresolved CRITICAL/HIGH experimental-validity findings.

Re-attack results

1. Development cases reused as validation -> BLOCKED. Existing 12 cases are development/regression only.
2. Manual cherry-picking after seeing challenger difficulty -> BLOCKED by frozen HoldoutCaseSamplingPolicy + complete selection ledger + deterministic fallback reservoir.
3. Policy tuning after seeing labels -> BLOCKED; any post-label tuning reclassifies affected data out of independent holdout evidence.
4. Reviewer remembers prior case -> surfaced by prior_case_exposure; non-clean cases preserved but excluded.
5. Reviewer manually mixes during curation -> surfaced by sequence-only attestation; contaminated reviews preserved but excluded from clean holdout metrics.
6. Transition preview leaks into curation judgment -> BLOCKED by phase ordering; transition artifact unavailable until append-only curation commit.
7. Transition A better while curation B better -> preserved as separate evidence streams; no single mixed winner.
8. Missing transition spec -> transition auditory judgment not_assessable.
9. Missing vocal evidence -> vocal collision not_assessable.
10. Tempo good / harmony bad -> preserved as separate dimensions.
11. Legacy R1 masquerades as R2 -> explicitly forbidden; requires new schema/protocol identity.
12. Execution results mutate curation calibration -> prohibited by artifact/type isolation and required hash-invariance tests.
13. Shadow results influence case selection -> prohibited as sampling input and testable through selection invariance.
14. Human labels influence replacement order -> prohibited by frozen fallback reservoir.
15. Clustered multi-DJ reviews treated as independent rows -> prohibited for primary general-DJ uncertainty analysis.
16. A/B slot inversion breaks source mapping -> deterministic identity binding remains a mandatory negative test.
17. Near-equivalent cases become fake decisive wins -> stratified/reporting status preserved; ties remain evidence.
18. Dimension arithmetic overwrites explicit preference -> prohibited; inconsistency is diagnostic only.
19. Algorithm-identity guess influences label -> prohibited; diagnostic only after immutable review.
20. Single-DJ result marketed as DJ consensus -> blocked by explicit personal vs general claim scope.

Remaining MEDIUM/non-blocking implementation requirements

M5. Record playback/audition environment metadata when available (software/device/gain-normalization mode) for diagnostics; do not make it a hidden calibration dependency.

M6. Treat n=24 personal and n=60/5-DJ general floors as minimum screening floors, not evidence that statistical power is universally sufficient. Actual claim thresholds remain pre-registered/versioned.

M7. Report track/case overlap and repeated-track exposure across holdout cases as a diagnostic because familiarity can affect preference even when exact cases are fresh.

These do not invalidate the separation architecture and can be represented as optional/diagnostic evidence.

Final tester gate

PROTOCOL_R2_2_DESIGN_VALIDITY=PASS_FOR_IMPLEMENTATION

This means the design may be implemented and tested. It does not authorize merge, release, optimizer activation, production activation, PDM training, or any claim that real human validation has passed.

Next required engineering gate: implement on an isolated feature branch and run adversarial tests that encode the invariants above.