Skill Tester blocker: separate curation review from transition execution

[nulleimy]

Source

Independent APPLAYLIST Skill Tester / adversarial QA pass over canonical Human DJ Review R1 + Human Preference Calibration R2.

Tester verdict

EXPERIMENT_VALIDITY=FAIL

The current real R4 calibration protocol mixes curation quality with uncontrolled DJ transition execution.

Finding 1 — transition metrics are not reproducible

The Bundle 63 reviewer workspace presents only Plan A / Plan B track ordering while requiring numeric scores for:

• transition_smoothness
• phrase_alignment

It does not bind a canonical transition execution recipe such as:

• outgoing/incoming mix windows
• bar length
• tempo handling
• EQ policy
• FX policy
• stems policy
• cue/loop behavior
• rendered reference transition

When the DJ performs each transition differently, those ratings measure a mixture of APPLAYLIST transition potential and reviewer execution skill/style. They are therefore not reproducible ground truth for APPLAYLIST transition quality.

Finding 2 — curation calibration label is contaminated

Bundle 68 requires the complete six-dimension HumanDJReview and resolves the single overall human preference directly against the Bundle 67 Competitive Curation shadow challenger.

Because the overall preference may be influenced by uncontrolled transition execution, the calibration can mark a curation challenger correct/incorrect for reasons outside the curation model being evaluated.

Required separation

CURATION_REVIEW != TRANSITION_FEASIBILITY_REVIEW != HUMAN_EXECUTION_REVIEW

A. Curation Review

Suitable for current A/B track-order cases:

• energy_flow
• dramaturgical_fit
• set_coherence
• alternative_usefulness
• track-selection / sequence preference
• confidence

B. Transition Feasibility Review

Must evaluate APPLAYLIST's proposed transition, not arbitrary DJ execution. Candidate dimensions:

• phrase-window quality
• energy handoff
• spectral compatibility
• vocal collision risk when explicit vocal evidence exists
• tempo/key feasibility where relevant
• transition-strategy suitability

This requires either a deterministic proposed mix recipe or a rendered/standardized preview.

C. Human Execution Review

Optional separate experiment. Only compare execution quality when mix instructions are standardized or the experiment explicitly studies the DJ's execution choices.

Protocol changes required before R4 calibration can be considered valid

1. Split curation preference from transition preference.
2. Do not require transition_smoothness / phrase_alignment for curation-only calibration unless transition execution is standardized.
3. Add not_assessable / missing-evidence semantics per transition dimension instead of forcing a fabricated 1..5 score.
4. Prevent Bundle 68 curation calibration from consuming a preference contaminated by transition-execution dimensions.
5. Preserve tie/abstain.
6. Preserve qualitative notes as evidence rather than coercing them into numeric ratings.
7. Add tests proving curation calibration cannot depend on human execution-only ratings.
8. Keep optimizer activation / production / PDM training authority false.

Current R4 implication

The existing qualitative track-selection review remains useful curation evidence.

The current 12-case full 6D review should be PAUSED as calibration evidence until this protocol boundary is corrected. No historical qualitative feedback should be fabricated into missing numeric fields.

Authority

IMPLEMENTATION_AUTHORIZATION=NO
MERGE_AUTHORIZATION=NO
OPTIMIZER_RANKING_ACTIVATION=NO
RELEASE_AUTHORIZATION=NO
DEPLOY_AUTHORIZATION=NO
PRODUCTION_ACTIVATION=NO
PDM_TRAINING=NO

[nulleimy]

Protocol V2 proposal — PRE-IMPLEMENTATION ONLY

This comment is a design candidate for Skill Tester attack. It is not implementation authority.

1. Three independent evidence streams

CURATION_REVIEW_V2 != TRANSITION_FEASIBILITY_REVIEW_V1 != HUMAN_EXECUTION_REVIEW_V1

No aggregate or calibration layer may silently substitute one stream for another.

---

A. Curation Review V2 — valid for the existing 12 blinded A/B cases

Purpose

Measure whether the DJ prefers one track sequence / whole-set arc over the other.

Reviewer instruction

Judge the selection and ordering as a set plan. Ignore how well or badly you happened to execute any live transition. The curation preference asks: which sequence would you choose to play/build from?

Required dimensions, each Plan A / Plan B score 1..5

1. energy_flow
2. dramaturgical_fit
3. set_coherence
4. alternative_usefulness
5. track_selection_fit

transition_smoothness and phrase_alignment are explicitly not part of Curation Review V2.

Required case-level fields

• curation_preference = plan_a | plan_b | tie | abstain
• curation_confidence = 0..1
• optional reason_codes
• optional free-text notes
• real observed_at
• algorithm_identity_was_hidden=true

Semantics

• plan_a/plan_b: sequence preferred as a curation plan
• tie: no material curation preference
• abstain: reviewer cannot make a valid curation judgment
• no preference may be inferred from dimension arithmetic; the human preference is explicit

Binding

Reuse existing immutable:

• case_id
• assignment_id
• blinded Plan A / Plan B ordering
• packet fingerprint / source snapshot binding

No MIR rerun or audio regeneration required.

---

B. Transition Feasibility Review V1 — separate evidence stream

Purpose

Evaluate the proposed APPLAYLIST transition, not arbitrary DJ execution.

Required prerequisite

A transition is numerically reviewable only when APPLAYLIST supplies a deterministic transition proposal containing enough execution-independent evidence, e.g.:

• outgoing mix window
• incoming mix window
• phrase/bar alignment
• target tempo / allowed tempo delta
• transition strategy
• optional bass/EQ handoff guidance
• optional stem/loop instructions when explicitly part of the proposal
• evidence refs / source analysis revision

A standardized rendered preview may be used as an alternative binding if its recipe and source identity are immutable.

State before that prerequisite exists

transition_review_status = not_assessable_no_standardized_transition

No forced numeric score.

Candidate feasibility dimensions once assessable

• phrase_window_quality
• energy_handoff
• spectral_compatibility
• tempo_key_feasibility
• transition_strategy_suitability
• vocal_collision_risk only when explicit vocal evidence exists; otherwise not_assessable_missing_vocal_evidence

The protocol evaluates feasibility of the proposed transition, not the DJ's motor/performance skill.

---

C. Human Execution Review V1 — optional separate experiment

Only used when the research question explicitly concerns execution.

If comparing execution quality across plans, the execution recipe must be standardized enough to make repetitions comparable. Otherwise record qualitative notes only; do not use as model calibration truth.

---

2. Human Preference Calibration V3 boundary

Bundle 67 Competitive Curation challenger may calibrate only against CurationReviewV2.curation_preference.

It MUST reject as curation ground truth:

• Bundle 63 legacy overall HumanDJReview.preference unless a separately collected V2 curation preference exists;
• transition preference;
• execution preference;
• a preference derived from numerical dimension sums;
• qualitative notes coerced into a preference.

Calibration continues to preserve:

• blind Plan A/B -> source path identity resolution after review
• tie
• abstain
• confidence weighting
• exact/decisive agreement
• false-winner-on-human-tie
• challenger-tie-on-human-decisive
• set-role coverage
• INCOMPLETE | DOES_NOT_SUPPORT_ACTIVATION | SUPPORTS_FURTHER_EVALUATION only

SUPPORTS_FURTHER_EVALUATION != OPTIMIZER_ACTIVATION

---

3. Backward compatibility / existing evidence

• Existing qualitative track-selection review remains a separate valid evidence stream.
• Existing empty 12-case CSV is not treated as completed evidence.
• Existing Bundle 63 6D protocol remains historical evidence schema; it is not silently reinterpreted as V2.
• No existing human ratings are fabricated, migrated, or rewritten.
• Existing R2 blinded packet/case assignments may be reused to generate a new Curation Review V2 form because the track order and blinding remain valid.

---

4. Required negative tests before implementation may be accepted

1. curation calibration rejects legacy overall preference without explicit V2 curation preference
2. changing transition-only ratings cannot change curation calibration outcome
3. changing human-execution notes cannot change curation calibration outcome
4. missing one required Curation V2 dimension fails closed
5. tie/abstain stay first-class and cannot be coerced
6. transition numeric ratings rejected when standardized proposal binding is absent
7. vocal collision numeric rating rejected when vocal evidence is absent
8. algorithm identity exposure invalidates blinded curation review
9. assignment/case/path identity mismatch fails closed
10. no audio/MIR/network/cloud side effects
11. no optimizer/ranking mutation
12. activation/PDM/release/deploy authority remains structurally false

---

5. Current R4 state under this proposal

CURATION_REVIEW_V2=NOT_YET_EXECUTED
TRANSITION_FEASIBILITY_REVIEW=NOT_ASSESSABLE_NO_STANDARDIZED_TRANSITION
HUMAN_EXECUTION_REVIEW=NOT_REQUIRED_FOR_CURATION_CALIBRATION
R4_CALIBRATION=PAUSED_PENDING_PROTOCOL_V2_VALIDATION

IMPLEMENTATION_AUTHORIZATION=NO
MERGE_AUTHORIZATION=NO
OPTIMIZER_RANKING_ACTIVATION=NO
RELEASE_AUTHORIZATION=NO
DEPLOY_AUTHORIZATION=NO
PRODUCTION_ACTIVATION=NO
PDM_TRAINING=NO

[nulleimy]

Skill Tester attack #1 — Protocol V2 candidate

Tester result: FAIL_WITH_FIXABLE_BLOCKERS

The separation is directionally correct, but the candidate is not yet implementation-ready.

Blocker A — curation preference can still be contaminated by live execution

The instruction says to ignore how well/badly a transition was executed, but the protocol does not bind how the curation judgment was obtained. A reviewer can still live-mix A and B differently and unconsciously prefer the plan that happened to be mixed better.

Required fix:

• add explicit audition_mode=sequence_curation_only for Curation Review V2;
• state that transition execution quality is out of scope;
• add execution_quality_excluded_from_curation_judgment=true acknowledgement;
• if the reviewer cannot separate sequence quality from execution quality, they MUST choose abstain, not force a preference.

Blocker B — old/new packet schema ambiguity

Reusing the same case/assignment identities is correct, but simply reusing the legacy packet fingerprint risks accidental ingestion of Bundle 63 evidence as V2.

Required fix:

• create a distinct applaylist-curation-review-packet-v2 schema;
• include source_blinded_packet_fingerprint as provenance only;
• create a new V2 packet fingerprint over V2 fields;
• V2 calibration must reject the legacy Bundle 63 submission schema.

Blocker C — reviewer population scope is undefined

Twelve cases from one DJ can support a bounded personal-DJ calibration result, but cannot establish general market/product superiority.

Required fix:

• report an explicit evidence scope:
  • personal_dj_calibration for one reviewer identity;
  • multi_dj_product_evaluation only when independent reviewer coverage policy is met;
• never translate a personal-DJ result into a generalized competitive claim.

Blocker D — alternative_usefulness semantics are underspecified

Each A/B plan is already an alternative. Without a definition, reviewers can interpret this as novelty, transition ease, track quality, or willingness to play it.

Required fix:
Define it as:

alternative_usefulness: how useful this complete sequence is as a viable set-plan option the DJ would realistically keep, adapt, or play.

It must explicitly exclude transition-execution quality.

Blocker E — no contamination invariant in calibration API

A design statement is insufficient. The calibration contract must structurally accept only CurationReviewV2 evidence and expose no transition/execution rating fields in its input path.

Required negative proof:

• varying transition/execution evidence while keeping CurationReviewV2 fixed yields byte-for-byte identical curation calibration report identity and metrics.

Tester status

PROTOCOL_V2_IMPLEMENTATION_READY=NO

No code branch should be created until these blockers are incorporated and re-attacked.

[nulleimy]

Protocol V2.1 — blockers incorporated, PRE-IMPLEMENTATION

This supersedes the previous V2 candidate for tester purposes only.

A. New schema boundary

Create a distinct reviewer packet:

schema = applaylist-curation-review-packet-v2

Required provenance:

• protocol_version = curation-review-v2
• source_blinded_packet_fingerprint = immutable Bundle 63 source provenance only
• source_snapshot_ref
• same immutable case_id / assignment_id / blinded Plan A/B ordering
• new curation_packet_fingerprint computed only over the V2 packet fields
• algorithm_identity_hidden=true
• all activation/training authority false

Legacy Bundle 63 submission JSON is not valid Curation Review V2 input.

---

B. Curation Review V2.1

Audition mode

Required constant:

audition_mode = sequence_curation_only

Reviewer instruction:

Judge only the track selection, ordering and whole-set arc. Do not score or use the quality of a transition you personally performed as evidence. You may cue/preview tracks or use a neutral hard switch/simple unscored fade only to hear the sequence. If you cannot separate sequence quality from your own transition execution, choose abstain.

Per completed review:

execution_quality_excluded_from_curation_judgment = true

A review with this field false/missing is invalid for curation calibration.

Required Plan A/B dimensions 1..5

1. energy_flow

   • whether the track order creates an appropriate energy trajectory for the case/set role;
   • excludes smoothness of a manually performed transition.

2. dramaturgical_fit

   • whether the sequence supports the intended opening/build/mid-set/peak/reset/closing role and overall narrative.

3. set_coherence

   • whether the sequence feels intentionally connected as a set plan rather than a collection of individually plausible tracks.

4. alternative_usefulness

   • how useful the complete sequence is as a viable set-plan option the DJ would realistically keep, adapt or play;
   • excludes execution difficulty of a personally performed transition.

5. track_selection_fit

   • how well the chosen tracks belong in the intended set role/style/energy context independent of live mixing technique.

Case-level outcome

• curation_preference = plan_a | plan_b | tie | abstain
• curation_confidence = 0..1
• optional reason codes
• optional notes
• real observed_at
• algorithm identity hidden
• execution-quality exclusion acknowledgement true

The human preference is explicit. It MUST NOT be derived from score sums.

---

C. Transition Feasibility stream

Separate schema and evidence path. It is not required for Curation Review V2.1.

Until APPLAYLIST supplies a deterministic proposed transition recipe or immutable standardized preview:

transition_review_status = not_assessable_no_standardized_transition

No transition numeric scores are required or permitted as curation evidence.

When a future transition proposal is assessable, its feasibility dimensions live in a separate contract and cannot alter curation calibration input.

---

D. Human Execution stream

Optional separate experiment only.

No execution score/note/preference field is accepted by the curation calibration API.

---

E. Calibration V3 structural boundary

Calibration input type must bind only:

• CuratedReviewCase
• BlindedPlanAssignment
• CurationDJReviewV2
• Bundle 67 ShadowPathComparison

It MUST NOT accept legacy HumanDJReview as curation truth.

Structural contamination invariant:

With the same CurationDJReviewV2 evidence, adding/changing/removing any transition-feasibility or human-execution evidence must produce the same curation calibration report identity and metrics.

Required negative test proves this invariant.

---

F. Evidence scope

Every calibration report contains explicit scope:

personal_dj_calibration

Valid when evidence comes from one genuine reviewer identity across required cases/set roles.

Meaning:

• evidence about how well challenger matches that DJ;
• may support personal preference evaluation;
• MUST NOT be described as general product/market superiority.

multi_dj_product_evaluation

Requires a policy-defined minimum number of independent genuine reviewer identities plus required case/role coverage per reviewer or per aggregate policy.

R1 implementation proposal: minimum 3 independent reviewers before this scope may be emitted.

Even this scope yields only:

• INCOMPLETE
• DOES_NOT_SUPPORT_ACTIVATION
• SUPPORTS_FURTHER_EVALUATION

Never optimizer activation.

---

G. Required tests

1. legacy Bundle 63 submission rejected by Curation Review V2 parser
2. missing/false execution-quality exclusion acknowledgement rejected
3. curation preference cannot be derived from dimension arithmetic
4. transition/execution evidence variation cannot affect calibration report identity or metrics
5. one reviewer can emit only personal_dj_calibration
6. fewer than 3 independent reviewers cannot emit multi_dj_product_evaluation
7. tie preserved
8. abstain preserved/excluded from accuracy denominator
9. missing required curation dimension rejected
10. duplicate dimensions/reviews rejected
11. algorithm identity exposure rejected
12. assignment/case/path mismatch rejected
13. legacy qualitative notes are not auto-converted
14. transition numeric scoring is unavailable without standardized proposal binding
15. no MIR/audio/network/cloud side effects
16. no optimizer/TransitionAssessment mutation
17. activation/PDM/release/deploy authority structurally false

V2.1 current state

IMPLEMENTATION_READY=PENDING_SKILL_TESTER_REATTACK
R4_CALIBRATION=PAUSED

No implementation branch exists yet.

[nulleimy]

Skill Tester attack #2 — Protocol V2.1

Tester result: FAIL_WITH_ONE_CRITICAL_METHODOLOGY_BLOCKER

V2.1 successfully closes the transition-execution contamination boundary, but it still risks evaluation leakage / circular validation.

Critical blocker — the existing 12 cases are no longer a clean validation set

The qualitative findings from these exact R4 cases directly influenced later system design:

• excessive UKG / UK bassline concentration;
• sudden rave-techno drift;
• weak PEAK energy;
• technically different but musically near-equivalent A/B alternatives.

Bundle 65 and especially Bundle 67 were explicitly designed and regression-tested around those findings.

Therefore the existing 12 cases are legitimate development/calibration evidence, but they cannot independently prove that the new challenger generalizes or outperforms the prior optimizer.

Using them as the only post-change validation set would be circular:

HUMAN FEEDBACK ON CASES -> MODEL/POLICY DESIGN -> TEST MODEL ON SAME CASES -> CLAIM IMPROVEMENT

Required fix — evidence role must be explicit

Add to every Curation Review V2 / calibration report:

evidence_role = development_calibration | holdout_validation

Existing 12 R2 cases

Must be marked:

evidence_role = development_calibration

They may be used to:

• quantify the DJ's previously observed preferences;
• calibrate thresholds/reason codes;
• debug disagreement;
• evaluate whether the new protocol captures the known failures correctly.

They MUST NOT by themselves support a claim of generalization, competitive superiority, or optimizer activation.

Holdout validation

Before a future optimizer-activation decision, require genuinely new blinded cases that were not used to design/tune Bundle 65/67 or their thresholds.

Holdout requirements:

• new case identities / scenario fingerprints;
• hidden algorithm identity;
• no human preference available to implementation/tuning before challenger decision is frozen;
• same Curation Review V2 protocol;
• required set-role coverage;
• source optimizer/challenger identities frozen before human review is revealed;
• no threshold changes after holdout labels are revealed without invalidating that holdout as validation evidence.

A holdout may reuse already-existing immutable library/MIR evidence if identity-safe case generation is possible; new audio analysis is not automatically required.

Additional scope rule

personal_dj_calibration on the old 12 cases is still useful and valid as development calibration for that DJ.

multi_dj_product_evaluation and any future activation evidence must report development vs holdout results separately.

Tester status

TRANSITION_CONTAMINATION_BLOCKER=RESOLVED_BY_V2_1_DESIGN
VALIDATION_LEAKAGE_BLOCKER=OPEN
PROTOCOL_IMPLEMENTATION_READY=NO

Revise once more before implementation.

[nulleimy]

Protocol V2.2 — validation leakage fix, PRE-IMPLEMENTATION

This supersedes V2.1 for tester purposes.

All V2.1 separation rules remain in force.

---

H. Evidence role is mandatory

Every Curation Review V2 packet, review, calibration case, and aggregate must bind one of:

• development_calibration
• holdout_validation

The role is immutable for that evidence lineage.

Existing R2 12-case set

Hard classification:

evidence_role = development_calibration

Reason: qualitative findings from these cases influenced Bundle 65 / Bundle 67 design and regression policy.

Allowed use:

• personal-DJ preference calibration;
• disagreement analysis;
• threshold/reason-code development;
• protocol verification;
• regression testing against known failures.

Forbidden use by itself:

• generalization claim;
• product superiority claim;
• optimizer activation evidence;
• holdout-validation claim.

---

I. Holdout Validation Manifest R1

A new holdout may be evaluated only when a manifest is frozen before human labels are available.

Required immutable fields:

• holdout_id
• holdout_version
• evidence_role = holdout_validation
• source library snapshot ref/fingerprint
• case-selection policy id/version
• deterministic selection seed commitment / fingerprint
• complete selected case IDs + scenario fingerprints
• required set-role coverage
• source optimizer canonical SHA
• challenger implementation SHA / policy id / policy version
• challenger configuration digest
• calibration threshold/policy digest
• source evidence revisions
• generated/frozen timestamp
• human_labels_available_at_freeze = false
• algorithm_identity_hidden = true
• all activation/training authority false

Holdout selection constraints

• case inclusion/exclusion rules are defined before labels;
• no case may be added/removed after labels to improve metrics;
• selection may not depend on human preference labels;
• if difficulty/role stratification is used, it must be declared in the selection policy before execution;
• recommended R1 shape: at least 12 new cases with all six set roles represented, preferably two per role;
• old development case IDs/scenario fingerprints are excluded.

A holdout may reuse existing immutable MusicDNA/MIR/library evidence; new audio analysis is not required merely to create a new evaluation set.

---

J. Freeze-before-reveal invariant

Before holdout human review starts, freeze:

1. source optimizer revision;
2. challenger revision;
3. challenger weights/policy/config;
4. curation calibration thresholds;
5. holdout case-selection manifest;
6. reviewer protocol version.

After any holdout human label is revealed:

• changing challenger behavior/weights/thresholds invalidates that holdout as independent validation for the changed system;
• the old holdout may then be retained as development_calibration evidence;
• a new fresh holdout is required for the changed candidate.

---

K. Curation calibration semantics by evidence role

Development calibration

May yield the existing bounded verdict vocabulary, but:

SUPPORTS_FURTHER_EVALUATION means only ready to proceed to a fresh holdout.

It never means validated improvement.

Report must expose:

• evidence_role=development_calibration
• independent_validation=false

Holdout validation

Report must expose:

• evidence_role=holdout_validation
• exact frozen manifest ref/fingerprint
• independent_validation=true only if all freeze/binding checks pass

Even a successful holdout still yields only bounded evaluation evidence; optimizer activation remains a separate future governed decision.

---

L. Anti-cherry-pick tests

Required additional negative tests:

18. existing R2 12 cases cannot be labeled holdout
19. duplicate development case/scenario fingerprint rejected from holdout
20. holdout manifest with human_labels_available_at_freeze=true rejected
21. challenger/config/threshold digest mismatch rejects holdout validation
22. post-label threshold/config mutation invalidates independent-validation status
23. adding/removing cases after manifest freeze changes fingerprint and fails binding
24. holdout role coverage below policy remains INCOMPLETE
25. development report cannot emit independent_validation=true
26. holdout report without frozen manifest cannot emit independent_validation=true

V2.2 state

TRANSITION_CONTAMINATION=DESIGN_RESOLVED
VALIDATION_LEAKAGE=DESIGN_RESOLVED
IMPLEMENTATION_READY=PENDING_SKILL_TESTER_ATTACK_3
R4_CALIBRATION=PAUSED

No implementation branch exists yet.

[nulleimy]

Skill Tester attack #3 — Protocol V2.2

Tester result: FAIL_WITH_TWO_BIAS_BLOCKERS

The development/holdout separation is now correct, but two remaining bias channels can still create misleading validation.

Blocker F — holdout selection can still depend on challenger outputs

V2.2 forbids selection based on human labels, but it does not forbid selecting cases because the challenger already scores them highly, disagrees favorably with the source optimizer, or otherwise looks promising.

That would bias a general-performance estimate even without seeing human labels.

Required fix:

Add selection_scope:

• representative_holdout
• diagnostic_challenge_set

For representative_holdout:

• case selection MUST be independent of challenger score/preference/outcome;
• selection may use predeclared library eligibility, set role, deterministic/random seed, and other model-independent stratification only;
• this is the only scope eligible for generalization/product-evaluation evidence.

For diagnostic_challenge_set:

• cases may intentionally target disagreement/hardness/failure modes;
• results are diagnostic only and MUST NOT be presented as representative performance.

Blocker G — A/B slot position bias across reviewers

A single fixed Plan A/B assignment reused for all reviewers can create systematic position bias. If one strategy maps to Plan A for a case across every reviewer, an A-preference tendency can masquerade as algorithm preference.

Required fix:

For multi_dj_product_evaluation:

• assignments must be counterbalanced across reviewer identities;
• each reviewer gets a blinded reviewer-specific assignment manifest derived from the same frozen source plans;
• assignment generation must be deterministic/auditable from a private seed commitment while keeping algorithm identity hidden;
• aggregate calibration resolves each review through its exact reviewer-specific assignment binding;
• no reviewer sees another reviewer's mapping during judgment.

For personal_dj_calibration with one reviewer:

• require case-level A/B allocation to be balanced as far as possible across the full case set;
• report slot counts and position-balance diagnostics.

Tester status

TRANSITION_CONTAMINATION=RESOLVED
DEVELOPMENT_HOLDOUT_LEAKAGE=RESOLVED
HOLDOUT_SELECTION_BIAS=OPEN
A_B_POSITION_BIAS=OPEN
IMPLEMENTATION_READY=NO

[nulleimy]

Protocol V2.3 — selection + position bias fixes, PRE-IMPLEMENTATION

All V2.2 rules remain in force.

---

M. Holdout selection scope

Every holdout manifest must bind:

selection_scope = representative_holdout | diagnostic_challenge_set

representative_holdout

Purpose: estimate performance/generalization on a bounded representative evaluation slice.

Selection MUST be independent of:

• human preference labels;
• challenger score;
• challenger preference;
• source-vs-challenger disagreement direction;
• any post-hoc knowledge of which candidate looks better.

Allowed predeclared selection inputs:

• frozen source library eligibility;
• set role;
• deterministic/random seed commitment;
• predeclared BPM/style/energy strata that do not use challenger output;
• fixed inclusion/exclusion rules unrelated to model success.

Only this scope may contribute to representative product-evaluation evidence.

diagnostic_challenge_set

Purpose: intentionally stress failure modes/disagreements/hard cases.

May select based on:

• known failure class;
• disagreement;
• low confidence;
• style/energy edge conditions.

Mandatory report flag:

representative_performance_claim_allowed = false

Diagnostic results are useful for debugging but cannot be reported as representative success rate.

---

N. Reviewer-specific blinded assignment manifests

Multi-DJ evaluation

Each reviewer receives a distinct blinded assignment manifest:

• same frozen case/source plans;
• reviewer-specific assignment_id per case;
• Plan A/B mapping derived deterministically from a private assignment seed commitment + reviewer pseudonymous identity + case_id;
• algorithm identity hidden;
• no reviewer can inspect another reviewer mapping during judgment.

Aggregate resolution always uses the exact reviewer-specific assignment manifest.

Counterbalancing rule

Across the reviewer population and case set:

• GREEDY/BEAM placement in Plan A/B must be as balanced as mathematically possible;
• report slot counts by reviewer and globally;
• position imbalance above policy invalidates multi_dj_product_evaluation completeness.

Personal-DJ calibration

For one reviewer:

• A/B allocation across the case set must be as balanced as mathematically possible;
• report plan_a_greedy_count, plan_b_greedy_count, plan_a_beam_count, plan_b_beam_count;
• imbalance beyond one case for an even/odd bounded set fails the assignment-balance check unless a documented structural reason exists.

---

O. Additional negative tests

27. representative holdout selection rejects challenger-score-dependent selection metadata
28. representative holdout rejects challenger-preference/disagreement-dependent selection metadata
29. diagnostic challenge set cannot emit representative-performance claim
30. reviewer-specific assignment mismatch fails closed
31. multi-DJ aggregate with unbalanced A/B mapping above policy remains INCOMPLETE
32. personal-DJ assignment imbalance above bounded policy remains INCOMPLETE
33. cross-reviewer assignment leakage/identity mismatch fails closed
34. assignment seed commitment mismatch fails closed

V2.3 state

TRANSITION_CONTAMINATION=DESIGN_RESOLVED
DEVELOPMENT_HOLDOUT_LEAKAGE=DESIGN_RESOLVED
HOLDOUT_SELECTION_BIAS=DESIGN_RESOLVED
A_B_POSITION_BIAS=DESIGN_RESOLVED
IMPLEMENTATION_READY=PENDING_SKILL_TESTER_ATTACK_4
R4_CALIBRATION=PAUSED

No implementation branch exists yet.

[nulleimy]

Skill Tester attack #4 — Protocol V2.3

Tester result: FAIL_WITH_ONE_SURVIVORSHIP_BIAS_BLOCKER

The protocol now controls transition contamination, development/holdout leakage, selection bias and A/B position bias. One remaining failure mode can still make the system look better than it is.

Blocker H — silent exclusion of weak/unreviewable system outcomes

A representative holdout can become biased if cases are silently dropped or regenerated after the source optimizer/challenger produces:

• identical A/B paths;
• no material alternative;
• missing required evidence;
• invalid/failed path generation;
• one strategy producing no reviewable plan.

Those are product outcomes, not reasons to erase the case after selection.

If the protocol keeps regenerating until every case has two attractive reviewable alternatives, it measures only survivors and overstates alternative quality.

Required fix

Once a representative holdout case is frozen by the selection manifest:

• the case remains in the denominator even if generation/reviewability fails;
• no replacement case may be substituted after seeing the failure;
• failures are recorded with explicit machine outcome codes;
• human A/B review is required only when two blinded reviewable plans genuinely exist;
• lack of two reviewable plans is itself product evidence.

Required machine outcome vocabulary before human review:

• reviewable_pair
• technically_identical_pair
• no_meaningful_alternative
• missing_required_evidence
• source_generation_failed
• challenger_not_proven

A case with an outcome other than reviewable_pair must not be converted to human abstain; that would incorrectly blame reviewer uncertainty. It stays a machine/system outcome and counts in holdout coverage/product metrics.

Representative holdout metrics must expose

• selected_case_count
• reviewable_pair_count
• non_reviewable_system_outcome_count
• human_reviewed_case_count
• reviewable_pair_fraction
• meaningful_alternative_availability_rate
• machine outcome counts by reason

Anti-survivorship tests

35. frozen case cannot be replaced after identical-path outcome
36. frozen case cannot be removed after missing-evidence outcome
37. non-reviewable system outcome remains in selected-case denominator
38. system non-reviewability cannot be encoded as human abstain
39. regenerated replacement case cannot inherit original holdout identity
40. representative holdout with poor alternative availability cannot hide failures behind human-review coverage

Tester status

SURVIVORSHIP_BIAS=OPEN
IMPLEMENTATION_READY=NO

Revise once more; no implementation branch yet.

[nulleimy]

Protocol V2.4 — survivorship fix, PRE-IMPLEMENTATION

All V2.3 rules remain in force.

---

P. Frozen-case execution outcomes

After a representative_holdout manifest is frozen, every selected case receives exactly one pre-human system outcome.

Required enum:

• reviewable_pair
• technically_identical_pair
• no_meaningful_alternative
• missing_required_evidence
• source_generation_failed
• challenger_not_proven

A selected case can never disappear from the evaluation denominator.

Invariants

• no post-outcome case replacement;
• no post-outcome case deletion;
• no retry-until-reviewable loop that changes selected identity;
• retries for transient execution errors may only preserve the same frozen case identity and must be auditable;
• a regenerated/reselected scenario receives a new identity and belongs to a new manifest/evidence lineage, never the original holdout.

---

Q. Human review eligibility

Human A/B Curation Review V2 is executed only when:

system_outcome = reviewable_pair

For all other system outcomes:

• no forced human score;
• no forced human preference;
• no human abstain substitution;
• the machine/system outcome remains explicit evidence.

Human abstain is reserved for a genuinely reviewable pair where the human cannot make a valid curation judgment.

---

R. Representative holdout denominator metrics

Every holdout report must expose at least:

• selected_case_count
• reviewable_pair_count
• non_reviewable_system_outcome_count
• human_reviewed_case_count
• reviewable_pair_fraction
• meaningful_alternative_availability_rate
• outcome count for every system-outcome enum value
• set-role distribution for selected cases
• set-role distribution for reviewable pairs

No human-agreement metric may be presented without the reviewable-pair denominator beside it.

Example interpretation:

A 90% challenger/human agreement on only 4 reviewable cases out of 12 is not a 90% product-success result; the report must simultaneously show reviewable_pair_fraction=0.33.

---

S. Holdout completeness semantics

independent_validation=true requires freeze/binding integrity, not perfect product performance.

A poor reviewable-pair fraction is allowed to produce valid negative evidence; it must not make the evidence disappear.

Policy may yield:

• INCOMPLETE only for missing/invalid evidence or protocol coverage;
• DOES_NOT_SUPPORT_ACTIVATION for valid but poor product outcomes;
• SUPPORTS_FURTHER_EVALUATION only when both alternative availability and human-alignment thresholds pass.

Thus system failure cannot hide behind INCOMPLETE if the evidence proving failure is itself complete.

---

T. Multi-DJ disagreement transparency

For multi_dj_product_evaluation, do not collapse reviewer disagreement into one hidden majority label.

Report per case:

• reviewer count
• plan_a / plan_b / tie / abstain counts
• confidence distribution summary
• exact disagreement rate

Report aggregate:

• per-reviewer challenger agreement
• macro-average across reviewers
• pooled agreement separately
• reviewer disagreement rate

A population result may not describe a polarized case as unanimous/consensus merely because a plurality exists.

---

U. Additional tests

35. frozen case cannot be replaced after identical-path outcome
36. frozen case cannot be removed after missing-evidence outcome
37. non-reviewable system outcome remains in selected denominator
38. system non-reviewability cannot be encoded as human abstain
39. regenerated replacement case cannot inherit original holdout identity
40. poor alternative availability cannot be hidden behind human-review coverage
41. complete negative system outcomes produce valid negative evidence, not automatic INCOMPLETE
42. agreement metric always carries reviewable-pair denominator
43. multi-DJ disagreement remains visible per case and in aggregate
44. pooled and per-reviewer/macro agreement are reported separately

V2.4 state

TRANSITION_CONTAMINATION=DESIGN_RESOLVED
DEVELOPMENT_HOLDOUT_LEAKAGE=DESIGN_RESOLVED
HOLDOUT_SELECTION_BIAS=DESIGN_RESOLVED
A_B_POSITION_BIAS=DESIGN_RESOLVED
SURVIVORSHIP_BIAS=DESIGN_RESOLVED
IMPLEMENTATION_READY=PENDING_SKILL_TESTER_FINAL_ATTACK
R4_CALIBRATION=PAUSED

No implementation branch exists yet.

[nulleimy]

Skill Tester final attack — Protocol V2.4

Tester result: PASS_FOR_IMPLEMENTATION_DESIGN

No remaining critical experiment-validity blocker was found in the proposed protocol boundary.

What is now structurally protected

• curation judgment is separated from live transition execution;
• legacy 6D overall preference cannot silently become Curation V2 truth;
• transition scores are unavailable without standardized transition evidence;
• old 12 cases are development/calibration only, not holdout proof;
• fresh holdout is frozen before human labels;
• representative holdout selection cannot depend on challenger outcome;
• diagnostic challenge sets cannot masquerade as representative performance;
• A/B mapping is counterbalanced and reviewer-specific for multi-DJ evaluation;
• selected holdout cases cannot disappear after weak system outcomes;
• identical/no-alternative/missing-evidence/generation failures remain in denominator;
• human abstain is not used to hide machine non-reviewability;
• multi-DJ disagreement remains visible rather than collapsed into false consensus;
• optimizer/release/deploy/PDM authority remains separate.

Non-blocking wording constraints

Implementation/docs must avoid overstating evidence:

1. independent_validation=true means independent holdout-case validation under this protocol, not independent laboratory replication.
2. Three reviewers is only the proposed minimum to unlock multi_dj_product_evaluation scope; it is not a statistically universal proof of market superiority.
3. No p-value/significance/market-wide accuracy claim may be emitted unless a separately designed statistical protocol supports it.
4. Personal-DJ calibration remains personal even when internally very accurate.

Tester verdict

PROTOCOL_EXPERIMENT_VALIDITY=PASS_FOR_IMPLEMENTATION
IMPLEMENTATION_READY=YES
R4_CURRENT_LEGACY_6D_CALIBRATION=STILL_PAUSED

This PASS authorizes only moving the validated design into a governed implementation branch under the user's current GO. It does not authorize merge, optimizer activation, release, deploy, production effects, or PDM training.

[nulleimy]

Skill Tester closure for Bundle 69 implementation

Protocol V2.4 implementation has now passed the adversarial implementation audit on PR #161 exact HEAD aca2eacc54583e6aa9152e3756abc996c8288052.

Final gates:

• CI #939 PASS: 512 tests on Python 3.11 and 512 on Python 3.12
• PR Guard #338/#339/#340 PASS
• 8 new files / 0 pre-existing files modified
• optimizer and legacy review/calibration authority files byte-identical to canonical base
• review threads 0 / reviews 0

Implementation-only gaps discovered after earlier green CI were also closed before Ready:

1. prior development identities are excluded from holdout lineage;
2. representative selection uses an enumerated model-independent selection basis;
3. reviewer A/B assignments are deterministically counterbalanced and verified from a private seed commitment;
4. purity/security inspection includes the holdout guard service.

Tester verdict:
PROTOCOL_V2_4_DESIGN=PASS
BUNDLE_69_IMPLEMENTATION=PASS_FOR_REVIEW
LEGACY_R4_6D_CALIBRATION=PAUSED

This does not claim real human V2 evidence, independent lab replication, optimizer superiority, activation, release, deploy, production readiness, or PDM training authority. A real holdout still needs a persisted pre-label freeze receipt.