Bundle 70: Fresh Personal Holdout Runner R1

docs/bundles/BUNDLE_70_FRESH_PERSONAL_HOLDOUT_RUNNER_R1.md

@@ -0,0 +1,90 @@
+# Bundle 70 — Fresh Personal Holdout Runner R1
+
+## Goal
+
+Provide the local execution layer that turns the canonical Human Review Protocol R2 / Curation Calibration R3 contracts into a reproducible fresh personal holdout run.
+
+The runner must stop before human labels are collected unless all pre-label evidence is frozen and auditable.
+
+## Required execution order
+
+1. Verify the local checkout is on canonical branch `feature/bundle-0-bootstrap`, its HEAD exactly matches the supplied canonical SHA, and the working tree is clean.
+2. Validate private local-library snapshot R1.
+3. Deterministically generate a bounded candidate case pool without reading human labels or challenger scores.
+4. Materialize real-library optimizer evidence and blind A/B assignments locally, isolating per-case technical failures so one invalid candidate cannot abort the full pool.
+5. Build engineering-only `HoldoutCandidate` rows.
+6. Freeze `HoldoutCaseSamplingPolicy` and select at least 24 personal holdout cases with four cases per set role plus a frozen fallback reservoir.
+7. Freeze replacement policy and effective cohort.
+8. Compute Bundle 67 competitive-curation shadow comparisons for the selected cases and frozen fallback reservoir before reviewer workspace publication.
+9. Persist a private pre-registration manifest containing selection, assignments, challenger evidence, fingerprints, and authority=false.
+10. Require zero replacement events in the R1 effective cohort before reviewer publication.
+11. Before finalization, verify byte identity of the generated private manifest, reviewer packet, and CSV against their pre-finalization SHA-256 values.
+12. Require reviewer case order/identity to match the frozen effective cohort exactly, with assignment and set-role metadata matching private frozen evidence.
+13. Apply dual prior-exposure exclusion: reviewer-visible sequence matching plus stable track-ID matching from prior private manifests.
+14. Finalize a reviewer-safe workspace bound to the exact preregistration/selection/cohort and exposure-registry fingerprints.
+15. Publish a reviewer-safe packet containing only anonymous Plan A / Plan B track sequences and the four R2 curation dimensions.
+16. Create the review CSV with system binding metadata but leave all human judgments and clean-attestation assertions empty; no ratings, preferences, confidence, timestamps, or exposure claims may be fabricated.
+
+## Critical isolation
+
+`HOLDOUT_SELECTION_INPUTS` may include only frozen policy plus engineering/technical candidate metadata.
+
+`HOLDOUT_SELECTION_INPUTS` must not include:
+- human preference;
+- human ratings;
+- reviewer notes;
+- competitive challenger scores;
+- competitive challenger preference.
+
+The challenger comparison is computed only after holdout selection is frozen, but before reviewer workspace publication. Challenger evidence is also frozen for the fallback reservoir so a later protocol version can support bounded technical replacement without post-label challenger computation.
+
+For **Fresh Personal Holdout Run R1**, the effective cohort is immutable once reviewer workspace publication begins. A technical invalidity discovered after publication aborts/restarts the run; it does not authorize an in-review fallback substitution.
+
+## Freshness / prior-exposure rule
+
+A new `case_id` or changed display metadata is not evidence that a case is fresh.
+
+The formal run requires two independent historical exposure sources:
+
+1. prior blinded reviewer packet(s), used to reject exact reviewer-visible Plan A / Plan B sequence reuse;
+2. prior private review manifest(s), used to reject exact stable track-ID sequence reuse even if display metadata changed.
+
+Before Case 1 may be opened, the workspace finalizer must fail closed if an effective holdout case reproduces:
+- an exact previously exposed individual plan sequence; or
+- an exact previously exposed A/B sequence pair.
+
+After successful finalization the new private manifest receives an opaque stable-exposure registry so subsequent fresh runs can exclude this holdout without exposing track identities to the reviewer.
+
+## Reviewer-safe dimensions
+
+- energy_flow
+- dramaturgical_fit
+- set_coherence
+- alternative_usefulness
+
+Allowed preference values:
+- plan_a
+- plan_b
+- tie
+- abstain
+
+No transition execution is requested in this runner.
+
+The review CSV contains explicit R2 attestation fields, but the human-controlled fields remain empty at workspace freeze and must be completed only from the actual review session.
+
+## Privacy
+
+- local audio paths stay in private evidence only;
+- no audio upload;
+- no cloud MIR execution;
+- reviewer packet must not expose absolute paths, optimizer strategy identity, shadow scores, challenger preference, or stable track IDs;
+- prior-exposure sources are hashed/bound in private evidence and are not published through the reviewer packet.
+
+## Authority
+
+- optimizer ranking activation: NO
+- PDM training: NO
+- release: NO
+- deploy: NO
+- production activation: NO
+- merge: NO without separate `MERGE GO`

docs/runs/FRESH_PERSONAL_HOLDOUT_RUN_R1.md

@@ -0,0 +1,104 @@
+# Fresh Personal Holdout Run R1
+
+## Purpose
+
+Execute a fresh personal blind curation holdout only after Bundle 70 is merged.
+
+## Preconditions
+
+- local APPLAYLIST checkout on canonical branch `feature/bundle-0-bootstrap`;
+- local HEAD exactly matches the canonical SHA supplied to the runner;
+- local working tree is clean;
+- private `applaylist-local-library-snapshot-r1` JSON;
+- local audio files remain readable;
+- one or more prior blinded reviewer packets covering sequences already exposed to the reviewer, including the historical 12-case packet;
+- one or more prior private review manifests carrying stable track-identity sequence evidence, including the historical real-library private manifest;
+- no reviewer labels have been collected for the new run;
+- sampling and blinding seeds are chosen before review.
+
+## Local command
+
+```bash
+python scripts/applaylist_fresh_personal_holdout.py \
+  --snapshot "$SNAPSHOT" \
+  --output "$OUTPUT" \
+  --database "$OUTPUT/APPLAYLIST_FRESH_PERSONAL_HOLDOUT_R1.sqlite" \
+  --canonical-sha "$CANONICAL_SHA" \
+  --generated-at "$GENERATED_AT" \
+  --sampling-seed "$SAMPLING_SEED" \
+  --blinding-seed "$BLINDING_SEED" \
+  --exclude-reviewer-packet "$PRIOR_REVIEW_PACKET" \
+  --exclude-private-manifest "$PRIOR_PRIVATE_MANIFEST"
+```
+
+Both exclusion arguments are repeatable when more than one earlier exposure source must be covered.
+
+The runner must fail closed before evidence generation if the checkout is not the exact canonical commit or the working tree is dirty.
+
+## Expected private outputs
+
+- `APPLAYLIST_FRESH_PERSONAL_HOLDOUT_R1.private.json`
+- local SQLite database
+
+The private manifest includes frozen selection, replacement/effective-cohort provenance, blind assignments, challenger evidence for the selected + fallback reservoir, and after workspace finalization an opaque stable-exposure registry for future holdout runs. Challenger evidence must be frozen before the reviewer workspace is published.
+
+These outputs must not be published to a public repository because the private manifest is bound to local evidence and may contain private provenance.
+
+## Expected reviewer-safe outputs
+
+- `APPLAYLIST_FRESH_PERSONAL_HOLDOUT_REVIEWER_R1.json`
+- `APPLAYLIST_FRESH_PERSONAL_HOLDOUT_REVIEW_R1.csv`
+
+The reviewer packet contains anonymous Plan A / Plan B sequences and only the R2 curation dimensions:
+
+- `energy_flow`
+- `dramaturgical_fit`
+- `set_coherence`
+- `alternative_usefulness`
+
+The CSV contains system binding fields and explicit R2 clean-attestation columns, but all human-controlled fields must be empty when the workspace is frozen.
+
+## Prior-exposure exclusion
+
+A case is not fresh merely because it has a new case identifier or because display metadata changed.
+
+Before the reviewer workspace is finalized, the runner performs two independent exclusion checks:
+
+1. reviewer-visible sequence matching against prior blinded reviewer packets;
+2. stable track-ID sequence matching against prior private manifests.
+
+It must fail closed if an effective holdout case reproduces:
+
+- any exact previously exposed individual plan sequence; or
+- any exact previously exposed A/B sequence pair.
+
+The pre-finalization SHA-256 values of the generated private manifest, reviewer packet, and CSV are also verified before the finalizer is allowed to bind them. The reviewer case order/identity must exactly equal the frozen effective cohort, and reviewer assignment/set-role metadata must match the private frozen evidence.
+
+Do not open Case 1 if either exposure exclusion layer or any binding check failed.
+
+## Cohort immutability during review
+
+Fresh Personal Holdout Run R1 requires zero replacement events before reviewer publication. Once the reviewer workspace is finalized, the 24-case effective cohort is immutable for that run.
+
+If any selected case later becomes technically invalid, do **not** substitute a fallback during human review. Abort the run and create a new preregistered fresh holdout instead. The frozen fallback reservoir exists as pre-label provenance/future protocol support, not as authority for an in-review swap in R1.
+
+## Stop gate before Case 1
+
+Before opening the reviewer packet, verify:
+
+- exact canonical SHA matches the run preregistration and local HEAD;
+- canonical branch is `feature/bundle-0-bootstrap`;
+- working tree was clean at run start;
+- selected holdout has 24 effective cases;
+- all six set roles are represented with four cases each;
+- effective cohort has zero replacement events before publication;
+- reviewer case order/identity exactly matches the frozen effective cohort;
+- replacement policy and effective cohort fingerprints are frozen;
+- challenger comparisons for selected + fallback cases are present in private evidence and absent from reviewer-safe outputs;
+- reviewer-visible and stable track-ID prior-exposure registries were both applied successfully;
+- reviewer packet is bound to the frozen preregistration/selection/effective-cohort fingerprints;
+- all human review and clean-attestation fields are empty at freeze;
+- algorithm identity is hidden;
+- transition execution is not requested.
+
+If any check fails, do not review Case 1.

scripts/applaylist_fresh_personal_holdout.py

@@ -0,0 +1,142 @@
+from __future__ import annotations
+
+import argparse
+import json
+import subprocess
+from pathlib import Path
+from typing import Mapping
+
+from services.intelligence.fresh_holdout_reviewer_workspace import (
+    finalize_fresh_holdout_reviewer_workspace,
+)
+from services.intelligence.fresh_personal_holdout_runner import (
+    FreshPersonalHoldoutRunnerError,
+    materialize_fresh_personal_holdout_r1,
+)
+
+
+def _parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        description="Materialize a fresh blinded personal curation holdout locally."
+    )
+    parser.add_argument("--snapshot", required=True)
+    parser.add_argument("--output", required=True)
+    parser.add_argument("--database", required=True)
+    parser.add_argument("--canonical-sha", required=True)
+    parser.add_argument("--canonical-branch", default="feature/bundle-0-bootstrap")
+    parser.add_argument("--generated-at", required=True)
+    parser.add_argument("--sampling-seed", required=True)
+    parser.add_argument("--blinding-seed", required=True)
+    parser.add_argument(
+        "--exclude-reviewer-packet",
+        action="append",
+        required=True,
+        help="Prior blinded reviewer packet to exclude from visible-sequence exposure; repeatable.",
+    )
+    parser.add_argument(
+        "--exclude-private-manifest",
+        action="append",
+        required=True,
+        help=(
+            "Prior private review manifest carrying stable track-identity exposure evidence; "
+            "repeatable."
+        ),
+    )
+    parser.add_argument("--cases-per-role", type=int, default=8)
+    parser.add_argument("--candidate-scope-size", type=int, default=16)
+    parser.add_argument("--fallback-count", type=int, default=12)
+    return parser
+
+
+def _git(*args: str) -> str:
+    completed = subprocess.run(
+        ("git", *args),
+        check=True,
+        stdout=subprocess.PIPE,
+        stderr=subprocess.PIPE,
+        text=True,
+    )
+    return completed.stdout.strip()
+
+
+def verify_canonical_checkout(*, canonical_sha: str, canonical_branch: str) -> None:
+    """Fail closed unless the run is launched from the exact clean canonical checkout."""
+    try:
+        head = _git("rev-parse", "HEAD")
+        branch = _git("rev-parse", "--abbrev-ref", "HEAD")
+        dirty = _git("status", "--porcelain")
+    except (subprocess.CalledProcessError, FileNotFoundError) as exc:
+        raise FreshPersonalHoldoutRunnerError(
+            "fresh holdout run requires a readable local Git checkout"
+        ) from exc
+
+    expected_sha = str(canonical_sha).strip()
+    expected_branch = str(canonical_branch).strip()
+    if not expected_sha or not expected_branch:
+        raise FreshPersonalHoldoutRunnerError("canonical SHA/branch must not be empty")
+    if head != expected_sha:
+        raise FreshPersonalHoldoutRunnerError(
+            f"local HEAD {head} does not match declared canonical SHA {expected_sha}"
+        )
+    if branch != expected_branch:
+        raise FreshPersonalHoldoutRunnerError(
+            f"local branch {branch} does not match canonical branch {expected_branch}"
+        )
+    if dirty:
+        raise FreshPersonalHoldoutRunnerError(
+            "fresh holdout run requires a clean working tree before evidence generation"
+        )
+
+
+def verify_r1_fixed_effective_cohort(result: Mapping[str, str]) -> None:
+    """R1 publishes only a cohort with no replacement events already applied.
+
+    Once the reviewer workspace is finalized, later technical invalidity must abort and
+    restart the run rather than silently swap in a fallback during human review.
+    """
+    private_path = Path(str(result.get("private_manifest", "")).strip())
+    if not private_path.is_file():
+        raise FreshPersonalHoldoutRunnerError("fresh holdout private manifest is missing")
+    raw = json.loads(private_path.read_text(encoding="utf-8"))
+    cohort = raw.get("effective_cohort") if isinstance(raw, dict) else None
+    if not isinstance(cohort, dict):
+        raise FreshPersonalHoldoutRunnerError("fresh holdout effective cohort is missing")
+    events = cohort.get("replacement_events")
+    if events not in ([], ()):
+        raise FreshPersonalHoldoutRunnerError(
+            "Fresh Personal Holdout R1 requires zero replacement events before review publication"
+        )
+
+
+def main() -> int:
+    args = _parser().parse_args()
+    verify_canonical_checkout(
+        canonical_sha=args.canonical_sha,
+        canonical_branch=args.canonical_branch,
+    )
+    snapshot_path = Path(args.snapshot)
+    result = materialize_fresh_personal_holdout_r1(
+        snapshot_path=snapshot_path,
+        output_dir=Path(args.output),
+        database_path=Path(args.database),
+        canonical_sha=args.canonical_sha,
+        generated_at=args.generated_at,
+        sampling_seed=args.sampling_seed,
+        blinding_seed=args.blinding_seed,
+        cases_per_role=args.cases_per_role,
+        candidate_scope_size=args.candidate_scope_size,
+        fallback_count=args.fallback_count,
+    )
+    verify_r1_fixed_effective_cohort(result)
+    result = finalize_fresh_holdout_reviewer_workspace(
+        result,
+        snapshot_path=snapshot_path,
+        prior_reviewer_packet_paths=tuple(args.exclude_reviewer_packet),
+        prior_private_manifest_paths=tuple(args.exclude_private_manifest),
+    )
+    print(json.dumps(result, indent=2, sort_keys=True))
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())