Bundle 70: Fresh Personal Holdout Runner R1 by nulleimy · Pull Request #166 · eimyroot/APPLAYLIST

nulleimy · 2026-08-23T08:39:40Z

Closes #165

Summary

Adds the local-only execution layer for a fresh personal blinded curation holdout under the canonical Bundle 69 Human Review Protocol R2 / Curation Calibration R3 methodology.

Bundle Context

Bundle: 70 — Fresh Personal Holdout Runner R1
Canonical base: feature/bundle-0-bootstrap
Exact base SHA: 27af5d04e1f7971126adef74dff4d34ee54536df
Governing issue: Bundle 70: Fresh Personal Holdout Runner R1 #165
R2/R3 protocol source: Bundle 69 / PR Bundle 69: Human Review Protocol R2 Isolation + Curation Calibration R3 #164

Scope

deterministic candidate generation from frozen private snapshot identity only;
per-case local MIR analysis and optimizer materialization with failure isolation;
engineering-only holdout candidate ledger;
frozen 24-case personal holdout, four cases per set role, plus frozen fallback reservoir;
challenger evidence frozen only after selection and before reviewer publication;
full source CuratedReviewCase records frozen privately for later unblinding/calibration without optimizer rerun;
exact canonical branch/SHA + clean-working-tree preflight;
reviewer-safe Plan A/B packet with only four R2 curation dimensions;
private preregistration/provenance manifest;
pre-finalization SHA-256 verification of private/reviewer/CSV files;
reviewer case/order/assignment/role binding to the frozen effective cohort;
dual prior-exposure exclusion using reviewer-visible sequences + stable track-ID sequences from private manifests;
explicit human/attestation CSV fields left empty at freeze;
immutable R1 effective cohort after reviewer publication;
local CLI, runbook, adversarial/isolation/security tests.

Critical invariants

holdout selection receives no human preferences, ratings, notes, challenger scores, or challenger preferences;
a MIR/analyzer failure affects only cases that require the failed evidence and does not abort the entire pool;
frozen private source cases preserve plan IDs, path IDs, and ordered track IDs required for post-review unblinding/R3 calibration;
changed case IDs/display metadata cannot bypass prior-exposure exclusion;
reviewer-safe outputs contain no absolute paths, stable track IDs, optimizer strategy identity, challenger scores, or challenger preference;
transition execution is not requested;
no human labels, attestation claims, confidence, or timestamps are fabricated;
a technical invalidity discovered after reviewer publication aborts/restarts R1 rather than swapping a fallback in-review;
actual Fresh Personal Holdout Run must not execute until this bundle is canonical;
no optimizer/release/deploy/production/PDM authority.

Verification

Exact HEAD: baa7e8312ef02907e01042ab4fb6d841720e5108

PR Guard #382: SUCCESS
CI #1026: SUCCESS
Python 3.11.16: 533 passed, 6 warnings
Python 3.12.14: 533 passed, 5 warnings
compile: PASS on 3.11 + 3.12
critical Ruff E9,F63,F7,F82: PASS on 3.11 + 3.12
Python 3.11 artifact ID: 9492270480
Python 3.11 artifact SHA-256: 44adee1ea241e9b2620c3059a5761eaeab8788c2de40a7214c41ce0285bb4cb0
Python 3.12 artifact ID: 9492269541
Python 3.12 artifact SHA-256: 65d8089109b87b6b700a892a8790cf21558ec0b4be26bf504ec6607d852c57cd
scope audit: 9 added Bundle70 files, 0 existing files modified/deleted
branch behind canonical base: 0
late Codex P1 MIR-isolation finding: FIXED + regression test
late Codex P1 frozen-source-case finding: FIXED + regression test
both P1 review threads: RESOLVED
Set Engine blob unchanged: dd8f0cdb672202289530579d092cf0feb5cedf25
Set Path Optimizer blob unchanged: 819f6399181f4b37bb26ed8ccefb6cc0f4a9607f
Bundle 67 Competitive Curation blob unchanged: faadcae932e5f7b907503cbf2cd5bfdaab36462e

READY_FOR_MERGE=YES

Authority

MERGE_AUTHORIZATION=YES — explicit user MERGE GO
OPTIMIZER_RANKING_ACTIVATION=NO
PDM_TRAINING=NO
RELEASE_AUTHORIZATION=NO
DEPLOY_AUTHORIZATION=NO
PRODUCTION_ACTIVATION=NO

…ger evidence

…ation

nulleimy

Skill Tester final verification on exact HEAD e3c8d3e27e6b993527c6bd9c39a14bcd0cbca890: PASS. No unresolved HIGH/CRITICAL findings after candidate-failure isolation, exact canonical checkout preflight, empty human-attestation fields at freeze, pre-finalization SHA binding, exact effective-cohort/assignment/role binding, dual reviewer-visible + stable track-ID prior-exposure exclusion, reviewer leakage checks, and immutable R1 cohort/no in-review fallback substitution. CI #1020 PASS: 531 passed on Python 3.11 and 3.12; PR Guard #377 PASS. This review does not authorize merge, optimizer activation, release, deploy, production, or PDM training.

nulleimy

Post-Ready gate on unchanged exact HEAD e3c8d3e27e6b993527c6bd9c39a14bcd0cbca890: PR Guard #378 SUCCESS and PR Guard #379 SUCCESS after Ready-for-Review transition. CI #1020 remains SUCCESS. READY_FOR_REVIEW=YES. MERGE_AUTHORIZATION=NO; separate explicit MERGE GO is still required.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e3c8d3e27e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

nulleimy · 2026-08-23T11:11:19Z

@codex review

Please re-review exact HEAD baa7e8312ef02907e01042ab4fb6d841720e5108, with focus on the two previously reported P1 findings: per-case MIR failure isolation before materialization, and persistence of full frozen source CuratedReviewCase records needed for later unblinding/calibration without optimizer rerun.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: baa7e8312e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-08-23T11:14:45Z

+                snapshot_raw=snapshot_raw,
+                selection_raw=_single_case_selection(selection_raw, spec),
+            )
+        except RealLibraryPilotError as exc:


Catch actual MIR provider failures at the case boundary

When one selected audio file is corrupt, empty, non-finite, or silent, this boundary still aborts the entire run because it catches only RealLibraryPilotError. Fresh evidence at exact HEAD is that BaselineLibrosaMIR.analyze directly raises ValueError for several invalid-audio conditions and allows decoder exceptions from librosa.load to propagate, while analyze_real_tracks does not translate them. Catch and classify the expected provider/decoder exceptions here, or normalize them inside analyze_real_tracks, so only cases depending on that track become technically ineligible.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-08-23T11:14:45Z

+            case_evidence = analyze_real_tracks(
+                snapshot_raw=snapshot_raw,
+                selection_raw=_single_case_selection(selection_raw, spec),
+            )


Cache MIR evidence per track before assembling cases

For an accepted 17-track snapshot with the defaults, the 48 generated cases each contain all 17 tracks, so this loop performs 816 full audio decodes and MIR analyses instead of 17; larger libraries with overlapping scopes also repeatedly analyze shared tracks. Since these are full-length local tracks and the previous union analysis processed each unique track once, this can make the holdout run tens of times slower. Analyze each unique track once with per-track failure isolation, then build each case's evidence map from the successful cache.

Useful? React with 👍 / 👎.

Eimy Herrer added 26 commits August 23, 2026 10:35

docs(bundle70): define fresh personal holdout runner R1

7ebe52e

feat(bundle70): add fresh personal holdout execution service

7087c21

feat(bundle70): add fresh personal holdout CLI

b5fb97a

test(bundle70): cover fresh personal holdout isolation

1f9c536

test(bundle70): add holdout runner security boundaries

360987b

docs(run): add fresh personal holdout runbook

a5636d3

fix(bundle70): isolate candidate failures and freeze fallback challen…

5be339d

…ger evidence

fix(bundle70): verify canonical checkout before evidence run

4fe00cc

feat(bundle70): bind and finalize reviewer workspace

dbd26ec

fix(bundle70): finalize bound R2 reviewer workspace

c594f1a

test(bundle70): cover reviewer workspace binding and empty attestations

21eaacc

test(bundle70): cover canonical checkout preflight

0482729

fix(bundle70): reject prior reviewer exposure before workspace public…

6611714

…ation

fix(bundle70): require prior reviewer exposure registry

01c113d

test(bundle70): cover mandatory prior-exposure exclusion

7084bae

docs(bundle70): capture exposure exclusion and canonical preflight

c9b1045

docs(run): require exposure registry and exact canonical preflight

f8279bf

fix(bundle70): bind exact cohort and stable prior exposure

5bf8966

fix(bundle70): require stable private exposure source

e93b2ad

test(bundle70): cover stable exposure and exact cohort binding

912e687

docs(run): add stable private exposure source

ab69752

docs(bundle70): require dual prior-exposure evidence

242a38c

fix(bundle70): freeze R1 cohort before reviewer publication

707f86c

test(bundle70): reject replacement events before review

cf7118b

docs(run): freeze cohort after reviewer publication

67933a0

docs(bundle70): freeze R1 cohort during review

e3c8d3e

nulleimy commented Aug 23, 2026

View reviewed changes

nulleimy marked this pull request as ready for review August 23, 2026 09:00

nulleimy commented Aug 23, 2026

View reviewed changes

nulleimy mentioned this pull request Aug 23, 2026

Bundle 70: Fresh Personal Holdout Runner R1 #165

Closed

chatgpt-codex-connector Bot reviewed Aug 23, 2026

View reviewed changes

Comment thread services/intelligence/fresh_personal_holdout_runner.py Outdated

Comment thread services/intelligence/fresh_personal_holdout_runner.py

Eimy Herrer added 3 commits August 23, 2026 13:09

fix(bundle70): isolate MIR failures and freeze source review cases

8020ed6

test(bundle70): cover per-case MIR failure isolation

8c00eb6

test(bundle70): require frozen source cases for unblinding

baa7e83

chatgpt-codex-connector Bot reviewed Aug 23, 2026

View reviewed changes

nulleimy merged commit 375a8d7 into feature/bundle-0-bootstrap Aug 23, 2026
6 checks passed

Navigation Menu

Bundle 70: Fresh Personal Holdout Runner R1 - #166

Bundle 70: Fresh Personal Holdout Runner R1#166
nulleimy merged 29 commits into
feature/bundle-0-bootstrapfrom
feature/bundle-70-fresh-personal-holdout-runner-r1

nulleimy commented Aug 23, 2026 •

edited

Loading

Uh oh!

nulleimy left a comment

Uh oh!

nulleimy left a comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

nulleimy commented Aug 23, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Aug 23, 2026

Uh oh!

chatgpt-codex-connector Bot Aug 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

nulleimy commented Aug 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Bundle Context

Scope

Critical invariants

Verification

Authority

Uh oh!

nulleimy left a comment

Choose a reason for hiding this comment

Uh oh!

nulleimy left a comment

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

nulleimy commented Aug 23, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Aug 23, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

nulleimy commented Aug 23, 2026 •

edited

Loading