R4 Pilot Run 1: real DJ evidence collection

[nulleimy]

Purpose

Execute the first governed real-DJ pilot against canonical APPLAYLIST after Bundle 64.

This is an operational evidence run, not a feature bundle and not an authorization for merge, release, deploy, production activation, optimizer/ranking activation, or Personal DJ Model training.

Canonical checkpoint

• branch: feature/bundle-0-bootstrap
• canonical SHA: b8ad056e34be27db5ff2dd09e71814e01e2c70bf
• Bundle 63 Human DJ Review Execution R1: merged
• Bundle 64 R4 Bounded DJ Pilot Evidence R1: merged

Pilot objective

Answer with genuine DJ evidence:

1. Does APPLAYLIST reduce time from import to a usable set?
2. Do DJs inspect recommendations?
3. Do they accept/reject them rather than ignore them?
4. How often must they reorder, replace, or lock tracks?
5. Can they complete export successfully?
6. Do they return in a later ISO week?
7. Would they pay?
8. Do they trust and understand the system?
9. Does blinded Human DJ Review evidence support continuing the current intelligence direction?

No answer may be inferred from CI, test fixtures, or synthetic submissions.

Required real inputs

A. Bundle 63 Human DJ Review

Private/trusted local inputs:

• APPLAYLIST_REAL_LIBRARY_RUNTIME_EVIDENCE_R1.private.json
• APPLAYLIST_BLINDED_HUMAN_DJ_REVIEW_PACKET_R1.json

Reviewer artifact:

• APPLAYLIST_HUMAN_DJ_REVIEW_WORKSPACE_R1.html

Per real reviewer:

• reviewer reference
• blinded Plan A/B review
• all six required dimensions
• confidence
• observed timestamp
• algorithm_identity_was_hidden=true

Append-only ledger:

• APPLAYLIST_HUMAN_DJ_REVIEW_R1.sqlite

Aggregate:

• APPLAYLIST_HUMAN_DJ_REVIEW_AGGREGATE_R1.json

B. Bundle 64 workflow evidence

Append-only ledger:

• APPLAYLIST_R4_PILOT_EVIDENCE_R1.sqlite

Required real event vocabulary:

• import_started
• usable_set_reached
• recommendation_presented
• recommendation_inspected
• recommendation_accepted
• recommendation_rejected
• manual_reorder
• manual_replace
• manual_lock
• export_completed

C. Explicit human exit survey

For each participating DJ where possible:

• willingness_to_pay = yes | no | unsure
• optional stated monthly price + explicit currency
• trust score 1–5
• explainability score 1–5
• optional bounded note/reason codes

Local execution commands

Generate blinded reviewer workspace

python scripts/applaylist_human_dj_review.py workspace \
  --packet APPLAYLIST_BLINDED_HUMAN_DJ_REVIEW_PACKET_R1.json \
  --output APPLAYLIST_HUMAN_DJ_REVIEW_WORKSPACE_R1.html

Ingest one genuine reviewer submission

python scripts/applaylist_human_dj_review.py ingest \
  --packet APPLAYLIST_BLINDED_HUMAN_DJ_REVIEW_PACKET_R1.json \
  --submission reviewer-01.json \
  --ledger APPLAYLIST_HUMAN_DJ_REVIEW_R1.sqlite \
  --receipt reviewer-01.receipt.json

Build Human DJ Review aggregate

python scripts/applaylist_human_dj_review.py report \
  --packet APPLAYLIST_BLINDED_HUMAN_DJ_REVIEW_PACKET_R1.json \
  --private-manifest APPLAYLIST_REAL_LIBRARY_RUNTIME_EVIDENCE_R1.private.json \
  --ledger APPLAYLIST_HUMAN_DJ_REVIEW_R1.sqlite \
  --output APPLAYLIST_HUMAN_DJ_REVIEW_AGGREGATE_R1.json

Record a real workflow event

python scripts/applaylist_r4_pilot.py event \
  --ledger APPLAYLIST_R4_PILOT_EVIDENCE_R1.sqlite \
  --input event.json \
  --receipt event.receipt.json

Record explicit exit survey

python scripts/applaylist_r4_pilot.py survey \
  --ledger APPLAYLIST_R4_PILOT_EVIDENCE_R1.sqlite \
  --input survey.json \
  --receipt survey.receipt.json

Build R4 pilot report

python scripts/applaylist_r4_pilot.py report \
  --ledger APPLAYLIST_R4_PILOT_EVIDENCE_R1.sqlite \
  --human-review-aggregate APPLAYLIST_HUMAN_DJ_REVIEW_AGGREGATE_R1.json \
  --output APPLAYLIST_R4_PILOT_REPORT_R1.json

Pilot boundaries

• local-first evidence only
• no analytics SaaS
• no hidden/background telemetry
• no cloud upload of review/audio/private runtime evidence without separate authorization
• no fabricated reviewer ratings
• no synthetic workflow evidence presented as product evidence
• no identity unblinding in reviewer workspace
• no destructive ledger edits
• no implicit currency conversion
• no PDM training
• no optimizer/ranking activation
• no production activation

Readiness / decision semantics

INCOMPLETE until all required product metric categories are assessable and Bundle 63 review evidence is complete.

READY_FOR_HUMAN_DECISION means evidence is sufficiently complete for a separate governed human decision. It does not mean GO.

Separate later decision vocabulary:

• CONTINUE
• ADJUST
• STOP-PIVOT

STOP conditions

Stop the run and preserve evidence if:

• algorithm identity becomes exposed during blinded review,
• packet/private-manifest fingerprint binding fails,
• event correlation fails,
• immutable ledger rejects a conflicting retry,
• private paths/evidence are about to be published,
• participant evidence would need to be fabricated to complete a metric,
• an operation would require production/release/deploy authority.

Completion gate

This issue closes only after genuine evidence exists and the generated R4 report has been inspected. Until then the correct product state is INCOMPLETE.

RELEASE_AUTHORIZATION=NO
DEPLOY_AUTHORIZATION=NO
PRODUCTION_EFFECTS=NO
PDM_TRAINING=NO

[nulleimy]

Execution checkpoint discovered

Library preflight found the governed 55B proof/recovery tooling and the recovered canonical snapshot/cases, but not the generated private manifest or blinded packet as persisted ChatGPT Library files.

The latest known local proof root encoded by the recovery/diagnostic tooling is:

$HOME/00_DEV/APPLAYLIST-EVIDENCE/real-library-55b-proof-r1/20260817T074454Z

Expected local inputs beneath/alongside that checkpoint include:

• $PROOF_ROOT/APPLAYLIST_REAL_LIBRARY_ANALYSIS_EVIDENCE_R1.sqlite3
• $PROOF_ROOT/blinding_seed.private.txt
• a completed warm/APPLAYLIST_REAL_LIBRARY_RUNTIME_EVIDENCE_R1.private.json
• a completed warm/APPLAYLIST_BLINDED_HUMAN_DJ_REVIEW_PACKET_R1.json
• $HOME/Downloads/APPLAYLIST_RECOVERED_CANONICAL_SNAPSHOT_R1.json
• $HOME/Downloads/APPLAYLIST_RECOVERED_CANONICAL_CASES_R1.json

The recovered snapshot contains real audio paths under /Volumes/EXTRAZALOHA/..., so any re-materialization that needs actual audio must execute locally with that volume available.

Current canonical product SHA is now b8ad056e34be27db5ff2dd09e71814e01e2c70bf; old proof scripts pin earlier SHAs and must not be rerun unchanged as current-canonical proof.

Immediate governed local action

On the current canonical checkout, first validate the existing warm blinded packet/private manifest with the Bundle 63 tooling. If they validate, reuse them rather than rerunning expensive audio analysis. Then generate a fresh local reviewer workspace and begin genuine blinded reviews. If validation fails, stop and re-materialize from the recovered snapshot/cases under the current canonical code.

No human ratings are to be synthesized. No release/deploy/production/PDM authority is granted.

[nulleimy]

Pilot Run 1 path correction after local preflight on canonical b8ad056:

The assumed legacy path $PROOF_ROOT/warm/APPLAYLIST_BLINDED_HUMAN_DJ_REVIEW_PACKET_R1.json is not authoritative and was locally confirmed missing.

Authoritative verified R2 root from prior independent verification:
$HOME/00_DEV/APPLAYLIST-EVIDENCE/real-library-55b-proof-r1/20260817T074454Z/r2-independent-verification-and-blinded-packet/20260818T145225Z

Expected immutable inputs there:

• APPLAYLIST_REAL_LIBRARY_RUNTIME_EVIDENCE_R2.private.json
  SHA256 03a9b3c71d32647cb97f3203f6782ab19fb6e7bcc4f8c14f4c48e4294da4d569
• APPLAYLIST_BLINDED_HUMAN_DJ_REVIEW_PACKET_R1.json
  SHA256 8fa3454ae1ef0c6cc8da0f7059a67b3c3f305ca32f4d047b68cd45b642d94e12
• APPLAYLIST_R2_INDEPENDENT_VERIFICATION_REPORT.json
  SHA256 0a5a00878caa93260b420d7e23953050623450168f723b29974ab68e0931b997

A prior blinded local audio workspace was designed at:
$VERIFY_ROOT/human-dj-review-r1/20260818T151935Z
with 12 cases, PLAN_A/PLAN_B M3U8 files, algorithm identity hidden, and an explicit CSV review form.

Next gate is local existence/hash/path preflight of these immutable artifacts and, if the workspace exists, verification that all M3U8 source paths are still readable. No human ratings may be fabricated; no release/deploy/activation/PDM authority.

[nulleimy]

R4 Pilot continuation after Bundle 65

Canonical is now bb85ee2e73a4a439ebeab820e2ff024e0ff5e049 (Bundle 65 merged).

Runtime preflight found an operational provenance blocker: the Bundle 65 real-library wrapper calls analyze_real_tracks(...), which would hash/decode and re-run MIR over the already-complete historical R2 selection. We are not regenerating that evidence.

Governed follow-up Issue #153 / PR #154 adds Historical Meaningful Review Replay R1: evaluate Bundle 65 against the existing snapshot + curated selection + private R2 runtime manifest only, with zero audio reads/provider calls and one new immutable replay report.

Until that replay path passes CI and is separately merge-authorized, R4 Pilot Run 1 remains INCOMPLETE. No release/deploy/production/optimizer/PDM authority.

[nulleimy]

Human Review Run 1 — STARTED

Canonical APPLAYLIST checkpoint for this operational run:

• branch: feature/bundle-0-bootstrap
• canonical SHA: efd6207fbbc9265773fd113a5450a0aaf53f3149
• Bundle 68 Human Preference Calibration R2: merged/canonical

This is an operational evidence run only. No feature bundle is being opened.

Required genuine reviewer evidence

Each of the 12 blinded cases must contain all six Bundle 63 dimensions for both Plan A and Plan B:

1. transition_smoothness
2. phrase_alignment
3. energy_flow
4. dramaturgical_fit
5. set_coherence
6. alternative_usefulness

Each dimension score must be 1..5. Each case also requires:

• preference = plan_a | plan_b | tie | abstain
• confidence = 0..1
• observed timestamp
• algorithm_identity_was_hidden=true

Qualitative track-selection notes already collected remain a separate evidence stream and MUST NOT be converted into missing numeric 6D ratings.

Fail-closed boundaries

• no fabricated ratings
• no algorithm identity unblinding during review
• no private manifest inspection while judging A/B
• no MIR/audio regeneration merely to complete review
• no optimizer/ranking activation
• no PDM training
• no release/deploy/production activation

Current execution state

REAL_R4_HUMAN_PREFERENCE_CALIBRATION=IN_PROGRESS

Completion requires genuine complete 6D review evidence for all required cases, ingestion into the append-only human-review ledger, aggregate generation, then Bundle 68 calibration against the competitive challenger.

Until that evidence exists, any calibration/product verdict remains INCOMPLETE.

[nulleimy]

Skill Tester gate — R4 review PAUSED

Independent adversarial QA found the current Human DJ Review / Calibration protocol experimentally invalid for curation calibration:

• reviewer workspace requires transition_smoothness and phrase_alignment but provides no standardized transition execution recipe or rendered reference;
• human overall A/B preference can therefore be influenced by uncontrolled DJ execution;
• Bundle 68 compares that single preference directly against the Bundle 67 curation challenger, contaminating curation ground truth.

Governing blocker: #159 Skill Tester blocker: separate curation review from transition execution.

Current state:

• existing qualitative track-selection review remains valid as separate curation evidence;
• do not fabricate missing 1..5 transition ratings;
• pause the 12-case full-6D calibration run until curation review and transition execution are separated or transition execution is standardized;
• no optimizer activation / release / deploy / production / PDM authority.

R4_HUMAN_CALIBRATION_PROTOCOL_VALIDITY=FAIL
R4_HUMAN_CALIBRATION=PAUSED

[eimyroot]

Canonical correction after Bundle 70 audit

Current canonical branch is feature/bundle-0-bootstrap at 375a8d7b53492af2a4c9f07983f31c4bce8cc8ff (Bundle 70 merge commit). Independent compare against the branch is identical / 0 ahead / 0 behind.

The older Bundle 62 checkpoint bba282136590be91e8910aa0bc49cdd4402acc63 is no longer current. Bundle 63 Human DJ Review Execution R1 already exists and was merged through PR #146; it must not be recreated as a parallel review layer.

Bundle 69 subsequently isolated curation review from transition feasibility / human execution and explicitly preserved historical Bundle 63/R1 code and evidence as immutable. Bundle 70 is now the canonical execution boundary for a fresh personal blinded curation holdout.

Governed next operational step

Use docs/runs/FRESH_PERSONAL_HOLDOUT_RUN_R1.md and scripts/applaylist_fresh_personal_holdout.py on a local clean checkout at exact canonical SHA 375a8d7b53492af2a4c9f07983f31c4bce8cc8ff.

Before Case 1, fail closed unless:

• local branch is exactly feature/bundle-0-bootstrap at the canonical SHA;
• working tree is clean;
• private local-library snapshot and audio are readable;
• historical reviewer-visible and stable-ID exposure sources are supplied for exclusion;
• fresh effective cohort is 24 cases, four per set role;
• reviewer packet contains only R2 curation dimensions and no transition execution request;
• challenger evidence is frozen privately and absent from reviewer-visible output;
• all human-controlled review/attestation fields remain empty at freeze.

No genuine human judgement, pilot PASS, optimizer activation, PDM training, release, deploy, or production activation is claimed by this checkpoint.

R4_PILOT_RUN_1=INCOMPLETE
OPTIMIZER_RANKING_ACTIVATION=NO
PDM_TRAINING=NO
RELEASE_AUTHORIZATION=NO
DEPLOY_AUTHORIZATION=NO
PRODUCTION_ACTIVATION=NO

[eimyroot]

Fresh Personal Holdout Run R1 — execution authorization checkpoint

Live canonical verification on 2026-08-30 confirms feature/bundle-0-bootstrap still points exactly to 375a8d7b53492af2a4c9f07983f31c4bce8cc8ff.

Execution state:

• FRESH_PERSONAL_HOLDOUT_RUN_R1=AUTHORIZED_TO_EXECUTE_LOCALLY
• CASE_1_GO=NO
• R4_PILOT_RUN_1=INCOMPLETE
• OPTIMIZER_RANKING_ACTIVATION=NO
• RELEASE_AUTHORIZATION=NO
• DEPLOY_AUTHORIZATION=NO
• PRODUCTION_ACTIVATION=NO
• PDM_TRAINING=NO

Required pre-Case-1 evidence remains fail-closed:

1. exact canonical branch/SHA and clean local tree;
2. valid private applaylist-local-library-snapshot-r1 with readable local audio;
3. historical reviewer packet + private manifest exposure sources verified;
4. finalized 24-case cohort, four cases per each of six set roles;
5. zero replacement events before reviewer publication;
6. dual prior-exposure exclusion and binding checks PASS;
7. reviewer packet contains only the four R2 curation dimensions and no private/model leakage;
8. all human-controlled review fields empty at freeze;
9. artifact SHA-256/fingerprints captured.

Do not open Case 1 until the local runner result and independent stop-gate verification are attached/audited. This checkpoint authorizes only the bounded local evidence run, not human-review PASS or any activation authority.