Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: aca2eacc54
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if review.review_id in review_ids: | ||
| raise CurationPreferenceCalibrationV3Error("duplicate curation review identity") | ||
| review_ids.add(review.review_id) |
There was a problem hiding this comment.
When the same assignment is supplied multiple times with distinct review_id values, every copy is accepted and included in the exact, decisive, and confidence-weighted agreement denominators. A caller can therefore duplicate a favorable review enough times to change DOES_NOT_SUPPORT_ACTIVATION into SUPPORTS_FURTHER_EVALUATION; track reviewed assignment IDs and reject duplicates before appending calibration evidence.
Useful? React with 👍 / 👎.
| case_ids = tuple(item.case_id for item in cases) | ||
| scenario_fingerprints = tuple(item.scenario_fingerprint for item in cases) | ||
| if case_ids != manifest.selected_case_ids: | ||
| raise CurationPreferenceCalibrationV3Error("holdout selected case identities changed after freeze") | ||
| if scenario_fingerprints != manifest.scenario_fingerprints: | ||
| raise CurationPreferenceCalibrationV3Error("holdout scenario fingerprints changed after freeze") |
There was a problem hiding this comment.
If supplied cases retain the frozen case IDs and scenario fingerprints but have a different snapshot_ref, these checks still accept them and the report is emitted with independent_validation=True and representative claims allowed. Since the manifest explicitly freezes source_snapshot_ref, validate every case's snapshot reference against it here so evidence generated from a later or different library snapshot cannot pass as the frozen holdout.
Useful? React with 👍 / 👎.
| a_greedy, b_greedy, a_beam, b_beam = _assignment_position_counts( | ||
| cases_by_id=cases_by_id, | ||
| assignments=assignments, | ||
| ) | ||
| assignment_imbalance = abs(a_greedy - b_greedy) |
There was a problem hiding this comment.
For multi-DJ evaluations this computes only a batch-wide imbalance, so opposing biases cancel: one reviewer can receive every greedy plan in A and another every greedy plan in B while the aggregate imbalance is zero and completeness passes. Because slot placement is reviewer-specific and position bias acts within each reviewer's judgments, calculate the imbalance per reviewer and fail when any reviewer exceeds the policy limit.
Useful? React with 👍 / 👎.
1 participant
Closes #160
Governing Skill Tester blocker/design: #159
Bundle Context
feature/bundle-0-bootstrapf3292d6ac3db81e9d741eedfd2467ecc90c933c0aca2eacc54583e6aa9152e3756abc996c8288052Summary
Implements the Skill Tester-approved Protocol V2.4 design that separates curation judgment from uncontrolled transition execution and adds development/holdout-safe calibration semantics.
Exact scope
sequence_curation_onlyaudition modeSkill Tester attacks closed before Ready-for-Review
The existing R2 12-case set remains
development_calibrationevidence only. It is not independent holdout validation.Verification — exact HEAD
aca2eacc54583e6aa9152e3756abc996c8288052CI
CI #939 (
32623400807): SUCCESS512 passed, 6 warnings94890370415f4a45bd4c6a764ae72786a41c5638a403201bf7549a64505ef1dbc286c6d641512 passed, 5 warnings9489037043da3d73e8cf3246028a50411da2c91e10727d16da5c835f8f5c2c41d6e0c7e5a2PR Guard #338: SUCCESS on exact HEAD.
Earlier failures preserved as evidence
Bundle Context; metadata was corrected without bypassing the guard.development_evidence_not_independent_validation, while only successful development evaluation recommends a fresh holdout.Exact diff / source authority
Compare vs canonical base:
f3292d6ac3db81e9d741eedfd2467ecc90c933c0Canonical authority files remain byte-identical between base and feature branch:
services/intelligence/set_engine.pyblobdd8f0cdb672202289530579d092cf0feb5cedf25services/intelligence/set_path_optimizer.pyblob819f6399181f4b37bb26ed8ccefb6cc0f4a9607fservices/intelligence/human_dj_review_execution.pyblob195a4f5f6ef62421be07413807072244f48c4b57services/intelligence/competitive_curation.pyblobfaadcae932e5f7b907503cbf2cd5bfdaab36462eservices/intelligence/human_preference_calibration.pyblob61899fda033c5d77513a780602c227ccabe36b79No Set Engine, path optimizer, TransitionAssessment, legacy Bundle 63 execution, Bundle 67 challenger, or Bundle 68 calibration implementation was modified.
Security / privacy / experiment validity
sequence_curation_onlyOperational holdout constraint
independent_validation=trueis protocol-scoped evidence, not laboratory replication. A real holdout must additionally persist an immutable freeze receipt before the first human label is collected. The pure in-memory protocol does not claim to prove wall-clock ordering by itself.Current product evidence state
Authority
MERGE_AUTHORIZATION=NO
OPTIMIZER_RANKING_ACTIVATION=NO
RELEASE_AUTHORIZATION=NO
DEPLOY_AUTHORIZATION=NO
PRODUCTION_ACTIVATION=NO
PDM_TRAINING=NO
Merge requires separate explicit
MERGE GO.