Skip to content

eval: preregister fresh-to-refusal 50-case TRAIN comparison - #80

Merged
Air000000 merged 1 commit into
mainfrom
codex/refusal-prospective-train-prep
Oct 16, 2026
Merged

Air000000 merged 1 commit into
mainfrom
codex/refusal-prospective-train-prep

Conversation

@Air000000

Copy link
Copy Markdown
Owner

Scope

Freeze an inert prospective comparison, not another prompt: v5.2.1/v5.3 prompts and strict validator remain unchanged before any new labels. Reuse frozen450 TRAIN snapshot/results and full ordered Top14; exclude the prior selection's recorded exclusions/family exclusions/selected30, then normalized-question/leading-document proxy against old cases. Deterministic SHA256 seed priority, within-batch same-question/leading-document dedup, exactly50, no smaller fallback/post-label balancing. Eligible114 before within-batch dedup; all selected50 have distinct leading documents. No new retrieval, gold-based selection or DEV.

Interpretation

Fresh relative to recorded refusal selections/visible family proxy, NOT untouched project data: all TRAIN previously used in retrieval/automated analysis; leading-document proxy does not guarantee independent semantic families. Proposed Qwen drafts plus current-assistant preprediction evidence review remain AI proxy, not independent expert gold. Report actual class support; no label/sample changes to reach the old minimum-five/class rule. No formal acceptance/production gate, generated-answer accuracy or unseen DEV claim regardless of results.

Proposed two-stage authorization (NOT executed)

  1. Up to50 Singapore qwen3.6-plus-2026-04-02 annotation calls, max4096 output tokens, existing blind prompt. Assistant reviews questions/drafts/cited evidence with recorded depth, audits recoverable citation formatting without overwriting originals. Freeze all50 reviewed labels/counts/hashes before any classifier prediction. If defensible complete labels cannot be frozen, stop before classification.
  2. Up to100 fresh paired Singapore qwen3.5-plus-2026-04-20 calls,50 per arm, unchanged payload/schema/validator, max2048 output tokens. Fixed question order and alternating arm order. No target labels, review reasons, IDs, retrieval metadata or secrets in classifier requests.

Both stages use temperature0/thinking off/JSON, SDK retries0; at most150 calls total, no paid retries. Proposed stage ledger caps annotation CNY10, classification CNY15, total CNY25, no transfer/recharge. Planning estimates annotation CNY7.6092975 + classification CNY10.848336 = CNY18.4576335, using official International rates checked2026-10-16 and fixed FX7.5; byte/framing proxy is not certified tokens or an invoice guarantee. New isolated durable checkpoints, pending before POST, raw/usage before parsing, exclusive lock; saved-raw recovery without new calls, no repeated completions, unknown charge blocks resume. Classifier schema errors remain terminal null outcomes. New explicit authorization required for both data/model/stage/budget scopes.

Artifacts and verification

  • Reuse selection, full-context builders, annotation prompt and existing dependencies; no paid runner/client/checkpoint framework added.
  • Bind source/protocol/request/prompt/input/packet/previous-selection hashes. Only public aggregate metadata/QIDs/fingerprints; real questions/source bodies/blank annotation packets remain under ignored data/.
  • 317 offline tests with external-network guard and temporary DB passed; Ruff/in-memory compilation/diff checks passed.
  • Selection, unique-leading-document50, private payloads and preflight reproduce exactly; previous v5.3 contract/journal hashes unchanged.
  • Zero provider calls; no .env read, labels, predictions, DEV, relabeling, serving or primary dirty-checkout changes.

@Air000000
Air000000 merged commit ad5a22e into main Oct 16, 2026
1 check passed
@Air000000
Air000000 deleted the codex/refusal-prospective-train-prep branch October 16, 2026 12:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant