Repository navigation
eval: preregister fresh-to-refusal 50-case TRAIN comparison - #80
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scope
Freeze an inert prospective comparison, not another prompt: v5.2.1/v5.3 prompts and strict validator remain unchanged before any new labels. Reuse frozen450 TRAIN snapshot/results and full ordered Top14; exclude the prior selection's recorded exclusions/family exclusions/selected30, then normalized-question/leading-document proxy against old cases. Deterministic SHA256 seed priority, within-batch same-question/leading-document dedup, exactly50, no smaller fallback/post-label balancing. Eligible114 before within-batch dedup; all selected50 have distinct leading documents. No new retrieval, gold-based selection or DEV.
Interpretation
Fresh relative to recorded refusal selections/visible family proxy, NOT untouched project data: all TRAIN previously used in retrieval/automated analysis; leading-document proxy does not guarantee independent semantic families. Proposed Qwen drafts plus current-assistant preprediction evidence review remain AI proxy, not independent expert gold. Report actual class support; no label/sample changes to reach the old minimum-five/class rule. No formal acceptance/production gate, generated-answer accuracy or unseen DEV claim regardless of results.
Proposed two-stage authorization (NOT executed)
Both stages use temperature0/thinking off/JSON, SDK retries0; at most150 calls total, no paid retries. Proposed stage ledger caps annotation CNY10, classification CNY15, total CNY25, no transfer/recharge. Planning estimates annotation CNY7.6092975 + classification CNY10.848336 = CNY18.4576335, using official International rates checked2026-10-16 and fixed FX7.5; byte/framing proxy is not certified tokens or an invoice guarantee. New isolated durable checkpoints, pending before POST, raw/usage before parsing, exclusive lock; saved-raw recovery without new calls, no repeated completions, unknown charge blocks resume. Classifier schema errors remain terminal null outcomes. New explicit authorization required for both data/model/stage/budget scopes.
Artifacts and verification