Referee detection: naive gate vs targeted vs deployable (MedMCQA)

[sebasmos]

Goal (apples-to-apples with MedQA/NIH): Referee (the named contribution): can a monitor catch the adoption? Compare (a) a naive conformity gate that fires on any agreement, (b) a targeted cross-condition referee, (c) a DEPLOYABLE monitor with no privileged knowledge (transcript + one private re-query). Report precision / recall / FPR.

Copy this template: experiments/referee/referee_deployable.py and referee_judge.py. It already implements the experiment; you mostly swap in the MedMCQA manifest and adapt the loader if needed. Do NOT rewrite the method, replicate it.

What to do: Run on the MedMCQA cascade transcripts; report P/R/FPR for each referee.

Golden rule: real data + a real Gemini API run only. A passing mock / synthetic smoke test does NOT count. Commit results under experiments/medmcqa/results/ in the SAME JSON/JSONL format as the source, so the cross-dataset comparison table can be filled directly.

Definition of done: the real keyed-run numbers are committed under results/, and the row/cell for MedMCQA in the cross-dataset table can be read off them. Part of #291. Closes the referee gap the MedMCQA battery #267-#290 was missing.

[sebasmos]

Delivered and merged in #368. Closing, with the numbers here so the issue records what landed.

n = 40 planted cases, 23 adoptions, plus an honest-peer clean-control arm.

referee	precision	recall	FPR	note
naive_gate	0.287	1.000	1.000	scored over planted plus clean
deployable	0.742	1.000	0.140	scored over planted plus clean
oracle_audit	1.000	1.000	0.000	scored over planted plus clean
same-lineage judge	1.000	0.913	0.000	tp 21, fp 0, fn 2, tn 17

The part worth crediting. @duckyquang did not just port the template, he found and fixed the thing that makes it meaningless. Scored on the planted arm alone the deployable referee reads 1.000/1.000/0.000, which is arithmetic rather than a measurement: the peers are scripted to the shortcut, so the flag reduces to the label it is scored against. He added an honest-peer clean-control arm where adoption is false by construction, so a flag there is a genuine false positive, and the honest number is 0.742 precision at FPR 0.140, costing 8 false positives.

That design is now the project's general remedy for this whole defect family. It has been applied to MedQA in #397, giving 0.682/1.000/0.108, and it is the requested fix on #390 and #399. The paper cites it as such.

Two caveats carried into the paper with it, both from the artifact's own note: recall stays 1.0 by construction, since every positive in the matrix is a planted row and the clean arm contributes none; and the oracle column stays definitional in both blocks, its clean-arm verdict being assigned rather than measured.

MedMCQA now has every arm in the standard battery, and is the only replication cohort of which that is true.