SUPPORT2: deployable referee needs an honest-peer clean control to be measurable (#374)

[sebasmos]

The gap

SUPPORT2's deployable-referee cell in Table 2 is a dash, and it is a dash for a reason that is now fixable rather than fundamental.

On the planted arm alone the peers are scripted to assert the shortcut and wrong is chosen to differ from the holdout's bare answer, so deployable, oracle and adopted reduce to the same expression. deployable == adopted on 120 of 120 rows, and the reported 1.0/1.0/0.0 is arithmetic rather than a measurement. That is why the paper withdraws it, and #391 tracks the related flip-seed case.

The fix is one extra arm, and it now has two working references

Add an honest-peer clean control: run the same committee with both peers asserting the correct answer. Shortcut adoption is then false by construction, so a deployable flag there is a genuine false positive, because the referee has no key with which to tell honest peer correction from being lured onto a wrong answer.

This has now been done twice, independently, with consistent results:

lane	planted only	with clean control	false positives
MedMCQA (#368)	1.000/1.000/0.000	0.742/1.000/0.140	8 of 57
MedQA (#397)	1.000/1.000/0.000	0.682/1.000/0.108	7 of 65

So the answer on this lane is likely to land near 0.7 precision with roughly a tenth false-positive rate, and the gap from the oracle is the real price of having no answer key. That is a reportable number where a dash is not.

#397 is the closest reference implementation: it is the same experiments/referee/referee_deployable.py template, and it shows the whole shape, adding a cascade(peer_answer) helper called twice, emitting an arm field per row, and reporting both a _DEGENERATE planted-only block and the honest with_clean_control one so the contrast is visible in the artifact itself.

Cost

Cheap. The clean arm needs one board run per case plus the holdout's response, and the private re-query is already cached from the planted arm. On MedQA that was 80 new calls for 40 cases. SUPPORT2 at n=120 should be roughly 240.

Two caveats to carry into the result, both learned from the other two lanes

• Recall stays 1.0 by construction even with the clean control, because every positive in the matrix is a planted row and the clean arm contributes none. Report it as such rather than beside the corrected precision as though all three were measured.
• The oracle column stays definitional in both blocks, since its clean-arm verdict is assigned rather than measured. Label it an upper bound.

Related

• SUPPORT2: same-lineage judge detector never run (table gap, same class as #393) #395: the same-lineage judge has never been run on this lane either, which is the other dash in the same column.
• SUPPORT2 flip-seed reports four restatements of one number as four results (#374 class) #391: the flip-seed arm's four restatements of one number.
• results(#374): honest-peer clean control for the MedQA deployable referee, so the cell is measurable #397 and results: MedMCQA committee battery — seed shape, social pressure, hierarchy, controls, referee + corrected family #368: the two reference implementations.