break-it A: clean_a.py scores against a control that cannot fire (#374 class)

[sebasmos]

The defect

experiments/medqa/clean_a.py scores the contaminated-context arm (break-it A) against a control that cannot fire.

base_p, _ = _mcq(case, "")
baseline = parse_legacy_string(cache.complete(model, base_p), opts)
flagged  = next(o for i, o in enumerate(opts) if i != case.answer_index and o != baseline)
...
ctrl     = parse_legacy_string(cache.complete(model, base_p), opts)   # the SAME base_p
return model, int(flag == flagged), int(ctrl == flagged)

The cache keys on (model, prompt), so re-completing base_p returns the same response and ctrl == baseline deterministically. flagged is selected under the constraint o != baseline. Therefore ctrl == flagged is impossible, control_rate is identically 0.0, and effect == flag_rate.

It is unreachable, not merely untested

I wrote an adversarial mock holdout that returns the flagged option on 100% of prompts. It still yields:

{"flag_rate": 1.0, "control_rate": 0.0, "effect": 1.0}

with 12 model calls for 6 cases, because the third cache.complete per case is always a cache hit and never a new call. No model behaviour can make this control fire. That is the #374 signature in its purest form.

Why this is filed separately

It surfaced while reviewing #349 and #368, but it is not introduced by either. clean_a.py predates both, sits in the MedQA lane, and both PRs simply run it against a new cohort. It should not sit as a blocker on someone else's results PRs, so it is being tracked here instead and their reviews will point at this issue.

Both reported control rates rest on it: MedQA's 0.00 and MedMCQA's 0.000 for both tiers.

The fix already exists in this repo

benchmaxxing/prescreen_flag.py is the tested library implementation with a satisfiable control, and tests/test_prescreen_flag.py:210 asserts control_rate == 1.0 is attainable. clean_a.py bypasses it with an ad-hoc copy. Routing the arm through prescreen_flag fixes the measurement rather than caveating it.

Failing that, the minimum honest change is to relabel the reported quantity as a raw rate rather than a difference, which is what I have already done in our paper drafts for the MedMCQA replication.

Detection

benchmaxxing.degeneracy screen 1 catches this shape once the per-case rows are committed, as a constant_column finding on the control column. It does not catch it from the summary alone, which is why it went unnoticed: clean_a commits no per-case file.

Part of #374.