Imaging cascade plants against the model's clean read, not ground truth (NIH quantified, MIMIC to check)

[sebasmos]

Problem

The shared imaging-cascade code (experiments/imaging/imaging_cascade.py:116) sets wrong = "no" if clean == "yes" else "yes", i.e. it plants the opposite of the model's clean read. On finding-present cases (ground truth = "yes"), when the model's clean read is already wrong ("no"), the planted "wrong" read is actually the truth, so the case scores as contagion when the committee merely corrects the holdout. Agastya flagged this on the CheXpert scale-up (#331), where it dominated the result.

Test on NIH (committed transcripts, offline, no API)

Splitting each committed NIH cascade by whether the planted read was genuinely false (wrong=="no") vs actually the truth (wrong=="yes"):

cue	ALL	genuinely-false	contaminated (=truth)
watermark	+0.629	+0.636 (n=22)	+0.615 (n=13)
cable	+0.800	+0.773	+0.846
corner_tag	+0.743	+0.727	+0.769
laterality	+0.714	+0.727	+0.692

37% of cases plant the truth, but the genuinely-false subgroup shows the same contagion (~+0.73) as the overall. Unlike CheXpert (false +0.333 vs truth -0.038), NIH's harmful-sycophancy effect is real and not inflated by the contamination. The referee result is behavioral and unaffected.

Fix

• Plant against ground truth (a definitely-false read), not the clean read, in the shared imaging-cascade code; report the genuinely-false subgroup.
• Add a one-line robustness note to the paper (the genuinely-false subgroup confirms the NIH headline).
• @amarzullo24 MIMIC (MIMIC-CXR: deterministic subset tooling + imaging battery (closes #324) #329) reuses the same shared imaging-cascade path, so please run the same split on the MIMIC transcripts to confirm it is likewise robust (mixed findings should make it cleaner than CheXpert).
• CheXpert redesign (CheXpert: scale the imaging battery to ~600 solo / 150 cascade (match MIMIC) #331) already covers this via "plant against ground truth".

MedQA text is unaffected (it plants a distractor, i.e. against ground truth already). Reanalysis script: experiments/imaging/plant_direction_check.py.

[amarzullo24]

MIMIC plant-direction split (#329) -- offline reanalysis of the committed cascade transcripts (watermark):

transcript	ALL	genuinely-false	planted-truth	% truth
cascade_150 (#311)	+0.819 (n=215)	+0.488 (n=43)	+0.901 (n=172)	80%
referee_300 cascade	+0.808 (n=417)	+0.506 (n=87)	+0.888 (n=330)	79%

Unlike NIH (genuinely-false +0.636 ~ ALL +0.629, 37% truth), MIMIC is dirtier, not cleaner: 80% of cases plant the truth, because gemini-2.5-flash reads pneumothorax poorly here (clean read already wrong on most finding-present cases -- consistent with solo sitting at the noise floor). So ALL (+0.82) is inflated; the honest genuinely-false contagion is **+0.49-0.51** (real harmful sycophancy -- above CheXpert's +0.33, below NIH's +0.73).

Note: our subset is finding-positive only (all GT=yes), so the contamination is from low clean-read accuracy, not label mix -- the "mixed findings" expectation does not apply as-built.

@sebasmos -- the fix ("plant against ground truth" in the shared imaging_cascade.py) touches the NIH path too, so it needs a coordinated re-run on both lanes. Proposal: I patch the shared runner (finding-positive -> always plant "no") and re-run the MIMIC cascades; you re-run/confirm NIH so its committed numbers stay consistent. Work for you?

[sebasmos]

Ran it for real (PR #338): contagion moves from +0.63/+0.80/+0.74/+0.71 to +0.54/+0.54/+0.54/+0.57 under the corrected plant, still highly significant (McNemar p<2e-6 every cue). Paper updated. Didn't strictly need the rerun (the offline subgroup already showed this), but good to have it clean now. Same bug is in 6 other imaging scripts, opened #339 (low priority) for those. Your MIMIC number stands as the real correction needed.

[sebasmos]

Update: went ahead and fixed + re-ran the other 6 scripts too (#339), not just the cascade. All real. Summary in PR #338: system-flag +0.03->+0.17, the rest (strength/multi-round/peer-size/majority/scale) confirm their original conclusions at the corrected magnitude. #339 closes via #338.