CheXpert standard-battery replication (tracking)

[sebasmos]

Core battery (apples-to-apples, do these first; each maps to a cross-dataset table cell)

• Solo shortcut susceptibility + noise floor (image cues) (CheXpert) #304 Solo susceptibility + noise floor
• Cascade contagion: confident wrong read on a cued image (CheXpert) #305 Cascade contagion
• System-flag vs peer contagion (CheXpert) #306 Break-it A (system flag)
• Cue strength / natural-vs-injected cue sweep (CheXpert) #307 Break-it C (plausible seed)
• Blind-metric substitution (silent imaging gaming) (CheXpert) #308 Break-it D (blind-metric)
• Referee detection: naive gate vs targeted vs deployable (CheXpert) #309 Referee detection

---

Tracking issue: replicate the finished NIH ChestX-ray14 imaging standard battery on CheXpert (imaging, natural support-device cues) so imaging results are apples-to-apples across datasets. Real data + real API (multimodal Gemini) only; results under experiments/chexpert/results/, same format as experiments/imaging/results/.

Foundational (run first)

• Solo shortcut susceptibility + noise floor (image cues; flip-above-noise is the honest quantity)
• Cascade: shared-blackboard contagion of a confident wrong read (shared minus isolated)

Cue characterisation

• Cue families (watermark / corner-tag / cable / laterality, plus rotation/compression/brightness/soft-tissue) solo + cascade
• Polarity asymmetry (suppress a true finding vs hallucinate an absent one)
• Cue combinations / super-additivity (two cues at once)
• Strength x cascade (cue-strength sweep) + scale for tighter CIs

Committee structure

• System-flag vs peer contagion (does bare system authority transfer without a peer voice?)
• Majority / peer-size saturation

Referee

• Referee detection: naive gate vs targeted vs deployable (transcript + one private re-query)

Adapter/loader: build the loader (#46); Split any item into its own issue if you prefer; this tracker is the home.

[Yehudha-kennedy]

Hey @sebasmos, I just finished replicating the full imaging pipeline (solo susceptibility, cascade onset, and referee evaluation) on the CheXpert dataset (CheXpert-v1.0-small, N=35 cases) with Gemini 2.5 Flash.

Key Results:

• Solo Susceptibility: Watermark flip rate = 40.0% (flip-above-noise = +17.1%), Noise floor = 22.9%.
• Cascade Contagion: Shared adoption = 100.0% vs. Isolated adoption = 40.0% (Contagion = +60.0%), confirming the strong cascade effect matches ChestX-ray14.
• Referee Evaluation: Precision = 68.0%, Recall = 81.0%.

All 674 tests in pytest passed, ruff clean, and cached responses included for 100% offline reproducibility. I've opened the PR here: https://github.com/criticaldata/benchmaxxing/pull/new/feat/chexpert-replication (Closes #294, #304, #305, #309). Ready for review!

Side note -- potential improvements I noticed while working on this:
While replicating the pipeline, I noticed the _Cache class (the JSONL-backed API response cache) and the _yesno parser are copy-pasted across 50+ experiment scripts. If we extract them into a shared utility module, it would eliminate ~1,250 lines of duplicated code and make future bug fixes apply everywhere at once. I can open an issue to track this for after the workshop deadline if the team agrees. Just flagging it for now.