MIMIC-CXR imaging: no committed runner script for cascade/solo/system_flag/blind_metric (mirrors #337's CheXpert gap)

[sebasmos]

Problem

experiments/mimic_cxr_image/ has no committed script that actually calls the Gemini API to generate the cascade/solo/system_flag/blind_metric results. The only committed .py files there are build_subset.py (deterministic manifest/subset selection, no API calls) and plant_direction_check.py (an offline re-analysis that reads already-generated jsonl files, also no API calls).

The committed result files under experiments/mimic_cxr_image/results/ (e.g. cascade_150/imaging_cascade_summary.json, imaging_blind_metric_summary.json, imaging_system_flag_summary.json, imaging_strength_cascade_summary.json) were therefore produced by a script that either never got committed, or was run and then removed. Nobody looking at this repo can regenerate these numbers from what is here, and the code that was actually used to produce them cannot be audited for correctness (see below).

This is the exact same gap flagged on the CheXpert redesign in #331/PR #337 (also missing a committed runner), except CheXpert's gap is being actively worked on and MIMIC's is not yet tracked anywhere.

Why this matters for the paper

1. Reproducibility. The project's standing rule is that every real-data claim reproduces from a committed cache with no API key required. That is impossible here since the generating script itself is absent, not just its cache.
2. The ground-truth-plant fix (Imaging cascade plants against the model's clean read, not ground truth (NIH quantified, MIMIC to check) #332/Imaging cascade: plant against ground truth, not the clean read (#332) #333/Imaging: ground-truth plant fix, real re-run across all 7 affected scripts (#332, #339) #338/Audit the other 6 imaging scripts for the same plant-against-clean-read pattern (#332 follow-up) #339) cannot be verified as applied. PR Imaging cascade: plant against ground truth, not the clean read (#332) #333/Imaging: ground-truth plant fix, real re-run across all 7 affected scripts (#332, #339) #338 fixed a real bug in the shared experiments/imaging/imaging_cascade.py (and 6 sibling scripts): the old design planted wrong = flip(clean_read), which on finding-present cases coincides with ground truth whenever the model's own clean read was already wrong, understating what a genuine correction looks like versus real sycophancy. Aldo's plant_direction_check.py (MIMIC-CXR: deterministic subset tooling + imaging battery (closes #324) #329) is a valid, honest retrospective correction of the existing committed data (splits already-collected transcripts into genuinely-false vs coincides-with-truth subgroups), but it cannot confirm the next run (e.g. any rerun toward n=600, or any new cue/arm) would use the fixed design, because there is no committed generator to inspect or fix going forward.

What "done" looks like

• Commit the script(s) that actually drive GeminiBackend/cache.ask calls for each MIMIC-CXR imaging arm (solo, cascade, system_flag, strength_cascade, blind_metric), under experiments/mimic_cxr_image/, mirroring the shape of the already-fixed experiments/imaging/*.py scripts.
• The cascade runner must plant against ground truth (the definitely-false read), not against the model's own clean read, matching the Imaging cascade: plant against ground truth, not the clean read (#332) #333/Imaging: ground-truth plant fix, real re-run across all 7 affected scripts (#332, #339) #338 fix. Do not reintroduce the old wrong = "no" if clean == "yes" else "yes" pattern.
• A README or docstring documents the exact command to regenerate every arm from a fresh manifest, so plant_direction_check.py's inputs are themselves reproducible, not just its outputs.
• Re-verify (or re-derive) that the currently-committed result summaries match what the committed runner actually produces, since they may currently be stale relative to any local-only version of the script.

Real-data rule

Any fix here needs a real Gemini API run to confirm the committed runner reproduces the existing (or corrected) summary numbers, not just a code read. Mock/offline verification alone does not satisfy this project's standing rule.

Related: #331, #332, #333, #338, #339, PR #329, PR #337.

[Agastya191]

PR #347 covers the first three checklist items. It commits run_battery.py as the binding from the per-arm manifests to the shared, already-fixed experiments/imaging/ runners, rather than forking five copies into this directory (the fork is what gave #332 seven files to fix, and #331's v2 refactor went the other way for the same reason). The runner refuses to launch any cascade-family arm still planting against the clean read, so the second item is enforced rather than assumed.

Verifying the documented recipe turned up three defects worth flagging on their own: the unpinned --cache sends MIMIC reads into the NIH lane's committed cache files, --n defaults to the NIH arm sizes so an 834-image arm can silently run 35, and imaging_system_flag's vs_peer_assertion_cascade block has been silently dropping out since the cascade moved to its own --out in b8b4905.

The fourth item has a definite answer: the committed cascade-family summaries cannot match the fixed runner. plant_direction_summary.json records ~80% of cases planting the truth, which is impossible under wrong = "no" on an all-finding-present cohort, so five of the nine summaries are pre-fix. The PR documents the affected set and the genuinely-false subgroup as a floor. The real-data rerun is still open and asked of @MohShahin there, subject to his PhysioNet access covering the imaging release.

Note the PR bases on #329 (feat/mimic-subset), the only branch where experiments/mimic_cxr_image/ exists, so its Closes #343 will not auto-close on merge.

[armaanvgrewal]

@MohShahin looks from your whatsapp post you want to take this on, so I'll skip this and move to another one.

[MohShahin]

@MohShahin looks from your whatsapp post you want to take this on, so I'll skip this and move to another one.

Yes, I offered to help with this on WhatsApp. At the moment, I'm still waiting for and trying to run my priority task #297, as I was resolving the dependency needed for it.

In the meantime, I'm happy to support with this as well. Just to set expectations, this task involves processing a very large volume of patient data, so each run can take several hours to complete, even when everything works correctly on the first attempt.

It's currently running in the background, so fingers crossed everything completes successfully. I'll keep you posted as soon as I have an update.

[Agastya191]

@MohShahin on task #297 would you like to coordinate our efforts? I just started looking into it and would be happy to know where you are at on it right now

[armaanvgrewal]

@MohShahin got it! I bet it's the images that usually take longer. If there's a clean way to collaborate on this, I'm happy to. I'm working on another one in the meantime.

[MohShahin]

Thanks @Agastya191! I think the best way to coordinate would be for you to start implementing the fix for #297 while I focus on getting the real data run completed.

I'm currently in the middle of a large MIMIC-CXR imaging run, and most of the issues I've been working through are related to making sure the data is set up correctly before launching the experiments, verifying that the proper subsets are being queried, that the manifests and inputs are aligned with the code, and generally ensuring the run proceeds smoothly without wasting several hours on an incorrect configuration.

As soon as this imaging run finishes, I'll switch over to rerunning the data for #297 with the updated implementation and verify the results on the real dataset.

That way we can make progress in parallel instead of blocking on the long-running experiments. I'll keep everyone updated once the current run completes or if I hit anything that changes the timeline.

[amarzullo24]

@MohShahin quick heads-up before your rerun, so nih_match comes out right: that arm had a sizing bug (35 studies expanded to 50 images vs NIH's n=35 one-per-case), now fixed on feat/mimic-subset @ d3331d0 -- it's sized by image count (exactly 35 images, one-per-observation).

So please run on feat/mimic-subset at that commit (or, since the manifests are gitignored, regenerate them first with build_subset select --seed 20240724 using the updated build_subset.py) -- otherwise run_battery.py's nih_match_solo arm would regenerate the old 50-image version. Everything else in the battery is unaffected. Thanks, and shout if anything in the subset/manifest setup is unclear -- happy to help since we have the same data staged locally.