MIMIC-CXR: deterministic subset tooling + imaging battery (closes #324)

experiments/mimic_cxr_image/.gitignore

@@ -0,0 +1,11 @@
+# Generated by build_subset.py and reproducible from the seed (see README).
+# The per-arm CSVs embed credentialed MIMIC-CXR report text, so they must not be committed.
+manifests/*.csv
+manifests/download_list.txt
+manifests/provenance.json
+
+# Experiment results: commit the aggregate *_summary.json only. The per-case *.jsonl transcripts
+# and caches link MIMIC case_ids to reads (same credentialed-data caution as the manifests) and
+# regenerate from the runners; run logs reference host paths.
+results/**/*.jsonl
+results/**/*.log

experiments/mimic_cxr_image/README.md

@@ -0,0 +1,125 @@
+# MIMIC-CXR imaging subset (Lane A): download + battery (#92, #295)
+
+Replicates the NIH ChestX-ray14 imaging standard battery on MIMIC-CXR so imaging results are
+apples-to-apples across datasets. The full release is ~600 GB; we never download it. We build the
+manifest from the small metadata CSVs, select a deterministic nested study subset, and download
+only the images those studies reference (~800 images for the 600-study superset).
+
+## 0. Data prerequisites (credentialed, one time)
+
+You need PhysioNet credentialed access to MIMIC-CXR-JPG v2.1.0 (images/labels; v2.0.0 is not
+accessible) and MIMIC-CXR v2.0.0 (reports).
+Download only the small metadata and reports first (tens of MB, no images):
+
+```bash
+mkdir -p ~/mimic-cxr-raw && cd ~/mimic-cxr-raw
+
+wget -N -c --user YOUR_PHYSIONET_USER --ask-password \
+  https://physionet.org/files/mimic-cxr-jpg/2.1.0/mimic-cxr-2.0.0-metadata.csv.gz \
+  https://physionet.org/files/mimic-cxr-jpg/2.1.0/mimic-cxr-2.0.0-chexpert.csv.gz
+wget -N -c --user YOUR_PHYSIONET_USER --ask-password \
+  https://physionet.org/files/mimic-cxr/2.0.0/mimic-cxr-reports.zip
+
+gunzip -f mimic-cxr-2.0.0-*.csv.gz
+unzip -q mimic-cxr-reports.zip     # unpacks files/pXX/pYYYY/sZZZZ.txt (the adapter's report path)
+```
+
+The adapter (`benchmaxxing/datasets/mimic_cxr.py`) builds the manifest from these CSVs alone; no
+image needs to be on disk to select the subset.
+
+## 1. Select the nested study subset (`build_subset.py select`)
+
+Ranks studies by a stable hash of the seed and takes the top N per arm. The arm sizes nest, so
+every arm is a strict subset of the 600-study superset, and a 35-study set nests inside the 600
+for the matched NIH comparison. One seed fixes the whole set; the same seed reproduces the identical
+`download_list.txt` byte for byte.
+
+```bash
+python -m experiments.mimic_cxr_image.build_subset select \
+  --raw-root ~/mimic-cxr-raw \
+  --out-dir experiments/mimic_cxr_image/manifests \
+  --seed 20240724
+```
+
+Writes per-arm manifests (`solo_600.csv`, `cross_modal_600.csv`, `referee_300.csv`,
+`cascade_150.csv`, `blind_metric_100.csv`, `nih_match_35.csv`), a `download_list.txt` of the image
+URLs for the 600-study superset (the union of every arm), and `selection_summary.json`.
+
+Defaults (from the tracker): solo/cross-modal 600 studies, referee 300, cascade 150, blind-metric
+100, NIH-match 35. Only finding-positive (pneumothorax) studies are selected, matching the imaging
+pipeline; pass `--include-no-finding` to keep negatives too. A study contributes all of its images
+(frontal + lateral), so image counts run a little above the study counts.
+
+Building the full manifest reads ~227k report files (~30 s). To iterate faster, build it once and
+reuse it with `--full-manifest path/to/full_manifest.csv`.
+
+## 2. Download only the selected images
+
+The list is plain URLs, so the fetch is a standard-tool one-liner with your own credentials (no
+secrets in the repo). This is the reproducible download: same seed gives the same list.
+
+```bash
+cd ~/mimic-cxr-raw
+wget -N -c --user YOUR_PHYSIONET_USER --ask-password \
+  -i /path/to/experiments/mimic_cxr_image/manifests/download_list.txt \
+  -x -nH --cut-dirs=3
+# -x -nH --cut-dirs=3 recreates files/pXX/... under the current dir, matching image_ref.
+```
+
+(The requester-pays AWS mirror `s3://mimic-cxr-jpg-2.1.0.physionet.org/` works too; read the same
+list with `aws s3 cp`.)
+
+## 3. Verify + provenance (`build_subset.py verify`)
+
+After the download, confirm every referenced image resolves and pin a sha256 per image (same
+auditable-pixels guarantee as the NIH manifest):
+
+```bash
+python -m experiments.mimic_cxr_image.build_subset verify \
+  --manifest experiments/mimic_cxr_image/manifests/solo_600.csv \
+  --image-root ~/mimic-cxr-raw \
+  --provenance experiments/mimic_cxr_image/manifests/provenance.json
+```
+
+## 4. Run the battery
+
+One-time, install the imaging + model deps (full setup in `CONTRIBUTING.md`). The imaging runners
+need the `image` extra (Pillow/OpenCV cue injection) and the Gemini client; `.[models]` (torch +
+transformers) is only needed for the open-weights cross-lineage arm.
+
+```bash
+pip install -e ".[image,changepoint,stats]" && pip install "google-genai>=0.3"
+export GEMINI_API_KEY=...   # multimodal Gemini; compute is API-side, no local GPU
+```
+
+Point the dataset-agnostic imaging runners at the per-arm manifests. `--n 9999` means "every image
+in this arm" (the runners cap at the manifest size, so a smaller `--n` would silently truncate).
+Each run caches per call, so an interrupted run resumes cheaply on re-run.
+
+```bash
+M=experiments/mimic_cxr_image/manifests ; O=experiments/mimic_cxr_image/results ; RAW=~/mimic-cxr-raw
+
+python -m experiments.imaging.imaging_solo             --manifest $M/solo_600.csv         --image-root $RAW --out $O --n 9999               # #310
+python -m experiments.imaging.imaging_solo             --manifest $M/nih_match_35.csv     --image-root $RAW --out $O/nih_match_35 --n 9999  # NIH-matched (35 images)
+python -m experiments.imaging.imaging_cascade          --manifest $M/cascade_150.csv      --image-root $RAW --out $O/cascade_150 --n 9999   # #311
+python -m experiments.imaging.imaging_system_flag      --manifest $M/solo_600.csv         --image-root $RAW --out $O --n 9999               # #312
+python -m experiments.imaging.imaging_strength_cascade --manifest $M/solo_600.csv         --image-root $RAW --out $O --n 9999               # #313
+python -m experiments.imaging.imaging_blind_metric     --manifest $M/blind_metric_100.csv --image-root $RAW --out $O --n 9999               # #314
+
+# #315 referee consumes a cascade transcript over referee_300. Give each cascade run its OWN --out:
+# for the watermark cue imaging_cascade writes a fixed imaging_cascade.jsonl, so a shared --out would
+# silently overwrite the #311 arm. Run the referee_300 cascade into its own dir, then run the referee.
+python -m experiments.imaging.imaging_cascade          --manifest $M/referee_300.csv      --image-root $RAW --out $O/referee_300 --n 9999
+python -m experiments.imaging.imaging_referee          --manifest $M/referee_300.csv      --image-root $RAW --cascade-jsonl $O/referee_300/imaging_cascade.jsonl --out $O/referee_300
+```
+
+Results land under `experiments/mimic_cxr_image/results/`, same format as
+`experiments/imaging/results/`, and feed issue #295's cross-dataset table.
+
+## Reproducibility
+
+Three artifacts pin a run, mirroring the NIH lane:
+
+- `download_list.txt` + the seed: which images (deterministic from the seed).
+- the per-arm manifests: which cases each arm ran on (pinned case_ids).
+- `provenance.json`: the exact pixels (sha256 per image).

experiments/mimic_cxr_image/build_subset.py

@@ -0,0 +1,216 @@
+"""Select a deterministic, nested MIMIC-CXR study subset and emit its download list (#92, #295).
+
+The full MIMIC-CXR-JPG release is ~600 GB, but the imaging battery only needs a few hundred
+studies per arm, and the arm sizes nest (blind 100 < cascade 150 < referee 300 < solo/cross-modal
+600, with a 35-study set nested inside the 600 for an apples-to-apples comparison against the NIH
+ChestX-ray14 run). So we never download the whole release: we build the manifest from the small
+metadata CSVs (no images required on disk), rank *studies* by a stable hash of the seed, take the
+top N for each arm, and write one ``download_list.txt`` of just the images in the 600-study
+superset. The union of every arm is exactly that superset, so a single download covers all arms.
+
+Determinism and nesting both come from ``benchmaxxing.budget.subsample_cases`` (hash-ranked, so
+the selection depends only on study identity, not on the order the pool arrives in). Same seed
+gives the identical list byte for byte, which is what makes the download reproducible: the list
+plus the per-arm manifests plus the ``verify`` provenance checksums are the reproducibility
+artifacts, exactly as the NIH imaging manifest pins its cases and sha256s its pixels.
+
+Two modes:
+
+    select   build/reuse the full manifest, rank studies, write per-arm manifests + download_list.txt
+    verify   after the image download: confirm every referenced image resolves and write provenance.json
+
+Selection needs only the metadata CSVs + reports (what ``mimic_cxr.build_manifest`` reads); it does
+not need any image on disk. ``verify`` is the step that needs the downloaded images.
+"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import tempfile
+from pathlib import Path
+
+from benchmaxxing.budget import RunBudget, subsample_cases
+from benchmaxxing.data import load_cases
+from benchmaxxing.datasets import mimic_cxr
+from benchmaxxing.datasets.base import finalize
+
+# Arm -> number of studies. solo and cross-modal share the 600 superset; every smaller arm is a
+# strict subset of it, so the download only ever needs the 600. nih_match is the 35 nested inside
+# the 600 for the matched NIH comparison.
+DEFAULT_SIZES = {
+    "solo": 600,
+    "cross_modal": 600,
+    "referee": 300,
+    "cascade": 150,
+    "blind_metric": 100,
+    "nih_match": 35,  # IMAGES, not studies (see IMAGE_SIZED_ARMS): matched one-per-observation to NIH n=35
+}
+
+# Arms sized by image count rather than study count. nih_match is the matched-to-NIH cell: NIH runs
+# one image per case, so counting studies here (each expanded to all its views) would ship correlated
+# multi-view clusters and a mismatched n. Sizing by images keeps it exactly n one-per-observation.
+IMAGE_SIZED_ARMS = frozenset({"nih_match"})
+
+DEFAULT_URL_BASE = "https://physionet.org/files/mimic-cxr-jpg/2.1.0/"
+
+
+def _study_key(case) -> str:
+    """Stable per-study identity: ``patient_id/study_id``. study_id alone is globally unique in
+    MIMIC, but pairing it with the subject id is defensive and self-documenting."""
+    return f"{case.patient_id}/{case.meta.get('study_id')}"
+
+
+def _is_finding(case) -> bool:
+    """True when the case carries a real finding label (the adapter emits 'pneumothorax' vs
+    'no finding'). Mirrors the eligibility filter the NIH imaging pipeline uses."""
+    return bool(case.label) and case.label.strip().lower() not in ("no finding", "", "none")
+
+
+def select_arms(cases, seed: int, sizes: dict[str, int], require_finding: bool = True) -> dict[str, list]:
+    """Group cases into studies, rank studies deterministically by ``seed``, and return the nested
+    per-arm case lists.
+
+    Each arm gets *all* images of its top-N studies. Because every arm ranks the same study pool
+    with the same seed, the smaller arms are strict subsets of the larger ones: the top-100 studies
+    are a prefix of the top-300, which are a prefix of the top-600 (the prefix property of
+    :func:`subsample_cases`). Returns ``{arm_name: [Case, ...]}``.
+
+    Arms in :data:`IMAGE_SIZED_ARMS` (``nih_match``) are instead sized by image count: the first N
+    images in study-rank order, so the matched-to-NIH cell is n images one-per-observation rather
+    than N studies expanded to more (correlated) views. It stays a prefix of the ranking, so it
+    still nests inside the study-sized arms.
+    """
+    eligible = [c for c in cases if not require_finding or _is_finding(c)]
+    studies: dict[str, list] = {}
+    for case in eligible:
+        studies.setdefault(_study_key(case), []).append(case)
+    study_keys = list(studies)
+
+    arms: dict[str, list] = {}
+    for arm, n in sizes.items():
+        if arm in IMAGE_SIZED_ARMS:
+            # Rank studies, then take the first n images in that order. Ranking n studies always
+            # yields >= n images (each study has >= 1 image), and the slice is a prefix of the study
+            # ranking, so the arm still nests inside the study-sized arms.
+            picked = subsample_cases(study_keys, RunBudget(max_cases=min(n, len(study_keys)), seed=seed))
+            arms[arm] = [case for key in picked for case in studies[key]][:n]
+        else:
+            picked = subsample_cases(study_keys, RunBudget(max_cases=n, seed=seed))
+            arms[arm] = [case for key in picked for case in studies[key]]
+    return arms
+
+
+def _download_lines(cases, url_base: str) -> list[str]:
+    """One URL per unique image in ``cases``, in a deterministic order, for ``wget -i``."""
+    base = url_base.rstrip("/") + "/"
+    seen: dict[str, None] = {}
+    for case in cases:
+        if case.image_ref and case.image_ref not in seen:
+            seen[case.image_ref] = None
+    return [base + ref for ref in seen]
+
+
+def _build_full_cases(raw_root: str, full_manifest: str | None):
+    """Return every case in the release. Reuses ``--full-manifest`` when given (fast reruns),
+    otherwise builds it once into a temp file (discarded) via the registered adapter."""
+    if full_manifest and Path(full_manifest).is_file():
+        return load_cases(full_manifest)
+    with tempfile.TemporaryDirectory() as tmp:
+        return mimic_cxr.build_manifest(raw_root, Path(tmp) / "full_manifest.csv")
+
+
+def _sha256(path: Path) -> str:
+    h = hashlib.sha256()
+    with path.open("rb") as f:
+        for chunk in iter(lambda: f.read(1 << 20), b""):
+            h.update(chunk)
+    return h.hexdigest()
+
+
+def do_select(args) -> None:
+    out_dir = Path(args.out_dir)
+    out_dir.mkdir(parents=True, exist_ok=True)
+    sizes = dict(DEFAULT_SIZES)
+
+    all_cases = _build_full_cases(args.raw_root, args.full_manifest)
+    arms = select_arms(all_cases, seed=args.seed, sizes=sizes, require_finding=not args.include_no_finding)
+
+    written = {}
+    for arm, cases in arms.items():
+        path = out_dir / f"{arm}_{sizes[arm]}.csv"
+        finalize(cases, path)
+        n_studies = len({_study_key(c) for c in cases})
+        written[arm] = {"manifest": str(path), "studies": n_studies, "images": len(cases)}
+
+    # The download superset is the largest arm; every other arm is nested inside it.
+    superset = max(arms.values(), key=len)
+    lines = _download_lines(superset, args.url_base)
+    (out_dir / "download_list.txt").write_text("\n".join(lines) + "\n", encoding="utf-8")
+
+    summary = {
+        "seed": args.seed,
+        "require_finding": not args.include_no_finding,
+        "url_base": args.url_base,
+        "arms": written,
+        "download_list": str(out_dir / "download_list.txt"),
+        "download_images": len(lines),
+    }
+    (out_dir / "selection_summary.json").write_text(json.dumps(summary, indent=2), encoding="utf-8")
+    print(json.dumps(summary, indent=2))
+
+
+def do_verify(args) -> None:
+    image_root = Path(args.image_root)
+    cases = load_cases(args.manifest)
+    missing = [c.image_ref for c in cases if not (image_root / c.image_ref).is_file()]
+    if missing:
+        raise SystemExit(
+            f"{len(missing)}/{len(cases)} images not found under {image_root} "
+            f"(first few: {missing[:5]}). Run the download first."
+        )
+    provenance = {
+        "manifest": str(args.manifest),
+        "image_root": str(image_root),
+        "n_cases": len(cases),
+        "cases": [
+            {
+                "case_id": c.case_id,
+                "image_ref": c.image_ref,
+                "label": c.label,
+                "sha256": _sha256(image_root / c.image_ref),
+                "bytes": (image_root / c.image_ref).stat().st_size,
+            }
+            for c in cases
+        ],
+    }
+    Path(args.provenance).write_text(json.dumps(provenance, indent=2), encoding="utf-8")
+    print(json.dumps({"n_cases": len(cases), "provenance": str(args.provenance), "all_present": True}, indent=2))
+
+
+def main() -> None:
+    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    sub = ap.add_subparsers(dest="mode", required=True)
+
+    sel = sub.add_parser("select", help="rank studies and write per-arm manifests + download_list.txt")
+    sel.add_argument("--raw-root", required=True, help="dir with the metadata/chexpert CSVs and unpacked reports")
+    sel.add_argument("--out-dir", required=True, help="output dir for per-arm manifests + download list")
+    sel.add_argument("--seed", type=int, default=0, help="selection seed (fixes the nested study set)")
+    sel.add_argument("--url-base", default=DEFAULT_URL_BASE, help="prefix for image_ref paths in download_list.txt")
+    sel.add_argument("--full-manifest", default=None, help="reuse a prebuilt full manifest instead of rebuilding")
+    sel.add_argument("--include-no-finding", action="store_true", help="keep 'no finding' studies too (default: findings only)")
+    sel.set_defaults(func=do_select)
+
+    ver = sub.add_parser("verify", help="check images resolve on disk and write provenance.json")
+    ver.add_argument("--manifest", required=True, help="a per-arm manifest (usually the 600-study superset)")
+    ver.add_argument("--image-root", required=True, help="dir holding the downloaded images")
+    ver.add_argument("--provenance", required=True, help="output provenance JSON path (sha256 per image)")
+    ver.set_defaults(func=do_verify)
+
+    args = ap.parse_args()
+    args.func(args)
+
+
+if __name__ == "__main__":
+    main()

experiments/mimic_cxr_image/manifests/selection_summary.json

@@ -0,0 +1,39 @@
+{
+  "seed": 20240724,
+  "require_finding": true,
+  "url_base": "https://physionet.org/files/mimic-cxr-jpg/2.1.0/",
+  "arms": {
+    "solo": {
+      "manifest": "experiments/mimic_cxr_image/manifests/solo_600.csv",
+      "studies": 600,
+      "images": 834
+    },
+    "cross_modal": {
+      "manifest": "experiments/mimic_cxr_image/manifests/cross_modal_600.csv",
+      "studies": 600,
+      "images": 834
+    },
+    "referee": {
+      "manifest": "experiments/mimic_cxr_image/manifests/referee_300.csv",
+      "studies": 300,
+      "images": 417
+    },
+    "cascade": {
+      "manifest": "experiments/mimic_cxr_image/manifests/cascade_150.csv",
+      "studies": 150,
+      "images": 215
+    },
+    "blind_metric": {
+      "manifest": "experiments/mimic_cxr_image/manifests/blind_metric_100.csv",
+      "studies": 100,
+      "images": 141
+    },
+    "nih_match": {
+      "manifest": "experiments/mimic_cxr_image/manifests/nih_match_35.csv",
+      "studies": 26,
+      "images": 35
+    }
+  },
+  "download_list": "experiments/mimic_cxr_image/manifests/download_list.txt",
+  "download_images": 834
+}