Dataset status: staged vs coded vs blocked (single source of truth)

[sebasmos]

Purpose. One place to see, for every dataset, exactly what is staged (data on disk), what is coded (an adapter merged), and what has actually produced results, plus what is blocked and why. This mirrors the per-dataset coverage grid in the paper draft. Update this issue whenever a dataset changes state.

Legend

• ✅ done / available
• 🔄 in progress / in review
• ⬜ planned, not started
• ⛔ blocked (see reason)

Coverage matrix

Dataset	Lane	Data staged	Adapter (code)	Solo susceptibility	Cascade	Overall
MedQA-USMLE	text (A-E MCQ)	✅	✅ medqa.py	✅ flash 0.79 / flash-lite 0.89 (noise floor 0.000)	✅ null; break-it + n=150 scale done	✅ LIVE, complete
NIH ChestX-ray14	imaging	✅	✅ nih_cxr14.py	✅ watermark +0.11 above a 0.23 floor (others at/below)	⬜ pending (multimodal committee)	🔄 LIVE, cascade is next
PubMedQA	text (yes/no/maybe)	⬜	✅ pubmedqa.py (#148 merged)	⬜	⬜	🔄 code-ready, no data run
MedMCQA	text (A-E MCQ)	⬜	🔄 #142 (has conflicts)	⬜	⬜	🔄 adapter in review
CheXpert	imaging (natural cue: support devices)	⛔	✅ chexpert.py	⬜	⬜	⛔ blocked (Stanford registration)
MIMIC-CXR	imaging + reports	⛔	✅ mimic_cxr.py	⬜	⬜	⛔ blocked (PhysioNet credentialing)
Structured EHR	tabular	⬜	🔄 ehr.py (NotImplementedError stub)	⬜	⬜	⬜ deferred (stage 5)

Bottom line: only MedQA and NIH ChestX-ray14 are actually producing results today. PubMedQA and MedMCQA are code-only (no data run). CheXpert, MIMIC-CXR, and EHR are not runnable yet.

Per-dataset detail

MedQA-USMLE (text) — LIVE, complete

• Data: train/dev/test = 10,178 / 1,272 / 1,273; manifest built, answer_index round-trips against ground truth.
• Adapter: benchmaxxing/datasets/medqa.py (merged in feat: dataset adapters + stage runners + scrutiny + prompt registry (ready to run on real data) #66).
• Results: solo susceptibility (flash 0.79 / flash-lite 0.89, noise floor 0.000); cascade null (verified real, seed differs from committee baseline); break-it A/C/D; C plausibility dose-response at n=150 (McNemar p<1e-4). All reproducible from the committed call cache. PRs results: MedQA experiments (reproduction script + saved results); closes #93, #96 #135, results: break-the-cascade experiments (A/C/D) on MedQA; C anchored-seed is the lever #140, results: cascade at scale (C significant p<1e-4, clean-A contamination) + imaging solo #141.
• Next: nothing required; it is the primary text-lane result.

NIH ChestX-ray14 (imaging) — LIVE, cascade next

• Data: metadata (112,120 rows) + one image batch (4,999 X-rays) staged; every sampled image resolves on disk; all four image cues inject on a real X-ray with the diagnosis preserved.
• Adapter: benchmaxxing/datasets/nih_cxr14.py.
• Results: imaging solo done. With the noise floor computed in-script (temperature>0 resample), only the watermark cue clears it (+0.11); laterality/corner-tag/cable fall at or below the 0.23 floor at n=35. PR results: cascade at scale (C significant p<1e-4, clean-A contamination) + imaging solo #141.
• Next: the multimodal imaging cascade (shared vs isolated committee on the watermark cue first). This is the natural next experiment and does not need new data.

PubMedQA (text, yes/no/maybe) — code-ready, no data run

• Data: not staged. Needs the official ori_pqal.json (pqa_labeled) downloaded locally.
• Adapter: benchmaxxing/datasets/pubmedqa.py (merged in feat: PubMedQA dataset adapter (closes #114) #148). 3-way answer shape (yes/no/maybe), which differs from A-E MCQ.
• Results: none yet.
• Next: download pqa_labeled, build a manifest, run solo (and cascade). Tracks the adapter issue Dataset adapter: PubMedQA (text QA) #114 (closed) for code; the run is still to do.

MedMCQA (text, A-E MCQ) — adapter in review

• Data: not staged.
• Adapter: PR feat: MedMCQA dataset adapter (closes #112) #142 (open, has a merge conflict in tests/test_datasets.py and ~17 ruff errors; author to rebase). Issue Dataset adapter: MedMCQA (text/MCQ) #112.
• Results: none yet.
• Next: land feat: MedMCQA dataset adapter (closes #112) #142, stage MedMCQA, run solo (and the cross-dataset cue-consistency check, Experiment: cross-dataset consistency of a cue (MedQA vs MedMCQA) #129).

CheXpert (imaging, natural cue) — BLOCKED

• Why blocked: requires Stanford CheXpert registration/agreement before the images can be downloaded.
• Adapter: benchmaxxing/datasets/chexpert.py exists (code) but is unvalidated on real data.
• Results: none. This is the intended primary MICCAI cascade (support-device natural confounder), so unblocking it is high value. Issues CheXpert loader with device labels #46 (loader), Validate injected vs natural cues on real CheXpert Support Devices #94 (validate cues), Cascade (primary redesign): natural-cue imaging cascade on CheXpert Support Devices #120 (cascade).
• Next: obtain CheXpert access, stage the support-device subset, validate the natural cue, then run the natural-cue cascade.

MIMIC-CXR (imaging + reports) — BLOCKED

• Why blocked: requires PhysioNet credentialed access (CITI training + data-use agreement).
• Adapter: benchmaxxing/datasets/mimic_cxr.py exists (code), unvalidated on real data.
• Results: none. Issues MIMIC-CXR loader and image-to-report pairing #45 (loader), Run stage-1 solo baselines on real MIMIC-CXR (pneumothorax + chest tube) #92 (run solo on pneumothorax + chest tube).
• Next: complete PhysioNet credentialing, stage the pneumothorax-with-chest-tube subset, run solo then cascade.

Structured EHR (tabular) — deferred

• Why deferred: stage-5 resource-constraint scenarios; not needed for the MICCAI submission.
• Adapter: benchmaxxing/datasets/ehr.py raises NotImplementedError by design (documented stub). Issue Structured EHR loader for resource-constraint scenarios #49.

Access / credentialing checklist (the two real blockers)

• PhysioNet (MIMIC-CXR): create a PhysioNet account, complete CITI "Data or Specimens Only Research" training, sign the MIMIC-CXR data-use agreement, then download.
• Stanford (CheXpert): register on the Stanford ML Group CheXpert page and accept the research-use agreement, then download.

Related issues (consolidated here)

• Adapters: Dataset adapter: MedMCQA (text/MCQ) #112 (MedMCQA), Dataset adapter: PubMedQA (text QA) #114 (PubMedQA, closed), Dataset adapter: a second imaging set (VinDr-CXR or Open-i) #119 (second imaging set), Docs: "Your first dataset adapter in 20 minutes" tutorial #123 (adapter tutorial).
• Loaders (older, overlapping): MIMIC-CXR loader and image-to-report pairing #45 (MIMIC), CheXpert loader with device labels #46 (CheXpert), NIH ChestX-ray14 loader (open replication arm) #47 (NIH — done, closing), Structured EHR loader for resource-constraint scenarios #49 (EHR).
• Runs: Run stage-1 solo baselines on real MIMIC-CXR (pneumothorax + chest tube) #92 (MIMIC solo), Run the stage-0 pilot on real MedQA and freeze the pre-registration #93 (MedQA pilot — closes with results: MedQA experiments (reproduction script + saved results); closes #93, #96 #135), Validate injected vs natural cues on real CheXpert Support Devices #94 (CheXpert cue validation).
• Cascade / experiment: Cascade (primary redesign): natural-cue imaging cascade on CheXpert Support Devices #120 (CheXpert natural-cue cascade), Experiment: cross-dataset consistency of a cue (MedQA vs MedMCQA) #129 (cross-dataset cue consistency).
• Staging/provenance: Dataset acquisition and local staging with provenance manifests #103 (dataset acquisition + provenance manifests).

Close/label reconciliation done alongside this issue: #47 closed (NIH staged + solo done); #45/#46/#92/#94/#120 labelled blocked: data-access.

[sebasmos]

De-scoping for the 5-day workshop sprint (overlaps the dataset-status PR #239 anyway). Priority 3 for now, reopening after the deadline. Thank you!