Experiment runners: emit incremental progress during long runs

[amarzullo24]

The experiment runners (imaging lane, and the text/medqa runners look the same) print nothing until the final summary JSON. On real-data + API runs of thousands of calls that take hours, there is no built-in way to see how far along a run is, or whether it has stalled on a retry/rate-limit. Today progress has to be inferred out-of-band by wc -l on the cache JSONL.

Felt concretely running the MIMIC-CXR battery (#295): the solo arm alone is ~6-7k Gemini calls.

Proposal: a lightweight, shared progress signal, a periodic line to stderr (so stdout/the summary stays clean) every N cases with completed X/Y, elapsed, rate, ETA. Dependency-light (no hard tqdm dependency; use it only if already present). Prefer a small shared helper reused across runners over per-script code.

Nice-to-have: distinguish cache hits from live calls in the count.

Non-blocking; developer-experience enhancement.

[armaanvgrewal]

@amarzullo24 @duckyquang @sebasmos

I’d like to help with this, if @duckyquang is OK with it.

Based on the issue text, I’m thinking of this for the first pass:

• add a dependency-light shared progress helper, probably in benchmaxxing/progress.py
• emit periodic progress to stderr so stdout/final JSON stays clean
• include completed/total, elapsed, rate, and ETA
• add unit tests for the helper, likely tests/test_progress.py
• wire it into one representative MedQA runner and one representative imaging runner first

I'll leave nice-to-haves cache-hit vs live-call counts as a follow-up unless you want that included in the first pass.

@sebasmos Does this scope sound right, or should the first PR update all experiment runners at once (or is there a different priority order)?

[amarzullo24]

Fine with me! I would also add some initial statement about the dataset, number of instances and any other informative print

[armaanvgrewal]

@sebasmos @amarzullo24 @duckyquang

#351 addresses pass 1. If it passes muster, I can expand from there.

[duckyquang]

Yeah, go for it, and thanks for picking this up. Pass 1 (#351) looks good and it's merged.

For pass 2 the loops that actually eat wall-clock are the experiment runners, so I'd point the helper at those next: benchmaxxing/runner.py (the solo + cascade stages), experiments/medqa/reproduce.py, and experiments/imaging/imaging_solo.py + imaging_cascade.py. Same stderr / every-n pattern you already used in analysis and ablations. Cache-hit vs live-call counts can stay a follow-up like you said.

Keeping this open until those runners are covered.

[sebasmos]

Sounds good to me @armaanvgrewal

[sebasmos]

Closing as part of the pre-release cleanup. The paper is submitted and the repository is being prepared to go public, so the board is being reduced to the single item that still gates a published number: #393, the MIMIC-CXR same-lineage judge cell. Everything else is out of scope for this release. This is not a judgement on the work: unmerged branches were archived to git bundles before deletion, and anything here can be reopened if the lane is picked up again.