MIMIC-CXR text: replicate the hidden-rubric break-it arm (Table 1, row D)

[sebasmos]

Table 1 (break-it experiments) has two rows never replicated on MIMIC-CXR text: D (hidden rubric rewarding the longest option) and B (deliberation framing ladder). This issue is for D specifically, the hidden-rubric arm, since that's the one with no script at all for this cohort.

What exists already, for reference

• Row B (deliberation framing) is already replicated on MedMCQA (experiments/medqa/deliberation_framing.py, run against the MedMCQA manifest, n=120, ladder 0.74/0.56/0.37/0.28), merged via results: MedMCQA foundational arms — solo, cascade, break-it A + C #349/results: MedMCQA committee battery — seed shape, social pressure, hierarchy, controls, referee + corrected family #368. Not yet run on MIMIC-CXR text.
• Row D (hidden rubric) exists only on MedQA (Table 1: 0.24 -> 0.24, null). No equivalent script or run exists for MedMCQA or MIMIC-CXR text.

What's needed

Run the hidden-rubric break-it arm (row D) on the MIMIC-CXR text cohort: a rubric that secretly rewards the longest-option decoy, paired against a no-rubric control, scored as decoy-selection rate. The existing MedQA script is the template; it will need pointing at the MIMIC-CXR text manifest and cache the same way the other break-it arms (A, C) were adapted for this cohort in experiments/mimic_cxr_text/push_c.py.

Why this cohort specifically

MIMIC-CXR text is already one of the two replication cohorts for rows A and C in Table 1 (paper-miccai.tex, main-full.tex). Row D is the only cell in that table with a real gap on this cohort, since MedMCQA now covers B and (soon) will need D too, but nobody has started D on either replication cohort yet.

Assigning to @felipeocampoos as the owner of the MIMIC-CXR text lane (per master-table.md's reproducibility ledger).

[sebasmos]

Correction, and closing: the premise of this issue is wrong. Row D already exists on both cohorts, and both cells are now in Table 1.

I filed this and did not check the artifacts carefully enough first. It claims "No equivalent script or run exists for MedMCQA or MIMIC-CXR text" for row D. Both exist and are committed on main:

cohort	artifact	value
MedMCQA	experiments/medmcqa/results/break_it_summary.json, key D_blind_metric_incentive	control 0.3077, incentivised 0.3077, decoy_drift 0.0, n=13
MIMIC-CXR text	experiments/mimic_cxr_text/results/blind_metric_summary.json	baseline 0.00, blind 0.05, delta +0.05, n=40

Table 1's row D now reads 0.24 -> 0.24 for MedQA, 0.31 -> 0.31 for MedMCQA and 0.00 -> 0.05 for MIMIC-CXR text. The MIMIC-CXR value is a raw rate rather than a difference, because the decoy is defined as an option other than the baseline answer, so the control arm cannot select it and is structurally 0. That is the same structural caveat row A carries, and the caption says so.

So nothing needs running for row D. There is no script named break_it_d.py on this cohort, which is what misled me, but the arm is implemented as blind_metric.py and the run is committed.

The one genuine gap in Table 1 is row B on MIMIC-CXR text, which this issue also mentions in passing. That now has its own issue with the numbers from both cohorts where it has run, so @felipeocampoos has one clear ask rather than two overlapping ones:

• MIMIC-CXR text: deliberation-framing arm (break-it B) never run, the last blank in Table 1 #398: MIMIC-CXR text deliberation-framing arm (break-it B) never run.

Closing this as already satisfied, and superseded by #398 for the part that is real. Apologies for the noise, @felipeocampoos.

[sebasmos]

Reopening: closing this as not_planned was my mistake.

I closed it on the basis that row D was already covered for this cohort by blind_metric.py (#320/#356). Those measure different mechanisms. blind_metric rewards an arbitrary decoy chosen to differ from the model's own baseline; this issue asks for the literal MedQA design, where the rubric rewards the longest option. Only the second is comparable across cohorts on the same rule.

That distinction turned out to matter beyond the tracker: our break-it table labelled row D "hidden rubric rewards the longest option" while the MIMIC-CXR column came from blind_metric, so the row claimed one mechanism and one column measured another. The label is now corrected and the difference stated.

@felipeocampoos spotted this and built the literal version in #404, which is the right call. That PR has an open change request from me on two of its claims, not on the port, so this issue stays open until it lands.