[E07] Canonical missing-modality rerun with bootstrap CIs

[duckyquang]

Task metadata

Field	Value
Task ID	E07
Category	Evaluation
Priority	High
Complexity	Medium
Est. days	2
Blocked by	M10

Features / method

Canonical missing-modality rerun with bootstrap confidence intervals and persisted per-example predictions.

Description

results/missing_modality.json carries its own caveat: the numbers are point estimates transcribed from the E02 evaluation, with bootstrap CIs pending a canonical rerun. Per-example predictions were never saved, which is why CIs could not be computed after the fact.

Rerun scripts/evaluate_missing_modality.py against the canonical fused checkpoint from M10 and fix that properly.

Steps:

1. Score the canonical fused checkpoint under all three conditions (full, echo_dropped, ecg_dropped) on the manifest test split.
2. Persist per-example predictions alongside the aggregate metrics, not just the summary numbers.
3. Compute bootstrap CIs for MAE and EF<=40% AUROC in every condition.
4. Regenerate the degradation curve figure (E02) from the new predictions.
5. Record the masking strategy used, and resolve the open decision in TECHNICAL.md 11 (zero-out vs learned null token vs branch dropout) with the reason written down.

Acceptance criteria:

• results/missing_modality.json replaced with a run that includes CIs and a provenance field naming the checkpoint, embedding source, seed, and device.
• Per-example predictions written to disk and reusable by E08, E09, and E10 without rescoring.
• Degradation curve regenerated from real predictions.
• No transcribed numbers anywhere in the output.

[duckyquang]

Found and fixed a blocker on this that wasn't obvious until #73's wiring landed.

evaluate_missing_modality still required a cohort table plus two separate embedding tables, so E07 could not read a joined manifest even after the probes learned to. The canonical rerun would have trained a checkpoint off the manifest in #63 and then had nothing able to score it off the same table.

Fixed in 30ab118: both embedding paths are optional, matching the probes, and --manifest supersedes the two-table flags on the CLI. checkpoint_path keeps its positional slot but now raises a named error instead of failing inside Path(None).

Covered by test_missing_modality_scores_a_manifest_trained_checkpoint, which drives the real chain — train the fused probe off a manifest, save the checkpoint, score all three conditions off the same manifest. Verified it bites: restore the required-argument signature and both new tests fail. Suite 95 → 97.

What this means for the issue: steps 1–2 (score all three conditions, persist per-example predictions) are now runnable the moment a manifest is reachable — the machinery is in place and predict_missing_modality already returns per-example arrays, so the CI work in step 3 has what it needs.

Still blocked on:

• the manifest itself (@kevzho has it — asked on Retain clip-level echo tokens so attentive pooling actually pools #73)
• [M10] Retrain all four probes on the synchronized manifest #63's canonical checkpoint, which is the input here

One thing to decide before the rerun. Step 5 asks to resolve the masking strategy in TECHNICAL.md 11. That decision is now entangled with #72: mask_echo/mask_ecg were verified to behave correctly (output is bit-independent of the masked modality), but they mask a fusion block with no cross-modal interaction. So whatever numbers come out describe degradation of a concat model, and they will change if #72's re-extraction lands. Worth deciding whether to run this now for a baseline or wait — I'd run it now and label it clearly, since #72 has no timeline.

[duckyquang]

Ran the E07 missing-modality rerun from the M10 pooled fused checkpoint.

Command shape:

python scripts/evaluate_missing_modality.py \
  --manifest <1,208-row manifest> \
  --checkpoint <M10 pooled fused checkpoint> \
  --output <ignored local json> \
  --embed-dim 256 --echo-dim 1024 --ecg-dim 768 \
  --n-bootstrap 1000

Device/config: local Mac CPU, seed 42, batch size 64, fusion_dim 256, hidden 256. Wall-clock: 5.71s.

Test n is 245 after dropping the 3 non-finite HuBERT-ECG test rows.

condition	MAE	95% CI	EF<=40 AUROC	95% CI
full	10.4218	9.2770-11.6448	0.7708	0.6968-0.8386
echo dropped	20.5507	19.0182-22.0169	0.6890	0.6050-0.7654
ECG dropped	11.2167	9.9998-12.3924	0.3827	0.2793-0.4934

The output JSON includes per-example predictions for all three conditions, and I regenerated the degradation curve from that JSON rather than transcribing point estimates.

Masking strategy is now recorded in TECHNICAL.md on this branch: inference-time branch masks (mask_echo / mask_ecg), no retrained unimodal model, no learned null token, no branch dropout for this canonical rerun.

Caveat: this is the pooled-manifest checkpoint. It is the right baseline for this PR, but the clip-level echo comparison in #69 is still separate.

[duckyquang]

Promoted the canonical pooled E07 artifacts into the default ignored local paths:

results/missing_modality.json
results/figures/degradation_curve.{pdf,png,svg}
results/figures/degradation_curve_caption.txt
probes/cross_attn_fused/cross_attn_fused.pt

I also updated README's results table to the real rerun with 95% bootstrap CIs instead of the older transcribed point estimates. The README now states the exact caveat that this is the pooled-manifest baseline and that results/ / probes/ stay gitignored.

Smoke check against the promoted checkpoint reproduced the same point estimates:

full         MAE 10.4218 | EF<=40 AUROC 0.7708
echo_dropped MAE 20.5507 | EF<=40 AUROC 0.6890
ecg_dropped  MAE 11.2167 | EF<=40 AUROC 0.3827