[M11] Fused probe has no cross-modal interaction — one direction still dead

[duckyquang]

Task metadata

Field	Value
Task ID	M11
Category	Model
Priority	High
Complexity	Medium
Est. days	3
Related	#67 (baseline gap), #69 (pooling), #64 (missing-modality numbers)

Features / method

The cross-attention fused probe has no cross-modal interaction. Half of it is now fixed; the other half needs token-level ECG embeddings.

Description

Splitting this out of #67 because it is a defect in the headline model, not a hypothesis about the baseline gap.

CrossAttentionFusion runs two attention blocks:

echo_ctx, _ = self.echo_to_ecg(echo_tokens, ecg_tokens, ecg_tokens)   # keys/values = ECG
ecg_ctx,  _ = self.ecg_to_echo(ecg_tokens, echo_tokens, echo_tokens)  # keys/values = echo

nn.MultiheadAttention degenerates twice when its keys/values are identical tokens: identical keys force exactly uniform softmax weights regardless of the query, and identical values make the weighted sum equal that value regardless of the weights. Both modalities arrived as a single pooled vector tiled into identical tokens, so both blocks were degenerate and the probe reduced to a concat-MLP with a linear pre-map per modality — with the two halves of the concat swapped, since echo_ctx depended only on ECG and ecg_ctx only on echo.

Clip-level echo retention (--max-clips, on feat/manifest-probe-training) fixes exactly one direction. Measured attention-weight spread, D=32, eval mode — 0 means uniform means degenerate:

block	keys	tiled echo	clip-level echo
ecg_to_echo	echo	0.000e+00	3.724e-02
echo_to_ecg	ECG	0.000e+00	0.000e+00

echo_to_ecg stays dead because its keys and values are the ECG tokens, and _ensure_ecg_tokens still tiles one pooled 768-d HuBERT-ECG vector into 4 identical tokens. No amount of echo work reaches it. Sensitivity of echo_ctx to a 100x perturbation of the echo input is exactly 0.000e+00 both before and after.

Steps:

1. Establish whether token-level ECG embeddings are obtainable — per-lead, per-time-window, or the pre-pooling HuBERT-ECG hidden states. The current Parquet stores one pooled vector per record, so this may require re-extraction rather than a loader change.
2. If they are, feed them as real tokens and confirm echo_to_ecg weight spread goes non-zero the same way ecg_to_echo did.
3. If they are not, decide honestly what to do with the fused probe. A cross-attention block that provably cannot attend in one direction should either be replaced with something that does fuse, or the paper should describe the model as what it is.
4. Re-run [E07] Canonical missing-modality rerun with bootstrap CIs #64 afterwards. The missing-modality numbers were measured on a model with no cross-modal interaction, so they describe degradation of a concat model.

Acceptance criteria:

• Stated yes/no on token-level ECG availability, with evidence.
• If yes: echo_to_ecg demonstrably attends, pinned by a test in the style of test_cross_attention_fusion_ignores_padded_clips.
• If no: a decision recorded on whether M09 stays cross-attention, plus a docs correction so the architecture is not described as doing something it cannot.
• Note for [W07] Restate reported numbers from the canonical run #68 either way — TECHNICAL.md 6.4 calls this the headline model.

Caveat: all measurements above are mechanism checks on synthetic tensors. data/ is empty on the machine this was found on, so nothing here says what fixing it is worth in MAE or AUROC.

[duckyquang]

Rescoping this from a loader change to a re-extraction task. @kevzho confirmed on #73:

I don't think we can get token-level ecg without re-extraction. The interim parquet is the pooled 768-d vector per record; there aren't any pre-pooling hidden states on disk.

So step 1 above ("establish whether token-level ECG embeddings are obtainable") is answered: not from anything currently on disk. The interim Parquet holds one pooled 768-d vector per record, and _ensure_ecg_tokens tiles that into 4 identical tokens — which is exactly what keeps echo_to_ecg degenerate.

HuBERT-ECG is wav2vec2-family, so re-running it over the MIMIC-IV-ECG waveforms with the pooling step removed would yield genuine per-window tokens. That makes this a GPU extraction job, not a loader fix, and it needs:

• PhysioNet + waveform access ([I07] Access check for the canonical rerun (PhysioNet + HF gate) #70)
• reserved GPU and storage for the re-extracted token embeddings, which are substantially larger than one pooled vector per record — [I08] Right-size compute for probe training and evaluation #71, rescoped to match
• a decision on token granularity, since that sets both the storage cost and what echo_to_ecg can actually attend over

Revised steps:

1. Size the job: how many ECG records in the paired cohort, what token granularity, resulting storage.
2. Confirm HuBERT-ECG can be run with pooling disabled to expose the hidden states.
3. Re-extract for the paired cohort only, not the full 800K records.
4. Feed as real tokens; confirm echo_to_ecg weight spread goes non-zero the way ecg_to_echo did with clip-level echo.
5. Re-run [E07] Canonical missing-modality rerun with bootstrap CIs #64 afterwards.

Revised acceptance criteria:

• Sized and costed re-extraction plan, or a recorded decision not to do it.
• If done: echo_to_ecg demonstrably attends, pinned by a test.
• If not done: M09 stops being described as cross-attention in TECHNICAL.md 6.4 and the paper, since it provably cannot fuse in one direction. Flag on [W07] Restate reported numbers from the canonical run #68.

The decision in step 5 of the original text still stands and is now the crux: if re-extraction isn't worth it, the honest move is to call the model what it is rather than ship a cross-attention label over a concat-MLP.