[E10] Diagnose fused vs single-modality baseline gap

[duckyquang]

Task metadata

Field	Value
Task ID	E10
Category	Evaluation
Priority	High
Complexity	High
Est. days	4
Blocked by	M10

Features / method

Work out why the fused model underperforms published single-modality baselines.

Description

The reported fused numbers are 10.28 MAE and 0.766 EF<=40% AUROC. The baselines in TECHNICAL.md 8 are EchoJEPA alone at 5.97 MAE and ECG-FM alone at 0.929 AUROC. Fusion is currently losing to both published solo baselines on their own headline metric.

Those baselines are on different cohorts and different setups, so this is not a like-for-like comparison and the gap may be entirely explained by cohort. But the deployment-risk framing needs the full condition to be a credible model — "degrades gracefully" is a weak claim if the starting point is well behind what either modality does alone. This needs an answer before the numbers go into a paper.

M10 gives the in-cohort solo baselines, which is the part that makes this diagnosable.

Steps:

1. Compare in-cohort ECG-only and echo-only probes against the fused probe on identical splits. If fusion beats both in-cohort, the gap is cohort, not architecture.
2. Check label noise: build_cohort.py falls back across several LVEF measurement sources (lvef, biplane_lvef, rest_lvef, rest_biplane_lvef, lvef_3d). Compare error stratified by lvef_measurement to see if a fallback source is noisier.
3. Check probe capacity and training budget — 829 training rows is small, so confirm the fused probe is not simply overfitting or underfitting relative to the concat baseline.
4. Check the echo pooling step. Clip-level EchoJEPA embeddings are mean-pooled to one vector per study, which discards the spatial structure TECHNICAL.md 4.1 says an attentive probe needs.
5. Write up which of these explains the gap, with evidence.

Acceptance criteria:

• In-cohort fusion vs solo comparison on identical splits, with CIs.
• A stated, evidence-backed answer for the gap: cohort size, label noise, pooling, probe capacity, or some combination.
• If the cause is fixable within Task A scope, a follow-up issue opened for the fix.
• If the cause is cohort size, this feeds D07 and the finding is documented as a limitation.

[duckyquang]

Candidate explanation for the gap, found while wiring #63 and verifiable without data.

Echo embeddings reach the probes as one mean-pooled vector per study, which _embedding_to_tokens tiles into identical tokens. AttentivePool over identical tokens is an exact identity (uniform softmax), so the echo branch is an MLP on a mean-pooled vector — no attentive pooling happens anywhere in the current pipeline. Details and numbers in #69.

Why this is a plausible driver of 10.28 MAE vs EchoJEPA's published 5.97: docs/echo_hubert_loader.md records ~525K clip rows collapsing to 7,251 study rows. Whatever per-clip structure EchoJEPA encodes is averaged away before any probe sees it, and TECHNICAL.md 4.1 states plainly that a plain head on pooled embeddings underperforms and that attentive pooling is required. The architecture to do better is implemented — it just has no varying tokens to attend over.

This does not rule out cohort size or label noise, and it is not measured against real data. It moves step 4 of this issue near the front of the queue, since it is cheap to test: retain clip-level tokens for a subset and compare.

[duckyquang]

Independently verified the pooling claim above, and it holds bit-exactly — but the fused probe is a separate and more serious problem than I described. Correcting and extending.

1. The echo-only collapse is provable, and training cannot escape it.

AttentivePool(tiled tokens) == tokens.mean(1) bit-exactly across 56 configs (dim 8–1152 × T 2–32, float32 and float64); softmax weights deviate from 1/T by exactly 0.0. Bit-exact against the original vector at the repo's defaults (T=8 echo, T=4 ecg — any power of two); off by ≤1 ulp at T=3 or 5, which nothing uses.

More importantly, AttentivePool.query.grad is exactly 0.0, analytically: d(softmax_t)/dq_j ∝ softmax_t(x_tj − Σ_k softmax_k x_kj) = x_j − x_j = 0. After 500 Adam steps the query is bit-identical to its initialisation. EchoOnlyProbe and a mean-pool MLP sharing head init and dropout RNG produce bit-identical predictions and bit-identical head weights after 300 steps; the parameter counts differ by exactly dim — the entire dead query. So this is not "attention that happens to be uniform at init and might learn" — the gradient path does not exist.

2. The fused probe (M09) has zero cross-modal interaction, and its concat halves are swapped.

This is the part I had wrong. I expected the echo branch to be inert; it isn't, and what's actually happening is stranger.

nn.MultiheadAttention degenerates twice on identical tokens: identical keys force uniform weights regardless of the query, and identical values make the weighted sum equal the value regardless of the weights. Consequence at layers.py:64-65:

• echo_ctx is an affine function of the ECG vector only
• ecg_ctx is an affine function of the ECHO vector only

Verified by scaling the echo input 100× under a fixed dropout RNG state: echo_ctx is bit-identical. The echo→ecg_ctx map is full rank (64/64), so echo information does reach the head at full dimensionality — the fused probe is not an ECG-only model. What it actually is: a concat-MLP with a linear pre-map per modality and no cross-modal interaction whatsoever.

Dead parameters per fusion block: 2·(2·D²) for the q/k projections plus 2·D for the two pool queries. In eval mode those grads are exactly 0.0 bitwise; in train mode over 200 seeds, max |q grad| 1.3e-8 vs |v grad| 1.9 — a ratio of 1.7e-8, below float32 epsilon. With genuinely varying tokens, q_proj grad is 2.2e-1 and pool query grad norm is 87.2.

mask_echo/mask_ecg still behave correctly, so the missing-modality results in #64 aren't invalidated by this — but they measure degradation of a concat model, not of a cross-attention model.

A trap for anyone re-checking this: the LayerNorm at layers.py:49 spans the full dim*2 concat, so perturbing echo does move the echo_pool half of the output by 0.37 even though the pre-LayerNorm delta there is exactly 0.0. That's normalisation coupling and it reads as evidence the attention works. It isn't.

3. The attentive-vs-linear ablation measures nothing.

LinearEchoProbe is nn.Linear over tokens.mean(1); EchoOnlyProbe is MLPHead over a bit-identical vector. attentive_beats_linear_val_mae in results.json is an MLP-vs-linear capacity comparison on the same mean-pooled vector. Any reported gap should not be read as evidence for attentive pooling.

---

Scope: all of the above is a mechanism check on synthetic tensors — no real data exists on this machine and no MAE/AUROC is claimed or implied. It says the architecture cannot do what it claims; it does not say what fixing it is worth. That needs the manifest.

Fix in progress on feat/manifest-probe-training: opt-in clip-level token retention plus padding masks, so the tokens genuinely vary and the q/k paths come alive.

The M09 fusion degeneracy is arguably its own issue rather than a note on this one — happy to split it out.