[D07] Paired cohort recovery — 4,103 shared subjects to 1,208 rows

[duckyquang]

Task metadata

Field	Value
Task ID	D07
Category	Data
Priority	High
Complexity	Medium
Est. days	3

Features / method

Recover paired cohort size. 4,103 shared subjects currently collapse to 1,208 rows.

Description

The join summary in docs/echo_hubert_loader.md shows where the cohort goes:

Echo studies: 7,251 (4,585 subjects)
ECG studies: 800,035 (161,352 subjects)
Shared Echo + ECG subjects: 4,103
Echo studies among shared subjects: 6,617
Subject-only candidate Echo+ECG pairs: 116,754
Final joined manifest rows: 1,208

6,617 echo studies belong to subjects who also have an ECG, and the final manifest keeps 1,208. TECHNICAL.md 3.1 predicted "a few thousand to tens of thousands" of paired rows, so the current cohort is at the bottom of the expected range and below it on the echo side.

Cohort size plausibly explains the E10 baseline gap, and it is the hard ceiling on Task A's statistical power — a 248-row test split makes stratified fairness cells thin and bootstrap CIs wide. Worth understanding before treating the current numbers as final.

The first question is which constraint is actually binding: the 48h pairing window, the LVEF label requirement, the admission-matching mode, or the echo embedding coverage (7,251 studies pooled from the ~525K clip corpus, which is itself a small slice).

Steps:

1. Produce the funnel from 6,617 candidate echo studies down to 1,208 with a count dropped at each step. build_cohort.py already emits cohort_funnel.json — read it rather than reconstructing.
2. Isolate the binding constraint. Check the pairing window (24h vs 48h vs admission mode) and the LVEF label join separately.
3. Check whether echo embedding coverage is the limiter. 7,251 echo studies out of the full corpus is a small fraction — establish whether more studies can be pooled from the available EchoJEPA shards.
4. Run a sensitivity analysis on the pairing window and report cohort size vs window, closing the open decision in TECHNICAL.md 11.
5. If a defensible relaxation gets materially more rows, rebuild the manifest and note that M10 should be rerun on it.

Acceptance criteria:

• Exclusion counts published for every step from candidate pairs to final cohort.
• The binding constraint named, with numbers.
• Pairing window decision (24h vs 48h vs admission) fixed and justified rather than left open.
• A clear yes/no on whether cohort size can be materially increased without weakening the pairing protocol.

[duckyquang]

Bounding the prize before anyone spends BigQuery time on this. All arithmetic below is on the counts already published in docs/echo_hubert_loader.md — no new data, nothing measured here.

echo studies among shared subjects : 6,617
final manifest rows                : 1,208
lost in pairing + label join       : 5,409   (81.7%)
ceiling without more extraction    : 6,617 rows = 5.48x current
rows per shared subject            : 0.29 now, 1.61 at the ceiling

Two things follow.

1. There is real headroom, but it is capped at 5.5x. Echo embedding coverage is 7,251 studies, of which 6,617 belong to subjects who also have an ECG. Even a perfect pairing protocol — every echo study matched, every label present — cannot exceed 6,617 rows. So the honest framing of this issue is "recover up to 5.5x", not "get to tens of thousands". TECHNICAL.md 3.1's "a few thousand to tens of thousands" is unreachable from the current embedding set no matter what happens to the window.

Getting past 6,617 is a different task: extracting EchoJEPA embeddings for more studies. Worth separating from this one.

2. 81.7% is lost between the candidate set and the final join, and we cannot yet say to what. There are 116,754 subject-level candidate pairs feeding 1,208 rows, so the timestamp window and the LVEF label requirement are together doing enormous filtering — but the published summary doesn't split them, and cohort_funnel.json isn't in the repo (it's gitignored). Which of the two dominates completely changes the fix:

• if the LVEF label is binding, relaxing the window buys nothing and the answer is label recovery
• if the 24–48h window is binding, a sensitivity sweep is the whole job
• if it is multi-match dedup, the fix is keeping more than one pair per subject — note 0.29 rows per shared subject today against a 1.61 ceiling, which is the shape you would expect if dedup is aggressive

I can't distinguish these without the funnel or BigQuery, and I'd rather not guess — the three point at different work.

Cheapest next step, for whoever has access: read cohort_funnel.json from an existing build_cohort.py run. It already emits per-step exclusion counts, so this needs no new query — it answers step 1 and probably step 2 outright.

Scope note: this issue's premise is that cohort size may explain the #67 baseline gap. Worth holding that loosely — 5.5x more rows would help statistical power and would narrow the fairness strata in #65, but the echo branch was provably inert for every run measured so far (#67, #72), so cohort size is not the only candidate and possibly not the main one.

[duckyquang]

Checked what we can answer from the local repo plus the PR #73 files. There is still no cohort_funnel.json or window-sensitivity output available locally, and the manifest summary only has final joined counts. It does not split the 6,617 candidate echo studies down to 1,208 by LVEF label, pairing window, admission filter, or dedup.

So the binding constraint is still not identifiable from the files in hand. I do not want to guess here because each possible answer points to different work.

Current status:

• Known ceiling without more EchoJEPA extraction: 6,617 rows.
• Current manifest: 1,208 rows.
• Missing evidence: the actual build_cohort.py funnel output, or a new BigQuery run/sensitivity sweep.

Next real step is still the cheap one: get cohort_funnel.json from the original cohort build if it exists. If not, rerun build_cohort.py with the current defaults and then do the 24h/48h/admission sensitivity sweep from those same inputs.

[duckyquang]

Added scripts/run_cohort_sensitivity.py on PR #73 so the D07 window/admission sweep is one real command instead of a hand-run sequence. It reuses build_cohort.py's existing SQL helpers and writes one funnel per scenario plus summary.csv/json.

Default scenarios:

python scripts/run_cohort_sensitivity.py --project "$GCP_PROJECT_ID"

That runs 24h, 48h, and admission-mode funnels without downloading cohorts. The new unit coverage checks the summary math (final_rows, gain vs 24h baseline, etc.).

I also tried to run it here. This machine has no gcloud, no ADC, and no GCP_PROJECT_ID, so the real BigQuery run stops before querying:

RuntimeError: Set GCP_PROJECT_ID to your billing GCP project (export GCP_PROJECT_ID=<project> and run `gcloud auth application-default login`).

So the code path is now in place and tested, but the actual binding-constraint answer still needs a credentialed BigQuery environment or the original cohort_funnel.json.