Task metadata
| Field |
Value |
| Task ID |
E09 |
| Category |
Evaluation |
| Priority |
Medium |
| Complexity |
Low |
| Est. days |
1 |
| Blocked by |
M10, E07 |
Features / method
Produce the EF<=40% calibration artifacts that E04 never emitted.
Description
Same situation as E08: E04 (#10) is closed, src/primed_ai/evaluation/calibration.py implements Platt scaling from continuous LVEF to EF<=40% probabilities, and there is no calibration output in the repo.
A deployment-risk paper that reports AUROC without calibration is only telling half the story. AUROC is rank-based and says nothing about whether the probabilities are usable as a risk score at the bedside.
Steps:
- Fit the Platt scaler on val predictions, apply to test. Do not fit on test.
- Report expected calibration error and a reliability diagram for the
full condition.
- Repeat under
echo_dropped and ecg_dropped — the question is whether the model stays honest about its own confidence when a modality goes missing, or gets overconfident.
- Write the results into
results/ with provenance.
Acceptance criteria:
- ECE and reliability diagram for all three conditions.
- Scaler fit on val only, stated explicitly in the output.
- Result file records checkpoint, seed, and prediction source.
Task metadata
E09Features / method
Produce the EF<=40% calibration artifacts that E04 never emitted.
Description
Same situation as E08: E04 (#10) is closed,
src/primed_ai/evaluation/calibration.pyimplements Platt scaling from continuous LVEF to EF<=40% probabilities, and there is no calibration output in the repo.A deployment-risk paper that reports AUROC without calibration is only telling half the story. AUROC is rank-based and says nothing about whether the probabilities are usable as a risk score at the bedside.
Steps:
fullcondition.echo_droppedandecg_dropped— the question is whether the model stays honest about its own confidence when a modality goes missing, or gets overconfident.results/with provenance.Acceptance criteria: