Skip to content

[E09] EF≤40% calibration artifacts #66

Description

@duckyquang

Task metadata

Field Value
Task ID E09
Category Evaluation
Priority Medium
Complexity Low
Est. days 1
Blocked by M10, E07

Features / method

Produce the EF<=40% calibration artifacts that E04 never emitted.

Description

Same situation as E08: E04 (#10) is closed, src/primed_ai/evaluation/calibration.py implements Platt scaling from continuous LVEF to EF<=40% probabilities, and there is no calibration output in the repo.

A deployment-risk paper that reports AUROC without calibration is only telling half the story. AUROC is rank-based and says nothing about whether the probabilities are usable as a risk score at the bedside.

Steps:

  1. Fit the Platt scaler on val predictions, apply to test. Do not fit on test.
  2. Report expected calibration error and a reliability diagram for the full condition.
  3. Repeat under echo_dropped and ecg_dropped — the question is whether the model stays honest about its own confidence when a modality goes missing, or gets overconfident.
  4. Write the results into results/ with provenance.

Acceptance criteria:

  • ECE and reliability diagram for all three conditions.
  • Scaler fit on val only, stated explicitly in the output.
  • Result file records checkpoint, seed, and prediction source.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions