Task metadata
| Field |
Value |
| Task ID |
E08 |
| Category |
Evaluation |
| Priority |
High |
| Complexity |
Low |
| Est. days |
2 |
| Blocked by |
M10, E07 |
Features / method
Produce the fairness stratification artifacts that E03 never emitted.
Description
E03 (#9) was closed and src/primed_ai/evaluation/fairness.py exists and is unit-tested, but results/fairness/ is an empty directory. There is no fairness number anywhere in the repo, and equity is one of the three evaluation pillars in the README. This closes that gap.
Run the audit on the canonical predictions from E07, using the demographics already joined into the manifest (sex, age, race columns are present).
Steps:
- Stratify the canonical test predictions by sex, age band, and race.
- Report per-stratum LVEF MAE and EF<=40% AUROC, with per-stratum n.
- Flag strata too small for a stable estimate rather than reporting a number that looks solid. The test split is 248 rows across 201 subjects, so some cells will be thin —
evaluation/aggregate.py already has the small-stratum marker.
- Repeat the stratification under
echo_dropped, since an equity gap that only appears when echo is missing is the deployment-relevant one.
- Note the known MIMIC gender-curation bias in the output docs.
Acceptance criteria:
results/fairness/ contains a real result file with per-stratum metrics and per-stratum n.
- Strata below the reportable threshold are marked, not silently included.
- Fairness gaps reported for
full and echo_dropped conditions.
- README fairness claim backed by an artifact that exists.
Task metadata
E08Features / method
Produce the fairness stratification artifacts that E03 never emitted.
Description
E03 (#9) was closed and
src/primed_ai/evaluation/fairness.pyexists and is unit-tested, butresults/fairness/is an empty directory. There is no fairness number anywhere in the repo, and equity is one of the three evaluation pillars in the README. This closes that gap.Run the audit on the canonical predictions from E07, using the demographics already joined into the manifest (
sex,age,racecolumns are present).Steps:
evaluation/aggregate.pyalready has the small-stratum marker.echo_dropped, since an equity gap that only appears when echo is missing is the deployment-relevant one.Acceptance criteria:
results/fairness/contains a real result file with per-stratum metrics and per-stratum n.fullandecho_droppedconditions.