Skip to content

[E08] Fairness stratification artifacts #65

Description

@duckyquang

Task metadata

Field Value
Task ID E08
Category Evaluation
Priority High
Complexity Low
Est. days 2
Blocked by M10, E07

Features / method

Produce the fairness stratification artifacts that E03 never emitted.

Description

E03 (#9) was closed and src/primed_ai/evaluation/fairness.py exists and is unit-tested, but results/fairness/ is an empty directory. There is no fairness number anywhere in the repo, and equity is one of the three evaluation pillars in the README. This closes that gap.

Run the audit on the canonical predictions from E07, using the demographics already joined into the manifest (sex, age, race columns are present).

Steps:

  1. Stratify the canonical test predictions by sex, age band, and race.
  2. Report per-stratum LVEF MAE and EF<=40% AUROC, with per-stratum n.
  3. Flag strata too small for a stable estimate rather than reporting a number that looks solid. The test split is 248 rows across 201 subjects, so some cells will be thin — evaluation/aggregate.py already has the small-stratum marker.
  4. Repeat the stratification under echo_dropped, since an equity gap that only appears when echo is missing is the deployment-relevant one.
  5. Note the known MIMIC gender-curation bias in the output docs.

Acceptance criteria:

  • results/fairness/ contains a real result file with per-stratum metrics and per-stratum n.
  • Strata below the reportable threshold are marked, not silently included.
  • Fairness gaps reported for full and echo_dropped conditions.
  • README fairness claim backed by an artifact that exists.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions