Add explainers/bootstrap-confidence-intervals.md.
Every fairness gap this repo reports - in every metric explainer, every frozen benchmark number - comes with a 95% confidence interval and a p-value, computed by faircode/significance.py's bootstrap resampling (CI) and permutation test (p-value); see that module's own docstring for the exact methodology. Nothing explains what these actually are or why a raw gap number is meaningless without them, even though 'is this gap real or just noise on a small subgroup' is the single most repeated question this repo's own detection code exists to answer.
Suggested structure: one-sentence definition -> why it matters for fairness (small subgroups, the difference between a gap and a significant gap) -> core concept (resampling with replacement for the CI, label-shuffling for the p-value - both distribution-free, assuming nothing about the underlying data shape) -> concrete example tied to a repo audit, using real frozen CI/p-value numbers already in paper/results-frozen/results_fairness.csv -> runnable detection code (a minimal from-scratch bootstrap + permutation implementation; faircode/significance.py itself is a frozen file and shouldn't be imported directly, see other explainers' detection-code sections for the established pattern of self-contained code) -> limitations (small-n instability, multiple-comparisons risk across six metrics x many protected attributes) -> related concepts/projects/further reading.
Related: base-rate-fallacy, equal-opportunity, intersectional-bias - all three already lean on CI/p-value numbers without explaining where they come from.
See CONTRIBUTING.md: add explainers/<slug>.md plus a one-line entry in assets/explainers-data.json; the build script generates the page.
Freeze-safe: explainers are fully open for contribution during the paper freeze. If you quote any Fair Code benchmark result, use the frozen numbers in paper/results-frozen/ (never re-run your own) - see CLAUDE.md.
Add
explainers/bootstrap-confidence-intervals.md.Every fairness gap this repo reports - in every metric explainer, every frozen benchmark number - comes with a 95% confidence interval and a p-value, computed by
faircode/significance.py's bootstrap resampling (CI) and permutation test (p-value); see that module's own docstring for the exact methodology. Nothing explains what these actually are or why a raw gap number is meaningless without them, even though 'is this gap real or just noise on a small subgroup' is the single most repeated question this repo's own detection code exists to answer.Suggested structure: one-sentence definition -> why it matters for fairness (small subgroups, the difference between a gap and a significant gap) -> core concept (resampling with replacement for the CI, label-shuffling for the p-value - both distribution-free, assuming nothing about the underlying data shape) -> concrete example tied to a repo audit, using real frozen CI/p-value numbers already in
paper/results-frozen/results_fairness.csv-> runnable detection code (a minimal from-scratch bootstrap + permutation implementation;faircode/significance.pyitself is a frozen file and shouldn't be imported directly, see other explainers' detection-code sections for the established pattern of self-contained code) -> limitations (small-n instability, multiple-comparisons risk across six metrics x many protected attributes) -> related concepts/projects/further reading.Related: base-rate-fallacy, equal-opportunity, intersectional-bias - all three already lean on CI/p-value numbers without explaining where they come from.
See CONTRIBUTING.md: add
explainers/<slug>.mdplus a one-line entry inassets/explainers-data.json; the build script generates the page.