Skip to content

Fix calibration dict/dict multi-population contract #290

Description

@uriahf

Problem

create_calibration_curve() currently has inconsistent internal expectations for probs/reals dictionaries.

For matching population keys such as:

probs = {
    "Train": train_probs,
    "Test": test_probs,
}
reals = {
    "Train": train_y,
    "Test": test_y,
}

_make_deciles_dat_binary() computes the sum of all population sizes and requires each probability vector to have that total length. This rejects the natural keyed-population representation, including the example currently shown in Getting Started.

At the same time, _calculate_smooth_curve() and the time-dependent dataframe builder already treat matching keys as population-by-population pairs. The two calibration paths therefore use incompatible contracts.

Desired contract

When probs and reals are dictionaries with matching keys, pair each population independently:

probs["Train"] <-> reals["Train"]
probs["Test"]  <-> reals["Test"]

Each pair must have matching lengths, but different populations may have different sample sizes.

Acceptance criteria

  • create_calibration_curve() works for equal-size matching-key populations.
  • create_calibration_curve() works for unequal-size matching-key populations.
  • mismatched keys raise a clear ValueError.
  • within-population probability/outcome length mismatches raise a clear ValueError naming the population.
  • both calibration_type="discrete" and calibration_type="smooth" work under the same public input contract.
  • create_calibration_curve_times() continues to work with the same matching-key population contract after filtering by horizon.
  • regression tests cover equal and unequal population sizes.

Scope

This should be fixed before treating unequal-population calibration as a documented limitation. It is a correctness/API-consistency issue, not just a docs gap.

Related docs PR: #289

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions