Skip to content

Stop microdf's warning on formula reads with ADD or DIVIDE in a Microsimulation - #657

Closed
MaxGhenis wants to merge 8 commits into
masterfrom
stop-microseries-warning-on-option-reads
Closed

Stop microdf's warning on formula reads with ADD or DIVIDE in a Microsimulation#657
MaxGhenis wants to merge 8 commits into
masterfrom
stop-microseries-warning-on-option-reads

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Nov 14, 2026

Copy link
Copy Markdown
Contributor

This branch starts from #647's head (99e8a02e0) and adds two commits. #647 gives Microsimulation.calculate_add and calculate_divide the decode_enums argument that Population.__call__ passes with every options=[ADD] or options=[DIVIDE] read. Until #647 merges, this pull request's diff against master includes #647's changes.

Stop microdf's warning on a read with ADD or DIVIDE

In a Microsimulation, calculate_add and calculate_divide return a MicroSeries. In #647, Population.__call__ copies it into a NumPy array with result.to_numpy(copy=True) (population.py#L174-L175). microdf warns on every MicroSeries.to_numpy call (microseries.py#L321-L337):

UserWarning: Calling .to_numpy() on a MicroSeries returns a plain numpy array without weights.

microdf reports the warning at the caller's line, so each ADD or DIVIDE read in a formula raises it at population.py. With microdf-python 1.5.11, #647's example tests raise 20 of these warnings, and its property tests raised 492 in one run.

Population.__call__ now copies the values with numpy.array(result). numpy.array copies by default, so the formula still gets a new array that it can write to, and the call reaches neither MicroSeries.to_numpy nor MicroSeries.values, which warns the same way. The read returns the same values: #647's 37 tests pass unchanged.

Test that each Microsimulation calculate method takes Simulation's arguments

tests/core/test_microsimulation_calculate_signatures.py finds every calculate* method that Microsimulation overrides. For each one, it binds every keyword call that Simulation's method accepts to the override: the required arguments plus each subset of the optional ones, 20 calls in all.

Method Simulation Microsimulation on master Microsimulation on this branch
calculate variable_name, period=None, map_to=None, decode_enums=False variable_name, period=None, map_to=None, use_weights=True, decode_enums=True as on master
calculate_add variable_name, period=None, decode_enums=False variable_name, period=None, map_to=None, use_weights=True adds decode_enums=False (#647)
calculate_divide variable_name, period=None, decode_enums=False variable_name, period=None, map_to=None, use_weights=True adds decode_enums=False (#647)
calculate_dataframe variable_names, period=None, map_to=None variable_names, period=None, map_to=None, use_weights=True as on master

On master (003d09e), the four calls that pass decode_enums to calculate_add or calculate_divide fail, and the other 16 bind. If a later change gives one of Simulation's calculate methods an argument that the override lacks, the test fails again.

Tests

  • test_population_read_with_an_option_gives_no_microseries_warning reads a person variable and a household variable with each option and checks that no warning mentions MicroSeries. With result.to_numpy(copy=True) in place of numpy.array(result), its four cases fail.
  • With numpy.asarray(result), five cases of test_population_read_with_an_option_matches_a_simulation fail, because pandas 3 hands the formula a read-only view.
  • With decode_enums removed from Microsimulation.calculate_add, two cases of the signature test fail.
  • On this branch, with Python 3.13.9, pandas 3.0.6, NumPy 2.5.3 and microdf-python 1.5.11: 29 tests pass in test_microsimulation_option_reads.py, 21 in test_microsimulation_calculate_signatures.py and 12 in test_microsimulation_option_reads_property.py, and none of the three files reports a warning. ruff format --check . passes.
  • I ran neither core's full suite (make test) nor the documentation build on this branch. CI runs both.

🤖 Generated with Claude Code

MaxGhenis and others added 8 commits November 3, 2026 07:16
@MaxGhenis @claude
Accept decode_enums in Microsimulation.calculate_add and calculate_di…
d18c54c 
…vide

Population.__call__ passes decode_enums=False to calculate_add or
calculate_divide for every options=[ADD] or options=[DIVIDE] read in a
formula, as it has since 0abe43a. Simulation's methods take that
argument, but Microsimulation's overrides did not, so every such read
raised a TypeError in a Microsimulation.

Both overrides now take decode_enums, with Simulation's default of
False, and pass it to super(). Simulation.calculate_add and
calculate_divide accept the flag without using it, so every call that
worked before returns the same values.

Example tests run formulas that read with ADD and DIVIDE on a small
country-template system. Hypothesis properties check that a
Microsimulation's unweighted reads equal a Simulation's over generated
data and read sequences, and that formula results equal the same
arithmetic in NumPy. A strict xfail records that an Enum read with
DIVIDE returns labels in a Microsimulation and an EnumArray in a
Simulation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis @claude
Return NumPy arrays from formula reads with ADD or DIVIDE
753bef2 
In a Microsimulation, calculate_add and calculate_divide return a
MicroSeries, and pandas pairs two Series' values by index label.
GroupPopulation.project labels each member's value with its group's
position. So a formula that projected such a read and combined it with
another Series raised a ValueError, or, when every group had one member
and the data listed them out of group order, paired members with another
group's value. policyengine-us's wi_wheap_categorically_eligible has
that shape (lines 38-44).

Population.__call__ now returns the NumPy array a Simulation returns for
these reads. They raised a TypeError in every Microsimulation on master,
so no value that computed before moves.

The fixture now takes each person's household, so data can list members
in any order, and adds fee_or_all_paid, a reader with the PE-US
formula's shape. The Enum DIVIDE test now records both classes' results
instead of failing as a strict xfail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis @claude
Read options over periods they accept in the property, in any person …
141219e 
…order

Population calls and calculate_add and calculate_divide requests now
pair each variable with periods its option accepts, and population calls
read only variables the population can read, so they return values
instead of mostly raising in both classes. The data lists people in any
order. hypothesis.event reports the mix of values, zeros and errors.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis @claude
Test that option reads return arrays, and check every formula in each…
99a808e 
… run

A new property checks that each population call a formula makes with
ADD or DIVIDE returns a NumPy array in both classes. The NumPy-arithmetic
property now runs for each formula, 25 examples each: with 100 examples
drawn across all ten, one seeded run never drew fee_or_all_paid or
members_grant_divided.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis @claude
Return a writable copy from formula reads with ADD or DIVIDE
885be7a 
pandas 3 returns a read-only view from Series.to_numpy(), so in a
Microsimulation a formula that wrote to such a read would raise
"ValueError: assignment destination is read-only" where a Simulation
computes. Copy the values instead, and test that population calls return
writable arrays in both classes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis @claude
Draw yearly inputs in multiples of 12 in the option-read properties
99e8a02 
The NumPy reference sums a household's grants and then divides by 12,
while a household's DIVIDE read divides each member's float32 grant by 12
and then sums. Whole grants that nearly cancel, such as 9,990 and -10,000
in one household, made members_grant_divided return -0.83331299 against
-0.83333333, past rtol=1e-6, so the property could fail by chance. With
yearly amounts in multiples of 12, every monthly share is whole, every sum
and share is exact in float32, and the property checks exact equality.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis @claude
Test that Microsimulation's calculate methods take Simulation's argum…
9b538bb 
…ents

Population.__call__ sends a formula's read to calculate, calculate_add or
calculate_divide with one set of keyword arguments. For each calculate
method that Microsimulation overrides, the test binds every keyword call
that Simulation's method takes to the override. On master, the four calls
that pass decode_enums to calculate_add or calculate_divide fail.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@MaxGhenis @claude
Copy option reads with numpy.array, so microdf raises no warning
9ddef01 
In a Microsimulation, Population.__call__ gets a MicroSeries from a read
with ADD or DIVIDE and copied it with result.to_numpy(copy=True). microdf
warns on every MicroSeries.to_numpy call that the result carries no
weights, and it reports the warning at the caller's line, so each such
read in a formula raised a UserWarning at population.py.

numpy.array copies the values into a new array that the formula can write
to, and it calls neither MicroSeries.to_numpy nor MicroSeries.values.

The new test reads a person variable and a household variable with each
option and checks that no warning mentions MicroSeries.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@MaxGhenis MaxGhenis closed this Nov 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@MaxGhenis