[MaxGhenis]

On master, a formula that reads a variable with options=[ADD] or options=[DIVIDE] raises this error in a Microsimulation, though a Simulation computes it:

TypeError: Microsimulation.calculate_add() got an unexpected keyword argument 'decode_enums'

A policyengine-us impact run hit the error on spm_unit("snap", period, options=[ADD]) in mt_liheap_categorical_eligibility.py (PolicyEngine/policyengine-us#10141 (comment)). At PolicyEngine/policyengine-us@2b39b88e2f, 21 formulas read with options=[ADD], and each raised the error whenever it ran in a Microsimulation. In PolicyEngine/policyengine-us#10730, we replaced the options=[ADD] reads in 26 formula files with plain reads to avoid the error. No formula on policyengine-us main (PolicyEngine/policyengine-us@61c743cb61) reads with an option.

Cause

Population.__call__ has passed decode_enums=False with every read since 0abe43a (December 2022), and it sends reads with options=[ADD] or options=[DIVIDE] to calculate_add or calculate_divide (population.py#L147-L163). Simulation.calculate_add and calculate_divide take it (simulation.py#L2057-L2062, #L2126-L2131), but Microsimulation overrides both without it (microsimulation.py#L84-L118), so Python rejects the call.

Plain reads go to calculate, which takes the flag. When a formula reads a monthly flow over a year or a yearly flow over a month, Simulation._calculate calls calculate_add or calculate_divide without the flag (simulation.py#L1741-L1753), so those reads kept working.

Fix

Both overrides now take decode_enums: bool = False, Simulation's default, and pass it to super(). Simulation.calculate_add and calculate_divide accept the flag without using it, and the new parameter goes last, so every direct call that worked before returns the same values.

Population.__call__ now returns a NumPy array for a read with ADD or DIVIDE, as a Simulation does (population.py#L167-L178). In a Microsimulation the overrides return a MicroSeries, and pandas pairs two Series' values by index label. GroupPopulation.project labels each member's value with its group's position (group_population.py#L366-L370). So a formula that projected such a read and combined it with another Series would raise microdf's ValueError: Cannot propagate weights unambiguously to these rows (microdf 1.5.11 and 1.5.12). If every group had one member and the data listed them out of group order, members would get other groups' values without an error. At 2b39b88e2f, policyengine-us's wi_wheap_categorically_eligible projected its ADD read of snap to members and combined it with a person-level read of ssi (lines 33-44).

Population.__call__ copies the values with numpy.array, which returns a new writable array that shares no memory with the Series and keeps its dtype for numbers and booleans. pandas 3 returns a read-only view of a Series' values, so a formula that wrote to the view would raise ValueError: assignment destination is read-only, while the same formula computes in a Simulation. From version 1.2.2, microdf warns on every MicroSeries.to_numpy or .values call that the result carries no weights (PolicyEngine/microdf#278), and numpy.array calls neither.

On master these reads raised the TypeError in every core Microsimulation, and a Simulation already returns arrays, so the conversion moves no value that computed before. Converting in Population.__call__ leaves direct calls alone: calculate_add(..., use_weights=False) worked on master and still returns a MicroSeries.

Invariants and tests

tests/fixtures/microsimulation_option_reads.py adds to the country template ten formulas that read with ADD or DIVIDE. They read person and household variables, sum members' values into households, compare a sum with zero and call the add helper. One of them, fee_or_all_paid, projects a household's ADD read to its members. tests/core/test_microsimulation_option_reads_property.py checks three properties with Hypothesis:

• A Microsimulation's unweighted reads equal a Simulation's, or both raise the same type of error. Hypothesis generates one to four households of one to three people, listed in any order, with weights, whole monthly amounts, yearly amounts in multiples of 12 and up to six reads in sequence. The reads cover the formulas, the population calls they make, calculate_add and calculate_divide with either value of decode_enums, and plain reads at months, years and a three-month period. Population calls and the two methods read each variable over periods its option accepts. hypothesis.event records each read's outcome, and in two 200-example runs, population calls returned nonzero values in 19% and 31% of examples and raised in none.
• Each of those population calls returns a NumPy array that a formula can write to, in both classes (50 examples).
• Each formula returns what the same arithmetic gives in NumPy, in both classes (25 examples for each formula). The test compares them without a tolerance, since whole monthly amounts and yearly amounts in multiples of 12 keep every sum and monthly share exact in float32.

Example tests in tests/core/test_microsimulation_option_reads.py check hand-computed values for each formula and for fee_or_all_paid on two one-person households listed in reverse order. They also check that seven population calls return writable arrays equal to a Simulation's, the weights on a weighted result, and both methods with each value of decode_enums. Four more check that population calls raise no microdf warning about MicroSeries values. microdf added that warning in 1.2.2, so those four can fail only under microdf 1.2.2 or later. uv.lock pins 1.2.1, and CI installs without the lock and resolves 1.5.12.

tests/core/test_microsimulation_calculate_signatures.py binds each keyword call that Simulation.calculate, calculate_add, calculate_divide and calculate_dataframe take, with every subset of their optional arguments, to Microsimulation's override. It checks only that each keyword call binds. So it can't catch an override that takes a flag and ignores it, a default that differs from Simulation's, a keyword that Population.__call__ passes and Simulation lacks (unweighted), or an override in a country package. Positional calls already differed on master: Microsimulation.calculate_add takes map_to third, where Simulation.calculate_add takes decode_enums. So Microsimulation.calculate_add(name, period, True) passes True as map_to and raises KeyError: True. No caller in core or on policyengine-us main passes a third argument by position.

On master, 44 of the 62 new tests fail (21 signature tests, 29 example tests and 12 property tests). In 40 the Microsimulation raises the TypeError, and the signature test rejects the 4 keyword calls that pass decode_enums to calculate_add or calculate_divide. The other 18 check the NumPy reference alone or bind calls that master's overrides already take. Without the conversion in Population.__call__, 10 fail, and Hypothesis falsifies all three properties. fee_or_all_paid raises the ValueError, the reverse-order case returns [True, False] against the Simulation's [False, True], and five population calls return a MicroSeries. With .to_numpy() in place of the copy, 10 fail: five population calls return read-only arrays, four calls raise microdf's warning, and Hypothesis falsifies the second property. With .to_numpy(copy=True), the four warning tests fail.

In CI, on this branch merged with master, core's full suite passes, the 62 new tests included, on Ubuntu and Windows under Python 3.11 to 3.14 with microdf 1.5.12. ruff format --check . passes too.

policyengine-us

No formula on policyengine-us main reads with an option, so this change moves no policyengine-us result there. With it, a policyengine-us formula can again read with options=[ADD] or options=[DIVIDE] in a Microsimulation. No policyengine-us Microsimulation has run on this branch.

A formula can still get a MicroSeries from a plain read that Simulation._calculate sends to calculate_add, such as add(person, period, ["ssi"]). That MicroSeries labels values by position, so it pairs correctly with arrays and with unprojected reads of its entity. On main, wi_wheap_categorically_eligible projects a plain read of snap and ORs it with add(person, period, ["ssi"]) > 0, so the second gap below applies to it. By our reading of the code, a Microsimulation would raise microdf's ValueError there only when this formula reads both ssi and snap over the year before any other formula does.

Remaining gaps

• Simulation.calculate_add and calculate_divide read each term with self.calculate, which in a Microsimulation decodes Enums and applies weights by default (simulation.py#L2093-L2096, #L2152-L2160). So an Enum read with DIVIDE over a year returns member names in a Microsimulation and an EnumArray in a Simulation. On master that read raised the TypeError. Now a formula that compares it with a member, as read == Tenure.OWNER does in the test, gets False for every person, where a Simulation gets [True, False, True]. test_enum_read_with_divide_over_a_year_gives_member_names_in_a_microsimulation records both results. For the same reason, an Enum read with ADD, or with DIVIDE over a month, raises a TypeError in a Microsimulation, and calculate_add(..., use_weights=False) returns a MicroSeries that still carries weights.
• The first time a Microsimulation computes a plain read that Simulation._calculate sends to calculate_add or calculate_divide, the read still returns a weighted MicroSeries to the formula, as on master. A formula that projects one and combines it with another Series pairs values by label too. Once the Microsimulation has cached the value, after an earlier plain read or a read with ADD or DIVIDE, a later plain read returns the cached array instead. The holder stores it with numpy.asarray, which under pandas 3 gives a read-only view of the MicroSeries. So a formula that writes to that read raises ValueError: assignment destination is read-only, while a Simulation computes the same formula. On master, a formula that wrote to a second such read raised the same error. No formula on policyengine-us main writes in place to a value it takes straight from an entity read.
• In a Simulation, a read with ADD or DIVIDE can hand the formula the array the simulation stores. DIVIDE over a year returns the stored yearly value, so a write to it changes every later read. ADD over a year of a monthly flow and DIVIDE over a month of a yearly flow return the value they cache. A write to that value changes later plain reads at that period, while a later read with the option computes the value again. In a Microsimulation, Population.__call__ copies these reads, so after such a write the two classes return different values. No formula on policyengine-us main reads with ADD or DIVIDE.
• For a simulation class that doesn't subclass core's Microsimulation but has "micro" in its name, such as policyengine-uk's, Population.__call__ passes unweighted=True to the calculate methods. Simulation.calculate_add and calculate_divide don't accept that argument, so a read with ADD or DIVIDE in such a class raises a TypeError. On policyengine-uk main (PolicyEngine/policyengine-uk@d3db7ac07d), only two test reads use an option, and both expect policyengine-uk's StoredDrawError before the read reaches core.

impact: no output moves for any variable that computed before; formulas that read with options=[ADD] or options=[DIVIDE] now compute in a Microsimulation (policyengine-us main has none since PolicyEngine/policyengine-us#10730)

axiom: n/a: core infrastructure fix

🤖 Generated with Claude Code

[MaxGhenis]

I'm handing this PR to the US + core hub to land. The independent review approves, and CI passes at 99e8a02e0.

• Head: 99e8a02e0. Five commits after d18c54c6b act on both review rounds. 753bef26d returns NumPy arrays from formula reads with ADD or DIVIDE, and 885be7a81 copies them so formulas can write to them. 141219eb6 and 99a808e6e extend the tests, and 99e8a02e0 draws the properties' yearly inputs in multiples of 12, so float32 rounding can't fail the NumPy comparison. I also updated the title and description.
• Independent review: Opus 5.5 reviewed the PR in two rounds. Round 1, a Subfleet review job, requested changes at d18c54c6b (~/reviews/core-647-microsim-decode-enums/review-r1.md): a projected MicroSeries paired members with other groups' values. Round 2, an in-session agent, reviewed d18c54c6b..99e8a02e0, asked for changes to the description only, and approved the revised description at the same head (review-r2.md in the same folder).
• Passed locally at 99e8a02e0: the 37 new tests, core's full suite (2,886 passed, 4 skipped, 3 xfailed), ruff format --check ., and the three properties at 1,000, 1,000 and 300 examples for each formula.
• CI at the head: all 19 checks passed in run 42896284226. GitHub reports MERGEABLE, with merge state CLEAN.
• Merge: squash.
• Depends on: nothing. No hold or decision applies.
• Follow-ups: the four gaps under "Remaining gaps" in the description. None of them changes what the 21 policyengine-us formulas return.

[MaxGhenis]

US + core hub merge audit, core#647 at 43b27e4da95bbcf2371b7a318aa4832f02568dba (squash): Fix formula reads with ADD or DIVIDE in a Microsimulation.

• Reviews:
  • An Opus 5.5 reviewer requested changes at d18c54c6b7: a projected MicroSeries paired members with other groups' values. 753bef26d returns NumPy arrays from formula reads with ADD or DIVIDE, as a Simulation does, and 885be7a81 copies them so a formula can write to them.
  • A second Opus 5.5 review of d18c54c6b..99e8a02e0 found the code right and asked for description changes only. Its probes found that every Microsimulation read with ADD or DIVIDE returned a writable array that shared no memory with stored values.
  • Two commits followed. 9b538bb58 tests that each Microsimulation calculate method takes the keyword arguments Simulation's takes, and 9ddef016f copies with numpy.array, so microdf raises no warning about MicroSeries values.
  • A third Opus 5.5 review read the whole diff at 9ddef016f. It found the code right and asked for a description that matches the head and for one changelog fragment. 43b27e4da deletes the second fragment and states the first in one sentence, and the description now gives this head's counts, CI run and policyengine-us impact.
  • A fourth Opus 5.5 check at 43b27e4da found the description, the changelog fragment and CI at this head consistent with the code and tests.
• Tests: the PR adds 62 tests: 21 signature tests, 29 example tests and 12 property tests. On master, 44 of the 62 fail. With .to_numpy() in place of the copy, 10 fail, and with .to_numpy(copy=True), 4 fail.
• CI: run 43639955708 passed 19 of 19 jobs at this head, merged with master 003d09e15. Each of the four Ubuntu jobs passed 3,880 tests and each of the four Windows jobs passed 3,845, under Python 3.11 to 3.14 with microdf 1.5.12. All eight ran the 62 new tests.
• Classification: a crash fix. On master, a formula read with options=[ADD] or options=[DIVIDE] raises TypeError in every core Microsimulation, so the change moves no value that computed before. No formula on policyengine-us main reads with an option since Add a national LIHEAP total, an Illinois regular heating payment and LIHEAP cooling registry notes policyengine-us#10730, so the change moves no policyengine-us result there. The description's "Remaining gaps" section lists four gaps that the PR leaves open.
• PE-US core gate: with this head merged into master 003d09e15 and PE-US main 7c516c74bb, the 43 PE-US core test files that exercise subsampling, branches, inputs or holders passed all 1,203 tests.
• Gates the lander checked at merge:
  • every check passes at the head;
  • MERGEABLE;
  • not a draft;
  • no CHANGES_REQUESTED review;
  • the merge pins the head (--match-head-commit).
• Axiom: n/a; the PR changes no policy.

🤖 Generated with Claude Code