Skip to content

Lock policyengine-uk 2.174.1 and follow what its releases change - #1511

Merged
juaristi22 merged 18 commits into
mainfrom
uk-pe-uk-bump-2174
Nov 5, 2026
Merged

juaristi22 merged 18 commits into
mainfrom
uk-pe-uk-bump-2174

Conversation

@juaristi22

@juaristi22 juaristi22 commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

This PR also carries #1512 (the occupation imputation), which was merged into this branch at 86043f2. Closes #1490.

The UK lock moves from policyengine-uk 2.165.1 to 2.174.1, the newest release on PyPI (microcosm#1095). 2.174.1 carries the SOC 2020 occupation inputs of 2.173.0 (PolicyEngine/policyengine-uk#2858), so #1490's occupation PR can stack on this one. policyengine-core stays at 3.33.3; uv.lock moves policyengine-uk alone. The plan, with what each release from 2.165.2 needs, is María's of 4 November; she took its recommended options. The branch is rebased on main after #1391 (875506e8a), and its generated files are regenerated on the merged tree.

The seven new inputs, signed at their defaults

Each is in UK_ENGINE_DEFAULT_INPUTS with the error it can make, for María's signature:

What the build's engine reads return now

  • 2.165.2: the producer declares the savings income projection rule obr_savings_income_2022_2025, in every H5 it writes and in every dataset it hands the engine in process. The engine then grows savings interest, other investment income and ISA interest by the OBR's savings income per person in 2022-23 to 2025-26, the path on which the SPI income stage and the SPI Table 3.7 interest targets already grow (María's ruling of 2026-10-29). Before, it grew the frame's interest to 2025 by GDP per head. A file that declares no rule still loads as GDP per head and keeps that reading, and the export check refuses a release without the producer's rule. The SPI income receipt names the engine's new projection index for savings interest and other investment income, as 2.165.2's uprating_indices.yaml does; the older two-leg savings interest concept keeps GDP per head, and no SPI target follows it.
  • 2.166.0: Income Tax reductions no longer offset the High Income Child Benefit Charge or the annual allowance charge.
  • 2.168.1: the declared energy spending grows by one retail energy price index per fuel instead of CPI. The energy price basis engine test reads each fuel's index.
  • 2.170.0: a couple who both care for the same person give Carer's Allowance to the partner with the larger award.
  • 2.170.4: the gambling duty inputs grow per person by their duty bases instead of keeping their cash level.
  • 2.172.1: Housing Benefit keeps the Working Tax Credit disregard back before the childcare deduction in tax-credit years.
  • 2.173.3 (Treat income derived from capital as capital in the legacy means tests policyengine-uk#1995): the legacy means tests and Council Tax Reduction treat income from capital as capital. They count sublet_income less £20 a week and no longer count property_income; Council Tax Reduction counts land and property other than the home as capital. The build keeps the FRS sub-letting rent (SUBRENT) in property_income, as Move SUBRENT from property_income to sublet_income, after policyengine-uk reads sublet_income policyengine-uk-data#508 orders: income tax and HBAI income still count it, and the means tests count it once that issue moves it, after Model rent from boarders, lodgers and sub-tenants for the householder policyengine-uk#2002.
  • 2.174.0: each person's cash ISA and stocks and shares ISA balances count in their own benefit unit for Housing Benefit, income-related ESA, income-based JSA and Income Support, and through Universal Credit's person-level sources. The concept mapping's liquid-asset reason and its two lockstep tests follow.
  • The other releases change nothing the build reads at 2024 or 2025, or nothing at all.

What does not move

  • The vendored uprating pins and the bonus-sacrifice anchor change only their version stamps, and the CGT projection path keeps its values, so no gate digest moves.
  • No target reference, membership or compile-parity receipt changes, so none is regenerated.
  • UK_HBAI_GROSS_INCOME_DEDUCTIONS keeps its parity: policyengine-uk's HBAI subtractions do not change.

Texts and derived files

The texts that describe the engine follow 2.174.1: frs_take_up and its stage note (the education condition), uc_capital_coherence and its stage note (Council Tax Reduction's capital), frs_property_income (SUBRENT) and the LISA and ISA docs. The concept coverage report (369 to 376 inputs), the release input coverage manifest, the charter-H2 fixture, the cited versions and APPROVED_UV_LOCK_SHA256 are regenerated.

Measurement

Two arms run on this machine, one at a time: a control on main 875506e8a and this branch's head, each a spine and a calibration, at review date 2026-11-04 on feed 08962f1b. Their results follow in a comment: both arms' gates, the fit, the first stage where the spines differ and whether the redraws follow the engine-computed predictors, and the rows each release moves, with rows of the SPI channel as classes. The control is also #1095's owed first whole build with #1426 and #1391 on the current base.

Tests

  • Engine tier on policyengine-uk 2.174.1, run in full locally before the rebase, because CI stops at the first failure: 295 tests, of which 293 passed and 2 failed. Both failures were lockstep tests that hold the SPI receipt's projection index to the engine's (test_uk_hmrc_uprating.py, test_uk_spi_income.py): 2.165.2 moved that index to the OBR series. The last commit follows it, and both files then pass. The full tier runs again on the rebased head beside the control arm; its result follows with the arms.
  • After the rebase: the engine files either side touched, with the H parity test, 69 tests, none failed.
  • Targeted engine-free files on the rebased head (one process each run, counted by --junitxml): the 30 files this branch touches, 2,531 tests, none failed or skipped; Calibrate the QRF sign gates out of fold, for every gated UK imputation #1391's touched files with this branch's SPI and uprating files and microcosm-data's contract and dense release files, 18 files, 1,001 tests, one skipped because the licensed SPI donor is not staged on this machine. No gate digest moved.
  • Checks: tools/ci_test_plan.py verify; tools/spec_engine_coverage.py --check; tools/refresh_concept_coverage.py --engine policyengine-uk --check; both pin tools and the coverage manifest with --check; the H2 fixture regenerated on the merged tree (before the rebase a second regeneration was byte-identical), and the H parity test; ruff on the edited files.
  • Mutation check: a builder that does not pass the savings rule fails the new engine test.

Signatures

Awaiting María's signature: the seven signed defaults, with the error each can make (UK_ENGINE_DEFAULT_INPUTS).

Not in this PR

Disclosure

No figure in this PR, its commits or its changelog rests on survey records: the growth rates quoted are policyengine-uk's parameters. Measured rows of the SPI channel are given as classes only.

🤖 Generated with Claude Code

juaristi22 and others added 6 commits November 4, 2026 13:13
The UK lock moves from 2.165.1 to 2.174.1, the newest release, which
carries the SOC 2020 inputs of 2.173.0 (policyengine-uk#2858), so that
microcosm#1490 can stack on it (microcosm#1095). uv.lock moves
policyengine-uk alone; policyengine-core stays at 3.33.3.

Seven new inputs stay at their defaults and are signed in
UK_ENGINE_DEFAULT_INPUTS, each with the error it can make, for María's
signature: income_not_otherwise_charged (2.165.3),
machine_games_standard_rate_higher_stake_net_takings (2.167.0),
trading_allowance_claim_status (2.172.0, at UNKNOWN, so #1428's
receipts encoding stands until microcosm#1439), soc2020_major_group and
soc2020_sub_major_group (2.173.0), and is_in_full_time_education and
has_education_maintenance_funding (2.173.2).

From 2.174.0 Universal Credit counts each member's cash and stocks and
shares ISA balances through counted_person_sources, in the holder's own
benefit unit, beside the Lifetime ISA. The concept mapping's
liquid-asset reason and its two lockstep tests follow.

No economic assumption moves: the vendored uprating pins and the
bonus-sacrifice anchor change only their version stamps. The three
pyproject floors, the approved lock digest, the engine-core pin test,
the concept coverage report (369 to 376 inputs), the stage dependency
strings and the version mentions follow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
policyengine-uk 2.168.1 projects the declared energy spending by one
retail energy price index per fuel (ENERGY_PRICE_SERIES under
ENERGY_PRICES), from the declared period's level to the projected
fiscal year's, and removes the single ENERGY_PRICE_INDEX the test
imported. The end-to-end test now expects each column's own fuel ratio,
calendar 2024 to the fiscal year, and no longer asks a ratio to exceed
one: gas prices fall into 2025. The undeclared file still keeps its
stored spending in every year.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- frs_take_up and its stage note: is_uc_eligible also needs one claimant
  to meet the condition of not receiving education (2.173.2), which
  every claimant meets, since the build flags nobody in full-time
  education.
- uc_capital_coherence and its stage note: from 2.173.3 Council Tax
  Reduction counts the same property, with owned land, at the same
  value, and takes the Universal Credit assessment for a working-age
  applicant with Universal Credit.
- frs_property_income: from 2.173.3 the legacy means tests and Council
  Tax Reduction count sublet_income, not property_income. SUBRENT stays
  in property_income, where income tax and HBAI income count it, until
  policyengine-uk-data#508 moves it after policyengine-uk#2002.
- docs/uk-lisa-1003.md and docs/uk-isa-subscriptions-1415.md: from
  2.174.0 the means tests count each member's ISA balances in the
  holder's own benefit unit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The SPI income stage carries the donors' interest to the frame's year,
and the SPI Table 3.7 interest targets reach the calibration year, on
the OBR's savings income per person (María's ruling of 2026-10-29,
microcosm#1418). policyengine-uk 2.165.2 grows a dataset's savings
income by that series only where the dataset declares the rule, and by
GDP per head otherwise. On María's ruling of 2026-11-04 the producer
now declares obr_savings_income_2022_2025:

- the adapter defines the rule, its root attribute and the engine's
  default, and passes the frame's rule to UKSingleYearDataset, so the
  stages' in-process projections (dataset_period) follow it too;
- national_frame writes the rule in every H5, and a file that declares
  none loads as GDP per head and keeps that reading when written again;
  only a rule other than the producer's travels in frame metadata;
- graph_terminal's export check refuses a release without the
  producer's rule, and the serializer registry lists the new writer.

The ISA contract engine test's 2025 values move from x1.0462 (GDP per
head) to x1.0032 (the OBR series per person). New tests: the engine
grows a declared file by the OBR series and an undeclared one by GDP per
head, and the stages' in-process projection matches; the writer and
loader keep each file's rule and refuse an unknown one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The release input coverage manifest follows the stage manifest's
digests (the policyengine-uk floor in two stage dependencies and the
two restated stage notes). The charter-H2 fixture follows the same stage
texts and the adapter's source, which now declares the savings income
projection rule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From 2.165.2 policyengine-uk's uprating_indices.yaml projects
savings_interest_income, other_investment_income and the cash interest
in non-cash ISAs by the OBR's savings income per person, which the
engine applies in 2022 to 2025 under the rule the producer declares, and
by GDP per head in other years. SPI_HISTORY_INDEX_DIFFERS_FROM_PROJECTION
names that index for both rebased columns, so the SPI income receipt's
dataset_projection_index is the engine's again, and the two engine
lockstep tests that compare it with the engine (test_uk_hmrc_uprating,
test_uk_spi_income) pass.

The two-leg savings interest concept keeps GDP per head as its
projection leg: no SPI target has followed it since microcosm#1418, and
the uprating test now fails if one does. No compiled target moves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vahid-ahmadi

Copy link
Copy Markdown
Contributor

Review of fd651b3. I read the whole diff against main at 875506e (the branch's merge base), diffed policyengine-uk 2.165.1 against 2.174.1 (variables, parameters and CHANGELOG for every release in between), and read the seven new inputs at the 2.174.1 tag. I did not run tests locally; this rests on reading the code and on CI.

Must

  1. engine-free fails on both Pythons because of this PR. test_us_plan.py::TestUsSources::test_no_incumbent_data_package_references_in_live_tree (1 failed, 17,992 passed on 3.13) finds the retired package name policyengine-uk-data in two new texts: changelog.d/uk-policyengine-uk-2-174-1.changed.md (PolicyEngine/policyengine-uk-data#508) and policyengine_uk_concepts.py:428-429 (policyengine-uk-data#641, policyengine-uk-data#640). The guard skips uk_runtime/, which is why the same reference in frs_spine.py passes.
    Fix: use the tree's usual short form, uk-data#508, uk-data#641 and uk-data#640, in both files. Since this also changes the signed text, item 3's wording can go in at the same time.

Should

  1. The rowwise and ladder paths drop the savings rule a source file declares. rowwise_dataset.py carries isa_income_contract through every step but never savings_income_projection: _read_uk_single_year_h5 (about line 1016) and _dataset_tables (about 998) don't read it, expand_uk_geographic_pool (455-464), assign_uk_geographic_pool (488-497) and load_uk_rowwise_dataset (826-834) call uk_national_frame without it, and the legacy branch of write_uk_rowwise_dataset (858-867) leaves the writer's default. uk_national_frame treats a missing rule as the producer's, so the rule is lost.
    Failure case: run clone_uk_dataset_with_ladder_geography on an H5 with no policyengine_uk.savings_income_projection attribute, such as any national written before this PR. The source projects as gdp_per_head, but the ladder output is written with obr_savings_income_2022_2025. So the local dataset grows 2024 savings interest to 2025 by x1.0032 while its national parent grows it by GDP per head (x1.0462 in the docstring this PR replaces in test_uk_isa_income_contract.py). That is about 4% less interest in the local cut than in its own source. This breaks the guarantee the new national_frame test states ("a reference file never gains the producer's rule"). Producer-built frames aren't affected, since they all carry the producer's rule.
    Fix: read the attribute in _read_uk_single_year_h5 (_read_savings_income_projection), add it to both _dataset_tables branches, and pass savings_income_projection= wherever isa_income_contract= is passed in this file. Add a test that ladder-clones an undeclared H5 and checks that the output still reads gdp_per_head.

  2. The signed error for income_not_otherwise_charged gets the size wrong (policyengine_uk_concepts.py:418-431). "Income tax is overstated by up to GBP 1,000" gives the relieved income as if it were tax. The trading allowance relieves up to the allowance in income, so tax is overstated by that amount times the marginal rate. Also, from 6 April 2027 the allowance is £3,000 (2.170.0, trading_allowance.yaml 2027-04-06). That raises the cap for 2027-28 onwards, which is exactly the year a raise would be scored. Suggested wording: "taxable income is overstated by up to the trading allowance (GBP 1,000; GBP 3,000 from 2027-28), so income tax by that amount at the person's marginal rate".

Nit

  1. hmrc_calibration._uk_single_year_dataset (hmrc_calibration.py:458-466) builds a UKSingleYearDataset without savings_income_projection, so the engine takes gdp_per_head. This does no harm today, because materialize_uk_hmrc_calibration_frame calculates only at the dataset's own year (line 122) and nothing is projected. But the PR body says every dataset handed to the engine in process declares the rule. Pass savings_income_projection=uk_savings_income_projection(frame) so the statement holds if the period ever moves.

Checked and fine

  • uv.lock: only policyengine-uk moves (2.165.1 to 2.174.1, plus the three >=2.174.1 specifiers). policyengine-core stays at 3.33.3 for UK and 3.32.5 for US, and 2.174.1 still asks for policyengine-core>=3.33.0. APPROVED_UV_LOCK_SHA256 equals the sha256 of the head uv.lock.
  • The seven defaults match 2.174.1 exactly: income_not_otherwise_charged and machine_games_standard_rate_higher_stake_net_takings are 0; trading_allowance_claim_status is UNKNOWN, which the engine documents as "the current treatment"; both SOC 2020 inputs are NONE, and no formula in 2.174.1 reads either; is_in_full_time_education and has_education_maintenance_funding are False. The error directions are right. uc_receiving_education is full_time & (...), so False means everyone meets the condition and UC eligibility is overstated for uncovered full-time students. The higher-stake input at 0 leaves the 25% rate (rates/higher.yaml) unapplied from April 2027. UC Regs 2013 regs. 12 and 13 are "receiving education" and "undertaking a course", as cited.
  • The linked items exist and say what the PR says: microcosm#1418 item 5 (Machine Games split by Gambling Commission category), UK trading allowance: store the claim status once policyengine-uk#2765 is released #1439 (store the claim status once pe-uk#2765 ships), UK: carry the FRS occupation (SOC 2020 major group), and redraw it on the SPI channel #1490 (FRS SOC major group, sub-major imputed), SPI donor savings rebased to SPI's 2023-24 bands (savings follow-up to #1418) #1428 (adds trading_allowance_receipts); uk-data#641 (set the input from FRS odd-job income, open PR), Rename POPULACE_LEDGER_* client env vars to POPULACE_LOGBOOK_* #640 and Diagnose the SSI aged-band collapse: frozen pre-calibration thresholds + #477's scorecard-only band misses (#507) #508 (open); pe-uk#2765, #2858 and #1995 merged; pe-uk#2002 still open.
  • Releases 2.165.2 to 2.174.1: no variable file is removed or renamed. Engine inputs go from 369 to 376, and the seven new inputs are exactly these. The parameters 2.166.0 removes (income_tax_additions, income_tax_subtractions) appear nowhere in microcosm. The 2.171.0 stored-ISA-totals fix doesn't reach the release, because the export drops the household cash_isa and stocks_and_shares_isa (national_frame.py:749-761). The 2.174.0 AFIP fix and the 2.174.1 Council Tax Reduction fix touch nothing the build writes or targets: no microcosm package writes armed_forces_independence_payment, and there is no CTR target. The other changes take effect in 2022-23 or from 2026-27 (the 2.166.1 Housing Benefit accommodation disregards start on 2026-10-05), or they touch only reporting and CI.
  • The savings rule's constants match policyengine_uk/data/savings_income_projection.py. SPI_HISTORY_INDEX_DIFFERS_FROM_PROJECTION matches 2.174.1's uprating_indices.yaml, which maps savings_interest_income and other_investment_income to per_capita.savings_income.
  • The generated files agree with each other. Every source_manifest_sha256 in the coverage manifest equals the sha256 of the head source_stages.json. The concept coverage counts (376 inputs, 324 uncovered, person 200, household 106) agree with seven added inputs. The branch sits directly on 875506e.

CI at fd651b3: engine-free (3.13, 3.14) fails on the one test in item 1. lint, wheels, select-countries, installed-emitter, integration-uk and engine-uk (3.13, 3.14) pass. engine-us (3.13, 3.14) is still running.

Verdict: the lock bump and the seven defaults hold up. Item 1 turns CI green again. Item 2 should be fixed before this leaves draft, and item 3 before María signs the defaults.

🤖 Generated with Claude Code

…s rule

- Must 1: the changelog fragment and the concept mapping cited the
  retired package name, which test_us_plan.py's live-tree guard refuses
  (engine-free red on 3.13 and 3.14). They now cite uk-data#508,
  uk-data#641 and uk-data#640, as the tree does; frs_spine.py too.
- Should 2: the rowwise and ladder paths dropped the savings income
  projection rule a source H5 declares, so a clone of an undeclared
  file (any national written before the rule) came out declaring the
  producer's. rowwise_dataset now reads the rule with the ISA contract,
  carries it through every step and writes it; a new test ladder-clones
  a declared and an undeclared H5 and checks the output, the written
  file and its reload keep each source's rule (it fails when the H5
  reader drops the rule).
- Should 3: the signed error of income_not_otherwise_charged gave the
  relieved income as tax. Taxable income is overstated by up to the
  trading allowance (GBP 1,000; GBP 3,000 from 2027-28), so income tax
  by that amount at the person's marginal rate.
- Nit 4: hmrc_calibration hands the engine the frame's rule too.

Producer-built frames carry the producer's rule either way, so the
national build's data does not change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Thanks, Vahid. All four items are in 938d4b2.

  1. Must, the retired package name. The changelog fragment and the concept mapping now cite uk-data#508, uk-data#641 and uk-data#640, as the tree does. frs_spine.py uses the same short form too, although the guard skips uk_runtime/. test_us_plan.py passes locally.
  2. Should, the rowwise and ladder paths. rowwise_dataset now reads the savings income projection rule with the ISA income contract (_read_uk_single_year_h5 and both branches of _dataset_tables), and passes savings_income_projection= wherever it passes isa_income_contract=: the row-wise clone and its result, expand_uk_geographic_pool, assign_uk_geographic_pool, the ladder clone, load_uk_rowwise_dataset and the legacy branch of write_uk_rowwise_dataset. A new test, test_ladder_clone_keeps_the_source_files_savings_income_rule, ladder-clones a declared and an undeclared H5 and checks the result's frame, the written file and its reload: the undeclared source stays gdp_per_head, and the declared one keeps the producer's rule. With the H5 reader dropping the rule, the test fails.
  3. Should, the signed error. It now reads: "taxable income is overstated by up to the trading allowance (GBP 1,000; GBP 3,000 from 2027-28), so income tax by that amount at the person's marginal rate".
  4. Nit, hmrc_calibration. _uk_single_year_dataset passes savings_income_projection=uk_savings_income_projection(frame).

Tests at 938d4b2: the touched engine-free files (test_us_plan.py, the ladder, rowwise and rowwise-driver files, test_uk_national_frame.py, test_uk_hmrc_calibration.py, test_uk_terminal_gates.py and the concept mapping files) pass, 446 tests; the concept coverage check is current. I did not rerun the H2 check locally while the arms hold the machine; CI's engine-uk job runs the H parity test, and I run it locally after the arms.

The arms: the control on 875506e and the bump arm on fd651b3 are running. By the code, 938d4b2 changes no data in the national build: producer-built frames carry the producer's rule either way, and the other changes are text. So the bump arm measures this head's national data, and its results follow in a comment.

juaristi22 and others added 4 commits November 4, 2026 15:22
…rocosm-fit (#1490)

Imputing a code (an occupation group, an industry) cannot go through the
quantile forest, which interpolates between donor values. microcosm-fit
gains three country-agnostic pieces for it:

- categorical.CategoricalClassifier: a ConditionalModel for one
  categorical target under the package's weight contract. It fits a
  regularised histogram gradient-boosted classifier, a multinomial logit
  on spline features (arcsinh for long-tailed amounts), or their mixture
  by boosted_weight (1 and 0 fit one learner), and draws by inverse CDF at
  caller uniforms (predict_from_uniforms), so a build's keyed uniforms fix
  each row's draw. It refuses a logit that does not converge, categories
  it never saw, and non-finite numeric predictors.
- held_out: grouped stratified folds and the calibration slope (reusing
  the sign-gate machinery in qrf), out-of-fold probabilities for the model,
  each learner and the training folds' class shares, and the scores a
  stage records (log loss, slope, class-share gaps within strata).
- donors: keyed nearest-donor choice within cells, falling back to coarser
  cells, chosen by weight at the caller's uniform.

Refit through this API on FRS 2024-25 coded adults (five household-grouped
folds), the blend reproduces the measured one: log loss 1.819 against
1.820, pay-band share gap 0.013, slope 1.01.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
)

frs_employment now reads adult.tab's SOC2020, which codes the SOC 2020
major group as 1000-9000 (0 undefined), and writes policyengine-uk's
soc2020_major_group: the SOC2020MajorGroup member names, held once in
uk_runtime/occupation_constants.py with the FRS view derived beside them.
The FRS asks the employment questions of everyone who has ever worked, so
the group is the main job's, or the last job's for an adult not in work
(2024-25 question instructions, details of employment, p. 183; PDF
p. 184). Child-table people are NONE. An adult with a blank or unknown
code refuses the build, naming the codes and never a count, as EMPSTATI
does; soc2020 is direct-indexed like sic.

The column joins the export allow-list, and a terminal gate,
uk_soc2020_major_group_enum_domain, keeps it inside the engine's enum,
which policyengine-uk has from 2.173.0. The gate moves the battery's
three vintage digests and the release-cut part's two (spine and
calibration seam are unchanged); the coverage manifest's source digests
and the H2 fixture (a SOC2020 column in the synthetic adult.tab, 30
normalised string columns) are regenerated.

Checked on the real FRS 2024-25 adult.tab (digest-verified): every adult
maps to its own code's group and every child-table person is NONE.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…coherent industry (#1490)

The SPI support copies and income band donors are whole FRS households
whose claimants and partners then draw SPI incomes, so a drawn row kept
its twin's SOC 2020 major group, SIC division and sector, all chosen with
the twin's pay. A new person stage, spi_employment_shell, runs after
company_cars (so the tape rows' pay is net of car benefits, as FRS pay
is) and before spi_housing_shell:

- It redraws the major group of every drawn row whose twin had an
  occupation, or that drew the tape's own PAY or a profit; a drawn row
  with neither keeps NONE, and FRS rows and undrawn copies are untouched.
- It trains on the FRS rows' adults aged 16 and over with an occupation
  and positive household weight, on age, sex, region, employment-status
  group and employment, self-employment, investment (savings interest
  plus dividends) and private pension income: what a drawn row holds of
  its own. The classifier is microcosm-fit's CategoricalClassifier, the
  declared equal blend of a regularised boosted classifier and a spline
  logit, drawn at uniforms keyed on person_id.
- Each drawn row copies the SIC division and sector of an FRS adult with
  the drawn group and status group among the 50 nearest on age, incomes
  and region, falling back to the group and then to everyone.
- The receipt records out-of-fold evidence on five household-grouped
  folds (log loss for the model, each learner and the constant; the
  calibration slope; occupation-share gaps within pay bands and status
  groups; the industry copy's section and public-sector gaps), with counts
  banded by receipt_counts. A stage-health gate,
  uk_stage_spi_employment_shell_fit, holds them to declared bounds; it is a
  population fact check, so the synthetic smoke records it without
  blocking.

Training rows and donors are taken in person_id order, so no draw depends
on how the frame orders its rows. On a fixture the knob that caps the
boosted learner also turns its early stopping off, since a few households
leave classes too thin for the stratified split.

The registration moves the derived surfaces: the spec and its packaged
projection, the schema branch (every nested block closed), the operation
kind, the graph's cells, consumes and reader isolation, the kernel and
helper modules, the seed harvest, the gate's registrations and the
battery and spine-part digests, the roster pins (53 stages) and the
coverage manifest. The H2 fixture is regenerated separately.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…#1490)

aps_occupation draws each person's soc2020_sub_major_group within the major
group that the FRS carry and the SPI employment shell settle. The donors are
the Annual Population Survey's adults with a SOC 2020 minor group, April 2024
to March 2025 (UK Data Service SN 9408, End User Licence): the main job's
group for people in employment, the last job's otherwise. Employees without a
pay figure leave the donors; those with one carry the income weight PIWTA22,
and everyone else the person weight PWTA22. Each major group gets microcosm-
fit's CategoricalClassifier blend over its sub-major groups. The predictors
are age (capped at 80), sex, region, employment-status group, annual pay and
SIC 2007 section, the set that held out best on the APS. The tab is
caller-supplied (--aps-tab) and pinned by size and sha256. Region comes from
GOVTOF, which is numeric and agrees with GOR9DCENSUS2021 on every record; the
pinned reader blanks string columns.

Draws key on person_source_id, so an undrawn SPI copy draws its FRS twin's
group. SPI income recipients key on their own person_id under a separate
salt. A person with no major group keeps NONE.

The receipt records:
- the cleaning's drops and the donors by major group;
- held-out evidence on five person folds per major group: the pooled log
  loss of the model, each learner and the constant; each group's slope; and
  share gaps within pay bands, regions, age bands and status groups;
- the FRS rows' expected shares against the donors'.

Two new gates. A stage-health gate, uk_stage_aps_occupation_fit (check
sub_major_draw), holds the evidence to proposed bounds. A terminal gate,
uk_soc2020_sub_major_group_enum_domain, keeps the column inside the engine's
enum. The column joins the export allow-list, and this commit retires its
signed NONE default.

microcosm-fit's CategoricalClassifier gains unseen_categories="ignore", which
gives a level the fit never saw no indicator, as the experiment's one-hot
encoding did. It also gains unseen_category_rows; the receipt counts the
households that held one.

The gates move the battery's three vintage digests and the spine and
release-cut parts' two each. The coverage manifest's source digests are
regenerated. receipt_counts records that SN 9408's catalogue entry sets no
condition beyond the End User Licence (checked 2026-11-04). The H2 fixture
tool gains a synthetic APS donor (three per sub-major group) and the
54-stage count; the fixture itself is regenerated once the policyengine-uk
bump is measured.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…oyee pay, nesting guard

The employment shell's industry copy now draws within one cell: the drawn
group crossed with the row's own status group, taking the 50 nearest donors,
or all of the cell's donors where it holds fewer. A copied sector is
therefore always one the row's status group holds, and every (group, section)
pair occurs. The group-only and everyone levels are gone.

A row now draws only among the groups that FRS adults of its own status
group hold: the other groups' probabilities go to zero and the rest are
renormalised. The cell is never empty, and a row whose status group holds
none of the groups refuses the stage. On FRS 2024-25 at full scale every
such cell holds at least 50 adults, so the restriction removes nothing
there. The receipt reports the rows whose draw it restricts, and the donors
of another status group (zero by construction). The held-out industry
evidence leaves out, and counts, any adult whose cell has no donor in the
other folds.

Both stages read the SPI tape's PAY (hmrc_spi_pay) as an SPI income
recipient's pay. Its employment_income also holds EPB and TAXTERM, and
termination pay says nothing of an occupation.

In aps_occupation, pay is an employee's on both sides, because the APS asks
GRSSWK of employees only. It is zero outside the EMPLOYEE status group on
the frame, and zero for non-employee donors. On the pinned APS tab and
FRS 2024-25 the stage's held-out figures barely move (pooled log loss 0.5526,
against 0.5527 before). The gate's notes now cite slopes of 0.95 to 1.01.

A new test holds that no stage after aps_occupation writes either occupation
column, so the sub-major group stays nested in the major group on the
shipped frame.

The operations declare the new pay and draw-support rules. The gate notes
move the battery's manifest digest and fingerprint and the spine part's
manifest digest, and the coverage manifest is regenerated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Arm results for this PR. Both arms ran locally on 4 November, one at a time, with the same review date (2026-11-04) and ledger (08962f1b). Each arm built a full spine and ran a national calibration. Nothing was uploaded.

  • Control: main at 875506e, policyengine-uk 2.165.1.

  • Bump arm: fd651b3, policyengine-uk 2.174.1. The head is now 938d4b2, one commit later. By the code, that commit changes no national data:

    • the frs_spine.py change is one docstring line;
    • hmrc_calibration simulates at the frame's own year (2024), where the savings rule has no effect;
    • the rowwise and ladder change carries the rule for frames loaded from an H5 file, and producer-built frames write the producer's rule either way.

    The bump arm's spine and national files already declare obr_savings_income_2022_2025.

Gates and fit

  • Spine gates pass in both arms (56 of 56), and the terminal gates pass in both arms (7 of 7).
  • Final loss: 0.00533 on the control, 0.00532 on the bump arm. The design loss is lower on the bump arm.
  • 98.7% of the 1,339 rows are within 10% in both arms, and no row is in one arm only.
  • One row crosses 10%: hmrc/property_income_income_band_12_570_to_15_000@2025 moves from within 10% to outside 10%. It rests on the SPI channel, so I give no figures for it. Its design value is the same in both arms, so the move comes from the solve. This stretch row has also moved near the 10% line in earlier arms.

Rows for each release

  • Savings rule (2.165.2): the SPI savings interest amount rows move at design, mostly down, because the engine's 2025 interest now grows by the OBR series. They stay on target after calibration in both arms.
  • Income tax: obr.income_tax and the 84 hmrc.itl rows are within 10% in both arms. The final obr.income_tax error does not move.
  • HICBC and the annual allowance (2.166.0): the HICBC readout (liable people and charge, 2024 and 2025) and the annual allowance charge move by less than 1%.
  • Housing Benefit (2.166.1, 2.173.3, 2.174.0): the four pension-age HB rows stay within 0.2 points of target. In the engine, HB units and spend at 2025 fall by about 0.5%.
  • Energy and consumption taxes (2.168.1, 2.170.4): the bound electricity and road fuel rows stay within 0.1%. The gas diagnostic row (at 2025) moves about 7 points down, to within 2% of the ONS value: exactly the move from CPI (×1.0337) to the gas price index (×0.9661) from calendar 2024 to 2025. VAT does not move. The betting and gaming duties and the other excise rows move by less than 1 point; they are measured at 2024, so the new gambling projection into 2025 does not reach them.
  • Carer's Allowance (2.170.0): both rows stay within 0.2 points of target. The stage receipt moves by rounding only.
  • Trading allowance (2.172.0): the receipt moves by rounding only, as expected at UNKNOWN.
  • Pension Credit (2.173.3, 2.174.0): all 14 rows are within 10% in both arms, and the finals move by at most 0.6 points. The take-up stage sees fewer entitled units in its first band, as the new means-test income and the ISA capital for units with no recorded capital predict.

Where the spines differ

  • The first stage whose records differ is was_wealth, the first stage that reads engine income at 2024, as the plan expected. The SPI income stage differs only in its version strings and the declared projection index.
  • No frame predictor of was_wealth or lcfs_consumption moved. Most moves are very small (median under 0.1%), and about 0.2% of the weight moves by more than 10%. So the redraws follow the engine-computed predictors.

Tests on 938d4b2

  • The H2 fixture regenerates byte-identically (tools/graph_uk_spine_fixture.py --check).
  • The full engine-uk tier passes locally: 295 of 295 tests, with no failures, errors or skips. This includes the H parity test.

@vahid-ahmadi

Copy link
Copy Markdown
Contributor

Round 2 at 938d4b2. I read the diff from fd651b3, the new ladder test, both of your replies and the arm results, then rechecked every caller that carries the ISA income contract. I did not run tests locally; this rests on reading the code and on CI.

  1. Must, retired package name: fixed. The changelog fragment and policyengine_uk_concepts.py now cite uk-data#508, uk-data#641 and uk-data#640. engine-free passes on both Pythons.
  2. Should, rowwise and ladder savings rule: fixed. _read_uk_single_year_h5 and both _dataset_tables branches now carry the rule. savings_income_projection= now goes everywhere isa_income_contract= goes in rowwise_dataset.py: the clone, expand_uk_geographic_pool, assign_uk_geographic_pool, the ladder clone, load_uk_rowwise_dataset and the legacy writer. An undeclared file reads back as gdp_per_head (national_frame.py:278-295), so that value is now written out explicitly, not dropped to the producer's default. The new test_ladder_clone_keeps_the_source_files_savings_income_rule checks the frame, the written attribute and the reload for both cases, which covers the failure case I gave.
  3. Should, signed error text: fixed. It now says taxable income, not tax, is overstated by up to the allowance (GBP 1,000, or GBP 3,000 from 2027-28), and income tax by that amount at the marginal rate.
  4. Nit, hmrc_calibration._uk_single_year_dataset: fixed (hmrc_calibration.py:467).

New findings: none.

Checked and fine

  • Three files still carry the ISA contract without the savings rule: graph_build.py, full_build_cli.py and measure_simulation.py. None needs it. measure_simulation only reads the contract's representation. The other two load a spine H5 and require the producer's ISA contract, and a spine made before this PR is refused anyway by _assert_spine_engine_is_pinned (calibration_run.py:333-353), because its recorded engine is 2.165.1.
  • The only two UKSingleYearDataset(...) constructors outside tests (hmrc_calibration.py:461 and the adapter, policyengine_uk.py:348) both pass the rule.
  • Your reason for measuring the national arm at fd651b3 holds. Producer-built frames write the producer's rule either way, hmrc_calibration runs at the frame's own year, and the other changes are text. So the arm results stand for this head.

CI at 938d4b2: all 11 checks pass (lint, wheels, select-countries, installed-emitter, integration-uk, and engine-free, engine-uk and engine-us on 3.13 and 3.14).

Verdict: all four round-1 items are resolved and I found nothing new. From my side, this can leave draft.

🤖 Generated with Claude Code

…#1512)

The charter-H2 fixture now follows the branch:
- fixture.json carries the two new stages, and the synthetic APS donor
  sources/aps.csv (three people per sub-major group, under the codebook's
  names);
- uk_spine.json carries the 54-stage graph.

The fixture tool's legacy string audit lists soc2020_sub_major_group beside
the major group, so it audits 31 cells, and NORMALIZATION.md follows.
test_h2_uk_spine_parity and the integration smoke test pass.

The employment shell's note no longer says the stage follows company_cars
for its pay. A drawn row's pay predictor is the tape's PAY, which is cash pay
without benefits in kind, as FRS pay is; the car benefits sit in EPB, which
company_cars carves out of employment_income (Vahid's second review of
#1512). The coverage manifest's source digests follow the note.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22
juaristi22 marked this pull request as ready for review November 4, 2026 18:14
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Follow-up to the arm results: the match for 938d4b2 is now measured, not only read from the code.

A full control arm on 938d4b2 (a spine and a national calibration, with the same review date and ledger as the other arms) gives the same calibrated dataset as the bump arm on fd651b3:

  • The spine has the same frame content identity.
  • The terminal gates pass (7 of 7), and the initial and final loss are the same.
  • All 1,339 target rows have the same final error.
  • The calibrated file has the same 3,786 objects, with the same shapes, types and attributes, and all 3,434 datasets have the same values. The files are not byte-identical: only bytes in the HDF5 internal structure differ.

So the arm results apply to this head without change.

juaristi22 and others added 3 commits November 4, 2026 19:07
…olumns (#1512)

The head arm's spine peaked at 12.8 GiB, against 9.6 on the control, and the
APS read caused it. The shared read_pinned_tab parses the whole 548-column
tab in one pass to keep 12 of its columns, a 2.4 GiB transient.

aps_occupation now reads through read_pinned_columns. It checks the same
size and sha256 pins, then reads only the named columns as text, a chunk of
rows at a time, and converts them to numbers as the shared reader does. On
the pinned tab the read no longer raises peak memory, takes the same second,
and returns an identical table.

The licensed-only resource test reads through the new reader, and a
synthetic test holds it equal to read_pinned_tab.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Carry, redraw and impute SOC 2020 occupation in the UK spine (#1490)
The branch carries #1512, merged into it at 86043f2, so main comes in
by a merge and no history is rewritten.

- spec/sources.yaml: company_cars takes main's version (its notes, and
  hmrc_spi_pay and hmrc_spi_employment_benefits among its rewrites), and
  #1490's spi_employment_shell and aps_occupation follow it.
- spi_employment_shell's note no longer says that company_cars leaves PAY
  alone: on main, company_cars moves the part of the car benefits above
  EPB from PAY into EPB (source_stages.json too).
- The coverage manifest and the H2 fixture are regenerated on the merged
  tree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
juaristi22 and others added 2 commits November 5, 2026 08:28
#1512, merged into this branch at 86043f2, adds the categorical, donor
and held-out exports to microcosm/fit/__init__.py. That module defines
fit(), so its bytes are part of fit.qrf@1's implementation hash, and
test_fit_qrf_tolerance_source_hash_pin_is_current failed on both
engine-free jobs at f06ca86. #1512 was stacked on this branch, so it
had no CI run of its own.

`.venv/bin/python tools/graph_parity_repin.py fit.qrf` on
arm64/darwin/py3.14 (Python 3.14.6), the fixture's authoring platform,
with the locked numpy 2.4.6, pandas 3.0.3, quantile-forest 1.4.2 and
scikit-learn 1.8.0:

  implementation_hash  8d4dfdf3 -> c3c8c99e  (the value CI computed)
  arm64/darwin/py3.14  9c6faf71 -> b8acf52a  (produced: graph executed here)
  x86_64/linux/py3.13  a74f0e59 -> d9ff6813  (derived)
  x86_64/linux/py3.14  a86ca61c -> c925ebf4  (derived)

Before it wrote, the tool reproduced all three old keys from the old
hash. direct.csv does not change on any platform (7b8dbd56 before and
after), so the change is additive: the kernel draws what it drew before.
pins.json is the only file this commit changes. The same tool, run on
calibrate and simulate, rewrites their pins byte for byte.

No other pin moves: the seed protocol attests microcosm.fit.qrf, not the
package's __init__, and the one other kernel whose hash moves with #1512
(uk.full.certification-readiness@1, through microcosm.data.contract) has
no pinned value.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#1512 adds UKAPSOccupationStageTransform, which declares
fit_weight_records. The #1428 roster pins the stage transforms that
declare this hook, and a new one must be reviewed. Thus
test_only_the_reviewed_stage_transforms_declare_fit_weight_records
failed on both engine-free jobs at bf12c39 (1 failed, 26112 passed).

The review against the #1428 rule (record each fit, or declare nothing):

- The stage fits one sub-major-group model for each of the 9 SOC 2020
  major groups. A major group with no APS donors raises before its fit,
  so the stage cannot declare the hook and fit nothing.
- Each fit records one FitWeightRecord named
  <stage>:<output>:<major group>, so each name is unique.
- Each record carries the kind that microcosm.fit resolved. The fits use
  the APS donor weights, so the kind is "explicit", not "none".
  test_uk_aps_occupation.py asserts 9 records of that kind.
- Before a run, the property returns an empty tuple, as etb_vat does.

The roster is the only change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
juaristi22 added a commit that referenced this pull request Nov 6, 2026
Since this branch last merged main, lcfs_consumption reads the NTS vehicle
tab (#1426) and aps_occupation the APS April 2024-March 2025 person tab
(#1511, #1490), so a cut from the merged head would refuse at the spine.
The runner now extracts vehicle_eul_2002-2024.tab from the hosted NTS zip
it already fetches, fetches the APS tab from aps_2024_25.zip (UKDS SN 9408,
a repack of the extracted files) at policyengine-uk-data-private
ebebd1e0, and passes --nts-vehicle-tab and --aps-tab. A new test fails if
a stage pins a licensed tab the runner does not fetch and pass.

The gate battery's registry check imports microcosm.data.gate_signing_keys,
which brings the microcosm.data package into the transport bindings'
source closure; main's binding-identity test now lists those modules.

The doc's inputs table names the new tabs and the Chronicle 64753803
artifact, now hosted under chronicle-uk/ in microcosm-uk-build-inputs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

UK: carry the FRS occupation (SOC 2020 major group), and redraw it on the SPI channel

2 participants