[juaristi22]

This PR adds the published occupation totals #407 asks for, for PolicyEngine/microcosm#1490's report-only occupation check: ONS's Annual Population Survey estimates of people in employment by SOC 2020 group, from Nomis, and ONS's ASHE Table 2 pay by two-digit SOC. It covers every ask but one: Nomis's July-to-June and October-to-September windows, including the latest (Jul 2025-Jun 2026), have no Chronicle period type (see Open points).

What it adds

Fifteen packages, 73,039 facts. scripts/build_uk_occupation_packages.py writes them from the pinned bytes and asserts the publisher's text cell by cell; --check compares instead of writing. Field contracts and caveats are in docs/data-sources/uk-occupation-facts.md.

• APS, Nomis APS218 (NM_218_1): ons-aps218-occupation-{apr-2024-mar-2025,jan-dec-2025,apr-2025-mar-2026} (24,956).
  • People in employment for all occupations, the 9 SOC 2020 major groups and the 26 sub-major groups, by sex and by employment type (all, employees, self-employed), for the UK, its four countries and the nine English regions.
  • Each estimate (Nomis "Value") has a second fact for its interval (Nomis "Confidence") where Nomis publishes one, of concept ons.aps_people_in_employment_confidence_interval_half_width and statistic: confidence_interval_half_width. Nomis doesn't state the interval's level, so the id names none.
  • Periods:
    • Apr 2024-Mar 2025 is fiscal_year 2024, the FRS 2024-25 year;
    • Jan-Dec 2025 is calendar_year 2025;
    • Apr 2025-Mar 2026 is fiscal_year 2025;
    • exact dates are in period_coverage, as the house-price packages store a year ending in March.
  • Nomis flags every cell:
    • A cell it blanks (F, a sample of 0 to 2; H, an estimate under 500) emits no fact.
    • Nomis may withhold an estimate's interval (G, "unreliable since the group sample size is small (3-9)", or F) while printing the estimate. The estimate is ported with the interval's status cell guarded, so the flag is in its lineage, and the interval emits no fact.
  • Nomis's one revision of the dataset (27 January 2023, October 2021 to September 2022, "Estimates revised due to identified error") is on every record set's notes; no ported period is affected.
  • Query:
    • the UK and the four countries by their country codes, and the nine English regions by their region codes;
    • totals over full-time/part-time and job type;
    • Nomis's count measure.
  • Not ported: Nomis's "Other" employment type (which "includes unpaid family worker"), the full-time/part-time split and the "Rate" measure.
• ASHE Table 2: ons-ashe-table2-{weekly,annual}-pay-gross[-cv]-{2024-revised,2025-revised,2026-provisional} (48,083).
  • Tables 2.1a and 2.7a, United Kingdom:
    • gross weekly and gross annual pay (the median, the mean and ten percentiles) and the number of jobs;
    • for all employee jobs and each sex by working-pattern sheet;
    • by all employees, the 9 major groups and the 26 sub-major groups.
  • Tables 2.1b and 2.7b give the coefficient of variation of each estimate. The estimate tables show the CV bands only by colour, which Chronicle doesn't read. So each CV is its own fact: its estimate's concept with the suffix _coefficient_of_variation (for example ons.ashe_gross_weekly_pay_coefficient_of_variation), the estimate's dimensions and statistic: coefficient_of_variation.
  • Periods:
    • Weekly pay is calendar_year Y, with the survey reference date the bulletins give (17 April 2024, 30 April 2025, 22 April 2026).
    • Annual pay is "for the tax year that ended on 5th April in the reference year", so it is tax_year Y-1.
  • Cells marked "x" (CV over 20 per cent) emit no fact, nor do the 2024 CV tables' "." cells ("unavailable"). A row with such a cell sits in a record set whose measures leave that column out (…without_p90).
  • Corrections:
    • The 19 December 2025 police note: the 2025 provisional Table 2.7a suppresses sub-major group 33's UK pay in every sheet, and the 2025 revised edition prints it.
    • The 2024 revised zip is the table ONS corrected on 3 November 2025.

Reading the facts

• Select by concept. An estimate, its interval and its CV have different concepts and statistic values, so selecting by concept and dimensions gives one fact.
• One SOC level at a time. A record set's detail rows hold both major and sub-major groups. Select one level by soc2020_major_group or soc2020_sub_major_group and never sum across both.

Labels

• The values these packages share are added to chronicle/dimension_values/canonical_labels.json:
  • soc2020_group (the groupby axis, 36 values) and soc2020_sub_major_group (26);
  • employment_status and working_pattern (3 each);
  • earnings_percentile (10);
  • two statistic values.
• soc2020_major_group 3 changes from the SOC 2010 name the ASHE pension tables print to SOC 2020's "Associate professional occupations", which ASHE Table 2 and Nomis print. Fifteen packages word it that way against five, and CHOSEN_LABELS records the choice. The five pension packages' declarations follow the register; their facts and row headers don't change. That's its own commit.

Licences

Nomis's copyright page puts ONS material under the OGL; the ASHE dataset page states the OGL v3.0. The manifests quote both.

Parity

Asserted by the generator and by tests/test_source_uk_occupation_facts.py:

• Each major group's sub-major groups, the four countries and the nine English regions add to their totals within Nomis's rounding to the nearest hundred (50 per summed cell).
• The nine major groups add to less than the all-occupations total, never more beyond rounding.
  • The gap is up to 0.51 per cent of the total, and 0.13 to 0.14 per cent for the UK's all persons.
  • Nomis publishes no residual category, so the issue's "major groups sum to the total" holds as a bound, not an equality.
• Each ASHE edition's full-time all-employees median matches its bulletin's headline to the bulletin's rounding:
  • weekly £728.27, £766.60 and £811.20;
  • annual £37,439, £39,157 and £40,911.

Pins

• Measured from the fifteen packages' own builds at 2023. CI's merged-bundle test passed with them at c250f44, and the concept change in 6c995aa moves no count.
• fact_count/row_count 1,662,950 → 1,735,989 (+73,039, all of them person facts from ONS).
• Packages 443 → 458, and source tables 385 → 400.
• country:K02000001 +49,991 and country:E92000001 +1,902.
• Eight existing period keys move and none is new; there is no new geography or entity key. Every geography the packages use already appears in other packages, with the register's names.
• CI's merged-bundle step notes a local peak near 15.6 GB on a 16 GB runner, and this adds about 4.4 per cent more facts.

Chronicle Governance

• Approved Chronicle agent role: ledger-source-ingestor. Paths outside its list, as earlier source PRs declared:
  • db/data/** and .gitattributes;
  • chronicle/source_package.py, chronicle/bundle.py (registration) and chronicle/dimension_values/canonical_labels.json;
  • scripts/build_uk_occupation_packages.py (new), and CHOSEN_LABELS in scripts/build_dimension_value_register.py with its count in tests/test_dimension_value_register.py;
  • docs/** and changelog.d/**.
• Deterministic checks run: see Tests below.
• LLM judge verdicts, not run yet:
  • ledger-source-fidelity: pending.
  • ledger-contract: not applicable; no schema, identity or consumer-contract change.
  • ledger-boundary: pending.

Tests

• validate-package on all fifteen: valid, with no warnings.
• build-suite and agent acceptance on all fifteen: valid, with no errors or row-semantic errors. The only warning is concept_alignment_validation_skipped, since Axiom isn't run.
• publish-raw uploaded the six artifacts, and verify-raw confirms each object's key, sha256 and size.
• tests/test_source_uk_occupation_facts.py: 13 passed. It covers:
  • registration, the generator rebuilding each YAML byte for byte, the manifests, their R2 blocks and the consumer contract;
  • that concept, aggregation, period, geography and every dimension but statistic select exactly one fact;
  • every fact re-read from its file with csv or openpyxl and found by its own dimensions;
  • Nomis's flags, the parity checks above and the police group.
• The register, label, alias-drift and UK pension facts for the 2025 calibration: DWP State Pension and Pension Credit caseloads from Stat-Xplore (UK residents), Northern Ireland pensions, workplace pension contributions, and the unported pension rows of files Chronicle already archives #302 pension tests pass. So does the rest of the suite without the merged bundle: 5,439 passed, 5 skipped.
• ruff check on every edited Python file.

Review

Vahid's review of c250f44 is addressed in 6c995aa. The interval and CV facts now have concepts of their own (Should 1). The doc's R2 paragraph is replaced, and the tests require the R2 blocks and a valid consumer contract (nit 1). The doc and package headers say to select one SOC level (nit 2).

Open points

• Periods. Nomis publishes a rolling twelve-month period every quarter. Chronicle's period types fit only the April-to-March and January-to-December windows, so the July-to-June and October-to-September ones, including the latest (Jul 2025-Jun 2026), are not ported. Carrying them needs a new period type, a schema change.
• Nomis's pages don't say whether the occupation is that of the main job. Its metadata also gives Jan 2025-Dec 2025 a start date of 1 September 2025; the package follows the period's label.
• Chronicle facts have no quality field, so Nomis's reliability flags live in lineage and in the missing interval fact.

🤖 Generated with Claude Code

[vahid-ahmadi]

Review of c250f44. I checked the source pins against the live publishers, re-fetched a sample of Nomis cells and all three ASHE zips, read the generator's status handling and parity checks, the registration and pin changes, and how the facts line up with what PolicyEngine/microcosm#1512 will compare against. I found nothing blocking. One convention point and two small doc fixes are listed below.

Should

1. The interval and CV facts share their estimate's concept, unit and aggregation, and only the statistic filter tells them apart. The ACS packages handle this differently: their margins of error get their own concepts (census_acs.household_count_margin_of_error, census_acs.median_household_income_margin_of_error and others) as well as statistic: margin_of_error. Here is how it goes wrong. Take packages/ons/aps218_occupation_apr_2024_mar_2025 and ask for concept ons.aps_people_in_employment, UK, sex all, employment status all, soc2020_major_group: 1, fiscal_year 2024. You get two facts, both unit: count and aggregation: sum: 3,648,300 and 79,700. A reader that doesn't filter on statistic either sums them to 3,728,000 or keeps whichever comes first. The ASHE CV packages have the same problem: the jobs CV is ons.ashe_weekly_pay_jobs with aggregation: sum and the median CV is ons.ashe_gross_weekly_pay with aggregation: median, both in percent. Suggested fix, following the ACS packages: give the interval facts ons.aps_people_in_employment_confidence_interval_half_width and the CV facts <estimate concept>_coefficient_of_variation, and keep the statistic dimension. If you've kept the shared concept on purpose, a line in the doc saying "always filter on statistic" would do instead.

Nit

1. docs/data-sources/uk-occupation-facts.md:28-30 still says "none is in R2 yet, so the consumer export fails ... until chronicle publish-raw uploads them". c250f44 adds a storage.r2 block to all six manifests, so that sentence is now out of date. The same goes for test_consumer_contract_waits_only_on_the_r2_location (tests/test_source_uk_occupation_facts.py:228): it still accepts a report whose only error is a missing source.raw_r2_uri. If a manifest later lost its R2 block, the test would still pass. Suggest asserting report.valid for all fifteen packages and dropping the paragraph.
2. Each record set mixes two nested levels among its detail rows: soc_1 and soc_11/soc_12 sit side by side under groupby_dimension: soc2020_group. Anything that sums a record set's detail rows counts everyone twice. For the UK, all persons, Apr 2024-Mar 2025, the nine majors give 32,954,800 and the 26 sub-majors give another 32,954,800, against a total of 32,997,500. The filters do tell the two levels apart, so this is only a doc fix. One sentence in the doc's Dimensions bullets, saying "select one level by soc2020_major_group or soc2020_sub_major_group; never sum across both", would cover it.

Checked and fine

• Pins. The sha256 and size of all six files match their manifests. A fresh download of the three ASHE zips from the source_urls today matches byte for byte (684813…, 99c6b4…, a376d7…). The Nomis query is NM_218_1, measures 20100/20701 (Value/Confidence), dates 2025-03 / 2025-12 / 2026-03. Each manifest records Nomis's revision metadata (revision 0; released 17 Jul 2025, 21 Apr 2026, 21 Jul 2026) and fetched_at of 4 Nov 2026. The licence fields read "Open Government Licence v3.0", quote Nomis's copyright page and the ONS dataset page, and give license_verified_at: 2026-11-04.
• Nomis spot check. I made a live API call today for the UK, Scotland and London, all three periods, all persons and females, all and self-employed, and Total/1/31/63/92, with Value and Confidence. That is 360 cells, and all 360 match the pinned CSVs in both value and status. Examples: UK total 32,997,500 (±239,700), 33,263,700 and 33,320,100. UK major group 1: 3,648,300, 3,809,800 and 3,791,600. UK sub-major 92, Apr 2024-Mar 2025: 2,619,100. Scotland 63, Apr 2025-Mar 2026: blank, status H.
• ASHE spot check (from the workbooks in the re-downloaded zips), full-time, 2025 revised. Weekly median: all 766.6, group 1 1,111.6, 33 924.3, 63 617.6, 92 545.1. Annual median: all 39,157.2, 33 47,939.5 (so the police row is printed, as the PR says). 2026 provisional weekly: all 811.2, group 1 1,185.6. 2024 revised: all 728.3 weekly and 37,439 annual. The reference dates (17 Apr 2024, 30 Apr 2025, 22 Apr 2026) and the headlines (£728.27, £766.60, £39,157, £40,911) match the 2025 and 2026 bulletins word for word.
• Suppression. Across all six CV workbooks, "." falls in a ported column only in 2024: 3 cells in 2.1b and 5 in 2.7b, all in column D. Every other dot is in the percentage-change columns E and G, which aren't ported. "x" and ":" are skipped. In the Nomis files, Value rows carry only A, F or H (F and H always blank); G appears only on Confidence rows. Where an interval is flagged but its estimate is printed, the estimate is ported and the interval is not.
• Parity. In my own sum for the UK, all persons, the majors fall short of the total by 0.129% in Apr 2024-Mar 2025 and 0.126% in Apr 2025-Mar 2026. The sub-majors add to the majors within 100. The generator's tolerance of 50 × (n+1) is the right bound for rounding to the nearest hundred.
• SOC 2020. There are 9 major and 26 sub-major groups, including 63 "Community and civil enforcement occupations". The codes 11-92 are the ones policyengine-uk's SOC2020_SUB_MAJOR_GROUP_CODES (Add SOC 2020 occupation inputs: major and sub-major group of the main or last job policyengine-uk#2858) uses, so microcosm can join on code. Changing major group 3's register label to "Associate professional occupations" matches both publishers.
• Periods. Apr 2024-Mar 2025 is fiscal_year 2024, the same year as Carry, redraw and impute SOC 2020 occupation in the UK spine (#1490) microcosm#1512's APS donors. ASHE annual pay is tax_year Y-1, which fits "the tax year that ended on 5th April". The jobs count uses value_scale: 1000. The digit filters render to ints, the same as the pension packages' quoted '1'.

For the PolicyEngine/microcosm#1512 report-only check (not a change here):

• APS218 counts people in employment only. Microcosm's column also carries the last job's group for adults not in work, so the comparison needs Microcosm restricted to people in work.
• Within the check's scope (all persons, all employment types, UK/countries/regions), the only estimates without an interval are sub-major 63 at country and region level: 12 cells in Apr 2024-Mar 2025 and 9 in Apr 2025-Mar 2026. One example is Scotland 63, Apr 2024-Mar 2025, at 500 with an F interval. Dropping estimates that have no interval fact handles them.
• For shares, the sum of the nine majors is a better denominator than the Nomis total, since the 0.13% gap has no occupation.

CI at c250f44: Chronicle bundle, Chronicle surface and Chronicle checks all pass. The surface job runs tests/test_source_uk_occupation_facts.py, and the bundle job ran the merged-bundle pins that the body says had not had a full local run. I didn't run the tests locally.

Verdict: the sources are pinned and reproducible, and the sample matches the live publishers exactly. Only the concept point (Should 1) and the two nits remain.

🤖 Generated with Claude Code

[juaristi22]

Thanks, Vahid. All three points are in 6c995aa:

• Should 1. The intervals and CVs now have concepts of their own, following the ACS margins of error:

  • the APS intervals are ons.aps_people_in_employment_confidence_interval_half_width;
  • each ASHE CV takes its estimate's concept with the suffix _coefficient_of_variation (for example ons.ashe_gross_weekly_pay_coefficient_of_variation);
  • the statistic dimension stays.

  A new test, test_a_concept_and_its_dimensions_select_one_fact, asserts that concept, aggregation, period, geography and every dimension but statistic select exactly one fact across all fifteen packages.

• Nit 1. The doc's R2 paragraph is replaced. test_consumer_contract_is_valid now asserts a valid contract and a raw_r2_uri on every fact, and the manifest test requires each storage.r2 block with its canonical key.

• Nit 2. The doc and every package header now say that a record set's detail rows hold both SOC levels: select one level by soc2020_major_group or soc2020_sub_major_group, and never sum across both.

No fact, period, geography or table count moves. Your notes for PolicyEngine/microcosm#1512 are about the Microcosm check (people in work only, dropping estimates without an interval, the nine majors as the share denominator), so nothing changes here for them.

🤖 Generated with Claude Code