Skip to content

Add SPI 2023-24 Table 3.10 and the OBR October 2026 EFO vintage (PolicyEngine/microcosm#1418) - #394

Merged
juaristi22 merged 11 commits into
mainfrom
spi-table-3-10-1418
Oct 29, 2026
Merged

juaristi22 merged 11 commits into
mainfrom
spi-table-3-10-1418

Conversation

@juaristi22

@juaristi22 juaristi22 commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

This PR carries two Budget 2026 data requests for PolicyEngine/microcosm#1418. It is rebased onto main after #392 (ISA) merged, so its pins include #392's:

  • SPI 2023-24 Table 3.10 in hmrc-spi-income-bands-2023-24 (below, unchanged);
  • the OBR's October 2026 Economic and fiscal outlook as a vintage beside March 2026. This implements chronicle#393; María closes that issue herself.

1. SPI 2023-24 Table 3.10

This PR adds HMRC Survey of Personal Incomes 2023-24 Table 3.10 to hmrc-spi-income-bands-2023-24: income of individuals with self-employment income, banded by self-employment income. Microcosm needs its small-profit bands for the Budget 2026 trading allowance measure (Table 6.1 line 15; PolicyEngine/microcosm#1418, item 13).

What it adds

  • Source: the workbook the package already reads (Collated_Tables_3_1_to_3_11_2324.ods), sheet Table_3_10.
  • 159 facts in two record sets:
    • hmrc.spi_2324.table_3_10: 14 bands from GBP 1 to GBP 100 000 and over, plus All ranges as total, 10 measures each;
    • hmrc.spi_2324.table_3_10.subset_1: the GBP 0 to GBP 1 band. HMRC suppresses its self-employment amount, so it carries the other 9 measures.
  • Measures: the counts (thousands) and amounts (GBP million) of self-employment, employment, pension, and property/interest/dividend/other income, and total income. The mean and percentage columns are derived, so they are not ported.
  • Dimension: a new axis, hmrc.self_employment_income_band, with self_employment_income_lower_bound and self_employment_income_upper_bound. Both edges are carried, as for the total-income bands.

Universe

Unlike the package's other sheets, Table 3.10 is not taxpayers only: per its scope line, it counts every individual with a self-employment source, taxpayer or not. So its facts carry entity_role: individual_with_self_employment_source.

  • Over all ranges that is 5,340 thousand people, against Table 3.6's 3,670 thousand self-employed taxpayers.
  • With the taxpayer role, the All-ranges row would have shared 10 semantic keys, all with different values, with Table 3.11's UK row and Tables 3.5 and 3.3.

Concepts

  • Self-employment and property/interest/dividend/other income reuse the ids Tables 3.6, 3.7 and 3.11 name.
  • Employment income has a source concept of its own, hmrc.spi_table_3_10_employment_income_{count,amount}, mapped to Table 3.6's concepts as a broad_match. Table 3.10's note 4 defines it as ITEPA 2003 income plus other miscellaneous income, which is wider than Table 3.6's (note 6). The canonical concepts, and so the fact keys, are unchanged.
  • Pension uses Table 3.11's combined-pension concept.
  • 'Total income' maps to total income before tax with an exact relation, as on Table 3.11.

Pins (Table 3.10)

tests/test_chronicle_bundle.py moves by 159 in fact_count/row_count (1,657,279 → 1,657,438 on main after #392), in hmrc (36,513 → 36,672), in tax_year:2023, in country:K02000001 (36,500 → 36,659) and in person (596,595 → 596,754).

Semantic duplicates (2,324), warnings (133), source tables, and the geography, period and package counts are unchanged. A single-source build and a four-package build (bands, region, area, Income Tax liabilities) both show +159 with no new duplicate or warning.

2. OBR EFO October 2026 (chronicle#393)

The OBR published the October 2026 EFO with the Budget on 28 October 2026. policyengine-uk moves its economic assumptions to it (PolicyEngine/policyengine-uk#2717), and PR C (PolicyEngine/microcosm#1420) needs receipts Table 4.5's savings income. obr.uk refuses scripted downloads, so the five detailed forecast workbooks were fetched with a browser and pinned with their manifests (OGL). scripts/build_obr_efo_october_2026.py writes five packages, 4,317 facts:

Package Facts What
obr-efo-receipts-october-2026 1,551 The March package's 41 lines, and every other row of tables 4.1 to 4.22 except 4.9 and 4.19, with each restated row ported once
obr-efo-expenditure-october-2026 165 The March package's 24 lines
obr-efo-economy-october-2026 56 The March package's 6 lines, plus the deposit rate and household financial assets (policyengine-uk#2638)
obr-efo-aggregates-october-2026 25 The March package's 4 lines
obr-efo-policy-october-2026 2,520 Table 1.1: each policy decision's costing by head and category, 2026-27 to 2031-32, and the summary rows

Semantics (docs/data-sources/uk-obr-efo-october-2026.md has the detail):

  • Vintage. A March line keeps its line id, label and concept under obr.efo_2026_10, so a consumer changes vintage by package. Its October table is found by title (receipts 3.x are now 4.x, expenditure 4.x now 5.x) and its row by label; each record set carries its October table and row. The 33 concepts Microcosm selects are all there.
  • Outturn or forecast follows each column's band: 2025-26 is now outturn, and 2031-32 is new.
  • Table 4.5. Savings income is obr.self_employed_income_growth.savings_income, measure growth_rate: a percentage change on a year earlier from 2024-25, all under the table's 'Forecast' band. The table has no 2023-24 column. 2024-25 and 2025-26 give the ×1.358 path from 2023-24.
  • Northern Ireland domestic rates are carried both places the OBR shows them: receipts table 4.2 row 6 and expenditure table 5.1 row 16. Their values differ.
  • Changed rows against March. 'England council tax receipts' gains footnote 1 (its outturn is MHCLG's pre-measures data). 'Total net council tax receipts' renumbers its footnote. Table 4.10 now starts in 2026-27.
  • Left out: table 4.9, which states no unit, and table 4.19, which prints '£ billion' over thresholds in pounds.
  • Restated rows are ported once. The build asserts each copy equal to the row kept:
    • table 4.7's total SDLT and ATED rows (table 4.8's);
    • table 4.22's VAT refunds, CIL, EPR and five environmental levies (tables 4.8, 4.2 and 4.20);
    • table 4.21's '(c * d + e + f) Forecast' row (table 4.20's contracts for difference, in £ million);
    • table 4.1's betting and gaming row from 2026-27 (table 4.10's total).
  • Cash and accrued taxes. Tables 4.1 (accrued) and 4.8 (cash) print eight of the same taxes: customs duties, landfill tax, aggregates levy, the soft drinks levy, diverted profits tax, plastic packaging tax, RPDT and CBAM. Each pair shares one concept with concept_relation: exact and differs in accounting_basis, as chronicle#381's excise duties do. Each keeps its table's source concept. Table 4.1's vaping tax and 2025-26 betting and gaming join the March lines' obr.vaping_tax and obr.betting_and_gaming_duties. Row labels say '(cash)' or '(accrued)', and the March lines' measure labels name their basis.
  • No 2024-25. All 33 lines Microcosm selects run from 2025-26 to 2031-32 in October. So the 2024-25 part of the calendar-2025 window has to come from the March package's outturn (obr.efo_2026_03.<kind>.<line id>.fy2024) or from ONS. The docs' "For Microcosm" section says so.
  • Policy labels. Ten Table 1.1 measure names start with a tab or end in a space. Their labels are trimmed; the guard cells keep the published text.
  • Table 4.22 prints '£ million' over £ billion cells, so it carries value_scale 1e9. The build asserts the evidence.
  • R2. publish-raw uploaded the five workbooks to ledger-raw, and verify-raw confirms each object's sha256.
  • Annex A was checked against the policy workbook. It is the EFO's detailed summary tables, not policy measures, so it has no missing measure rows and is not archived. The optional devolved forecasts are not ported.

Pins (OBR)

tests/test_chronicle_bundle.py moves by the five packages:

  • fact_count/row_count 1,657,438 → 1,661,755;
  • obr 648 → 4,965;
  • packages and source tables +5 (423 → 428 and 365 → 370);
  • period_count 553 → 554: 2031-32 is a fiscal year no other package carries;
  • the fiscal- and calendar-year periods;
  • country:K02000001 +4,219, K03000001 +56 and K04000001 +28;
  • entities: government +2,601, person +1,547, institutional sector +113 and firm +56.

Semantic duplicate keys go from 2,324 to 2,687 (+363); #392 added none. These are the March lines' projections for the years both vintages forecast, kept as published beside the March packages. A consumer picks a vintage by package; #393 asks that March stay addressable. A bundle of the five October packages alone is valid with no warning and no duplicate key.

Role and paths

ledger-source-ingestor. Paths outside the role's list, as #391 and #392 declared:

Checks run

  • validate-package on both SPI packages, and build-suite on this one;
  • the dimension-label report;
  • the two package tests in tests/test_chronicle_source_package.py that touch HMRC packages;
  • seven quick bundle unit tests;
  • ruff.

The hour-long test_build_bundle_writes_merged_consumer_contract was not run locally; CI runs it.

  • OBR: validate-package and the consumer contract on all five packages: valid, no warnings, no duplicate key within any package.
  • OBR: tests/test_source_obr_efo_october_2026.py: 23 passed. It covers:
    • registration, manifests and licence;
    • the generator rebuilding each YAML byte for byte;
    • March-line parity;
    • Table 4.5, both NI domestic rates placements and sample values, re-read by label with pandas;
    • each policy measure's rows summing to its total;
    • the restated rows left out, the eight cash and accrued pairs, and the trimmed policy labels.
  • SPI: tests/test_source_hmrc_spi_table_3_10.py: 1 passed (Table 3.10's employment source concept and notes).
  • tests/test_chronicle_bundle.py without the merged build: 26 passed; ruff check passed.

Questions for review, answered

Vahid answered all five in his review of 9bb348f (#394 (comment)). His three shoulds and two nits are addressed in 49d9992..2ba21f3.

  1. individual_with_self_employment_source stays: the role keeps Table 3.10 off the taxpayer keys.
  2. Table 3.10's employment income has a source concept of its own, as above.
  3. source_table stays as it is.
  4. Table-scoped concepts stay, but restated rows are ported once and the cash and accrued pairs share concepts, as above.
  5. Both vintages stay in the bundle.

The Microcosm side (national targets for the small-profit counts and amounts) waits for the next feed re-pin. PR C rebuilds the bundle from this PR's head and re-pins once it is review-complete.

🤖 Generated with Claude Code

@juaristi22 juaristi22 changed the title Add SPI 2023-24 Table 3.10: income of individuals with self-employment sources by range of self-employment income Add SPI 2023-24 Table 3.10 and the OBR October 2026 EFO vintage (PolicyEngine/microcosm#1418) Oct 29, 2026
@juaristi22

Copy link
Copy Markdown
Collaborator Author

@PavelMakarchuk the OBR October 2026 EFO packages that chronicle#393 asks for are on this PR now, beside Table 3.10:

  • 71f2e8d: the five pinned workbooks, scripts/build_obr_efo_october_2026.py and its five packages (4,377 facts), with their tests;
  • 19cc96f: the docs;
  • 968c5e6: the merged-bundle pins.

The PR body gives the scope, the semantics, the pins and three OBR questions (4 to 6). CI is running.

🤖 Generated with Claude Code

juaristi22 and others added 5 commits October 29, 2026 10:02
…ine/microcosm#1418

HMRC's Survey of Personal Incomes 2023-24 Table 3.10 bands individuals
with a self-employment source by their self-employment income, from the
GBP 0 to GBP 1 range (a loss or profits wholly offset, counted at nil,
and unknown profits) to GBP 100 000 and over, with the counts and
amounts of their self-employment, employment, pension, and property,
interest, dividend and other income and total income. It reads the
workbook the SPI bands package already reads. Microcosm needs its
GBP 1 to GBP 500 and GBP 500 to GBP 1 000 bands for the trading
allowance.

Unlike the other sheets, Table 3.10 is not taxpayers only, so its facts
carry entity_role individual_with_self_employment_source and their own
axis, hmrc.self_employment_income_band, with
self_employment_income_lower_bound and self_employment_income_upper_bound.
With the taxpayer role, its all-ranges row would have shared 10 semantic
keys, all with different values, with Table 3.11's UK row and Tables 3.5
and 3.3. The concepts are the ones Tables 3.6, 3.7 and 3.11 already name;
'Total income' maps to total income before tax with an exact relation,
as on Table 3.11. The mean and percentage columns are derived and not
ported. HMRC suppresses the GBP 0 to GBP 1 band's self-employment amount,
so that band is its own subset record set carrying the other nine
measures.

159 facts: 15 rows of 10 measures and the subset row's 9. A single-source
build of the package goes from 1,078 to 1,237 facts, and a build with the
region, area and Income Tax liabilities packages adds no semantic
duplicate or warning, so the bundle pins move by 159 in fact_count,
hmrc, tax_year:2023, country:K02000001 and person. The region package's
note that Table 3.10 is not ported is brought up to date.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#393 asks for the Economic and fiscal outlook the OBR
published with the Budget on 28 October 2026. obr.uk refuses scripted
downloads, so the five detailed forecast workbooks were fetched with a
browser and are pinned under db/data/obr with their manifests (OGL, per the
OBR's copyright page). scripts/build_obr_efo_october_2026.py writes five
packages from them, 4,377 facts:

- obr-efo-receipts-october-2026 (1,611): the March package's 41 lines, and
  every other row of tables 4.1 to 4.22 except 4.9 (no unit) and 4.19 (a
  '£ billion' unit over thresholds in pounds). Table 4.5's savings income is
  obr.self_employed_income_growth.savings_income, a growth rate from 2024-25.
- obr-efo-expenditure-october-2026 (165), -economy- (56) and -aggregates-
  (25): the March packages' lines, plus the deposit rate and household
  financial assets that policyengine-uk#2638's interest projection reads.
- obr-efo-policy-october-2026 (2,520): Table 1.1's costing of each policy
  decision by head and category, 2026-27 to 2031-32, and its summary rows.

A March line keeps its line id, label and concept under obr.efo_2026_10, so
a consumer changes vintage by package. Its October table is found by title
(receipts 3.x are now 4.x, expenditure 4.x now 5.x) and its row by label.
Each year's assertion follows the table's Outturn/Forecast band: 2025-26 is
now outturn, and 2031-32 is new. Northern Ireland domestic rates are carried
in both places the OBR shows them. The build asserts each equality the notes
cite (restated rows left out, table 4.22's mislabelled unit).

The manifests carry each file's canonical R2 key; publish-raw has not run
yet (no Wrangler on this machine).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A data-source page with the files, conventions and caveats (the two tables
left out, Table 4.5's growth rates, the two Northern Ireland domestic rates
placements, table 4.22's unit, the restated rows and the outturn bands),
the Annex A check, what Microcosm selects, a checklist section and a
changelog fragment.

Annex A is the EFO's detailed summary tables, not policy measures, so it
has no measure row the policy workbook lacks and is not archived.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The five packages add 4,377 facts (obr 648 -> 5,025), five packages and
source tables, and their fiscal- and calendar-year periods. Their entities:
government 2,601, people 1,601, institutional sectors 119 and firms 56. The
United Kingdom gains 4,279 facts, Great Britain 56 and England and Wales 28
(table 5.9's DWP lines). The March lines' projections for the years both
vintages forecast add 363 semantic duplicate keys, kept as published beside
the March packages. Measured on bundles of the OBR packages; CI runs the
merged build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI's bundle job on 968c5e6 stopped at the summary counts: period_count is
554, not 553. Every other summary count matched. 2031-32, the October 2026
EFO's new forecast year, is a fiscal_year key no other package carries.
The by_period expectation already adds its count.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Rebased onto main now that #392 is merged, so the commits in my last comment have new SHAs:

The first bundle run stopped at period_count, 554 against 553. 2031-32, the October EFO's new forecast year, is a fiscal_year key no other package carried. Every other summary count matched.

The rebased pins sum #392, Table 3.10 and the OBR packages: 1,661,815 facts, 2,687 semantic duplicate keys, 428 packages. The PR body's numbers are updated to match.

🤖 Generated with Claude Code

@vahid-ahmadi

Copy link
Copy Markdown

Review of 9bb348f (draft, so a comment rather than a formal review). This is the first review round: there are no earlier review comments or inline threads, and María's two comments (02:39 and 10:08) are author updates. The rebased pins in the body match the head.

No musts. Three shoulds and two nits.

Findings

1. should: October carries no 2024-25 for the 33 lines Microcosm selects, and the docs don't say so.
docs/data-sources/uk-obr-efo-october-2026.md:124-135 ("For Microcosm") says to select October by package and leaves the calendar-2025 window to the consumer. But #393's window is 3/12 of 2024-25 plus 9/12 of 2025-26, and in October every one of the 33 concepts runs 2025-26 to 2031-32 only. Receipts table 4.8 row 5 starts at 2025-26, while March had *.fy2024 record sets (for example obr.efo_2026_03.receipts.income_tax.fy2024). So the window can't be built from the October package alone. 2024-25 has to come from March's outturn or from ONS. Please add a line saying so in that section and on #393.
Checked: I built the October facts and listed the periods for all 33 concepts. All 33 are present, each with exactly 2025 to 2031.

2. should (your Q4): table-scoped concepts carry the same published series under two or three ids.
The build drops table 4.7's SDLT and ATED rows because they restate table 4.8 (scripts/build_obr_efo_october_2026.py:676-678). Other restatements are kept under separate concepts, though, with nothing linking them. Each of these pairs is identical in all 6 or 7 years:

  • VAT refunds: cash_receipts.vat_refunds (efo_receipts_october_2026/source_package.yaml:62405) and fiscally_neutral_receipts.vat_refunds (:96269);
  • diverted profits tax: cash_receipts.* (:55944) and other_hmrc_taxes.* (:18212);
  • contracts for difference, under three ids: environmental_levies.* (:87168), fiscally_neutral_receipts.* (:93455) and contracts_for_difference_drivers.c_d_e_f_forecast_gbp_million (:92327);
  • also RPDT, CIL, EPR, renewables obligation, capacity market, green gas levy and warm home discount.

A consumer summing a table-scoped concept family will double count these. The fix could follow either the 4.7 rule (drop the restated row) or this PR's own SPI precedent: Table 3.5's 'Other income' carries Table 3.7's concept with concept_relation: exact. Separately, obr.betting_and_gaming_duties (:4619) and obr.other_hmrc_taxes.betting_and_gaming_taxes (:16196) agree from 2026-27 but differ in 2025-26 (3.831 against 3.892). The labels should name the basis difference.
Checked: I grouped all 1,611 receipts facts by period and value and listed the concept pairs that coincide in 5 or more years.

3. should (your Q2): Table 3.10's employment income should get its own source concept.
Note 4 on Notes_3_10 defines it as ITEPA 2003 income plus other miscellaneous income. Table 3.6 note 6 defines it as pay, taxable benefits and contributory IB, ESA and JSA. So the two are not the same measure. Yet spi_income_bands_2023_24/source_package.yaml:6221 and :6441 reuse hmrc.spi_employment_income_* with source_label. The same record set already handles 'Total income' this way: source_concept: hmrc.spi_table_3_10_total_income_count with a relation (:6345). I suggest the same shape here: hmrc.spi_table_3_10_employment_income_*, mapped to the 3.6 concept with broad_match (it adds miscellaneous income). The other reuses look right to me:

  • self-employment: note 1a points to Table 3.6;
  • property, interest, dividend and other income: note 5 applies Table 3.7's notes;
  • pension: one combined column, as on 3.11;
  • total income exact: the four parts sum to 196,800 against 197,000.

4. nit (your Q1): the role name is fine.
entity_role is free text: there is no enum, and source_package.py:1837 passes it through. build_semantic_fact_key includes asdict(fact.entity), so the role is what keeps Table 3.10 off the taxpayer keys. individual_with_self_employment_source is unique in the repo and matches the sheet's scope line ("individuals with sources of self-employment income, whether taxpayers or not"). On Q3, I agree with leaving source_table. Re-keying 1,078 facts for a cosmetic label isn't worth it.

5. nit: three policy measure labels start with a tab.
efo_policy_october_2026/source_package.yaml:148 is measure_61: "\tProperty Income: ...", and measure_62 and measure_80 are the same. The OBR cell has the tab, so the guard cells are right to pin it. Strip it in dimension_value_labels only.

On your Q5, I agree with keeping both vintages:

I recomputed the 363: all are projection against projection, and 5 values are identical. No March package file is in the diff, and Chronicle has no "latest" or default-vintage selector, so existing consumers see no behaviour change until they re-point.

What I ran

Fresh shallow clone of 9bb348f, fresh venv, deleted afterwards.

SPI Table 3.10

  • Downloaded Collated_Tables_3_1_to_3_11_2324.ods from the gov.uk 2023-24 page. Its sha256 matches the manifest (ad063b06…).
  • Compared all 159 facts to Table_3_10, located by band and column header rather than the package's coordinates: 0 mismatches. I also checked both band edges (the next band's lower limit is the upper edge, and £100,000 and over has no upper edge) and the All-ranges row: 5,340 thousand individuals, £197,000m.
  • The only cell not ported is the £0-£1 band's self-employment amount ('[Not available]'). The mean and percentage columns (L and M) are derived, as stated.
  • validate-package: valid, no warnings, 1,237 records (1,078 + 159).

OBR October 2026

  • Hashes: all five pinned workbooks match their manifests. I fetched economy and aggregates from obr.uk myself, and they are byte-identical to the pinned copies. obr.uk refused scripted downloads of receipts, expenditure and policy. Their hashes match the copies pinned independently in policyengine-uk and the AB2026 dashboard repos, but I did not fetch those bytes myself.
  • Spot-checks: 24 values read from the workbooks by row label and column header, all matching. I also checked that the outturn and forecast flags match each table's band.
    • Economy: CPI, RPI, Bank Rate, deposit rate, nominal GDP, employment, the unemployment rate and household financial assets.
    • Aggregates: PSNB, PSND (% of GDP), TME and current receipts.
    • Receipts: VAT, NICs, CGT, fuel duties, tobacco, SDLT, and Table 4.5's savings income (34.96% and 0.65%, ×1.358; the pre-measures note is carried).
    • Expenditure: England council tax, UC in the cap, NI domestic rates and the state pension.
    • Policy: measure 61's income tax line and total.
  • Average earnings and house prices are not in either vintage's economy package, so I had nothing to check there.
  • Coverage: all 33 concepts from OBR October 2026 EFO: archive the receipts and expenditure detailed forecast tables as a new vintage (PolicyEngine/microcosm#1095) #393 are present.
  • Pins: the entity, geography and period recounts match them (+4,377; government 2,601, person 1,601, institutional sector 119, firm 56; fiscal year 2031 = 668).

Tests

  • tests/test_source_obr_efo_october_2026.py: 21 passed.
  • tests/test_chronicle_bundle.py without test_build_bundle_writes_merged_consumer_contract: 26 passed.
  • tests/test_chronicle_source_package.py -k hmrc: 12 passed.
  • No test file exercises the SPI bands package directly. Table 3.10 is covered only by the bundle count pins, as Tables 3.3 to 3.8 are.

Not checked

  • The merged-bundle test. CI's Chronicle bundle job is still pending on 9bb348f; Chronicle surface (including lint) passed.
  • Annex A and the devolved tables.
  • The other 1,500-odd OBR rows beyond the spot-checks.

Verdict

Sound and well evidenced. Nothing blocks. Items 1 to 3 are worth doing before this leaves draft, and the bundle job needs to go green.

🤖 Generated with Claude Code

juaristi22 and others added 6 commits October 29, 2026 13:01
…ued taxes

Vahid's second point on chronicle#394. Table 4.22's VAT refunds, CIL, EPR and
five environmental levies restate tables 4.8, 4.2 and 4.20, and table 4.21's
forecast row restates table 4.20's contracts for difference. Each is now ported
once, and the build asserts every copy equal to the row kept. Table 4.1's
betting and gaming row restates table 4.10's total from 2026-27, so only
2025-26 is kept, under that total's concept.

Tables 4.1 (accrued) and 4.8 (cash) print eight of the same taxes. Each pair now
shares one concept with an exact relation and differs in accounting_basis, as
chronicle#381's excise duties do; each keeps its table's source concept. Table
4.1's vaping tax joins obr.vaping_tax. Row and measure labels name the basis.

The receipts package drops from 1,611 to 1,551 facts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Vahid's fifth point on chronicle#394. In Table 1.1, three measure names start
with a tab and seven end in a space. Their labels in dimension_value_labels are
trimmed; the guard cells keep the published text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Vahid's third point on chronicle#394. Table 3.10's employment income is ITEPA
2003 income plus other miscellaneous income (its note 4), so it is wider than
Table 3.6's (note 6). Its count and amount now carry
hmrc.spi_table_3_10_employment_income_{count,amount} as source concepts, mapped
to Table 3.6's as a broad match, as the table's total income already is. The
canonical concepts, and so the semantic fact keys, don't change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Vahid's first and second points on chronicle#394. "For Microcosm" now says that
October's 33 lines start at 2025-26, so the 2024-25 part of the calendar-2025
window has to come from the March package's outturn or from ONS. The receipts
section lists the rows ported once and the eight cash and accrued pairs, and
the counts move to 4,317 facts (receipts 1,551).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The October receipts package has 60 fewer facts: 54 person and 6 institutional
sector facts, ten in each of fiscal years 2026 to 2031, all UK-wide. So the fact
count is 1,661,755, obr 4,965, K02000001 40,878, person 598,301 and
institutional_sector 8,919. Duplicate keys, periods and tables don't change: a
bundle of the March and October OBR packages still has 363 shared keys.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The receipts package's header comment still named only table 4.7's two rows.
It now lists each row ported once and the eight cash and accrued pairs, as the
docs do. Only the comment changes; no fact moves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

@vahid-ahmadi thanks for the review, and for re-deriving the restatement pairs from the facts. All five points are addressed in six pushed commits, 49d9992..2ba21f3. The PR body is updated, and CI is re-running on 2ba21f3.

  1. should, no 2024-25: done in 95b9c45. "For Microcosm" now says that all 33 lines run from 2025-26 to 2031-32 in October, so the calendar-2025 window can't be built from the October packages alone. Its 2024-25 part has to come from the March package's outturn (obr.efo_2026_03.<kind>.<line id>.fy2024) or from ONS. I checked that March has a 2024-25 fact for each of the 33, all observations. The same caveat is on OBR October 2026 EFO: archive the receipts and expenditure detailed forecast tables as a new vintage (PolicyEngine/microcosm#1095) #393.

  2. should, restated rows: done in 49d9992, with the package header in 2ba21f3. I used both of your routes.

    • The 4.7 rule for same-basis copies. Each of these is now left out, and the build asserts every copy equal to the row kept:

      • table 4.22's VAT refunds (table 4.8's);
      • its CIL and EPR (table 4.2's);
      • its CfD, renewables obligation, capacity market, green gas levy and warm home discount (table 4.20's);
      • table 4.21's '(c * d + e + f) Forecast' row (table 4.20's CfD, in £ million).

      So CfD has one id now, and receipts go from 1,611 to 1,551 facts.

    • Shared concepts for the cash and accrued pairs, as chronicle#381 does. Tables 4.1 (accrued) and 4.8 (cash) print eight of the same taxes: customs duties, landfill tax, aggregates levy, the soft drinks levy, DPT, plastic packaging tax, RPDT and CBAM. Each pair now shares one concept (obr.diverted_profits_tax, ...) with concept_relation: exact, and differs in accounting_basis. The source concepts stay table-scoped. Table 4.1's vaping tax joins the March line's obr.vaping_tax.

    • Betting and gaming. From 2026-27, table 4.1's row restates table 4.10's accrued total, so only its 2025-26 value is kept. That value (3.892) is now obr.betting_and_gaming_duties on the accrued basis, the concept table 4.10's total already carries. So the accrued series runs from 2025-26, and the cash series (3.831 in 2025-26) is table 4.8's.

    • Labels name the basis. The new rows end in '(cash)' or '(accrued)', and the March lines' measure labels say it too, for example 'Tobacco duties, cash basis, FY2025-26'.

    I re-ran your check on the pushed package, listing the row pairs equal in five or more years. Only these are left:

    • DPT (all seven years) and RPDT (six). Each is now one concept on two bases.
    • Table 4.4's self-assessment row against table 4.8's. They are equal in five years but £17.6bn apart in 2029-30 (84.7 against 102.3), so it isn't a restatement, and each keeps its own concept.
    • Four rows that are zero in most years: bingo duty, feed-in tariffs, onshore CT 'Other' and the PPF levy.
  3. should, Table 3.10's employment income: done in 024b080, in the shape you suggest. Its count and amount now carry hmrc.spi_table_3_10_employment_income_{count,amount} as source concepts, mapped to Table 3.6's with broad_match, as the record set's total income already is. The notes quote both definitions: 3.10's note 4 and 3.6's note 6. The canonical concepts don't change, and so neither do the semantic keys. There is also a new tests/test_source_hmrc_spi_table_3_10.py, which checks the 32 facts' source concept, relation and notes, so a test now covers the bands package directly.

  4. nit, the role and source_table: both stay as they are, as you suggest.

  5. nit, policy labels: done in e81320f. Three names start with a tab (measures 61, 62 and 80) and seven end in a space. dimension_value_labels trims all ten, and the guard cells keep the published text. A test checks that no label has surrounding whitespace.

On Q5, agreed: both vintages stay.

Pins: done in 8277a07. The 60 facts left out are all UK-wide: ten in each fiscal year from 2026-27 to 2031-32, 54 of them person and 6 institutional_sector. The new pins:

  • fact count 1,661,755;
  • obr 4,965;
  • country:K02000001 40,878;
  • person 598,301 and institutional_sector 8,919.

Duplicate keys (2,687), periods and tables don't change. A bundle of the March and October OBR packages still has its 363 shared keys, and the October packages alone have none.

Tests:

  • tests/test_source_obr_efo_october_2026.py: 23 passed;
  • tests/test_source_hmrc_spi_table_3_10.py: 1 passed;
  • tests/test_chronicle_source_package.py -k "hmrc or obr": 16 passed;
  • tests/test_chronicle_bundle.py without the merged build: 26 passed;
  • tests/test_source_package_alias_drift.py: 29 passed;
  • scripts/build_obr_efo_october_2026.py --check: clean, at 49d9992 and at the head;
  • validate-package on the three changed packages: valid, no warnings;
  • ruff check: passed.

The merged build runs in CI.

🤖 Generated with Claude Code

@juaristi22
juaristi22 marked this pull request as ready for review October 29, 2026 13:22
@vahid-ahmadi

Copy link
Copy Markdown

Review of 2ba21f3, round 2

Thanks for the quick turnaround. I re-checked the six commits (49d9992..2ba21f3) against the round 1 findings, on a fresh clone and venv.

Round 1 findings

  1. should, no 2024-25: resolved. docs/data-sources/uk-obr-efo-october-2026.md:143-147 says it, and the caveat is on OBR October 2026 EFO: archive the receipts and expenditure detailed forecast tables as a new vintage (PolicyEngine/microcosm#1095) #393. I checked the March packages: all 33 concepts have a fy2024 fact, each an observation (for example obr.efo_2026_03.receipts.income_tax.fy2024).
  2. should, restated rows: resolved.
    • The copies are left out in scripts/build_obr_efo_october_2026.py:666, :779 and :798, and check_restatements (:880) asserts each one against the row kept: betting and gaming against 4.10 (exact), the seven 4.22 rows against 4.20 and 4.2 (:902, rel 1e-12), 4.21's £m row against 4.20 ×1,000 (:920), and VAT refunds against 4.8. --check is clean, so the committed YAML is what those asserts guard.
    • I re-ran my pair check on the pushed receipts package (1,551 facts, 212 series). Pairs equal in 5 or more years: DPT (7/7) and RPDT (6/7), now one concept on two bases, and six pairs among bingo duty, feed-in tariffs, onshore CT 'Other' and the PPF levy, which are zero in most years. Nothing else. Self assessment (4.4 against 4.8) matches in six years to rounding, not five, but 2029-30 really differs (84.7 against 102.3), and 4.8's income tax total adds up with 102.3, so separate concepts are right.
    • The eight cash and accrued pairs share obr.<tax> with exact and differ only in accounting_basis (:816). No semantic key repeats inside the package.
    • Betting and gaming: 2025-26 accrued (3.892) is obr.betting_and_gaming_duties on accrual, next to the cash 3.831, and 4.10's total carries the same concept from 2026-27. That matches how beer, wine and spirits already sit.
  3. should, Table 3.10 employment income: resolved. spi_income_bands_2023_24/source_package.yaml:6229, :6249, :6449 and :6469 carry hmrc.spi_table_3_10_employment_income_{count,amount} with broad_match, and the notes quote both definitions. The canonical concept is unchanged, so the semantic keys are too. The new test covers all 32 facts.
  4. nit, role and source_table: no change, as agreed.
  5. nit, policy labels: resolved. dimension_value_labels is trimmed (:1429; efo_policy_october_2026/source_package.yaml:148), and the guard cells still hold the tab (expected_value: "\tProperty Income: ...").

Pins. The 60 facts left out reconcile with the deltas: 8 rows of 4.22, 4.21's £m row and 4.1's betting row, 6 years each, 10 per year from 2026-27 to 2031-32, all UK-wide, 54 person and 6 institutional_sector. That gives receipts 1,611 to 1,551, obr 5,025 to 4,965, country:K02000001 40,938 to 40,878, person 598,355 to 598,301 and institutional sector 8,925 to 8,919. I compared the base and head facts directly: 60 removed, none added. 120 facts changed semantic key (the 17 rows that now share a concept, 7 years each, plus 4.1's 2025-26 betting fact), which is expected, and nothing outside this PR reads them yet. Of the new shared concepts, only obr.vaping_tax exists in another package (March's cash line), and the bases keep the keys apart.

New findings

No musts or shoulds. Two nits.

6. nit: table 4.22's total and environmental levies no longer have their parts in the same table. Total fiscally neutral receipts (178.9 in 2026-27) is kept, but only three of its six parts are (environmental levies, depreciation, council tax). VAT refunds, EPR and CIL now live under 4.8 and 4.2's concepts, and the five levies under 4.20's. That's right, but only the package header and the build script say where they went. One sentence in the notes of the 4.22 total and its environmental levies row, naming the lines that complete them, would help anyone summing 4.22.

7. nit: four March lines now say the basis twice. The new suffix at :520 gives 'Betting and gaming duties (accrued), accrued basis, FY2025-26', and the same for beer and cider, wine and spirits _accrued. Skip the suffix when the display label already says '(accrued)'. These labels also differ from the March package's now ('Tobacco duties, FY2025-26' there, 'Tobacco duties, cash basis, FY2025-26' in October). That's harmless, because measure labels aren't in the semantic key and alias drift passes. The header's "with their line ids, display labels and concepts" still holds for the row labels.

Verdict

All five round 1 points are done, and the numbers check out. Fine to merge once the bundle job is green; the two nits can go in now or later.

What I ran

  • Fresh clone at 2ba21f3 and a fresh venv. Both deleted afterwards.
  • The five pinned October workbooks in db/data/obr/ match my round 1 downloads byte for byte.
  • scripts/build_obr_efo_october_2026.py --check: clean.
  • test_source_obr_efo_october_2026.py, test_source_hmrc_spi_table_3_10.py and test_source_package_alias_drift.py: 53 passed. test_chronicle_source_package.py -k "hmrc or obr": 16 passed. test_chronicle_bundle.py without the merged build: 26 passed.
  • validate-package on the receipts, policy and SPI bands packages: no errors, no warnings.
  • ruff check at the locked version over the CI paths: passed.
  • Semantic keys across the four March and five October OBR packages: 363 shared between the vintages, none repeated within either.
  • My own scripts: the pair check, the March fy2024 check, the concept and basis listing, a base against head fact diff, and a read of tables 4.4, 4.8 and 4.22 in the workbook.

Not verified

  • The merged-bundle test: the bundle-wide duplicate count (2,687) and the absolute person and institutional sector pins. I checked the deltas and the OBR keys, not the full bundle. CI's Chronicle bundle and Chronicle surface jobs were both still pending on 2ba21f3 when I posted.

🤖 Generated with Claude Code

@juaristi22
juaristi22 merged commit 53815b2 into main Oct 29, 2026
3 checks passed
juaristi22 added a commit to PolicyEngine/microcosm that referenced this pull request Oct 29, 2026
… 3.10

PolicyEngine/chronicle#394 brings the October 2026 EFO and SPI Table 3.10
(María's rulings of 2026-10-29).

- The SPI Table 3.7 interest and other-income rows by band grow from
  2023-24 to the 2025 calibration period by the OBR's savings income
  growth (October 2026 EFO, Table 4.5), vendored as
  obr_savings_income_growth_facts.json, in place of ONS household
  interest income then GDP per head.
- SPI Table 3.10's individuals with self-employment income, and that
  income, bind by self-employment band from GBP 1 to GBP 10,000. The
  targets fan out over every published band; the zero band and the bands
  from GBP 10,000 are signed out row by row in
  target_reference_signed_exclusions.json (a list in dimension_values
  would have summed the bands instead). They are declared
  signature_incomplete, since the band is the groupby, not a measurement
  filter, and the SPI fan-out family's rationale names them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
juaristi22 added a commit to PolicyEngine/microcosm that referenced this pull request Oct 29, 2026
The UK Chronicle feed moves to 53815b2, the merge of
PolicyEngine/chronicle#394, on top of the 0b721a4 merge of
PolicyEngine/chronicle#392 (1,511,063 rows; the schema digest is
unchanged). The merge commit's tree is the pull request head's (2ba21f3),
so the artifact was built from that head's bundle and carries the merge
commit's identity.

The vendored resources and the constituency energy resource carry the
same rows under the new feed identity, and the high-value home stock's
inputs are unchanged (the feed-gated check passes). On the national
surface only the intended values move: the 13 SPI interest rows (up
13.4%), the 13 other-income rows and the 16 Table 3.10 rows.
obr.winter_fuel_allowance is pinned to March's expenditure table, since
October's FY2025-26 outturn (0.02% above March's projection) would match
it twice; the OBR targets that window FY2024-25 stay on March because
only March carries that year. On the local surface the statuses and the
references are unchanged; the membership moves its feed label and only
the 977 DESNZ gas cells, back to the Energy Trends level after commit 30
(x1.067). The local references, the national and
local compile-parity receipts, the coverage manifest and the H2 fixture
are regenerated, and the two tests that restate the pin follow it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants