Repository navigation
Add SPI 2023-24 Table 3.10 and the OBR October 2026 EFO vintage (PolicyEngine/microcosm#1418) - #394
Conversation
|
@PavelMakarchuk the OBR October 2026 EFO packages that chronicle#393 asks for are on this PR now, beside Table 3.10:
The PR body gives the scope, the semantics, the pins and three OBR questions (4 to 6). CI is running. 🤖 Generated with Claude Code |
…ine/microcosm#1418 HMRC's Survey of Personal Incomes 2023-24 Table 3.10 bands individuals with a self-employment source by their self-employment income, from the GBP 0 to GBP 1 range (a loss or profits wholly offset, counted at nil, and unknown profits) to GBP 100 000 and over, with the counts and amounts of their self-employment, employment, pension, and property, interest, dividend and other income and total income. It reads the workbook the SPI bands package already reads. Microcosm needs its GBP 1 to GBP 500 and GBP 500 to GBP 1 000 bands for the trading allowance. Unlike the other sheets, Table 3.10 is not taxpayers only, so its facts carry entity_role individual_with_self_employment_source and their own axis, hmrc.self_employment_income_band, with self_employment_income_lower_bound and self_employment_income_upper_bound. With the taxpayer role, its all-ranges row would have shared 10 semantic keys, all with different values, with Table 3.11's UK row and Tables 3.5 and 3.3. The concepts are the ones Tables 3.6, 3.7 and 3.11 already name; 'Total income' maps to total income before tax with an exact relation, as on Table 3.11. The mean and percentage columns are derived and not ported. HMRC suppresses the GBP 0 to GBP 1 band's self-employment amount, so that band is its own subset record set carrying the other nine measures. 159 facts: 15 rows of 10 measures and the subset row's 9. A single-source build of the package goes from 1,078 to 1,237 facts, and a build with the region, area and Income Tax liabilities packages adds no semantic duplicate or warning, so the bundle pins move by 159 in fact_count, hmrc, tax_year:2023, country:K02000001 and person. The region package's note that Table 3.10 is not ported is brought up to date. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#393 asks for the Economic and fiscal outlook the OBR published with the Budget on 28 October 2026. obr.uk refuses scripted downloads, so the five detailed forecast workbooks were fetched with a browser and are pinned under db/data/obr with their manifests (OGL, per the OBR's copyright page). scripts/build_obr_efo_october_2026.py writes five packages from them, 4,377 facts: - obr-efo-receipts-october-2026 (1,611): the March package's 41 lines, and every other row of tables 4.1 to 4.22 except 4.9 (no unit) and 4.19 (a '£ billion' unit over thresholds in pounds). Table 4.5's savings income is obr.self_employed_income_growth.savings_income, a growth rate from 2024-25. - obr-efo-expenditure-october-2026 (165), -economy- (56) and -aggregates- (25): the March packages' lines, plus the deposit rate and household financial assets that policyengine-uk#2638's interest projection reads. - obr-efo-policy-october-2026 (2,520): Table 1.1's costing of each policy decision by head and category, 2026-27 to 2031-32, and its summary rows. A March line keeps its line id, label and concept under obr.efo_2026_10, so a consumer changes vintage by package. Its October table is found by title (receipts 3.x are now 4.x, expenditure 4.x now 5.x) and its row by label. Each year's assertion follows the table's Outturn/Forecast band: 2025-26 is now outturn, and 2031-32 is new. Northern Ireland domestic rates are carried in both places the OBR shows them. The build asserts each equality the notes cite (restated rows left out, table 4.22's mislabelled unit). The manifests carry each file's canonical R2 key; publish-raw has not run yet (no Wrangler on this machine). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A data-source page with the files, conventions and caveats (the two tables left out, Table 4.5's growth rates, the two Northern Ireland domestic rates placements, table 4.22's unit, the restated rows and the outturn bands), the Annex A check, what Microcosm selects, a checklist section and a changelog fragment. Annex A is the EFO's detailed summary tables, not policy measures, so it has no measure row the policy workbook lacks and is not archived. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The five packages add 4,377 facts (obr 648 -> 5,025), five packages and source tables, and their fiscal- and calendar-year periods. Their entities: government 2,601, people 1,601, institutional sectors 119 and firms 56. The United Kingdom gains 4,279 facts, Great Britain 56 and England and Wales 28 (table 5.9's DWP lines). The March lines' projections for the years both vintages forecast add 363 semantic duplicate keys, kept as published beside the March packages. Measured on bundles of the OBR packages; CI runs the merged build. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI's bundle job on 968c5e6 stopped at the summary counts: period_count is 554, not 553. Every other summary count matched. 2031-32, the October 2026 EFO's new forecast year, is a fiscal_year key no other package carries. The by_period expectation already adds its count. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
968c5e6 to
9bb348f
Compare
|
Rebased onto main now that #392 is merged, so the commits in my last comment have new SHAs:
The first bundle run stopped at The rebased pins sum #392, Table 3.10 and the OBR packages: 1,661,815 facts, 2,687 semantic duplicate keys, 428 packages. The PR body's numbers are updated to match. 🤖 Generated with Claude Code |
|
Review of 9bb348f (draft, so a comment rather than a formal review). This is the first review round: there are no earlier review comments or inline threads, and María's two comments (02:39 and 10:08) are author updates. The rebased pins in the body match the head. No musts. Three shoulds and two nits. Findings1. should: October carries no 2024-25 for the 33 lines Microcosm selects, and the docs don't say so. 2. should (your Q4): table-scoped concepts carry the same published series under two or three ids.
A consumer summing a table-scoped concept family will double count these. The fix could follow either the 4.7 rule (drop the restated row) or this PR's own SPI precedent: Table 3.5's 'Other income' carries Table 3.7's concept with 3. should (your Q2): Table 3.10's employment income should get its own source concept.
4. nit (your Q1): the role name is fine. 5. nit: three policy measure labels start with a tab. On your Q5, I agree with keeping both vintages:
I recomputed the 363: all are projection against projection, and 5 values are identical. No March package file is in the diff, and Chronicle has no "latest" or default-vintage selector, so existing consumers see no behaviour change until they re-point. What I ranFresh shallow clone of 9bb348f, fresh venv, deleted afterwards. SPI Table 3.10
OBR October 2026
Tests
Not checked
VerdictSound and well evidenced. Nothing blocks. Items 1 to 3 are worth doing before this leaves draft, and the bundle job needs to go green. 🤖 Generated with Claude Code |
…ued taxes Vahid's second point on chronicle#394. Table 4.22's VAT refunds, CIL, EPR and five environmental levies restate tables 4.8, 4.2 and 4.20, and table 4.21's forecast row restates table 4.20's contracts for difference. Each is now ported once, and the build asserts every copy equal to the row kept. Table 4.1's betting and gaming row restates table 4.10's total from 2026-27, so only 2025-26 is kept, under that total's concept. Tables 4.1 (accrued) and 4.8 (cash) print eight of the same taxes. Each pair now shares one concept with an exact relation and differs in accounting_basis, as chronicle#381's excise duties do; each keeps its table's source concept. Table 4.1's vaping tax joins obr.vaping_tax. Row and measure labels name the basis. The receipts package drops from 1,611 to 1,551 facts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Vahid's fifth point on chronicle#394. In Table 1.1, three measure names start with a tab and seven end in a space. Their labels in dimension_value_labels are trimmed; the guard cells keep the published text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Vahid's third point on chronicle#394. Table 3.10's employment income is ITEPA
2003 income plus other miscellaneous income (its note 4), so it is wider than
Table 3.6's (note 6). Its count and amount now carry
hmrc.spi_table_3_10_employment_income_{count,amount} as source concepts, mapped
to Table 3.6's as a broad match, as the table's total income already is. The
canonical concepts, and so the semantic fact keys, don't change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Vahid's first and second points on chronicle#394. "For Microcosm" now says that October's 33 lines start at 2025-26, so the 2024-25 part of the calendar-2025 window has to come from the March package's outturn or from ONS. The receipts section lists the rows ported once and the eight cash and accrued pairs, and the counts move to 4,317 facts (receipts 1,551). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The October receipts package has 60 fewer facts: 54 person and 6 institutional sector facts, ten in each of fiscal years 2026 to 2031, all UK-wide. So the fact count is 1,661,755, obr 4,965, K02000001 40,878, person 598,301 and institutional_sector 8,919. Duplicate keys, periods and tables don't change: a bundle of the March and October OBR packages still has 363 shared keys. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The receipts package's header comment still named only table 4.7's two rows. It now lists each row ported once and the eight cash and accrued pairs, as the docs do. Only the comment changes; no fact moves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@vahid-ahmadi thanks for the review, and for re-deriving the restatement pairs from the facts. All five points are addressed in six pushed commits, 49d9992..2ba21f3. The PR body is updated, and CI is re-running on 2ba21f3.
On Q5, agreed: both vintages stay. Pins: done in 8277a07. The 60 facts left out are all UK-wide: ten in each fiscal year from 2026-27 to 2031-32, 54 of them
Duplicate keys (2,687), periods and tables don't change. A bundle of the March and October OBR packages still has its 363 shared keys, and the October packages alone have none. Tests:
The merged build runs in CI. 🤖 Generated with Claude Code |
|
Review of 2ba21f3, round 2 Thanks for the quick turnaround. I re-checked the six commits (49d9992..2ba21f3) against the round 1 findings, on a fresh clone and venv. Round 1 findings
Pins. The 60 facts left out reconcile with the deltas: 8 rows of 4.22, 4.21's £m row and 4.1's betting row, 6 years each, 10 per year from 2026-27 to 2031-32, all UK-wide, 54 New findingsNo musts or shoulds. Two nits. 6. nit: table 4.22's total and environmental levies no longer have their parts in the same table. Total fiscally neutral receipts (178.9 in 2026-27) is kept, but only three of its six parts are (environmental levies, depreciation, council tax). VAT refunds, EPR and CIL now live under 4.8 and 4.2's concepts, and the five levies under 4.20's. That's right, but only the package header and the build script say where they went. One sentence in the notes of the 4.22 total and its environmental levies row, naming the lines that complete them, would help anyone summing 4.22. 7. nit: four March lines now say the basis twice. The new suffix at VerdictAll five round 1 points are done, and the numbers check out. Fine to merge once the bundle job is green; the two nits can go in now or later. What I ran
Not verified
🤖 Generated with Claude Code |
… 3.10 PolicyEngine/chronicle#394 brings the October 2026 EFO and SPI Table 3.10 (María's rulings of 2026-10-29). - The SPI Table 3.7 interest and other-income rows by band grow from 2023-24 to the 2025 calibration period by the OBR's savings income growth (October 2026 EFO, Table 4.5), vendored as obr_savings_income_growth_facts.json, in place of ONS household interest income then GDP per head. - SPI Table 3.10's individuals with self-employment income, and that income, bind by self-employment band from GBP 1 to GBP 10,000. The targets fan out over every published band; the zero band and the bands from GBP 10,000 are signed out row by row in target_reference_signed_exclusions.json (a list in dimension_values would have summed the bands instead). They are declared signature_incomplete, since the band is the groupby, not a measurement filter, and the SPI fan-out family's rationale names them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The UK Chronicle feed moves to 53815b2, the merge of PolicyEngine/chronicle#394, on top of the 0b721a4 merge of PolicyEngine/chronicle#392 (1,511,063 rows; the schema digest is unchanged). The merge commit's tree is the pull request head's (2ba21f3), so the artifact was built from that head's bundle and carries the merge commit's identity. The vendored resources and the constituency energy resource carry the same rows under the new feed identity, and the high-value home stock's inputs are unchanged (the feed-gated check passes). On the national surface only the intended values move: the 13 SPI interest rows (up 13.4%), the 13 other-income rows and the 16 Table 3.10 rows. obr.winter_fuel_allowance is pinned to March's expenditure table, since October's FY2025-26 outturn (0.02% above March's projection) would match it twice; the OBR targets that window FY2024-25 stay on March because only March carries that year. On the local surface the statuses and the references are unchanged; the membership moves its feed label and only the 977 DESNZ gas cells, back to the Energy Trends level after commit 30 (x1.067). The local references, the national and local compile-parity receipts, the coverage manifest and the H2 fixture are regenerated, and the two tests that restate the pin follow it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This PR carries two Budget 2026 data requests for PolicyEngine/microcosm#1418. It is rebased onto main after #392 (ISA) merged, so its pins include #392's:
hmrc-spi-income-bands-2023-24(below, unchanged);1. SPI 2023-24 Table 3.10
This PR adds HMRC Survey of Personal Incomes 2023-24 Table 3.10 to
hmrc-spi-income-bands-2023-24: income of individuals with self-employment income, banded by self-employment income. Microcosm needs its small-profit bands for the Budget 2026 trading allowance measure (Table 6.1 line 15; PolicyEngine/microcosm#1418, item 13).What it adds
Collated_Tables_3_1_to_3_11_2324.ods), sheetTable_3_10.hmrc.spi_2324.table_3_10: 14 bands from GBP 1 to GBP 100 000 and over, plus All ranges astotal, 10 measures each;hmrc.spi_2324.table_3_10.subset_1: the GBP 0 to GBP 1 band. HMRC suppresses its self-employment amount, so it carries the other 9 measures.hmrc.self_employment_income_band, withself_employment_income_lower_boundandself_employment_income_upper_bound. Both edges are carried, as for the total-income bands.Universe
Unlike the package's other sheets, Table 3.10 is not taxpayers only: per its scope line, it counts every individual with a self-employment source, taxpayer or not. So its facts carry
entity_role: individual_with_self_employment_source.Concepts
hmrc.spi_table_3_10_employment_income_{count,amount}, mapped to Table 3.6's concepts as abroad_match. Table 3.10's note 4 defines it as ITEPA 2003 income plus other miscellaneous income, which is wider than Table 3.6's (note 6). The canonical concepts, and so the fact keys, are unchanged.Pins (Table 3.10)
tests/test_chronicle_bundle.pymoves by 159 infact_count/row_count(1,657,279 → 1,657,438 on main after #392), inhmrc(36,513 → 36,672), intax_year:2023, incountry:K02000001(36,500 → 36,659) and inperson(596,595 → 596,754).Semantic duplicates (2,324), warnings (133), source tables, and the geography, period and package counts are unchanged. A single-source build and a four-package build (bands, region, area, Income Tax liabilities) both show +159 with no new duplicate or warning.
2. OBR EFO October 2026 (chronicle#393)
The OBR published the October 2026 EFO with the Budget on 28 October 2026. policyengine-uk moves its economic assumptions to it (PolicyEngine/policyengine-uk#2717), and PR C (PolicyEngine/microcosm#1420) needs receipts Table 4.5's savings income. obr.uk refuses scripted downloads, so the five detailed forecast workbooks were fetched with a browser and pinned with their manifests (OGL).
scripts/build_obr_efo_october_2026.pywrites five packages, 4,317 facts:obr-efo-receipts-october-2026obr-efo-expenditure-october-2026obr-efo-economy-october-2026obr-efo-aggregates-october-2026obr-efo-policy-october-2026Semantics (
docs/data-sources/uk-obr-efo-october-2026.mdhas the detail):obr.efo_2026_10, so a consumer changes vintage by package. Its October table is found by title (receipts 3.x are now 4.x, expenditure 4.x now 5.x) and its row by label; each record set carries its October table and row. The 33 concepts Microcosm selects are all there.obr.self_employed_income_growth.savings_income, measuregrowth_rate: a percentage change on a year earlier from 2024-25, all under the table's 'Forecast' band. The table has no 2023-24 column. 2024-25 and 2025-26 give the ×1.358 path from 2023-24.concept_relation: exactand differs inaccounting_basis, as chronicle#381's excise duties do. Each keeps its table's source concept. Table 4.1's vaping tax and 2025-26 betting and gaming join the March lines'obr.vaping_taxandobr.betting_and_gaming_duties. Row labels say '(cash)' or '(accrued)', and the March lines' measure labels name their basis.obr.efo_2026_03.<kind>.<line id>.fy2024) or from ONS. The docs' "For Microcosm" section says so.value_scale1e9. The build asserts the evidence.publish-rawuploaded the five workbooks toledger-raw, andverify-rawconfirms each object's sha256.Pins (OBR)
tests/test_chronicle_bundle.pymoves by the five packages:fact_count/row_count1,657,438 → 1,661,755;obr648 → 4,965;period_count553 → 554: 2031-32 is a fiscal year no other package carries;country:K02000001+4,219,K03000001+56 andK04000001+28;Semantic duplicate keys go from 2,324 to 2,687 (+363); #392 added none. These are the March lines' projections for the years both vintages forecast, kept as published beside the March packages. A consumer picks a vintage by package; #393 asks that March stay addressable. A bundle of the five October packages alone is valid with no warning and no duplicate key.
Role and paths
ledger-source-ingestor. Paths outside the role's list, as #391 and #392 declared:
db/data/**;chronicle/source_package.pyandchronicle/bundle.py(registration);docs/**andchangelog.d/**;scripts/build_obr_efo_october_2026.py, beside UK excise facts for alcohol, tobacco and gambling: the OBR duty and clearance lines already archived, the revised Consumer Trends release, Gambling Commission GGY and HMRC betting and gaming receipts (PolicyEngine/microcosm#1359) #381's builders.Checks run
validate-packageon both SPI packages, andbuild-suiteon this one;tests/test_chronicle_source_package.pythat touch HMRC packages;The hour-long
test_build_bundle_writes_merged_consumer_contractwas not run locally; CI runs it.validate-packageand the consumer contract on all five packages: valid, no warnings, no duplicate key within any package.tests/test_source_obr_efo_october_2026.py: 23 passed. It covers:tests/test_source_hmrc_spi_table_3_10.py: 1 passed (Table 3.10's employment source concept and notes).tests/test_chronicle_bundle.pywithout the merged build: 26 passed;ruff checkpassed.Questions for review, answered
Vahid answered all five in his review of 9bb348f (#394 (comment)). His three shoulds and two nits are addressed in 49d9992..2ba21f3.
individual_with_self_employment_sourcestays: the role keeps Table 3.10 off the taxpayer keys.source_tablestays as it is.The Microcosm side (national targets for the small-profit counts and amounts) waits for the next feed re-pin. PR C rebuilds the bundle from this PR's head and re-pins once it is review-complete.
🤖 Generated with Claude Code