[anth-volk]

Fixes #199

Incident

On 2026-10-20 it was disclosed that eight eMudhra (emSign) intermediate CA certificates were issued with private keys anyone can recover from the public certificate, so anyone can issue a certificate for any domain that chains to "emSign ECC Root CA - G3" (and, through cross-signatures, to Sectigo's "AAA Certificate Services"). Browsers and current operating systems already block them; what stays exposed is software that carries its own CA list or reads an operating-system CA store that has not been updated. Details: https://emudhra.we-have-a-problem.com

What in this repository trusted the affected roots

The hosted MCP server (Modal app policyengine-macro-mcp, built from integration/modal_app.py) makes outbound HTTPS requests from its containers: Hugging Face microdata downloads for population_reform_impact, and PolicyEngine data fetches. Its image carries two CA lists, and both trusted the compromised root:

• certifi 2026.7.22 (used by requests and httpx). The pip_install layer was last built on 2026-08-12 (deploy run 31608860051: Successfully installed ... certifi-2026.7.22 ... hf-xet-1.6.0 ... requests-2.34.2). Every deploy since has reused that cached layer: the 2026-10-08 deploy (run 37774203311) reports Requirement already satisfied: certifi in ./usr/local/lib/python3.12/site-packages ... (2026.7.22), and the 2026-10-12 deploy built no layers at all. The fixed release is certifi 2026.10.19.
• Debian bookworm ca-certificates (used by urllib, aiohttp, and hf-xet 1.6.0, which reads the operating-system store). Modal's debian_slim base is bookworm (the apt output in the same deploy logs shows bookworm, bookworm-updates, bookworm-security), and no step in the image refreshes ca-certificates. The layers above it were built on 2026-08-12, before Debian published the fixed 20250419~deb12u2 to bookworm-security.

No grpcio is installed in the image (not in the 2026-08-12 install list), and no code in the repository overrides TLS verification or sets a custom CA bundle.

Not affected:

• The Vercel project macromod serves this repository as a static site (no api/ directory, no package.json; the latest production deployment has a single static build and no functions), so there is no server-side code on Vercel.
• The GitHub Actions workflows run on GitHub-hosted runners and install unpinned packages fresh on every run; astral-sh/setup-uv@v8.3.2 has no version input, so it installs the latest uv.

Changes

1. CA lists in the image

integration/modal_app.py appends two layers at the end of the image chain, after the editable installs:

• .apt_install("ca-certificates"), which Modal runs as apt-get update then apt-get install -y ca-certificates, upgrading the package to 20250419~deb12u2 from bookworm-security.
• .pip_install("certifi>=2026.10.19"), which keeps the fixed certifi minimum whatever the layers above resolve to (certifi has no dependencies).

2. Move to policyengine 6.11.1

The image's pip_install layer pinned policyengine-us==1.764.6, which PyPI no longer hosts, so the layer could not be rebuilt, and the nightly contract job has failed on every run since because its uv pip install -e ".[models]" cannot resolve that pin. Every policyengine release from 4.17.3 to 5.3.1 pins a policyengine-us 1.x version that PyPI no longer hosts. The only 4.x releases whose pins PyPI still hosts are 4.13.1 to 4.17.2, and those would move the models back to older versions than the server runs now.

The image and the [models] extra in integration/pyproject.toml now pin policyengine[models]==6.11.1, the newest release, and list the country packages that release pins: policyengine-us 2.2.1, policyengine-uk 2.151.0 and policyengine-core 3.33.3 (spm-calculator 1.0.0 comes with the extra). policyengine's bundled manifest certifies its US and UK microdata for exactly these versions, and import policyengine fails certification with any other policyengine-us, so a newer policyengine-us (PyPI has 2.75.0) cannot be used. There is no lockfile in integration/.

3. US households need a Supplemental Poverty Measure geography

policyengine-us 2.x computes the Supplemental Poverty Measure (SPM) in every default US household output, and a state alone does not identify an SPM area, so every US calculate_household and household_reform_impact call raised SPM_GEOGRAPHY_REQUIRED.

This follows how the other PolicyEngine projects supply it through policyengine.py 6.x: the household's five-digit county_fips, or an explicit spm selection, with no default.

• policyengine.py's household docs pass household={"state_code": ..., "county_fips": ...} or spm={"geography_kind": "national"}. They say national measurement is a conscious analytical choice.
• The PolicyEngine API (docs/canonical-spm.md) takes spm beside the household and does not infer national geography when it is omitted.
• The web app (SPMGeographyChoice.tsx) requires the user to choose national thresholds or a county FIPS code before it calculates.

What changes in this repository:

• calculate_household, household_reform_impact, core.pe_household, core.pe_household_impact and the pe-macro household / household-impact commands take an optional spm dict (--spm for pe-macro), passed unchanged to policyengine.py's calculate_household(spm=...).
• A US household with neither a county_fips nor a national or metro selection is rejected before PolicyEngine is imported. The error names the three choices.
• policyengine-us takes the state and the county independently. A county with no state was run as a California household in that county (checked: county_fips="36061" alone gave state_code=CA). The adapter therefore requires the state with a county, and checks that the county's first two digits match the state's FIPS code.
• The state can be given as state_code, the key policyengine.py documents, or as state_code_str, the key this package documented before. policyengine-us 2.2.1 reads both; this was checked for CA and TX. The adapter used to reject state_code.
• US results gain an spm entry: the resolved selection and the SPM area each year was measured in. policyengine.py's full receipt is about 690 KB for one household, so it is not returned.
• The published US examples had no state or county. These are the commands on /connect and /forecasts/us (the latter regenerated with economy/topics.py), the /connect example prompt, and the policyengine.py snippet on /pe/code. They now name Los Angeles County (06037), as policyengine.py's own examples do.

The UK population default dataset stays enhanced_frs_2023_24, which is in policyengine 6.11.1's manifest. policyengine's certified UK default is now microcosm_uk_2024_25 (policyengine/populace-uk-private). Switching to it would change the published UK costing, and /pe/validation records that a newer UK data release is deliberately not adopted, so this PR does not switch. With no dataset argument, the US population tools follow policyengine's certified default, as before. In 6.11.1 that default is Populace US 2024 (populace_us_2024).

Which tool outputs change

All figures are from real runs. "Before" means policyengine 4.22.3, policyengine-us 1.764.6 and policyengine-uk 2.89.2, which is what the deployed server reports in its provenance. "After" means policyengine 6.11.1 with this branch.

Tool and input	Before	After
calculate_household UK: age 35, £50,000, 2026	income tax £7,486.00; NI £2,994.40; HBAI net income £39,519.60	identical (household tax £10,660.45 becomes £10,660.40)
household_reform_impact UK: same person, basic rate 25%	income tax +£1,871.50	identical
list_reform_parameters	13 curated paths, all live	same baseline values for all 13
calculate_household US: NY head of household aged 30, $30,000 wages, child aged 5	household_benefits $17,910.02 (SNAP 287.68 + school meals 1,130.96 + Head Start 16,491.37); net income $51,334.70	county 36061 (New York County): benefits $1,908.08, net income $35,374.59. County 36001 (Albany), or spm national: benefits $1,446.08, net income $34,870.77
calculate_household US: single, 35, $60,000, CA (county 06037 after)	state income tax $1,602.86; net income $48,001.14	state income tax $1,595.10; net income $48,008.90
calculate_household US: single, 35, $50,000, CA (county 06037 after)	state income tax $1,002.86; net income $40,697.14	state income tax $995.10; net income $40,704.90
population_reform_impact US: CTC adult-dependent amount $1,000, 2026	not measured: on the deployed server the call through start_job hit its 1,800 s job limit, and a local run with the old versions ended without output	costs $7.935bn per year; household net income +$7.935bn; 12,496,986 households gain, none lose (populace_us_2024, 57,240 households; 2,903 s on a laptop)
population_reform_impact UK: basic rate 21%, 2026	raises £6.463bn (deployed server, enhanced_frs_2023_24, 53,508 households)	not measured: no available Hugging Face token can read the UK microdata (see below)

Why the New York household's benefits fall. Geography does not cause it. In policyengine-us 2.x, household_benefits counts Head Start and Early Head Start only when gov.simulation.include_head_start_benefits_in_net_income is true. That parameter defaults to false. The child, aged 5, still qualifies for Head Start (New York's per-enrollee value is $16,491.37 in both versions), but the value is no longer counted. With that parameter set to true, policyengine-us 2.2.1 gives benefits of $17,937.45 for Albany County. The remaining $27.43 against the old $17,910.02 is SNAP (+$16.16) and school meals (+$11.28).

The county matters too. Without a county, both versions apply the state's first county, Albany. New York County raises SNAP by $462 and lowers state income tax by $41.83 compared with Albany. Geography accounts for nothing else in this example. The earlier figures in this PR (26,172.92 falling to 4,919.13) came from a household whose inputs were not recorded, and the same Head Start rule is the likely cause there.

Committed outputs

• Rechecked and unchanged. The UK household figures on /pe/code, /models, the registry text in capabilities.py and the pe-microsim paper (£7,486, £2,994, £39,520, +£1,872) match the runs above, so they are left as they are.

• Left alone because I could not rerun them. These all come from UK population runs:

  • the £6.46bn costing of 1p on the basic rate and £7.38bn for 2030, with the OBR-emulator results derived from it, on /obr, /obr/validation, /obr/methodology, /models, /pe, /pe/validation, validation/figures/chart_data.json, and the pe-microsim and obr-macro papers and their PDFs;
  • the £0.09bn SVAR inflation incidence on /svar.

  I have no Hugging Face token that can read the UK microdata, so publishing a new figure would break the rule that every number comes from a model run. test_population.py::test_population_impact_uk_real_basic_rate compares the costing with £6.46bn to within ±5%. Run it with a working token, and if it fails, regenerate those pages.

• Fixed. US commands and examples, listed under "Changes" above. None of them publishes a US figure.

• Tests. The US household tests in integration/tests/test_core.py now pass a county or a national selection. New fast tests cover the geography rules.

Checks

• Image. Ephemeral uvx modal run of a no-op function on the image from integration/modal_app.py, with POLICYENGINE_MACRO_IMAGE_SOURCE=github, in workspace policyengine. Nothing was deployed. The rebuilt pip_install layer took 118 s and installed policyengine 6.11.1, policyengine-core 3.33.3, policyengine-uk 2.151.0, policyengine-us 2.2.1, spm-calculator 1.0.0 and certifi 2026.10.23 on Python 3.12.10. The apt step installed ca-certificates 20250419~deb12u2.

• Tools inside the image.

  • calculate_household for the New York example (county 36061) and for the UK example gave the "After" figures in the table above.
  • A US state-only household was rejected with the geography message.
  • US population_reform_impact with run_tool_job's resources (4 CPUs, 8 GiB) did not finish within the 1,800 s job limit, and no memory error was reported. That run included the first 827 MB data download. On the deployed server the same call through start_job also hit the 1,800 s limit, so this is not new.
  • UK population_reform_impact failed for both enhanced_frs_2023_24 and microcosm_uk_2024_25: "Hugging Face rejected the configured credentials".

• CA trust, checked inside the built image.

  Host	requests + certifi	urllib + OS store
  dv-g3b-emudhra.we-have-a-problem.com (emSign chain)	certificate error	certificate error
  dv-g3b-aaa-emudhra.we-have-a-problem.com (Sectigo cross-signed chain)	certificate error	certificate error
  pypi.org	HTTP 200	HTTP 200
  huggingface.co	HTTP 200	HTTP 200

  The same before-and-after result was reproduced earlier in Docker (python:3.12-slim-bookworm), starting from the unfixed state (ca-certificates 20230311+deb12u1, certifi 2026.7.22). Before the fix, requests + certifi gave HTTP 200 for the emSign chain, and urllib + the OS store gave HTTP 200 for both eMudhra hosts.

• Resolution. uv pip compile integration/pyproject.toml --extra models resolves for Linux on Python 3.11, 3.12 and 3.13, all with the versions above. Python 3.10 did not resolve before this change either, because boe-var needs 3.11 or later.

• Nightly contract job. Dispatched on this branch: the uv pip install resolved and the run gave 19 passed, 3 skipped. The three FRB/US tests skip because they need an editable FRB/US checkout, which this job's non-editable install does not provide. The same job also passed locally on Python 3.13.

• Integration tests, run as test-integration.yml does (base install plus the OBR and SVAR packages, Python 3.13, uv run): tool surface 32 passed, reform input 28 passed, full fast suite 304 passed and 77 skipped. With --runslow on Python 3.12, all 13 household tests in test_core.py pass against the real models.

• Site. site_contract.py and site_nav.py pass, as do all 14 generator --check commands from site-provenance.yml, forecast-archive.yml and data-immutability.yml. python3 -m pytest tests gives 1564 passed and 19 skipped, and forecasts/tests passes.

This repository does not use towncrier: CHANGELOG.md is written at release time, so this PR has no changelog fragment.

Operator actions after merge

1. Replace the Hugging Face token in the Modal secret macromod-hf (workspace policyengine). Its key is HUGGING_FACE_TOKEN (checked inside a container; HF_TOKEN is not set). Hugging Face rejects its value: whoami-v2 returns HTTP 401, and so do policyengine/policyengine-uk-data-private and policyengine/populace-uk-private.
   • Use a token that can read policyengine/policyengine-uk-data-private, which holds enhanced_frs_2023_24, the UK default. If the UK default later moves to policyengine's certified microcosm_uk_2024_25, the token also needs policyengine/populace-uk-private.
   • Do this before the merge if possible. The volume policyengine-macro-pe-data holds per-year files written by policyengine 4.x. policyengine 6.11.1 rebuilds per-year files that do not record their data year, and the volume does not hold the source file, so every UK population tool (population_reform_impact, score_reform with the UK microsimulation, svar_inflation_incidence, define_scenario_incidence, the OBR reform scoring) will need a download. With the current token that download fails.
2. Merging to main changes integration/**, so deploy-mcp.yml runs modal deploy integration/modal_app.py. In that run's Deploy step, confirm:
   • the pip_install layer rebuilds with policyengine-6.11.1, policyengine-us 2.2.1 and policyengine-uk 2.151.0;
   • ca-certificates reports 20250419~deb12u2;
   • the certifi step reports certifi 2026.10.19 or later.
3. The Smoke-test the deployed server step has failed on every run since 2026-09-22 while Deploy succeeded. On 2026-10-12, test_remote_mcp.py::test_obr_shock_cgg failed with Redirect to https://policyengine--policyengine-macro-mcp-serve.modal.run/mcp not followed. That failure is not caused by this change, so judge the deploy by the Deploy step. After the token is replaced, run pytest integration/tests/test_population.py -k uk_real --runslow with that token to recheck the £6.46bn costing.
4. US clients of calculate_household and household_reform_impact must now send county_fips with the state, or spm={"geography_kind": "national"}. US results change as shown above.

Environment-variable verification

This PR adds no environment variables. Every variable the image, its functions and the workflows that build or test it read:

Variable	Provider	Wiring	Status
MODAL_TOKEN_ID, MODAL_TOKEN_SECRET	GitHub Actions repository secrets (also defined at organization level; the repository secrets take precedence)	deploy-mcp.yml Deploy step env: from secrets.*	Verified: both names exist (set 2026-07-14), and the 2026-10-08 and 2026-10-12 deploys authenticated
POLICYENGINE_MACRO_IMAGE_SOURCE	Literal github in deploy-mcp.yml	Deploy step env:; read by modal_app.py when the image is defined	Verified (the ephemeral build above used the same value)
HF_HOME, POLICYENGINE_MACRO_PE_DATA_DIR	Literals in modal_app.py	image.env({...}) on serve, run_tool_job and check_overlay_gate; read by huggingface_hub and core._pe_pop_data_folder	Verified
POLICYENGINE_MACRO_SOURCE_REVISION_*	Constants in modal_app.py	.env(SOURCE_REVISION_ENV) image layer; read by core._provenance	Verified
HUGGING_FACE_TOKEN	Modal secret macromod-hf in workspace policyengine	secrets=[modal.Secret.from_name("macromod-hf")] on all three functions. Read by policyengine 6.11.1 (provenance/manifest.py, which also accepts HF_TOKEN), by policyengine-core, and by core._pe_pop_dataset's error hint	Key name verified; value rejected by Hugging Face (HTTP 401). See operator action 1
HF_TOKEN	Not set in macromod-hf	serve and run_tool_job copy it to HUGGING_FACE_TOKEN when only it is set	Not used
Populace US microdata (policyengine/populace-us)	Public Hugging Face repository	No token needed (unauthenticated API request returns HTTP 200)	Verified
POLICYENGINE_MACRO_REMOTE_TESTS	Literal "1" in deploy-mcp.yml (smoke-test step) and validate-deployment.yml	Step env:; read by test_remote_mcp.py	Verified
POLICYENGINE_MACRO_REQUIRE_PE	Literal "1" in contract.yml	Test step env:; read by test_contract.py	Verified (dispatched run above)

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
macromod	Ready	Preview	Oct 28, 2026 12:47pm UTC