[anth-volk]

Fixes #225

Summary

• Upgrades .py to exactly policyengine[us]==6.7.5 (not 6.7.6) for the hosted runtime (backend/requirements.txt) and the root evaluation harness (pyproject.toml).
  • Its us extra pins the same packages as 6.4.1: policyengine-us 2.2.1, policyengine-core 3.33.0 and spm-calculator 1.0.0.
  • Its bundle certifies the same national dataset, populace-us-2024-spm-20260915. The US build ID, certification, certified data artifact (sha256 6496cc43…aee84), default dataset URI, release manifest and every file record that hosted_release.py checks are unchanged.
  • What 6.7.5 does change is the ACS local-area dataset and the state and congressional-district region datasets. The hosted runtime does not load them.
• Fixes WIC take-up in hosted calculations.
  • The bug: the reviewed H5 stores each person's decision as would_claim_wic. policyengine-us 2.2.1 renamed that input to takes_up_wic_if_eligible (default True) and skips the stored column. Because the hosted runtime constructs policyengine_us.Microsimulation directly, production treated every WIC-eligible person as enrolled.
  • The fix: every hosted simulation, both national and California, now applies .py's stored-input renames. The root evaluation harness does the same, so both paths agree.

How the data is loaded

• Why not managed_microsimulation: .py's public loader doesn't fit the hosted runtime, for three reasons.
  • It takes only a dataset path or URI. A local file needs allow_unmanaged=True, and then loses its certified identity.
  • It passes that path to Microsimulation(dataset=path), which allocates every economic-assumption year. The hosted runtime builds a one-year USMultiYearDataset to avoid that.
  • It cannot take the in-memory California subset.
• What the code calls instead: after constructing the simulation, both paths call the step managed_microsimulation itself calls: policyengine.tax_benefit_models.us.legacy_inputs.apply_legacy_input_renames_to_microsimulation.
  • It reads the per-year entity tables that the one-year USMultiYearDataset holds.
  • It sets takes_up_wic_if_eligible from the stored would_claim_wic for each month.
  • It stops applying once the data stores the live name, or once the engine defines the legacy name again.
  • This repository does not copy the mapping.
• The supported-API gap: the function has a public name and is documented in its module, but policyengine.tax_benefit_models.us does not re-export it. That gap is what Provide an API to resolve, verify and load the datasets .py certifies policyengine.py#667 describes.
• New response field: each calculation response records the renames applied per scope, in execution.legacy_input_renames (for example {"national": {"would_claim_wic": "takes_up_wic_if_eligible"}}).
• Image build-time check: verify_hosted_runtime.py, which runs during the image build, now binds the .py function. It also records the renames the pinned model needs, as .py computes them.
• SPM results don't change:
  • In 6.7.5, .py's legacy SPM handling is registered only for two populace_us_2024_acs_local SHA-256s. The national file is not one of them.
  • resolve_spm_selection(None) equals policyengine-us's default provider: forecast 3d86d5c4…, scenario ce_trend, county geography, 2020 vintage.
  • The deployment check values below confirm it.

Commits

Commit	Red (before the implementation)	Green
35dff74 Pin .py 6.7.5 for the hosted runtime and the evaluation harness	19 failing: 17 hosted (in test_model_dependency_pins.py, test_hosted_release.py, test_hosted_runtime_verification.py and test_modal_endpoint.py) and 2 root. The cause in each case: the config still named 6.4.1 while 6.7.5 was installed or expected.	hosted 45 passed (the 4 files plus test_variable_catalog.py); root 38 passed
bc63330 Apply .py's stored-input renames in hosted calculations and the harness	4 failing: 2 hosted WIC tests (the child who declined WIC gets $701.55), the image-verification receipt (missing the renames record), and 1 root harness WIC test ($701.55)	hosted 52 passed (8 files); root 39 passed

• Lock changes: both locks were regenerated with their documented commands, run in place so the existing pins act as preferences. The only pin that moves is policyengine 6.4.1 → 6.7.5.
• PyPI: every pin exists and none is yanked (118 distinct pins: 87 in the hosted lock, 100 in uv.lock). Every hash in backend/requirements.lock is one PyPI publishes.
• Variable catalog: regenerated from the reviewed H5. Its 6,167 variables are unchanged; only its recorded .py version moves.

WIC evidence

Fixture tests:

• frontend/tests/test_stored_wic_take_up.py covers the national scope and the California scope through calculate_variables.
• tests/test_microcosm_stored_wic_take_up.py covers the harness.
• The fixture: one California household with no income: a parent and two WIC-eligible children, one stored with would_claim_wic = False and one with True.
• Before: both children get $701.55 of WIC in 2024.
• After: the child who declined has takes_up_wic_if_eligible = False and wic = 0, and the claiming child keeps $701.55.

Real data: one before-and-after run on the reviewed national file, using a local, sha256-verified copy.

• Both runs use .py 6.7.5 and the hosted one-year builder. "Before" is today's construction; "after" adds the .py step.
• The run took 373 s for four simulations, with peak resident memory of 4.89 GB.

	Before	After
National wic, 2024 total	$11,517,241,344	$6,633,045,504
Weighted WIC recipients, any month of 2024	12,653,956	6,661,355
Person records with WIC	6,914	3,648
Person records with takes_up_wic_if_eligible, January	166,321 (all)	7,470 (the stored True count)
Weighted WIC-eligible people, January	12,460,186	12,460,186
California wic, 2024 total	$1,167,677,340	$593,302,036
National spm_unit_spm_threshold (deployment check)	3,887,638,030,556.44	3,887,638,030,556.44
ca_income_tax, 3,876 tax units (deployment check)	121,198,406,078.07	121,198,406,078.07

Other variables that include WIC change as well, such as spm_unit_benefits, household_benefits and the SPM resource and poverty measures built on them.

Deploy implications

• What runs on merge: merging triggers deploy.yml after CI. It deploys the Modal backend, checks real national (spm_unit_spm_threshold) and California (ca_income_tax) calculations, then promotes. Today's deploy took 179 s for those calculations, of 790 s allowed.
• Image build: it installs the new lock and runs verify_hosted_runtime.py, which now imports .py.
  • Run locally, the verification takes 11.4 s, compared with 6.8 s without the .py import (warm). On a first run, it took 53.7 s while Python compiled the bytecode.
• First calculation in a cold container: it also imports .py. That import builds .py's us_latest, which repopulates the variable and parameter metadata from the already loaded policyengine-us system. That took about 5 s locally.
  • The same import sends one GET for the US release_manifest.json to huggingface.co, with a 30 s timeout. If the request fails, .py falls back to its bundled certification; locally, the import succeeded with DNS blocked.
  • The import writes no files. The runtime never calls .py's materialize_dataset, whose default destination is ./data. The H5 is still downloaded only by the existing hf_hub_download call, into /tmp/huggingface/hub.
• Effect on the deployment checks: their values are unchanged (table above), so the added time stays well within the 790 s limit.
• Memory: the Modal allocation stays at 16 GiB.

Deployment verification: environment variables

Variable	Read or declared by	Provider	Wiring into the Modal image and runtime	Status
CALIBRATION_MODAL_APP_NAME	backend/modal_app.py	deploy.yml step output steps.identity.outputs.modal_app	Step env: of "Deploy Modal backend"; baked into the image by .env()	Unchanged. Wiring confirmed in deploy.yml and modal_app.py
CALIBRATION_SOURCE_COMMIT, CALIBRATION_SOURCE_TREE_SHA256	modal_app.py, runtime_identity.py	Computed from Git by modal_app.py at deploy time	Image .env()	Unchanged
CALIBRATION_REQUIREMENTS	verify_hosted_runtime.py	modal_app.py image .env() (/root/backend/requirements.txt)	Image environment, used by the build-time run_commands	Unchanged; exercised locally with the /root layout
PYTHONPATH	Python	modal_app.py image .env() (/root)	Image environment	Unchanged
HF_HOME, HF_HUB_CACHE, XDG_CACHE_HOME, HF_HUB_DISABLE_XET	microcosm_variable_core.py (os.environ.setdefault)	Code defaults under /tmp	Not set by the image, so the defaults apply. /tmp is the container's writable disk	Unchanged
POPULACE_HF_REPO, POPULACE_HF_REVISION	backend/variable_endpoint.py	None; must be absent	Not set on Modal	Unchanged
POLICYENGINE_SKIP_COUNTRY_IMPORTS (new to the runtime, through the .py import)	policyengine/__init__.py	None; must stay unset so the US model module loads	Not set by the image .env(). web_app attaches no Modal secrets	Unset in the repository's configuration. The live container environment has not been observed: unverified
HUGGING_FACE_TOKEN, HF_TOKEN, HUGGINGFACE_HUB_TOKEN (new to the runtime, through the .py import)	policyengine.provenance.manifest.hugging_face_auth_headers, for the release-manifest GET	None for Modal; optional	web_app has no secrets= and the image sets none. HF_RELEASE_READ_TOKEN reaches only Vercel, as HF_TOKEN (deploy-dashboard.sh deploy_frontend)	Not required, because policyengine/populace-us is public: the request is unauthenticated, and a failure falls back to the bundled certification. Whether the GET succeeds from the Modal container is unverified; the container already downloads the H5 from the same host
MODAL_TOKEN_ID, MODAL_TOKEN_SECRET	Modal CLI in deploy.yml	GitHub Actions secrets	Step env: in deploy.yml	Unchanged by this PR. The secrets' presence was not read; today's successful deploy is indirect evidence only

Checks run locally

These mirror CI's hosted-python job, in a fresh Python 3.12 environment:

• uv pip install --require-hashes -r backend/requirements.lock, followed by pytest, httpx 0.28.1 and modal 1.5.5;
• uv pip check, which was clean;
• verify_hosted_runtime.py, which passed;
• pytest on the changed and added files plus the certificate and catalog tests: 52 passed;
• the Modal image's build-time command, run with a copy of the /root layout and CALIBRATION_REQUIREMENTS/PYTHONPATH set as the image sets them: passed;
• verify_hosted_runtime.py --dataset-file with the real H5 and the published manifests: passed;
• the root tests touched: 39 passed.

Not changed

• frontend/HOSTED_RUNTIME.md: left unchanged because Extract the calibration tree builder into a self-contained workspace package #220 edits it.
  • Its statements are still accurate: 6.4.1 certifies this dataset, and its memory note is labelled as a 6.4.1 measurement.
  • It does not yet describe the rename step. That can follow once Extract the calibration tree builder into a self-contained workspace package #220 merges.
• frontend/scripts/generate_variable_catalog.py: still builds a plain simulation, because it lists variable metadata only and stored input values do not affect it.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
calibration-diagnostics	Ready	Preview	Oct 19, 2026 2:14pm UTC