Skip to content

Add simulation and report definitions to the simulation schema - #900

Draft
anth-volk wants to merge 5 commits into
v2plan/1b-104-persistence-foundationfrom
v2plan/1b-105-simulations-and-reports
Draft

anth-volk wants to merge 5 commits into
v2plan/1b-104-persistence-foundationfrom
v2plan/1b-105-simulations-and-reports

Conversation

@anth-volk

@anth-volk anth-volk commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #899

Same-Repository Draft PR Check

  • This PR is a draft.
  • This PR is from a branch in PolicyEngine/policyengine-sim-api, not a personal fork.

Summary

Second of four stacked PRs; it builds on #898. Revision 0002 adds the two definition tables of the simulation schema. After it deploys, both exist in staging and production with no rows; nothing writes them until stage 2.

  • simulation.simulations: kind (economy or household), country, policy (null for current law), dynamic, and either a household or a region code (state/ca) with an optional dataset name (null means the region's default dataset in each catalog version). No column refers to a catalog version, so the same simulation can run again on a newer one; each run resolves the region and dataset in its own version (next PR). A check requires the population to match the kind.
  • simulation.reports: baseline and reform simulations of the report's own country and kind (composite foreign keys on simulation, country and kind), the year (1900 to 2200) and options (a JSON object). An economy report needs a reform; a household report may have only a baseline.
  • Foreign keys into the Public API's tables in the same database: (policy_id, country) to policies (id, country_id), (household_id, country) to households (id, country_id) and dynamic_id to dynamics, all ON DELETE RESTRICT, each with an index. Creating them needs REFERENCES on those tables for the migrator role; a test shows the migration fails and applies nothing without it. The runtime role needs no privilege on them.
  • v1 IDs: unique legacy_simulation_id and legacy_report_id (plain columns, since the Public API's v1 database is separate). SimulationStore.create_migrated and ReportStore.create_migrated return the existing row when the same v1 ID is migrated again.
  • Definitions never change: the runtime role may read and insert them but not update or delete them. NewSimulation and NewReport check every field before a write (a kind given as text becomes its enum), and the database checks the same fields again for any other writer; a test inserts directly to reach each database check.

Environment variables

This PR reads no new environment variable. The migration job from #898 applies revision 0002 with SIMULATION_MIGRATION_DATABASE_URL as described there.

Waits for

Testing

Rebased with #898 onto main at 2a9212f on 2026-10-28, without conflicts in this branch. Every commit was checked again after the rebase: persistence tests on PostgreSQL 17 with database tests required (100 passed at the tip), the contract library's tests, the entrypoint suite (797 passed), the rules guard and quality guards, Ruff, uv lock --check and Pyright. One review round found no bugs; its quality findings (the README's roles table, untested database checks, a kind given as text) are fixed here. A later review of the PR above this one found that revision 0002 gave its check constraints doubled names (ck_simulations_ck_simulations_kind), which alembic check does not compare; the last commit fixes the names and adds a test comparing every constraint name in the migrated database with the table definitions. The checks above were run locally; this push's GitHub Actions runs report their own results.

🤖 Generated with Claude Code

anth-volk and others added 5 commits October 28, 2026 01:00
A simulation is one policy applied to one population: a household, or an
economy named by a region code (state/ca) with an optional dataset name,
where a null dataset name means the region's default dataset in whichever
catalog version a run uses. A null policy means current law. The row holds
no catalog version, so the same simulation can run again on a newer one; each
run resolves the region and dataset in its own version (item 1.06).

Revision 0002 creates the table with:

- the population checked against the kind (a household simulation has a
  household and no region or dataset; an economy simulation has a region
  and no household);
- foreign keys into the Public API's tables in the same database, all
  ON DELETE RESTRICT: (policy_id, country) to policies (id, country_id),
  (household_id, country) to households (id, country_id), and dynamic_id to
  dynamics, with an index on each referencing column;
- a unique legacy_simulation_id for simulations migrated from the Public
  API's v1 database, and a unique (id, country, kind) for the reports'
  foreign keys;
- no UPDATE privilege for the runtime role, since definitions never change.

Creating the foreign keys needs the operator's REFERENCES grant on the
Public API tables; a test shows the chain fails without it and leaves
nothing applied. SimulationStore creates, reads and finds simulations by v1
ID, and create_migrated is idempotent by legacy_simulation_id for the
one-time migration (item 2.05). Importing the package now registers every
table, so the migration environment always compares the whole schema.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A report is a definition: its baseline simulation, its reform simulation
(required for an economy report; a household report may have only a
baseline), the year (1900 to 2200) and its options as a JSON object. Composite
foreign keys (simulation id, country, kind) into simulation.simulations make
both simulations share the report's country and kind; they are ON DELETE
RESTRICT, so a simulation a report uses cannot be deleted. A report migrated
from the Public API's v1 database keeps its v1 report ID in the unique
legacy_report_id.

Revision 0002 now creates the reports table too, and revokes UPDATE on both
definition tables from the runtime role. ReportStore creates, reads and finds
reports by v1 ID; create_migrated is idempotent by legacy_report_id, for the
one-time migration (item 2.05). NewReport checks the country, year, options
and v1 ID before any write; the database checks them again for any other
writer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
NewSimulation and NewReport typed kind as an enum but neither checked nor
converted it, so a kind given as plain text (as a mapping of v1 records
easily does) passed their checks and then failed with AttributeError when
the store read kind.value. Both now convert the kind to its enum member
when they are created, and reject an unknown kind with a ValueError that
names the field.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The database repeats each check NewSimulation and NewReport make, for any
other writer, but only the report year and options checks had a test, so a
mistake in another check's expression would have gone unnoticed. New tests
insert rows directly as the runtime role and expect each of the other
checks to refuse them: the kinds, countries, region code, dataset name,
both v1 IDs, and an economy report without a reform.

The README's roles table still gave the runtime role UPDATE on every table,
and did not say the migrator needs REFERENCES on the Public API tables
before revision 0002 deploys; both are corrected. The unused
REFERENCED_PUBLIC_API_TABLES constant is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Revision 0002 passed full check constraint names, such as
ck_simulations_kind, which the naming convention prefixed a second time,
so the database held ck_simulations_ck_simulations_kind while the table
definitions name it ck_simulations_kind. alembic check does not compare
check constraints, so nothing noticed; a later revision dropping one by its
defined name would have failed. The revision now marks the names as final
(op.f), and a test compares every constraint name in the migrated database
with the names the table definitions give.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add simulation and report definitions to the simulation schema

1 participant