v3 variables on the v2 routes: the household builder and breakdown from /v2/model-catalog/variables, unused until the switch by anth-volk · Pull Request #1441 · PolicyEngine/policyengine-app-v2

anth-volk · 2026-10-27T13:58:32Z

Stacked on #1444 (v3 parameters on demand, the generated Public API client and V3_API_ROUTES).

What changes

On the v2 routes (V3_API_ROUTES = 'v2'), v3's household builder, its variable picker, the household breakdown and the views that name variables now work from GET /v2/model-catalog/variables instead of v1's whole metadata.

Background load: every variable of the catalog version loads once in the background after a v3 page renders: on the workspace pages through WorkspaceLayout, and on the report and bill report pages. The variables go into the metadata store in the shape v1's metadata gave them, so the existing builder, picker and breakdown code reads them unchanged.
Caching and model catalog loads: the load is keyed by the model catalog load, as the parameters are. A new load asks for the variables again, and an earlier load's cache entry is removed, so only the current load's variables stay cached. Nothing is requested when the store already holds them, or when the store's model catalog is for another country than the page's. Nothing is kept in localStorage.
Parts named by a parameter: the household breakdown (VariableArithmetic) takes a parameter-named adds or subtracts list for the report's year from the dated values the route sends (part_parameter_values), instead of from the whole parameter list.
Waiting and failure: views that read the variables wait for them, with no time limit, and say when loading failed, with Try again. Those views are:
- the report's household section;
- the budgetary impact by programme chart and its CSV, whose download stays unavailable, with the reason as its tooltip and accessible description;
- calibration matches and the geography fit;
- changes that list variables, in the provision list and the report's changes.
Old catalog versions: a catalog version published before catalog format 2 fails with a clear message, rather than giving variables without input flags, enum labels or parts.

Field conversion and parity with v1's metadata

Each catalog variable becomes the record v1's GET /{country}/metadata gave, checked against records serialized from .py's public Variable records in the catalog's shape:

Enums: .py publishes every Enum variable with data type str and its values. A variable with enum values is v1's Enum, with possibleValues {value, label} and labels kept as declared.
Defaults: as v1 gave them: a value for numbers, yes-or-no values and enums (the default member's name), none for text or date variables.
Unit and definition period: read as v1 gave them, from catalog format 2. A variable without a unit formats as a plain number and is "Number" in the picker; a period that is neither a year nor a month gets no "per" suffix, as on v1.
Parts named by a parameter: a parameter-named adds or subtracts keeps the parameter's path, as on v1.
Other fields: label (the name when there is none), entity, documentation, input and hidden flags, and module.
Dated values: these and the parameters' values share one converter, which checks each value where it arrives.

Merged unused

Every new path is chosen when its module loads, from V3_API_ROUTES, which stays 'v1' until the switch. With it off:

the variables hooks report 'ready' at once;
the waiting wrapper shows its content;
nothing is requested;
every view renders as before.

Tests cover both settings.

Note for whoever makes the switch

The other segment's PR #1392 adds a v2-only standalone household report and a builder panel that read state.metadata.variables directly, without V3VariablesGate or useV3Variables. The background load fills the store, so they do get the variables. But until the variables arrive they will show empty fields, and nothing at all if the load fails. When the switch makes that page reachable, it will need the same wait.

Environment variables

None read or added. No workflow changed.

What this waits for before it can merge

v3: load metadata on demand from the Public API's v2 routes, unused until the switch #1444, the PR below it.
The Public API PRs it reads from, deployed: Serve v2 metadata in parts loaded on demand, cacheable per catalog version policyengine-api#4042, #4046 and #4049, and the stack below them. That stack waits for the .py release containing Publish the model metadata the Public API can only read through a private method policyengine.py#701.
Review.

Tests and evidence

GitHub Actions runs on v2plan/* branches are cancelled at the user's request, so the evidence is local, on Node 26.10.0. Every commit was written test-first: its new tests failed for the reason stated in the commit, then passed. The exceptions are the commits marked as refactors or test-only.

On the final commit (9b986d87):

app suite: 6,139 passed, 10 skipped (501 files);
website tests: 483 passed;
make typecheck and make lint clean;
bun run check (type check, ESLint, Prettier) and bun run stylelint clean;
bun run public-api:check: the generated types match openapi.json.

Review

Three review rounds:

Round 1: one major bug. Enum inputs showed a free-text box, because the catalog's str data type was copied as the value type. Also quality findings: the store filled only when a reader mounted; duplicated converters; typing; test placement; documentation.
Round 2: one minor bug. The programme CSV listed raw names while the variables loaded or after they failed. Also quality findings: report pages did not start the load; earlier loads stayed cached; duplicated query code; test organisation; a real parameter-named variable was untested.
Round 3: two minor bugs. The variables could be downloaded again after their cache entry expired, and the CSV's unavailable reason could not be reached. Also quality findings: status names; catalog versions before format 2; test sections and fixtures.

All are fixed in this branch.

🤖 Generated with Claude Code

vercel · 2026-10-27T13:58:40Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policyengine-calculator-next	Ready	Preview	Nov 4, 2026 9:13am UTC
policyengine-website	Ready	Preview	Nov 4, 2026 9:13am UTC

On the v2 routes the metadata store starts with no variables. After a v3 page renders, every variable of the catalog version now loads once from `GET /v2/model-catalog/variables`, in the shape v1's metadata gave each variable, so the household builder, its variable picker and the breakdown read them unchanged. The dated values of each parameter a variable's adds or subtracts names come with them and join the store as well. The response is cached per catalog version for the session, and a response for another country or version is dropped. `useV3Variables` says whether they are in, for views that wait for them. On the v1 routes (still the default), nothing changes and nothing is requested. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… from v2's part values Where a variable's adds or subtracts names a parameter, the household breakdown (`VariableArithmetic`) used the list that parameter holds in the report's year, read from the whole parameter list. On the v2 routes there is no whole parameter list, so it now reads the dated values the variables response carries for that parameter, and the whole parameter list on v1, as before. A parameter with no values lists nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

On the v2 routes the variables load in the background, so the views that read them now show a placeholder until they are in, with no time limit, and a failure says so and asks again: - the report's household section (the builder, its variable picker and the breakdown), behind `V3VariablesGate`; - the budgetary impact by programme chart, whose programme names are variable labels; - calibration matches and the geography fit, whose target names are; - a reform's changes that list variables, in the provision list and in the report's changes. The household builder's default fields and entities come from `/v2/model-catalog` through the metadata store, and a test pins that they, and the variable picker's entered, hidden and enum fields, read correctly from the v2 variables. On the v1 routes (still the default) every view shows as before and nothing is requested. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The Public API's catalog format 2 now stores each variable's unit and definition period, and `/v2/model-catalog/variables` serves both (null for catalog versions published earlier). `fetchCatalogVariables` now maps them into v1's variable shape (`unit`, `definitionPeriod`), so on the v2 routes the household breakdown formats currencies and rates, and the variable picker says "per year" or "per month", as on v1. A null unit reads as v1's variables without a unit do: a plain number, and "Number" in the picker. A null period gives no "per" suffix, as v1 shows for a period that is neither a year nor a month. The generated client on this branch predates the two fields, so a small, documented local extension of its variable type carries them until the client is regenerated from the final Public API branch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… failed load in every view that waits On the rebase onto 1.19's final branch: - The variables load is now keyed by the model catalog load (`v2LoadId`), as 1.19 keys parameters, so a new load (a retry, or a newer catalog) empties the store and asks for the variables again instead of refilling it from an earlier load's response; a response asked for under an earlier load is dropped. - The local extension of the generated variable type is gone: the client regenerated from the final Public API branch carries each variable's unit and definition period, and its other catalog format 2 fields are required and nullable. - Every view that waits for the variables on the v2 routes now also says when loading them failed, with Try again: the provision list and the report's changes (where a change lists variables), and calibration matches and the geography fit. The household section and the programme chart already did, through `V3VariablesGate`. `useV3Variables` reports this as `failed`, only ever on the v2 routes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The catalog stores each variable's Python type name, and .py publishes every Enum variable with data type "str" and its values. The conversion copied that into v1's `valueType`, so on the v2 routes every enum input (the default fields `state_name`, `region`, `brma`, `local_authority` among them) showed a free-text box instead of a list, and the picker called them "Number". A variable with enum values is now v1's "Enum". v1 also gave a default only for numbers, yes-or-no values and enums (the default value's name), none for a text or date variable; the conversion now does the same. Tests use records serialized from .py's public `Variable` records in the catalog's shape, and the small fixture's enum uses data type "str" as the catalog does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…g of another country The background load filled only the query cache, so the metadata store got the variables only once a view using `useV3Variables` mounted; a view that reads `state.metadata.variables` directly saw none. The background load now puts them in the store when they arrive, under the model catalog load they were asked for (a failure stays in the query, where waiting views report it). Both the background load and `useV3Variables` now also skip a store whose model catalog is for another country than the page's, which a client-side country change leaves until the new catalog loads, instead of starting a download that would be dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…takes Parameters (`catalogParameters`) and the parts of variables (`catalogVariables`) each had their own copy of the converter from v2's dated values to v1's, with its earliest-date constants; both now use `v1DatedValues` in `api/publicApi/datedValues`, which checks each value where it arrives with 1.19's `parameterValueFromApi`, so a malformed part value also fails there, naming its parameter. The variables request takes its country through `publicApiCountryId`, like its siblings, instead of a cast, and `catalogVariablesLoaded` takes `V1ShapedVariable` records. The existing tests cover both converters. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…und variables load - Tests import from `@test-utils`. The breakdown test's metadata fixtures move under the fixtures folder. The builder and list tests are split to mirror the modules they cover (`metadataUtils`, `VariableResolver`, `VariablePickerDialog`, `householdValues`, `ProvisionList`, `ReportChanges`), with the store seeding they share in a fixture. - The `v3ApiRoutes` and `modelCatalog` comments and the country ID architecture document describe the background variables load, and the comment on `MetadataState.v2LoadId` says a new load empties the variables too. No behaviour changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The UK "Budgetary impact by programme" card built its CSV outside the waiting chart, so on the v2 routes a CSV downloaded while the variables loaded, or after they failed, named programmes by raw variable name. The card's CSV button now shows disabled until the variables are in, with "Loading programme names" or "Could not load the programme names" as its tooltip (`ReportChartCard`'s new `csvUnavailableReason`); on v1, and once the variables are in, it downloads as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The background variables load ran only from `WorkspaceLayout`, which the report and bill report pages do not use, so on those pages the download started only when a view that reads the variables mounted. Both pages now start it when they render, like the workspace pages; on v1 it does nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…e the views one state to read - `useV3Variables` and the background load now build their query from one helper (`catalogVariablesQuery`), whose fetch also puts the variables in the store under the load they were asked for, so the key, the options, the country check and the store update live in one place. - The query's `gcTime` is now finite, and the background load removes an earlier model catalog load's query when a new load starts, so only the current load's variables stay cached for the session. - `V3VariablesStatus` is one `state` ('ready', 'waiting' or 'failed'), which the lists, the calibration sections, the programme CSV and `V3VariablesGate` all read; on v1 it is always 'ready', so the waiting wrapper is one component and shows its content at once there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- The real-shape fixture now also holds `above_the_line_deductions`, as the catalog publishes it, and the dated values of `gov.irs.ald.deductions` that name its parts, in the route's date format. A breakdown test on it checks that a 2017 report adds `domestic_production_ald` and a 2018 report does not (it passes, and fails when the breakdown ignores the part values). - The `catalogVariablesLoaded` reducer cases sit with `catalogParametersLoaded`'s in the metadata reducer's v2 tests. - The variables route path, the list provision and a fail-once-then-answer route are shared fixtures, and the country-change test waits for the query client to be idle instead of a fixed 50 ms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…he store lacks them The variables query kept its entry for five minutes, and the background load fetches without an observer, so on a page with no reader the entry went after five minutes and the next mount downloaded the whole response again, although the store already held that load's variables. The current load's entry is now kept for the session (an earlier load's entry is still removed when a new load starts), and neither the background load nor `useV3Variables` asks for the variables when the store already holds those of the current load. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The CSV button showed why it was unavailable only in a tooltip on a `disabled` button, which cannot take focus or open its tooltip, so the reason was never shown. While unavailable the button now stays focusable with `aria-disabled`, does nothing when clicked, opens its tooltip on hover or focus, and carries the reason ("Loading programme names" or "Could not load the programme names") as its accessible description. An available CSV button renders as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…bles status as its siblings do - `fetchCatalogVariables` now fails, naming the catalog version, when the response's `catalog_format` is not 2: such a version publishes no input flags, enum labels, modules or parameter-named parts, so v3's builder and breakdown would otherwise get variables with none of them and show no inputs, no dropdowns and wrong parts lists. Views report it as a failed load, and `V3VariablesGate` shows the message. - `V3VariablesStatus` uses the same field and values as `V3ParametersStatus` and `V3SearchCatalog`: `status`, one of 'ready', 'loading' and 'error'. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ixtures and request checks - `v1DatedValues` has its own tests: real dated values keyed by the day each starts, Python's earliest date written as v1 wrote it, no values, and a value that is not a parameter value failing with the parameter's name. - The breakdown and parts tests have Given, When and Then sections; their parts values, and the catalog test's unlabelled variable, are fixtures. - The report and bill report page tests check that the variables request is sent, not only that the background load hook ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…iables-on-v2 The base now holds main (with the missing-parameter checks and the household tab's changes) and the fixes that merge needed. The one conflict, in ReportView's household tab, keeps both: main's notice that households cannot be calculated when the report uses parameters the model no longer has, and its heading passed into the household section, with this branch's gate that waits for the model's variables. The section draws its own heading once it shows, so on the v2 routes the heading was missing while the variables loaded or after they failed. V3VariablesGate takes an optional heading, shown above its placeholder and above a failure, and the household tab passes it. Tests (both failed first): on the v2 routes, the gate shows a heading above the placeholder while the variables load, and above the failure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

main's household "What changes" walks a total's parts, reading a part list a parameter names from the parameter list alone. On the v2 routes that list holds only the parameters a report changes, so a total such as the US's refundable credits was named instead of its parts, while the Net income breakdown, which reads the catalog's part values, listed them. "What changes" now reads such lists with getParameterNamedParts, as the breakdown does. The v1 routes read the same parameter as before. Test (failed first): with the parts held as the v2 catalog's part values, the changed part is named rather than its total. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

main hides the UK budget by programme card for now, keeping its chart, CSV rows and tests so a one-line change shows it again. The v2 tests of that card (its names waiting for the model's variables, and its CSV) looked it up and failed once the card was hidden. The flag moves to libs/flagship/reportCards, still false, and the v2 test module shows the card through it, so those tests keep checking the card as it will show. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ReportView built the household heading in three places: the gate's, the section's and the one shown while the policies are not known. They now share one element. The v2 report test checks that the heading shows while the household builder waits for the variables (it failed with the gate's heading removed). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…rts in its mock The comment on the shared household heading now names the states that draw it; the state where households cannot be calculated draws none, as on main. The v2 card tests' mock of reportCards spreads the module, as the file's other mocks do, so a later setting there keeps its value. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Brings in 77f0980, which updates the useFlagshipReport test that failed on every run: the report now reads its policies at once for the missing-parameter check, so the test asserts that they are read but not parsed before the metadata loads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

PavelMakarchuk

Reviewed with origin/main merged in: no conflicts now (#1511 is already in the merge base, and householdProgrammeChanges.ts already uses getParameterNamedParts). tsc is clean, and 312 test files across the touched areas pass (3,982 tests, including all 36 of this PR's). No blocking issues.

The 'v1' claim holds. useV3Variables and useV3VariablesInBackground pick their branch at module load (useV3Variables.ts:146,188); on v1 the gate always reports ready, the background hook does nothing, and no request goes out. V3VariablesGate around HouseholdImpactSection (ReportView.tsx:573) only renders its children, and the key stays on the section. ProvisionList, ReportChanges and CalibrationMatches take their old branches, and no v2 household or legacy page imports the new code.

The v2 path avoids #1444's race patterns. Variables load in one request per catalog load, so there are no overlapping per-name requests, and catalogVariablesLoaded (metadataReducer.ts:155-163) drops stale responses, with readers deciding "ready" from v2VariablesLoadId === v2LoadId rather than query success. All variables and part_parameter_values arrive in one action, and every reader of the adds/subtracts tree sits under the gate, so the tree never sees a partial state.

Should-fix (after the switch):

One failure in the background load is not retried (useV3Variables.ts:173). fetchQuery does not retry by default, and the app's QueryClient sets no retry. Readers already mounted share that request, so one transient error shows "Could not load the model's variables" on the Household tab; only a reader that mounts later gets useQuery's retries. Adding retry: 3 to catalogVariablesQuery would cover both.
The full variable download repeats on each page move (useV3Variables.ts:166-171). Each catalog load gets a new loadId, and the earlier entry is removed, so with #1444's reload between v3 pages and the old builder, going v3 → builder → v3 downloads every variable again and the Household tab waits for it. Fixing #1444's reload fixes this too; keying by catalog version rather than load would also stop it.

Nits:

(Uncertain) When part_parameter_values lacks a path that an adds list names, householdValues.ts:159-162 falls back to metadata.parameters, which on v2 holds only the parameters loaded so far. The parts list is then empty with no signal: VariableArithmetic shows no children and householdProgrammeChanges lists the total. No wrong sum, but worth confirming the API always sends those paths.
(Parameters, not variables) HouseholdVariationKeeper (HouseholdBreakdowns.tsx:137-148) builds policy payloads from metadata.parameters, which on v2 are on demand, so value coercion can lack metadata for a provision. Belongs to the #1444/#1511 boundary rather than this PR; flagging for the switch.
Without quantity_type in the catalog, a stock money amount gets an "Annually" label; may match v1.

For the switch: as the body says, #1392's standalone household report reads state.metadata.variables without the gate and needs the same wait.

Happy to approve.

PavelMakarchuk

Approving: no change with the switch at 'v1', and the two should-fix items in my review apply only after the switch.

vercel Bot deployed to Preview – policyengine-calculator-next October 27, 2026 13:58

View deployment

vercel Bot deployed to Preview – policyengine-website October 27, 2026 13:58

View deployment

PavelMakarchuk mentioned this pull request Oct 27, 2026

v3 validation: one related score per measure and year, calibration headlines first #1442

Merged

anth-volk added this pull request to stack #1445 October 27, 2026 14:06

anth-volk and others added 17 commits October 30, 2026 20:28

anth-volk force-pushed the v2plan/1f-120-v3-variables-on-v2 branch from 9b986d8 to 1c274dd Compare October 30, 2026 17:22

vercel Bot deployed to Preview – policyengine-website October 30, 2026 17:23

View deployment

vercel Bot deployed to Preview – policyengine-calculator-next October 30, 2026 17:23

View deployment

PavelMakarchuk mentioned this pull request Nov 3, 2026

Hide the UK budget by programme card in the v3 report for now #1514

Merged

Anthony Volk and others added 3 commits November 4, 2026 02:35

Anthony Volk and others added 2 commits November 4, 2026 02:51

vercel Bot deployed to Preview – policyengine-website November 3, 2026 22:59

View deployment

vercel Bot deployed to Preview – policyengine-calculator-next November 3, 2026 22:59

View deployment

vercel Bot deployed to Preview – policyengine-website November 4, 2026 09:13

View deployment

vercel Bot deployed to Preview – policyengine-calculator-next November 4, 2026 09:13

View deployment

Base automatically changed from v2plan/1f-119-v3-parameters-on-demand to main November 4, 2026 09:24

anth-volk marked this pull request as ready for review November 4, 2026 09:24

anth-volk requested a review from PavelMakarchuk November 4, 2026 09:24

PavelMakarchuk reviewed Nov 4, 2026

View reviewed changes

PavelMakarchuk approved these changes Nov 4, 2026

View reviewed changes

anth-volk merged commit e71a113 into main Nov 4, 2026
11 checks passed

anth-volk deleted the v2plan/1f-120-v3-variables-on-v2 branch November 4, 2026 15:13

Navigation Menu

Repository navigation

v3 variables on the v2 routes: the household builder and breakdown from /v2/model-catalog/variables, unused until the switch - #1441

v3 variables on the v2 routes: the household builder and breakdown from /v2/model-catalog/variables, unused until the switch#1441
anth-volk merged 23 commits into
mainfrom
v2plan/1f-120-v3-variables-on-v2

anth-volk commented Oct 27, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Oct 27, 2026 •

edited

Loading

Uh oh!

PavelMakarchuk left a comment

Uh oh!

PavelMakarchuk left a comment

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

anth-volk commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes

Field conversion and parity with v1's metadata

Merged unused

Note for whoever makes the switch

Environment variables

What this waits for before it can merge

Tests and evidence

Review

Uh oh!

vercel Bot commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

PavelMakarchuk left a comment

Choose a reason for hiding this comment

Uh oh!

PavelMakarchuk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

anth-volk commented Oct 27, 2026 •

edited

Loading

vercel Bot commented Oct 27, 2026 •

edited

Loading