Skip to content

Pool main's default ASEC years in Route A (d2603) - #1559

Draft
MaxGhenis wants to merge 2 commits into
mainfrom
route-a-default-asec-pool
Draft

MaxGhenis wants to merge 2 commits into
mainfrom
route-a-default-asec-pool

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Max's d2603 (7 November 2026, "From the next build") moves Route A from income years 2022-2024 to main's default ASEC pool, income years 2023-2025 (ASEC 2024-2026, asec_sources.ASEC_DEFAULT_POOL_INCOME_YEARS). This PR moves the driver and its Modal base template, so the next certified US build becomes the first on the default pool. The November 2026 certified build keeps 2022-2024: it runs from its own worktree at 9434c504b with a frozen driver copy, and nothing here reaches it.

Refs #296. I'll close it with a comment once this merges.

Status: draft. Targeted local tests pass. The GPT-6.1 Sol review waits in Subfleet's queue behind its disk hold.

What changes

  • Driver. tools/route_a/route_a.sh passes census_cps_2025.h5, census_cps_2024.h5 and census_cps_2023.h5, newest first as before, with their pins, and the Census person archives asecpub24csv.zip to asecpub26csv.zip. input_rows lists the new files with the registries' digests and byte counts. Next to its H5 check, check_commit now also refuses a build commit whose education_assistance_source.py lacks a pooled archive pin.
  • Modal template. docs/us-modal-stage-route-a-rerun-base-plan.json becomes the next certified build's template. It names a new registration, us-puf-support-base-income-2023-2025, and lists every input in CAS form (volume://cas/sha256/<sha>/<name>) with the registries' digests. Its commit placeholder, PICK_MAIN_COMMIT_FOR_THE_NEXT_CERTIFIED_BUILD, makes validate refuse it until an operator fills it.
  • Registration. tools/modal_us_stage_plan.py adds AsecPool and builds the base command's input roles and argv from a pool. us-puf-support-base-income-2023-2025 runs the rerun registration's command, crosswalk pin and resources on the default pool, plus one seed-only input, asec_weeks_unemployed_zip. us-puf-support-base and us-puf-support-base-rerun keep every field but the argv builder, and their builders return the same argv as before, so plans made before this PR still validate on main.
  • Local reference. tools/route_a/make_local_reference.py reads the pool from the registration and hashes a seeded input where the local tool read it, in the home cache.
  • Docs. A new "ASEC pool" section in the driver README, the env example, a "Route A" section in the pins doc, "The default-pool template" in the runbook, dated notes on the 18 October rerun record and its prep report, and a changelog fragment.

Why the years stay written out

d2603's brief asked whether to derive the years or write them out. The driver and the template write them out, and CI holds them to the default pool:

  • Each year carries its own digest, byte count, file name and CAS upload, and the operator stages new files at a rollover. A derived year list would leave all of that to do by hand.
  • The driver hashes its inputs and checks the build commit's pins in bash (verify_inputs, check_commit) before it creates the build worktree and its venv, so it has no Python to ask.
  • When the pool moves is a ruling, as d2603 shows. A derived pool would move Route A silently the day someone pins a 2026 file.
  • test_route_a_asec_pool.py fails as soon as the driver's or the template's years leave ASEC_DEFAULT_POOL_INCOME_YEARS, so the PR that pins a newer year has to move Route A too, or say why not.

The LKWEEKS sidecar

The base reads the income-2022 Census archive, asecpub23csv.zip, for weeks unemployed in every mode; it isn't a pooled year. The driver passes no --asec-2023-weeks-unemployed-source, so the base calls fetch_asec_2023_weeks_unemployed_source(), which reads ~/.cache/microcosm/cps/asec_2023/, downloads from Census only when the file is missing, and checks the pin. The driver's prefetch stage warms that cache. The older registrations seed it from asec_education_2022_zip, which the default pool drops, so the new registration stages the same file as asec_weeks_unemployed_zip and seeds the cache from that. Neither path passes the flag, so local and Modal bases record the same run config.

ASEC order

The base pools the --asec-h5 years in the order given and records that order in the run context it locks, so the order belongs to the build. Route A keeps newest first (2025, 2024, 2023), as it passed 2024, 2023, 2022 before. The archives stay oldest first.

Invariants

Each holds for every input, and test_route_a_asec_pool.py executes it:

  1. The driver's and the template's pooled years equal ASEC_DEFAULT_POOL_INCOME_YEARS.
  2. Every pooled year sits in all four year-keyed registries: ASEC_SOURCE_ARTIFACTS, ASEC_EDUCATION_ASSISTANCE_ARCHIVES, ASEC_SPM_ROLE_SOURCES and ASEC_PUBLIC_ASSISTANCE_TYPE_AUDIT_PINS.
  3. Every pin in the driver and the template matches its registry: digest, byte count and file name, and the LKWEEKS archive's digest and name.

The same file also runs:

  • a differential test: the driver's base command, as bash expands it, tokenizes to the registration's planned command, flag for flag and value for value except file locations;
  • check_commit against a commit whose registries lack each pooled year's pins in turn, which must report exactly those two blockers;
  • a check that the new registration differs from the rerun registration only in its name, pool and seeded archive;
  • a Hypothesis property: the argv builder maps any pool year for year, in pool order, and at the 2022-2024 pool returns the historical builder's argv;
  • AsecPool refusing a repeated year, or archives that don't match the pooled years;
  • the pre-d2603 template, frozen byte for byte as a fixture, still validating on us-puf-support-base-rerun with its 2022-2024 years and pins.

#1044's "not affected" note

The PR that set the default pool, #1044, said Route A "reads explicit local paths and is not affected". That held for the driver of its day, which named 2022-2024, and d2603 ends it. The pins doc and the 18 October rerun record (its "ASEC 2025" row) now say so.

The student debt pilot (d2606)

The EqRC hub chose (a) on 9 November: the pilot's base keeps the November build's pool, income years 2022-2024 (ASEC 2023-2025), because d2606's terms come from the November build on that pool, and a new pool would reopen them inside a booking with no room for it. The pilot runs a copy of the driver from the last main commit before this merge, with the pre-merge template on us-puf-support-base-rerun. EqRC asked that this registration stay unchanged and that the pre-merge template keep validating on main. This PR keeps both, and the fixture test above holds them. EqRC also said the merge need not wait for the pilot.

Left alone

  • The November 2026 certified build, its worktrees and its kit.
  • us/spec/sources.yaml asec_raw_stage (51e9fafc…, which pools 2022-2024). The spec engine reads it; Route A's base passes no spec and writes its own raw-stage checkpoint. The Microcosm non-dynamics session pins the first default-pool base's checkpoint there once that base exists.
  • The driver's default CROSSWALK_SHA (c7cb040b…, the historical in-tree file). Each build's route_a.env sets the current pin, as the 18 October rerun's did.
  • Launchd jobs and builds: nothing ran, locally or on Modal.

Tests

  • Local, Python 3.14, serial: 571 passed at b377f7851 across test_route_a_asec_pool.py, test_route_a_driver.py, test_route_a_lineage_class_check.py, test_us_route_a_local_reference.py, test_us_modal_stage_puf_support_base.py, test_us_asec_sources.py and test_us_modal_stage_plan_tool.py. After the second commit, the three files it touches passed again (220), and the new fixture test passed by node id.
  • Mutation smoke: one changed digit in the driver's ASEC_EDU_2025_SHA and one in the template's asec_2023_h5 digest failed the driver-pin, template-pin, differential and check_commit tests, as they should.
  • A direct check with the module at 9b7a4544b and at this head: the pre-merge rerun template and the historical plan parse on both, with identical argv.
  • ruff check and ruff format --check pass on the changed Python files, and tools/ci_test_plan.py verify passes.
  • Independent review: GPT-6.1 Sol through Subfleet, pending.

🤖 Generated with Claude Code

MaxGhenis and others added 2 commits November 9, 2026 13:47
Max's d2603 (7 November 2026, "From the next build") moves Route A from
income years 2022-2024 to main's default pool, income years 2023-2025
(ASEC 2024-2026). route_a.sh now passes census_cps_2025/2024/2023.h5 and
asecpub24-26csv.zip with their registry pins, and check_commit refuses a
build commit whose registries lack a pooled pin. The Modal base template
names a new registration, us-puf-support-base-income-2023-2025, which
stages the 2023 ASEC archive as its own LKWEEKS cache seed; the historical
and rerun registrations keep their inputs, argv and seeds.
test_route_a_asec_pool.py holds the driver and the template to
ASEC_DEFAULT_POOL_INCOME_YEARS and the four year-keyed registries.

The November 2026 certified build keeps 2022-2024 and its own driver.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The student debt pilot's base (d2606) keeps the November build's pool on
us-puf-support-base-rerun, through the template as it stood at 9b7a454
(EqRC, 2026-11-09). A byte copy of that template now sits in the
modal_us_stage fixtures, and test_route_a_asec_pool.py checks that it still
validates on main with its 2022-2024 years, order, seed and pins.

The default-pool registration takes its stage from the rerun registration
it extends. The README describes what the education fetcher does, the pins
doc drops a build reference this PR can't verify, and the runbook states
the run-id convention without claiming the validator enforces it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant