refactor(config): hardcode the Cornerstone model schema, delete use_cornerstone_2026_model_schema - #561
Open
briantobin-99 wants to merge 1 commit into
Conversation
…ornerstone_2026_model_schema The Cornerstone 2026 model schema is the only supported schema; every config set the flag True. All code paths now assume the flag's True side unconditionally: - eeio/derived.py: legacy wrappers collapse to their cornerstone implementations; derive_B_usa_via_vnorm and scale_abq_via_summary.py (orphaned legacy-B helpers) are deleted, along with the unused inflate_A_matrix/inflate_B_matrix/inflate_q_or_y helpers in inflation_helpers_ceda.py. - allocation/derived.py: map_fbs_sectors_to_model_schema always maps into Cornerstone_2025; the E collapse reindexes directly (the BEA->CEDA groupby collapse for the legacy schema is gone). - The allocator chain (MECS mapping selectors, get_allocation_sectors, load_bea_use_table) is pinned to the Cornerstone frame — the side that generated the pinned *_CEDA allocation FBA parquets on GCS, which carry Cornerstone-frame activity codes (33131B, 335220, disaggregated 562*). The use-table alignment helpers for CEDA-vocabulary sectors are kept verbatim. Rebuilding all three allocation FBAs (foam, refrigeration, MECS energy) reproduces the pinned parquets bit-identically. - EF diagnostics always align old (CEDA v7) vs new (Cornerstone) vectors and always emit the sector_mapping_notes tab. - publish loses require_cornerstone_config; nowcast loses its _force_cornerstone_model_schema shim. - The field is removed from USAConfig and stripped from all config YAMLs (behavior-neutral: every YAML set it True). Verified: black/ruff/mypy clean, unit suite green, the v0.3 snapshot integration tests (test_usa.py, 9 tests) pass against pinned snapshots c60bdf4, and the three allocation FBA rebuilds match their pinned parquets exactly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
briantobin-99
marked this pull request as ready for review
August 1, 2026 00:12
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cc:
Closes:
What changed? Why?
use_cornerstone_2026_model_schemais deleted. Every config set it True — the Cornerstone 2026 model schema is the only supported schema — so all code paths now assume the flag's True side unconditionally. 40 files, net −719 lines.Model pipeline.
transform/eeio/derived.py: the legacy wrappers collapse to their cornerstone implementations;derive_B_usa_via_vnormandscale_abq_via_summary.py(orphaned legacy-B helpers) are deleted, along with the unusedinflate_A_matrix/inflate_B_matrix/inflate_q_or_yhelpers ininflation_helpers_ceda.py.transform/allocation/derived.py:map_fbs_sectors_to_model_schemaalways maps intoCornerstone_2025(the electricity-disaggregation retargeting hook is unchanged); the E collapse reindexes to the Cornerstone columns directly, and the legacy BEA→CEDA groupby collapse is gone.Allocator chain. The MECS mapping selectors,
get_allocation_sectors(), andload_bea_use_table()are pinned to the Cornerstone frame — the side that generated the pinned*_CEDAallocation FBA parquets on GCS, which carry Cornerstone-frame activity codes (33131B,335220, disaggregated562*). The use-table alignment helpers for CEDA-vocabulary sectors are unchanged.Diagnostics. EF diagnostics always align old (CEDA v7) vs new (Cornerstone) EF vectors, always emit
comparison_typeand thesector_mapping_notestab.Guards and shims.
require_cornerstone_config(publish) and the_force_cornerstone_model_schemacontext manager (nowcast) are deleted with their call sites — they only asserted or forced the flag.Config. The field is removed from
USAConfigand the key stripped from all config YAMLs. Behavior-neutral: every YAML set it True, andUSAConfig's strict validation requires removing the key in the same change.Testing
test_usa.py, 9 tests) pass against the pinned snapshots.BEA_CEDA_foam_allocation_2023,BEA_CEDA_refrigeration_allocation_2023, andEIA_MECS_Energy_Allocation_CEDA_2018(2,835 rows; rebuilt underusa_ghg_data_year: 2023, the vintage the pin was generated with — the EPA allocation tables used by the MECS builders do not yet support 2024).Snapshot integration run for this branch: https://github.com/cornerstone-data/bedrock/actions/runs/30674351415