chore: examples/pypsa/ cuts pypsa.yaml into 24 topic fragments that merge back to its rows

[FBumann]

Prompt: "please explore how we can decompose the big pypsa yaml file into many small, topic related yaml files. And how we can test if everything works. I imagine that the merging of those yaml files must ultimatively result in the same spec. Start from #620. We should probably merge main into it. Then use the feature from 690. For testing, we might need 718"

Prompt: "I think we can be sure that we need to add parameters and expressions to given... Lets just do it as a stacked PR onto 690. Then retry. Iterate until the split is clean, reasonable and its easy to add new features via a fragment. Present your final design of given, merge, overrride and potential additions as you are satisfied with the api, ux and typesetting."

Prompt: "Present your design, changes etc in a draft PR,"

Prompt: "What do you think of this idea? Then the flag on the reader's given: entry is the design." (and the follow-ups that settled it)

Note

The following content was generated by AI.

examples/pypsa.yaml splits into 24 topic fragments that merge back to its rows. given: reads parameters and expressions (#728), and a reader marks a sum that fragments add terms to (#729). A new component is a new family of files.

This PR presents the design and holds the experiment. Do not merge it. The language changes are reviewed in #728 and #729; the split lands in a migration PR once the stack is on main (plan in #722).

The stack

PR	Base	What it does
#571, #690	main	fragments and patches: merge, override, given: for variables and constraints
#728	#690	given: also reads parameters and named expressions
#729	#728	a reader marks a given expression additive: true; contributors declare ordinary terms; merge sums them
#530, #718	main	the canonical form, which the equality check compares in
#620	main	the one-file PyPSA model
this PR	#620 + all of the above	the split, the splitter, the checks, two canonical-form fixes

The design

given: — what a file reads and does not declare

A fragment is a whole model: it loads, lowers and prints on its own. What it reads from a sibling it states under given:, one kind at a time.

Kind	States	This file reads it as	New in
parameters	dims, dtype	a parameter: in expressions, where masks and bounds	#728
variables	dims, domain	a column	#571
expressions	dims, additive	a quantity over at most the frame, of degree one; the definer owns the body	#728, #729
constraints	dims	a row family, named only in dual()	#571

# examples/pypsa/generator_ramping.yaml
given:
  parameters:
    Generator_committable: { dims: [generator], dtype: bool }
  variables:
    Generator_status: { dims: [scenario, snapshot, generator], domain: integer }
  expressions:
    Generator_p_nom_committed: { dims: [scenario, generator] }

A given expression's dims are an upper bound. A definition whose body carries a dimension the reader does not state is refused. A body over fewer dimensions is folded, and the composed model decides: it refuses a row that would repeat, and accepts one where another term carries the dimension.

A sum other files add to

The file that gives a sum its meaning marks it once, with its frame and description. Every contributor declares an ordinary named expression under the name.

# examples/pypsa/network.yaml
given:
  expressions:
    Bus_injection:
      dims: [scenario, snapshot, bus]
      additive: true
      description: what every component puts into a bus, less what it takes out of it; ...
constraints:
  Bus_nodal_balance: { dims: [scenario, snapshot, bus], expression: Bus_injection == 0 }

# examples/pypsa/load.yaml
expressions:
  Bus_injection: sum(Load_demand, by=Load_bus, over=load, into=bus)

Three kinds of file, one rule:

The file	Carries	Reads the name as
a reader	a given: entry; one reader marks it additive, others state the frame	the sum
a contributor	an ordinary expressions: entry, and the name nowhere else	its term, alone
both, carrying the marked entry: a hub that adds its own term, or a composed model	the marked entry and a term	the sum so far

A contributor that reads the name without the marked entry would mean two models, its term alone and the sum composed, and merge refuses it.

merge — fragments as peers

The entry	What merge does
a dimension or a relation	every fragment may declare it, and those that do agree; the first description is carried
any other declaration	one fragment declares it; a second is refused, both named
a given: entry	checked against the fragment that declares the name, then folded into it
a given expression	its definition carries no dimension the reader's dims do not name
a name read as one kind and declared as another	refused, both fragments named
a name a given: entry marks additive	every declaration is a term, summed in fragment-name order, each in parentheses; the marked entry is kept, so a later merge adds more
a term written as cases:, or over a dimension the sum does not state	refused, the fragment named
a contributor that reads the sum without the marked entry	refused, with one term or several
two declarations of a name no entry marks	refused as a collision; the message says a sum needs a marked entry
a marked name no file adds to	stays under given:; zero would leave its row without a variable
the objectives	summed in fragment-name order; the senses agree; the first description is carried
a given entry nothing declares	stays under given: for a host model

The result goes through to_spec, so the composed model is held to every rule again. In the composed program a marked entry that has a definition folds into it (ExpressionDeclaration.additive), so merged.program.given is empty for a whole model; the file keeps the entry.

override — patches over a base

Unchanged from #690. A patch edits, adds or removes (null) declarations field by field, and lays given: over one kind at a time. merge adds a term; override replaces one.

Typesetting

• Given lists the names a file reads: "data another file declares", "an expression another file defines", and "a sum other files add terms to" for a marked entry.
• Once a marked entry folds into a definition, Definitions lists it as "a sum other files add terms to". The line prints =: in any one model the sum is the terms that model has.

The split

Fragment	Lines	Holds
core	144	the scenario, snapshot and period weights, the risk preference, how a snapshot counts in a global constraint, and the marked entries of seven sums
network	39	marks Bus_injection; Bus_nodal_balance: Bus_injection == 0
power_flow	33	marks Cycle_angle_sum; Kirchhoff's voltage law
cost	52	the operating cost in expectation and at the tail (CVaR)
carrier	54	the growth limits
global_constraints	132	the primary-energy, operational and expansion limits
security	157	the branch-outage rows
generator, link, process	265, 304, 279	dispatch, capacity, bounds, modular builds
*_commitment	261 each	status, start-up and shut-down, up and down times, their costs
*_ramping	237 each	the ramp rows
*_maintenance	224 each	maintenance scheduling
storage_unit, store, line, transformer, load	455, 351, 247, 252, 50

4980 lines in total against 4033: the frames each fragment declares to load alone, and its given: entries. Nothing is copied. Nine names are sums, each marked once, and each carries its description there.

Where a sum is marked. In the reader where a model without it is never wanted, so leaving the reader out collides: Bus_injection in network, Cycle_angle_sum in power_flow, since lines without Kirchhoff's law are not a PyPSA model. In core where the reader may go: scenario_opex, Carrier_additions and the five global-constraint totals.

Adding a feature is a fragment.

• Leaving out any of carrier, cost, global_constraints, security, load, storage_unit, store, a generator, link or process family, or a *_ramping file alone still merges into a whole model.
• security needs lines or transformers. Line and transformer, with security and power_flow, leave together, as a transport model.
• Leaving out network, or power_flow with the branches kept, is refused as a collision that names the fix.

How it is tested

tests/test_pypsa_split.py, 49 cases:

1. Each fragment loads on its own (24).
2. Same model. merge(examples/pypsa/*.yaml) has the canonical form of pypsa.yaml with its hubs written as marked sums, objective description included, and merged.program.given is empty.
3. Same rows. That rewritten file states the rows of pypsa.yaml. Each new hub is substituted back, every term of a row is moved to one side, and the terms are compared as a multiset. A flipped sign, a dropped term and a changed operator each fail it.
4. The fragments are what tools/pypsa_split.py writes from pypsa.yaml.
5. 16 ways of leaving topics out each leave a whole model; leaving out network, or power_flow alone, is refused (2).
6. Every sum is marked in exactly one fragment, the one named above.
7. Every fragment and the composition print in LaTeX, Typst and Markdown (3).

Open points and possible additions

• Given frames in order. A given parameter or variable is compared as an ordered list (open since feat(language): merge and override load every model they compose and return one #690). A given expression is compared as a set.
• Italic or upright. A given expression prints italic, as a column does. The composition prints it upright where its body reads only data. A symbol table overrides either.
• A sum with no term. It stays under given:. A link-only model with global constraints therefore leaves global_constraints out, or a row gets a mask.
• A contributor alone does not say it is a term. Accepted: the file loads as it reads.
• canonical over several files. feat(cli): canonical --check fails a model file that is not in the canonical form, and --write rewrites it #718's --check takes one file; checking a composition in CI without pytest would need canonical to take fragments.
• Folders. generator/commitment.yaml would need merge to name a fragment by its relative path.
• Rejected:
  • copying declarations between fragments;
  • the flag on each term (the first form of feat(language): fragments add terms to a named expression that its reader marks as a sum, and merge sums them #729): a missing reader merged silently, a self-reading share was missed when alone, and each hub's description was copied into every term;
  • a += operator in the typesetter;
  • reading a sum no fragment adds to as zero: the reader's row would hold no variable.

The changes in each PR

#728 — given: reads parameters and expressions

• GivenParameterBlock (dims, dtype, description) and GivenExpressionBlock (dims, description); GivenTargets.parameters and .expressions.
• Resolution reads a given parameter as a parameter and a given expression as a Variable node; the dim rules, bounds, the flat namespace and the frame checks cover both.
• merge folds both, bounds a definition by the reader's dims, and refuses a kind mismatch. override lays them over through GIVEN_KINDS.
• The legend, typeset_declaration and advice cover both. Docs in declarations.md, compose.md, reading.md, errors.md, limits.md.
• test_given.py from 40 cases to 67; six guards, each with a test that fails without it.

#729 — a sum its reader marks

• GivenExpressionBlock.additive, GivenDeclaration.additive and ExpressionDeclaration.additive.
• At load, where a file carries the marked entry and a term: the term's frame and form are checked, and the entry folds into the definition.
• merge sums the terms of a marked name, keeps the entry, agrees readers on the frame, and refuses a cased term, a term over more than the sum and a self-reading contributor. The collision message names the fix. A composed objective keeps its first description.
• The legend notes; docs in declarations.md, compose.md, reading.md.
• test_additive.py, 24 cases; ten guards, each with a test that fails without it.

This PR

• examples/pypsa/: 24 fragments, generated.
• tools/pypsa_split.py: split writes them, and check runs checks 1 to 3.
• tests/test_pypsa_split.py: the checks above.
• Two fixes to the canonical form of feat(language): two files that mean the same model write one text #530, each with a regression test that failed before it: canonical_dict sorts the names under given:, and normalises a named expression written on one line, which it passed through as written. Both belong with feat(language): two files that mean the same model write one text #530; the first only shows once feat(language): a file says what it reads, and fragments and patches compose into one model #571 is also on main.
• Merges of main, feat(language): merge and override load every model they compose and return one #690, feat(cli): canonical --check fails a model file that is not in the canonical form, and --write rewrites it #718, feat(language): a fragment reads a parameter or a named expression another file declares, under given: #728 and feat(language): fragments add terms to a named expression that its reader marks as a sum, and merge sums them #729 into docs(pypsa): a single spec covers every pypsa model class and component #620, with the rename fixes they need (the stack still imports math_spec).

Hashes, how to reproduce, gates

Input	Branch	Commit
#620	pypsa-complete	60c279e559ceaef3a5a8ebdf10e098b7d0a79909
main	main	ca2e04b8cdd20c565edb84ca142a0c1dad21e9fd
#690 (carries #571)	feat/compose-through-to-spec	4ff3483b6e8f9087140b6a34c1f46273021e2657
#718 (carries #530)	claude/clever-ramanujan-vsgh97	1affd55225d05506cdc6eb76a8128e301b01d2dc
#728	feat/given-parameters-expressions	d41d3ee31a56922376aad60b71b2835a699fb5d9
#729	feat/additive-expressions	a98ad6ce81c06107a25de24843e383a6ecebbf45
this PR	claude/great-galileo-6k1248	c7b1c878d5c59c4a14518f31522d82d907b8aee9

main has moved on since ca2e04b, so GitHub shows this PR as conflicting. It is not meant to merge, and was not brought up to date.

git checkout c7b1c878d5c59c4a14518f31522d82d907b8aee9
python -m tools.pypsa_split split /tmp/pypsa
python -m tools.pypsa_split check /tmp/pypsa
python -m pytest tests/test_pypsa_split.py

Pixi cannot be installed here. In a Python 3.12 uv virtualenv:

this PR    pytest 2433 passed, 26 skipped; ruff, reuse, typos, prettier clean; tools.gallery --check current
#728       pytest 1787 passed, 7 skipped; ruff, pyrefly (0 errors), typos clean
#729       pytest 1812 passed, 7 skipped; ruff, pyrefly (0 errors), reuse, typos clean

Not run: pixi run ci, compile-tex (the tectonic bundle download is blocked), mkdocs build --strict (the proxy refuses the docs.python.org inventory), reference.py --check, zizmor, taplo, and pyrefly on this branch.

No changelog line: this PR is not meant to merge, and #728 and #729 sit on a base that predates the ## Upcoming version rule.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Sr4SCw5A7GUF5ARhdCaKR1

[read-the-docs-community]

Documentation build overview

📚 mathspec | 🛠️ Build #34761327 | 📁 Comparing c7b1c87 against latest (f74ec15)

  🔍 Preview build  

24 files changed · + 7 added · ± 13 modified · - 4 deleted

+ Added

• examples/library/index.html
• howto/compare/index.html
• howto/compose/index.html
• examples/library/composed/index.html
• examples/library/generator/index.html
• examples/library/load/index.html
• examples/library/surface/index.html

± Modified

• changelog/index.html
• examples/index.html
• about/limits/index.html
• examples/pypsa/index.html
• examples/pypsa_linearized_uc/index.html
• reference/api/index.html
• reference/glossary/index.html
• reference/language/index.html
• reference/program/index.html
• reference/reading/index.html
• and 3 more...

- Deleted

• examples/pypsa_losses/index.html
• examples/pypsa_multi_period/index.html
• examples/pypsa_quadratic/index.html
• examples/pypsa_stochastic/index.html

[FBumann]

Superseeded by #736