feat(language): a spec is composed from files that each state part of it, and patched with files that each change part of it

[FBumann]

Prompt: "I think we should do 2 PRs. One doing the cononical stuff. And another doing all the composition stuff as one, coherent design. And the PR description should explain what it adds, changes, and how the composition works. Also with a tutorial"

Then: "You build it / Fold it into 732 / Close 734", and "Adjust the PR description of 732 after 738 merges."

Note

The following content was generated by AI.

A spec can now be written as several files. merge composes files that each state part of a spec. override lays patches over a spec. given: says what a file reads from another file, and term: says what it adds to a name another file has.

Stacked on #731 (the canonical form). This squashes #571, #690, #728, #729, #734, #738 and #739 into one design onto main after #721, in #721's words: a file states a spec, and a model is a spec with data.

How composition works

A fragment is a whole spec. Each file loads, prints and gets advice on its own. What it reads and does not declare, it states under given:, with the frame:

# emissions.yaml reads the fleet's dispatch, which generators.yaml declares
given:
  variables:
    dispatch: { dims: [snapshot, generator] }
constraints:
  emission_limit: { dims: [], expression: sum(dispatch * emission_rate) <= emission_cap }

merge composes fragments as peers, and returns one loaded Spec:

spec = ms.merge({'network': 'network.yaml', 'generators': 'generators.yaml', 'emissions': 'emissions.yaml'})

• A dimension or relation may appear in several fragments, and they must say the same thing.
• Every other declaration has one owner; a second is refused, both fragments named.
• A given: entry is checked against the fragment that declares the name, then folded into it. A name nothing declares stays under given: for a host model to provide.
• The objectives are summed.
• Prose is not a claim. Where fragments word one thing differently, the first wording in fragment-name order is carried, and a reader's description fills a declaration its owner left undescribed. The order the fragments are passed in reaches no field.
• The result goes through to_spec, so the composed spec is held to every rule again.

A term a file adds. A balance should not list every component. A component says what it puts in as an ordinary named expression, and names it as the term: of the entry it adds to. The file that reads the sum states only its frame:

# network.yaml reads the sum
given:
  expressions:
    injection: { dims: [snapshot, bus], description: what the components put into a bus }
constraints:
  balance: { dims: [snapshot, bus], expression: injection == 0 }

# generators.yaml adds a term
expressions:
  generation: { expression: sum(dispatch, by=gen_bus, over=generator, into=bus), description: what the fleet puts in }
given:
  expressions:
    injection: { dims: [snapshot, bus], term: generation }

merge defines injection as the definition one fragment writes, if any, plus every term by name, in fragment-name order: generation + consumption. Each term stays a named expression of the composed spec, with its description, so a host can read it and a patch can change it by name. A later merge adds to the composed definition the same way. A new component is a new file, and the network does not change.

The contributor decides alone. A file that defines the name does not opt in: it reads its own definition alone, and the definition plus every term once composed. Whoever composes the files answers for that sum. An opt-in or opt-out flag on the definition can be added later without breaking a file.

override lays patches over a base, the spec a framework ships, and returns the patched Spec:

spec = ms.override('base.yaml', {'carbon': 'carbon.yaml'})

A patch names only what it changes, field by field. A whole new entry is added. null removes a declaration or drops a field to its default. Sibling patches must write disjoint fields, so their order never matters. Layering is written as nesting, and the two verbs compose as override(merge({...}), {...}): build the spec, then configure the run.

What this adds

• given:, a new top-level section with four kinds:

  Kind	States	The file reads it as
  parameters	dims, dtype	a parameter: in expressions, where masks and bounds
  variables	dims, domain	a column
  expressions	dims, term	a quantity over at most that frame, of degree one; the definer owns the body
  constraints	dims	a row family, named only in dual()

• mathspec.merge and mathspec.override, in mathspec.composition. Both take paths, YAML text, mappings or loaded specs, and return a loaded Spec.

• term: on a given: expressions: entry: the name of an expression the same file declares. At load, the term must exist, carry no dimension the entry does not state, not read the name it adds to (directly or through another name), and stay within degree two. It counts as read by the math.

• Program.given (GivenTargets): what a program reads and does not build, by kind. GivenDeclaration.term is the Named node of the term. ExpressionDeclaration is unchanged, and program.given still means what a host must provide.

• Advice of kind given: one note per name a file reads and does not build. A term gets its own note, saying merge sums it.

• Typesetting: the legend gains Given. It says "data another file declares", "an expression another file defines", or "an expression this file adds generation to". A term prints under Definitions as the definition it is. A given name prints no line of its own, so typeset_declaration refuses it.

• Docs:

  • the tutorial A spec in several files (docs/several-files.md), with tests/test_several_files_page.py, which runs every step and compares what it prints with the page;
  • the how-to Compose a spec from several files (docs/howto/compose.md);
  • the given section and "A term a file adds" in the language reference;
  • "What a program does not build" in reading.md;
  • a glossary entry for provide;
  • the component library example (examples/library/, five generated pages).

What this changes

• A spec file has twelve declaration keys, not eleven, and the schema gains given.
• A name declared both in a section and under the matching given: kind is refused.
• A bound may name a given parameter.
• The canonical form (feat(language): two files that state the same spec write one text, and canonical --check fails a file that is not in it #731) places given after variables and sorts the names under each kind.
• The advice note says "declared elsewhere" rather than "built elsewhere", since a parameter is not built.
• Internal: the test fixture override is now varied, since override is the public verb; the bus balance the composition tests start from lives in tests/fixtures.py. The internal names this PR adds carry no #: comment, since none reaches the API reference (An attribute's #: comment does not reach the API reference #752).

Where each part came from, the rules in full, guards, gates, what was not done

Origin. #571 (claude/loving-thompson-vurdqx): given: for variables and constraints, merge, override, the library example. #690 (feat/compose-through-to-spec): both verbs load through to_spec and return a Spec, null makes what it names absent, and assumptions are kept. #728: given parameters and expressions. #729: a sum the reader marks additive, which #734 replaced. #734: the contributor's term:, which drops additive, the kept entry, defined_sums and the namespace exemption; then 1975438 makes the term a name and adds it by name. #739: descriptions follow fragment-name order, and a reader's fills an undescribed owner. #738: the ⋯ + term line and the ellipsis operator go, one agreement rule (_agreed) and one fold rule (_fits) serve every given kind, and the balance fixture is shared.

Why a name, not an inline expression. A named term takes cases:, a description and every rule of a named expression. It stays in the composed spec, so a host reads it and a patch changes it. The composed sum prints generation + consumption, each term defined once with its description. An inline term would join the sum as anonymous text. A fold: flag, which would substitute a term into the sum, was left out until a caller needs it.

The rules of merge in full.

The entry	What merge does
a fragment that does not load alone	refused under its own name, with to_spec's refusal
two fragments on different language versions	refused
a dimension or a relation	shared; the declarations agree; the first description in fragment-name order is carried
any other declaration	owned by one fragment; a second is refused, both named
a reading two fragments share, of any kind	agree on the frame, as a set, and on every other field; the first description in fragment-name order is carried
a given: parameter, variable or constraint	states the introducer's frame and says the same as it or less, then folds into it
a given expression	the composed body carries no dimension the reader's dims do not name; a narrower body is folded, and the composed load refuses a row it would repeat
a reading of a name its owner left undescribed	the reader's description fills it; the owner's own wins where it exists
a name read as one kind and declared as another	refused, both named
a given expression with a term	defined as the definition, if one fragment writes it, plus every term by name in fragment-name order; the terms stay named expressions
terms on a name no fragment defines, reads or uses	refused, with a near miss named if there is one
a definition written as cases: that a term adds to	refused: name the cased body, and define the name as that name
two definitions of one name	a collision; the message names term:, and names the term where one of them is one
the objectives	summed in fragment-name order; the senses agree; the first description in fragment-name order is carried
a given entry nothing declares	stays under given: for a host model

The rules of override. A partial entry edits a declaration the base has; a miss is refused with the near miss named. A whole entry creates one. null removes a declaration, and refuses a removal of what the base lacks. null on a field drops it to its default, so term: null makes an entry a plain reading. A dimension or relation may be added or restated word for word, never changed or removed. A section set to null is refused. Two patches that write one field are refused. given: is laid over one kind at a time.

Tests. test_given.py, test_composition.py, test_terms.py, test_library_example.py and test_several_files_page.py. test_terms.py replaces test_additive.py: the refusals of a marked hub and of a contributor reading its own term have no counterpart, since neither case exists now. test_the_file_that_defines_the_name_reads_the_extended_sum_once_composed pins the one thing the contributor-only design accepts. #739's tests were written first and failed on the head before it; #738 moved one guard, test_a_given_expression_prints_no_line_of_its_own, which covers a reader and a contributor.

Guards of the term, each deleted in turn:

Guard	Fails without it
a term names an expression of its file	a-term-written-inline, a-mistyped-term
a term counts as read by the math	test_a_term_is_read_by_the_math

A term that reads the sum through its name is refused by the existing walk, which steps into a Named body; a-term-reading-the-sum-through-its-name holds it.

Gates. Pixi cannot be installed in the containers this was built in. In a Python 3.12 virtualenv with the pins of pixi.toml, on the tree of 1339df3 (#738 merged):

pytest tests -n 8                         2071 passed, 27 skipped (the skips are the docs-feature tests, absent in that environment)
ruff check . / ruff format --check .      clean
pyrefly check                             0 errors (9 suppressed)
prettier --check docs CHANGELOG.md        clean
python -m tools.schema / tools.notation / tests.typesetting.golden   regenerated, committed
python -m tools.gallery --check           13 page(s) current

Earlier heads also ran typos, reuse lint and python -m zensical build --strict clean, the last apart from the docs.python.org inventory the proxy refuses. Not run: compile-tex (the tectonic bundle download is blocked), zizmor, taplo. CI runs all of them.

Not done.

• An opt-in or opt-out flag for the file that defines a name, and a fold: flag for a term. Either is an optional field later.
• merge names a fragment by its key, so a folder layout such as generator/commitment.yaml would need the relative path as the key.
• The order of the declarations in the composed spec follows the order the fragments are passed in. That is presentation: the spec compares equal either way, and the canonical form sorts it.
• One component library in the docs. The how-to shows the port pattern and the tutorial shows terms, two ways to write a balance; a sentence in each naming the other is the open suggestion.
• The PyPSA split that exercises this design is docs(pypsa): the pypsa spec is also 24 topic files that merge back to it, each component adding its share of a sum by name #736. It needs docs(pypsa): a single spec covers every pypsa model class and component #620 and lands after this PR.
• feat(language): a file says what it reads, and fragments and patches compose into one model #571, feat(language): merge and override load every model they compose and return one #690, feat(language): a fragment reads a parameter or a named expression another file declares, under given: #728, feat(language): fragments add terms to a named expression that its reader marks as a sum, and merge sums them #729 and feat(language): a file adds a term to an expression another file defines, under given: #734 are closed as superseded by this PR; refactor(language): a term prints as the definition it is, and one rule folds every reading #738 and fix(language): a merged spec's descriptions do not depend on the order the fragments are passed in, and a reader's fills one its owner left out #739 are merged into it.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Sr4SCw5A7GUF5ARhdCaKR1

[read-the-docs-community]

Documentation build overview

📚 mathspec | 🛠️ Build #34798298 | 📁 Comparing fc464a7 against latest (0efbf59)

  🔍 Preview build  

20 files changed · + 8 added · ± 12 modified

+ Added

• several-files/index.html
• examples/library/index.html
• howto/compare/index.html
• howto/compose/index.html
• examples/library/composed/index.html
• examples/library/generator/index.html
• examples/library/load/index.html
• examples/library/surface/index.html

± Modified

• changelog/index.html
• examples/index.html
• about/limits/index.html
• reference/api/index.html
• reference/glossary/index.html
• reference/language/index.html
• reference/program/index.html
• reference/reading/index.html
• reference/language/declarations/index.html
• reference/language/errors/index.html
• and 2 more...

[FabianHofmann]

trimmed the inline comments a bit which lead me to raise an issue on this. currenly we are risking to bloat to code with explainatory comments which neither help LLMs nor humans in my opinion but rather making thing clunky.