feat(language): a term names the sum it adds to with adds_to:, and no file marks a sum as open

[FBumann]

Prompt: Given defines what other files need to declare / Adds_to names a given expression, but is a write, not a read! … Add_to needs to reference an expression that's defined as a givenexpression block in that file. … Can we catch a misspelling of the given block? … Would it be better or safer if we only allow adding to given expressions? — Yes! … 763 needs to showcase and explain the new syntax in its pr description. And explain how edge cases are handled, whats allowed and what isnt … Then we do the rest on the PR directl as soon as 5 merged and 763 is updated

Prompt: Could we loosen the constraint of not allowing incremental merges in #763 ? So an expressions adds_to: can reference an expression thats a regular expression in another yaml? … I think i want it in merge. Then any expression is open always.

Note

The following content was generated by AI.

What this changes

A named expression adds itself to a sum with adds_to:. The target must be a given: expressions: entry of the same file. The given entry is the read, and adds_to: is the write. merge writes the terms after the body another file defines, or as the whole body where no file defines one. A merged spec takes more terms in a later merge. empty: true and given: … term: are removed.

The syntax

A file that uses the sum reads it under given:. The network reads injection because its balance uses it:

# network.yaml
dimensions:
  snapshot: { dtype: int }
  bus: { dtype: str }
given:
  expressions:
    injection:
      dims: [snapshot, bus]
      description: what the components put into a bus
constraints:
  balance:
    dims: [snapshot, bus]
    expression: injection == 0

A file that contributes also reads the sum. It writes its share as an ordinary named expression and names the sum with adds_to::

# fleet.yaml
dimensions:
  snapshot: { dtype: int }
  bus: { dtype: str }
  generator: { dtype: str }
relations:
  gen_bus: { key: generator, values: bus }
variables:
  gen_p: { dims: [snapshot, generator], bounds: { lower: 0 } }
given:
  expressions:
    injection: { dims: [snapshot, bus] }     # the read
expressions:
  generation:
    expression: sum(gen_p, by=gen_bus, over=generator, into=bus)
    adds_to: injection                         # the write

# load.yaml
dimensions:
  snapshot: { dtype: int }
  bus: { dtype: str }
parameters:
  demand: { dims: [snapshot, bus] }
given:
  expressions:
    injection: { dims: [snapshot, bus] }
expressions:
  consumption:
    expression: -demand
    adds_to: injection

Alone, each file loads and prints. A contributor reads injection as a column over its frame. check notes the term:

expression 'injection' is read here, and this file adds a term to it: merge() sums the term with what the other files write under the name. Until then, the program reads it and does not build it.

The legend names the term:

Symbol	Meaning
$\mathit{injection}$	injection over $\mathcal{T} \times \mathcal{B}$, an expression this file adds consumption to

Merged, merge(['network.yaml', 'fleet.yaml', 'load.yaml']) defines injection as generation + consumption, over the readers' frame. Each term stays a named expression, and its adds_to: is dropped. Nothing is left under given:, so the result is fully defined:

$$\mathit{injection}_{t,b} = \mathit{generation}_{t,b} + \mathrm{consumption}_{t,b} \qquad \forall\, t \in \mathcal{T},\ b \in \mathcal{B}$$

A body another file defines takes terms too. If network.yaml defines injection: {expression: slack} instead of reading it, the merge gives slack + generation + consumption. The body keeps its dims: and its description.

Incrementally, merge([merge(['network.yaml', 'fleet.yaml']), 'load.yaml']) gives generation + consumption. The terms join with a plain +, so the written body is the one-list merge's, and so is the canonical form. A merge defines every sum it has terms for, so the step that first merges the terms of a sum also merges a file that reads it. merge([merge(['fleet.yaml', 'load.yaml']), 'network.yaml']) is refused, as "Terms nothing else reads" below.

Before and after

On main	With this PR
The file that owns the sum declares it expressions: {injection: {dims: […], empty: true}}	Every file that uses the sum reads it: given: {expressions: {injection: {dims: […]}}}, or one file defines it with a body
A contributor names its term on the sum's entry: given: {expressions: {injection: {dims: […], term: generation}}}	The term names the sum: expressions: {generation: {expression: …, adds_to: injection}}
A term on a definition gives (body) + term, the body in brackets	It gives body + term, so a sum merged in steps is written as one merged in one list

Allowed

Each result is the output of the fixtures above on this branch's head.

Case	Result
Terms from several files	injection = generation + consumption
List order	merge([load, fleet, network]) gives consumption + generation. The canonical form of both orders is the same text
A body one file defines, e.g. injection: slack	slack + generation + consumption, with the definer's dims: and description
A merge onto a merged spec	the same written body and canonical text as one merge of every file. Four splits are tested: the last file later, the reader alone first, three steps, and a body a file defines
The reader adds its own part as a term, e.g. shed: {expression: slack, adds_to: injection} in network.yaml	shed + generation + consumption. The balance uses the sum, so the network counts as a reader
A contributor that also uses the sum in its own constraint (injection <= 10), beside another file that reads it	merges. Its math reads the sum, so it counts as a reader
Two terms from one file (consumption and loss)	generation + consumption + loss
A term that reads another sum its file adds to, where that sum's terms do not read back	loads
A term over fewer dimensions (demand over [bus] only)	merges. The term is constant along snapshot, and the sum keeps the readers' frame
A term written as cases: with otherwise:	added by its name like any other, onto a read sum or a defined body
A quadratic term	loads. It is held to degree two, as what reads the sum is
A reading no term fills	stays under given: for a host model to provide, as any given expression does
A one-line patch on a term: override(load, [{'expressions': {'consumption': '-2 * demand'}}])	{'expression': '-2 * demand', 'adds_to': 'injection'}. The one-line form replaces the body, expression: or cases:, and keeps the other fields

Refused when the one file loads

Each file is checked alone, so a mistake is named in the file that made it.

• adds_to: names something the file does not read under given: expressions:. This covers a typo:

  Named expression 'consumption': it adds to 'injecton', which this file does not read under 'given: expressions:'. A term writes into a name this file reads: declare the name there over its frame, or fix the spelling. Did you mean 'injection'?
  

• adds_to: names an expression the same file defines. A file writes its own body in one place:

  Named expression 'consumption': it adds to 'total', which this file defines. A file writes its own body in one place, so a term fills only a name read under 'given: expressions:': write the term into the body of 'total', or read 'total' there and add its body as a term of its own.
  

• The term reads the sum it adds to, directly or through another sum the same file adds to (fix(language): a file whose terms read each other's sums is refused at load #780). Each term in the loop is named:

  Named expression 'consumption': it reads 'injection', the sum it adds to, so the sum would define itself. A term is what this file puts in: write it in what this file declares.
  Named expression 'demand_injection': it reads 'injection', the sum it adds to, through 'withdrawal', so the sum would define itself. A term is what this file puts in: write it in what this file declares.
  

• The term carries a dimension the reading does not state:

  Named expression 'consumption': it adds to 'injection' over ['snapshot'], which the given entry's dims ['bus'] do not name. A term is read over the frame the given entry states: add ['snapshot'] to those dims, or leave them out of the term.
  

• The term is above degree two:

  Named expression 'generation': this product is degree 3. The language takes degree 2 and nothing above it.
  Multiply by a parameter instead, or give the inner product a name — a variable constrained to equal it is degree 1 wherever it is used.
  

• One file reads and defines the same name. This is the existing collision:

  Given expression 'injection' collides with the named expression of the same name. Names share one flat namespace — rename one of them.
  

• A frame with no body, what empty: true used to mean. The message names the new form:

  expressions.injection: a named expression is one `expression:` or a set of `cases:`, and this has neither. Cases are for a quantity whose value varies by region; one expression is everything else. A sum other files add every term to is read under 'given: expressions:', and each file names it with `adds_to:` on its term.
  

Refused at merge

• A term on a definition written as cases:. A term follows one body:

  fragment '#1' defines 'injection' as `cases:`, and fragment '#2' adds a term to it. A term follows one body, and a set of cases is no one body: name the cased body as its own expression, and define 'injection' as that name.
  

• A term that reads its own sum through another fragment. Each file loads alone. The composed load named the loop, x -> injection -> t -> x, but no fragment:

  fragment '#2' adds 't' to 'injection', and 't' reads 'injection' back through 'x' of '#1', so the sum would define itself. A term may not read what reads its sum: write 't' from something else, or define 'x' without 'injection'.
  

  Through another sum, the path names it: … reads 'injection' back through the sum 'withdrawal', 'inflow' of '#3', ….
• Two fragments that both define the sum. This is the existing collision, and its hint now says one may define it:

  fragments '#1' and '#2' both declare the expression 'injection'. … A sum several fragments add to is defined by one of them at most: each other reads it under 'given: expressions:' and adds its part with `adds_to:`.
  

• A term on a name a fragment declares as a variable, a parameter or a constraint:

  fragment '#1' declares 'injection' as a variable, and fragment '#2' adds a term to it. A term adds to a named expression: give the sum a name of its own, or read the variable under 'given: variables:' and add no term to it.
  

• A reader that states less than the definer's frame. This is the existing fold rule, now tested against a defined sum.
• Where no fragment defines the sum:
  • Terms nothing else reads. Some fragment must read the sum for more than adding to it: read it and add nothing, or use it in its math. A reported expression builds no row, so reading the sum there does not count. A name one fragment alone reads is refused too, even where that fragment uses it in its math. This also refuses a merge step that has the terms of a sum and no file that reads it.

    fragments '#1' and '#2' add a term to 'injection', and no other fragment reads it: none reads it without adding to it, or uses it in its math. A term writes into a sum the rest of the spec reads: add the fragment that reads it, or fix the spelling under 'given:'.
    

    With a misspelled given: entry, the suggested fix is drawn from the names the fragments read, never from term names: … Did you mean 'injection'?
  • Readers that state different frames:

    fragments '#1' and '#2' say different things about the given expression 'injection': {'dims': ['bus']} against {'dims': ['snapshot', 'bus']}. A declaration two fragments share is one both say the same thing about: make the two identical, or read it over one frame.
    

  • Readers that write one frame in different orders. No fragment defines the sum, so no fragment's order wins:

    fragments '#1' and '#2' read the sum 'injection' over ['bus', 'snapshot'] and ['snapshot', 'bus']. No fragment defines the sum, so its frame is the order its readers write: write the dims in one order in every file.
    

• Two fragments whose terms share a name. Terms share one namespace, so name each after its component:

  fragments '#2' and '#3' both declare the expression 'generation', which a fragment adds to 'injection' as a term. A term shares one namespace with every named expression: name each fragment's term apart, such as after its component.
  

Why

The old form had two problems:

• The link sat on the wrong side. A contributor named its term on the sum's given: entry. So given:, which reads, also carried a write.
• The owner had to opt in. The file that owned the sum had to declare it empty: true.

Now given: only reads, adds_to: writes, and no file marks a sum as open: every named expression with one body takes terms from other files.

Method, gate output, what goes, guards

This branch is on main after #761, #762, #766 and #780, with later main merged in. #766's fix, that a term keeps its definition line when expressions are inlined, is written against adds_to:: Walk.defined() keeps an entry whose adds_to is set. #780's loop check is written against adds_to: in the merge commit: _terms groups a file's terms by the sum they add to, and _loop follows each term's reads through those groups.

What goes.

• ExpressionBlock.empty, _check_empty and empty_sums.
• GivenExpressionBlock.term, and GivenDeclaration.term and .empty.
• The symbol = ⋯ definition line and Format.ellipsis.
• The empty branches in lowering, resolution, dimensions, the legend and walk.
• In composition.py: _readings, _landed and _summand. _read_elsewhere, _uses, _undeclared, _frame, _acyclic and _path replace them.

Terms on a defined body (5e51689). An earlier head refused a term on any name a fragment defined, so a merged spec took no further term. On request, _summed now starts from the definer's entry and appends the terms. _undeclared refuses only a cases: definition and the other kinds. _read_elsewhere and _frame apply only where no fragment defines the sum. _acyclic follows each fragment's reads, and each sum's terms, from every term back to its sum. Coverage moved: test_a_term_on_a_name_a_fragment_defines_is_refused is gone. Its file-definition and merged-sum cases are now test_a_term_follows_the_body_a_fragment_defines, and its cased case is test_a_term_on_a_cased_definition_is_refused.

Merges in steps (09cb6f2). test_a_sum_merged_in_steps_is_the_sum_merged_in_one takes four splits and asserts the written body of both sides, then their canonical text. The written body is the assertion that pins the plain +: the canonical form reads (a + b) + c as a + b + c, so it matches with or without brackets. test_a_step_where_no_other_file_reads_the_sum_is_refused pins the refusal of a step with no reader.

PyPSA. tools/pypsa_split.py writes adds_to: on each of the 40 terms. The home fragments (network, power_flow, settings) read their sums under given:, with the description. All 24 fragments load alone and merge to the one file's canonical form. The gallery pages are regenerated.

First review. Six findings. Five are fixed here; the sixth was #766.

• _uses walks only in-math expressions, so reporting a misspelled sum does not pass the misspelling check.
• adds_to: is stripped from every composed expression.
• merge()'s Raises: names both refusals.
• tools/pypsa_split.py keeps the body of a term written as Name: >-.
• tools/gallery.py builds the index with split_index(specs).

Second review. Ten findings.

• Loop through another sum of the same file: fixed on main in fix(language): a file whose terms read each other's sums is refused at load #780, carried here in the merge commit.
• A target the same file defines: already fixed in d9d0367.
• A one-line patch dropped adds_to:: _body reads it as a new body over the entry.
• A contributor that caps its own misspelt sum passed the misspelling check: a name one fragment alone reads is refused.
• A term on a sibling's variable got the misspelling message: _undeclared names the kind.
• The sum took the first reader's dims order, so the list order reached the canonical text: _frame refuses readers that order the frame apart.
• _term_block nested a flow mapping and dropped a continued plain body: it rereads such a block as YAML. The fragments it writes do not change.
• _uses reads through variables_of.
• Not changed: a reader that reads the sum only in a where: mask. It does not reproduce, since a mask that reads a given expression is refused at load ("may test parameters and dimension coordinates only").
• Not changed: _uses walks an in-math body twice. The cost is small and the code is shorter as it is.

Guards. Each check was deleted, and the suite run. For the second review's checks, each test below failed on the tree before its fix. The four rows for 5e51689 were taken by hand on that head, each against the full suite.

Check deleted	Failing test
target not read under given:	test_what_a_term_may_not_be_is_refused_at_load[a-mistyped-target]
a target the same file defines	[a-target-this-file-defines]. Before the dedicated check, this case got the "does not read under given:" message ending in "Declared: injection.", and the test failed on it
term reads its own sum	[a-term-reading-the-sum]
term reads its sum through another sum of its file	[two-terms-reading-each-other-s-sum]
term frame vs reading frame	[a-term-wider-than-the-entry]
the definer's body before the terms	test_a_term_follows_the_body_a_fragment_defines[a-definition-a-file-writes], [a-sum-a-merge-wrote], test_the_definer_keeps_its_description, the stepped-merge test, test_a_reader_that_states_less_than_the_definer_is_refused
the plain + (body bracketed instead)	test_a_term_follows_the_body_a_fragment_defines[…] (both), the stepped-merge test, by its written-body assertion
a term on a cased definition	test_a_term_on_a_cased_definition_is_refused
a loop across fragments (_acyclic)	test_a_term_that_reads_its_sum_through_another_fragment_is_refused[through-a-definition], [through-another-sum]. Without it the composed load refuses with a message that names no fragment
a term on a sibling's variable	test_a_sum_a_sibling_declares_as_another_kind_names_that_kind
the readers' frame order	test_readers_that_order_the_frame_apart_are_refused
the misspelling check	test_leaving_out_the_reader_of_a_sum_no_model_goes_without_is_refused[the-bus-balance]
one fragment alone reads the name	test_terms_only_their_own_files_read_are_refused[a-misspelt-given-entry-the-same-file-caps]
math use counts as a read	test_a_fragment_that_reads_the_sum_for_more_than_adding_lets_the_terms_land[a-contributor-whose-own-constraint-reads-it]
a reported read does not count	test_terms_only_their_own_files_read_are_refused[a-misspelt-given-entry-the-same-file-reports]
a one-line patch keeps adds_to:	test_a_patch_changes_a_term_s_body_and_keeps_what_it_adds_to[one-line]
the term body in every source form	test_a_term_block_carries_its_body_in_every_source_form[folded], [flow-mapping], [one-line-continued]
the gallery index's reader	test_the_split_index_names_a_hub_once_per_fragment_and_needs_a_described_reader
a term keeps its line when inlined (#766)	test_inlining_keeps_the_definition_of_a_term

Docs.

• declarations.md "Terms" (anchor #terms): a body another file defines, a later merge and the reader its first step needs, the cases: refusal, the cross-file loop with its message, and the rules that hold only where no file defines the sum.
• howto/compose.md: a merged spec takes a further term; the table row for adds_to:; a one-line patch replaces the body and keeps the other fields.
• The docstrings of merge, GivenExpressionBlock and ExpressionBlock, and the regenerated schema.
• Also updated earlier: named.md, reading.md, typeset.md, the tutorial several-files.md, and the PyPSA page intros.

Gates.

• On head 09cb6f2: pixi run lint clean; pixi run test 2611 passed.
• On head 5e51689: pixi run ci ran. docs-build stopped on one warning, an unresolved pathlib.Path autoref, because the session's proxy blocked docs.python.org/3/objects.inv. compile-tex could not download the tectonic bundle. Both are left to CI, and no file under examples/ changed.
• On head d9d03671: pixi run lint clean; pixi run test 2590 passed.

Deliberately not done. One file may add two terms to one sum. Both are summed, and the gallery index lists both. A term may not add to a body its own file defines: one file has one place for its body. The objectives of a merge in steps still nest their brackets, ((a) + (b)) + (c); the canonical text is the same, and a plain join changes every merged objective, so it is a PR of its own. The first commit's body quotes a wrong sentence count (n 197); the measured figures are in the docs commit.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AApRDPxpZcALYLrFMXEF7z
https://claude.ai/code/session_01AzXL2w5xdUm8FsYMNWf1KW
https://claude.ai/code/session_01JiNwuVCdEkSLKNxiCNaNAv

[read-the-docs-community]

Documentation build overview

📚 mathspec | 🛠️ Build #34855413 | 📁 Comparing e80c1fc against latest (685bbb2)

  🔍 Preview build  

31 files changed · ± 31 modified

± Modified

• changelog/index.html
• several-files/index.html
• examples/library/index.html
• examples/operators/index.html
• examples/pypsa/index.html
• howto/compose/index.html
• reference/api/index.html
• reference/program/index.html
• reference/reading/index.html
• reference/typeset/index.html
• and 21 more...

[FabianHofmann]

I like that. Thank you! Two minor points. First, I don't think a having a key with an underscore in it is super elegant like "adds_to". is there an alternative without it? - hard to think of one...
The second point is that I don't think it needs to add to a expression that is defined in the given block. It could also be added to a expression that is defined in the same yaml file, right?

[FBumann]

I like that. Thank you! Two minor points. First, I don't think a having a key with an underscore in it is super elegant like "adds_to". is there an alternative without it? - hard to think of one... The second point is that I don't think it needs to add to a expression that is defined in the given block. It could also be added to a expression that is defined in the same yaml file, right?

1. I'll try to come up with something better. adds_to is hard to misunderstand, and that's worth a lot, but not pretty...
2. This would mean that there are two ways of "reusing" expressions. Injecting with adds_to, and referencing the name in another expression. That's definitely sth we should avoid, as it doesn't help with anything and is strictly less capable (only ever expands to + expression)

[FBumann]

@FabianHofmann I think there aren't any viable alternatives.

1. into: is already used for sth else. This would change if we decide to change the relations design with feat(program): the axis a join opens names the relation column it stands for #665
2. onto: is too close to into
3. feeds: To physics based
4. to: not too bad
5. injects: does it inject into or does the mentioned one get injected into itself?
6. total: Also not sure

[FabianHofmann]

i see, what about

complements:

?

[FBumann]

i see, what about

complements:

?
complements implies its complete after this is added, which is not true.

[FabianHofmann]

okay, I would vote for adds_to (or perhaps add_to but rather not) - it is clear and says what is does

[FBumann]

@FabianHofmann Cool.
Should we have a chat about this or merge it now?