feat(language)!: a call names the columns of a relation as relation[column], and a sum through a relation is a sum over the axes its join opens

[FBumann]

Prompt: Try to do a new branch from #641 and rebase it onto 661. […] Now, let's discuss a better api for sum, at and potential future join or groupby operations. What's the best syntax imaginable in the yaml? It should be intuitive, safe and close to the internals […] Let's do this as 2 stacked PRs. One introducing the new syntax, but keeping the dot notation internally, and then the one adding the axis class […] Agreed, keep at() add this reasoning to the pr […] Review the stack of #665 and below. Should we merge it into a single PR for better reviewability? […] Merge them and update the PR description. And close 641 […] Please review 664 and 665 […] Fold 665 into 664 and merge main […] Close 665 and add the ! to the title […] Maybe we should always require the column name in over instead of the dim thats used in the relation! That would remove this problem. And only change over= if a relation uses a dim twice, where its also ok to be explicit! […] Implement this right in 664 as a new commit, and pdate the PR description and docs etc […] Coroas comment: It would probably be good to rename PulledBackPredicate to JoinedPredicate […] Update 664 and resove conflicts

Note

The following content was generated by AI.

A call reads one relation, and names its columns as relation[column, …]. The program writes each such call as a Join, which opens one Axis for each column it drops. A Sum over those axes closes them. This PR includes #662 and #665.

What this changes

The syntax

was	is
sum(p, by=gen_bus, over=generator, into=bus)	sum(p, over=generator, by=gen_bus[bus])
sum(p, by=slot_of, over=[generator, period], into=[bus, technology])	sum(p, over=[generator, period], by=slot_of[bus, technology])
at(cap, by=gen_zone, over=zone, into=generator)	at(cap, by=gen_zone[zone])
shift(p, …, by=cal, within=[day, week])	shift(p, …, within=cal[day, week])
position(h, by=cal, within=day) == 0	position(h, within=cal[day]) == 0
calendar.month == 'jan', ends.bus0 != ends.bus1	calendar[month] == 'jan', ends[bus0] != ends[bus1]

• In a sum, over= names what leaves. Beside by=, a name in over= is a column of the relation where it has one, and a dimension otherwise, summed away after the group-by. So sum(p, over=[generator, snapshot], by=gen_bus[bus]) is one call, and sum(x, over=from, by=nbr[to]) says which of two columns over one dimension the operand is joined on.
• Every column a call touches is written in it. A relation may gain a value column without changing what any call means, which is what composing a relation across files needs. The declaration already refuses a column named after a dimension it is not over, so a name reads the same as a column and as a dimension.
• A lookup names the value columns it reads, and the key arrives. It joins on each key column over a dim the operand carries, unless a column in by= already matches that dim. So at(x, by=rep_of[rep]) reads x at rep(t).
• The grammar allows one table per call. A list of relations cannot be written.
• A macro formal can stand inside a selection. sum(x, over=d, by=rel[col]) binds rel and col at the call site.
• A join operator apart from sum and at is refused, and limits.md records this. The axis must close in the call that opens it.

The program

• Join and JoinColumns replace Direction, GroupSum and Pullback. A sum through a relation lowers to Sum(Join(x, columns), over=axes), and at() lowers to the bare Join. feat(program): a sum through a relation is a sum over the axes its join opens #641 has the design and its guard table.
• JoinedPredicate replaces PulledBackPredicate. at(has_curve, by=zone_of[zone]) in a where is the predicate's lookup Join, and the two names pair as Translate and TranslatedPredicate do.
• Column(relation, name) and Axis(dimension, column=None) are public in mathspec.program. Sum.over and JoinColumns.axes are tuple[Axis, ...]. A declared dimension is Axis(dimension). A join opens Axis(dimension, column) for each column it drops, so a map into its own dimension gives Axis('snapshot', Column('rep_of', 'snapshot')), which is not Axis('snapshot'). The type keeps them apart, where feat(program): a sum through a relation is a sum over the axes its join opens #662 used a dotted name. str() gives the user spelling, generator or zone_of[generator].
• Frames stay sets of dimension names. dims_of reads a sum over a join as one rule, so a join's axes never reach a frame (chore(program): a frame stays a set of dimension names, and only the sum over a join reads the axes it opens #799).
• Separability, boundedness, advice and the typesetter read Join, and separability reads axis.column to tell a grouping from a plain sum.

Why

This is a break. Every file that writes by=R, over=, into= or R.c stops loading, with no hand-written message for the old spelling. The release that carries it raises the minor version.

Every relation call lowers to one Join and a reduction over the axes it opens. The syntax now has the same shape. over= keeps its meaning from a plain sum. by= holds the group-by key, which a pandas or SQL reader expects. The relation is written once, so a call cannot mix tables.

over= reads column names so that a relation can grow. An earlier version of this PR read over=snapshot as a dimension and inferred the column over it, with over=rep_of[snapshot] for the case where two columns are over one dimension. Under that rule, adding alt: snapshot to rep_of made an existing sum(x, over=snapshot, by=rep_of[rep]) stop loading, because two columns now matched. A column name never matches twice.

With Axis, a consumer reads axis.dimension or axis.column, and pyrefly checks it. It does not parse strings. The axes are needed only between a join and the sum that closes it, and the resolver is the only code that builds that pair, so frames do not carry axes.

Method, departures, alternatives, new refusals, coverage moved, gates, not done

How this PR was made

This PR was three stacked PRs. #662 moved #641's program change onto main, with the syntax by=R, over=, into=. This PR replaced that syntax and kept dotted axis names in the program. #665 replaced those names with Axis. The review of the stack found that this PR removed about a third of the lines #662 added, and that #665 removed a convention this PR introduced, so both are folded in: #662's commits are in this branch, and #665 is the squash commit 6a899c9. The branch is not rebased and not force-pushed.

33b8eab merges main, which had moved by #763 (adds_to:), #795, #809 and the 0.2.1 release. The 25 conflicts take main's structure with this branch's spelling of each relation call. The one old-syntax expression main added outside the conflicts, in tests/test_pypsa_split.py, and a prose line on the library generator page are rewritten the same way. The changelog gets a fresh ## Upcoming version heading above 0.2.1.

24c7354 changes over= beside by= from dimensions to column names, after a comparison of the two syntaxes on 29 situations found that a value column added to a relation broke an existing sum under the dimension reading and nothing else. The three PyPSA maintenance constraints that wrote over=X_maintenance_cover[start] now write over=start.

7bb84bc renames PulledBackPredicate to JoinedPredicate, as coroa asked in the comments, and the prose that called a lookup a pullback follows.

9d68a67 merges main again, with the nine PyPSA docs PRs #786 to #817. The one conflict is the changelog. Three at() calls those PRs added in the line and transformer fragments, and in the one-file spec, are rewritten from by=snapshot_period, over=period, into=snapshot to by=snapshot_period[period], and the generated pages follow.

#662 took #641's net diff against its merge base with main and applied it with git apply --3way. #641 still carried #638 as 13 commits that main holds squashed, so a rebase would have applied #638 twice. #662's body lists how each conflict was resolved.

Departures from the discussion

• at(x, by=R[c]) keeps a keyword. The discussion wrote at(cap, gen_zone[zone]), positional. limits.md says everything a modeller passes goes in a keyword value, so that a macro can pass it. by= is that keyword, and it reads the same in sum and at.
• Old spellings get no hand-written message, as AGENTS.md says. into= is a call-shape error, and R.c is a parse error.

Considered and not done: a lookup as an index, cap[gen_zone[zone]]

The index form reads like the printed math, $\mathrm{cap}_{\mathrm{zone_of}(g)}$, and it is shorter. It changes nothing that a file can say, and it has five costs:

1. Brackets would have two meanings. name[...] here means "columns of this relation". cap[...] would mean "read this array at". The parser cannot tell which applies, and the reader has to.
2. The operand is any expression. at(p * eff, by=…) would become (p * eff)[gen_bus[bus]]: a postfix operator on arithmetic, with a precedence against ** and unary minus.
3. It breaks the pairing with sum. sum(x, over=d, by=R[c]) and at(x, by=R[c]) are the two directions through one table. The same by= shows this.
4. It invites indexing that the language does not have. cap['north'], p[t - 1] and p[0] would read as if they should work. Those are label selection, shift and position.
5. It adds nothing to the program. The node, the typesetter output and the refusals are the same.

New refusals, each with the test that fails without it

Each guard was deleted in turn, the named test file run, and the tree restored. #641 has the guard table for the program change.

guard	caught by
a formal in a selection is substituted	test_a_call_site_binds_the_relation_and_its_columns[names]
a selection formal takes only a name	test_a_formal_inside_a_selection_takes_a_name_and_nothing_else
a template formal inside a selection is left bare	test_a_formal_stands_where_a_call_site_will_bind_it[a-relation-and-its-columns]
a selection in arithmetic is refused	test_a_selection_bound_into_arithmetic_is_refused
a selection in over= is refused, with the bare name	test_a_refused_call_is_not_read_by_the_call_around_it, [a-selection-in-over-beside-by], [a-selection-in-over-without-by]
a name in over= that is no column and no dimension	[a-name-neither-a-column-nor-a-dimension]
a sum reading nothing through its relation is refused, naming the columns over the dimension written	[a-sum-reading-nothing-through-its-relation], [a-dimension-two-columns-are-over-is-no-column]
a lookup through a bare relation is refused	[at-through-a-bare-relation]
a lookup of a key column is refused	[a-read-of-a-key-column]
a lookup joins only on dims its columns do not match	test_dim_inference[and-so-does-its-lookup]
a comparison of several columns names the rewrite	test_a_relation_column_is_named_in_brackets
a join's axis is not the dimension's own axis	5 tests, e.g. test_dim_inference[a-map-into-its-own-dimension-keeps-the-frame]
separability reads a join axis as a grouping	test_a_grouping_that_sums_the_axis_away_couples_it

test_a_value_column_added_to_a_relation_changes_no_call is the case 24c7354 was made for. On the tree before it, the test fails with "over=snapshot matches 2 columns of 'rep_of'".

Two asserts in dimensions.py from #799 have no test that fails without them: a sum over a join closes exactly join.columns.axes, and a join that opens axes is read only by the sum that closes them. The resolver makes both true, and no probe builds such a node by hand.

Coverage that moved

• Two refusals are now legal lookups. at(zone_cap, by=gen_zone[zone]), where zone_cap lacks the key's snapshot, and at(load, by=diag[rep]), where rep and the key share snapshot. Both are in test_a_lookup_joins_on_the_key_columns_the_operand_carries_and_the_read_does_not_match, with their frames.
• Three refusals of the dimension reading of over= are gone with it. "matches 2 columns" cannot arise from a column name. "over= and by= name two tables" cannot be written. "sums away the dim it groups onto" is now the case "over= and by= both name the column", [over-and-by-the-same-column].
• A lookup can no longer group by a value column, because the syntax cannot say it. Three cases for that are removed.
• "names 2 relations" is gone, because the grammar cannot write it. by=[lk, lk2] is refused as "takes columns of one relation".
• The "leaves within= unsaid" cases are gone, because within= now carries the relation.
• The lookup "several rows per group" check is removed. A lookup reads value columns at the whole key, so it cannot fail.
• GroupSum and Pullback tests read Join. tests/test_lowering.py and tests/test_separability.py take feat(program): a sum through a relation is a sum over the axes its join opens #641's names and messages.
• tests/test_dimensions.py compares frames to string sets. The self-map test asserts node.over == (Axis('snapshot', Column('rep_of', 'snapshot')),) and the frame of the sum, and no longer the frame of the bare join, which no longer exists.
• tests/test_lowering.py and tests/test_expansion.py build Sum with Axis values. tests/typesetting/test_golden.py lists Axis and Column in CARRIERS.

Generated files

tools.schema, tools.gallery, tools.notation, tools.spec_math, tools.home_math, tools.expansion_math and tests.typesetting.golden were run again after each merge of main and after each later commit. The schema carries the relation block's docstring; the three maintenance pages carry over=start; the notation page's heading reads "Shift and lookup in a condition". The one golden change against main is the legend's shift(within=relation[c]).

Gates, on 9d68a67

gate	result
pixi run lint	clean
pixi run test	2665 passed
pixi run typecheck	0 errors
docs-build, compile-tex	not run here: the proxy refuses docs.python.org and the tectonic bundle host. render-tex rendered all 55 documents. CI runs both.

Not done

• Relation columns are still called roles internally (JoinColumns, _role_name, key_roles). JoinColumns.kept_dims has no reader. That cleanup is separate.
• The plain dims in over= are summed after the group-by, as Sum(Sum(Join(x)), plain). Summing them before the join would make sum(lim * p, over=[generator, bus], by=gen_bus[bus]) legal, where today the clash refusal fires. That is a language decision, separate from this PR.
• No new reduction. A mean(x, over=a, by=R[b]) would lower the same way. It needs a language decision first: whether an empty group is absent.
• Consumers of the program, such as fluxopt/lpspec, are not updated. GroupSum, Pullback, Direction and PulledBackPredicate are gone, and Sum.over holds Axis values.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MKjqJocyxdRYvx7xin6NCM
https://claude.ai/code/session_01YVJQzL5G4boHcJDwrhgfag

[read-the-docs-community]

Documentation build overview

📚 mathspec | 🛠️ Build #34896710 | 📁 Comparing 9d68a67 against latest (ae4a44c)

  🔍 Preview build  

50 files changed · + 1 added · ± 49 modified

+ Added

• about/relations-as-linear-maps/index.html

± Modified

• changelog/index.html
• contributing/index.html
• several-files/index.html
• about/limits/index.html
• about/what-counts-as-language/index.html
• examples/library/index.html
• examples/operators/index.html
• examples/pypsa/index.html
• examples/pypsa_linearized_uc/index.html
• howto/compose/index.html
• and 39 more...

[coroa]

I like what has happened since i last looked.

The Program nodes are used for typesetting and dimensions checks and so on. I think that was inevitable. cool that this has landed (even though this was not part of this PR).

The lowered code in this PR is a lot more straightforward:

sum(p, over=generator, by=gen_bus[bus])
# → Sum(Join(p, JoinColumns(joined=('generator',), grouped=('bus',))), over=('gen_bus.generator',))

and #665 with the Axis node also seemed to make sense here.

I still have a bit of troubles with the interpretation of the actual syntax, i do like the arrow stuff better. Here it is a bit unlcear that generator comes in by the gen_bus relation, and again basically mirrors the over -> into pattern.

[coroa]

It would probably be good to rename PulledBackPredicate to JoinedPredicate, due to the symmetry with:

at(cap, by=zone_of[zone])          # Join(cap, cols)                        expression
at(has_curve, by=zone_of[zone])    # PulledBackPredicate(Mask(has_curve), cols, dims)   predicate

[FBumann]

I still have a bit of troubles with the interpretation of the actual syntax, i do like the arrow stuff better. Here it is a bit unlcear that generator comes in by the gen_bus relation, and again basically mirrors the over -> into pattern.

If you are talking about syntax, are you talking about the yaml syntax or the "program syntax" (Sum(Join(p, JoinColumns(joined=('generator',), grouped=('bus',))), over=('gen_bus.generator',)))

[coroa]

The program syntax is a definite improvement. The actual yaml syntax i am still at odds with, but hopefully we can improve that with the better program syntax.

[FBumann]

@FabianHofmann We have @coroa 's go to merge this as its a clear improvement, although he might want to discuss some follow up syntax changes as he has had some more time to look into the details.