feat(language): a relation call is a join on the columns it names, and a sum through a relation is a sum over that join

[FBumann]

Prompt: Let's change everything to this groupby and join concept. Only leave the yaml syntax unchanged unless it bites. I'm most worried about internal classes, Nodes, attribute names and docs

Would it be clearer if we do a larger refactor along this rename? [...] Do this as a follow up!

Note

The following content was generated by AI.

What this changes

Every relation call is described by two questions per column, joined on and grouped by, and the program has one relation node, Join. at(x, by=R, over=a, into=b) lowers to Join(x, columns). sum(x, by=R, over=a, into=b) lowers to Sum(Join(x, columns), over=<the dims the join drops>), the Sum node that already exists. The YAML surface is unchanged. #607 landed here, so this is the vocabulary and the refactor it points at in one PR.

The break, for a consumer of math_spec.program.

• Direction, GroupSum and Pullback are gone. Join is the one relation node, with operand and columns. JoinColumns holds what a call named: joined and grouped roles, with dropped, added and kept derived, each with a _dims form.
• Partition.group is Partition.grouped. The parser node DirectionNode is JoinNode, with columns.
• fan_in reports Join as one-to-one and the Sum over it as many-to-one, as before.
• No alias and no deprecation, per the alpha stream. lpspec follows on its own branch once this is released.

main is merged in at alpha.111; its new uses of the relation nodes in Program.names_read, the typesetter and test_lowering take the one Join.

Where the words changed

• The frame rule reads (operand − joined) ∪ grouped in _join_dims; _direction in resolution is _join, and the single-valued test is one_row_per_group.
• Refusals say "joins on", "groups by" and "a join with no group-by" in place of "consumes", "lands on" and "a read". tests/test_validation.py and tests/test_dimensions.py assert the new wording in place.
• The relations page rebuilds "How a relation is used" around a four-row table, one row per answer to the two questions, and reworks the worked calls and rules. The expressions, operators, contributing, declare-a-column and what-counts-as-language pages, three example descriptions, the golden model's constraint names and comments, and the generated pages follow.
• A new explanation page, docs/about/relations-as-linear-maps.md, states what a relation is as an indicator, why the join and group-by are one contraction, and where the built model departs from the matrix. Its argument was checked numerically in the session: rows built by lpspec through sum(by=) equal the indicator matrix from the data, and the at rows equal its transpose.

Why one node is the same model

The absence page already describes the composition. Out of a Sum, an absent summand is one term fewer and the row stands; out of a bare join, absence spreads. Those are today's sum(by=) and at semantics, so nothing on that page moves. The frame rule composes the same way: the join adds the grouped dims and keeps the joined ones, the sum drops the dims over= named. The dim checker and the typesetter read the resolved AST, not the program, so neither changed beyond the names.

Separability keeps its verdicts: a Sum whose operand is a Join reports the grouping message for the dims it drops, and that Join is not also reported as an undecided coordinate read. tests/test_separability.py passes unchanged.

Coverage that moved

• test_lowering.py: the lowering cases and test_a_relation_lowers_with_the_join_each_call_names assert Sum(Join(...), over) and Join(...); the fan-in table has the same two rows in the new shape; test_a_divisor_under_a_join_is_still_named descends through Join.
• tests/fixtures/every_program_node.yaml is unchanged: grouped and looked_up both reach Join, and reduced reaches Sum, so test_every_program_node_is_one_some_file_lowers_to holds with two nodes fewer.
• test_golden.CARRIERS names JoinColumns in place of Direction.

Gates

pixi is unavailable here, so the gates ran from a uv venv on Python 3.12, on the head with main merged.

gate	result
pytest	1534 passed, 6 skipped
ruff check, ruff format --check	clean
mkdocs build --strict	clean before the merge, with the unreachable docs.python.org inventory dropped for the run; tests/test_docs.py holds every generated page current after it
prettier --check	clean on the touched pages
tools.schema, tests.typesetting.golden, the four page generators	re-run and diffed; the schema moved by one docstring, the golden .out files by the three renamed constraints
compile-tex, typos, reuse lint, pyrefly	not run, not installed here

Why

The three roles, consumed, produced and joined on, were bookkeeping that hid the mechanism. Each column of a relation is either joined on or not, and either grouped by or not, and every rule on the relations page falls out of that: over= is joined on and not grouped by, into= is grouped by and not joined on, an unnamed key column is both, an unnamed value column is neither and so is not read. A sum whose grouped columns hold the whole key groups nothing, which is what at is.

Two nodes that carried the same columns and differed only in whether a group-by followed welded the join and the sum into a third kind of reduction. With the join as its own node, the program is the relational algebra the words describe: one join, and the sum you already have over it. lpspec already computes it that way, one join with no aggregate and a drop of dims, both collapsing in the terminal group-by, so its engine loses a branch rather than gaining one.

🤖 Generated with Claude Code

https://claude.ai/code/session_01SWBcNGLjNH2i4AqRsfyxaN

[read-the-docs-community]

Documentation build overview

📚 math-spec | 🛠️ Build #34695951 | 📁 Comparing 53ba4a2 against latest (1e00c23)

  🔍 Preview build  

16 files changed · + 1 added · ± 15 modified

+ Added

• about/relations-as-linear-maps/index.html

± Modified

• CHANGELOG/index.html
• contributing/index.html
• about/what-counts-as-language/index.html
• examples/operators/index.html
• howto/declare-a-column/index.html
• reference/notation/index.html
• reference/reading/index.html
• reference/language/expressions/index.html
• reference/language/operators/index.html
• reference/language/relations/index.html
• and 5 more...

[FabianHofmann]

@FBumann thanks for tackling this. this logic is indeed much more aligned to what happens. the process essentially is joining and summing. good that we reflect it now

[FBumann]

Yes. Very sharp of @coroa to press on it!

I would leave this here until he has time next week.

If you want, we could discuss syntax based on this better internal implementation.

The neat thing here is that joining and suming is now essentially just chained operations. So we can also joind and then do a mean (if we impement mean).

This could enable a different syntax naturally. Maybe sth in the direction of groupby().sum()

Maybe its not a good idea, but the Architecture would enable it now I think. Just a first idea.

[FBumann]

Superseded by #641