Skip to content

feat(program): a sum through a relation is a sum over the axes its join opens - #662

Closed
FBumann wants to merge 36 commits into
mainfrom
claude/serene-goldberg-m8xwru
Closed

FBumann wants to merge 36 commits into
mainfrom
claude/serene-goldberg-m8xwru

Conversation

@FBumann

@FBumann FBumann commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Prompt: Try to do a new branch from #641 and rebase it onto 661. Any issues? […] Dont change anything on 641 […] PR […] Conflicting […] Now, let's discuss a better api for sum, at and potential future join or groupby operations. What's the best syntax imaginable in the yaml? It should be intuitive, safe and close to the internals […] Fold this into 662

Note

The following content was generated by AI.

#641's change, on top of #661, and the syntax decision it implies. The call shape sum(x, by=R, over=a, into=b) and at(...) stays. The docs now say by= names the relation, and a separate join operator is refused.

What this changes

  • feat(program): a sum through a relation is a sum over the axes its join opens #641's change (1e30b80). Join and JoinColumns replace Direction, GroupSum and Pullback. A join opens one axis named relation.column for each column it drops. sum(by=) becomes Sum(Join(x, cols), over=axes), and at() becomes the bare Join. feat(program): a sum through a relation is a sum over the axes its join opens #641's body has the design, the guard table and the rationale.
  • The relation-call syntax, in the docs (e88efcf):
    • relations.md says by= names the relation and not the columns grouped by. It says over= names what leaves and into= names what the result lands on, as over= does in sum(p, over=d).
    • expressions.md drops "the consumed dimension goes and the produced one arrives" from the predicate at() paragraph. That wording is from before the join design.
    • limits.md records a join operator apart from sum and at as refused, and names what to write instead.

This PR stacks on #661, so its diff is the above on the #661 stack and nothing else.

Why

The user asked for #641 on top of #661, and then for the syntax discussion to be folded in.

The call shape is kept because it already matches the nodes. Every relation call lowers to Reduce(Join(x, JoinColumns(joined, grouped)), over=axes), and each keyword maps to one part of that node:

  • by= is the relation.
  • over= is joined and not grouped: the axes the reduction closes.
  • into= is grouped and not joined.
  • The operator name is the reduction, and at is no reduction.

The operator name is a claim the loader checks. sum claims several rows per group and at claims one. The key decides which is true, and a mismatch is refused at load. A named into= means that a new value column changes no call.

A separate join is refused because the axis must close in the call that opens it. An example is sum(join(p, R), over=R.a). This form would let an open relation.column axis into an expression's frame, where no declaration can carry it. It would also need a new "every axis closes" rule, and it would make the author name the relation twice. In the one-call form, no file can leave an axis open.

by= keeps its name. A pandas reader can take by= for the group-by keys, which are into= here. The fix is to say so in the prose. sum p by gen_bus reads well, so the keyword is not renamed.

Method, conflicts, gates, what was not done

Why one commit and not a rebase

#641 still carries #638 as its 13 original commits. main holds #638 as the single squashed commit b3cee88. A git rebase would apply #638 a second time. So I took #641's net diff against its merge base with main (1e010ca), which is 42 files, +742/−466, the same as the #641 PR. I applied that diff to #661's head f0dc3e3 with git apply --3way (1e30b80).

#661 then gained b535973, which removes UnexpandedCurveError. 5c0814c merges that head in. The branch is not rebased and not force-pushed.

Conflicts and how they were resolved

file resolution
resolution.py The stack moved the expression walk and the where walk out (a2db22c). #641's hunks go to _expression_resolver.py (the sum/at node build, _relation_ref, _direction → _join with its refusal wording) and to _where_resolver.py (join_dims, the predicate at() check). resolution.py keeps the stack's side.
lowering.py The stack deleted inline(), because a program carries Named. #641's Join case in it goes with it.
advice.py #661's current imports (LanguageError, to_spec, after b535973), with Join in place of GroupSum and Pullback.
boundedness.py Sum | Join | Translate | WindowSum | Cases | Named: #641's Join and the stack's Named.
program.py __all__ holds Join, JoinColumns and the stack's Link.
typesetting/walk.py The stack's self.program.relations[...].values, with #641's join.
tests/test_lowering.py, tests/test_separability.py #641's names and messages, with the stack's to_spec(...).program.

One line merged without a conflict but needed a change: #641's new test_a_sum_over_a_lookup_is_a_sum_and_a_lookup_not_a_grouping called ms.to_program, which the stack removed. It now calls to_spec(...).program.

Generated files

python -m tools.schema and python -m tests.typesetting.golden were run again after each step. The output is the same as the merged text.

Docs sentences

After (before): relations.md n 45, avg 16.3, median 15, over 25 words 8 (43, 16.2, 14, 8). expressions.md n 92, avg 15.3, median 14, over 25 words 11 (91, 15.4, 14, 11). limits.md is unchanged at n 55, avg 18.3, median 19, over 25 words 14, because the new text is a table row.

Gates

Pixi is not reachable here (the proxy refuses pixi.sh), so the gates ran from a uv venv on Python 3.12, with the pinned ruff and pyrefly. The results below are for e88efcf unless a row says otherwise.

gate result
pytest -q -n auto 1624 passed, 1 skipped
ruff check, ruff format --check clean on 5c0814c; e88efcf changes only docs/
pyrefly check 0 errors on 5c0814c
mkdocs build --strict clean, with the docs.python.org inventory left out for the run (the proxy refuses it)
typos clean on docs/
reuse lint clean on 1e30b80; not run again
prettier --check clean on the three changed pages; warns on mkdocs.yml and the schema, as #661's head does
compile-tex, zizmor, the full lefthook lint not run

Not done

🤖 Generated with Claude Code

https://claude.ai/code/session_01PHNF149r8B4tGmGUKmAdUd

… and every curve is written out before a model becomes one

Resolved is gone. Lowering builds the Program as a model loads and holds
it on the Spec; to_program(spec) returns that object, and returns the
expansion's for a model with a curve, so a program holds the rows a curve
states and never the curve. Named joins program.Expression, so a use of an
expressions: entry stands where it is read and the typesetter prints from
the program beside the file. ExpressionDeclaration carries its dims, and
piecewise.curve reads a block's typed links and frame off the model for
the expansion and the walk. PiecewiseDeclaration, Program.piecewise,
Spec.resolved, lowering.inline and the refusal of a model still carrying
a curve are removed.

The typesetter's output is unchanged: the golden files and the generated
pages regenerate byte for byte.

Docs sentences, after (before): reading.md n 75 avg 16.1 median 15
over25 10 (72, 16.4, 14, 13); piecewise.md n 70 avg 17.6 median 15
over25 13 (69, 17.8, 15, 14).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TrvhjFQCQJ6ATBQkfhcoMi
…s included, and the typesetter reads it alone

to_program lowers the model as it arrived: a piecewise: block still in it
is a curve on the program, typed, with its links, signs, method, gate, mask
and frame, and the program of spec.expand('piecewise') carries the rows
instead. Every declaration carries its description, and the program the
file's. The typesetter takes a Program and reads nothing else; the walk no
longer re-resolves a curve's links at print time, and piecewise.curve is
gone. relations_of moves from Spec to Program. advice() writes curves out
itself and refuses a program still carrying one.

The typeset output is unchanged: the golden files and the generated pages
regenerate byte for byte.

Docs sentences, after (before): reading.md n 75 avg 16.8 median 15
over25 12 (75, 16.1, 15, 10); piecewise.md n 70 avg 17.6 median 15
over25 14 (70, 17.6, 15, 13).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TrvhjFQCQJ6ATBQkfhcoMi
… lowering, before any expression is read

The nine after-validators on Spec that read one declaration against the
others — name collisions, frames over declared dimensions, relation
targets, bound names, set shapes and bounds, curve references and the names
an expansion would collide with — are functions in validation.py, collected
by reference_errors and run first by lowering.lower. Spec keeps the shape
rules pydantic decides per block, and no longer imports sos or the operator
table; side_columns is the public name of the relation helper the rules
share. Every message is the same string, and every refusal the same class.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TrvhjFQCQJ6ATBQkfhcoMi
… is advised as its rows are

The guard landed in the previous commit without the test that fails without
it. With the guard deleted, the new test fails: a program with a block is
advised on the file's own rows as if the curve stated none.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TrvhjFQCQJ6ATBQkfhcoMi
… the program owns the vocabulary the file declares in

Loading no longer expands every curve to validate the expansion: the rows a
curve states are held to the language when expand() writes them out, since
an expansion is a model like any other. A model with a curve is validated
once at load rather than twice.

The dtype, domain, absence, sense, set-order and method vocabulary moves
from model.py into program.py, which model.py imports. program.py no longer
imports model.py, so the type-only import of Program on Spec and its noqa
go, and sos.py imports Spec plainly instead of inside a function.

Load cost, before (after): examples/piecewise.yaml validated 2 (1) times
and parsed 9 (4) expressions; examples/sos.yaml the same.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TrvhjFQCQJ6ATBQkfhcoMi
…he one door to both states is to_spec

to_program is gone. Spec.program is the model typed, section for section,
built once as the model loads; spec.expand(...).program is its rows. A
consumer building rows reads the sections it takes and refuses a curve or a
set it finds, the way it already had to for a set; advice() does so for a
Program handed to it, and writes curves out itself for a file or a Spec.
typeset and typeset_declaration take a Program as before.

Docs sentences, after (before): reading.md n 75 avg 17.1 median 15
over25 14 (76, 16.8, 15, 12); piecewise.md n 70 avg 17.7 median 15
over25 13 (70, 17.6, 15, 14); what-counts-as-public-api.md n 18 avg 16.0
median 14 over25 3 (unchanged count); limits.md n 55 avg 18.3 median 19
over25 14 (unchanged count).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TrvhjFQCQJ6ATBQkfhcoMi
…the model loads

The private attribute lowering filled and the property asserting over it
are one cached_property computing lower(self); the after-validator forces
it, so a Spec in hand has still passed the whole language.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TrvhjFQCQJ6ATBQkfhcoMi
…ules beside the namespace

_Resolver, one class over both grammars, is ExpressionResolver in
_expression_resolver.py and WhereResolver in _where_resolver.py, the where
walk building a side that is an expression through the expression walk.
resolution.py keeps the Namespace and the doors lowering calls. The three
methods the where walk reads from the expression walk are public on it;
names_in and the literal-number helper move to the parser module beside
the other helpers over parsed nodes. Every message is the same string.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TrvhjFQCQJ6ATBQkfhcoMi
…he program carries, and nothing else

The Curve alias, a tacit protocol between the pydantic block and the
program declaration, is gone: assumptions_of, Emitted.of and the curvature
rule take a PiecewiseDeclaration. The expansion keeps the block for the
link text its rows repeat and takes the declaration for the frame and the
names it writes. The two emitted-name collision rules read the program
rather than the file, so they run once the declarations exist; every
message is the same string.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TrvhjFQCQJ6ATBQkfhcoMi
…s use off the program, and the walk keeps no record

The glossary and the three kinds of note move from walk.py to legend.py.
What they explain is read off the program before anything prints, by
notice(), so the walk no longer fills a Noticed record as it prints and a
subscript no longer mutates the walk through its context. Walk.line refuses
a name declared as none of the five kinds or as two, so typeset_declaration
is one call. Symbols is a frozen record built by symbols_for. The typeset
output is unchanged: the golden files match byte for byte.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TrvhjFQCQJ6ATBQkfhcoMi
…, the errors, the advice and the typesetter

Thirteen names leave math_spec's top level: the message builders
call_shape_error, edge_error, unknown_operator_message and schema_error,
the tables BUILTIN_NAMES and EDGE_WRAP, the vocabulary sets ADVICE_KINDS,
DIMENSION_DTYPES, PARAMETER_DTYPES, VARIABLE_DOMAINS, VARIABLE_ABSENCE and
CURVATURES, and SosBlock. The message builders and tables stay in their
modules for the resolver; the vocabulary sets are deleted, since the
Literals they were the set form of are what a consumer pins against, and
nothing in the package read them. did_you_mean stays: it is the one wording
a consumer's own refusals share with the language's.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TrvhjFQCQJ6ATBQkfhcoMi
…vocabulary with no Literal form

A consumer pins its operator table against BUILTIN_NAMES, and unlike the
dtype vocabularies it has no Literal on math_spec.program to pin against
instead; the module it lives in is package-private.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TrvhjFQCQJ6ATBQkfhcoMi
…d advice writes nothing out on the caller's behalf

UnexpandedCurveError carries the one sentence every consumer building rows
says of a program still carrying a piecewise: block, naming the blocks and
the expansion to pass. The language does not raise it at load, since a
model with a curve is printed and edited as written; advice() raises it
for a file, a Spec or a Program alike, and no longer expands a file or a
Spec itself. The check verb and the check how-to expand first, as a front
end may.

Docs sentences, after (before): reading.md n 76 avg 17.1 median 15
over25 14 (75, 17.1, 15, 14); check.md n 20 avg 12.5 median 14 over25 1
(19 sentences before, one added).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TrvhjFQCQJ6ATBQkfhcoMi
…ck verb takes --expand like every other

The check verb read a file with its curves written out, the one verb that
read a file differently from the rest. Every verb now reads the file as
written and takes --expand; check refuses a curve model in the one wording
until asked. The premise is on the public-API page, beside the other things
every function keeps, and the reading page and the check how-to say it
where a consumer meets it.

Docs sentences, after (before): check.md n 22 avg 13.0 median 14 over25 2
(20, 12.5, 14, 1); what-counts-as-public-api.md n 22 avg 16.4 median 14
over25 4 (18, 16.0, 14, 3); reading.md n 77 avg 17.1 median 15 over25 14
(76, 17.1, 15, 14).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TrvhjFQCQJ6ATBQkfhcoMi
UnexpandedCurveError is gone. advice words the refusal of a curve left as
written itself, as a LanguageError, and a consumer building rows words its
own; the reading page shows the idiom with the consumer's own error. A
sentence is not a thing one tool should export for another to reuse, where
a computation such as did_you_mean is.

Docs sentences, after (before): reading.md n 76 avg 17.4 median 15
over25 15 (77, 17.2, 15, 14); piecewise.md n 70 avg 17.7 median 15
over25 13 (70, 17.7, 15, 14).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TrvhjFQCQJ6ATBQkfhcoMi
…in opens

The change of #641, carried onto #661 as one commit. #641 carries #638
as its separate commits, and main holds #638 squashed, so its net diff
against its merge base with main (1e010ca) is applied here, not its
history. The resolver edits move into _expression_resolver and
_where_resolver, where #661's stack split resolution.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PHNF149r8B4tGmGUKmAdUd
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PHNF149r8B4tGmGUKmAdUd
…, and a join is never written apart from its reduction

The relations page says by= is not the grouping, so a reader does not
read it as a group-by key. The predicate at() paragraph drops the
consumed and produced wording the join design retired. The limits page
records a join operator apart from sum and at as refused, with the
reason: the axis a join opens must close in the call that opens it.

Docs sentences, after (before): relations.md n 45 avg 16.3 median 15
over25 8 (43, 16.2, 14, 8); expressions.md n 92 avg 15.3 median 14
over25 11 (91, 15.4, 14, 11); limits.md n 55 avg 18.3 median 19
over25 14 (unchanged; the new text is a table row).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PHNF149r8B4tGmGUKmAdUd
…s expand

"states rows rather than being one" was hard to parse on first read, and
the sentence named the fix with a different word from the method the user
calls. The refusal now says the block is still a curve, that advice reads
the rows a curve is expanded into, and to expand first.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TrvhjFQCQJ6ATBQkfhcoMi
The unboundedness pass read a set as it is, every variable it restricts
being named by a row, and refused a curve. A curve states its rows the
same way: each link names the variables a link row would, so the pass
reads the links and the answer is the expansion's with nothing expanded.
The refusal goes, and check loses --expand, a flag that would change no
answer.

Guard: with the line reading the links deleted, three tests fail: the
advice test over the file, the Spec and the Program; the boundedness
case carried-by-a-curve; and the existing test that a curve holds its
variables, which now runs on the block.

Docs sentences, after (before): check.md n 20 avg 12.8 median 14 over25
1 (21, 13.3, 14, 2); what-counts-as-public-api.md n 20 avg 18.7 median
15 over25 4 (20, 18.1, 15, 4).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TrvhjFQCQJ6ATBQkfhcoMi
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PHNF149r8B4tGmGUKmAdUd
@FBumann
FBumann added this pull request to stack #652 September 23, 2026 22:47
@FBumann
FBumann removed this pull request from stack #652 September 24, 2026 08:35
@FBumann
FBumann changed the base branch from claude/confident-faraday-coc7mj-11 to main September 24, 2026 09:44
@FBumann
FBumann added this pull request to stack #670 September 24, 2026 09:44
…rg-m8xwru

main landed the stack this branch was built on as one squash commit per
PR. The squash of #661 has the same tree as 279997a, the stack head
this branch contains, so this merge records it and changes no file. The
merge of main that follows then brings only what landed after it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GpnFcQDuvt2bExGngb6MYk
After the squash of the stack was recorded, main brought #666, #668 and
the release. One conflict: the golden census imports this branch's
Join in place of GroupSum and Pullback, and main's legend module.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GpnFcQDuvt2bExGngb6MYk
FBumann pushed a commit that referenced this pull request Sep 24, 2026
…662

No textual conflict. The legend test main added in #666 wrote a grouped
shift as by=season_of, within=season; this branch spells it
within=season_of[season], so the test does too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GpnFcQDuvt2bExGngb6MYk
@FBumann FBumann added the area: relations relations and dimensions: the relation design label Sep 24, 2026
#633 retired fan_in, FanIn, quotients and divisor_parameters. The branch's
Join-aware fan_in and its tests go with them, and the divisor test under a
join asserts through parameters_of.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012JFQDAjuZ3sCndQ4cD77Gc
FBumann pushed a commit that referenced this pull request Sep 24, 2026
…u-2, through the updated #662

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012JFQDAjuZ3sCndQ4cD77Gc
Ports #662 onto main's rename to mathspec, its autoref docstrings and the model-to-spec wording, and adds its changelog line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YUK4sePJqJfxJJZLhdvZnS
@read-the-docs-community

read-the-docs-community Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019TMqXLvfDGYm8hRND63cs8
FBumann pushed a commit that referenced this pull request Sep 30, 2026
FBumann pushed a commit that referenced this pull request Sep 30, 2026
…wru-3, with #662 and #664 as one changelog line

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MKjqJocyxdRYvx7xin6NCM

FBumann commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor Author

Prompt: Review the stack of #665 and below. Should we merge it into a single PR for better reviewability? […] Merge them and update the PR description. And close 641

Note

The following content was generated by AI.

Closed in favour of #664, which holds this PR's commits and now targets main. #664 replaced this PR's by=R, over=, into= syntax, so the two PRs are reviewed as one. The branch is kept.


Generated by Claude Code

@FBumann FBumann closed this Sep 30, 2026
@FBumann
FBumann removed this pull request from stack #670 September 30, 2026 13:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: relations relations and dimensions: the relation design

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants