feat(language): a lookup keyed by several dimensions, walked along the key a call names

[FBumann]

Prompt: "What is our proposal, using coroa's design? I'd like to implement it as a PR on top of 422. With all rules written down. The PR description should be the artifact to discuss the decisions taken"

Note

The following content was generated by AI. This is the proposal for #275's remaining question and #161, built on the design @coroa sketched in this comment. It is an alternative to #428 (per:), which it supersedes if adopted; both are open so the two can be compared on the same base.

The proposal, on one complete model

Generators bid in zones, and a generator's zone changes by period, so the map has two keys. This model loads on the branch as written, and the dimensions in the comments are what the loader reports.

dimensions:
  generator: {}
  zone: {}
  period: { dtype: int }

lookups:
  zone_of: { over: [generator, period], into: zone }   # one row per (generator, period): the zone it bids in

parameters:
  cost:   { dims: [generator] }
  demand: { dims: [zone, period] }
  price:  { dims: [zone, period] }

variables:
  p: { foreach: [generator, period], bounds: { lower: 0 } }

Every walk of zone_of names the key it walks after a dot. The other key is joined on and kept.

constraints:
  zone_balance:                                  # one row per (zone, period)
    foreach: [zone, period]
    expression: sum(p, by=zone_of.generator) >= demand
    #  p[generator, period] ── consume generator, join period, produce zone ──▶ [zone, period]  ≥  demand[zone, period]

  history:                                       # one row per (generator, zone)
    foreach: [generator, zone]
    expression: sum(p, by=zone_of.period) <= 100
    #  p[generator, period] ── consume period, join generator, produce zone ──▶ [generator, zone]  ≤  100

  capped_revenue:                                # one row per (generator, period)
    foreach: [generator, period]
    expression: at(price, by=zone_of.generator) * p <= 1000
    #  price[zone, period] ── consume zone, join period, produce generator ──▶ [generator, period]
    #  × p[generator, period] ──▶ [generator, period]

  first_in_zone:                                 # one row per (generator, period) the mask admits
    foreach: [generator, period]
    where: "position(period, by=zone_of.period) == 0 AND zone_of == 'A'"
    #  position counts period within each (generator, zone) group; the comparison reads both keys ──▶ mask over [generator, period]
    expression: p <= 10

objective: { sense: minimize, expression: sum(p * cost) }
#  p[generator, period] × cost[generator] ──▶ [generator, period], summed over both ──▶ scalar

The data for zone_of is one table with columns generator, period, zone, one row per (generator, period). That one table serves zone_balance, history, capped_revenue and first_in_zone. Under #428 history needs a second declaration of the same table, since per: fixes which key is consumed.

One block, one kind: a lookup is a single-valued map from its key dimensions into a target. over: is one dimension or a list. A call walks one key, named after a dot, and joins on the rest.

What each walk does to the dimensions

call	operand carries	consumed	joined on and kept	produced	result
sum(p, by=zone_of.generator)	generator, period	generator	period	zone	zone, period
sum(p, by=zone_of.period)	generator, period	period	generator	zone	generator, zone
at(price, by=zone_of.generator)	zone, period	zone	period	generator	generator, period
at(x, by=zone_of.period)	zone, generator	zone	generator	period	generator, period
shift(p, over=period, offset=1, by=zone_of.period)	generator, period	—	generator	—	generator, period; the step stays inside each (generator, zone) group
position(period, by=zone_of.period) == 0 in a where	frame generator, period	—	generator	—	true at the first period of each generator's stay in a zone
zone_of == 'A' in a where	frame generator, period	—	both keys	—	true where the map says A

sum consumes the walked key and produces the target; at consumes the target and produces the walked key; shift, sum_back and position walk the key they already name and group within the other keys. In every row the keys not walked stay in the result. A one-key lookup is the same table with an empty "joined" column, and by=gen_bus is by=gen_bus.generator.

The rules, each decided at load

#	Rule	Refused with
1	over: is one declared dimension or a list of them, each named once; into: is a declared dimension that is not a key	the key list, or "maps into itself"
2	by=name.key names the key the operator walks. Required where the lookup has several keys; with one, by=gen_bus and by=gen_bus.generator are the same call	"keyed by [generator, period], and the call has to say which key sum walks — by=zone_of.generator or by=zone_of.period"
3	The walked key is consumed by sum, produced by at; every other key is joined on: the operand carries it and the result keeps it	"joins on ['period'], which the expression does not carry"
4	shift, sum_back and position walk the dimension they already name, so the dotted key must be that dimension	"walks 'generator' but groups along 'period'"
5	A by=[a.k, b.k] list is one grouping, so every entry walks the same dimension; each joins on its own other keys, and no two target the same dimension	"groups through lookups along different dimensions"
6	A where naming a lookup — bare, compared, or compared to another — reads every key, so the frame carries all of them, and two compared lookups have the same keys	the existing frame refusal; "compares lookups keyed by different dimensions"
7	At bind the table has one column per key, named after its dimension, plus the value column, and is single-valued per key tuple	the consumer's DataError

Rule 7 is the one #161 asked for: a generator in two zones in one period is a refusal, where a 0/1 membership parameter said it legally and silently.

Decisions taken, and why

The dot instead of per: (#428). per: fixes the consumed key in the declaration, so sum(p, by=zone_of.period) needed a second declaration of the same table. The dot moves that choice to the call, which is where the direction of a walk is already chosen (sum vs at). Nothing per: said is lost: a per list of any length is the same as a longer over: list with one key walked. The consumer nodes are the same shape either way, with keys in place of per.

coroa's function kind is taken whole; a bare relation (no into:) is a parameter, and the placement table now says so. Asked use by use what a relation without into: says that the language cannot: selecting on it is a bool parameter; the weighted aggregation in coroa's own example, sum(efficiency * p, by=connection.bus), is sum(efficiency * p, over=entity) today, because efficiency[entity, bus] has a row exactly where the pair exists; an unweighted fan-out is sum(p * connection, over=entity) with a 0/1 parameter; and at, shift(by=) and position(by=) are undefinable through a many-to-many map, since a row has several values or several groups. The kind makes no cardinality claim, so it checks nothing a parameter does not. What it would add is the legend printing connection as structure rather than data — the purity argument that kept the label-space kind alive until #422. The rules also do not close: with no declared side, the dot would name the produced column and the rest would be consumed, the opposite of its meaning on a function. The real many-to-many shapes are not relations on inspection: a PyPSA multi-link is {over: [link, port], into: bus}, a function of two keys; a cycle incidence is signed weights; a technology present at some nodes is a mask. The "Dimension, lookup or parameter?" table gains the row relates members of two dimensions many-to-many → a parameter over both, with the rewrite beside it.

Columns are named after their dimensions, not roles. coroa's over: {bus0: bus, bus1: bus} sugar is the natural spelling for a self-map (#424) and for two columns over one dimension. Nothing in this PR can walk such a lookup: a walk needs one axis per dimension in the operand, so a second bus column has nothing to join on. Deferred with #424, where the self-map is what needs it (#436 takes it up).

lookups stays the block name. groups reads wrong for at(by=) and for a where filter; maps is the better noun but the rename buys a reader nothing the legend does not already show. A one-day change if the discussion prefers another word.

The dot is not admitted in where comparisons. A comparison reads the whole key table, so there is no key to choose; position(d, by=l.k) is the one where form that walks, and there k must be d (rule 4).

What consumers see

LookupDeclaration(name, target, keys), with keys in declared order. GroupSum and At gain keys: tuple[tuple[str, ...], ...], one per coordinate, with over still the walked dimension. LookupComparisonNode, LookupPairComparisonNode and LookupDefinedNode carry keys in place of over; DimensionPositionNode carries the lookup's keys beside by. Program.lookups is a dict by name (a two-key lookup sits under both DimensionDeclarations, once in the dict). The legend prints zone_of: 𝒢 × 𝒫 → 𝒵, and every application zone_of(g, p).

The golden model's new lines, rendered

gen_zone: { over: [generator, snapshot], into: zone }
zonal:          { foreach: [snapshot, zone],      expression: sum(p, by=gen_zone.generator) <= zone_cap }
zonal_history:  { foreach: [generator, zone],     expression: sum(p, by=gen_zone.snapshot) <= zone_cap }
zonal_pullback: { foreach: [snapshot, generator], where: "gen_zone == 'north' AND position(generator, by=gen_zone.generator) == 0",
                  expression: p <= at(spill * zone_cap, by=gen_zone.generator) }

$$\sum_{g \in \mathcal{G} : \mathrm{gen_zone}(g, t) = z} p_{t,g} \le \mathrm{zone_cap}_{z} \qquad \forall t, z$$

$$\sum_{t \in \mathcal{T} : \mathrm{gen_zone}(g, t) = z} p_{t,g} \le \mathrm{zone_cap}_{z} \qquad \forall g, z$$

$$p_{t,g} \le \mathit{spill}{t} \cdot \mathrm{zone_cap}{\mathrm{gen_zone}(g, t)} \qquad \forall t, g : \mathrm{gen_zone}(g, t) = \text{'north'} \wedge \mathrm{pos}_{\mathrm{gen_zone}(g, t)}(g) = 0$$

Where it lands

• _expression_parser.py, _where_parser.py: a dotted name is a token in a by= value and nowhere else.
• model.py: over: str | list[str] with a keys property; rule 1 in _lookup_targets.
• resolution.py: _walked (rules 2, 4 for position), the list rule (5), the pair rule (6); Namespace.lookups is name -> (keys, into).
• dimensions.py: _check_joined (rule 3), the walked-key check for translations (rule 4).
• program.py, lowering.py: keys on every node above; Program.lookups by name.
• typesetting/walk.py: a map prints its keys in declared order, the walked one as the reduction's dummy.
• docs: the lookups field table, a "Keyed by several dimensions" section carrying the rules above, the many-to-many row and rewrite in the placement table, rows in operators and expressions; notation regenerated.
• tests: 5 declaration refusals, 5 call-site refusals, 5 dim inferences, 4 dim refusals, 3 predicate readings, one lowering test over every node, the golden case in three formats.

Verified

On the current head, in a Python 3.13 venv built from PyPI (pixi.sh is blocked in this environment, so the tool versions are the latest rather than the pinned ones):

• pytest -q: 1231 passed, 6 skipped.
• ruff check .: all checks passed. ruff format --check .: 127 files already formatted.
• Every generated file was regenerated rather than hand-merged — tools.schema, tools.notation, tools.spec_math, tools.home_math, tools.gallery, tests.typesetting.golden — and tests/test_docs.py passes on the result.

Not run: pyrefly, reuse lint, typos, prettier --list-different, compile-tex, and pixi run ci as a whole. mkdocs build --strict reaches its last step and then aborts on the one error the proxy causes — it cannot fetch https://docs.python.org/3/objects.inv for mkdocstrings.

Four conflicts against main were resolved by hand:

• resolution.py: main lifted the kwarg-name reading into a shared names_in() (chore: the separability walk leaves program.py, and every table keyed by an operator is pinned to the closed set #418) where this branch inlined the same two isinstance arms; _lookup_ref now calls names_in and keeps this branch's written spellings, which carry the dotted key.
• typesetting/walk.py: main names the where-precedence level comparison (feat(program): a where predicate's operands are walked with where_children, as an expression's are with children #416/chore: the separability walk leaves program.py, and every table keyed by an operator is pinned to the closed set #418) where this branch still wrote the literal 2; the three lookup predicates keep main's name and this branch's node.keys arguments.
• tests/test_lowering.py: test_a_binary_variable_lowers_to_a_vtype is test_a_binary_variable_lowers_to_a_binary_domain on main (refactor(program): a variable's domain is called domain in the program, as the file calls it #427); the new two-key lowering test sits beside it under main's name.
• docs/reference/notation.md and tests/typesetting/golden/markdown.out are generated, so they were regenerated.

The model at the top loads through to_spec and to_program on the branch, and the frames in its comments are the ones the program reports — unchanged by the rebase, and re-checked by the suite above.

Why

Bryn's complaint in #7 was that lookups: mixed what could act as a dimension with what could not; #422 removes the second kind. coroa's comment shows the remaining gap is not a second block but a second key: the same map varies along more than one dimension and can be walked along either. The dot says which, once, at the call, and every rule above follows from that one sentence.

🤖 Generated with Claude Code

https://claude.ai/code/session_014FHQEdh9u2ppraSniF3iJd

[read-the-docs-community]

Documentation build overview

📚 math-spec | 🛠️ Build #34487039 | 📁 Comparing 2d8735a against latest (62564a2)

  🔍 Preview build  

43 files changed · + 6 added · ± 34 modified · - 3 deleted

+ Added

• about/limits/index.html
• about/what-counts-as-public-api/index.html
• howto/check/index.html
• howto/installation/index.html
• howto/print/index.html
• howto/regimes/index.html

± Modified

• index.html
• CHANGELOG/index.html
• contributing/index.html
• examples/index.html
• about/what-counts-as-language/index.html
• examples/commitment/index.html
• examples/dispatch/index.html
• examples/operators/index.html
• examples/pypsa/index.html
• examples/pypsa_linearized_uc/index.html
• and 24 more...

- Deleted

• installation/index.html
• about/ceiling/index.html
• about/what-counts-as-a-verb/index.html

[FBumann]

Prompt: "#437 was merged / Close all related issues and stale PRs"

Note

The following content was generated by AI.

Closed: #275 is decided, and #437 merged as 3284926, released in v0.0.0-alpha.89.

This PR's design is the one #437 was built out of, and its own body says so: "#433 answered coroa's design by keeping the lookup a function and letting the call pick which key it walks. The objection was that a table has no direction until something walks it, and the direction belongs to the verb." #437 takes that one step further — the declaration keeps only the claim the data can be held to (key:), and every direction is the call's.

What carried over: keys as the cardinality claim, walking a table from either key column, and the bind check per key tuple. What did not: the dot at the call site, which became over= and into= naming columns, and the lookup staying a function, which a keyless relation is not.

Capability rows where the two differed, from #437's comparison: several value columns walked to one, landing on a product of two value columns, reading a two-column slot at once, removing two key columns at once, a line's two ends in one table, a calendar table partitioning at three levels, and a masked sum where the produced dimension is already carried.

#436 built the self-map on top of this branch and is closed with it; #424 is answered by roles instead.

---

Generated by Claude Code