feat(data): a lookup is a relation, and one short of a key it declares total is refused at bind

[FBumann]

Prompt: "Please do a PR in lpspec that implement the new mathspec feature of this PR stack energy-models/mathspec#443"

Note

The following content was generated by AI. The consumer side of
energy-models/mathspec#443 and the #437 it is stacked on, pinned to
cb38f25 — #443's commit rebased onto #437's current head.

What this changes

A lookup is a relation — a table with one column per dimension it relates —
and the two claims a declaration makes about it are now held where the data is:
key: says a row is identified by those columns, and coverage: says whether
every key tuple has one, which is #443's feature and the reason a generator
lost in preparation is now named rather than dropping its terms in the join that
places them.

lookup 'gen_bus' is total, so every 'generator' has a row, and 1 do not: generator='g3'. Add their rows, or declare coverage: masked on the lookup if a key with no row is meant — under masked a walk through it places that key's terms in no group, where total says the gap is a hole in the data.

The relational engine walks the table as the call names it. One remap
serves both verbs, because the declaration fixes no direction and the Walk
carries the one the call made: the dims it consumes, the dims it produces, and
the key columns it joins on all become join keys of one inner join. Five shapes
the engine could not build before now build, each with the answer worked out by
hand in tests/test_lookups.py:

shape	the model	the optimum
a composite key, walked from either column	sum(p, by=zone_of, from=generator, into=zone)	12.0, and 18.0 from from=period
two value columns landed on their product	sum(p, by=gen_bt, into=[bus, technology])	9.0
a bare relation, both ends named	sum(p, by=connection, from=generator, into=bus)	4.0
a self-map	p == at(p, by=rep_of)	10.0
a produced dim the operand carries	sum(load * p, by=gen_bus), a masked sum	4.0

A partition keyed by two columns joins on the one it does not walk, so a
per-plant calendar ranks each plant's days apart.

Three walks part the lanes, and the linopy lane says so before linopy is
asked. It holds a lookup as a dense array over the dimension its key is over,
which is what .groupby() and .sel() read — so a composite key, a bare
relation and a self-map have no array, and each raises LaneError naming which
and pointing at the engine that does build it. A bare relation a where tests
is refused the same way, and a grouped position() is held to the same rule.
docs/about/linopy.md gains that as the lane's third wall, beside the objective
constant and the relational lane's own.

What moved in the corpus. Every model migrates once —
{over: generator, into: bus} becomes {over: [generator, bus], key: generator}
— across 29 YAML files under examples/ and bench/, the 16 generated
differential/ rungs, and the test fixtures. Two declarations in examples/
and 22 in tests/ gained coverage: masked, because in each the gap is the
subject: reserves' two open line ends, and the fixtures for a dangling at,
a partitioned shift, a per-group width and a partial group.

The PyPSA parity gate read the corpus ref with grep -o 'math-spec@v[^"]*',
so a pin at a commit — what a stacked language PR is pinned to before it has a
tag — read as empty and actions/checkout quietly took math-spec's default
branch: a corpus the pinned language does not accept. The step now takes any ref
and fails when it finds none. differential/pypsa/rungs/ and its tables are the
runner's own output and were regenerated; a lookup's tidy table is now named
after the column the file declares rather than after the lookup.

Why the pin is a rebase of #443 rather than its head

#443's commit sat on the #437 branch as it stood before math-spec's
v0.0.0-alpha.85, while #437 itself had since been rebased onto main. Pinning
the stale head meant regenerating every gallery page with a typesetter that
predates math-spec#446, which reverted GitHub's math delimiters across 62
pages — 1276 lines of the diff — and reverted rung 10's description to the
prose math-spec#442 had rewritten. Neither was this change's to make.

So #443's one commit is rebased onto #437's current head as cb38f25, and the
pin follows it. The diff fell from 182 files (+5097/−5435) to 163
(+3012/−1838), and docs/ alone from +2755/−3980 to +672/−381.

The rebase brings math-spec#444 with it: a grouped position carries the Walk
it counts within rather than naming its pieces, so predicates.py reads
partition straight and the walk it used to rebuild is deleted.

cb38f25 is pushed to energy-models/math-spec on
claude/mathspec-feature-lpspec-bcgaw5; #443's own branch still points at the
pre-rebase 31a6595 and wants a force-push to cb38f25 to show it.

Every guard deleted in turn

Re-taken on this head after the merge and the rebase, through tools/mutate.py
where it could be expressed as a deletion and by hand otherwise, with the same
three precautions — git checkout --, __pycache__ dropped on both sides, the
tree asserted clean after. The baseline is 10 failed, 3595 passed: the ten
are gurobi and xpress tests that do not skip when the solver package is
absent, and they fail identically on origin/main in this environment.

mutation	result
the coverage: total guard (sources.py:288-289)	caught
the one-row-per-key guard (sources.py:287)	caught
the column-holds-labels guard (sources.py:284-285)	caught
the null-row guard (sources.py:387-395)	caught
the linopy-lane walk refusal (builder.py:149-151)	caught
the bare-relation-in-a-where refusal (builder.py:152-154)	caught
a produced dim the operand carries is landed rather than joined on	caught — 11 failed
a walk stops joining on the key columns it does not walk	caught — 12 failed
a partition ranks without the key columns it does not walk	caught — 11 failed

The last three are hand-taken: each is a changed expression rather than a
deletable statement, so tools/mutate.py cannot express them.

Verified

uv venv on Python 3.12.3 with the repo's own pins; pixi is not installed here,
so each gate was run as the task defines it.

• pytest -q -n auto: 10 failed, 3595 passed, 328 skipped, 1 xfailed. The
  ten are the environment's, as above and unchanged from origin/main.
• ruff check ., ruff format --check ., pyrefly check: clean.
• The PyPSA parity gate, reproduced locally in the workflow's own
  environment (pypsa==1.3.0, highspy==1.15.1) against the corpus at the
  pinned commit: every rung matches PyPSA as deep as the engines allow, and a
  second run confirms the gate's own git diff --exit-code -- differential/pypsa
  rewrites nothing.
• On the rebased math-spec commit itself: its suite 1271 passed, 6 skipped;
  ruff, pyrefly clean; all six generators re-run with no drift;
  mkdocs build --strict clean once the python.org inventory is dropped, which
  the proxy answers 403.
• pytest bench/test_harness.py -q --i-know-another-is-running: 127 passed,
  9 skipped. The override is because the harness refuses a loaded machine and
  no number here is a measurement — this is the harness's own correctness.
• tools.gallery_math and tools.ladder re-run and committed; the walkthrough
  golden regenerated (VariableDeclaration.variable_type is domain now).
• CI is green on this head, parity included — the first run that resolves the
  corpus at a commit ref through the fix above.

No benchmark was taken and none is claimed.

What this deliberately does not do

• The linopy lane is not taught the three walks. Making it read a relation
  would mean a join in xarray, which is the thing this lane is a lane instead
  of. Both the wall and the route around it are stated, tested and documented.
• No number is measured. The walk is one inner join where it was one inner
  join, so nothing here claims a cost, and bench/ is untouched beyond the
  model's own declaration. CodSpeed agrees: 16 untouched benchmarks.
• tests/test_label_coords.py is gone, and tests/test_lookups.py replaces
  it: its subject was the split between a targeted and a label-space lookup,
  and math-spec#422 removed that kind. Everything still true moved over — the
  flat namespace, the supplied-relation contract, the where cases, the
  ordering and containment rules, A coordinate targeting an unused dimension is refused with a false reason: 'no coordinate set to check against' #488's column over a dimension nothing spans,
  two lookups between one pair of dimensions, and the positional-parameter index
  order — and the coverage and walk sections are new.
• One deleted guard has no replacement: GroupSum no longer carries two
  tuples that could disagree, so the zip(strict=True) that caught a
  hand-built mismatch is unreachable, and
  test_a_hand_built_node_whose_tuples_disagree_is_refused went with it.
• tests/test_assertions.py's ratchet falls from 209 to 204.

🤖 Generated with Claude Code

https://claude.ai/code/session_01SrEY2ZhFD1YXamRn4FzpMa

[codspeed]

Merging this PR will not alter performance

✅ 16 untouched benchmarks
⏩ 42 skipped benchmarks¹

---

_{Comparing claude/mathspec-feature-lpspec-bcgaw5 (b561077) with main (005b106)²}

Footnotes

1. 42 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

2. No successful run was found on main (4d71a85) during the generation of this report, so 005b106 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[read-the-docs-community]

Documentation build overview

📚 lpspec | 🛠️ Build #34494243 | 📁 Comparing b561077 against latest (45842b7)

  🔍 Preview build  

70 files changed · ± 70 modified

± Modified

• index.html
• about/index.html
• examples/index.html
• guide/index.html
• interactive/index.html
• about/architecture/index.html
• about/linopy/index.html
• examples/dispatch/index.html
• examples/facility_location/index.html
• examples/genx_piecewise_fuel/index.html
• and 60 more...

[FBumann]

Note

The following content was generated by AI.

Closing: the language half of this is in main — #1646 took math-spec's relations: with columns:/key:, which is the squashed form of this branch's own stack under a different spelling. The engine half lands in two pieces: a key of several columns is #1650, and what is left — a relation whose key determines several columns, and a bare relation — is filed as #1651.

Re-filed rather than annotated: this branch implements against program.lookups and from=/to=, which the rename retired, so merging main into it reconciles two implementations of one feature across 122 files rather than updating one.

Coverage that moved: the keyed-coverage commit here (cb38f25) is on math-spec's claude/lookup-coverage, which #1516 now pins.

---

Generated by Claude Code