Skip to content

feat(data): a relation may be keyed by several columns, so a generator's zone can change by period - #1650

Closed
FBumann wants to merge 2 commits into
mainfrom
feat/relation-keyed-by-several-columns
Closed

FBumann wants to merge 2 commits into
mainfrom
feat/relation-keyed-by-several-columns

Conversation

@FBumann

@FBumann FBumann commented Sep 15, 2026 •

Copy link
Copy Markdown
Collaborator

Prompt: the engine work (widen refusal, then attaching/compiler/reindex/predicates for a multi-column key), using #1567 and #1575 as the reference and closing them

Note

The following content was generated by AI.

The single-valued map is now one value column per key, whatever the key names. sum(by=) groups per condition, at(by=) reads back per condition, and a where: tests the value at every key dimension. What #1567 asked for — a generator's zone that changes by period — is one relation now, not a relation per period.

The language already lowered it (math-spec#437's Walk carries joined roles); lanes.lowered refused it. Refused still: a relation with no key, a relation whose key determines two columns, and a partitioned shift/sum_back/position grouped by a conditioned map — the last of which the language admits, so its refusal says the limit is this package's group table and names the walked dimension in the rewrite. #1651 carries all three.

What moved
  • relations.py — key_role/key_dim become key_roles/key_dims; refusal turns away only a keyless relation and one with several value columns, and gains the partition rule above, which needs a walk over the plan (_partitioning).
  • sources.py — a relation arrives with one column per key column; the null, duplicate and stray-label checks are per key tuple and per key dimension.
  • compiler.py — _mapping takes the node's walks and names every column by its dimension; _remap_fragment joins on the consumed dims and the joined ones. _empty_groups and the pullback's presences carry the condition too.
  • predicates.py — a relation comparison joins on every key dimension.
  • the eager lane — one array per relation over its key's product (loader.py), so dim_coords becomes a flat relations map; operator_grouped_sum groups by (value, condition) and masks the operand where a pair is unmapped rather than dropping a label; operator_at fills and masks back for the same reason.
Mutation table

Baseline for this environment: 10 failures, all gurobi/xpress not installed.

mutation result
the partition-by-a-conditioned-map refusal (relations.py:47-50) caught
the single-value walk guard (compiler.py:700-704) caught
the relational join drops the condition (compiler.py:834) caught — 13 failed, 3 beyond the baseline
the eager group keys drop the condition (operators.py:99) caught — 12 failed, 2 beyond the baseline

The first two through python -m tools.mutate, which reported the tree clean afterwards; the last two by hand (a changed line, which the tool does not express), each with the same three precautions — committed tree, git checkout --, __pycache__ dropped on both sides.

Verified

ruff check · ruff format --check · pyrefly check · pytest -q -n 4 — the suite reaches the same 10 failures as main in this environment, all of them a solver that is not installed (gurobi, xpress). Run in a uv venv rather than pixi (no pixi here).

Not run: pixi run test-floors (the bare-install shape) and test-bench — no pixi environment here.

New coverage in tests/test_conditioned_relations.py: the grouping, an unmapped pair, the pullback, the where, both refusals, and three door checks — the first four differential, the first also against the LP file.

Coverage that moved: test_label_coords.py's a-key-of-two-columns-joins-on-one-the-walk-does-not-trade was a refusal case and is now the feature, so it is deleted and its shape is the new module's subject. test_sum_by_relations.test_a_hand_built_walk_onto_two_columns_is_refused now meets a named assertion rather than a zip crash.

What I did not do

🤖 Generated with Claude Code

https://claude.ai/code/session_014pdMgEVkGBfh1f2AyiCSKA

…r's zone can change by period

The single-valued map is now one value column per key, whatever the key
names: `sum(by=)` groups per condition, `at(by=)` reads back per condition,
and a `where:` tests the value at every key dimension. A relation the key
does not determine a single column of, and a partitioned shift, sum_back or
position grouped by a conditioned map, are still refused at the door.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014pdMgEVkGBfh1f2AyiCSKA
@read-the-docs-community

read-the-docs-community Bot commented Sep 15, 2026 •

Copy link
Copy Markdown

Documentation build overview

📚 lpspec | 🛠️ Build #34583566 | 📁 Comparing 827010a against latest (28cceaf)

  🔍 Preview build  

2 files changed
± about/architecture/index.html
± reference/data/index.html

@codspeed

codspeed Bot commented Sep 15, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 24 untouched benchmarks
⏩ 58 skipped benchmarks1


Comparing feat/relation-keyed-by-several-columns (827010a) with main (1fd8926)2

Open in CodSpeed

Footnotes

  1. 58 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. No successful run was found on main (28cceaf) during the generation of this report, so 1fd8926 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

The language admits a partition through a map keyed on more than the
dimension it walks — the walk joins on the rest of the key and the operand
carries it. What cannot build it is the group table here, and the sentence
now says so and names the walked dimension in its rewrite.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014pdMgEVkGBfh1f2AyiCSKA
FBumann added a commit that referenced this pull request Sep 16, 2026
…#1657)

> **Prompt:** Then let's fix the bug first

> [!NOTE]
> The following content was generated by AI.

Closes #1652. A self-map — `rep_of: {columns: {snapshot: snapshot, rep:
snapshot}, key: snapshot}`, the language's own example of the form —
passed `check` and then raised: polars' `DuplicateError` relationally,
linopy's coordinate mismatch eagerly, and on `sum(by=)` the two lanes
disagreed, one raising where the other built.

The walk named its value column after the dimension it lands on, which
for a self-map is the name the key column already holds. The mapping
table names its columns **by side** now — `__walk key__`, `__walk value
{relation}__` — and the join names its sides (`left_on` / `right_on`),
so the landing is aliased to its dimension at the end rather than
colliding at the start. On the eager lane the pullback puts the fine
labels back as the coordinate: a row lands at the snapshot that *read* a
value, not at the one it read.

<details><summary>What the probe says, before and after</summary>

| | `lps.check` | relational | eager |
| --- | --- | --- | --- |
| `at(price, by=rep_of, over=rep)` before | passes | `DuplicateError` |
`ValueError: Coordinate mismatch` |
| `sum(p, by=rep_of, over=snapshot, into=rep)` before | passes |
`DuplicateError` | builds, unchecked |
| both, after | passes | builds | builds, and the two agree |

</details>

<details><summary>Mutation table</summary>

Baseline for this environment: 10 failures, all `gurobi`/`xpress` not
installed.

| mutation | result |
|---|---|
| the one-value-column-per-relation guard (`compiler.py:703-706`) |
**caught** |
| the walk selects its landing by the dimension's own name
(`compiler.py:845`) | **caught** — 218 failed |
| the eager pullback keeps the labels it read (`operators.py:120`) |
**caught** — 11 failed, 1 beyond the baseline |
| the join renames the mapping onto the fragment's dims instead of
naming both sides (`compiler.py:846`) | **not caught** — recorded rather
than hidden: it is an equivalent implementation, since what the bug
needed was the *landing* to keep a name of its own, which the row above
is the guard for |

The first through `python -m tools.mutate`, which reported the tree
clean afterwards; the rest by hand (changed lines, which the tool does
not express), each with the same three precautions.

</details>

<details><summary>Verified</summary>

`ruff check` · `ruff format --check` · `pyrefly check` · `pytest -q -n
4` — the suite reaches the same 10 failures as `main` in this
environment, all a solver that is not installed. Run in a `uv` venv
rather than pixi (no pixi here).

**Not run**: `pixi run test-floors` and `test-bench` — no pixi
environment here.

The bug was reproduced first: `tests/test_self_map.py` landed as two
**strict xfails** naming #1652 (commit `0324845`), which XPASS on the
fix and lose their markers in the commit that makes them pass. What was
wrong is in their docstrings. The `where:` case and the duplicate-key
refusal pass beside them, which is what said the data path was never the
problem.

Coverage that moved:
`test_sum_by_relations.test_a_hand_built_walk_onto_two_columns_is_refused`
met a `zip` crash inside the mapping builder; the guard it probes is now
a named assertion that fires before any relation is read, and the test
matches that.

</details>

## What I did not do

- **The other three shapes stay refused** — #1654, #1655, #1656 under
#1653. This is the bug of the four; those are gaps.
- **No example or gallery page.** A self-map is exercised by the new
test module.
- **#1650 touches the same two functions.** It is not a dependency
either way — this branches off `main` so it can merge alone — but
whichever lands second wants a small merge in `_mapping` and
`operator_at`. I will do that when you merge one.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_014pdMgEVkGBfh1f2AyiCSKA

---
_Generated by [Claude
Code](https://claude.ai/code/session_014pdMgEVkGBfh1f2AyiCSKA)_

---------

Co-authored-by: Claude <noreply@anthropic.com>
FBumann added a commit that referenced this pull request Sep 16, 2026
…language admits (#1659)

> **Prompt:** "Let's catch up and support everything from mathspecs
relations feature. If the linopy lane is to hard, leave it stale. The
relational lane is what we care about. Also take the time to potentially
refactor the code about relations, as the concept has changed a bit"

> [!NOTE]
> The following content was generated by AI.

The relational lane builds every relation shape the language admits: a
key of several columns, several value columns, a bare relation, a
partition by a conditioned map, a self-map, and two columns over one
dimension. The eager lane keeps the single-valued map and refuses the
rest, naming the lane that builds it.

Closes #1654, #1655, #1656 and #1653. Supersedes #1650, whose eager-lane
half is carried here. #1657 (the self-map fix, #1652) landed on `main`
meanwhile and is merged in: the general walk carries its relational
half, and the eager lane keeps its `operator_at` fix.

<details><summary>What moved</summary>

- **The transport.** A relation is attached as the table it declares,
one column per column under its own name. `sources.py` checks every
column against its dimension, one row per key tuple (every column, for a
bare relation), and a null anywhere; `attaching.py` casts every column
to its dimension's `Enum`. The archive stores the table as supplied, so
`supplied()` no longer renames.
- **The compiler** reads the plan's `Walk`. `_walked` puts consumed and
joined columns under their dimensions and produced ones under a landing
name until the consumed column is dropped, which is what a self-map
needs. `_remap_fragment` joins on consumed, joined, and any produced
dimension the operand already carries (the masked sum). `_empty_groups`
and `_pulled_back_presences` carry the joined dimensions and several
produced ones. `partitioned` takes the `Walk` and ranks inside `(joined
dims…, group columns…)`.
- **Partitions** (`reindex.py`) key every side, edge and presence by the
dimension and the partition's joined dimensions; a per-group offset or
width is read under the group column. **Predicates** read a relation at
every key dimension, a bare existence at every column, and a grouped
position at the rest of the key.
- **`lpspec/relations.py` is deleted.** Its accessors assumed the
two-column map; the language's `Walk` carries the fact, and
`lanes.lowered` refuses no relation any more.
- **The eager lane** takes #1650's multi-key support (one array per
relation over its key's product, `groupby` on value and condition),
reads a relation's key columns by role so a self-map's two columns stay
apart, and refuses the other three shapes in `linopy/loader.py` with a
`LaneError` naming the relational lane.
- **The parity tables** (`differential/pypsa/tables`) are regenerated
with `parity.py` against the pinned corpus: every rung still matches
PyPSA, and the only change is each relation table's header, which
`tidy_sources` no longer renames.
- **Docs:** `architecture.md` (module rows, the `GroupSum` row, the
relational-lane paragraph), `reference/data.md` (relation rows),
`about/linopy.md` (a third wall between the lanes).

</details>

<details><summary>Coverage that moved</summary>

- `tests/test_relation_shapes.py` (new): every shape on the relational
lane, each optimum hand-derived and the written LP file re-solved to it;
the masked sum; a bare relation holding a pair twice. The self-map walks
live in `main`'s `test_self_map.py`, which is differential; this module
keeps the self-map partition.
- `tests/test_conditioned_relations.py` (new): #1650's differential
cases for the multi-key map, the eager lane's three refusals, and the
door checks.
- `test_label_coords.py`: the two refusal cases became "passes check"
cases.
`test_sum_by_relations.test_a_hand_built_walk_onto_two_columns_is_refused`
is deleted, the shape now building in the new module.
- Door messages reworded for the general shape, regexes moved with them
(`test_self_map.py`'s included): `maps 1 key(s) more than once:
generator='g1'`, `has value(s) in 'generator' that are not 'generator'
labels`, `null in 'bus'`, `is a relation with a column over 'g'`.

</details>

<details><summary>Mutation table</summary>

Taken with `python -m tools.mutate` on 419870d, before the merge; the
tree came back clean. The two-columns-over-one-dimension guard was
retired by the merge, the eager lane building the shape since #1657.

| mutation | result |
|---|---|
| the eager lane refuses a bare relation (`loader.py:40-41`) |
**caught** |
| the eager lane refuses a key determining several columns
(`loader.py:42-48`) | **caught** |
| the eager lane refuses a partition by a conditioned map
(`loader.py:53-60`) | **caught** |
| a relation holding a row twice is refused (`sources.py:341-352`) |
**caught** |

</details>

<details><summary>Verified</summary>

`ruff check` · `ruff format --check` · `pyrefly check` · `pytest -q -n
4` on the merged head 4d133ea: the suite reaches the same 10 failures
as `main` (3e825f0) in this environment, all of them a solver that is
not installed (`gurobi`, `xpress`). `differential/pypsa/parity.py`
against math-spec at v0.0.0-alpha.89: every rung matches. Run in a `uv`
venv with the `linopy` extra rather than pixi (no pixi here).

**Not run:** `docs-build` and `test-floors`, no pixi environment here.

</details>

## What I did not do

- **The eager lane is left at the single-valued map**, as asked. The
three other shapes are refused there, not built; their tests use the LP
file as the second opinion.
- **No new example or gallery page.** A page for a line-ends table is a
`docs:` PR of its own.
- **Two lanes still each answer "which dimensions do a node's walks join
on"** (`compiler.joined_dims`, `builder._joined_dims`).
energy-models/mathspec#488 puts `joined` on `GroupSum` and `At`; the
two copies go with the pin bump once it releases.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01DT1r52zPp9BdnrzFdp1xoY

---------

Co-authored-by: Claude <noreply@anthropic.com>
@FBumann

FBumann commented Sep 16, 2026

Copy link
Copy Markdown
Collaborator Author

Superseeded

@FBumann FBumann closed this Sep 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants